CPIO(5)

💖 NAME 💖

okay so bestie, welcome to cpio(5), the File Formats Manual page, and it's literally all about the format of cpio archive files. yes the whole page. it's a big one. hydrate first. 🥤

---

🌈 DESCRIPTION 🌈

the cpio archive format is out here collecting any number of files, directories, and other file system objects — we're talking symbolic links, device nodes, the whole friend group — and packing them into a single stream of bytes. one stream. iconic behavior. ✨

🦋 General Format 🦋

so every file system object inside a cpio archive is made of three parts, in order:

1. a header record with basic numeric metadata,
2. the full pathname of the entry,
3. the file data.

the header record stores a whole series of integer values that generally follow the fields in struct stat. (go peek at stat(2) for the details, iykyk.) the variants of this format differ primarily in how they store those integers — binary, octal, or hexadecimal, pick your fighter. 💅

then after the header comes the pathname of the entry (the length of that pathname is stored in the header, so it's self-describing, we love that), and then any file data.

and the end of the archive? marked by a special record whose pathname is TRAILER!!!. that's it, that's the goodbye. 👋

---

🎀 PWB format 🎀

the PWB binary cpio format is the original format — it showed up when cpio was first introduced as part of the Programmer's Work Bench system, which was a variant of 6th Edition UNIX. it stores numbers as 2-byte and 4-byte binary values.

each entry begins with a header in this exact format:

struct header_pwb_cpio {
        short   h_magic;
        short   h_dev;
        short   h_ino;
        short   h_mode;
        short   h_uid;
        short   h_gid;
        short   h_nlink;
        short   h_majmin;
        long    h_mtime;
        short   h_namesize;
        long    h_filesize;
};

the short fields here are 16-bit integer values, while the long fields are 32 bit integers. and since PWB UNIX — like the 6th Edition UNIX it was based on — only ever ran on PDP-11 computers, they're in PDP-endian format: little-endian shorts, big-endian longs. so the long integer whose hexadecimal representation is 0x12345678 would be stored in four successive bytes as 0x34, 0x12, 0x78, 0x56. yes that's chaotic. no we're not fixing it. it's historical. 🫠

the fields are as follows:

• h_magic — the integer value octal 070707.
• h_dev, h_ino — the device and inode numbers from the disk. programs that read cpio archives use these to figure out when two entries refer to the same file. and if you're a program that synthesizes cpio archives, be careful bestie: set these to distinct values for each entry. ✍️
• h_mode — the mode specifies both the regular permissions and the file type, and it also holds a couple of bits that are straight-up irrelevant to the cpio format, because this field is actually a raw copy of the mode field in the inode representing the file. those extra bits are the IALLOC flag (shows the inode entry is in use) and the ILARG flag (shows the file is large enough to have indirect block pointers in the inode). the mode is decoded like this:
  • 0100000 — IALLOC flag — irrelevant to cpio.
  • 0060000 — this masks the file type bits.
  • 0040000 — file type value for directories.
  • 0020000 — file type value for character special devices.
  • 0060000 — file type value for block special devices.
  • 0010000 — ILARG flag — irrelevant to cpio.
  • 0004000 — SUID bit.
  • 0002000 — SGID bit.
  • 0001000 — Sticky bit.
  • 0000777 — the lower 9 bits specify read/write/execute permissions for world, group, and user following standard POSIX conventions.
• h_uid, h_gid — the numeric user id and group id of the owner.
• h_nlink — the number of links to this file. directories always have a value of at least two here. and heads up: hardlinked files include file data with every copy in the archive. so yeah, that data is duplicated. 😭
• h_majmin — for block special and character special entries, this field holds the associated device number, with the major number in the high byte and the minor number in the low byte. for all other entry types, writers should set it to zero and readers should just ignore it. 🙈
• h_mtime — modification time of the file, indicated as the number of seconds since the start of the epoch, 00:00:00 UTC January 1, 1970.
• h_namesize — the number of bytes in the pathname that follows the header. this count includes the trailing NUL byte.
• h_filesize — the size of the file. and ngl, this archive format is limited to 16 megabyte file sizes, because PWB UNIX (like 6th Edition) only used an unsigned 24 bit integer for the file size internally. so that's the ceiling. 🚧

the pathname immediately follows the fixed header. if h_namesize is odd, an additional NUL byte is added after the pathname. then the file data gets appended, again with an additional NUL appended if needed to get the next header at an even offset. alignment queen. 💅

hardlinked files are not given special treatment here — the full file contents are included with each copy of the file.

---

🔥 New Binary Format 🔥

the new binary cpio format showed up when cpio was adopted into late 7th Edition UNIX. it is exactly like the PWB binary format described above, except for three changes:

First, UNIX now ran on more than one hardware type, so the endianness of 16 bit integers must be determined by observing the magic number at the start of the header. the 32 bit integers are still always stored with the most significant word first, though — so each of those two, in the struct shown above, was stored as an array of two 16 bit integers, in the traditional order. and those 16 bit integers, like all the others in the struct, were accessed using a macro that byte swapped them if necessary.

Next, 7th Edition had more file types to store, and the IALLOC and ILARG flag bits were re-purposed to accommodate them. the revised use of the various bits is as follows:

• 0170000 — this masks the file type bits.
• 0140000 — file type value for sockets.
• 0120000 — file type value for symbolic links. for symbolic links, the link body is stored as file data.
• 0100000 — file type value for regular files.
• 0060000 — file type value for block special devices.
• 0040000 — file type value for directories.
• 0020000 — file type value for character special devices.
• 0010000 — file type value for named pipes or FIFOs.
• 0004000 — SUID bit.
• 0002000 — SGID bit.
• 0001000 — Sticky bit.
• 0000777 — the lower 9 bits specify read/write/execute permissions for world, group, and user following standard POSIX conventions.

Finally, the file size field now represents a signed 32 bit integer in the underlying file system, so the maximum file size has increased to 2 gigabytes. glow up. ⭐

⚠️ okay so here's the tea: there is no obvious way to tell which of the two binary formats an archive uses, other than to see which one makes more sense. the typical error scenario is that a PWB format archive unpacked as if it were in the new format will create named sockets instead of directories, and then fail to unpack files that should go in those directories. 💀

running bsdcpio -itv on an unknown archive will make it obvious which one it is: if it's PWB format, directories will be listed with an s instead of a d as the first character of the mode string, and the larger files will have a ? in that position. mystery solved, detective. 🔍

---

🌸 Portable ASCII Format 🌸

version 2 of the Single UNIX Specification (SUSv2) standardized an ASCII variant that is portable across all platforms. it's commonly known as the “old character” format or as the odc format. it stores the same numeric fields as the old binary format, but represents them as 6-character or 11-character octal values.

struct cpio_odc_header {
        char    c_magic[6];
        char    c_dev[6];
        char    c_ino[6];
        char    c_mode[6];
        char    c_uid[6];
        char    c_gid[6];
        char    c_nlink[6];
        char    c_rdev[6];
        char    c_mtime[11];
        char    c_namesize[6];
        char    c_filesize[11];
};

the fields are identical to those in the new binary format. the name and file body follow the fixed header.

and here's the cute part — unlike the binary formats, there is no additional padding after the pathname or file contents. so if the files being archived are themselves entirely ASCII, then the resulting archive will be entirely ASCII, except for the NUL byte that terminates the name field. pure. clean. ASCII-coded. ✨

---

🧚 New ASCII Format 🧚

the "new" ASCII format uses 8-byte hexadecimal fields for all numbers and separates device numbers into separate fields for major and minor numbers.

struct cpio_newc_header {
        char    c_magic[6];
        char    c_ino[8];
        char    c_mode[8];
        char    c_uid[8];
        char    c_gid[8];
        char    c_nlink[8];
        char    c_mtime[8];
        char    c_filesize[8];
        char    c_devmajor[8];
        char    c_devminor[8];
        char    c_rdevmajor[8];
        char    c_rdevminor[8];
        char    c_namesize[8];
        char    c_check[8];
};

except as specified below, the fields here match those specified for the new binary format above.

• magic — the string 070701.
• check — this field is always set to zero by writers and ignored by readers. see the next section for more details. 👀

the pathname is followed by NUL bytes so that the total size of the fixed header plus pathname is a multiple of four. likewise, the file data is padded to a multiple of four bytes. and note: this format supports only 4 gigabyte files — unlike the older ASCII format, which supports 8 gigabyte files. 📏

in this format, hardlinked files are handled by setting the filesize to zero for each entry except the first one that appears in the archive. first one carries the data, everyone else just vibes. 🫶

---

💜 New CRC Format 💜

the CRC format is identical to the new ASCII format described in the previous section, except that the magic field is set to 070702 and the check field is set to the sum of all bytes in the file data.

this sum is computed treating all bytes as unsigned values and using unsigned arithmetic. only the least-significant 32 bits of the sum are stored. so if your sum is bigger than that, it gets trimmed. snip snip. ✂️

---

🍓 HP variants 🍓

the cpio implementation distributed with HPUX used XXXX but stored device numbers differently XXX.

---

💀 Other Extensions and Variants 💀

Sun Solaris uses additional file types to store extended file data — including ACLs and extended attributes — as special entries in cpio archives.

XXX Others? XXX

---

👑 SEE ALSO 👑

• cpio(1)
• tar(5)

✨ STANDARDS ✨

Okay so bestie, here's the tea on where cpio actually stands in the standards world: it's not part of POSIX anymore, and it's not part of the Single Unix Standard either. 😭 It last showed up in Version 2 of the Single UNIX Specification (aka “SUSv2”), and then subsequent standards were like "we're replacing this" and brought in pax(1) instead. So now pax(1) is the main character, and the portable ASCII format is currently part of the spec for the pax(1) utility. The way cpio got gently nudged out of the group chat… 💀

---

🌸 HISTORY 🌸

Story time, because this is genuinely iconic. 🫶

• The original cpio utility was written by Dick Haight while he was working in AT&T's Unix Support Group. 👑
• It appeared in 1977 as part of PWB/UNIX 1.0 — that's the “Programmer's Work Bench”, which was derived from Version 6 AT&T UNIX and used internally at AT&T. No cap, internal-only legend behaviour.
• By 1980, both the new binary and old character formats were in use — we know this from the System III source released by SCO under their “Ancient Unix” license. 🧚
• The character format got adopted as part of IEEE Std 1003.1-1988 (“POSIX.1”). Slay. ✨

XXX when did "newc" appear? Who invented it? When did HP come out with their variant? When did Sun introduce ACLs and extended attributes? XXX

(The original page literally left those questions in there, and honestly? Same. The mystery is part of the vibe. 🙈)

---

💀 BUGS 💀

Okay so here's where we keep it 100, because there are some real ones:

• The “CRC” format is mis-named — it actually uses a simple checksum, not a cyclic redundancy check. The name is giving what it is not, fr fr. 😭
• The binary formats are limited to 16 bits for user id, group id, device, and inode numbers. They're limited to 16 megabyte and 2 gigabyte file sizes for the older and newer variants, respectively.
• The old ASCII format is limited to 18 bits for the user id, group id, device, and inode numbers. It's limited to 8 gigabyte file sizes.
• The new ASCII format is limited to 4 gigabyte file sizes.
• None of the cpio formats store user or group names — which is essential when you're moving files between systems with dissimilar user or group numbering. 🫠
• Especially when writing older cpio variants, it may be necessary to map actual device/inode values to synthesized values that fit the available fields. And with very large filesystems, this may be necessary even for the newer formats. So yeah, sometimes you gotta fake the numbers to make the fields fit. It's giving "make it work" energy. 💅

---

🌈 and that's the whole page, bestie! this lil manual page came from the Debian crew, dated December 23, 2011, and it's CPIO(5) — section 5, the file formats one. go forth and archive responsibly. mwah! 💖🎀🦋