XPAR(1)

📦 version 1.1, in alpine 3.24, nixos 26.05, nixpkgs unstable

💖 XPAR(1) 💖

okay so bestie, this is xpar(1) out of the General Commands Manual, and I'm gonna walk you through literally every single bit of it — no skips, no cap, we're reading the whole thing together 🫶

🌸 NAME 🌸

xpar — an error/erasure code system guarding data integrity ✨

🦋 SYNOPSIS 🦋

xpar [ -Je / -Jd ] [ -hfvqVc ] [ -i/--interlacing # ] [ -H/--integrity=<algo> ]
[ --auth=<keyfile> ] [ --no-mmap ] [ names ...  ]
xpar [ -Jse / -Jsd ] [ -hfvqVc ] [ --no-mmap ] [ names ...  ]
xpar [ -We / -Le ] [ -hfvqV ] [ --out-prefix # , --dshards # ,
--pshards # ] [ -H/--integrity=<algo> ] [ --auth=<keyfile> ] [ --no-
mmap ] [ names ...  ]
xpar [ -Wd / -Ld / -Wt / -Lt ] [ -hfvqV ] [ --auth=<keyfile> ] [ names
...  ]

💜 DESCRIPTION 💜

The xpar command is the command-line interface to an error/erasure code system which fortifies data and corrects errors in the data or reconstructs missing shards. like, it's the bodyguard for your files 💅

🎀 joint mode, the classic

xpar in joint encoding mode ( -Je ) generates a slightly inflated (by about 12%) parity-guarded file from a given data file with the extension .xpar data is corrupted. xpar internally uses a (255,223)-RS code over an 8-bit Galois field. At most two filenames intertwined with flags are expected as arguments.

xpar will by default not overwrite existing files — if this behaviour is intended, use the -f option. If no file names are specified, xpar will process data from the standard input and write it to the standard output, refusing to output binary data to a terminal 🙈. If two files are specified, the first one is used in place of standard input, and the second one is used in place of standard output. If the -c flag is present, xpar will read from the specified file and output data to standard output instead. Otherwise, if decoding, xpar will try to guess the decompressed filename by removing the .xpar extension — if not present, an error will be reported. If encoding, the output filename will be generated by appending the .xpar extension to the input filename.

✨ interlacing, aka the burst-error slayer

In joint mode, the flag -i/--interlacing can be used to specify the interlacing factor. This does not affect the nominal error correction capability (in the sense of a noisy memory-less channel model), but can be used to increase the robustness against burst errors. The default interlacing factor is 1 and the maximum correctible burst error length is 16 (as given by the Singleton bound). The interlacing factor of 2 transposes 65025-byte blocks of data and hence can correct burst errors of length 4080. The interlacing factor of 3 transposes approx. 16MB blocks of data and can correct burst errors of length approx. 1MB.

In theory, with the interlacing factor of one, up to 32 burst errors could be corrected — this problem is however NP-hard and reduces to polynomial reconstruction (the basis of e.g. Shamir's secret sharing scheme). As an added benefit, higher interlacing factors tend to result in faster processing (up to 50%) as the workload is more parallelisable 🔥

🧚 systematic mode, the glow-up

The joint-mode flag -s/--systematic switches to a systematic output layout. The encoder writes only the 32 parity bytes per 223-byte input block (plus an 8-byte block header) into the .xpa file; the original data file is not copied and remains on disk under its original name, directly accessible without any decoding step. The .xpa is consequently around 18% of the input size rather than 114%. slay 💅

On decode, xpar reads the (possibly corrupted) original as the first positional argument and the parity .xpa as the second (inferred as original.xpa when only one argument is given), and writes the corrected data to standard output.

⚠️ Systematic mode is incompatible with --interlacing: each 223-byte block is handled independently and can correct up to 16 byte errors. A file produced with -s cannot be decoded without -s, and vice versa; xpar detects the mismatch and refuses with a clear error. periodt.

💀 the joint-mode file format (v1.0 core lore)

The joint-mode file format binds each lace to its position via a sequence number in the block header and records the total input size in the file header. Truncation at a lace boundary and lace reordering are both detected on decode and rejected with a clear error 👑

By default, the per-lace integrity tag is a 32-bit CRC32C. The -H/--integrity flag selects the tag algorithm:

  • crc32c (default)
  • blake2b (128-bit BLAKE2b tag, giving 2^64 birthday-bound collision resistance instead of CRC32C's 2^16)

BLAKE2b mode grows the per-lace block header from 12 to 24 bytes; at interlacing factor 1 this raises the .xpa overhead from ~18% to ~24%; at factors 2 and 3 the change is negligible (one header per 65 kB or 16 MB respectively).

The -H/--integrity flag picks the per-lace / per-shard integrity tag: the default crc32c is a 32-bit hardware-accelerated CRC, and blake2b produces an unkeyed 128-bit BLAKE2b hash instead. In sharded modes this grows each shard header from 20 to 32 bytes.

🔐 --auth, for the girlies who don't trust anyone

The --auth=<keyfile> flag reads a 1–64-byte key from keyfile and switches the integrity tag to a keyed BLAKE2b-128 MAC. It implies -H blake2b. An encode with --auth can only be decoded with the same key; decoders presented with the wrong key, no key, or an extra key against a non-keyed file all refuse 😤

🔑 Keep the key file outside the .xpa (xpar never embeds it).

All three modes (CRC32C, BLAKE2b hash, BLAKE2b MAC) are supported in both joint and sharded modes, selected by the version byte of the file/shard header so decoders auto-detect 💖

🌈 compatibility tea ☕

The v1.0 file format is incompatible with v0.x tools: magic bytes and header fields moved, and a version check in the decoder rejects older archives. Re-encode anything you still need. ngl that's just how it is.

The sharded modes' per-shard header also changed at v1.0. The integrity tag (CRC32C, BLAKE2b hash, or BLAKE2b MAC) now covers the full header (version byte, shard count, shard number, total size) as well as the shard body, so a shard_number bit-flip or a header-field swap between shards is detected 🔥

🍓 sharded mode bestie

xpar in sharded encoding mode ( -We or -Le ) generates a set of parity-guarded files from a given data file. The amount of files generated is given by the sum of the mandatory parameters --dshards and --pshards — which specify the number of data and parity shards, respectively. The data shards contain the file contents verbatim ( xpar uses a systematic code). The data is reconstructible as long as there are more than --dshards shards available 👑

✨ OPTIONS ✨

  • -W, --van-sharded — Enable sharded processing mode using Vandermonde matrices in O(n^3) via the Berlekamp-Welch algorithm. Slowest sharded mode, parallelisable.
  • -L, --fft-sharded — Enable sharded processing mode using FFT in O(n log n) via the Leopard algorithm. Fastest sharded mode 🚀
  • -J, --joint — Enable joint processing mode.
  • -e, --encode — Enable encoding mode.
  • -d, --decode — Enable decoding mode.
  • -t, --test — Enable integrity-check mode. Reads the same inputs as -d and writes no output; exits non-zero if the input cannot be recovered. For joint mode, runs the full error-correction pipeline and reports any unrecoverable block or CRC mismatch. For sharded mode, validates each shard's header and CRC and verifies that enough valid shards remain to reconstruct the original. Works with -J, -Js, -W, and -L. (vibe check for your archive 🫶)
  • -h, --help — Print a help message and exit.
  • -f, --force — Force operation. Ignore errors and try to recover from them, if possible. Overwrite existing files.
  • -v, --verbose — Be verbose. Print additional information.
  • -q, --quiet — Be quiet. Suppress most output, except fatal errors.
  • -V, --version — Print version information of the package and exit.
  • -c, --stdout — Write resultant data to standard output. Joint mode only.
  • -i, --interlacing — Specify the interlacing factor. Joint mode only.
  • -s, --systematic — Joint mode only. Write a parity-only .xpa file that does not contain the original data; pair with -e to encode. On decode, pass the (possibly corrupted) original and the parity .xpa as positional arguments; corrected data is written to standard output. Incompatible with --interlacing.
  • -H, --integrity=<algo> — Joint mode only. Selects the per-lace integrity algorithm: crc32c (default, 32-bit hardware-accelerated CRC) or blake2b (128-bit BLAKE2b tag). BLAKE2b gives stronger collision resistance at a modest encode-time throughput cost and a larger per-lace block-header footprint.
  • --auth=<keyfile> — Joint mode only. Reads a 1–64-byte key from keyfile and enables keyed-MAC mode (implies -H blake2b). Encode writes a keyed BLAKE2b tag per lace; decode requires the same key and rejects any mismatch.
  • --no-mmap — Disable memory mapping. Generally results in worse performance, as the fallback method is to read the file in chunks using the standard I/O library. 😭
  • -j, --jobs — Specify the amount of CPU cores to use. Not setting this value or setting it to zero will result in the program automatically deciding the amount of cores to use. Setting it to one will disable parallel processing.
  • --out-prefix — Specify the prefix for the output files (shards). Sharded mode only.
  • --dshards — Specify the number of data shards. Sharded mode only. Mandatory.
  • --pshards — Specify the number of parity shards. Sharded mode only. Mandatory.

🐛 REPORTING BUGS 🐛

Report bugs to: k@iczelia.net or open a ticket at https://github.com/iczelia/xpar. please actually do it, we wanna know 💖

Copyright © 2022-2026 Kamila Szewczyk

Thanks to: Phil Karn (KA9Q), for the original implementation of the RS-joint code, licensed under the GPLv2-or-later license. Marshall Lochbaum for noticing an optimisation to the syndrome computaton code. Chris A. Taylor for the FFT-based Leopard algorithm. absolute icons, all of them 🫶

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to process this file through troff and print the results, provided the printed document carries copying permission notice identical to this one except for the removal of this paragraph (this paragraph not being relevant to the printed manual).

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Foundation.

and that's the whole page bestie 🎀 this was XPAR(1) from the General Commands Manual, and yeah — that's the footer, that's the vibe, go guard some data 💅✨

📦 where this page comes from