FPSYNC(1)

💖 FPSYNC(1) 💖

okay bestie, so this is fpsync, a General Commands Manual page, and honestly? she's giving parallel-sync main character energy. let's get into it 🦋

---

✨ NAME ✨

fpsync — Synchronize directories in parallel using fpart and an external tool

---

🌸 SYNOPSIS 🌸

fpsync [-p] [-n jobs] [-w wrks] [-m tool] [-T path] [-f files] [-s size]
       [-E] [-o toolopts] [-O fpartopts] [-S] [-t tmpdir] [-d shdir]
       [-M mailaddr] [-v] src_dir/ dst_url/
fpsync -l
fpsync -r runid [-R] [OPTIONS...]
fpsync -a runid
fpsync -D runid

---

💅 DESCRIPTION 💅

so fpsync synchronizes directories in parallel using fpart(1) and rsync(1), cpio(1), pax(1) or tar(1). periodt. it computes subsets of src_dir/ and then spawns jobs to synchronize them over to dst_url/. like a little crew of workers doing the most 💖

those synchronization jobs can run either locally or remotely (via SSH workers — peep option -w), and they get executed on-the-fly while the filesystem crawling is still going on. that combo is literally why fpsync is such a good tool for migrating large filesystems. she ate 🔥

---

🎀 COMMON OPTIONS 🎀

• -t tmpdir — set the fpsync temporary directory to tmpdir. this directory stays local and does not need to be shared amongst SSH workers when you're using the -w option. Default: /tmp/fpsync
• -d shdir — set the fpsync shared directory to shdir. this option is mandatory when using SSH workers, and it's set by default to tmpdir when running locally. the directory you specify must be an absolute path; it will be used to handle communications with SSH hosts (sharing partitions and log files), and as a consequence it must be made available to all participating hosts (e.g. through a r/w NFS mount) — including the master one running fpsync.
• -M mailaddr — send an e-mail to mailaddr after a run. you can specify multiple -space-separated- addresses. that option requires the mail(1) client to be installed and configured on the master host running fpsync.
• -v — Verbose mode. can be be specified several times to increase the verbosity level. yes bestie, stack 'em 😌
• -h — Print help

---

🌈 SYNCHRONIZATION OPTIONS 🌈

• -m tool — the external copy tool used to synchronize files. currently supported tools are: rsync, cpio, pax, tar, and tarify. Default: rsync. the special tool tarify generates tarballs into the destination directory 🍓
• -T path — specify the absolute path of the copy tool (guessed by default). if you force a specific path, the copy tool must be present at that path on each worker. and heads up: that path cannot be changed when resuming a run. no takebacks 💀
• -f files — transfer at most files files or directories per sync job. 0 means unlimited, but you must at least specify one file or size limit. Default: 2000
• -s size — transfer at most size bytes per sync job. 0 means unlimited, but you must at least specify one file or size limit. you can use a human-friendly unit suffix here (k, m, g, t, p). Default: 4294967296 (4 GB)
• -E — work on a per-directory basis (rsync tool only). in that mode, fpsync works with lists of directories instead of files. that mode may generate coarse-grained lists but it enables rsync(1)'s --delete option by default ( WARNING!!! ), which makes it a good candidate for a final (cleaning) pass after several incremental passes using standard (file) mode. when option -E is specified twice, it enables 'aggressive' mode, which isolates erroneous directories and enables recursive synchronization for them. this advanced mode can be useful to try to overcome transcient errors such as Linux SMB client deferring opendir() calls to support compound SMB requests. iykyk 😭
• -o toolopts — override default rsync(1), cpio(1), pax(1) or tar(1) options with toolopts. use this option with care, because certain options are incompatible with a parallel usage (e.g. rsync's --delete). Default for rsync: -lptgoD -v --numeric-ids. Empty for cpio, pax, tar and tarify.
• -O fpartopts — override default fpart(1) options with fpartopts. options and values must be separated by a pipe character. Default: -x|.zfs|-x|.snapshot*|-x|.ckpt
• -S — Sudo mode. use sudo(8) for filesystem crawling and synchronizations. she's got the keys to the kingdom 👑
• src_dir/ — Source directory. it must be absolute and available on all participating hosts (including the master one, running fpsync).
• dst_url/ — Destination directory or URL (rsync tool only). if a remote URL is provided, it must be supported by rsync(1). all participating workers must be able to reach that target.

---

🧚 JOB HANDLING AND DISPATCHING OPTIONS 🧚

• -n jobs — start jobs concurrent sync jobs (either locally or remotely, see below) per run. Default: 2
• -w wrks — use remote SSH wrks to synchronize files. synchronization jobs are executed locally when this option is not set. wrks is a space-separated list of login@machine connection strings and can be specified several times. you must be allowed to connect to those machines using a SSH key to avoid user interaction. no password-prompting drama, bestie 🙈

---

🦋 RUN HANDLING OPTIONS 🦋

• -p — Prepare mode: prepare, test the synchronization environment, start fpart(1) and create partitions, but do not actually start transfers. that mode can be used to create a run that can then be resumed using option -r.
• -l — List previous runs and their status.
• -r runid — Resume run runid and restart synchronizing the remaining partitions from a previous run. runid is displayed when using verbose mode (see option -v) or prepare mode (option -p), and can be retrieved afterwards using option -l. note that filesystem crawling is skipped when resuming a previous run. as a consequence, options -m, -f, -s, -E, -o, -O, -S, src_dir/, and dst_url/ are ignored.
• -R — Replay mode: when using option -r, force re-synchronizing the run's all partitions instead of the remaining ones only. that mode can be useful to skip filesystem crawling when you have to replay a final pass several times and you know the directory structure has not changed in the meantime (you may miss files if you use replay mode with a standard, file-based, run). so be careful, okay?
• -a runid — Archive run runid (including partition files, logs, queue and work directories) to tmpdir. that option requires the tar(1) client to be installed on the master host running fpsync.
• -D runid — Delete run runid (including partition files, logs, queue and work directories).

---

⭐ RUNNING FPSYNC ⭐

each fpsync run generates a unique runid, which is displayed in verbose mode (see option -v) and within log files. you can use that runid to resume a previous run (see option -r). fpsync will then restart synchronizing data from the parts that were being synchronized at the time it stopped. lowkey iconic ✨

this unique feature gives the administrator the ability to stop fpsync and restart it later, without having to restart the whole filesystem crawling and synchronization process. note that resuming is only possible when the filesystem crawling step has finished. so like, finish the crawl first bestie, it's a whole thing 💅

during synchronization, you can press CTRL-C to interrupt the process. the first CTRL-C prevents new synchronizations from being submitted, and the process will wait for the current synchronizations to be finished before exiting. if you press CTRL-C again, the current synchronizations will be killed and fpsync will exit immediately. when using option -E to enable directory mode and rsync's --delete option, keep in mind that killing rsync processes may lead to a situation where certain files have been updated and others not deleted yet (because the deletion process is postponed using rsync's --delete-after option). ⚠️ so messy, watch out ⚠️

on certain systems, CTRL-T can be pressed to get the status of the current and remaining parts to be synchronized. this can also be achieved by sending a SIGINFO to the fpsync process. 🌸

whether you use verbose mode or not, everything is logged within shdir/log/. everything. she's a documenter 💜

---

🍓 EXAMPLES 🍓

here are some examples:

fpsync -n 4 /usr/src/ /var/src/

• Synchronizes /usr/src/ to /var/src/ using 4 local jobs.

fpsync -n 2 -w login@machine1 -w login@machine2 -d /mnt/fpsync /mnt/src/
        /mnt/dst/

• Synchronizes /mnt/src/ to /mnt/dst/ using 2 concurrent jobs executed remotely on 2 SSH workers (machine1 and machine2). the shared directory is set to /mnt/fpsync and mounted on the machine running fpsync, as well as on machine1 and machine2. the source directory (/mnt/src/) is also available on those 3 machines, while the destination directory (/mnt/dst/) is mounted on SSH workers only (machine1 and machine2). look at her go 🔥

---

💀 LIMITATIONS 💀

parallelizing rsync(1) can make several options not usable, such as --delete. if your source directory is live while fpsync is running, you will have to delete extra files from the destination directory. this is traditionally done by using a final -offline- rsync(1) pass that will use this option, but you can also use fpsync and option -E to perform the same task using several workers. two paths, same slay 😌

fpsync enqueues synchronization jobs on disk, within the tmpdir/queue directory. be careful to host this queue on a filesystem that can handle fine-grained mtime timestamps (i.e. with a sub-second precision) if you want the queue to be processed in order when fpart(1) generates several jobs per second. on FreeBSD, VFS(9) timestamps' precision can be tuned using the 'vfs.timestamp_precision' sysctl. see vfs_timestamp(9).

contrary to rsync(1), fpsync enforces the final / on the source directory. it means that directory contents are synchronized, not the source directory itself (i.e. you will not get a subdirectory of the name of the source directory in the target directory after synchronization). no nesting surprise, we love that 🫶

before starting filesystem crawling, fpsync changes its current working directory to src_dir/ and generates partitions containing relative paths (all starting with ./). this is important to keep in mind when modifying toolopts or fpartopts dealing with file or directory paths.

if you want to run fpsync in the background, your shell must support enabling job control without a tty in non-interactive mode. as of this writing, Dash does not support that feature and will send a SIGTTIN to the whole process group, suspending fpsync execution. not Dash being the hater 💀

---

👑 SEE ALSO 👑

cpio(1), fpart(1), mail(1), pax(1), rsync(1), tar(1), sudo(8)

---

💖 AUTHOR, AVAILABILITY 💖

Fpsync has been written by Ganaël LAPLANCHE and is available under the BSD license on http://contribs.martymac.org

---

🌸 BUGS 🌸

No bug known (yet). she's flawless, your honor ✨

---

and that's the whole page bestie 💅 this came out of the Debian distribution, dated January 27, 2015 — yeah she's an old soul but she still eats. mwah, go sync something 🎀🦋💖