CLSYNC(1)
๐ธ NAME ๐ธ
clsync โ a live sync tool, written in GNU C ๐
โจ (this is clsync(1) from the User Manuals section, bestie, and it is about to be your whole personality)
MONITOMONITORR.
๐ฆ SYNOPSIS ๐ฆ
clsync [ ... ] -- [ sync-handler-arguments ]
โจ DESCRIPTION โจ
okay so here's the tea ๐: clsync executes sync-handler with the appropriate arguments whenever FS events go down in the directory watch-dir, and it watches that directory using inotify(7) or some other FS monitoring subsystem. that's it. that's the vibe. iconic behavior, no cap. ๐
๐ OPTIONS ๐
so these options can be passed as arguments or used inside the configuration file. both. choose your fighter. ๐ซถ
to disable a numeric option, set it to zero:
=0
to disable a string option (like, say, a path to a file), set it to an empty string:
=
also โ and this is the fun part โ you can use previously set values while you're setting new options. the substring %option_name% gets substituted with the previously set value of the option option_name. (peep the CONFIGURATION FILE section, iykyk)
sync-handler-arguments applies only to these modes:
simpledirectshellrsyncdirectrsyncshell
to set sync-handler-arguments in the config file, use '--'. for example:
-- = -aH --exclude-from %EXCLUDE-LIST%
--include-from=%INCLUDE-LIST% --exclude '*' %watch-dir%/
%destination-dir%/
-W,--watch-dirwatch-dir โ the root directory thatclsyncis gonna monitor. ๐
Required. (like, actually required. don't skip her.)
-S,--sync-handlersync-handler โ path to the sync-handler thatclsyncwill use for syncing. (see--mode)
it's required for every mode except "direct" and "rsyncdirect" [see SYNC HANDLER MODES].
-R,--rules-filerules-file โ path to the file holding the filter rules for which objects get monitored. (see RULES)
not set by default.
-D,--destination-dirdestination-directory โ sets the directory to sync to, for the modes"rsyncdirect","rsyncso"and"so". (see--mode)
not set by default.
-M,--modemode โ sets the syncing mode. possible values are:simpleโ calls sync-handler for every single eventdirectโ calls sync-handler for every sync, passing the file lists as argumentsshellโ calls sync-handler for every sync, passing the file lists inside a filersyncdirectโ callsrsyncby path sync-handler directlyrsyncshellโ calls sync-handler, which is supposed to runrsync, for every sync (this is the recommended mode)rsyncsoโ loads the shared object at path sync-handler withdlopen(3)and calls the functionclsyncapi_rsyncfor every syncsoโ loads the shared object at path sync-handler withdlopen(3)and calls the functionclsyncapi_syncfor every sync
see SYNC HANDLER MODES.
Required.
-b,--backgroundโ daemonize, which forcesclsynctofork()on start. ๐ฃ
not set by default.
-H,--config-fileconfig-file-path โ use the configuration from the file config-file-path (see CONFIGURATION FILE).
set it to "/NULL/" if you don't want any config files read at all. ๐ซ
not set by default.
-K,--config-blockconfig-block-name โ use the configuration block named config-block-name (see CONFIGURATION FILE).
the default value is "default".
--config-block-inheritsconfig-parent-block-name โ use the configuration block named config-parent-block-name as the parent for config-block-name (see CONFIGURATION FILE). options from config-parent-block-name get inherited into config-block-name. it's giving โจgenerational wealthโจ but for config.
the default value is "default".
--custom-signalscustom-signals โ set a list of signals paired with their corresponding config block names. the config block gets used when the matching signal is caught.
the format is:
signal:config-block-name[,signal:config-block-name[,...]]
for example:
--custom-signals=29:debug,28:normal
in that line, the signals "28" and "29" get added to the sighandler. and clsync will use options from the config block "debug" on signal 29, and "normal" on signal 28.
to reset all custom signals, use the 0-th signal (e.g. "--custom-signals=0").
the default value is "".
-z,--pid-filepath-to-pidfile โ writes the pid to the file at path-to-pidfile. ๐
not set by default.
--status-filestatus-file-path โ writes a status description into the file at path status-file-path.
possible statuses:
startingโ initializing subsystems and marking the file tree with the FS monitor subsysteminitsyncโ processing the initial syncingrunningโ waiting for events or syncingsynchandler errorโ waiting between synchandler execution tries (after a failure) [only used while--threading=off]rehashingโ reloading configuration filesthread gcโ running the threads' garbage collectorpreexitโ executing the--pre-exit-hookterminatingโ running the last iteration (if required) and preparing to die ๐exitingโ executing the--exit-hookand cleaning up [forvalgrind(1)]
not set by default.
-r,--retriesnumber-of-tries โ the tries limit for syncing with sync-handler.
clsync will die after number-of-tries tries. ๐ญ
set it to "0" to try infinitely.
the delay between tries is equal to the --delay-sync value.
the default value is "1".
--ignore-failuresโ don't die on sync failures. we stay unbothered. ๐
not set by default.
--exit-on-sync-skipโ exit if some event could be skipped for literally any reason.
for example, FreeBSD has a very short BSM event queue (1024). so it can overflow and some events can't climb into the queue. this option forces clsync to exit if the queue has been overflowed.
not set by default.
-p,--threadingthreading-mode โ [Not available on Debian/kFreeBSD]
use pthreads(7) to parallelize the syncing processes. for example, if clsync (with --threading=off) is already syncing a huge file, then every other sync gets suspended until the huge file's syncing finishes. to prevent that suspension, you can use "safe" or "full" threading mode.
possible values:
offโ disable threading for the syncing processes.safeโ parallelize syncs, but suspend the syncings of an object that's already syncing in another process (until that process finishes).fullโ parallelize syncs without any suspendings.
characteristics:
offโ new modifications won't get synced until the old ones finish.safeโ theoretically the best way. but it may use a lot of CPU if there's a lot of simultaneous parallel syncs. (also this way isn't well tested)fullโ may cause multiple simultaneous syncings of the same file, which in turn can cause a bug inside sync-handler (see below).
โ ๏ธ if you're runningclsyncwith--threading=fullalongsidersyncwith the--backupoption, you may catch a bug due to the nonatomicity of rsync's file replace operation. (see DIAGNOSTICS)
the default value is "off".
-Y,--outputlog-destination โ sets the destination for log writing (errors, warnings, infos and debugging).
possible values:
stderrstdoutsyslog
the default value is "stderr".
--one-file-systemโ don't follow to different devices' mount points. this option just adds the option"FTS_XDEV"for thefts_open(3)function.
โ ๏ธ Warning! if you're using this option (but not--exclude-mount-points),clsyncwill write neither includes nor excludes of the content of mount points. this may cause problems, e.g. if you're usingrsyncfor sync-handler without a similar"--one-file-system"option.
not set by default.
-X,--exclude-mount-pointsโ forces--one-file-system, but also adds excludes so mount points don't get synced.
this requires doing stat(2) syscalls on every dir and can reduce performance. ๐ฎโ๐จ
not set by default.
--socketsocket-path โ create a control socket at path socket-path.
this is a very experimental feature. ๐
not set by default.
--socket-ownsocket-owner-user[:socket-owner-group]โ sets the control socket owner user (and group).
not set by default
--socket-modsocket-mode โ sets the control socket mode [seechmod(2)].
not set by default.
--standby-filestandby-file-path โ sets a file at that path which gets checked before every sync. if the file exists, the sync gets suspended until the file is deleted. it may be useful if you need to freeze the destination directory while running some scripts. ๐ง
not set by default.
--max-iterationsiterations-count โ sets a limit on synchronization iterations. one iteration means one sync-handler execution.
iterations-count:
- set to
0means no limit (infinite loop). - set to
1means only the initial sync will be done. - set to n means only the initial sync and (n-1) sync-ups after that will be done.
Hint: this option may be useful in conjunction with --exit-on-no-events to prevent infinite sync-up processes. ๐ง
the default value is "0".
--modification-signaturesignature-mask โ sets the file/dir modification recheck signature. if a file isn't modified (according to the signature), then don't sync it.
see struct stat in lstat(2) for the possible fields.
for example, reasonable signature-mask-s can be "dev,ino,mode,uid,gid,rdev,size,atime,mtime,ctime" (there's an alias for that โ "*") or "uid,gid".
โจ Examples of use cases โจ
chown/chmodโ so like, if you're using clsync for fixing file/dir privileges [withchown(1)and/orchmod(1)] then a reasonable signature is gonna be"uid,gid", bestie.
Full example, no cap:
clsync -w5 -t5 -T5 -x1 -W /var/www/site.example.org/root -Mdirect -Schown --uid 0 --gid 0 -Ysyslog -b1 --modification-signature uid,gid -- --from=root www-data:www-data %INCLUDE-LIST%
- bi-directional syncing โ if you're setting up bi-directional syncing then you can lowkey use
--modification-signature "*"to stop a sync loop happening between servers. ๐ฆ - Not enough CPU โ if rsync is eating way too much CPU rechecking hashsums of files on their dry
open()/close()because of some hacky script (imagine achown -R www-data:www-datajust vibing in cron ๐) then you can use--modification-signature "dev,ino,mode,uid,gid,rdev,size,atime,mtime"(that's without"blksize","blocks","nlink"and"ctime").
Warning! This option may eat a lot of memory on huge file trees. Like, she's hungry. ๐ญ
This option cannot be used together with --cancel-syscalls=mon_stat.
To disable file/dir modification rechecking, use the empty value โ "".
The default value is "".
-k,--timeout-syncsync-timeout โ sets the timeout for syncing processes. clsync will literally die if a syncing process stays alive for more than sync-timeout seconds. ๐
Set it to "0" to disable the timeout entirely.
The default value is "86400" ["24 hours"].
-w,--delay-syncadditional-delay โ sets the minimal delay (in seconds) between syncs. She waits, she's patient, she's iconic. ๐
The default value is "30".
-t,--delay-collectordinary-delay โ sets the delay (in seconds) for collecting events about ordinary files and directories.
The default value is "30".
-T,--delay-collect-bigfilebigfiles-delay โ sets the delay (in seconds) for collecting events about "big files" (peep--threshold-bigfile).
The default value is "1800".
-B,--threshold-bigfilefilesize-threshold โ sets the file size threshold (in bytes) that separates ordinary files from "big files". Events about "big files" get processed in another queue with its own separate collecting delay. The whole point is unloading IO resources, fr fr.
To disable detection of "big files" set "0" (zero). That can improve performance by removing the need for an extra lstat() syscall. Slay.
The default value is "134217728" ["128 MiB"].
--cancel-syscallssyscalls-mask โ sets syscalls to be bypassed. This can be used to squeeze out more performance. ๐ฅ
Possible values:
mon_statโ skiplstat()calls while handling files/dirs events. This makes it impossible to determine file sizes (which is what the--threshold-bigfileoption uses) and to use the--modification-signatureoption. Big yikes.
You can combine these values using commas.
To disable this option just use the empty value โ "".
The default value is "".
-L,--lists-dirtmpdir-path โ sets the directory path for outputting temporary events-lists files. ๐
See SYNC HANDLER MODES.
Is not set by default.
--have-recursive-syncโ use the action"recursivesync"instead of"synclist"for directories that were just marked (see SYNC HANDLER MODES case shell).
Is not set by default.
--synclist-simplifyโ removes the first 3 parameters in list files of the action"synclist"(see SYNC HANDLER MODES case shell).
Is not set by default.
--rsync-inclimitrsync-includes-line-limit โ sets a soft limit for the lines count in files by path rsync-listpath. Unfortunately, rsync works very slowly with huge--include-fromfiles. So clsync splits that list into approximately rsync-includes-line-limit lines per list if it's too big, and executes one rsync instance per list part. Use the value"0"to disable the limit.
The default value is "20000".
--rsync-prefer-includeโ forces clsync to prefer a "lot of includes" method instead of an "excludes+includes" one for rsync on recursive syncing.
See cases rsyncshell, rsyncdirect and rsyncso of SYNC HANDLER MODES.
This option is not recommended. ๐
Is not set by default.
-x,--ignore-exitcodeexitcode โ forces clsync to not process exitcode of sync-handler as an error. You can set multiple ignores by passing this option multiple times.
Recommended values for the rsync case is "24". You can set multiple values by listing a lot of -x options (e.g. -x 23 -x 24) or via commas (e.g. -x 23,24). To drop the list use zero exitcode (e.g. -x 0). For example you can use -x 0,23 to drop the list and set the "23"-th exitcode to be ignored.
Is not set by default (or equally is set to "0").
-U,--dont-unlink-listsโ do not delete list-files after sync-handler has finished.
This may be used for debugging purposes. ๐ง
Is not set by default.
--fts-experimental-optimizationโ enable experimental features to optimize file tree scanning while usingfts(3). The features will be enabled by default after appropriate testing.
At the moment the option doesn't do anything but can be used in the future.
Is not set by default.
-F,--full-initialsyncโ ignore filter rules from rules-file on initial sync.
This may be useful for a quick start or e.g. if it's required to sync the "/var/log/" tree but not sync every change from there. ๐
Is not set by default.
--only-initialsyncโ exit after initial syncing on clsync start.
Is not set by default.
--exit-on-no-eventsโ exit if there's no events. Works like--only-initialsync, but also syncs events collected while the initial syncing.
Unlike --only-initialsync, this option uses the FS monitor subsystem to monitor for new events while the initial syncing. This may reduce performance. On the other hand this way may be used to be sure that everything is synced at the moment before clsync will exit. ๐
Is not set by default.
--skip-initialsyncโ skip initial syncing on clsync start.
Is not set by default.
--sync-on-quitโ onSIGQUIT: clsync will perform one more final syncing to empty the queue of collected events before exit.
Is not set by default.
--exit-hookpath-of-exit-hook-program โ sets the path of the program to be executed on clsync exit.
If this parameter is set then clsync will exec on exit:
path-of-exit-hook-program label
The execution will be skipped if the initial sync wasn't complete.
Is not set by default.
--pre-exit-hookpath-of-pre-exit-hook-program โ sets the path of the program to be executed before the last sync iteration (see--max-iterations,--exit-on-no-eventsand SIGNALS).
If this parameter is set then clsync will exec on exit:
path-of-pre-exit-hook-program label
The execution will be skipped if the initial sync wasn't complete.
If clsync finishes due to --exit-on-no-events and --pre-exit-hook is set then the pre-exit hook will be executed and an additional sync iteration will be triggered.
Is not set by default.
-v,--verboseโ this option is supposed to increase verbosity. But at the moment there's no "verbose output" in the code, so the option does nothing. :) ๐ญ
Is not set by default.
-d,--debugโ increases debugging output. This may be supplied multiple times for more debugging information, up to a maximum of fivedflags (more will do nothing), for example-d -d -d -d -dor-d5(equivalent cases).
Is not set by default.
--dump-dirโ directory to write clsync's instance information by signal 29 (see SIGNALS). The directory shouldn't exist before dumping. ๐
Is set to "/tmp/clsync-dump-%label%" by default.
-q,--quietโ suppresses error messages. ๐คซ
Is not set by default.
--monitormonitor-subsystem โ switches the FS monitor subsystem.
Possible values:
inotifyโinotify(7)[Linux, (FreeBSD via libinotify)]
Native, fast, reliable and well tested Linux FS monitor subsystem. ๐
There's no essential performance profit to use "inotify" instead of "kevent" on FreeBSD using "libinotify". It backends to "kevent" anyway.
FreeBSD users: the libinotify on FreeBSD is still not ready and unusable for clsync to sync a lot of files and directories. ๐ญ
gioโ use the gio library.
Crossplatform and tested library that backends to kqueue on FreeBSD and inotify on Linux. See the inotify and kqueue sections here for details.
Not well tested. Use with caution! ๐จ
kqueueโkqueue(2)[FreeBSD, (Linux via libkqueue)]
A *BSD kernel event notification mechanism (inc. timer, sockets, files etc).
okay so this monitor subsystem literally cannot tell when a file gets created, bestie โ it can only tell you which directory something happened in. So every time anything changes, clsync has to rescan that whole dir. Every. Single. Time. ๐ญ
Also kqueue requires an open() on every watched file and dir. But FreeBSD won't let you open() a symlink itself (like, without following it), and open()-ing pipes and devices is highkey invasive, so clsync just... doesn't. It won't call open() on anything except regular files and directories. Which means it cannot determine if something changed inside a symlink / pipe / socket / etc. ๐ซ However, it still can determine if one gets created or deleted, by watching the parent directory and rescanning it on every appropriate event. so that's something ๐
Also this API opens every monitored file and directory, so it may produce a huge amount of file descriptors. Make sure kern.maxfiles is big enough (on FreeBSD, babe). ๐
CPU/HDD expensive way. fr fr. ๐
Not well tested. Use with caution!
Linux users: the libkqueue on Linux is not working. He-he :) ๐
bsm
bsm(3) [FreeBSD]
Basic Security Module (BSM) Audit API. โจ
Okay so this isn't actually a FS monitor subsystem โ it's just an API to get at audit information (including logs). clsync can set up audit to watch FS events and report them into the log, and after that it just parses the log via auditpipe(4) [FreeBSD]. ๐ฆ
Reliable, but a hacky way to do it. It requires global audit reconfiguration, which may hopple audit analysis. ngl, that's a real one.
Warning! FreeBSD has a limit for queued events. In the default FreeBSD kernel it's only 1024 events. So pick one of these, bestie: - To patch the kernel to increase the limit. - Don't use clsync on systems with too many file events. - Use bsm_prefetch mode (but there's no guarantee in this case anyway).
See also the option --exit-on-sync-skip.
Not well tested. Use with caution! Also the file /etc/security/audit_control will be overwritten with:
#clsync
dir:/var/audit
flags:fc,fd,fw,fm,cl
minfree:0
naflags:fc,fd,fw,fm,cl
policy:cnt
filesz:1M
...unless it already starts with "#clsync\n" ("\n" is a new line character, just so we're clear ๐).
bsm_prefetch
Same as bsm, but all BSM events get prefetched by an additional thread to prevent BSM queue overflow. This may utilize a lot of memory on systems with a high FS events frequency. ๐
However โ and this is the tea โ โ the thread may not be fast enough to unload the kernel BSM queue. So it may overflow anyway. ๐
The default value on Linux is "inotify". The default value on FreeBSD is "kqueue".
-l,--labellabel โ Sets a label for this instance of clsync. The label will be passed to sync-handler every execution. The default value is"nolabel". ๐ท๏ธ-h,--helpโ Outputs options list and exits with exitcode"0". Is not set by default. ๐ฅฐ-V,--versionโ Outputs clsync version and exits with exitcode"0". Is not set by default. โญ--cgroup-group-namecg-group-name โ Set cgroup group name [seecgroup_new_cgroup()]. Is set to"clsync/%PID%"by default. ๐
โจ SECURITY OPTIONS โจ
okay bestie, this is the part where we lock everything DOWN ๐ because running clsync with root or capabilities is powerful and powerful needs boundaries, fr fr.
--secure-splittingโ the one-flag-to-rule-them-all โจ it implies--splitting=process,--check-execvp-arguments,--seccomp-filterand--forbid-devicesall at once.-u,--uiduid โ drops user privileges down to uid withsetuid(2). if there'scapabilities(7)support hanging around, the default value is"nobody"(or"65534"if"nobody"can't be found anywhere ๐ญ), otherwise this option just isn't set by default;-g,--gidgid โ same energy but for the group side: drops group privileges to gid withsetgid(2). ifcapabilities(7)support exists, the default value is"nogroup"(or"65534"if"nogroup"is MIA), otherwise it's not set by default;--privileged-uidsync-handler-uid โ the user ID that the privileged process gets to use (go peep--splitting=processfor the full lore).
default value is "$UID".
--privileged-gidsync-handler-gid โ the group ID used for the privileged process (again, see--splitting=process).
default value is "$GID".
--sync-handler-uidsync-handler-uid โ the user ID used for the sync-handler.
check out --preserve-capabilities.
default value is the same as for --privileged-uid.
--sync-handler-gidsync-handler-gid โ the group ID used for the sync-handler.
check out --preserve-capabilities.
default value is the same as for --privileged-gid.
-C,--preserve-capabilitiescapabilities-list โ [Linux only, requires capabilities] usescapset(2)andprctl(2)to preserve"CAP_DAC_READ_SEARCH","CAP_SETUID"or/and"CAP_SETGID"[seecapabilities(7)] Linux capability for the process usingfts(3),inotify(7)andexecve(2). this is how it keeps just enough FS privileges to watch a file tree and execute the sync-handler with the required uid and gid [see--sync-handler-uidand--sync-handler-gid] after dropping privileges viasetuid(2)andsetgid(2)[see--uidand--gid].
possible values:
CAP_DAC_READ_SEARCHโ to bypass FS read checks (for fts and inotify).CAP_SETUIDโ to be able to usesetuid(2)beforeexecve(2)on the sync-handler.CAP_SETGIDโ to be able to usesetgid(2)beforeexecve(2)on the sync-handler.CAP_KILLโ to be able to killsetuid()-ed processes
any combinations of these values are supported too ๐ซถ the list can be comma separated values, like:
CAP_DAC_READ_SEARCH,CAP_SETUID,CAP_SETGID
the default value is "CAP_DAC_READ_SEARCH,CAP_SETUID,CAP_SETGID,CAP_KILL" if the clsync runner has such privileges.
--inherit-capabilitiesโ [Linux only, requires capabilities] sets a mode for capabilities inheriting.
possible values:
permittedโ inherits all permitted capabilitiesdont-touchโ don't change the inheritable capabilities setclsyncโ use clsync's effective capabilities setemptyโ reset all capabilities
the default value is "empty".
--splittingsplitting-type โ splits the process/thread into a privileged part and a non-privileged part. it's an extra way to secure your system from any bug in clsync while it's running with capabilities or root privileges. but heads up, clsync may use a few times more CPU resources ๐ so yeah, it's a performance vs security trade off, pick your fighter.
you can basically cut the overhead by using "high load locks" ("--enable-highload-locks" of "./configure" file).
if you're using this option AND running the sync-handler with the root user, it's highly recommended to enable--check-execvp-argumentstoo. otherwise, in case of a clsync security bug, a hacker would be able to useexecvp()with any arguments with root privileges ๐ not the vibe.
possible values:
offโ disables this featurethreadโ [Linux only, requires capabilities] creates a separate thread for privileged operations.
it's highly recommended to enable --seccomp-filter in this case. but that will forbid --threading.
processโ a more secure and portable way, but it uses a separate process and:- forbids fanotify (which isn't implemented yet anyway);
- more complex code (and higher probability of error).
- slower due to copying data between private and shared memory pages.
recommended โจ
it's set to "off" by default.
--check-execvp-argumentsโ [Requires--splitting=[thread|process]] [Blocks--mode=direct] enablesexecvp()arguments recheck in the privileged process, just in case they got swapped out for exploit-given arguments ๐ค.
this option doesn't use a lot of CPU resources, but it forbids run-time changing of sync-handler-arguments and hook file paths.
this option cannot be used in conjunction with --mode=direct because of the arbitrary number of arguments in that mode.
it's not set by default.
--add-permitted-hook-files[hook-path0,[hook-path1[,...]]]โ [Requires--check-execvp-arguments] adds paths to the list of permitted hook paths so they can bypass--check-execvp-argumentschecks. you might need this if you're planning to change the hooks at run-time using--custom-signalsor--socket.
it's not set by default.
--seccomp-filterโ [Linux only] uses a seccomp filter to forbid syscalls that clsync shouldn't be using.
it forbids all syscalls for the non-privileged process/thread, but:
futex inotify_init1 alert stat fstat lstat open write close wait4
unlink tgkill clock_gettime rt_sigreturn brk mmap munmap wait4 rmdir
exit_group select read rt_sigprocmask rt_sigaction nanosleep
it's not set by default.
--permit-mprotectโ [Requires--seccomp-filter] permits themprotect(2)syscall.
this syscall is required by pthread_create(3), so it's required for --threading.
it makes --shm-mprotect useless.
it also enables the ability to change memory of the privileged thread from non-privileged, so using --splitting=thread with this option is useless too ๐
it's set to "0" by default if --splitting is set. otherwise "1".
--shm-mprotectโ [Requires--splitting=process] forbids writing to or reading from shared memory when it shouldn't be.mprotect(2)is used for the protection.
this option is useless while --permit-mprotect is enabled.
--chrootchroot-directory โ clsyncchroot()-s [seechroot(2)] into directory chroot-directory before any syncing processes.
this option may be used together with --uid, --gid or/and --pivot-root for security reasons.
remember! if you'rechroot()-ing somewhere, the sync-handler will be limited by the chroot-environment too. and if you're using rsync, you might wanna"mount --bind"some directories to the chroot-directory.
it's not set by default.
--pivot-rootpivot-root-way โ [Linux only, requires--chroot] sets a way of using thepivot_root(2)syscall on the chroot-directory (toumount(2)the old rootfs).
possible values:
autoโ creates a directory"/dev/shm/clsync-rootfs",unshare(2)-ing the mount namespace,mount(2)-s the chroot-directory to the directory and thenpivot_root(2)-ing,chroot(2)-ing andumount(2)-ing old rootfs. the directory"/dev/shm/clsync-rootfs"won't be deleted after clsync finishes.auto-roโ the same asautobut mounts the directory with the read-only option (MS_RDONLY).directโunshare(2)-ing the mount namespace,pivot_root(2)-ing,chroot(2)-ing andumount(2)-ing old rootfs. the directory"old_root"should be created in the chroot-directory before running clsync in this mode.offโ don'tpivot_root(2).
the default value is "off". if --chroot is used then the recommended value is "auto-ro".
--mountpoints[mountpoint[,mountpoint[,mountpoint]]]โ [Linux only] umounts (withMNT_DETACH) everything except the listed mountpoints.
supposed to be used for security reasons as an alternative to the --pivot-root option.
it's not set by default.
--detach-networkdetach-network-mode โ [Linux only] removes network in the clsync instance.
possible values:
everywhereโ removes network for all processes.non-privilegedโ removes network from the non-privileged process if option--process-splittingis enabled, otherwise it doesn't do anything.offโ don't do anything.
the default value is "non-privileged".
--detach-ipcโ [Linux only] makes its own IPC namespace.
it's set by default. ๐
--detach-miscellaneaโ [Linux only]unshare(2)on everything not listed above.
it's not set by default.
--forbid-devicesโ [Linux only] forbids any access to all devices except the listed ones:- read access to:
/dev/console/dev/zero/dev/urandom/dev/random
- write access to:
/dev/console/dev/null
- read access to:
it's not set by default.
๐ PERFORMANCE ๐
Okay bestie, if clsync is being a lil slow girl and you want her to zoom, here's the tea โ these are the recs to improve the performance:
- Disable thread/process splitting. no splitting, no drama ๐
- Don't use clsync rules (put the rules on the sync-handler side instead) or/and use option
--full-initialsync - Use option
-B0. - Use option
--cancel-syscalls=mon_stat. - Use option
-p safeor-p full. - Disable debugging with
-d0โ or, even better, disable debugging support at all with the./configureoption--enable-debug=no - Don't use option
--exclude-mount-points - Free memory for disk cache
โ ๏ธ Do not follow all this recommendation blindfold, okay? ๐ญ Pick only the ideas that actually fix performance problems in your specific use case. And only if it's actually necessary. Periodt.
๐ SYNC HANDLER MODES ๐
So clsync executes a sync-handler that's supposed to take care of the actual syncing process. Which means clsync is literally just a convenient way to run a syncing script. That's the whole vibe ๐ซถ
clsync can run sync-handler in seven ways. Which way gets used depends on the specified mode (see --mode).
sync-handler-arguments are used only in modes:
simpledirectshellrsyncdirectrsyncshell
If sync-handler-arguments are not set, then the default setting is used (see below). โจ
case simple
Executes for every syncing file/dir:
sync-handler sync-handler-arguments
Default sync-handler-arguments are:
sync %label% %EVENT-MASK% %INCLUDE-LIST%
In this case, sync-handler is supposed to non-recursively sync file or directory by path %INCLUDE-LIST%. With %EVENT-MASK% it's passed bitmask of events with the file or directory (see "/usr/include/linux/inotify.h").
Additional substitutions:
%EVENT-MASK%โ Is replaced by integer of events IDs.%INCLUDE-LIST%โ Is replaced by absolute path of a file/dir to be synced.
case direct
Executes for every sync:
sync-handler sync-handler-arguments
Default sync-handler-arguments are:
%INCLUDE-LIST% %destination-dir%/
Additional substitutions:
%INCLUDE-LIST%โ Is replaced by a list of relative paths of files/dirs to be synced.
case shell
Executes for every sync (if recursivesync is not used instead):
sync-handler sync-handler-arguments
Default sync-handler-arguments are:
synclist %label% %INCLUDE-LIST-PATH%
Default sync-handler-arguments for initial sync if --have-recursive-sync is set are:
initialsync %label% %INCLUDE-LIST%
In this case, sync-handler is supposed to non-recursively sync files and directories from list in a file by path %INCLUDE-LIST-PATH% on "synclist".
Also sync-handler is supposed to recursively sync data from directory by path %INCLUDE-LIST-PATH% with manual excluding extra files on "initialsync".
Additional substitutions:
%TYPE%โ Is replaced by"sync"/"initialsync".%INCLUDE-LIST-PATH%โ Is replaced by the path of the include list file.%INCLUDE-LIST%โ Is replaced by a list of relative paths of files/dirs to be synced.
๐ Not recommended. Not well tested.
case rsyncdirect
Executes for every sync:
sync-handler sync-handler-arguments
sync-handler is supposed to be a path to rsync binary.
Default sync-handler-arguments are:
-aH --delete --exclude-from %EXCLUDE-LIST-PATH%
--include-from %INCLUDE-LIST-PATH% --exclude='*'
%watch-dir%/ %destination-dir%/
if option --rsync--prefer-include is not set and:
-aH --delete --include-from %INCLUDE-LIST-PATH%
--exclude='*' %watch-dir%/ %destination-dir%/
if the option is set.
Error code "24" from sync-handler will be ignored in this case. We also recommend to ignore exitcode "23".
Additional substitutions:
%INCLUDE-LIST-PATH%โ Is replaced by the path of the include list file%EXCLUDE-LIST-PATH%โ Is replaced by the path of the exclude list file%RSYNC-ARGS%โ Is replaced by default sync-handler-arguments, but without"%watch-dir%/ %destination-dir%/"
๐ Recommended case.
case rsyncshell
Executes for every sync:
sync-handler sync-handler-arguments
Default sync-handler-arguments are:
rsynclist %label% %INCLUDE-LIST-PATH%
[%EXCLUDE-LIST-PATH%]
In this case, sync-handler is supposed to run "rsync" application with parameters:
-aH --delete-before --include-from %INCLUDE-LIST-PATH% --exclude
'*'
if option --rsync-prefer-include is enabled.
And with parameters:
-aH --delete-before --exclude-from %EXCLUDE-LIST-PATH%
--include-from %INCLUDE-LIST-PATH% --exclude '*'
if option --rsync-prefer-include is disabled.
Additional substitutions:
%INCLUDE-LIST-PATH%โ Is replaced by the path of the rsync include list file%EXCLUDE-LIST-PATH%โ Is replaced by the path of the rsync exclude list file
๐ Recommended case.
case rsyncso
In this case there's no direct exec*() calling. In this case clsync loads sync-handler as a shared library with dlopen(3) and calls function "int clsyncapi_rsync(const char *inclist, const char *exclist)" from it for every sync.
inclist is a path to file with rules for "--include-from" option of rsync. This argument is always not NULL.
exclist is a path to file with rules for "--exclude-from" option of rsync. This argument is NULL if --rsync-prefer-include is set.
Excludes takes precedence over includes. โจ
Also may be defined functions "int clsyncapi_init(ctx_t *, indexes_t *)" and "int clsyncapi_deinit()" to initialize and deinitialize the syncing process by this shared object.
To fork the process should be used function "pid_t clsyncapi_fork(ctx_t *)" instead of "pid_t fork()" to make clsync be able to kill the child.
See example file "clsync-synchandler-rsyncso.c".
๐ Recommended case.
case so
In this case there's no direct exec*() calling. In this case clsync loads sync-handler as a shared library with dlopen(3) and calls function "int clsyncapi_sync(int n, api_eventinfo_t *ei)" from it for every sync. n is number of elements of ei. ei is an array of structures with information about what and how to sync (see below).
api_eventinfo_t is a structure:
struct api_eventinfo {
uint32_t evmask; // event bitmask for
file/dir by path path.
uint32_t flags; // flags of "how to
sync" the file/dir
size_t path_len; // strlen(path)
const char *path; // the path to
file/dir need to be synced
eventobjtype_t objtype_old; // type of object by
path path before the event.
eventobjtype_t objtype_new; // type of object by
path path after the event.
};
typedef struct api_eventinfo api_eventinfo_t;
The event bitmask (evmask) values can be learned from "/usr/include/linux/inotify.h".
There may be next flags' values (flags):
enum eventinfo_flags {
EVIF_NONE = 0x00000000, // No modifier
EVIF_RECURSIVELY = 0x00000001 // sync the file/dir
recursively
};
Flag "EVIF_RECURSIVELY" may be used if option --have-recursive-sync is set.
Is that a file or directory by path path can be determined with objtype_old and objtype_new.
objtype_old reports about which type was the object by the path before the event.
objtype_new reports about which type became the object by the path after the event.
objtype_old and objtype_new have type eventobjtype_t.
enum eventobjtype {
EOT_UNKNOWN = 0, // Unknown
EOT_DOESNTEXIST = 1, // Doesn't exist (not
created yet or already deleted)
EOT_FILE = 2, // File
EOT_DIR = 3, // Directory
} typedef enum eventobjtype eventobjtype_t;
Also may be defined functions "int clsyncapi_init(options_t *, indexes_t *)" and "int clsyncapi_deinit()" to initialize and deinitialize the syncing process by this shared object.
To fork the process should be used function "pid_t clsyncapi_fork(options_t *)" instead of "pid_t fork()" to make clsync be able to kill the child.
See example file "clsync-synchandler-so.c".
๐ Recommended case.
๐ธ ENVIRONMENT VARIABLES ๐ธ
Recognized variables โ variables used by clsync.
Recognized variables
HOMEโ user's HOME directory, clsync looks for configuration file hereTMPDIRโ use this directory as a prefix to create clsync temporary directory inside, if unset/tmpwill be used
Output variables โ variables that are set by clsync before calling sync-handler.
Output variables
CLSYNC_STATUSโ clsync's status (see possible statuses in description of--status-file)CLSYNC_ITERATIONโ count of done synchronizaton iterations after initial sync see--max-iterationsoption
๐ RULES ๐
Okay so bestie, filter rules are how you tell clsync which events it should be watching and which ones it should straight up ignore. ๐
โ ๏ธ Caution! These rules do not guarantee that a filtered file/dir won't get synced. Why? Because a file or directory can literally appear in the exact moment the sync-handler is running (or after it started but before it reaches that directory), and by then it's too late to add an exclusion. So if you need a real, actual guarantee that a file won't sync, use the internal filter rules of the sync-handler program instead (rsync, for example, has the options--exclude,--exclude-fromand--filter), or just disable any "recursive" syncs in clsync (and drop the-avoption of rsync if you're using it). ๐ธ
To disable recursive syncs you can use:
simpleโ Already non-recursivedirectโ Already non-recursiveshellโ Don't enable option--have-recursive-sync.rsyncdirectโ Use option--rsync-prefer-includeand set sync-handler-arguments to-lptgoD --delete --include-from %INCLUDE-LIST-PATH% --exclude='*' %watch-dir%/ %destination-dir%/rsyncshellโ Use option--rsync-prefer-include.rsyncsoโ Use option--rsync-prefer-include.soโ Don't enable option--have-recursive-sync.
Filter rules go into a rules-file, one rule per line. ๐
Rule format: [+-][fdWwms*]regexp
+โ means include;-โ means exclude;fโ means file;dโ means directory (the same as an combination of "w" + "m" + "s");wโ means walking to directory;mโ means monitor events in the directory;sโ means sync the the directory if modified;Wโ means walk + monitor the directory (the same as an combination of "w" + "m");*โ means all.
For example: -*^/[Tt]est
It's not recommended to use w rules in modes rsyncdirect, rsyncshell and rsyncso. rsync(1) allows one to set syncing and walking only together in --include rules (--files-from is not appropriate due to problem with syncing files deletions). So there may be problems with clsync's w rules in this cases. ๐ญ
More examples:
Syncing pwdb files and sshd_config (non-rsync case):
# which files to sync:
+f^passwd$
+f^group$
+f^shadow$
+f^ssh/sshd_config$
# walk and monitor next directories:
+W^$
+W^ssh$
# forbid the rest:
-*
Syncing pwdb files and sshd_config (rsync case):
+f^passwd$
+f^group$
+f^shadow$
+f^ssh/sshd_config$
+d^$
+d^ssh$
-*
Syncing /srv/lxc tree (rsync case):
-d/sess(ion)?s?$
-f/tmp/
+*
๐ฆ SIGNALS ๐ฆ
1โ (HUP) rereads filter rules2โ (INT) exits without waiting of syncing processes ("hard kill", kills children)3โ (QUIT) waits for current syncing processes and exit ("soft kill", waits for children). See also--sync-on-quit.10โ runs threads' GC function12โ runs full resync15โ (TERM) exits without waiting of syncing processes ("hard kill", kills children)16โ interruptssleep()/select()andwait()[for debugging and internal uses]29โ dump information to dump-dir [for debugging]
If you need to kill clsync but leave children then you can use 9-th (KILL) signal. ๐
๐ฅ DIAGNOSTICS ๐ฅ
clsync misses events: Try to run command: sysctl fs.inotify.max_queued_events=1048576
Initial rsync process works very slow on clsync start Probably there's too huge exclude list is passed to rsync. This can happened if you're excluding with regex in clsync's rules a lot of thousands files. They will be passed to rsync's exclude list one by one.
To diagnose it, you can use -U option and look into rsync-exclude-listpath file (see SYNC HANDLER case d)
To prevent this, it's recommended to write such rules for rsync directly (not via clsync).
For example, often problem is with PHP's session files. You shouldn't exclude them in clsync's rules with -f/sess_.*, but you should exclude it in rsync directly (e.g with ยซ--exclude "sess_*"ยป).
The following diagnostics may be issued on stderr:
Error: Cannot inotify_add_watch() on [...]: No space left on device (errno: 28) Not enough inotify watching descriptors is allowed. It can be fixed by increasing value of sysctl fs.inotify.max_user_watches
Error: Got non-zero exitcode exitcode [...] sync-handler returned non-zero exitcode. Probably, you should process exitcodes in it or your syncer process didn't worked well. I case of using rsync, you can find the exitcodes meanings in man 1 rsync.
If exitcode equals to 23 and you're using clsync in conjunction with rsync, this may happen, for example, in the following cases:
- Not enough space on destination.
- You're running clsync with
--threading=fulland rsync with--backup. See a bugreportโจhttps://bugzilla.samba.org/show_bug.cgi?id=10081โฉ.
To confirm the problem, you can try to add "return 0" or "exit 0" into your sync-handler.
Bad system call If --use-seccomp option is enabled then the error is probably caused by using of forbidden syscall. It's a clsync bug or hack attack attempt.
To get support see SUPPORT.
โจ CONFIGURATION FILE โจ
clsync supports configuration file. ๐ซถ
By default clsync tries to read next files (in specified order):
~/.clsync.conf
/etc/clsync/clsync.conf
This may be overridden with option --config-file.
clsync reads only one configuration file. In other words, if option --config-file is not set and file ~/.clsync.conf is accessible and parsable, clsync will not try to open /etc/clsync/clsync.conf. Command line options have precedence over config file options.
Configuration file is parsed with glib's g_key_file_* API. That means, that config should consits from groups (blocks) of key-value lines as in the example:
[default]
background = 1
mode = rsyncshell
debug = 0
output = syslog
label = default
pid-file = /var/run/clsync-%label%.pid
[debug]
config-block-inherits = default
debug = 5
background = 0
output = stderr
[test]
mode=rsyncdirect
debug=3
Also glib's gkf API doesn't support multiple assignments. If you need to list some values (e.g. exitcodes) just list them with commas in single assignment (e.g. "ignore-exitcode=23,24").
In this example there're 3 blocks are set - "default", "debug" and "test". And block "debug" inherited setup of block "default" except options "debug", "background" and "output".
By default clsync uses block with name "default". Block name can be set by option --config-block.
๐ CLUSTERING ๐
Not implemented yet. Don't try to use cluster functionality. ๐ซ
Not described yet.
๐ EXAMPLES ๐
Mirroring a directory:
clsync -Mrsyncdirect -W/path/to/source_dir -D/path/to/destination_dir
Syncing 'authorized_keys' files:
mkdir -p /etc/clsync/rules
printf "+w^$\n+w^[^/]+$\n+W^[^/]+/.ssh$\n+f^[^/]+/.ssh/authorized_keys$\n-*" > /etc/clsync/rules/authorized_files_only
clsync -Mdirect -Scp -W/mnt/master/home/ -D/home -R/etc/clsync/rules/authorized_files_only -- -Pfp --parents %INCLUDE-LIST% %destination-dir%
Mirroring a directory, but faster:
clsync -w5 -t5 -T5 -Mrsyncdirect -W/path/to/source_dir -D/path/to/destination_dir
Instant mirroring of a directory:
clsync -w0 -t0 -T0 -Mrsyncdirect -W/path/to/source_dir -D/path/to/destination_dir
Making two directories synchronous:
clsync -Mrsyncdirect --background -z /var/run/clsync0.pid --output syslog -Mrsyncdirect -W/path/to/dir1 -D/path/to/dir2 --modification-signature '*'
clsync -Mrsyncdirect --background -z /var/run/clsync1.pid --output syslog -Mrsyncdirect -W/path/to/dir2 -D/path/to/dir1 --modification-signature '*'
Fixing privileges of a web-site:
clsync -w3 -t3 -T3 -x1 -W/var/www/site.example.org/root -Mdirect -Schown --uid 0 --gid 0 -Ysyslog -b1 --modification-signature uid,gid -- --from=root www-data:www-data %INCLUDE-LIST%
'Atomic' sync:
clsync --exit-on-no-events --max-iterations=20 --mode=rsyncdirect -W/var/www_new -Srsync -- %RSYNC-ARGS% /var/www_new/ /var/www/
Moving a web-server:
clsync --exit-on-no-events --max-iterations=20 --pre-exit-hook=/root/stop-here.sh --exit-hook=/root/start-there.sh --mode=rsyncdirect --ignore-exitcode=23,24 --retries=3 -W /var/www -S rsync -- %RSYNC-ARGS% /var/www/ rsync://clsync@another-host/var/www/
Copying files to slave-nodes using pdcp(1):
clsync -Msimple -S pdcp -W /opt/global -b -Y syslog -- -a %INCLUDE-LIST% %INCLUDE-LIST%
Copying files to slave-nodes using uftp(1):
clsync -Mdirect -S uftp -W/opt/global --background=1 --output=syslog -- -M 248.225.233.1 %INCLUDE-LIST%
A dry running to see rsync(1) arguments that clsync will use:
clsync -Mrsyncdirect -S echo -W/path/to/source_dir -D/path/to/destination_dir
An another dry running to look how clsync will call pdcp(1):
clsync -Msimple -S echo -W /opt/global -b0 -- pdcp -a %INCLUDE-LIST% %INCLUDE-LIST%
Automatically run 'make build' if any '*.c' file changed
printf "%s\n" "+f.c$" "-f" | clsync --have-recursive-sync -W . -R /dev/stdin -Mdirect -r1 --ignore-failures -t1 -w1 -Smake -- build
More working examples you can try out in "/usr/share/doc/clsync/examples/" directory. Copy this directory somewhere (e.g. into "/tmp"). And try to run "clsync-start-rsync.sh" in there. Any files/directories modifications in "testdir/from" will be synced to "testdir/to" in a few seconds. ๐ง
๐ AUTHOR ๐
Dmitry Yu Okunev <dyokunev@ut.mephi.ru> 0x8E30679C
๐ฌ SUPPORT ๐ฌ
okay so if you get stuck โ and bestie, you will get stuck at some point, it's fine, it happens to literally everyone โ you have two whole places to come cry about it:
- the official IRC channel over on Freenode:
#clsync๐ซถ - the github issue tracking system for the clsync repo:
https://github.com/clsync/clsyncโจ
Do <b>not</b> be afraid to ask about clsync configuration, ;). ๐
fr fr, the winking semicolon is doing a lot of emotional labour here but the message is real: nobody is gonna clown you for not knowing how to wire up your config. ask the question. that's what the channel is for. ๐
๐ SEE ALSO ๐
go read these too, they're the supporting cast and they ate:
and that's a wrap on the whole page, bestie ๐ you made it to the end, which is genuinely main character energy. this lil document is from the Linux world, dated APRIL 2020, and it lives under the name CLSYNC(1) ๐ go sync something. not me being proud of you. ๐๐ฆ
๐ฆ where this page comes from
- alpine 3.24: clsync-doc 0.4.5-r4 ยท GPL-3.0-or-later
- debian bookworm: clsync 0.4.5-2 ยท GPL-3.0-or-later
- debian trixie: clsync 0.4.5-2.1+b2 ยท GPL-3.0-or-later
- ubuntu jammy: clsync 0.4.5-2 ยท GPL-3.0-or-later
- ubuntu noble: clsync 0.4.5-2.1build1 ยท GPL-3.0-or-later
- ubuntu questing: clsync 0.4.5-2.2
- ubuntu resolute: clsync 0.4.5-2.2build1 ยท GPL-3.0-or-later
- ubuntu stonking: clsync 0.4.5-2.3 ยท GPL-3.0-or-later
- ๐ girlified by deepseek-ai/DeepSeek-V4.1-Flash on : a language model's rewrite of the original, under its terms