anything-sync-daemon(1)

📦 version 6.0.0, in arch rolling

💅 anything-sync-daemon — the manual, but make it make sense ✨

okay so bestie, you're reading the page for anything-sync-daemon, a.k.a. asd. This is anything-sync-daemon(1) — yes, section one, the icon herself. Buckle in, I'm about to make a daemon sound cute. 🎀

🌸 NAME 🌸

anything-sync-daemon — symlinks and syncs user specified dirs to RAM thus reducing HDD/SDD calls and speeding-up the system. 🚀 literally free speed, no cap.

🦋 DESCRIPTION 🦋

Anything-sync-daemon (asd) is a tiny pseudo-daemon designed to manage user specified directories — referred to as sync targets from here on out — in tmpfs, and to periodically sync them back to the physical disc (HDD/SSD). It does this via a bind mounting step and an innovative use of rsync to maintain synchronization between a tmpfs copy and media-bound backups. And bestie, asd also features several crash-recovery features. she's got range. 💖

The design goals of asd, in order of slay:

  • Completely transparent user experience. (you literally won't even notice her)
  • Reduced wear to physical discs (particularly SSDs). (protect the hardware, she's doing her part 🫶)
  • Speed. (periodt.)

Since the sync targets is relocated into tmpfs (RAM disk), the corresponding onslaught of I/O associated with system usage of them is also redirected from the physical disc to RAM — thus reducing wear to the physical disc and also improving speed and responsiveness. it's giving "work smarter not harder" 💅

✨ SETUP ✨

/etc/asd.conf contains all user managed settings. Optionally another file can be used by setting the ASDCONF environment variable. iykyk.

NOTE: edits made to /etc/asd.conf while asd is running will be applied only after asd has been restarted from the init service. so like. don't be surprised when nothing changes immediately 🙈

the whole checklist, bestie:

  • At a minimum, define the sync targets to be managed by asd in the WHATTOSYNC array. Syntax below.
  • Optionally uncomment and define the location of your distro's tmpfs* in the VOLATILE variable.
  • Optionally enable the use of overlayfs to improve sync speed even further and use a smaller memory footprint. Do this in the USE_OVERLAYFS variable. Note that this option requires your kernel to be configured to use either the 'overlay' or 'overlayfs' module. See the FAQ below for additional details on this feature. (yes the quotes matter, look at the FAQ, she explains the whole module-name situation 👀)
  • Optionally disable the use of crash-recovery snapshots. Do this in the USE_BACKUPS variable.
  • Optionally define the number of crash-recovery snapshots to keep. Do this in the BACKUP_LIMIT variable.

*Note that the default value of "/tmp" should work just fine for the VOLATILE setting. If using bleachbit, do NOT invoke it with the '--clean system.tmp' switch or you will remove a key dot file (.foo) from /tmp that asd needs to keep track of sync status. 💀 Also note that using a value of "/dev/shm" can cause problems with systemd's NAMESPACE spawning only when users enable the overlayfs option.

Example:

 WHATTOSYNC=('/var/lib/monitorix' '/srv/http' '/foo/bar')

or

 WHATTOSYNC=(
 '/var/lib/monitorix'
 '/srv/http'
 '/foo/bar'
 )

🌈 RUNNING ASD 🌈

🎀 PREVIEW MODE 🎀

The preview option can be called to show users exactly what asd will do/is doing based on the entries in /etc/asd.conf, as well printout useful information such as dir size, paths, and if any recovery snapshots have been created. she's showing you her whole plan, transparency queen 👑

 $ asd p

 Anything-sync-daemon on Arch Linux.

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs v23 is currently active.

 Asd will manage the following per /run/asd.conf settings:

 owner/group id:     root/0
 target to manage:   /srv/http/serve
 sync target:        /srv/http/.serve-backup_asd
 tmpfs target:       /tmp/asd-root/srv/http/serve
 dir size:           21M
 overlayfs size:     15M
 recovery dirs:      2 <- delete with the c option
  dir path/size:     /srv/http/.serve-backup_asd-crashrecovery-20141105_124948 (17M)
  dir path/size:     /srv/http/.serve-backup_asd-crashrecovery-20150124_062311 (21M)

 owner/group id:     facade/100
 target to manage:   /home/facade/logs
 sync target:        /home/facade/.logs-backup_asd
 tmpfs target:       /tmp/asd-facadey/home/facade/logs
 dir size:           1.5M
 overlayfs size:     480K
 recovery dirs:      none

🧼 CLEAN MODE 🧼

The clean mode will delete ALL recovery snapshots that have accumulated. Run this only if you are sure that you want to delete them. fr fr, read that twice. 🫣

Note that if a sync target is owned by root or another user, and if you call asd to clean, it will throw errors based on the permissions of your sync targets.

 $ asd c

 Anything-sync-daemon on Arch Linux.

 Deleting 2 crashrecovery dirs for sync target /srv/http/serve
  /srv/http/.serve-backup_asd-crashrecovery-20141105_124948
  /srv/http/.serve-backup_asd-crashrecovery-20150124_062311

👑 START AND STOP ASD FOR SYSTEMD USERS 👑

Both a systemd service file and timer are provided, and should be used to start or stop asd. she comes with accessories 💅

The role of the timer is update the tmpfs copies back to the disk. This occurs once per hour by default. The timer is started automatically with asd.service.

 # systemctl [option] asd

Available options: start stop enable disable

🧚 START AND STOP ASD FOR USERS OF OTHER INIT SYSTEMS 🧚

For distros not using systemd, another init script should be used to manage the daemon. Examples are provided and are known to work with Upstart.

Note that for these init systems, the supplied cron script (installed to /etc/cron.hourly) will run the resync option to keep the tmpfs copies sync'ed. Of course, the target system must have cron installed and active for this to happen. no cron = no resync, that's the deal 🕰️

⭐ SUPPORTED DISTROS ⭐

At this time, the following distros are officially supported but there is no reason to think that asd will not run on another distro:

  • Arch Linux

(she's an Arch girlie first and foremost, but she's not gonna gatekeep 🌸)

💜 FAQ 💜

Q1: What is overlayfs mode?

A1: Overlayfs is a simple union file-system mainlined in the Linux kernel version 3.18.0. Starting with asd version 5.54, overlayfs can be used to reduce the memory footprint of asd's tmpfs space and to speed up sync and unsync operations. The magic is in how the overlay mount only writes out data that has changed rather than the entire sync target. See Example 1 below. The same recovery features asd uses in its default mode are also active when running in overlayfs mode. Overlayfs mode is enabled by uncommenting the USE_OVERLAYFS= in /etc/asd.conf followed by a restart of the daemon. 🔥

There are several versions of overlayfs available to the Linux kernel in production in various distros. Versions 22 and lower have a module called 'overlayfs' while newer versions (23 and higher) have a module called 'overlay' — note the lack of the 'fs' in the newer version. Asd will automatically detect the overlayfs available to your kernel if it is configured to use one of them. she does the detecting so you don't have to 😌

See the example in the PREVIEW MODE section above which shows a system using overlayfs to illustrate the memory savings that can be achieved. Note the "overlayfs size" report compared to the total "dir size" report for each sync target. Be aware that these numbers will change depending on just how much data is written to the sync target, but in common use cases, the overlayfs size will always be less than the dir size.

Q2: Why do I see directory ".foo-backup_asd" ".foo-backup_asd-old"?

A2: The way the backup process of asd works is that it creates a hard linked clone of the original directory; this is known as .foo-backup_asd-old. The other .foo-backup_asd is just a bind mount to the original directory link which is used to access the contents of the original directory for overlay purposes. two files, two jobs, both iconic 🫶

Q3: My system crashed and asd didn't sync back. What do I do?

A3: The "last good" backup of your sync targets is just fine still sitting happily on your filesystem. Upon restarting asd (on a reboot for example), a check is preformed to see if asd was exited in some corrupted state. If it is detected, asd will snapshot the "last good" backup before it rotates it back into place. Note that, since asd tries to decrease the disk usage, it never really "copies" the full contents of the directory and just uses the hardlinks to the previous files. And during the rsync step, it creates new files so that the previous hardlinks are untouched. So trying to modify the directory during the time asd is trying to backup can leave the directory in some corrupted state. 😭 don't touch it while she's working!!

Q4: Where can I find this snapshot?

A4: You will find the snapshot in the same directory as the sync target and it will contain a date-time-stamp that corresponds to the time at which the recovery took place. For example, a /foo/bar snapshot will be /foo/.bar-backup_asd-crashrecovery-20141221_070112.tar.zstd — of course, the date_time suffix will be different for you. yours will be cuter and unique 💫

Q5: How can I restore the snapshot?

A5: Follow these steps:

  1. Stop asd.
  2. Confirm that the directories created by asd is not present. If they are, asd did not stop correctly for other reasons.
  3. Move the "bad" copy of the sync taget to a backup (don't blindly delete anything).
  4. Untar the snapshot directory to the expected sync target.

Example using /foo/bar:

  1. cd /foo
  2. mv bar bar-bad
  3. tar -xvf .bar-backup_asd-crashrecovery-20141221_070112.tar.zstd

At this point, check that everything is fine with the data on /foo/bar and, if all is well, it is safe to delete the snapshot. ✅

Q6: Can asd delete the snapshots automatically?

A6: Yes, run asd with the "clean" switch to delete snapshots.

🌻 CONTRIBUTE 🌻

Users wishing to contribute to this code, should fork and send a pull request. Source is freely available on the project page linked below. come join the group chat, we'd love to have you 🫶

💀 BUGS 💀

Discover a bug? Please open an issue on the project page linked below. be nice about it please 🙏

  • Currently, asd cannot handle open files on a sync target so if a hung process has something open there, it can be messy. (and by messy we mean messy messy, so maybe close your stuff 😬)

🔗 ONLINE 🔗

🍓 AUTHOR 🍓

graysky (graysky AT archlinux DOT us)

🧚 MAINTAINER 🧚

Manorit Chawdhry (manorit2001@gmail.com)

okay that's the whole page bestie, go forth and sync 💖✨ this iconic little manual is from 26 November 2016 — yeah she's been serving since then — and it's the anything-sync-daemon(1) page, section 1, forever and always. 🎀👑💅

📦 where this page comes from