STOREBACKUP(1)

🌸 STOREBACKUP(1) — storeBackup.pl 💅

okay bestie, so this is the manpage for the Perl script storeBackup.pl — filed under User Contributed Perl Documentation — and it's literally the most extra backup tool you'll ever meet. grab a strawberry drink 🍓, we're going in.

---

✨ NAME ✨

storeBackup.pl — a fancy compressing managing checksumming hard-linking deduplicating cp -ua 🫶

like, it's cp -ua but it went to the gym, got a personality, and now remembers everything. iconic.

---

🦋 DESCRIPTION 🦋

So the whole vibe: this program copies trees to another location. that's the assignment. 💖

Every file copied is potentially compressed (peep --exceptSuffix for the deets). and every backup after the very first one compares the files against the last stored version using an md5 checksum. if they match? slay, it just makes a hard link to it — no new copy, no wasted space. ✨

It will also check mtime, ctime and size to recognize identical files in older backups very fast — like, it clocks them instantly. 🔥

And it can also backup big image files fast and efficiently on a per block basis (that's data deduplication, iykyk). 👑

Also very important bestie: you can overwrite options in the configuration file on the command line. yes. command line wins. periodt.

---

🎀 SYNOPSIS 🎀

storeBackup.pl --help

or

storeBackup.pl -g configFile

or

storeBackup.pl [-f configFile] [-s sourceDir]
      [-b backupDirectory] [-S series] [--checkCompr] [--print]
      [-T tmpdir] [-L lockFile] [--unlockBeforeDel]
      [--exceptDirs dir1] [--contExceptDirsErr]
      [--includeDirs dir1]
      [--exceptRule rule] [--includeRule rule]
      [--exceptTypes types]
      [--specialTypeArchiver archiver [--archiveTypes types]]
      [--cpIsGnu] [--linkSymlinks]
      [--precommand job] [--postcommand job]
      [--followLinks depth] [--stayInFileSystem] [--highLatency]
      [--ignorePerms] [--lateLinks [--lateCompress]] [--autorepair]
      [--checkBlocksSuffix suffix] [--checkBlocksMinSize size]
      [--checkBlocksBS] [--checkBlocksCompr check|yes|no]
      [--checkBlocksParallel] [--queueBlock]
      [--checkBlocksRule0 rule [--checkBlocksBS0 size]
       [--checkBlocksCompr0 key] [--checkBlocksRead0 filter]
       [--checkBlocksParallel0]]
      [--checkBlocksRule1 rule [--checkBlocksBS1 size]
       [--checkBlocksCompr1 key] [--checkBlocksRead1 filter]
       [--checkBlocksParallel1]]
      [--checkBlocksRule2 rule [--checkBlocksBS2 size]
       [--checkBlocksCompr2 kdey] [--checkBlocksRead2 filter]
       [--checkBlocksParallel2]]
      [--checkBlocksRule3 rule [--checkBlocksBS3 size]
       [--checkBlocksCompr3 key] [--checkBlocksRead3 filter]
       [--checkBlocksParallel3]]
      [--checkBlocksRule4 rule [--checkBlocksBS4 size]
       [--checkBlocksCompr4 key] [--checkBlocksRead4 filter]
       [--checkBlocksParallel4]]
      [--checkDevices0 list [--checkDevicesDir0]
       [--checkDevicesBS0] [checkDevicesCompr0 key]
       [--checkDevicesParallel0]]
      [--checkDevices1 list [--checkDevicesDir1]
       [--checkDevicesBS1] [checkDevicesCompr1 key]
       [--checkDevicesParallel1]]
      [--checkDevices2 list [--checkDevicesDir2]
       [--checkDevicesBS2] [checkDevicesCompr2 key]
       [--checkDevicesParallel2]]
      [--checkDevices3 list [--checkDevicesDir3]
       [--checkDevicesBS3] [checkDevicesCompr3 key]
       [--checkDevicesParallel3]]
      [--checkDevices4 list [--checkDevicesDir4]
       [--checkDevicesBS4] [checkDevicesCompr4 key]
       [--checkDevicesParallel1]]
      [--saveRAM] [-c compress] [-u uncompress] [-p postfix]
      [--noCompress number] [--queueCompress number]
      [--noCopy number] [--queueCopy number]
      [--withUserGroupStat] [--userGroupStatFile filename]
      [--exceptSuffix suffixes] [--addExceptSuffix suffixes]
      [--compressSuffix] [--minCompressSize size] [--comprRule]
      [--doNotCompressMD5File] [--chmodMD5File] [-v]
      [-d level] [--progressReport number[,timeframe]]
      [--ignoreReadError] [--suppressInfo]
      [--suppressWarning key] [--linkToRecent name]
      [--doNotDelete] [--deleteNotFinishedDirs] [--maxHardLinks]
      [--resetAtime] [--keepAll timePeriod] [--keepWeekday entry]
      [[--keepFirstOfYear] [--keepLastOfYear]
       [--keepFirstOfMonth] [--keepLastOfMonth]
       [--firstDayOfWeek day] [--keepFirstOfWeek]
       [--keepLastOfWeek] [--keepDuplicate] [--keepMinNumber]
       [--keepMaxNumber]
        | [--keepRelative] ]
      [-l logFile
       [--plusLogStdout] [--suppressTime] [-m maxFilelen]
       [[-n noOfOldFiles] | [--saveLogs]]
       [--compressWith compressprog]]
      [--logInBackupDir [--compressLogInBackupDir]
       [--logInBackupDirFileName logFile]]
      [otherBackupSeries ...]

ngl that's a lot of brackets, but we're gonna get through every single one together, bestie. 🫶

---

💜 OPTIONS 💜

okay so here's the full menu. no skipping, no summarizing — this is the whole thing, fr fr.

• --help — show this help 🙈
• --generate, -g — generate a template of the configuration file ✨
• --checkCompr, -C — check compression for all files bigger than 1k to check if it makes sense to compress them. heads up: this overwrites the options exceptSuffix, addExceptSuffix, minCompressSize, comprRule 😭
• --print — print configuration read from configuration file or command line, and then stop. like, it just shows you what it read and peaces out. 👋
• --file, -f — configuration file (instead of or additionally to options on the command line) 🎀
• --sourceDir, -s — source directory (must exist) 🦋
• --backupDir, -b — top level directory of all backups (must exist) 💖
• --series, -S — series directory, default is 'default'. it's a relative path from backupDir ⭐
• --tmpdir, -T — directory for temporary files, default is </tmp> 🧚
• --lockFile, -L — lock file. if it exists, new instances will finish if an old one is already running. default is $lockFile. okay so important caveat: this type of lock file does not work across multiple servers, and it's not designed to separate storeBackup.pl and storeBackupUpdateBackup.pl or any other storeBackup process in a separate PID space. also, default lock file for root is in /var/lock, for others in $HOME 🔒
• --unlockBeforeDel — remove the lock file before deleting old backups. default is to delete the lock file after removing old backups 🗝️
• --exceptDirs, -e — directories to except from backing up (relative path). wildcards are possible and should be quoted to avoid replacements by the shell. use this parameter multiple times for multiple directories 🚫
• --contExceptDirsErr — continue if one or more of the exceptional directories do not exist (default is to stop processing) 🙏
• --includeDirs, -i — directories to include in the backup (relative path). wildcards are possible and have to be quoted. use this parameter multiple times for multiple directories ✅
• --exceptRule — Files to exclude from backing up. see README: 'including / excluding files and directories' 📖
• --includeRule — Files to include in the backup — like exceptRule. see README: 'including / excluding files and directories' 📖
• --writeExcludeLog — write a file name .storeBackup.notSaved.bz2 with the names of all skipped files 📝
• --exceptTypes — do not save the specified type of files, allowed: Sbcfpl
  • S - file is a socket
  • b - file is a block special file
  • c - file is a character special file
  • f - file is a plain file
  • p - file is a named pipe
  • l - file is a symbolic link
  • Sbc can only be saved when using option [cpIsGnu] 💀
• --archiveTypes — save the specified type of files in an archive instead saving them directly in the file system. use this if you want to backup those file types but your target file or transport (eg. sshfs or non gnu-cp) system does not support those types of files 📦
  • S - file is a socket
  • b - file is a block special file
  • c - file is a character special file
  • p - file is a named pipe
  • you also have to set --specialTypeArchiver when using this option ⚠️
• --specialTypeArchiver — possible values are 'cpio' or 'tar'. default is 'cpio'. tar is not able to archive sockets. cpio is not part of the actual posix standard any more 😬
• --cpIsGnu — Activate this option if your system's cp is a full-featured GNU version. in this case you will be able to also backup several special file types like sockets 🔥
• --linkSymlinks — hard link identical symlinks 🔗
• --precommand — exec job before starting the backup, checks lockFile (-L) before starting (e.g. can be used for rsync). stops execution if job returns exit status != 0. also, this parameter is parsed like a line in the configuration file and normally has to be quoted 🎬
• --postcommand — exec job after finishing the backup, but before erasing of old backups. reports if job returns exit status != 0. this parameter is also parsed like a line in the configuration file and normally has to be quoted 🎬
• --followLinks — follow symbolic links like directories up to depth. default = 0 -> do not follow links 🪢
• --stayInFileSystem — only store the contents of file systems named by --sourceDir and symlinked via --followLinks 🏠
• --highLatency — use this for a very high latency line (eg. vpn over the internet) for better parallelization 🐌➡️🚀
• --ignorePerms — If this option is selected, files will not necessarily have the same permissions and owner as the originals. this speeds up backups on network drives a lot. recovery with storeBackupRecover.pl will restore them correctly 🫶
• --lateLinks — do not write hard links to existing files in the backup during the backup. you have to call the program storeBackupWriteLateLink.pl later on your server if you set this flag to 'yes'. you have to run storeBackupUpdateBackup.pl later — see description for that program ⏳

---

and that's part 1, bestie — the rest is coming, don't go anywhere 💖✨

• --lateCompress — only does anything when you also pass --lateLinks. With this on, compression for files >= minCompressSize gets postponed to later — the file gets (temporarily) copied into the backup first, and the squeezing happens afterwards. ✨
• --autorepair, -a — fixes simple inconsistencies (the ones coming from lateLinks) all by itself, no questions asked, no "hey bestie can you click yes". It just handles it. 🫶
• --checkBlocksSuffix — files with this suffix get an md5 check on blocks of that file from storeBackup. Runs after --checkBlocksRule(n). You can repeat this option as many times as you want. 💅
• --checkBlocksMinSize — only bother checking the files named in --checkBlocksSuffix if their file size is at least this value. Default is 100M.
• --checkBlocksBS — block size for the files named in --checkBlocksSuffix. Default is $checkBlocksBSdefault (that's 1 megabyte, iconically).
• --checkBlocksCompr — if set, the blocks generated because of checkBlocksSuffix get compressed. Default is 'no'. If you set it to 'check', it tries to estimate whether compression actually helps. 🎀
• --checkBlocksParallel — reads the files named here in parallel with the "normal" ones. This only makes sense if they're on a different disk, fr. Default value is 'no'.
• --queueBlock — how long the queue is for storing files before block checking. Default = $queueBlock.

---

the numbered block-check rules (0 through 4, bestie, there are five of each) 🦋

Here's how the set works — 0 gets the full explanation, and 1 through 4 are literally the same thing with a bigger number glued on:

• --checkBlocksRule0 — files for which storeBackup will make an md5 check depending on blocks of that file.
• --checkBlocksBS0 — block size for the checkBlocksRule option. Default is $checkBlocksBSdefault (1 megabyte).
• --checkBlocksCompr0 — if set, the blocks generated because of this rule get compressed.
• --checkBlocksRead0 — a filter for reading the file, so it gets treated as a blocked file. e.g. 'gzip -d' if the file is compressed. Default is no read filter. Heads up: this parameter is parsed like the line in the configuration file, so normally you have to quote it, e.g. 'gzip -9'.
• --checkBlocksParallel0 — reads the files named here in parallel with the "normal" ones. Only makes sense if they're on a different disk. Default value is 'no'.

And then the rest of the squad, same energy, just different indices:

• --checkBlocksRule1, --checkBlocksBS1, --checkBlocksCompr1, --checkBlocksRead1, --checkBlocksParallel1
• --checkBlocksRule2, --checkBlocksBS2, --checkBlocksCompr2, --checkBlocksRead2, --checkBlocksParallel2
• --checkBlocksRule3, --checkBlocksBS3, --checkBlocksCompr3, --checkBlocksRead3, --checkBlocksParallel3
• --checkBlocksRule4, --checkBlocksBS4, --checkBlocksCompr4, --checkBlocksRead4, --checkBlocksParallel4

---

device checks, aka md5-ing your actual disks 💀

• --checkDevices0 — list of devices to md5 check, depending on blocks of these devices (e.g. /dev/sdb or /dev/sdb1).
• --checkDevicesDir0 — the directory where the backup of the device gets stored.
• --checkDevicesBS0 — block size of the checkDevices0 option. Default is 1M (1 megabyte).
• --checkDevicesCompr0 — compress the blocks resulting from checkDevices0. Possible values are 'check', 'yes' or 'no', default is 'no'.
• --checkDevicesParallel0 — read the devices named here in parallel to the rest of the backup. Only makes sense if they're on a different disk. Default value is 'no'.

And the matching quartet, 1 through 4, exactly like above:

• --checkDevices1, --checkDevicesDir1, --checkDevicesBS1, --checkDevicesCompr1, --checkDevicesParallel1
• --checkDevices2, --checkDevicesDir2, --checkDevicesBS2, --checkDevicesCompr2, --checkDevicesParallel2
• --checkDevices3, --checkDevicesDir3, --checkDevicesBS3, --checkDevicesCompr3, --checkDevicesParallel3
• --checkDevices4, --checkDevicesDir4, --checkDevicesBS4, --checkDevicesCompr4, --checkDevicesParallel4

---

• --saveRAM — writes the temporary dbm files into --tmpdir. Use this if you don't have enough RAM, no cap. 😭
• --compress, -c — the compress command (with its options). Default is <bzip2>. This parameter is parsed like the line in the configuration file, so normally it has to be quoted, e.g. 'gzip -9'.
• --uncompress, -u — the uncompress command (with options). Default is <bzip2 -d>. Also parsed like a config file line, so quote it, e.g. 'gzip -d'.
• --postfix, -p — postfix tacked on after compression. Default is <.bz2>.
• --exceptSuffix — nope, don't compress files with these suffixes (uppercase included, bestie — the case matters!). The defaults are:
  • '\.zip'
  • '\.bz2'
  • '\.gz'
  • '\.tgz'
  • '\.jpg'
  • '\.gif'
  • '\.tiff?'
  • '\.mpeg'
  • '\.mpe?g'
  • '\.mpe?[34]'
  • '\.ogg'
  • '\.gpg'
  • '\.png'
  • '\.lzma'
  • '\.xz'
  • '\.mov'

You can repeat this option multiple times. And if you want no compression at all, just set this option to '.*' and watch the chaos unfold. 💅

• --addExceptSuffix — like --exceptSuffix, except it doesn't replace the defaults, it adds to them. Iconic behaviour, honestly.
• --compressSuffix — like --exceptSuffix, but the files you mention here do get compressed. If you pick this option, then any files not touched by exceptSuffix, addExceptSuffix or these suffixes get rated by the rule function COMPRESSION_CHECK to decide whether to compress or not. 🧚
• --minCompressSize — files smaller than this size will never be compressed, they just get copied. Periodt.
• --comprRule — an alternative to --exceptSuffix, compressSuffix and minCompressSize: it's a definition of a rule for which files get compressed.
• --noCompress — max number of parallel compress operations. Default = chosen automatically. ✨
• --queueCompress — length of the queue to store files before compression. Default = 1000.
• --noCopy — max number of parallel copy operations. Default = 1.
• --queueCopy — length of the queue to store files before copying. Default = 1000.
• --withUserGroupStat — writes statistics about used space into the log file.
• --userGroupStatFile — writes statistics about used space into name file. Heads up: this gets overridden every single time. 🙈
• --doNotCompressMD5File — do not compress .md5CheckSumFile.
• --chmodMD5File — permissions of .md5CheckSumFile and the corresponding .storeBackupLinks directory. Default is 0600.
• --verbose, -v — verbose messages. She's chatty. 🗣️
• --debug, -d — generate debug messages. The levels are 0 (none, default), 1 (some) and 2 (many) messages, especially in --exceptRule and --includeRule.
• --resetAtime — resets the access time in the source directory — but bestie, be aware this will change ctime (time of last modification of file status information). Not a vibe if you're relying on that.
• --doNotDelete — check only, do not delete any backup. Look but don't touch. 👀
• --deleteNotFinishedDirs — delete old backups which weren't finished. This only happens if doNotDelete is set.
• --maxHardLinks — maximum number of hard links to use. Zero (the default) means the limit depends on the file system you're using, and only that.

---

the retention saga, aka how long we keep the girlies 🌸

• --keepAll — keeps backups that aren't older than the amount of time you specify. It's basically a default value for all the days in --keepWeekday. Deletion begins at the end of the script. The time range has to be in 'dhms' format, e.g. 10d4h means 10 days and 4 hours. Default = 20d.
• --keepWeekday — keeps backups for the specified days for the specified amount of time. It overwrites the default values chosen in --keepAll. So 'Mon,Wed:40d Sat:60d10m' means:
  • keep backups from Mon and Wed 40days + 5mins
  • keep backups from Sat 60days + 10mins
  • keep backups from the rest of the days like specified in --keepAll (default $keepAll)

And if you also use the 'archive flag', it means not deleting the affected directories via --keepMaxNumber. So a10d4h means 10 days and 4 hours and 'archive flag'. For example, 'Mon,Wed:a40d5m Sat:60d10m' means:

• keep backups from Mon and Wed 40days + 5mins + 'archive'
• keep backups from Sat 60days + 10mins
• keep backups from the rest of the days like specified in --keepAll (default 30d)

• --keepFirstOfYear — do not delete the first backup of a year. Format is timePeriod with a possible 'archive flag'. 👑
• --keepLastOfYear — do not delete the last backup of a year. Format is timePeriod with a possible 'archive flag'.
• --keepFirstOfMonth — do not delete the first backup of a month. Format is timePeriod with a possible 'archive flag'.
• --keepLastOfMonth — do not delete the last backup of a month. Format is timePeriod with a possible 'archive flag'.
• --firstDayOfWeek — default: 'Sun'. This value is used for calculating --keepFirstOfWeek and --keepLastOfWeek. 🗓️
• --keepFirstOfWeek — do not delete the first backup of a week. Format is timePeriod with a possible 'archive flag'.
• --keepLastOfWeek — do not delete the last backup of a week. Format is timePeriod with a possible 'archive flag'.
• --keepDuplicate — keeps multiple backups of one day up to timePeriod. Format is timePeriod, and 'archive flag' is not possible here. Default = 7d.
• --keepMinNumber — keep that minimum of backups. Multiple backups of one day are counted as one backup. Default is 10.
• --keepMaxNumber — tries to keep only that maximum of backups. If you have more backups than that, this exact deletion sequence happens:
  1. delete all duplicates of a day, beginning with the old ones, except the last of every day
  2. if that's not enough, delete the rest of the backups beginning with the oldest — but never a backup with the 'archive flag', and never the last backup. 🫶
• --keepRelative, -R — this is the alternate deletion scheme, and the moment you use it, all the other keep options are ignored. It preserves backups based on their relative age, like this:

-R '1d 7d 61d 92d'

will (try to) make sure there’s always:

• one backup between 1 day and 7 days old
• one backup between 5 days and ~2 months old
• one backup between ~2 months and ~3 months old

If there’s no backup for a specified timespan (e.g. because the last backup was done more than 2 weeks ago), the next older backup gets used for that timespan. Lowkey clever, ngl.

• --progressReport, -P — prints a progress report after each number of files. You can also add a time frame after which a message gets printed. If you want a report every 1000 files and after one minute and 10 seconds, use: -P 1000,1m10s. Obsessed with a progress update, fr.
• --printDepth, -D — prints the depth of the actual read directory during backup. It’s giving “watch me go deeper” energy.
• --ignoreReadError — tells it to ignore read errors in the source directory. Not-readable directories do not cause storeBackup.pl to stop processing. Slay, resilience.
• --suppressWarning — suppresses (unwanted) warnings in the log files. To suppress warnings, these keys are available:
  • excDir — suppresses the warning that excluded directories do not exist
  • fileChange — suppresses the warning that a file has changed during the backup
  • crSeries — suppresses the warning that storeBackup had to create the 'default' series
  • hashCollision — suppresses the warning if a possible hash collision is detected
  • fileNameWithLineFeed — suppresses the warning if a filename contains a line feed
  • use_DB_File — suppresses the warning that you should install perl module DB_File for better performance
  • use_MLDBM — suppresses the warning that you should install perl module MLDBM if you want to use rule functions MARK_DIR or MARK_DIR_REC together with option saveRAM
  • use_IOCompressBzip2 — suppresses the warning that you should install perl module IO::Compress::Bzip2 for better performance
  • noBackupForPeriod — suppresses the warning that there are no backups for certain periods when using option keepRelative

This option can be repeated multiple times on the command line. Periodt.

• --suppressInfo — suppresses (unwanted) infos in the log files. To suppress infos, these keys are available:
  • readCheckSums — suppresses the information that checksums of blocked files are read

This option can be repeated multiple times on the command line. Cute and consistent.

• --linkToRecent — after a successful backup, sets a symbolic link to that backup and deletes existing older links with the same name. Main character link behavior, honestly.
• --logFile, -l — the log file. Default is STDOUT. Very “tell me everything” of it.
• --plusLogStdout — if you specify a log file with --logFile, you can also print the output to STDOUT with this flag. Double output, double vibe.
• --suppressTime — suppresses output of time in the logfile. No timestamps, no stress.
• --maxFilelen, -m — maximal length of the log file, default = 1e6. Keep it under control, bestie.
• --noOfOldFiles, -n — number of old log files, default = 5. A little log-file family.
• --saveLogs — saves log files with date and time instead of deleting the old ones (with [-noOfOldFiles]). Archivists rise.
• --compressWith — compresses saved log files (e.g. with 'gzip -9'). Default is 'bzip2'. This parameter is parsed like a line in the configuration file and normally has to be quoted. Very “read the room, quote it” energy.
• --logInBackupDir — writes the log file (also) in the backup directory.

Be aware that this log does not contain all error messages of the one specified with --logFile! Not me warning you affectionately 😭

• --compressLogInBackupDir — compresses the log file in the backup directory. Smaller log, same lore.
• --logInBackupDirFileName — filename to use for writing the above log file. Default is .storeBackup.log. Iconic hidden-file moment.
• otherBackupSeries — list of other backup series to consider for hard linking. Relative path from backupDir! Format (examples):

backupSeries/2002.08.29_08.25.28 -> consider this backup
or
0:backupSeries ->last (youngest) in <backupDir>/backupSeries
1:backupSeries ->one before last in <backupDir>/backupSeries
n:backupSeries ->
  n'th before last in <backupDir>/backupSeries
3-5:backupSeries ->
  3rd, 4th and 5th in <backupDir>/backupSeries
all:backupSeries -> all in <backupDir>/backupSeries

You can also use wildcards in series names. See documentation, section 'Using Wildcards for Replication' for details. Default is to link to the last backup in every series. Fr fr, the linking possibilities are endless.

---

💖 COPYRIGHT 💖

Copyright (c) 2000-2022 by Heinz-Josef Claes (see README). Published under the GNU General Public License, either version 3 of the License, or (at your option) any later version. Respect the license, bestie 🫶

This iconic page is STOREBACKUP(1) from perl v5.34.0, dated 2022-04-18 — go back up your fave files and stay hydrated 💅✨