STOREBACKUP(1)

📦
version 3.5.2 · 4 versions
, in ubuntu stonking

✨ STOREBACKUP(1) ✨

the User Contributed Perl Documentation girlies know 🫶

okay so bestie, gather round, because we are about to talk about a script that is genuinely iconic and I need you to be seated for it 💅

🎀 NAME 🎀

storeBackup.pl — it's literally a fancy compressing, managing, checksumming, hard-linking, deduplicating cp -ua ✨ no cap, that's the whole vibe.

🌸 DESCRIPTION 🌸

okay so here's the tea. This program copies trees to another location. Every single file it copies is potentially compressed (peep --exceptSuffix for the deets 👀).

The backups after the first backup will compare the files with an md5 checksum against the last stored version. If they're equal, it literally just makes a hard link to it. That's it. That's the whole flex. 🔥

It will also check mtime, ctime and size to recognize identical files in older backups very fast — like, speedrun fast, no thoughts head empty fast 💀

And get this: it can also backup big image files fast and efficiently on a per block basis (that's data deduplication, bestie, iykyk 👑).

Also you can overwrite options in the configuration file on the command line. Main character energy honestly.

✨ SYNOPSIS ✨

        storeBackup.pl --help
or
        storeBackup.pl -g configFile
or
        storeBackup.pl [-f configFile] [-s sourceDir]
              [-b backupDirectory] [-S series] [--checkCompr] [--print]
              [-T tmpdir] [-L lockFile] [--unlockBeforeDel]
              [--exceptDirs dir1] [--contExceptDirsErr]
              [--includeDirs dir1]
              [--exceptRule rule] [--includeRule rule]
              [--exceptTypes types]
              [--specialTypeArchiver archiver [--archiveTypes types]]
              [--cpIsGnu] [--linkSymlinks]
              [--precommand job] [--postcommand job]
              [--followLinks depth] [--stayInFileSystem] [--highLatency]
              [--ignorePerms] [--lateLinks [--lateCompress]] [--autorepair]
              [--checkBlocksSuffix suffix] [--checkBlocksMinSize size]
              [--checkBlocksBS] [--checkBlocksCompr check|yes|no]
              [--checkBlocksParallel] [--queueBlock]
              [--checkBlocksRule0 rule [--checkBlocksBS0 size]
               [--checkBlocksCompr0 key] [--checkBlocksRead0 filter]
               [--checkBlocksParallel0]]
              [--checkBlocksRule1 rule [--checkBlocksBS1 size]
               [--checkBlocksCompr1 key] [--checkBlocksRead1 filter]
               [--checkBlocksParallel1]]
              [--checkBlocksRule2 rule [--checkBlocksBS2 size]
               [--checkBlocksCompr2 kdey] [--checkBlocksRead2 filter]
               [--checkBlocksParallel2]]
              [--checkBlocksRule3 rule [--checkBlocksBS3 size]
               [--checkBlocksCompr3 key] [--checkBlocksRead3 filter]
               [--checkBlocksParallel3]]
              [--checkBlocksRule4 rule [--checkBlocksBS4 size]
               [--checkBlocksCompr4 key] [--checkBlocksRead4 filter]
               [--checkBlocksParallel4]]
              [--checkDevices0 list [--checkDevicesDir0]
               [--checkDevicesBS0] [checkDevicesCompr0 key]
               [--checkDevicesParallel0]]
              [--checkDevices1 list [--checkDevicesDir1]
               [--checkDevicesBS1] [checkDevicesCompr1 key]
               [--checkDevicesParallel1]]
              [--checkDevices2 list [--checkDevicesDir2]
               [--checkDevicesBS2] [checkDevicesCompr2 key]
               [--checkDevicesParallel2]]
              [--checkDevices3 list [--checkDevicesDir3]
               [--checkDevicesBS3] [checkDevicesCompr3 key]
               [--checkDevicesParallel3]]
              [--checkDevices4 list [--checkDevicesDir4]
               [--checkDevicesBS4] [checkDevicesCompr4 key]
               [--checkDevicesParallel1]]
              [--saveRAM] [-c compress] [-u uncompress] [-p postfix]
              [--noCompress number] [--queueCompress number]
              [--noCopy number] [--queueCopy number]
              [--withUserGroupStat] [--userGroupStatFile filename]
              [--exceptSuffix suffixes] [--addExceptSuffix suffixes]
              [--compressSuffix] [--minCompressSize size] [--comprRule]
              [--doNotCompressMD5File] [--chmodMD5File] [-v]
              [-d level] [--progressReport number[,timeframe]]
              [--ignoreReadError] [--suppressInfo]
              [--suppressWarning key] [--linkToRecent name]
              [--doNotDelete] [--deleteNotFinishedDirs] [--maxHardLinks]
              [--resetAtime] [--keepAll timePeriod] [--keepWeekday entry]
              [[--keepFirstOfYear] [--keepLastOfYear]
               [--keepFirstOfMonth] [--keepLastOfMonth]
               [--firstDayOfWeek day] [--keepFirstOfWeek]
               [--keepLastOfWeek] [--keepDuplicate] [--keepMinNumber]
               [--keepMaxNumber]
                | [--keepRelative] ]
              [-l logFile
               [--plusLogStdout] [--suppressTime] [-m maxFilelen]
               [[-n noOfOldFiles] | [--saveLogs]]
               [--compressWith compressprog]]
              [--logInBackupDir [--compressLogInBackupDir]
               [--logInBackupDirFileName logFile]]
              [otherBackupSeries ...]

(and yes bestie, I know that kdey in there looks like a typo. I am reporting, not fixing. It's not my job. She stays. 💀)

💖 OPTIONS 💖

okay this is the long part, grab a snack 🍓

  • --help — show this help 🫶
  • --generate, -g — generate a template of the configuration file ✨ like, auto-magic setup, we love her
  • --checkCompr, -C — check compression for all files bigger than 1k to check if it makes sense to compress them 🌈 overwrites options:
    • exceptSuffix
    • addExceptSuffix
    • minCompressSize
    • comprRule
  • --print — print configuration read from configuration file or command line, and then stop 🛑 she said that's enough
  • --file, -f — configuration file (instead of, or additionally to, options on the command line)
  • --sourceDir, -s — source directory (must exist 💀 no imaginary dirs on her watch)
  • --backupDir, -b — top level directory of all backups (also must exist, she does not create it for you bestie)
  • --series, -S — series directory, default is 'default' ✨ it's a relative path from backupDir btw
  • --tmpdir, -T — directory for temporary files, default is </tmp> 🧚
  • --lockFile, -L — lock file. if it exists, new instances will finish if an old one is already running. default is $lockFile 🔒
heads up bestie: this type of lock file does not work across multiple servers, and it is not designed to separate storeBackup.pl and storeBackupUpdateBackup.pl or any other storeBackup process in a separate PID space. like, she is doing her best but she is not doing that.

Default lock file for root is in /var/lock, for others in \$HOME 🏠

  • --unlockBeforeDel — remove the lock file before deleting old backups 🔓 default is to delete the lock file after removing old backups
  • --exceptDirs, -e — directories to except from backing up (relative path). wildcards are possible and should be quoted to avoid replacements by the shell 🐚 use this parameter multiple times for multiple directories
  • --contExceptDirsErr — continue if one or more of the exceptional directories do not exist (default is to stop processing 🛑)
  • --includeDirs, -i — directories to include in the backup (relative path). wildcards are possible and have to be quoted 🌸 use this parameter multiple times for multiple directories
  • --exceptRule — Files to exclude from backing up 📝 see README: 'including / excluding files and directories'
  • --includeRule — Files to include in the backug up — like exceptRule (yes bestie, "backug". she said what she said, and what she said stays 💅) see README: 'including / excluding files and directories'
  • --writeExcludeLog — write a file name .storeBackup.notSaved.bz2 with the names of all skipped files 📄 byeee to those files, hello to receipts
  • --exceptTypes — do not save the specified type of files, allowed: Sbcfpl 👀
    • S — file is a socket
    • b — file is a block special file
    • c — file is a character special file
    • f — file is a plain file
    • p — file is a named pipe
    • l — file is a symbolic link
    • Sbc can only be saved when using option [cpIsGnu] 😤
  • --archiveTypes — save the specified type of files in an archive instead saving them directly in the file system 🗄️ use this if you want to backup those file types but your target file or transport (eg. sshfs or non gnu-cp) system does not support those types of files
    • S — file is a socket
    • b — file is a block special file
    • c — file is a character special file
    • p — file is a named pipe you also have to set --specialTypeArchiver when using this option ⚠️
  • --specialTypeArchiver — possible values are 'cpio' or 'tar'. default is 'cpio' 🎀 tar is not able to archive sockets 💀 cpio is not part of the actual posix standard any more 😭
  • --cpIsGnu — Activate this option if your system's cp is a full-featured GNU version. In this case you will be able to also backup several special file types like sockets 🔥
  • --linkSymlinks — hard link identical symlinks 🔗
  • --precommand — exec job before starting the backup, checks lockFile (-L) before starting (e.g. can be used for rsync) ⏮️ stops execution if job returns exit status != 0 🛑 This parameter is parsed like a line in the configuration file and normally has to be quoted ✍️
  • --postcommand — exec job after finishing the backup, but before erasing of old backups ⏭️ reports if job returns exit status != 0 📢 This parameter is parsed like a line in the configuration file and normally has to be quoted ✍️
  • --followLinks — follow symbolic links like directories up to depth 🐇 default = 0 → do not follow links
  • --stayInFileSystem — only store the contents of file systems named by --sourceDir and symlinked via --followLinks 🏡 stay home bestie, no wandering
  • --highLatency — use this for a very high latency line (eg. vpn over the internet) for better parallelization 🌐 yes queen, optimize that sad connection
  • --ignorePerms — If this option is selected, files will not necessarily have the same permissions and owner as the originals ⚠️ This speeds up backups on network drives a lot 🚀 Recovery with storeBackupRecover.pl will restore them correctly ✅ so it's fine, it's slay, don't panic
  • --lateLinks — do not write hard links to existing files in the backup during the backup 😳 you have to call the program storeBackupWriteLateLink.pl later on your server if you set this flag to 'yes' 📞 she will be waiting you have to run storeBackupUpdateBackup.pl later — see description for that program 📖
  • --lateCompress — only does anything when you also pass --lateLinks, bestie. With this on, compressing files that are >= minCompressSize gets pushed to later, and the file is (temporarily) copied into the backup in the meantime. ✨
  • --autorepair, -a — fixes simple inconsistencies (the ones that come from lateLinks) all by itself, no "hey bestie can I do this?" prompt. it's giving self-sufficient queen 💅
  • --checkBlocksSuffix — files with this suffix get an md5 check on blocks of that file from storeBackup. This one runs after --checkBlocksRule(n). you can repeat this option as many times as you want 🎀
  • --checkBlocksMinSize — only bother checking the files named in --checkBlocksSuffix if their file size is at least this value. default is 100M. 🫶
  • --checkBlocksBS — block size for the files you named with --checkBlocksSuffix. default is $checkBlocksBSdefault (that's 1 megabyte). 🌸
  • --checkBlocksCompr — if you set it, the blocks made because of checkBlocksSuffix get compressed. default is 'no'. set it to 'check' and it tries to estimate whether compression actually helps. lowkey smart 🧠
  • --checkBlocksParallel — reads the files you named here in parallel to the "normal" ones. heads up: this only makes sense if they live on a different disk. default value is 'no'. 💀
  • --queueBlock — how long the queue is for storing files before block checking. default = $queueBlock. 👑
  • --checkBlocksRule0 — the files that storeBackup will do an md5 check on, decided based on blocks of that file. 🌈
  • --checkBlocksBS0 — block size for the checkBlocksRule option. default is $checkBlocksBSdefault (that's 1 megabyte). ⭐
  • --checkBlocksCompr0 — if you set it, the blocks made because of this rule get compressed. 🎀
  • --checkBlocksRead0 — a filter for reading the file so it's treated as a blocked file. example: 'gzip -d' if the file is compressed. default is no read filter. this parameter gets parsed just like a line in the configuration file, so normally you gotta quote it, e.g. 'gzip -9'. ✨
  • --checkBlocksParallel0 — reads the files you specified here in parallel to the "normal" ones. again: only makes sense on a different disk. default value is 'no'. 🦋
  • --checkBlocksRule1, --checkBlocksBS1, --checkBlocksCompr1, --checkBlocksRead1, --checkBlocksParallel1 — literally the exact same set as the 0 versions above, just for rule 1. 💖
  • --checkBlocksRule2, --checkBlocksBS2, --checkBlocksCompr2, --checkBlocksRead2, --checkBlocksParallel2 — same deal, rule 2. 🌸
  • --checkBlocksRule3, --checkBlocksBS3, --checkBlocksCompr3, --checkBlocksRead3, --checkBlocksParallel3 — same deal, rule 3. 🎀
  • --checkBlocksRule4, --checkBlocksBS4, --checkBlocksCompr4, --checkBlocksRead4, --checkBlocksParallel4 — same deal, rule 4. that's the whole set, no cap 🫶
  • --checkDevices0 — a list of devices to md5 check, based on blocks of those devices (think eg. /dev/sdb or eg. /dev/sdb1). 👑
  • --checkDevicesDir0 — the directory where the backup of the device gets stored. 🌈
  • --checkDevicesBS0 — block size for the checkDevices0 option. default is 1M (1 megabyte). ⭐
  • --checkDevicesCompr0 — compress the blocks that come out of the checkDevices0 option. possible values are 'check', 'yes' or 'no', default is 'no'. 💅
  • --checkDevicesParallel0 — read the devices you named in parallel to the rest of the backup. once more: only makes sense if they're on a different disk. default value is 'no'. 💀
  • --checkDevices1, --checkDevicesDir1, --checkDevicesBS1, --checkDevicesCompr1, --checkDevicesParallel1 — the 0 crew again, but wearing a 1 bracelet. 🦋
  • --checkDevices2, --checkDevicesDir2, --checkDevicesBS2, --checkDevicesCompr2, --checkDevicesParallel2 — same, number 2. 🎀
  • --checkDevices3, --checkDevicesDir3, --checkDevicesBS3, --checkDevicesCompr3, --checkDevicesParallel3 — same, number 3. ✨
  • --checkDevices4, --checkDevicesDir4, --checkDevicesBS4, --checkDevicesCompr4, --checkDevicesParallel4 — same, number 4. and that's all five slots bestie 🌸
  • --saveRAM — writes the temporary dbm files into --tmpdir instead. use this when you do not have enough RAM, like a true survivor 😭
  • --compress, -c — the compress command (with its options). default is <bzip2>. this parameter gets parsed just like a line in the configuration file, so normally you have to quote it, e.g. 'gzip -9'. 🔥
  • --uncompress, -u — the uncompress command (with its options). default is <bzip2 -d>. same as above: parsed like a config file line, normally needs quotes, eg. 'gzip -d'. 💜
  • --postfix, -p — the postfix to slap on after compression. default is <.bz2>. 🎀
  • --exceptSuffix — do not compress files with the following suffix (uppercase included, periodt): '\.zip', '\.bz2', '\.gz', '\.tgz', '\.jpg', '\.gif', '\.tiff?', '\.mpeg', '\.mpe?g', '\.mpe?[34]', '\.ogg', '\.gpg', '\.png', '\.lzma', '\.xz', '\.mov'. you can repeat this option multiple times. and if you do not want any compression at all, set this option to '.*'. 🦋
  • --addExceptSuffix — like --exceptSuffix, but it does not replace the defaults, it adds to them. 🫶
  • --compressSuffix — like --exceptSuffix, except the files you mention here will be compressed. and heads up: if you pick this option, then files that aren't touched by exceptSuffix, addExceptSuffix or these suffixes get rated by the rule function COMPRESSION_CHECK to decide whether to compress or not. 👀
  • --minCompressSize — files smaller than this size will never be compressed, just copied. no notes ⭐
  • --comprRule — an alternative to --exceptSuffix, compressSuffix and minCompressSize: you define a rule for which files will be compressed. 🌈
  • --noCompress — the maximal number of parallel compress operations. default = chosen automatically. 💅
  • --queueCompress — length of the queue where files hang out before compression. default = 1000. 🎀
  • --noCopy — the maximal number of parallel copy operations. default = 1. 🌸
  • --queueCopy — length of the queue where files hang out before being copied. default = 1000. ✨
  • --withUserGroupStat — writes statistics about used space into the log file. 👑
  • --userGroupStatFile — writes statistics about used space into a name file — and yes, it will be overridden each time. 🙈
  • --doNotCompressMD5File — don't compress .md5CheckSumFile. 💜
  • --chmodMD5File — the permissions of .md5CheckSumFile and the corresponding .storeBackupLinks directory. default is 0600. 🔥
  • --verbose, -v — verbose messages. she's chatty, we love her 🗣️
  • --debug, -d — generates debug messages. levels are 0 (none, default), 1 (some), 2 (many) messages, especially in --exceptRule and --includeRule. 🐛
  • --resetAtime — resets the access time in the source directory — but plot twist, this will change ctime (time of last modification of file status information). 🫢
  • --doNotDelete — check only, do not delete any backup. 🛑
  • --deleteNotFinishedDirs — deletes old backups which have not been finished. heads up: this will only happen if doNotDelete is set. 💀
  • --maxHardLinks — maximum number of hard links to use. zero (default) means the limit depends on the file system used only. ⭐
  • --keepAll — keeps backups which are not older than the specified amount of time. think of it like a default value for all days in --keepWeekday. deleting begins at the end of the script. the time range has to be specified in format 'dhms', e.g.
10d4h

means 10 days and 4 hours. default = 20d. 💖

  • --keepWeekday — keeps backups for the specified days for the specified amount of time. overwrites the default values chosen in --keepAll. 'Mon,Wed:40d Sat:60d10m' means:
    • keep backups from Mon and Wed 40days + 5mins
    • keep backups from Sat 60days + 10mins
    • keep backups from the rest of the days like specified in --keepAll (default $keepAll)

and if you also use the 'archive flag' it means to not delete the affected directories via --keepMaxNumber:

  • a10d4h means 10 days and 4 hours and 'archive flag'

e.g. 'Mon,Wed:a40d5m Sat:60d10m' means:

  • keep backups from Mon and Wed 40days + 5mins + 'archive'
  • keep backups from Sat 60days + 10mins
  • keep backups from the rest of the days like specified in --keepAll (default 30d) 🗓️
  • --keepFirstOfYear — do not delete the first backup of a year. format is timePeriod with possible 'archive flag'. 🎊
  • --keepLastOfYear — do not delete the last backup of a year. format is timePeriod with possible 'archive flag'. 🎆
  • --keepFirstOfMonth — do not delete the first backup of a month. format is timePeriod with possible 'archive flag'. 🌙
  • --keepLastOfMonth — do not delete the last backup of a month. format is timePeriod with possible 'archive flag'. ⭐
  • --firstDayOfWeek — default: 'Sun'. this value is used for calculating --keepFirstOfWeek and --keepLastOfWeek. 📅
  • --keepFirstOfWeek — do not delete the first backup of a week. format is timePeriod with possible 'archive flag'. 🌅
  • --keepLastOfWeek — do not delete the last backup of a week. format is timePeriod with possible 'archive flag'. 🌇
  • --keepDuplicate — keep multiple backups of one day up to timePeriod. format is timePeriod, and nope, 'archive flag' is not possible here. default = 7d. 👯
  • --keepMinNumber — keep that minimum of backups. multiple backups of one day are counted as one backup. default is 10. 🧮
  • --keepMaxNumber — try to keep only that maximum of backups. if you have more backups than that, this is the sequence of deleting that happens:
    • delete all duplicates of a day, beginning with the old ones, except the last of every day
    • if this is not enough, delete the rest of the backups beginning with the oldest, but *never* a backup with the 'archive flag' or the last backup 💅
  • --keepRelative, -R — the alternative deletion scheme, bestie. The second you turn this on, every other keep option is ✨ignored✨, fr fr, no exceptions. This one preserves your backups by their relative age. Watch:
-R '1d 7d 61d 92d'

...will (try to) make sure there's always:

  • one backup that's between 1 day and 7 days old
  • one backup that's between 5 days and ~2 months old
  • one backup that's between ~2 months and ~3 months old

And if there's no backup for one of those timespans — like the last backup happened more than 2 weeks ago — then the next older backup gets used for that timespan instead. 🫶

  • --progressReport, -P — prints a progress report after each number of files. You can also slap a time frame on there so a message gets printed after that much time too. So if you want a report every 1000 files and after one minute and 10 seconds, you go: -P 1000,1m10s ⏱️
  • --printDepth, -D — prints the depth of the directory it's actually reading right now during the backup. yes it's as nosy as it sounds
  • --ignoreReadError — ignores read errors in the source directory, so directories that you can't read won't make storeBackup.pl stop doing its thing. 😌
  • --suppressWarning — suppresses the (unwanted) warnings in the log files. The keys you can use for this are:
    • excDir — suppresses the warning that excluded directories do not exist
    • fileChange — suppresses the warning that a file has changed during the backup
    • crSeries — suppresses the warning that storeBackup had to create the default series
    • hashCollision — suppresses the warning if a possible hash collision is detected
    • fileNameWithLineFeed — suppresses the warning if a filename contains a line feed
    • use_DB_File — suppresses the warning that you should install perl module DB_File for better performance
    • use_MLDBM — suppresses the warning that you should install perl module MLDBM if you want to use the rule functions MARK_DIR or MARK_DIR_REC together with option saveRAM
    • use_IOCompressBzip2 — suppresses the warning that you should install perl module IO::Compress::Bzip2 for better performance
    • noBackupForPeriod — suppresses the warning that there are no backups for certain periods when using option keepRelative

psst: you can repeat this option multiple times on the command line.

  • --suppressInfo — suppresses the (unwanted) infos in the log files. The keys for this one:
    • readCheckSums — suppresses the information that checksums of blocked files are read

Also repeatable multiple times on the command line, 💅 same energy.

  • --linkToRecent — after a successful backup, sets a symbolic link pointing to that backup, and deletes any existing older links with the same name. fresh link, who dis 🔗
  • --logFile, -l — the log file (default is STDOUT).
  • --plusLogStdout — if you gave a log file with --logFile, this flag also prints the output to STDOUT on top of that. Best of both worlds, honestly.
  • --suppressTime — suppresses the output of time in the logfile. 🕰️ bye bye timestamps
  • --maxFilelen, -m — the maximal length of the log file, default 1e6.
  • --noOfOldFiles, -n — how many old log files stick around, default 5.
  • --saveLogs — saves log files with a date and time instead of deleting the old ones (the ones handled by [-noOfOldFiles]).
  • --compressWith — compresses the saved log files (like, say, with gzip -9). Default is bzip2. Heads up: this parameter gets parsed exactly like a line in the configuration file, so normally you have to quote it. quote it or perish 🙈
  • --logInBackupDir — writes the log file (also) into the backup directory.
⚠️ Be aware that this log does not contain all the error messages that the one you specified with --logFile has!
  • --compressLogInBackupDir — compresses that log file in the backup directory. 🗜️
  • --logInBackupDirFileName — the filename to use when writing that log file from up above, default is .storeBackup.log.
  • otherBackupSeries — a list of other backup series to consider for hard linking. The path is relative to backupDir! The format looks like this (examples):
backupSeries/2002.08.29_08.25.28 -> consider this backup

or

0:backupSeries ->last (youngest) in <backupDir>/backupSeries
1:backupSeries ->one before last in <backupDir>/backupSeries
n:backupSeries ->
  n'th before last in <backupDir>/backupSeries
3-5:backupSeries ->
  3rd, 4th and 5th in <backupDir>/backupSeries
all:backupSeries -> all in <backupDir>/backupSeries

You can also throw wildcards into series names 🌈 — check the documentation, section 'Using Wildcards for Replication', for the deets. Default is to link to the last backup in every series.

Copyright (c) 2000-2022 by Heinz-Josef Claes (see README). Published under the GNU General Public License, either version 3 of the License, or (at your option) any later version. respect the license, bestie 📜

and that's a wrap on the whole page!! 🎀 This one's riding with perl v5.40.1 and dated 2026-05-23. go forth and back up literally everything, you've got this 💖✨🦋

📦 where this page comes from