FTPSYNC.CONF(5)

📦
this version · 2 versions
, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble

💖 FTPSYNC.CONF 💖 (section 5 of the manual, bestie)

okay so welcome to ftpsync.conf(5) — yes it's a manual page, section 5 (that's the "file formats and conventions" girlies), and it's literally just the config file for ftpsync. Sit down, grab a snack 🍓, we're doing this.

🌸 NAME 🌸

ftpsync.conf — the config for ftpsync. that's it. that's the name. iconic brevity. 👑

🦋 DESCRIPTION 🦋

so ftpsync is part of the whole ftpsync suite, and its whole job is mirroring Debian and Debian-like repositories of packages. 💅

here's the ✨lore✨: there are WAY too many Debian mirrors to have the machine that generates the archive (they call it "ftp-master") populate every single one of them. so instead, mirrors are organized into a tree-shaped hierarchy. that means:

  • every mirror has exactly one upstream it syncs from
  • each mirror can have any number of downstreams which then sync from it

it's giving family tree but make it packages. 🌳 no cap.

✨ OPTIONS ✨

👑 the main character options

  • MIRRORNAME — the mirrorname. this is what gets used for stuff like the trace file name, and it should always be the full hostname of the mirror. Default: $(hostname -f)
  • TO — destination of the mirrored files. should be an empty directory.
⚠️ CAREFUL!! this directory will contain the mirror. everything else that might have happened to be hanging out in there WILL BE GONE after the mirror sync. like, gone gone. bestie, please check before you run this. 😭

Default: /srv/mirrors/debian/

  • MAILTO — the script can send logs (or error messages) to a mail address. if this is unset, it defaults to the current user. Default: $LOGNAME
  • HUB — do we have leaf mirrors to signal that we're done and they should sync? if so, set it to true and make sure you configure runmirrors.mirrors and runmirrors.conf for your needs. Default: false

🔌 Connection options 🔌

  • RSYNC_HOST — the host we mirror from.
  • RSYNC_PATH — the upstream name of the rsync share. you can find out what share names your upstream mirror supports by running:
rsync YOURUPSTREAMSERVER::

(you might have to export RSYNC_USER/RSYNC_PASSWORD for this to work 💜) Default: debian

  • RSYNC_USER — in case we need a user to access the rsync share at our upstream host.
  • RSYNC_PASSWORD — if we need a user, we also need a password. duh. 🔐
  • RSYNC_TRANSPORT — set this to pick the transport used for rsync.
    • ssh — use rsync daemon over SSH. this requires a rsyncd.conf on the server. additional options for the ssh client should be configured in ssh_config.
    • ssl — use rsync daemon over SSL. this requires a SSL wrapper (e.g. stunnel) for the rsync daemon on the server. see the other RSYNC_SSL options below. 🫶
  • RSYNC_SSL_PORT — Default: 1873
  • RSYNC_SSL_CAPATH — Default: /etc/ssl/certs
  • RSYNC_SSL_METHOD — ftpsync can use either stunnel, stunnel-old, or socat to set up the encrypted tunnel.
    • stunnel — requires at least stunnel version 5.15 built against openssl 1.0.2 or later, such that the stunnel build supports the checkHost service-level option. this will cause stunnel to verify both the peer certificate's validity and that it's actually for the host we wish to connect to. thorough queen. 👑
    • stunnel-old — will skip the checkHost check. as such it will connect to any peer that is able to present a valid certificate, regardless of which name it is made out to. (so like… it trusts everybody. do with that what you will 🙈)
    • socat — will verify the peer certificate name only starting with version 1.7.3 (Debian 9.0).

to test if things work, you can run:

rsync -e `bin/rsync-ssl-tunnel -m socat -p 1873 -C
/etc/ssl/certs' SERVER::

Default: stunnel

  • RSYNC_PROXY — you may establish the connection via a web proxy by setting the environment variable RSYNC_PROXY to a hostname:port pair pointing to your web proxy. note that your web proxy's configuration must support proxy connections to port 873.

📇 Mirror information options 📇

these options add informations about the mirror to a publicly accessible file. yes, publicly. so don't be weird about it. ✨

  • INFO_MAINTAINER — groups and people responsible for this mirror. these contacts are used to report irregularities and problems with the mirror. they must actually accept mail. Example:
INFO_MAINTAINER=“Admins <admins@example.com>, Person <person@example.com>”
  • INFO_SPONSOR — organizations sponsoring this mirror. Example:
INFO_SPONSOR=“Example <https://example.com>”
  • INFO_COUNTRY — the ISO 3361-1 code of the country hosting this mirror. Example:
INFO_COUNTRY=DE
  • INFO_LOCATION — the location this mirror is hosted in. Example:
INFO_LOCATION=“Example”
  • INFO_THROUGHPUT — available throughput for this mirror per second. Example:
INFO_THROUGHPUT=10Gb

🧺 Include and exclude options 🧺

these options can be used to include or exclude specified architectures or other files from the mirror.

🚨 Notice: the source architecture needs to be included on an official/public mirror! no exceptions, bestie.
  • ARCH_INCLUDE — if you want to include only a subset of architectures, this is for you. use as space seperated list of architectures in the archive you are mirroring from, source counts as architecture. architecture all will be included automatically if one binary architecture is included. Mutually exclusive with ARCH_EXCLUDE. Example:
ARCH_INCLUDE=“amd64 i386 source”
  • ARCH_EXCLUDE — if you want to exclude an architecture, this is for you. use as space seperated list of architectures in the archive you are mirroring from, source counts as architecture. Mutually exclusive with ARCH_INCLUDE. Example:
ARCH_EXCLUDE=“alpha arm arm64 armel mipsel mips s390
sparc”
  • EXCLUDE — if you do want to exclude files from the mirror run, put –exclude statements here. see rsync(1) for the exact syntax, these are passed to rsync as written here.
💀 DO NOT TRY TO EXCLUDE ARCHITECTURES OR SUITES WITH THIS, IT WILL NOT WORK! like it will simply not do the thing. use the ARCH options above. fr fr.

📝 Log option 📝

  • LOGDIR — in which directory should logfiles end up. Default: ~/.local/log/ftpsync in the package, ${BASEDIR}/log otherwise
  • LOG — name of our own logfile. note that ${NAME} is set by the ftpsync script depending on the way it is called. see README for a description of the multi-archive capability and better always include ${NAME} in this path. Default: LOGDIR/{NAME}.log
  • ERRORSONLY — if you do want a mail about every single sync, set this to false. everything else will only send mails if a mirror sync fails. Default: true
  • FULLLOGS — if you want the logs to also include output of rsync, set this to true. careful, the logs can get pretty big, especially if it is the first mirror run. 🫣 Default: false
  • LOGROTATE — we do create three logfiles for every run. to save space we rotate them, this defines how many we keep. Default: 14

🎀 Other options 🎀

  • LOCKTIMEOUT — timeout for the lockfile, in case we have bash older than v4 (and no /proc). Default: 3600
  • UIPSLEEP — number of seconds to sleep before retrying to sync whenever upstream is found to be updating while our update is running. Default: 1200
  • UIPRETRIES — number of times the update operation will be retried when upstream is found to be updating while our update is running. note that these are retries, so: 1st attempt + retries = total attempts. Default: 3
  • TRACEHOST — the local hostname to be written to the trace file. Default: $(hostname -f)
  • RSYNC — the rsync program. (that's the whole description. she's lowkey mysterious. 💅)
  • RSYNC_EXTRA — extra rsync options as defined by the local admin. there is no default by ftpsync. please note that these options are added to EVERY rsync call. also note that these are added at the beginning of the rsync call, as the very first set of options. please ensure you do not add a conflict with the usual rsync options as shown below. ⚠️
  • RSYNC_BW — limit I/O bandwidth. value is KBytes per second, unset or 0 means unlimited.
  • RSYNC_OPTIONS — default rsync options every rsync invocation sees.
🔥 BE VERY CAREFUL WHEN YOU CHANGE THIS OPTION, BETTER DON'T!

Default: see ftpsync script

  • RSYNC_OPTIONS1 — options the first pass gets. we do not want the Packages/Source indices here, and we also do not want to delete any files yet.
🔥 BE VERY CAREFUL WHEN YOU CHANGE THIS OPTION, BETTER DON'T!

Default: see ftpsync script

  • RSYNC_OPTIONS2 — options the second pass gets. now we do want the Packages/Source indices too and we also want to delete files. we also want to delete files that are excluded.
🔥 BE VERY CAREFUL WHEN YOU CHANGE THIS OPTION, BETTER DON'T!

Default: see ftpsync script

  • CALLBACKUSER, CALLBACKHOST, CALLBACKKEY — the following three options are used in case we want to "callback" the host we got pushed from. 📞

🪝 Hooks 🪝

hook scripts can be run at various places during the sync. little pit stops, if you will. 🦋

  • HOOK1 — after lock is acquired, before first rsync.
  • HOOK2 — after first rsync, if successful.
  • HOOK3 — after second rsync, if successful.
  • HOOK4 — right before leaf mirror triggering.

note that Hook3 and Hook4 are likely to be called directly after each other. the difference is: Hook3 is called every time the second rsync was successful, but even if the mirroring needs to re-run thanks to a second push. Hook4 is only effective if we are done with mirroring. ✨

  • HOOK5 — after leaf mirror trigger, only if we have slave mirrors (HUB=true).

👀 SEE ALSO 👀

💜 AUTHORS 💜

Debian mirror team. (absolute legends, we stan 🫶)

and that's the whole page, bestie!! this one came from the ftpsync Manual (that's the source/version tag at the bottom, section FTPSYNC.CONF(5), you'll see it printed down there) 💅✨ go forth and mirror responsibly, and remember: don't touch RSYNC_OPTIONS. like, at all. ily, bye!! 🌈🎀

📦 where this page comes from