DPHYS-CONFIG(1)

📦 version 20130301~current, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute

💖 DPHYS-CONFIG 💖

okay bestie, welcome to DPHYS-CONFIG(1) — a General Commands Manual page and honestly? it's giving infrastructure royalty 👑 it lives in section 1 of the manual, and it's here to make your config files manage themselves. buckle up 🎀

✨ NAME ✨

dphys-config — daily auto-install/update and/or remove config files 🦋

✨ SYNOPSIS ✨

dphys-config [-f filter] [-cqvD]

dphys-config -h

✨ DESCRIPTION ✨

so like, dphys-config installs/updates and/or removes config files. that's the whole vibe. but wait, there's more 💅 it also triggers commands after a new/updated config file shows up, or before an existing config file is about to disappear. and it can be run by hand, from cron and/or from init.d — she's flexible like that ✨

here's the actual flow, no cap:

  • get a list of config files from a configuration server 🌐
  • for each file in that list, retrieve the file from the same server
  • only install it if it is new or changed relative to what's already chilling on your box
  • if a file is newly installed (or changed), run a postinstall script — which may trigger actions you want in order to process the new config (like, say, inserting data from a config file into a database 🗄️)
  • also remove unwanted files — and before doing that, run a preremove script first to tidy stuff up 🧹

this is part of the D-PHYS (ETH Zuerich, Departement of Physics) automatic system operation and maintenance setup. iconic institution behaviour, fr fr 🎓

✨ OPTIONS ✨

  • -c — configname: use this set of config files instead of the hostname set. lowkey perfect for chroot or vhost installs, or for tests 🧪
  • -f filter — filter: only process lines which match the filter spec 🔍
  • -q — quiet: don't produce a running report of activities 🤫
  • -v — verbose: give large volume output, where sensible 📢
  • -D — Debug: activate a debug option. see the source for how to use this 👀
  • -h — help: output help text, and then abort operation 🛑

✨ CONFIG ✨

the config files /etc/dphys-config (sitewide) and ~/.dphys-config (personal) let the admin and users set up the working environment for dphys-config 🛠️

these config files are sh script fragments full of assignments, which get sourced, in the above row, with later config files' assignments overriding earlier ones. standard sh syntax rules apply ✍️

assignments are:

  • CONF_TMP_DIR — sets the base directory in which all temporary files are stored. it defaults to /var/tmp (for enough size and safe operation). some users may like to use /tmp for higher speed (tmpfs) or automatic deletion at boot time 🚀
okay so the original page literally says "Standard sh syntax rules apply. Assignments are:" again right here at the end of this entry, and we are keeping it because the rule is keep every detail 💅
  • CONF_BASEURL — sets the base URL to which all */<hostname>/<filename> combinations are added when wget-ing config files. this can be an http: or ftp: or whatever other type of URL which wget understands and can fetch a file from 🌐 additionally it can be a file: (this may be from an NFS server) URL, in which case wget is bypassed and the files fetched directly using cp. it defaults to the error message generating and aborting invalid setting of http://not-configured-server.example.net/not/configured/directory, as there is no sensible default possible. you must set this to wherever your config files should be taken from 🚨
  • CONF_CONFNAME — selects the name for which set of configuration files shall be used for this host. defaults to `hostname` 🏷️
  • CONF_LINEFILTER — sets a regexp which selects which lines from the config file list are processed. defaults to .* (all) 🎯
  • CONF_LOG_DONE — log to syslog that dphys-config has run. good to see if cron and/or init.d have done their job. defaults to yes 📝

the config file list dphys-config.list, which is found via the above settings, and is downloaded to /etc/dphys-config.list or ~/.dphys-config.list, then allows the admin to list what config files are to be fetched and installed/updated or removed, and what scripts to run for them 📋 these can each be given for the entire site (= all hosts) and/or group and/or each host, or even merged from site+group+host subsections. choose your own adventure, bestie 🌈

✨ FILES ✨

  • /etc/dphys-config — site admin config 👑
  • ~/.dphys-config — users personal config 🧑‍💻
  • /etc/dphys-config.list — roots config file list gets stored here 🌳
  • ~/.dphys-config.list — users config file list gets stored here 📂
  • $CONF_BASEURL/hostname/dphys-config.list — site-global (all hosts) common (usually, or group-global or host specific) config file list 🌍
  • $CONF_BASEURL/hostname/dphys-config.list.* — facultative host-specific (usually) or group-specific include-able subsection(s) to be added to the above config file list dphys-config.list. we often use *.group (one per group of users) and *.host (per host), sometimes also *.base (all host types) and *.workstation (only workstation hosts) subsections 🦋
  • $CONF_BASEURL/hostname/<file-name> — actual config files referred to in config file list, common section (usually the only section) 📄
  • $CONF_BASEURL/hostname/<file-name>.* — facultative host-specific (usually) or group-specific include-able subsection(s) to be added to the above config file <file-name> ✨

✨ CONFIG FILE LIST ✨

the config file list to be used for checking what files need to be installed/updated or removed, plus its subsections included by #@include lines, are merged to one list file, analog to cpp #include 🧬

these are all fetched via wget (or cp for file:), adding their names to the user-defined base URL in CONF_BASEURL, and then merged. so CONF_BASEURL can be any URL that wget understands — http: or ftp: or whatever else, or file: 🌐

the format of the resulting concatenated file must consist of lines, one per config file, of the following format:

file-name:place-on-target:command-to-trigger

where the 3 fields have the following meanings:

  • file-name — name of the config file to be installed/updated. must be only the base part of the filename on the server, without URL and hostname before it, and without any .* subsection endings after it, as these are all auto-added whenever they are needed. if this is set to - the line specifies a config file to be removed 🗑️
  • place-on-target — full directory or full filename (directory+filename) of where the file is to be placed on the target system. if only a directory is given (any name that ends in /), then the above file-name (inclusive any directories in it) will be automatically added to it. for removing, this must be the full filename (or a directory name (without a /) if an entire directory and its contents shall disappear). a directory name ending with / is not processed, to prevent incomplete edits (filename replaced by -, but not added to directory) from killing entire directories (such as, say, all of /etc/ :-)) 😱
bestie, this one is genuinely load-bearing. do not goof around with it 🚨
  • command-to-trigger — full command (directory+filename, with parameters) of a command to be run, after this config file has been newly installed or changed/updated, or before this config file is removed. this can also be multiple commands separated by ; separators. useful for doing chown/chmod to files that need it 🔧 if the marker {} appears in the command, this will be substituted by the filename the config file is going to be installed as. this is analog to find -exec filename substitution 🔍

lines which begin with an # are regarded as comments, and don't have any effect on anything (lines extended with one are chopped off at that point). the same applies for empty lines 🫧

✨ PREPROCESSOR ✨

if the first line of the config file list, or any config file fetched on its behalf, has the special format #@dphys-config-preprocess [action...] then this line will be stripped, and the rest of the file will be preprocessed 🪄 depending on the list of actions present and their order (repeats are allowed) the file will be processed. valid actions are:

  • backtick — anything inside backticks (`) will be executed as a command, and its stdout will then be substituted for the ` expression. this is analog to sh backtick substitution 🐚
  • if — for any line beginning with #@if, the stuff between the #@if and the first ; character will be executed as a command, and if it returns true, everything after the ; will be left, else the entire line will be removed. this is analog to shell if ... ; conditional execution 🤔
  • include — for any line beginning with #@include, the rest of the line is regarded as a subsection name, which will be added to the base filename, and then the resulting subsection file fetched (also by wget or cp) and substituted for the line. this is analog to a C preprocessor #include or a shell . include 📥

✨ EXAMPLES ✨

okay so picture this: you wanna pull all your config file lists from an HTTP VirtualHost called www.admin.example.net, living under its subdirectory dphys-config. slay. here's how 💅

In the file /etc/dphys-config, on every host (so it can actually find the config file server):

# system will use ${CONF_BASEURL}/`hostname`/<file-name>*
CONF_BASEURL=http://www.admin.example.net/dphys-config

and bestie, we highly advise using a subdirectory here, because other /http://www.admin.example.net/* directories may already contain other admin stuff you put on the same VirtualHost. such as software packages, site news, etc. keep it tidy ✨

For dphys-config to be useful you then need to make config file lists for it. And provide the actual config files that can be installed, driven by the lists. This is the largest job, as it basically amounts to extracting all your relevant config work from your site. Also known as reengineering your site. 😭 not the fun part, we know.

Assuming your VirtualHost on www.admin.example.net has as its DocumentRoot /vhost/www.admin, you would then begin with an pseudo-host Directory for site-global common stuff: /vhost/www.admin/dphys-config/SITE/.

If your hosts are organised in groups with group-global common configs (such as professors, students, staff), make an pseudo-host for each group, such as: /vhost/www.admin/dphys-config/PROFS/ and */STUDENTS/ and */STAFF/.

Then for host specific stuff, assuming systems called prof1.example.com to prof3.example.com, stud1 to stud20, staff1 to staff5, server1 and server2, make for each its own directory: /vhost/www.admin/dphys-config/prof1/ (and so on).

Note that we suggest using CAPITALS for pseudo-hosts and lowercase for actual hosts. This avoids name space collisions. You can also use loops like for host in [a-z]* ; do ... ; done to work (say generating symlinks to an new config file in all hosts). Well at least you can do this so long no one goes and sets LANGUAGE= or similar junk, then bash (or libc?) will hapily screw up case sensitivity and produce random lossage (yes, it was painfull). 💀💀💀

After this add to /vhost/www.admin/dphys-config/SITE/, the actual config files as far as they are not host specific, or at least have an common section to all hosts. Example this would be /etc/hosts for all, an common section for /etc/motd, common or all for sendmail.cf, common for inetd.conf, nothing for the ssh hostkeys.

Then add, to an group, say /vhost/www.admin/dphys-config/STUDENTS/, whatever is specific to that group. Example this may be an entire special motd for the many changing users, or just an motd.group to #@include into the common one.

Then for each host in its /vhost/www.admin/dphys-config/prof1/ (or so) add all that is specific to it. Such as its ssh key files. And its own motd.host, it it needs one. Same its inetd.conf.host if it is going to offer special stuff. An configs for services only this host has such as httpd.conf.

Then for each host add symlinks to the SITE or group versions that it is to use for common stuff, like on /vhost/www.admin/dphys-config/stud1/:

 .../dphys-config/stud1/dphys-config.list -> ../SITE/dphys-config.list
 .../dphys-config/stud1/hosts -> ../SITE/hosts
 .../dphys-config/stud1/inetd.conf -> ../SITE/inetd.conf
 .../dphys-config/stud1/motd -> ../SITE/motd
 .../dphys-config/stud1/motd.group -> GROUP/motd
 .../dphys-config/stud1/GROUP -> ../STUDENTS
 .../dphys-config/stud1/sendmail.cf -> ../SITE/sendmail.cf

In the /vhost/www.admin/dphys-config/SITE/ directory place the site-global common dphys-config.list for all your hosts, containing stuff like this:

# SITE dphys-config.list - just example stuff, for our exemplaric site
# basics
hosts:/etc/                      # simply works, no command
motd:/etc/                       # this will be assembled group specific
inetd.conf:/etc/:/etc/init.d/inetd restart  # needs an command to reload
sendmail.cf:/etc/mail/:/etc/init.d/sendmail restart  # not in /etc
# ssh restart only after last file, and ensure file modes for each file
ssh_host_key:/etc/ssh/:chown root:root {}; chmod 600 {}
ssh_host_rsa_key:/etc/ssh/:chown root:root {}; chmod 600 {}
ssh_host_dsa_key:/etc/ssh/:chown root:root {}; chmod 600 {}; /etc/init.d/sshd restart
# load stuff into an existing database file
seed.debconf:/etc/:debconf-set-selections {}
# other stuff
daemon1-conf:/etc/daemon1/conf   # rename so names can differ on server
daemon2-conf:/etc/daemon2/conf
daemon1/conf:/etc/               # same as above, but with directories on server
daemon2/conf:/etc/
testing:/etc/                    # put something in there for an test
# delete some stuff
-:/etc/testing                   # change to above test to get rid of it again
-:/etc/                          # you will get a warning if you leave this
#-:/etc                          # you would reinstall your system after the resulting  rm -rf /etc  :-)
# and some errors
#only-an-name                    # you would get an error: no place on target
#only-an-name:                   # you would get an error: no place on target
#:only-an-place                  # you would get an error: no file to install
lowkey screaming at that rm -rf /etc line ngl. it's commented out for a REASON bestie 🙈

For special services add an dphys-config.list.host on each host that has special config files not present on others, such as on /vhost/www.admin/dphys-config/server2/:

# server2 dphys-config.list.host - only used on our web server
httpd.conf:/etc/apache/httpd.conf:/etc/init.d/apache restart

You can also use dphys-config to run arbitrary commands, whenever config files are installed/updated or removed, to modify existing config files, or more likely modify complex config databases which can not be provided as files, but where one can provide edit info as files.

dphys-config can even install scripts to use as above commands (or even just to run scripts while installing), such as into /usr/local/sbin/.

For this make an ../SITE/local/sbin/ directory, place the scripts in there (such as ../SITE/local/sbin/dphys-config-<whatever>), and symlink local to ../SITE/local on each host, and then add config lines for the scripts, with the command to trigger them, giving something like this:

local/sbin/dphys-config-<whatever>:/usr/:chmod 755 {}; {}  # chmod and run

It this script processes an config file your will want it to be run if either the script or the config file is updated, so add the script to the laters line as well:

dphys-config-<whatever>:/etc/:if [ -x /usr/local/sbin/dphys-config-<whatever> ] ; then /usr/local/sbin/dphys-config-<whatever> ; fi  # run also here

Finally, new hosts can then later simply be added, by making the new hosts directory and copying all files and symlinks from an existing host of the same group. Such as by doing:

mkdir student21
tar -cf - -C student1 . | tar xpf - -C student21

To then run dphys-config by hand (say for tests), type on the host:

dphys-config

But usually you will want to run dphys-config automatically, every night (or if a machine was/is switched off, at every boot), to keep your configs up to date. 🦋

For nightly updates the best thing is to use an cron job on every host. 03:00 to 03:59 is most likely idle time. Use an line like this one, with the cron option to avoid an load peak on the config file server, by random delaying the run by 0..3599 seconds, and with stdout and strerr thrown away to avoid getting an mail from every host, as error messages are also allways sent to syslog:

0 3 * * * root dphys-config cron > /dev/null 2>&1

To catch machines switched off over night, with no cron run on them, also run an init.d script. Use an script like this one, also with stdout and stderr thrown away to avoid cluttering your boot console output:

#!/bin/sh
# /etc/init.d/dphys-config - boot time automatic config updates, if no cron
case "$1" in
  start)
    dphys-config init > /dev/null 2>&1
    ;;
esac
exit 0

🔒 SECURITY 🔒

If dphys-config is to be used to distribute all config files, this will also include files which are security relevant, such as ssh private keys (host key or (root) user authentification), SSL certificates, passwd and shadow, lilo.conf, software license keys, etc. 😳 yeah. the whole vibe.

As all files are most likely fetched from an http: URL, measures must be taken to secure the config website from other people downloading them. We here use an restriction to only IP addresses registered as hosts in our NIS server, and additionally run identd on all allowed hosts, and require the wget process opening the HTTP connection to be running by user root, and so also require dphys-config to run as root.

To avoid sniffing it is recommended to give wget an https: URL. 👑

fr fr, please do this. it's giving "i left my front door open" energy otherwise.

🙈 GOTCHAS 🙈

Config files are read by wget from an webserver, so they lose their owners and modes. So the commands triggered on their lines must be used to chown/chmod them to proper values.

When used together with dphys-admin, dphys-config should run as first (earlier cron and init.d entries). This is needed to provide configs before new packages are installed, so dphys-admin can pretend that the packages were already once installed (and then non-purge removed), and so prevent questions on install, which is required for unattended installs.

[Note that this pretending does not go as far as setting debconf up. Broken packages that ignore config files and only look at debconf will still ask questions.]

As result of this, when installing for the first time on an new system (such as installing Debian by the dphys3 end2stage feature, which installs first dphys-config and then dphys-admin), any scripts installed by packages by dphys-admin, to be called on config file install/update will still be missing, and so not runnable. Either ignore the warnings, or better call the scripts by something like this:

file:place:if [ -x script ]; then script; fi

Note that in this case, trying to run dphys-config for a second time after dphys-admin has installed packages and scripts, will not automatically mend this, as the config files have not changed, and so dphys-config will not (re-)run their scripts. Therefore packages containing such scripts must also, as part of their postinst (or init.d which is called by postinst), check for existing config files and then run their scripts. This is the normal behaviour of quite a few packages anyway. Of course this requires the scripts to be idempotent, which is official Debian policy anyway. 💅

👑 AUTHOR 👑

neil@franklin.ch, http://neil.franklin.ch/

and that's the whole page bestie!! this one's the D-PHYS Configuration Tools, dated 2010.01.21, section DPHYS-CONFIG(1) — go forth and config responsibly 🫶✨🌸

📦 where this page comes from