INNDUCT(8)

📦 version 2.2+b1, in debian trixie, ubuntu resolute, ubuntu stonking

💖 innduct — INNDUCT(8) 💖

System Manager's Manual ✨ bestie edition, no cap ✨

🌸 NAME 🌸

innduct — stream Usenet articles to a remote site, fast and reliably, periodt 💅

🦋 SYNOPSIS 🦋

innduct [options] site [fqdn]

✨ DESCRIPTION ✨

okay so listen. innduct implements NNTP peer-to-peer news transmission including the streaming extensions, for sending news articles to a remote site. It's meant to be the glow-up replacement for innfeed or nntpsend and innxmit. 👑

You run one instance of innduct for each peer site. That's the whole vibe. Each instance manages its own interaction with innd, including flushing the feed as appropriate, etc., so that articles get transmitted quickly, and it also manages the retransmission of its own backlog. And yes bestie, innduct includes the locking necessary to avoid multiple simultaneous invocations — because two of them fighting over the same feed is not the move. 🙈

By default, innduct reads the default feedfile corresponding to the site site (ie pathoutgoing/site) and feeds it via NNTP, streaming if possible, to the host fqdn. If fqdn is not specified, it defaults to site. Simple. Iconic. 💫

innduct daemonises after argument parsing, and all logging (including error messages) are sent to syslog (facility news).

The best way to run innduct is probably to periodically invoke it for each feed (e.g. from cron), passing the -q option to arrange that innduct silently exits if an instance is already running for that site. 🕐✨

💅 GENERAL OPTIONS 💅

  • -f, --feedfile=DIR/|PATH — Specifies the feedfile to read, and indirectly specifies the paths to be used for various associated files (see FILES, below). If specified as DIR/ it is taken as a directory to use, and the actual feed file used is path/site. If PATH or DIR does not start with a /, it is taken to be relative to pathoutgoing from inn.conf. The default is site. 📂
  • -q, --quiet-multiple — Makes innduct silently exit (with status 0) if another innduct holds the lock for the site. Without -q, this causes a fatal error to be logged and a nonzero exit. 🤫
  • --no-daemon — Do not daemonise. innduct runs in the foreground, but otherwise operates normally (logging to syslog, etc.). 🚶♀️
  • --interactive — Do not daemonise. innduct runs in the foreground and all messages (including all debug messages) are written to stderr rather than syslog. A control command line is also available on stdin/stdout. 🎛️
  • --no-streaming — Do not try to use the streaming extensions to NNTP (for use eg if the peer can't cope when we send MODE STREAM). 🚫🌊
  • --no-filemon — Do not try to use the file change monitoring support to watch for writes by innd to the feed file; poll it instead. (If file monitoring is not compiled in, this option just downgrades the log message which warns about this situation.) 👀
  • -C, --inndconf=FILE — Read FILE instead of the default inn.conf. 📄
  • --port=PORT — Connect to port PORT at the remote site rather than to the NNTP port (119). 🔌
  • --chdir=PATHRUN — Change directory to pathrun at startup. The default is pathrun from inn.conf. 🏠
  • --cli=CLI-DIR/|CLI-PATH|none — Listen for control command line connections on CLI-DIR/site (if the value ends with a /) or CLI-PATH (if it doesn't). See CONTROLLING INNDUCT, below. Note that there is a fairly short limit on the lengths of AF_UNIX socket pathnames. If specified as CLI-DIR/, the directory will be created with mode 700 if necessary. The default is innduct/ which means to create that directory in PATHRUN and listen on PATHRUN/innduct/site. 🎀
  • --help — Just print a brief usage message and list of the options to stdout. 🆘

See TUNING OPTIONS below for more options. 🔥

👑 CONTROLLING INNDUCT 👑

If you tell innd to drop the feed, innduct will (when it notices, which will normally be the next time it decides to flush) finish up the articles it has in hand now, and then exit. It is harmless to cause innd to flush the feed (but innduct won't notice and flushing won't start a new feedfile; you have to leave that to innduct). 🧹

If you want to stop innduct you can send it SIGTERM or SIGINT, or the stop control command, in which case it will report statistics so far and quickly exit. If innduct receives SIGKILL nothing will be broken or corrupted; you just won't see some of the article stats. 💀 (not the scary kind of 💀, promise)

innduct listens on an AF_UNIX socket (by default, pathrun/innduct/site), and provides a command-line interface which can be used to trigger various events and for debugging. When a connection arrives, innduct writes a prompt, reads commands a line at a time, and writes any output back to the caller. (Everything uses unix line endings.) The cli can most easily be accessed with a program like netcat-openbsd (eg nc.openbsd -U /var/run/news/innduct/site) or socat. The prompt is site|. 🫶

The following control commands are supported:

  • h — Print a list of all the commands understood. This list includes undocumented commands which mess with innduct's internal state and should only be used by a developer in conjuction with the innduct source code. 🧙♀️
  • flush — Start a new feed file and trigger a flush of the feed. (Or, cause the FLUSH-FINISH-PERIOD to expire early, forcibly completing a previously started flush.) 🚽✨
  • stop — Log statistics and exit. (Same effect as SIGTERM or SIGINT.) ✋
  • logstats — Log statistics so far and zero the stats counters. Stats are also logged periodically, when an input file is completed and just before tidy termination. 📊
  • show — Writes summary information about innduct's state to the current CLI connection. 👁️
  • dump q|a — Writes the same information about innduct's state to a plain text file feedfile_dump. This overwrites any previous dump. innduct does not ever delete these dump files. dump q gives a summary including general state and a list of connections; dump a also includes information about each article innduct is dealing with. 🗑️📝
  • next blscan — Requests that innduct rescan for new backlog files at the next PERIOD poll. Normally innduct assumes that any backlog files dropped in by the administrator are not urgent, and it may not get around to noticing them for BACKLOG-SCAN-PERIOD. 🔍
  • next conn — Resets the connection startup delay counter so that innduct may consider making a new connection to the peer right away, regardless of the setting of RECONNECT-PERIOD. A connection attempt will still only be made if innduct feels that it needs one, and innduct may wait up to PERIOD before actually starting the attempt. 🔄

✨ to be continued, bestie — more parts incoming ✨

<h2>✨ TUNING OPTIONS ✨</h2>

okay bestie real talk — you should not normally need to touch any of these. Like, they're here, they exist, they're gorgeous, but if you're fiddling with them you probably have a reason. 🔧

Time intervals can be written as plain seconds, or as a number followed by one of these units: s m h d, sec min hour day, das hs ks Ms. 💅

  • --max-connections=max — caps how many simultaneous NNTP connections you're allowed to have per peer at max. There's no global limit on connections across all innducts, because each site's instances are completely independent from each other. The default is 10.
  • --max-queue-per-conn=per-conn-max — caps how many outstanding articles can be queued up on any single connection, at max. (Heads up: non-streaming connections can literally only handle one article at a time, so yeah.) The default is 200.
  • --max-queue-per-file=max — caps how many articles get read into core from any one input file, at max. The default is twice per-conn-max.
  • --feedfile-flush-size=bytes — tells innduct to flush the feed and start a fresh feedfile once the current one goes over bytes in size. The vibe: innduct tries to keep all those batchfiles from growing way past this size. The default is 100000.
  • --period-interval=PERIOD-INTERVAL — sets the wakeup interval and the period granularity. innduct wakes up every PERIOD-INTERVAL to do its lil housekeeping checks 🧹. Also, lots of the timeout and rescan intervals (the ones this manual calls PERIOD) get rounded up to the next multiple of PERIOD-INTERVAL. The default is 30s.
  • --connection-timeout=TIME — how long a connection setup attempt gets before we give up on it. The default is 200s.
  • --stuck-flush-timeout=TIME — how long we wait for innd to respond to a flush request before we give up. The default is 100s.
  • --feedfile-poll=TIME — how often we poll the feedfile for new articles written by innd, if file monitoring (inotify or equivalent) isn't available. (When file monitoring is available, there's no need for periodic checks — we wake up immediately whenever the feedfile changes. ✨) The default is 5s.
  • --no-check-proportion=PERCENT — if the moving average of the proportion of articles the peer is accepting (instead of declining) goes above this value, innduct flips into "no check mode" — meaning it just yeets the articles at the peer with TAKETHIS instead of checking first with CHECK whether the article is even wanted. This only affects streaming connections, fyi. The default is 95 (that's 95%).
  • --no-check-response-time=ARTICLES — that moving average mentioned above is an alpha-smoothed value with a half-life of ARTICLES. The default is 100.
  • --reconnect-interval=RECONNECT-PERIOD — limits starting new connections to one per RECONNECT-PERIOD. This applies both to reconnections when the peer has been down, and to ramping up how many connections we're using after startup or in response to an article flood. 🌊 The default is 1000s.
  • --flush-retry-interval=PERIOD — if our attempt to flush the feed failed (usually because innd isn't running, oops), try again after PERIOD. The default is 1000s.
  • --earliest-deferred-retry=PERIOD — when the peer answers our article offer with a 431 or 436 NNTP response code, which means the article's already been offered to it by another one of its peers and we should try again, we wait at least PERIOD before offering that article again. The default is 100s.
  • --backlog-rescan-interval=BACKLOG-SCAN-PERIOD — we scan the directory containing the feedfile for backlog files at least every BACKLOG-SCAN-PERIOD, just in case the administrator manually dropped a file in there for processing. 👀 The default is 300s.
  • --max-flush-interval=PERIOD — we flush the feed and start a new feedfile at least every PERIOD, even if the current feedfile instance never hit the size threshold. The default is 100000s.
  • --flush-finish-timeout=FLUSH-FINISH-PERIOD — if we flushed FLUSH-FINISH-PERIOD ago and we're still trying to finish processing articles that were written to the old feed file, we forcibly and violently make sure we can finish that old feed file: we abandon and defer all the work. That includes unceremoniously dropping any connections where we sent some of those articles but never got replies back, because they're probably stuck somehow. 💀 The default is 2000s.
  • --idle-timeout=PERIOD — connections with no activity for PERIOD get closed. This includes connections where we've sent commands or articles but haven't gotten the responses yet, so this same value doubles as the timeout after which we decide the peer is unresponsive or the connection has gone broken. The default is 1000s.
  • --stats-log-interval=PERIOD — log statistics at least every PERIOD. The default is 2500s.
  • --low-volume-thresh=WIN-THRESH --low-volume-window=PERIOD — if innduct has only one connection to the peer, and has processed fewer than WIN-THRESH articles in the last PERIOD, and also zero articles in the last PERIOD-INTERVAL, it'll close the connection quickly. So basically innduct switches into a mode where it opens a connection for each article (or, like, each handful of articles that arrive together 🍓). The default is to close if fewer than 3 articles in the last 1000s.
  • --max-bad-input-data-ratio=PERCENT — we tolerate up to this proportion of badly-formatted lines in the feedfile and other input files. Every badly-formatted line gets logged, but if there are too many, we conclude the corruption of our on-disk data is too severe and crash; to restart successfully after that, administrator intervention is required. This keeps the logs from getting flooded with warnings, and also makes it abort earlyish if someone tries to process a file in the wrong format. We need to tolerate a small proportion of broken lines, if for no other reason than a crash might leave a half-blanked-out entry. The default is 1 (that's 1%).
  • --max-bad-input-data-init=LINES — additionally, we tolerate this many extra badly-formatted lines, so that if the bad lines are only a few but they're right at the start of the file, we don't crash immediately. The default is 30 (which would be enough to ignore one whole corrupt 4096-byte disk block filled with random data, or one corrupt 1024-byte disk block filled with an inappropriate text file with a mean line length of at least 35).

<h2>🌈 INNDUCT VS INNFEED/NNTPSEND/INNXMIT 🌈</h2>

  • innfeed — does roughly the same thing as innduct. However, the way it receives info from innd can result in articles being lost (like, never offered to peers) if innfeed crashes for any reason. This is an inherent defect in the innd channel feed protocol, not an innfeed skill issue. innduct uses a file feed, constantly "tailing" the feed file, and where it's implemented it uses inotify(2) to cut down the latency you'd get from having to constantly poll the feed file. innduct is way smaller and simpler — under 5kloc versus innfeed's ~25kloc. innfeed needs a separate helper script or similar infrastructure (there's an example in its manpage), whereas innduct can just be run directly and doesn't need help from shell scripts. However, innfeed can feed multiple peers from a single innfeed instance, whereas each innduct process handles exactly one peer.
  • nntpsend — processes feed files in batch mode. Meaning: you have to periodically invoke nntpsend, and when you do, the feed gets flushed and the articles that arrived before the flush get sent to the peer. This introduces a batching delay, and it also means the NNTP connection to the peer has to be remade at every batch. nntpsend (which uses innxmit) cannot make use of multiple connections to a single peer site. However, nntpsend automatically figures out which sites need feeding by looking in pathoutgoing, whereas the administrator has to arrange to invoke innduct separately for each peer.
  • innxmit — is the actual NNTP feeder program used by nntpsend.

Here's the showdown 👑:

  • realtime feed — innfeed: Yes · innduct: Yes · nntpsend/innxmit: No
  • reliable — innfeed: No · innduct: Yes · nntpsend/innxmit: Yes
  • source code size — innfeed: 24kloc · innduct: 4.6kloc · nntpsend/innxmit: 1.9kloc
  • invoke once for all sites — innfeed: Yes · innduct: No · nntpsend/innxmit: Yes
  • number of processes — innfeed: one · innduct: 1/site · nntpsend/innxmit: 2/site, intermittently

<h2>💀 EXIT STATUS 💀</h2>

  • 0 — An instance of innduct is already running for this feedfile and -q was specified.
  • 4 — The feed has been dropped by innd, and we (or previous innducts) have successfully offered all the old articles to the peer site. Our work is done. 🎀
  • 8 — innduct was invoked with bad options or command line arguments. The error message gets printed to stderr, and also (if any options or arguments were passed at all) to syslog with severity crit.
  • 12 — Things are going wrong. Hopefully it's a shortage of memory, or system file table entries; disk IO problems; disk full; etc. The specifics of the error get logged to syslog with severity err (if syslog is working!).
  • 16 — Things are going badly wrong in an unexpected way: system calls that aren't supposed to fail are failing, or the protocol for talking to innd is being violated, or some such. Details get logged with severity crit (if syslog is working!).
  • 24-27 — These exit statuses are used by children forked by innduct to talk to the parent. You should not see them. If you do, it is a bug. 🐛

🗂️ FILES 🗂️

okay bestie, listen up, because this is the part where the ✨lore✨ gets real. innduct and innd do this whole elaborate little dance together — think ballroom, think choreo, think main character energy — just to make sure everything goes smoothly and there are no races. 🩰 (there are two whole ascii-art diagrams in README.states if you wanna see the protocol step by step, iykyk.)

🚨 do NOT touch the feedfile and its little friend group unless this page says you can. seriously. hands off. 🙅‍♀️

here's the whole cast, one by one:

  • pathrun — the default current working directory for innduct, and also the default grandparent directory for the command line socket. yes, grandparent. it's a whole family tree. 🌳
  • pathoutgoing/site — the default feedfile. that's it, that's the vibe.
  • feedfile — the main feed file, exactly as specified in newsfeeds(5). this one and the other batchfiles innduct uses are made of lines that each look like this:
token messageid

where token is the inn storage API token. lines like that can be written by Tf,Wnm in a newsfeeds(5) entry. while it's processing, innduct overwrites lines in the batch files that correspond to articles it's already handled — each of those lines gets replaced with one containing only spaces. 💀 and remember: only innd should create feedfile, and only innduct should remove it. periodt.

  • feedfile_lock — the lockfile, and its whole job is preventing multiple innduct invocations for the same feed. a process holds this lock once it has opened the lockfile, made an fcntl F_SETLK call, and then checked with stat and fstat that the file it now has open and has locked still has the name feedfile_lock. (only) the lockholder is allowed to delete the lockfile. and for your convenience, once the lockfile is locked, innfeed's pid, the site, feedfile and fqdn are all written into it. 🎀 NB: stale lockfiles can contain stale data, so don't go trusting that info for anything other than troubleshooting, fr fr.
  • feedfile_flushing — a batch file: the main feedfile gets renamed to this filename by innduct before it asks inn to flush the feed. only innduct should create, modify or remove this file.
  • feedfile_defer — a batch file holding details of articles whose transmission was very recently deferred at the request of the recipient site. created, written, read and removed (only) by innduct. no one else. not even you. 🙈
  • feedfile_backlog.time_t.inum — a batch file with details of articles whose transmission was less recently deferred at the request of the recipient site. created by innduct, and it'll also be read, updated and removed by innduct. however — you, the administrator, may also safely remove backlog files. you get a lil permission slip for this one. 🫶
  • feedfile_backlogsomething — a batch file manually provided by the administrator. innduct will automatically find, read and process any file matching this pattern (blanking out entries for processed articles) and eventually remove it. just know that something may not contain #, ~ or /.
😭 please please please finish writing the file before you rename it to match the pattern feedfile_backlog*. otherwise innduct might find and process the file — and even think it has finished it — before you've written the complete thing. the horror. you may also safely remove backlog files.
  • pathrun/innduct/site — the default AF_UNIX listening socket for the control command line. see CONTROLLING INNDUCT, above. 👑
  • feedfile_dump — when you request it via a control connection, innduct dumps a summary of its state into this text file. mostly a debugging thing. mostly. 🐛
  • /etc/news/inn.conf — used for pathoutgoing (to compute the default feedfile and its associated paths), for pathrun (to compute the default PATHRUN and therefore the effective default CLI-DIR and CLI-PATH), for figuring out how to communicate with innd, and also for sourceaddress and/or sourceaddress6. it's giving utility player. ⭐

📜 HISTORY 📜

written by Ian Jackson <ijackson@chiark.greenend.org.uk> 🖋️✨

👀 SEE ALSO 👀

inn.conf(5), innd(8), newsfeeds(5)

and that's a wrap on the whole page, bestie!! 🌈💖 the footer down at the bottom is just the page's own name, INNDUCT(8) — no extra version or date gossip attached to this one, so that's your lot. go forth and not delete the lockfile, okay? mwah. 💅🎀🦋

📦 where this page comes from