PULLNEWS(1)

okay so bestie, buckle up 🫶 — this is pullnews, straight out of the InterNetNews Documentation (iykyk 💅). She's a tiny script with main character energy and one whole job: pulling news from a bunch of servers and feeding it to another one. Let's get into it ✨

🌸 NAME 🌸

pullnews — pull news from multiple news servers and feed it to another 🎀

✨ SYNOPSIS ✨

pullnews [-BhnOqRx] [-a hashfeed] [-b fraction] [-c config] [-C width]
[-d level] [-f fraction] [-F fakehop] [-g groups] [-G newsgroups] [-H
headers] [-k checkpt] [-l logfile] [-L size] [-m header_pats] [-M num]
[-N timeout] [-p port] [-P hop_limit] [-Q level] [-r file] [-s to-server[:port][_tlsmode]]
[-S max-run] [-t retries] [-T connect-pause]
[-w num] [-z article-pause] [-Z group-pause] [from-server ...]

🫶 REQUIREMENTS 🫶

• The Net::NNTP module has got to be installed, no cap. Good news though: it's available as part of the libnet distribution and it literally comes with recent versions of Perl 🔥
• For older versions of Perl, you can download it from http://www.cpan.org/ — yes, actually go do that, she's not gonna run herself 💀

---

💜 DESCRIPTION 💜

okay so here's the vibe: pullnews reads a config file named pullnews.marks and connects to the upstream servers given there — as a reader client, btw, not some fancy peering situation 🌸

Where does it look for that file? If pullnews is run as the user set in runasuser in inn.conf (which is the news user by default), it goes looking in pathdb. Otherwise? It checks the running user's home directory instead 🏠 both paths, noted, don't @ me.

By default, pullnews connects to all the servers listed in the configuration file. But you can absolutely gate-keep and limit it to specific servers by listing them on the command line — a whitespace-separated list of server names, like from-server for one of them 💅 For each server it connects to, it pulls over articles and feeds them to the destination server via the IHAVE or POST commands. Which means: the system pullnews is run on must have feeding access to the destination news server. Non-negotiable 🔥

pullnews is designed for very small sites that do not want to bother setting up traditional peering — she is not meant for handling large feeds, bestie. Know her limits 😭

🎀 the fake-hop thing, listen up 🎀

In case you have running peers and you don't want to propagate them the articles you're pulling from upstream servers, you should add a fake hop with the -F flag to all the pulled articles — and then add that very fake hop in the exclusion sub-field of all the sites configured in your newsfeeds file which should not receive these articles.

Like, for example: using pullnews -F myserverimported, change sitename:*:Tm:innfeed! to sitename/myserverimported:*:Tm:innfeed! for every sitename in newsfeeds you don't want to feed the pulled articles to (think your outgoing peers and a possible inpaths! entry). 🌈

Entries like ME, controlchan!, innfeed! or nocem! do not need that exclusion — they're fine, they're chill, leave them alone 💖

✨ OPTIONS ✨

okay bestie, deep breath — this is the big one. there are so many flags and every single one matters, so grab a snack 🍓 and let's go. i'm not skipping a single thing, i promise 🫶

• -a hashfeed — this one's a deterministic way to control the flow of articles and to split a feed. the hashfeed parameter has to be in the form "value/mod" or "start-end/mod". the Message-ID of each article gets hashed with MD5 (that's a 128-bit hash, no cap ✨). by default the lowest 32 bits get taken as the hashfeed value (which is an integer). so if the hashfeed value modulus "mod" plus one equals "value" — or lands between "start" and "end" — pullnews will feed the article. all these numbers must be integers, periodt.

for instance:

pullnews -a 1/2      Feeds about 50% of all articles.
pullnews -a 2/2      Feeds the other 50% of all articles.

another example:

pullnews -a 1-3/10   Feeds about 30% of all articles.
pullnews -a 4-5/10   Feeds about 20% of all articles.
pullnews -a 6-10/10  Feeds about 50% of all articles.

you can also use an extended syntax of the form "value/mod:offset" or "start-end/mod:offset" (an underscore "_" instead of a colon ":" is recognized too, iykyk). since MD5 generates a 128-bit return value, you can specify from which byte-offset the 32-bit integer used by hashfeed starts. the default value for "offset" is ":0" and thirteen overlapping values from ":0" to ":12" can be used. only up to four totally independent values exist: ":0", ":4", ":8" and ":12".

so yeah, this lets you generate a second level of deterministic distribution. like, if pullnews feeds "1/2", it can go on splitting thanks to "1-3/9:4" for instance. up to four levels of deterministic distribution can be used. 💅

the algorithm is compatible with the one used by Diablo 5.1 and up. (slay.)

• -b fraction — backtrack on server numbering reset, bestie. specify the proportion ("0.0" to "1.0") of a group's articles to pull when the server's article number is less than our high for that group. when fraction is "1.0", pull all the articles on a renumbered server. the default is to do nothing (very demure of it).
• -B — feed is header-only, that is to say pullnews only feeds the headers of the articles, plus one blank line. it adds the Bytes header field if the article doesn't already have one, and keeps the body only if the article is a control article. 🎀
• -c config — normally, the config file is stored in pullnews.marks in pathdb when pullnews is run as the news user, or otherwise in the running user's home directory. if -c is given, config will be used as the config file instead. this is useful if you're running pullnews as a system user on an automated basis out of cron or as an individual user, rather than the news user. see "CONFIG FILE" below for the format of this file. 🗂️
• -C width — use width characters per line for the progress table. the default value is "50".
• -d level — set the debugging level to the integer level (up to "4"); more debugging output will be logged as this increases. the default value is "0".
• -f fraction — this changes the proportion of articles to get from each group to fraction and should be in the range "0.0" to "1.0" ("1.0" being the default). 🦋
• -F fakehop — prepend fakehop as a host to the Path header field body of articles fed.
• -g groups — specify a collection of groups to get. groups is a list of newsgroups separated by commas (only commas, no spaces). each group must be defined in the config file, and only the remote hosts that carry those groups will be contacted. note that this is a simple list of groups, not a wildmat expression, and wildcards are not supported. 🙈
• -G newsgroups — add the comma-separated list of groups newsgroups to each server in the configuration file (see also -g and -w).
• -h — print a usage message and exit. short and sweet, honestly iconic 💅
• -H headers — remove these named header fields (colon-separated list) from fed articles.
• -k checkpt — checkpoint (save) the config file every checkpt articles (default is "0", that is to say at the end of the session).
• -l logfile — log progress/stats to logfile (default is "stdout").
• -L size — specify the largest wanted article size in bytes. the default is to download all articles, whatever their size. when this option is used, pullnews will first retrieve overview data (if available) of each newsgroup to process so as to obtain articles sizes, before deciding which articles to actually download. lowkey smart ngl ✨
• -m header_pats — feed an article based on header field body matching. the argument is a number of whitespace-separated tuples (each tuple being a colon-separated header field name and regular expression). for instance:

-m "Hdr1:regexp1 !Hdr2:regexp2 #Hdr3:regexp3 !#Hdr4:regexp4"

specifies that the article will be passed only if the "Hdr1" header field body matches "regexp1" and the "Hdr2" header field body does not match "regexp2". besides, if the "Hdr3" header field body matches "regexp3", that header is removed; and if the "Hdr4" header field body does not match "regexp4", that header is removed. the ! and # are doing a lot of heavy lifting here, i know 🧚

• -M num — specify the maximum number of articles (per group) to process. the default is to process all new articles. see also -f.
• -n — do nothing but read articles — does not feed articles downstream, writes no rnews file, does not update the config file. it's giving "just here for the vibes" 💀
• -N timeout — specify the timeout length, as timeout seconds, when establishing an NNTP connection.
• -O — use an optimized mode: pullnews checks whether the article already exists on the downstream server, before downloading it. it may help for huge articles or a slow link to upstream hosts. ur bandwidth is precious, protect it 🫶
• -p port — connect to the destination news server on a port other than the default of "119". this option does not change the port used to connect to the source news servers. important distinction, bestie.
• -P hop_limit — restrict feeding an article based on the number of hops it has already made. count the hops in the Path header field body (hop_count), feeding the article only when hop_limit is "+num" and hop_count is more than num; or hop_limit is "-num" and hop_count is less than num.
• -q — print out less status information while running. 🤫
• -Q level — set the quietness level ("-Q 2" is equivalent to "-q"). the higher this value, the less gets logged. the default is "0".
• -r file — rather than feeding the downloaded articles to a destination server, instead create a batch file that can later be fed to a server using rnews. see rnews(1) for more information about the batch file format. 🎀
• -R — be a reader (use MODE READER and POST commands) to the downstream server. some posts will then be rejected because of unexpected injection header fields, obsolete or incorrectly formatted header fields, or with a date too far in the past. you may then want to set artcutoff to "0" in inn.conf, and use the -H flag to strip unwanted header fields. even with that, a few articles may still be rejected. 😭

the default is to behave like a feeder and use the IHAVE command. (you'll have to allow in incoming.conf the connections from pullnews so that it is recognized as a feeder.)

• -s to-server[:port][_*tlsmode*] — normally, pullnews will feed the articles it retrieves to the news server running on localhost. to connect to a different host, specify a server with the -s flag. you can also specify the port with this same flag or use -p. default port is "119". 🔥

the connection is by default unencrypted. to negotiate a TLS encryption layer, you can set tlsmode to "TLS" for implicit TLS (negotiated immediately upon connection on a dedicated port) or "STARTTLS" for explicit TLS (the appropriate command will be sent before authenticating or feeding messages). examples of use are:

pullnews -s news.server.com
pullnews -s news.server.com_STARTTLS
pullnews -s news.server.com:433_TLS

note that not all NNTP servers implement TLS for feeding articles. check before u commit, bestie 💅

• -S max-run — specify the maximum time max-run in seconds for pullnews to run. ⏰
• -t retries — the maximum number (retries) of attempts to connect to a server or reconnect to a server if the socket is unexpectedly closed (see also -T). the default is "0".
• -T connect-pause — pause connect-pause seconds between connection retries (see also -t). the default is "1".
• -w num — set each group's high water mark (last received article number) to num. if num is negative, calculate Current+num instead (i.e. get the last num articles). therefore, a num of "0" will re-get all articles on the server; whereas a num of "-0" will get no old articles, setting the water mark to Current (the most recent article on the server). the difference between 0 and -0 is so real, don't sleep on it 🦋
• -x — if the -x flag is used, an Xref header field is added to any article that lacks one. it can be useful for instance if articles are fed to a news server which has xrefslave set in inn.conf. 🌈
• -z article-pause — sleep article-pause seconds between articles. the default is "0".
• -Z group-pause — sleep group-pause seconds between groups. the default is "0".

---

okay that's literally all of them and you ate that, bestie 👑 — there's more coming so don't go anywhere 🫶

🗂️ CONFIG FILE 🗂️

ok so the config file is literally the whole personality of pullnews — it's split into blocks, and each block is one remote server you're connecting to. one block per server bestie, don't mix them up 💅

A block kicks off with the host line, and hear me out: that line must have no leading whitespace. none. zero. it contains just the hostname of the remote server, plus optionally the port and TLS mode (same semantics as the -s flag, iykyk), and then optionally followed by the authentication details — that's the username and password for that server. ✨

Also lowkey important: you can provide auth details for the downstream server too! just add a host line for "localhost" or the hostname you specified with the -s flag in the config file, with no newsgroup to fetch. easy.

After the host line, you drop in one or more newsgroup lines. these start with whitespace, followed by the name of a newsgroup to retrieve. and please — only one newsgroup per line, okay? one. per. line. 🎀

Here's the slay part: pullnews will update the config file itself, adding the time the group was last checked and the highest numbered article that was successfully retrieved and transferred to the destination server. it uses that data to avoid doing duplicate work the next time it runs. it's giving self-aware queen energy. 👑

The full syntax is:

<host>[:<port>][_<tlsmode>] [<username> <password>]
    <group> [<time> <high>]
    <group> [<time> <high>]

where the <host> line must not have leading whitespace and the <group> lines must. 🫶

---

A typical configuration file would be:

# Format: group date high
data.pa.vix.com
    rec.bicycles.racing 908086612 783
    rec.humor.funny 908086613 18
    comp.programming.threads
nnrp.vix.com pull sekret
    comp.std.lisp
news.server.com:563_TLS joe password
    news.software.nntp

Note that an earlier run of pullnews has filled in details about the last article downloads from the two rec.* groups. the two comp.* groups and the news.* group were just added by the user and have not yet been checked. 🌸

The nnrp.vix.com server requires authentication, and pullnews will use the username "pull" and the password "sekret" (without any encryption layer). no cap, that's just how it is 🙈

The connection to news.server.com will be encrypted with implicit TLS on port 563. Joe's password won't be sent in plaintext. we love a secure king 💖

---

📁 FILES 📁

• pathbin/pullnews — The Perl script itself used to pull news from upstream servers and feed it to another news server. the main character fr fr 🌟
• pathdb/pullnews.marks or ~/pullnews.marks — The default config file. It is stored in pullnews.marks in pathdb when pullnews is run as the news user, or otherwise in the running user's home directory. so it depends where you're standing lol 🦋

---

🕰️ HISTORY 🕰️

pullnews was written by James Brister for INN. The documentation was rewritten in POD by Russ Allbery <eagle@eyrie.org>.

Geraint A. Edwards greatly improved pullnews, adding no more than 16 new recognized flags, fixing some bugs and integrating the backupfeed contrib script by Kai Henningsen, adding again 6 other flags. absolute legend behaviour, we stan 👑

---

👀 SEE ALSO 👀

• incoming.conf(5)
• rnews(1)

---

and that's the whole page bestie!! 💜 this one's from INN 2.7.4, dated 2024-05-19 — go forth and pull those news 🎀✨