RFC 5537
Netnews Architecture and Protocols
RFC 5537: Netnews Architecture and Protocols 🎀
- Authors: R. Allbery, Ed. (Stanford University) and C. Lindsey
- Category: Standards Track
- Date: November 2009
- Obsoletes: RFC 1036
Abstract ✨
okay so this doc is the big picture for Netnews systems 📰 It lays out how the whole architecture hangs together and what software has to do properly when it creates, moves, stores, and shows Netnews articles. It also spells out what any transport protocol has to deliver if it wants to carry and serve these articles. main character energy, fr fr 💅
Status of This Memo 🌸
So this is an Internet standards track protocol doc for the Internet community, and the authors are literally asking for discussion and suggestions for improvements 🫶 If you want the current standardization state and status, check the latest edition of the "Internet Official Protocol Standards" (STD 1). Distribution of this memo is unlimited, so share away 💜
Copyright Notice 🦋
Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved 🔒
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date of publication. Please read those documents carefully, because they explain your rights and restrictions for this document. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come with no warranty, just like the BSD License says 📜
Table of Contents 🧚
1. Introduction ....................................................3
1.1. Basic Concepts .............................................3
1.2. Scope ......................................................3
1.3. Requirements Notation ......................................3
1.4. Syntax Notation ............................................3
1.5. Definitions ................................................4
2. Transport .......................................................5
3. Duties of Agents ................................................6
3.1. General Principles .........................................6
3.2. The Path Header Field ......................................7
3.2.1. Constructing the Path Header Field ..................8
3.2.2. Path Header Field Example ...........................9
3.3. Article History and Duplicate Suppression .................10
3.4. Duties of a Posting Agent .................................11
3.4.1. Proto-Articles .....................................12
3.4.2. Multiple Injection of Articles .....................13
3.4.3. Followups ..........................................14
3.4.4. Construction of the References Header Field ........15
3.5. Duties of an Injecting Agent ..............................15
3.5.1. Forwarding Messages to a Moderator .................18
3.6. Duties of a Relaying Agent ................................19
3.7. Duties of a Serving Agent .................................21
3.8. Duties of a Reading Agent .................................22
3.9. Duties of a Moderator .....................................22
3.10. Duties of a Gateway ......................................24
3.10.1. Duties of an Outgoing Gateway .....................25
3.10.2. Duties of an Incoming Gateway .....................25
3.10.3. Original-Sender Header Field ......................27
3.10.4. Gateway Example ...................................28
4. Media Types ....................................................29
4.1. application/news-transmission .............................30
4.2. application/news-groupinfo ................................31
4.3. application/news-checkgroups ..............................33
5. Control Messages ...............................................35
5.1. Authentication and Authorization ..........................35
5.2. Group Control Messages ....................................36
5.2.1. The newgroup Control Message .......................36
5.2.1.1. newgroup Control Message Example ..........37
5.2.2. The rmgroup Control Message ........................38
5.2.3. The checkgroups Control Message ....................38
5.3. The cancel Control Message ................................40
5.4. The Supersedes Header Field ...............................40
5.5. The ihave and sendme Control Messages .....................41
5.6. Obsolete Control Messages .................................42
6. Security Considerations ........................................42
6.1. Compromise of System Integrity ............................42
6.2. Denial of Service .........................................44
6.3. Leakage ...................................................44
7. IANA Considerations ............................................45
8. References .....................................................45
8.1. Normative References ......................................45
8.2. Informative References ....................................46
Appendix A. Changes to the Existing Protocols ....................47
Appendix B. Acknowledgements .....................................481. Introduction 🧚
1.1. Basic Concepts 📰
okay so "Netnews" is a whole family of protocols for making, storing, and fetching news "articles". Their format is defined in [RFC5536], and they're used to swap articles around a readership that can be spread out really widely 🌍 It's organized into "newsgroups", and the idea is that every reader can see all the articles posted to each newsgroup they take part in 👀 Usually these protocols use a flooding algorithm, which spreads copies through a network of participating servers. Typically each server keeps only one copy, and it hands that copy out on request to any reader who can reach that server ✨
"Usenet" is one particular worldwide, publicly accessible network built on the Netnews protocols. It's just one possible network though! Other Netnews deployments exist too, like ones inside particular organizations 🏢 This document talks about the more general Netnews architecture and protocols, so Usenet isn't the only star here 🌟
1.2. Scope 🎯
This document defines the architecture of Netnews systems and specifies the correct manipulation and interpretation of Netnews articles by software that originates, distributes, stores, and displays them. It deals with protocol issues that don't depend on transport protocols like the Network News Transfer Protocol (NNTP) [RFC3977], and it specifies the requirements Netnews puts on those underlying transport protocols. It also covers the handling of control messages 💅
The format and syntax of Netnews articles are specified in [RFC5536], and you should read that one alongside this doc, bestie 📖
1.3. Requirements Notation 🔑
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Those are the official BCP 14 key words, so pay attention to them 🙈
1.4. Syntax Notation ✍️
The syntax in this document is written in Augmented Backus-Naur Form (ABNF) notation, including the Core Rules, which are defined in [RFC5234]. It also uses constructs defined in [RFC5536] and [RFC5322] 🧩
These are the ABNF rules that are defined somewhere else but get used in this document:
CRLF = <see [RFC5234] Appendix B.1>
DIGIT = <see [RFC5234] Appendix B.1>
HTAB = <see [RFC5234] Appendix B.1>
SP = <see [RFC5234] Appendix B.1>
WSP = <see [RFC5234] Appendix B.1>
VCHAR = <see [RFC5234] Appendix B.1>
argument = <see [RFC5536] Section 3.2.3>
article-locator = <see [RFC5536] Section 3.2.14>
component = <see [RFC5536] Section 3.1.4>
control-command = <see [RFC5536] Section 3.2.3>
diag-keyword = <see [RFC5536] Section 3.1.5>
diag-match = <see [RFC5536] Section 3.1.5>
diag-other = <see [RFC5536] Section 3.1.5>
dist-name = <see [RFC5536] Section 3.2.4>
msg-id = <see [RFC5536] Section 3.1.3>
newsgroup-name = <see [RFC5536] Section 3.1.4>
path-diagnostic = <see [RFC5536] Section 3.1.5>
path-identity = <see [RFC5536] Section 3.1.5>
path-nodot = <see [RFC5536] Section 3.1.5>
tail-entry = <see [RFC5536] Section 3.1.5>
verb = <see [RFC5536] Section 3.2.3>
display-name = <see [RFC5322] Section 3.4>
local-part = <see [RFC5322] Section 3.4.1>
mailbox = <see [RFC5322] Section 3.4>1.5. Definitions 💜
Any term used in this document that is defined in Section 1.5 of [RFC5536] has the definition given there. On top of that, these terms get used too:
A "hierarchy" is the set of all newsgroups whose names share a first <component> (as defined in Section 3.1.4 of [RFC5536]). A "sub-hierarchy" is the set of all newsgroups whose names share several initial components 🌳
A "news server" splits further into the roles of "injecting agent", "relaying agent", and "serving agent" 🖥️ An "injecting agent" accepts a proto-article with the goal of distributing it to relaying and serving agents, and from there to readers. A "relaying agent" accepts articles from other relaying agents or injecting agents, then distributes them to other relaying agents or serving agents. A "serving agent" receives an article from a relaying agent or injecting agent and makes it available to readers 📬
A "user agent" splits into the roles of "posting agent" and "reading agent" 🙋 A "posting agent" is software that helps prepare a proto-article and then passes it to an injecting agent. A "reading agent" is software that fetches articles from a serving agent so they can be shown to a reader 👀
"Injecting" an article means an injecting agent processes a proto-article. Normally that happens once and only once for a given article 🎯 "Multiple injection" is passing the same article to multiple injecting agents, either one after another or all at once, by one or several posting agents 🔁
A "gateway" is software that does one of three things: it receives news articles and converts them into messages of some other kind (such as [RFC5322] mail messages), it receives messages of some other kind and converts them into news articles, or it conveys articles between two separate Netnews networks 🌉 Gateways are lowkey the bridge builders of the whole system 🫶
Part 1 of 11. Next up: Section 2 (Transport) 🦋
2. Transport ✨
okay so the transport layer is literally the part where articles get from one agent to another, and the spec is like "idk, use whatever you want" 💅 the most common choice is NNTP [RFC3977] (very iconic, very standard), but people have also used the Unix-to-Unix Copy Protocol [UUCP] (super vintage, very early Usenet vibes 🌸) and even physical magnetic and optical media, like literally mailing a disk 💀. Any mechanism is fine as long as it can meet the requirements written here 🫶
Okay now the important part 👀 Transports for Netnews articles MUST treat articles as uninterpreted sequences of octets. They have to leave alone the value %d00 (which may not show up in Netnews articles at all 🙈), %d13, and %d10. Only %d10 is allowed, and it MUST only appear as a pair in that order (%d13 then %d10), and together they mark a line separator 🍓 These octets are the US-ASCII [ASCII] characters NUL, CR, and LF, respectively.
NOTE: This lines up with the range of octets allowed in MIME 8bit data [RFC2045]. Transports for Netnews are not required to support transmission of MIME binary data.
Also, transports MUST convey all header fields unmodified, even the ones hiding inside message/rfc822 objects in article bodies, and even if they have octets in the range of 128 to 255 🎀 Furthermore, transports for relaying and serving agents MUST convey lines, and transports for other agents SHOULD convey lines too, even when those lines are longer than 998 characters, especially in article bodies. This is stricter than MIME 8bit data, so no cap, it's a big deal ✨ These requirements cover the transport paths between posting agents, injecting agents, serving agents, and reading agents 🦋
3. Duties of Agents
this section is all about who has to do what to a Netnews article 🧚 The whole life of a typical Usenet article gets walked through: a posting agent writes it, an injecting agent takes it in, one or more relaying agents pass it along, a serving agent accepts it, and finally a reading agent retrieves it 💜 Articles posted to moderated groups get an extra process that's described separately (see Section 3.5.1 and Step 7 of Section 3.5). And at the end, the extra duties and requirements of a gateway get talked about 👑
At each step, every agent has a set of checks and transformations it's required to do to the article. They're written as step-by-step sequences, but the EFFECT of those sequences is what really matters, and implementations can use any method that gets the same effect 💅
Lots of news servers bundle the injecting, relaying, and serving agent functions into one software package 📦 For this spec, a combined agent should conceptually be treated as an injecting agent that sends articles to a serving agent and, optionally, to a relaying agent. Even so, the requirements of all three agents MUST still be met when the news server is doing those jobs 🫶
On news servers that accept them, control messages may have extra effects beyond the ones described below 👀 Those effects are covered in Section 5. 🌈
3.1. General Principles
there are two super important principles news implementors and administrators need to keep in their heads ✨ The first is the famous Internet Robustness Principle:
Be liberal in what you accept, and conservative in what you send.
For Netnews, this mostly means that unwanted or non-compliant articles SHOULD be rejected as early as possible 🙅♀️ But once an article is already out in general circulation, relaying and serving agents may wish to accept it where they can, rather than losing information. So posting agents and injecting agents SHOULD be maximally strict about following this protocol and [RFC5536] 💖 and reading agents SHOULD stay robust when the Netnews article format gets violated, where possible 🦋
The second one is EVEN MORE important for Netnews 😭 It comes from a much older code of practice, the Hippocratic Oath, so we can call it the Hippocratic Principle:
First, do no harm.
It's so vital to realize that decisions that are only suboptimal in a small setting can turn into devastating mistakes once thousands of hosts amplify them within a few minutes 💀
No Netnews agent is ever required to accept any article. It's super common for injecting, relaying, and serving agents to reject articles that are perfectly well-formed, just because of local policy (like not wanting to carry a certain newsgroup, or trying to filter out unwanted articles) 🙈 This document says how articles should be handled if they are accepted, and it names some cases where they MUST be rejected. But an agent MAY always reject any article for reasons other than the ones stated here 🍓
A big goal of the Netnews protocol is making sure every reader who gets a particular article (identified uniquely by the content of its Message-ID header field) sees the identical article 🎀 The only allowed differences are in trace headers and local metadata. So agents (other than moderators) MUST NOT modify articles in ways other than described here. And unacceptable articles MUST be rejected rather than corrected 🚫✨
3.2. The Path Header Field
every news server component (injecting agents, relaying agents, and serving agents) MUST identify itself whenever it processes an article 🪞 It does this by prepending its <path-identity> (defined in Section 3.1.5 of [RFC5536]) to the Path header field. Injecting agents MUST also use that same identity in the Injection-Info header fields they add, and serving and relaying agents SHOULD use the same identity in any Xref header fields they add 💜
An agent can pick its <path-identity> using one of these methods (in decreasing order of preference):
- The fully qualified domain name (FQDN) of the system the agent is running on 👑
- A fully qualified domain name (FQDN) inside a domain that's affiliated with the agent's administrators and guaranteed to be unique by that domain's administrators. For example, the uniqueness of
server.example.orgcould be guaranteed by the administrator ofexample.org, even if there's no DNS record forserver.example.orgitself 🌸 - Some other arbitrary name in the form of a
<path-nodot>, which is believed to be unique and registered at least with all the other news servers that the relaying or injecting agent sends articles to. This option SHOULD NOT be used unless the earlier options aren't available, or unless the name has been around for a long time 🥹
Some existing implementations treat <path-identity> as case-sensitive, and some treat it as case-insensitive, which is chaotic 😭 So the <path-identity> SHOULD be all lowercase, and implementations SHOULD compare identities case-insensitively 🫶
3.2.1. Constructing the Path Header Field
If a relaying or serving agent gets an article from an injecting or serving agent that's part of the same news server, it MAY leave the Path header field unchanged 🦋 Otherwise, every injecting, relaying, or serving agent that accepts an article MUST update the Path header field like this. Note that the Path header field content gets built from right to left, by prepending elements 🧚
- The agent MUST prepend
!to thePathheader field content. - An injecting agent SHOULD prepend the
<path-diagnostic>!.POSTED, optionally followed by.and the FQDN or IP address of the source, to thePathheader field content 📝 - A relaying or serving agent SHOULD prepend a
<path-diagnostic>to thePathheader field content, and the<path-diagnostic>is picked like this 💅- If the expected
<path-identity>of the source of the article matches the leftmost<path-identity>of thePathheader field's content, use!(<diag-match>), which results in two consecutive!s 👯 - If the expected
<path-identity>of the source of the article does NOT match, use!.MISMATCH.followed by the expected<path-identity>of the source, or its IP address 🚨 - If the relaying or serving agent isn't willing or able to check the
<path-identity>, use!.SEEN.followed by the FQDN, IP address, or expected<path-identity>of the source 🙈
- If the expected
The "expected <path-identity> of the source of the article" is the <path-identity> of the injecting or relaying agent that passed the article to this relaying or serving agent. It's figured out from properties of the connection the article came in on, like an authentication identity or a peer IP address 🔍 Be aware that [RFC1036] didn't include <path-diagnostic> 👀 Implementations that predate this specification will just add single ! characters between <path-identity> strings.
- The agent MAY then prepend
!or!!to thePathheader field content, followed by an additional<path-identity>for itself, other than its primary one ✨ Using!!(and therefore adding a<diag-match>, since that<path-identity>is clearly verified) is RECOMMENDED 🥰 This step may be repeated any number of times. It's allowed for agents that have multiple<path-identity>s, like during a transition from one to another. Each of these<path-identity>s MUST meet the requirements set out in Section 3.2 🌈 - Finally, the agent MUST prepend its primary
<path-identity>to thePathheader field content. The primary<path-identity>is the one it normally advertises to its peers, so they can use it when generating<path-diagnostic>s like we talked about above 💖
Any agent that modifies the Path header field MAY fold it by inserting FWS (folding white space) immediately after any <path-identity> or <diag-other> it added (see Section 3.1.5 of [RFC5536] for the allowable locations for FWS) 🎀
3.2.2. Path Header Field Example ✨
Okay so here's a juicy Path header field example 💅 built by following the rules for injecting and relaying agents. Main character energy 👑
Path: foo.isp.example!.SEEN.isp.example!foo-news
!.MISMATCH.2001:DB8:0:0:8:800:200C:417A!bar.isp.example
!!old.site.example!barbaz!!baz.isp.example
!.POSTED.dialup123.baz.isp.example!not-for-mailThis article was injected by baz.isp.example, and you can tell because of the <diag-keyword> "POSTED" 🔥 The injector recorded that it got the article from dialup123.baz.isp.example. "not-for-mail" is a super common <tail-entry> 🍓
The article was relayed to the relaying agent known, at least to old.site.example, as "barbaz". That relaying agent confirmed to its satisfaction that "baz.isp.example" was an expected <path-identity> for the source of the article, so it used <diag-match> (!) for its <path-diagnostic> ✨
barbaz relayed it to old.site.example, which does not support <diag-keyword> and therefore used the old "!" delimiter. This indicates that the identity of "barbaz" was not verified and may have been forged 🙈
old.site.example relayed it to a news server using the <path-identity> of bar.isp.example, and claimed (by using the "!" <path-diagnostic>) to have verified that it came from old.site.example. Bold claim, bestie 💀
bar.isp.example relayed it to foo-news, which, not being convinced that it truly came from bar.isp.example, inserted the <diag-keyword> "MISMATCH" and then stated that it received the article from the IPv6 address [2001:DB8:0:0:8:800:200C:417A]. (This is not to say that bar.isp.example was not a correct <path-identity> for that source, but simply that the identity did not match the expectations of foo-news.) 🦋
foo-news then passed the article to foo.isp.example, which declined to validate its <path-identity> and instead appended the <diag-keyword> "SEEN" to indicate that it knows the source of the article as isp.example. This may be either an expected <path-identity> or the FQDN of the system from which it received the article. Presumably, foo.isp.example is a serving agent that then delivered the article to a reading agent 🧚
baz.isp.example, bar.isp.example, and foo-news all folded the Path header field 🌈
3.3. Article History and Duplicate Suppression 🫶
Netnews normally uses a flood-fill algorithm to propagate articles. Each news server offers the articles it accepts to multiple peers, and each news server may be offered the same article from multiple other news servers. So duplicate suppression is key 🔑 If a news server accepted every article it was offered, it may needlessly accept (and then potentially retransmit) dozens of copies of every single article. Yikes 😭
Relaying and serving agents therefore MUST keep a record of articles they have already seen, and use that record to reject additional offers of the same article. This record is called the "history" file or database 📚
Each article is uniquely identified by its message identifier, so a relaying or serving agent could satisfy this requirement by storing a record of every message identifier that agent has ever seen. But a history database like that would grow without bound, so it is common and permitted to optimize based on the Injection-Date or Date header field of an article. (In the following discussion, the "date" of an article is defined to be the date represented by its Injection-Date header field, if present; otherwise, by its Date header field.) ✨
- Agents MAY select a cutoff interval and reject any article with a date farther in the past than that cutoff interval. If this interval is shorter than the time it takes for an article to propagate through the network, the agent might reject an article it had not yet seen, so it ought not to be aggressively short. For Usenet, for example, a cutoff interval of no less than seven days is conventional 📅
- Agents that enforce such a cutoff MAY then drop records of articles that had dates older than the cutoff from their history databases. If such an article were offered to the agent again, it would be rejected due to the cutoff date, so the history record is no longer required to suppress the duplicate 💅
- Alternatively, agents MAY drop history records according to the date when the article was first seen by that agent rather than the date of the article. In this case, the history retention interval MUST be at least 24 hours longer than the cutoff interval to allow for articles dated in the future. This interval matches the allowable error in the date of the article (see Section 3.5) 🌸
These are just two implementation strategies for article history, albeit the most common ones. Relaying and serving agents are not required to use these strategies, only to meet the requirement of not accepting an article more than once. However, these strategies are safe and widely deployed, and implementors are encouraged to use one of them, especially if they do not have extensive experience with Netnews and the subtle effects of its flood-fill algorithm. Trust me on this one 🙏
3.4. Duties of a Posting Agent 💌
A posting agent is the component of a user agent that helps a poster create a valid proto-article and forward it to an injecting agent. It's basically the hype person for your post 📣
Posting agents SHOULD ensure that proto-articles they create are valid according to [RFC5536] and any other applicable policies. They MUST NOT create any Injection-Info header field; this header field may only be added by the injecting agent. Hands off, bestie 🙅♀️
If the proto-article already contains both Message-ID and Date header fields, posting agents MAY add an Injection-Date header field to that proto-article immediately before passing it to an injection agent. They SHOULD do so if the Date header field (representing the composition time of the proto-article) is more than a day in the past at the time of injection. They MUST do so if the proto-article is being submitted to more than one injecting agent; see Section 3.4.2 🕰️
The Injection-Date header field is new in this revision of the Netnews protocol, and it's designed to let the Date header field hold the composition date (as recommended in Section 3.6.1 of [RFC5322]), even if the proto-article isn't injected until some time after it was composed. However, note that all implementations predating this specification ignore the Injection-Date header field and use the Date header field in its stead when rejecting articles older than their cutoff (see Section 3.3), and injecting agents predating this specification do not add an Injection-Date header. Articles with a Date header field substantially in the past will still be rejected by implementations predating this specification, regardless of the Injection-Date header field, and hence may suffer poorer propagation. Old tech be like 🦖
Contrary to [RFC5322], which implies that the mailbox or mailboxes in the From header field should be those of the poster or posters, a poster who does not, for whatever reason, wish to use their own mailbox MAY use any mailbox ending in the top-level domain ".invalid" [RFC2606]. Privacy queen behavior 👑🫣
Posting agents meant for use by ordinary posters SHOULD reject any attempt to post an article that cancels or supersedes (via the Supersedes header field) another article of which the poster is not the author or sender. No messing with other people's posts, periodt 🚫
3.4.1. Proto-Articles 📝
A proto-article is an article in the format used by a posting agent when offering that article to an injecting agent. It may omit certain header fields that can be better supplied by the injecting agent, and it will not contain header fields that are added by the injecting agent. A proto-article is only for transmission to an injecting agent and SHOULD NOT be transmitted to any other agent. It's a pre-game, not the main event 🎀
A proto-article has the same format as a normal article, except that the Injection-Info and Xref header fields MUST NOT be present, the Path header field SHOULD NOT contain a "POSTED" <diag-keyword>, and any of the following mandatory header fields MAY be omitted: Message-ID, Date, and Path. In all other respects, a proto-article MUST be a valid Netnews article. In particular, the header fields that may be omitted MUST NOT be present with invalid content 🚨
If a posting agent intends to offer the same proto-article to multiple injecting agents, the header fields Message-ID, Date, and Injection-Date MUST be present and identical in all copies of the proto-article. See Section 3.4.2 🍓
3.4.2. Multiple Injection of Articles 🌀
Under some circumstances (for example, when posting to multiple, supposedly disjoint, networks, when using injecting agents with spotty connectivity, or when desiring additional redundancy), a posting agent may wish to offer the same article to multiple injecting agents. In this unusual case, the goal is not to create multiple independent articles. Instead, the goal is to inject the same article at multiple points and let the normal duplicate suppression facility of Netnews (see Section 3.3) make sure that any given agent accepts the article only once, even if supposedly disjoint networks have unexpected links 🔗
Whenever possible, multiple injection SHOULD be done by offering the same proto-article to multiple injecting agents. The posting agent MUST supply the Message-ID, Date, and Injection-Date header fields, and the proto-article as offered to each injecting agent MUST be identical 🫶
In some cases, offering the same proto-article to all injecting agents may not be possible (such as when gatewaying articles found on one Netnews network to another supposedly unconnected one after injection). In this case, the posting agent MUST remove any Xref header field and rename or remove any Injection-Info, Path, and other trace header fields before passing it to another injecting agent. (This converts the article back into a proto-article.) It MUST retain the Message-ID, Date, and Injection-Date header fields unmodified. It MUST NOT add an Injection-Date header field if it is missing from the existing article 🧩
NOTE: Multiple injection inherently risks duplicating articles. Multiple injection after injection, by converting an article back to a proto-article and injecting it again, additionally risks loops, loss of trace information, unintended repeat injection into the same network, and other problems. It should be done with care and only when there is no alternative. The requirement to retain Message-ID, Date, and Injection-Date header fields minimizes the possibility of a loop and ensures that the newly injected article is not treated as a new, separate article. Be careful out there 😭⚠️
Multiple injection of an article that lists one or more moderated newsgroups in its Newsgroups header field SHOULD only be done by a moderator, and MUST only be done after the proto-article has been approved for all moderated groups to which it is to be posted and after an Approved header field has been added (see Section 3.9). Multiple injection of an unapproved article intended for moderated newsgroups will normally only result in the moderator receiving multiple copies. And if the newsgroup status is not consistent across all injecting agents, it may result in duplication of the article or other problems 👀
3.4.3. Followups 💌
okay so a followup is an article that replies to the content of an earlier article, aka its precursor 💬. On top of the usual posting agent duties, a posting agent making a followup has some extra rules to follow 🫶. Whenever it says a header field is inherited by default from a precursor header field, it means the starting content is a copy of that precursor header's content (folding is allowed to change) ✨. But posters MAY override that default before posting 💅.
Despite what some posting agents have historically done, the Keywords header field SHOULD NOT be inherited by default from the precursor article 🙈.
- okay so if the precursor's
Followup-Toheader field consists of "poster", the followup MUST NOT be posted by default 🙅♀️. Instead, by default it gets emailed to the address in the precursor'sReply-ToorFromheader field, following the rules for an email reply [RFC5322]. The poster MAY override this, and then the posting agent should continue as if the precursor'sFollowup-Toheader field didn't exist 🦋. - The
Newsgroupsheader field SHOULD, by default, be inherited from the precursor'sFollowup-Toheader field if present 📰. Otherwise, it's inherited from the precursor'sNewsgroupsheader field. - The
Subjectheader field SHOULD, by default, be inherited from the precursor'sSubject🎀. The case-sensitive string "Re: " (including the space after the colon) MAY be prepended to the content of itsSubjectheader field, unless it already begins with that string 💖.
NOTE: Prepending "Re: " does literally nothing for the protocol, so it's not required. But everyone expects it, and skipping it would be so surprising 😭
- The
Distributionheader field SHOULD, by default, be inherited from the precursor'sDistributionheader field, if present 🌈. - The followup MUST have a
Referencesheader field referring to its precursor, constructed in accordance with Section 3.4.4 🔗.
3.4.4. Construction of the References Header Field 🧵
this is the procedure to use whenever some previous article (the "parent") gets referred to in the References header field of a new article 🧚. That could be because the new article is a followup and the parent is its precursor, or because of some other reason.
The content of the new article's References header field MUST be formed from the content of the parent's References header field if present, followed by the content of the parent's Message-ID header field 🫶. If the parent had a References header, FWS as defined in [RFC5536] MUST be added between its content and the Message-ID header field content 💜.
If the resulting References header field would, after unfolding, exceed 998 characters in length (including its field name but not the final CRLF), it MUST be trimmed (and otherwise MAY be trimmed) ✂️. Trimming means removing any number of message identifiers from its content, except that the first message identifier and the last two MUST NOT be removed 💅.
An essential property of the References header field, guaranteed by the above procedure and REQUIRED to be maintained by any extensions to this protocol, is that an article MUST NOT precede one of its parents 👑. this is the kind of rule that keeps the whole thread in order, bestie 🧚
3.5. Duties of an Injecting Agent 🧁
an injecting agent takes a proto-article from a posting agent and either forwards it to a moderator or passes it along to a relaying or serving agent (or agents) 🚀. It bears the primary responsibility for making sure any article it injects follows the rules of the Netnews standards. The administrator of an injecting agent is also expected to take on some responsibility towards the rest of the Netnews network it's connected to, for the articles the injecting agent accepts 🌍.
Injecting agents, when rejecting articles, are encouraged to communicate the reason for rejection to the posting agent, using whatever facility the underlying transport provides ✉️. The injecting agent is in a unique position to explain why, because relaying agents and serving agents normally have to reject messages silently 🤫. So the injecting agent carries much of the burden of diagnosing broken posting agents and communicating policy violations to posters 🙋.
An injecting agent MUST have available a list (possibly empty) of moderated groups for which it accepts articles, along with the corresponding submission addresses 📋. It SHOULD have available a list of valid newsgroups, to catch articles not posted to a valid newsgroup (which are therefore likely to be silently discarded by relaying and serving agents) 🗑️. Usually, an injecting agent is deployed alongside a serving agent and maintains these lists based on control messages received by the serving agent 🎀.
An injecting agent processes proto-articles as follows:
- It SHOULD verify that the article is from a trusted source (for example, by relying on the authorization capability of the underlying transport used to talk to the posting agent) 🔐.
- It MUST reject any proto-article that does not have the proper mandatory header fields for a proto-article, that has
Injection-InfoorXrefheader fields, that has aPathheader field containing the"POSTED"<diag-keyword>, or that is not syntactically valid as defined by [RFC5536]. It SHOULD reject any proto-article that contains a header field deprecated for Netnews (see, for example, [RFC3798]). It MAY reject any proto-article that contains trace header fields (e.g.,NNTP-Posting-Host) indicating that it was already injected by an injecting agent that did not addInjection-InfoorInjection-Date🚫. - It SHOULD reject any article whose
Injection-DateorDateheader field is more than 24 hours into the future (and MAY use a margin less than 24 hours) ⏰. It SHOULD reject any article whoseInjection-Dateheader field is too far in the past (older than the cutoff interval of a relaying agent that the injecting agent is using, for example). It SHOULD similarly reject any article whoseDateheader field is too far in the past, since not all news servers supportInjection-Date, and only the injecting agent can give a useful error message to the posting agent 📣. In either case, this interval SHOULD NOT be any shorter than 72 hours into the past 🕰️. - It SHOULD reject any proto-article whose
Newsgroupsheader field does not contain at least one<newsgroup-name>for a valid group. It also SHOULD reject one that contains a<newsgroup-name>reserved for specific purposes by Section 3.1.4 of [RFC5536], unless that specific purpose or local agreement applies to the proto-article being processed 🗂️. Crossposting to unknown newsgroups is not precluded, provided that at least one of the newsgroups in theNewsgroupsheader is valid ✅. - The
Message-IDandDateheader fields with appropriate contents MUST be added when not present in the proto-article 🪪. - The injecting agent MUST NOT alter the body of the article in any way (including any change of
Content-Transfer-Encoding) 🙅. It MAY add other header fields not already provided by the poster, but injecting agents are encouraged to use theInjection-Infoheader for such information and to minimize the addition of other headers 🧾. It SHOULD NOT alter, delete, or reorder any existing header field except thePathheader field. It MUST NOT alter or delete any existingMessage-IDheader field 🔒. - If the
Newsgroupsheader contains one or more moderated groups and the proto-article does not contain anApprovedheader field, the injecting agent MUST either forward it to a moderator as specified in Section 3.5.1 or, if that is not possible, reject it 👩⚖️. This forwarding MUST be done after adding theMessage-IDandDateheaders if required, and before adding theInjection-InfoandInjection-Dateheaders 🎯. - Otherwise, a
Pathheader field with a<tail-entry>MUST be added if not already present 🛤️. - The injecting agent MUST then update the
Pathheader field as described in Section 3.2.1 🧭. - An
Injection-Infoheader field SHOULD be added that identifies the source of the article and possibly other trace information, as described in Section 3.2.8 of [RFC5536] 🕵️♀️. - If the proto-article already had an
Injection-Dateheader field, it MUST NOT be modified or replaced 🙌. If the proto-article had both aMessage-IDheader field and aDateheader field, anInjection-Dateheader field MUST NOT be added, since the proto-article may have been multiply injected by a posting agent that predates this standard 🫠. Otherwise, the injecting agent MUST add anInjection-Dateheader field containing the current date and time 📅. - Finally, the injecting agent forwards the article to one or more relaying agents, and the injection process is complete 🦋✨.
3.5.1. Forwarding Messages to a Moderator 📬
3.5.1. Forwarding Messages to a Moderator 📬
Okay bestie, an injecting agent MUST forward the proto-article to the moderator of the leftmost moderated group listed in the Newsgroups header field, and that's customarily done over email 💌. There are two standard ways to do it:
- The whole proto-article gets wrapped up inside the email, header fields and all 🎁. This SHOULD be done by making an email with a
Content-Typeofapplication/news-transmissionand the usage parameter set to"moderate". The body SHOULD NOT contain anything except the message. The perk here is that it kills any possible clash between Netnews and email header fields, plus any changes those fields might pick up on the way through email ✨. - The proto-article gets sent as a regular email, with whatever header fields email needs as defined in [RFC5322] added on, and maybe other email-y ones like
ToandReceived💅. The existingMessage-IDheader field SHOULD be retained.
Both of these have been used before, and the first has clear technical perks, but the second is way more common and lots of moderators aren't set up to handle the first format 🙈. So the first method SHOULD NOT be used unless the moderator it's being forwarded to is known to be able to handle it 🫶.
NOTE: Figuring out the email address of a group's moderator is outside the scope of this document, so we're not going there 🙃. Still, it's worth knowing that a common move is using a forwarding service that handles submissions for a bunch of moderated groups. For maximum compatibility with existing news servers, these services usually build the submission address for a moderated group by replacing each"."in the<newsgroup-name>with"-", then using that value as the<local-part>of a<mailbox>formed by appending a set domain.
Forwarding proto-articles to moderators by email is the most general method and the most common one on big Netnews networks like Usenet 🌍, but any means of forwarding the article that preserves it without injecting it MAY be used. For example, if the injecting agent can get into a database the moderator uses to store proto-articles waiting to be processed, it may place the proto-article straight into that database 🗄️. These methods may be more appropriate for smaller Netnews networks 🧚.
3.6. Duties of a Relaying Agent 🔁
A relaying agent accepts injected articles from injecting agents and other relaying agents, then passes them on to relaying or serving agents 💫. To avoid anyone bypassing injecting agent policies or forging Path and Injection-Info headers, relaying agents SHOULD accept articles only from trusted agents 🔒.
An article SHOULD NOT be relayed unless the sending agent has been configured to supply, and the receiving agent to receive, at least one of the <newsgroup-name>s in its Newsgroups header field and at least one of the <dist-name>s in its Distribution header field (if present) 📍. Exceptionally, control messages that create or remove newsgroups (newgroup or rmgroup control messages, for example) SHOULD be relayed if the affected group appears in its Newsgroups header field and both the sending and receiving relaying agents are configured to relay a newsgroup of that name (whether or not such a newsgroup exists) 🌸.
To avoid pointless relaying attempts, an article SHOULD NOT be relayed if the <path-identity> of the receiving agent (or some known alias of it) shows up as a <path-identity> in its Path header field 🙅♀️. The only exceptions are inside the <tail-entry> or right after a "POSTED" <diag-keyword> 👀.
A relaying agent processes an article like this:
- It MUST reject any article without a
Newsgroupsheader field orMessage-IDheader field, or without either anInjection-DateorDateheader field 🚫. - It MUST examine the
Injection-Dateheader field or, if that's absent, theDateheader field, and reject the article if that date is more than 24 hours into the future ⏰. It MAY reject articles with dates in the future using a smaller margin than 24 hours. - It MUST reject any article that has already been accepted. If it implements one of the mechanisms described in Section 3.3, this means that it MUST reject any article whose date falls outside the cutoff interval, since it won't know whether or not such articles had been accepted previously 🧠.
- It SHOULD reject any article that does not include all the mandatory header fields. It MAY reject any article that contains header fields that do not have valid contents 📋.
- It SHOULD reject any article that matches an already-received cancel control message, or the contents of the
Supersedesheader field of an accepted article, provided that the relaying agent has chosen (based on local site policy) to honor that cancel control message orSupersedesheader field 🗑️. - It MAY reject any article without an
Approvedheader field that's posted to a newsgroup known to be moderated. This practice is strongly encouraged, but the information needed to do so is not required to be maintained by a relaying agent 💁♀️. - It MUST update the
Pathheader field as described in Section 3.2.1 🛤️. - It MAY delete any
Xrefheader field already present. It MAY add a newXrefheader field for its own use (but recall that [RFC5536] permits at most one such header field) 🏷️. - Finally, it passes the article on to other relaying and serving agents it's configured to send articles to 🚀.
Relaying agents SHOULD, where possible in the underlying transport, let the agent that passed the article to them know if the article was rejected 📣. Relaying agents MUST NOT inform any other external entity of the rejection of an article, unless that external entity has explicitly asked to be told about such errors 🤐.
Relaying agents MUST NOT alter, delete, or rearrange any part of an article except for the Path and Xref header fields. They MUST NOT modify the body of articles in any way. If an article isn't acceptable as is, the article MUST be rejected rather than modified 🙅♀️💖.
3.7. Duties of a Serving Agent 🗄️
A serving agent accepts articles from a relaying agent or injecting agent, stores them, and makes them available to reading agents 📚. Articles are normally indexed by newsgroup and <article-locator> (Section 3.2.14 of [RFC5536]), usually as a decimal number 🔢.
If the serving agent stores articles by newsgroup, control messages SHOULD NOT be stored in the newsgroups listed in the control message's Newsgroups header field. Instead, they SHOULD be stored in a newsgroup in the hierarchy "control", which is reserved for this purpose 🗂️. Conventionally, control messages are stored in newsgroups named after the type of control message (such as "control.cancel" for cancel control messages) 🏷️.
A serving agent MUST have available a list (possibly empty) of moderated groups it accepts articles for, so that it can reject unapproved articles posted to moderated groups 📝. Often a serving agent is deployed alongside an injecting agent and can use the same list as the injecting agent 👯♀️.
A serving agent processes articles like this:
- It MUST reject any article that does not include all the mandatory header fields, or any article that contains header fields that do not have valid contents 🚫.
- It MUST examine the
Injection-Dateheader field or, if that's absent, theDateheader field, and reject the article if that date is more than 24 hours into the future ⏰. It MAY reject articles with dates in the future using a smaller margin than 24 hours. - It MUST reject any article that has already been accepted. If it implements one of the mechanisms described in Section 3.3, this means that it MUST reject any article whose date falls outside the cutoff interval, since it won't know whether or not such articles had been accepted previously 🧠.
- It SHOULD reject any article that matches an already-received and honored cancel message or
Supersedesheader field, following the same rules as a relaying agent (Section 3.6) 🗑️. - It MUST reject any article without an
Approvedheader field that's posted to any newsgroup listed as moderated ✅. - It MUST update the
Pathheader field as described in Section 3.2.1 🛤️. - It MUST remove any
Xrefheader field from each article (except when specially configured to preserve the<article-locator>s set by the sending site). It then MAY (and usually will) add a newXrefheader field (but recall that [RFC5536] permits at most one such header field) 🏷️. - Finally, it stores the article and makes it available for reading agents 🎉.
Serving agents MUST NOT create new newsgroups simply because an unrecognized <newsgroup-name> shows up in a Newsgroups header field 🙅♀️. Newsgroups are normally created via control messages (Section 5.2.1) 🌱.
Serving agents MUST NOT alter, delete, or rearrange any part of an article except for the Path and Xref header fields. They MUST NOT modify the body of the articles in any way. If an article isn't acceptable as is, the article MUST be rejected rather than modified 💖.
3.8. Duties of a Reading Agent 📖
Since a reading agent is only a passive participant in a Netnews network, there are no specific protocol requirements placed on it 🫧. See [USEAGE] for best-practice recommendations 🌷.
3.9. Duties of a Moderator 👑
okay so a moderator is the bestie who gets news articles (customarily by email 💌), decides whether to approve them, and if so, either passes them to an injecting agent or forwards them to further moderators 🦋.
Articles usually reach the moderator in email, either wrapped up as an object of Content-Type application/news-transmission (or wrapped but with no explicit Content-Type header field 🙈), or just straight up as an email that already has every header field a Netnews article needs (see Section 3.5.1). Moderators who might get articles by email SHOULD be ready to accept either format 💅.
There are no protocol restrictions on what criteria you use to accept or reject messages, or what changes a moderator can make to a message (header fields and body) before injecting it 🫶. Recommended best practice tho is to make the minimal required changes 💖. Mods need to be aware that changes they make can invalidate signatures created by the poster or earlier moderators 💀. See [USEAGE] for more best-practice recs ✨.
Moderators process articles like this:
- They decide whether to approve or reject a proto-article 🧐 and, if approved, prep it for injection. If the proto-article came in as an unencapsulated email message, they'll have to convert it back into a valid Netnews proto-article. If the article is rejected, it's normally rejected for all the newsgroups it was posted to, and nothing further happens. If it's approved, the moderator moves on to the following steps.
- If the Newsgroups header field has further moderated newsgroups that haven't approved yet, they may either reach some agreement with the other moderators on what happens to the article or, more generally, add an indication (naming both the moderator and the name of the newsgroup) that they approve it, then forward it to the moderator of the leftmost unapproved newsgroup. This forwarding SHOULD follow the procedure in Section 3.5.1. It MAY be done by rotating the
<newsgroup-name>s in the Newsgroups header field so the leftmost unapproved newsgroup becomes the leftmost moderated newsgroup in that field, and then posting it, letting the injecting agent do the forwarding. However, when using this mechanism, they MUST first ensure that the article contains no Approved header field 🔥. - If the Newsgroups header field has no further unapproved moderated groups, they add an Approved header field (see Section 3.2.1 of [RFC5536]) identifying the moderator and, insofar as is possible, all the other moderators who have approved the article ✨. The moderator who takes this step takes on responsibility for making sure the article was approved by the moderators of all the moderated newsgroups it was posted to 👑.
- Moderators are encouraged to retain the Message-ID header field unless it's invalid or the article was significantly changed from its original form 🌸. Moderators are also encouraged to retain the Date header field unless it appears stale (72 hours or more in the past) for reasons the moderator understands (like delays in the moderation process). In that case, they MAY substitute the current date 🧚. Any Injection-Date, Injection-Info, or Xref header fields already present MUST be removed 🚫.
- Any Path header field MUST either be removed or truncated to only those entries following its
"POSTED" <diag-keyword>, if any 🦋. - The moderator then passes the article to an injecting agent, having first observed all the duties of a posting agent 💖.
3.10. Duties of a Gateway 🌈
okay gateways are the translators of the internet 🧚. A gateway transforms an article into the native message format of another medium, or translates the messages of another medium into news articles, or transforms articles into proto-articles for injection into a separate Netnews network. Encapsulating a news article into a message of MIME type application/news-transmission, or undoing that encapsulation later, is not gatewaying, since it involves no transformation of the article 💅.
There are two basic types of gateway. There's the outgoing gateway, which transforms a news article into a different type of message, and the incoming gateway, which transforms a message from another network into a news proto-article and injects it into a Netnews network. These get handled separately below ✨.
Transforming an article into another medium has a very high chance of discarding or interfering with the protection the news system has against duplicate articles 😭. The most common problem caused by gateways is loops that repeatedly reinject previously posted articles. To prevent this, a gateway MUST take precautions against loops, as detailed below 🔁.
The transformations applied to the message SHOULD be as minimal as possible while still getting the gatewaying done 💅. Every change a gateway makes potentially breaks a property of one of the media or loses information, so only those transformations made necessary by the differences between the media should be applied 🌸.
If bidirectional gatewaying (both an incoming and an outgoing gateway) is being set up between Netnews and some other medium, the incoming and outgoing gateways SHOULD be coordinated to avoid unintended reinjection of gated articles 🫶. Circular gatewaying (gatewaying a message into another medium and then back into Netnews) SHOULD NOT be done. Encapsulation of the article SHOULD be used instead where this is necessary 🙅.
Safe bidirectional gatewaying between a mailing list and a newsgroup is way easier if the newsgroup is moderated 👑. Posts to the moderated group and submissions to the mailing list can then go through a single point that does the necessary gatewaying and then sends the message out to both the newsgroup and the mailing list at the same time, which eliminates most of the chance of loops 🦋. Bidirectional gatewaying between a mailing list and an unmoderated newsgroup, in contrast, is difficult to do correctly and is far more fragile 💀. Newsgroups intended to be bidirectionally gated to a mailing list SHOULD therefore be moderated where possible, even if the moderator is just a simple gateway and injecting agent that correctly handles crossposting to other moderated groups and otherwise passes all traffic 🍓.
3.10.1. Duties of an Outgoing Gateway 🎀
From the perspective of Netnews, an outgoing gateway is just a special type of reading agent 👀. What the outgoing gateway needs to do to articles depends on the medium the articles are being gated to. Because it raises the danger of loops, thanks to the possibility of one or more corresponding incoming gateways gating things back from that medium to Netnews, the outgoing gateway's operation is subject to additional constraints 🙈.
The following practices are encouraged for all outgoing gateways, whether or not there's a known related incoming gateway. They're both precautionary measures and guidelines for quality of implementation ⭐:
- The message identifier of the news article should be preserved if at all possible, preferably as or within the corresponding unique identifier of the other medium. If it isn't preserved that way, then at least it should be preserved as a comment in the message. This helps a lot with preventing loops 🔁.
- The Date and Injection-Date of the news article should also be preserved if possible, for similar reasons 🗓️.
- The message should be tagged somehow so it can't be reinjected into Netnews 🏷️. This may be impossible without knowing about potential incoming gateways, but it's better to try to give some indication even if it doesn't work. At the very least, a human-readable indication that the article should not be gated back to Netnews can help locate a human problem 🧑💻.
- Netnews control messages should not be gated to another medium unless they would somehow be meaningful in that medium 🙅♀️.
3.10.2. Duties of an Incoming Gateway
okay so the incoming gateway has to make sure every rule in this protocol is met by the articles it forms 💅. On top of its special gateway jobs, it also carries all the duties of a posting agent, and it has the same job a relaying agent has: rejecting articles it has already gatewayed 🔥.
An incoming gateway MUST NOT gate the same message twice 🙈. Honestly, this might not be totally possible if the message gets mangled or modified 😭. But at the very least, if a gateway gets a copy of a message it already gated, and the only differences are trace header fields (like Received in Email or Path in Netnews), it MUST NOT gate that message again 💀. An incoming gateway SHOULD also take precautions against people sneaking past this rule with modifications it can predict 👑.
News articles made by gateways MUST be valid news proto-articles (see Section 3.4.1) ✨. This usually means the gateway has to build a conforming article out of messy, non-conforming input 🧚. Then the gateway MUST pass the article to an injecting agent, NOT straight to a relaying agent 🦋.
Incoming gateways MUST NOT pass control messages (articles with a Control or Supersedes header field) unless they remove or rename that header field 🚫. Gateways MAY, however, generate cancel control messages for messages they have gatewayed 🌸. If a gateway gets a message it can tell is a valid equivalent of a cancel control message in the medium it's gatewaying, it SHOULD discard that message without gatewaying it, generate a matching cancel control message of its own, and inject that cancel control message 🎀.
NOTE: It's not unheard of for mail-to-news gateways to be used to post control messages, but encapsulation should be used for these cases instead 🙈. Gateways are basically loop magnets. Spews of normal articles are bad enough, but spews of control messages with special meaning to the news system could mean high processing load, or even emails being sent for every message received, which is catastrophic 😭. It's way better to build a system specifically for posting control messages, one that can do proper consistency checks and authenticate who sent the message 🫶.
If there's a message identifier that does a similar job to the Message-ID header field in news, it SHOULD be used when forming the message identifier of the news article 🍓. You might need to transform it to meet Netnews's uniqueness requirement, and you'd remove any comments so it follows the syntax in Section 3.1.3 of [RFC5536] ✨. Those transformations SHOULD be designed so that two messages with the same identifier generate the same Message-ID header field 💜.
NOTE: Message identifiers are central to preventing duplicates, so when gateways use them correctly, that does a lot to stop loops 🌈. But Netnews requires message identifiers to be unique, so identifiers from other media may need modifying before they can be used 🤓. The gateway has to strike a balance between keeping the info that prevents loops and generating unique message identifiers 💅.
Exceptionally, if there are multiple incoming gateways for a particular set of messages, each going to a different newsgroup(s), each one SHOULD generate a message identifier that's unique to that gateway 🦋. Even so, each incoming gateway MUST ensure that it does not gate the same message twice 🔥.
NOTE: Picture two gateways for one mailing list, feeding two separate Usenet newsgroups, and both preserve the email message identifier 👀. Each newsgroup might get a portion of the messages, with different sites seeing different portions 😭. When there's no one "official" gateway like this, some other way of generating message identifiers has to be used to avoid collisions 🧚. It would obviously be preferable to have only one gateway that crossposts, but that may not be possible to coordinate 🙈.
If no date information is available, the gateway MAY supply a Date header field with the gateway's current date 📅. If only partial information is available (like a date but no time), it SHOULD be fleshed out into a full Date by adding default values, not by throwing that info away 🌸. Date information should only be discarded in very exceptional circumstances, because it plays an important role in preventing reinjection of old messages 💀.
An incoming gateway MUST add a Sender header field to the news article it forms, containing the <mailbox> of the administrator of the gateway 📬. Problems with the gateway may be reported to this <mailbox> 🫶. The <display-name> part of this <mailbox> SHOULD indicate that the entity responsible for injecting the message is a gateway 🎀. If the original message already had a Sender header field, it SHOULD be renamed to Original-Sender so its contents can be preserved 💖. See Section 3.10.3 for the specification of that header field ✨.
3.10.3. Original-Sender Header Field
okay this one is the Original-Sender header field 💅. It holds the content of a Sender header field from an original message that an incoming gateway received, so it's preserved while the gateway adds its own Sender header field 🎀. The content syntax uses syntax defined in [RFC5536] and [RFC5322] 🌈.
header =/ Original-Sender-header
Original-Sender-header
= "Original-Sender" ":" SP
Original-Sender-content
Original-Sender-content
= mailboxThe Permanent Message Header Field Repository entry for this header field is:
Header field name: Original-Sender
Applicable protocol: Netnews
Status: standard
Author/Change controller: IETF
Specification document(s): RFC 55373.10.4. Gateway Example
let's look at an example of the precautions that should be taken against loops 🧚. This is one particular combination of mail-to-news and news-to-mail gateways, built to handle two-way gatewaying between mailing lists and unmoderated groups 💜:
- The news-to-mail gateway preserves the message identifier of the news article in the generated email message. The mail-to-news gateway likewise preserves the email message identifier, as long as it's syntactically valid for Netnews. This lets the news system's built-in duplicate suppression act as the first line of defense against loops 🛡️✨.
- The news-to-mail gateway adds an X-* header field to all messages it generates. The mail-to-news gateway discards any incoming messages containing this header field. This holds up against mailing list managers that replace the message identifier, and against any number of email hops, as long as the other message header fields are preserved 🦋.
- The mail-to-news gateway prepends the host name it received the email message from to the content of the Path header field. The news-to-mail gateway refuses to gateway any message that contains the list server name in its Path header field (including in the tail section). This holds up against any amount of munging of the message header fields by the mailing list, as long as the email only goes through one hop 🌸.
- The mail-to-news gateway is designed never to generate bounces to the envelope sender. Instead, articles the news server rejects (for reasons that don't warrant silently discarding the message) result in a bounce message sent to an errors address that is known not to forward to any mailing lists. That way, the news administrators can handle them 🎀.
These precautions have worked well in practice at preventing loops for this particular application (two-way gatewaying between mailing lists and locally distributed newsgroups, where both gateways can be designed together) 🌈. General gatewaying to world-wide newsgroups poses extra difficulties 😬. You've got to be very wary of weird configurations, like a newsgroup gated to a mailing list that's in turn gated to a different newsgroup 💀.
4. Media Types ✨
this document defines several media types, which have been registered with IANA as provided for in [RFC4288] 📝.
The media type message/news, as previously registered with IANA, is hereby declared obsolete 🥀. The point of this media type was to define a standard way of sending news articles via mail for humans to read 📬. But it was never widely implemented, and agents that didn't recognize it treating it as application/octet-stream was counter-productive 😭. The media type message/rfc822 (defined in Section 5.2.1 of [RFC2046]) SHOULD be used in its place 💖.
The updated MIME media type definition of message/news is:
MIME type name: message
MIME subtype name: news
Required parameters: none
Optional parameters: none
Encoding considerations: same as message/rfc822
Security considerations: News articles may constitute "control
messages", which can have effects on a
host's news system beyond just addition
of information. Since control messages
may occur in normal news flow, most hosts
are suitably defended against undesired
effects already, but transmission of news
articles via mail may bypass
firewall-type defenses. Reading a news
article transmitted by mail involves no
hazards beyond those of mail, but
submitting it to news software for
processing should be done with care.
Interoperability considerations:
Rarely used, and therefore often
handled as application/octet-stream.
Disposition should by default be inline.
Published specification: RFC 5537
Applications that use this media type:
Some old mail and news user agents.
Intended usage: OBSOLETE
Author: Henry Spencer
Change controller: IETF⚠️ Security alert, bestie! News articles can be "control messages", and those can change a host's news system in ways way beyond just adding info 😱. Most hosts already defend against that stuff since control messages show up in normal news flow 🛡️. But sending news articles through mail can get around firewall-type defenses 💀. Reading one that came through mail is no riskier than mail already is, but feeding it into news software needs real care 🥺.
4.1. application/news-transmission 📨
okay bestie, this media type is for sending a complete news article so the recipient can inject it straight into Netnews 💌 it's one of the ways to mail articles to moderators (see Section 3.5.1), and it can also carry messages to an injecting agent. the best part is it skips turning an email into a Netnews proto-article, and it gives you a way to send a Netnews article using MIME through a transport that doesn't support 8bit data 🫶
The MIME media type definition of application/news-transmission is:
MIME type name: application
MIME subtype name: news-transmission
Required parameters: none
Optional parameters: One and only one of "usage=moderate",
"usage=inject", or "usage=relay".
Encoding considerations: A transfer-encoding different from that
of the article transmitted MAY be
supplied to ensure correct transmission
over some 7bit transport medium.
Security considerations: News articles may constitute "control
messages", which can have effects on a
host's news system beyond just addition
of information. Since control messages
may occur in normal news flow, most hosts
are suitably defended against undesired
effects already, but transmission of news
articles via mail may bypass
firewall-type defenses.
Published specification: RFC 5537
Body part: A complete proto-article ready for
injection into Netnews or an article
being relayed to another agent.
Applications that use this media type:
Injecting agents, Netnews moderators.
Intended usage: COMMON
Change controller: IETFusage=moderate means the article is meant for a moderator, usage=inject is for an injecting agent, and usage=relay is for a relaying agent 🦋 if the receiving entity only runs one type of agent, the parameter MAY be omitted, and the entity receiving the article may only implement one type of agent in that case. ✨
Quick heads up: the old registration allowed article batches as a body part, but that's no longer a thing 🙈 article batches are not permitted now. you can send multiple messages instead, or a message with multiple application/news-transmission parts. slay 💅
4.2. application/news-groupinfo 🌸
the application/news-groupinfo media type goes along with the newgroup control message (see Section 5.2.1). its body part holds brief info about a newsgroup: the newsgroup name, its description, and its moderation status 🎀
MIME type name: application
MIME subtype name: news-groupinfo
Required parameters: none
Optional parameters: charset, which MUST be a charset
registered for use with MIME text types.
It has the same syntax as the parameter
defined for text/plain [RFC2046].
Specifies the charset of the body part.
If not given, the charset defaults to
US-ASCII [ASCII].
Encoding considerations: 7bit or 8bit encoding MUST be used to
maintain compatibility.
Security considerations: None.
Interoperability considerations:
Disposition should by default be inline.
Applications that use this media type:
Control message issuers, relaying
agents, serving agents.
Published specification: RFC 5537
Intended usage: COMMON
Change controller: IETFthe body part is defined like this, and yes the formatting is a little unusual 👀
groupinfo-body = [ newsgroups-tag CRLF ]
newsgroups-line CRLF
newsgroups-tag = %x46.6F.72 SP %x79.6F.75.72 SP
%x6E.65.77.73.67.72.6F.75.70.73 SP
%x66.69.6C.65.3A
; case sensitive
; "For your newsgroups file:"
newsgroups-line = newsgroup-name
[ 1*HTAB newsgroup-description ]
[ *WSP moderation-flag ]
newsgroup-description
= eightbit-utext *( *WSP eightbit-utext )
moderation-flag = SP "(" %x4D.6F.64.65.72.61.74.65.64 ")"
; SPACE + case sensitive "(Moderated)"
eightbit-utext = VCHAR / %d127-255the reason it's so weird is backward compatibility 🧚 it matches how older Netnews software scanned the body of newgroup control messages for descriptions. the <newsgroups-tag> is optional, but it SHOULD be included for backward compatibility 💜
the <newsgroup-description> MUST NOT contain the string "(Moderated)" anywhere in it, and moderated newsgroups MUST be marked by adding the case-sensitive text " (Moderated)" at the very end ✨
ngl the charset stuff is a whole thing 😭 a charset parameter exists, but most software doesn't understand MIME header fields or handle descriptions in different charsets properly. so using US-ASCII where possible is RECOMMENDED, and if that's not possible, UTF-8 [RFC3629] SHOULD be used. whatever charset you pick, the grammar above MUST still be met, and the <newsgroup-name> MUST be represented in that charset using the same octets it would use with US-ASCII. no cap, that's a lot of rules 💀
4.3. application/news-checkgroups 🦋
the application/news-checkgroups media type is a list of newsgroups in one or more hierarchies, including their descriptions and moderation status. it's mainly for the checkgroups control message (see Section 5.2.3) 🍓
MIME type name: application
MIME subtype name: news-checkgroups
Required parameters: none
Optional parameters: charset, which MUST be a charset
registered for use with MIME text types.
It has the same syntax as the parameter
defined for text/plain [RFC2046].
Specifies the charset of the body part.
If not given, the charset defaults to
US-ASCII [ASCII].
Encoding considerations: 7bit or 8bit encoding MUST be used to
maintain compatibility.
Security considerations: This media type provides only a means
for conveying a list of newsgroups and
does not provide any information
indicating whether the sender is
authorized to state which newsgroups
should exist within a hierarchy. Such
authorization must be accomplished by
other means.
Interoperability considerations:
Disposition should by default be inline.
Applications that use this media type:
Control message issuers, relaying
agents, serving agents.
Published specification: RFC 5537
Intended usage: COMMON
Change controller: IETFthe body is defined as:
checkgroups-body = *( valid-group CRLF )
valid-group = newsgroups-line ; see Section 4.2the same charset, <newsgroup-name>, and <newsgroup-description> rules from application/news-groupinfo apply here too 🫶
now the important bit 🔥 one application/news-checkgroups message can cover one or more hierarchies, and it's treated as complete for any hierarchy it has a <valid-group> for, unless some outside info limits its scope (like a <chkscope> parameter to a checkgroups control message, see Section 5.2.3). so this body:
example.moderated A moderated newsgroup (Moderated)
example.test An unmoderated test groupbasically says the example.* hierarchy has exactly two newsgroups, example.moderated and example.test, and no others 👑 which means this media type MUST NOT be used to share partial info about a hierarchy. if a group from a hierarchy shows up, every group that exists in that hierarchy MUST be listed, unless outside info limits its scope. in that limited case, all groups SHOULD be listed instead 🌈
spaces are only there in that example for formatting reasons 🙈 in a real message, the newsgroup name and description MUST be separated by one or more tabs (HTAB, ASCII %d09), not spaces. tabs only, periodt 💅
5. Control Messages 💌
a control message is an article with a Control header field, and that tells some agent to take an action other than distribution and display ✨ any article with a Control header field (defined in Section 3.2.3 of [RFC5536]) counts as a control message. there's also a Supersedes header field, and an article with one isn't a control message, but it does describe an action that's similar to the cancel control message 🎀
the <control-command> in a Control header field has a <verb> (the action to take) and one or more <argument> values (the details). some control messages also care about the article body 👀 each recognized <verb>, which is the control message type, gets its own section below 🧚
agents MAY accept control message types that aren't listed below, but they MUST either ignore or reject control messages with unrecognized types 🍓 the valid <argument> values and any body restrictions are spelled out in the section for each control message type.
⚠️ heads up, this one's a classic. [RFC1036] used to say that aSubjectheader field starting with "cmsg " made an article a control message, but that's changed. the presence of such aSubjectheader field MUST NOT cause an article to be treated as a control message. agents MAY reject an article that has that kind ofSubjectheader field and noControlheader field, because it's ambiguous 🙈
and same energy for the other tricks: the presence of a <newsgroup-name> ending in ".ctl" in the Newsgroups header field, or an Also-Control header field, MUST NOT cause the article to be interpreted as a control message 🫶 these are all decoys, not vibes 😭
5.1. Authentication and Authorization 🔐
okay so control messages are the ones that tell a server to do extra stuff on top of the usual article processing 👀 which makes them a super juicy target for abuse and unauthorized action 💀. And rn there's no standardized way to check who actually sent a control message, or to confirm that its contents came from the sender it claims 😬. Some unstandardized authentication mechanisms are in common use though, like [PGPVERIFY] 🔑
Agents acting on control messages SHOULD take steps to authenticate control messages before acting on them, as determined by local authorization policy. Whether that happens through an unstandardized authentication protocol, by comparing against info from another protocol, by a human reviewing it, or some other way, is left unspecified by this document. Future extensions or revisions are expected to standardize a digital signature mechanism 🖋️✨
Agents are expected to have their own local authorization policies deciding which control messages will be honored. No Netnews agent is ever required to act on any control message. The descriptions below spell out what a control message asks for, but an agent MAY always decline to act on any given control message 🙅♀️
5.2. Group Control Messages 📚
A group control message is any control message type that requests some update to the list of newsgroups known to a news server. The standard group control message types are "newgroup", "rmgroup", and "checkgroups" 🌸
Before honoring any group control message, an agent MUST check the newsgroup or newsgroups affected by that control message and decline to create any newsgroups that don't conform to the restrictions in Section 3.1.4 of [RFC5536]. Bestie, this is a real must-do 💅
All of the group control messages MUST have an Approved header field (Section 3.2.1 of [RFC5536]). Group control messages without an Approved header field SHOULD NOT be honored. No approval, no vibes 🙅♀️
Group control messages that affect specific groups (newgroup and rmgroup control messages, for example) SHOULD include the <newsgroup-name> for the group or groups affected in their Newsgroups header field. Other newsgroups MAY be included in the Newsgroups header field so the control message reaches more news servers. But because of the special relaying rules for group control messages (see Section 3.6), this is normally unnecessary and may be excessive 🤏
5.2.1. The newgroup Control Message 🆕
The newgroup control message requests that the specified group be created or, if it already exists, that its moderation status or description be changed. The syntax of its Control header field is:
control-command =/ Newgroup-command
Newgroup-command = "newgroup" Newgroup-arguments
Newgroup-arguments = 1*WSP newsgroup-name
[ 1*WSP newgroup-flag ]
newgroup-flag = "moderated"If the request is honored, the moderation status of the group SHOULD be set according to whether the <newgroup-flag> "moderated" is present or absent. "moderated" is the only flag defined by this protocol. Other flags MAY be defined by extensions to this protocol and accepted by agents. If an agent doesn't recognize the <newgroup-flag> of a newgroup control message, it SHOULD ignore that control message 🧐
The body of a newgroup message SHOULD contain an entity of type application/news-groupinfo specifying the description of the newsgroup, either as the entire body or as an entity within a multipart/mixed object [RFC2046]. If that entity is present, the moderation status specified in it MUST match the moderation status specified by the <newgroup-flag>. The body of a newgroup message MAY contain other entities (encapsulated in multipart/mixed) that give extra info about the newsgroup or the circumstances of the control message 📎
Without an application/news-groupinfo entity, a news server MAY search the body of the message for the line For your newsgroups file: and take the following line as a <newsgroups-line>. Before application/news-groupinfo was standardized, this was the convention for providing a newsgroup description. Old-school but iconic 🕰️
If the request is honored and contains a newsgroup description, and the news server honoring it stores newsgroup descriptions, the stored description SHOULD be updated to the one in the control message, even if no other property of the group has changed ✨
5.2.1.1. newgroup Control Message Example 🧾
A newgroup control message requesting creation of the moderated newsgroup example.admin.info, bestie, looks like this:
From: "example.* Administrator" <admin@noc.example>
Newsgroups: example.admin.info
Date: 27 Feb 2002 12:50:22 +0200
Subject: cmsg newgroup example.admin.info moderated
Approved: admin@noc.example
Control: newgroup example.admin.info moderated
Message-ID: <ng-example.admin.info-20020227@noc.example>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="nxtprt"
Content-Transfer-Encoding: 8bit
This is a MIME control message.
--nxtprt
Content-Type: application/news-groupinfo; charset=us-ascii
For your newsgroups file:
example.admin.info About the example.* groups (Moderated)
--nxtprt
Content-Type: text/plain; charset=us-ascii
A moderated newsgroup for announcements about new newsgroups in
the example.* hierarchy.
--nxtprt--Spaces are used in this example for formatting reasons. In an actual message, the newsgroup name and description MUST be separated by one or more tabs (HTAB, ASCII %d09), not spaces. Tabs only, no spaces, periodt 🍓
5.2.2. The rmgroup Control Message 🗑️
The rmgroup control message requests that the specified group be removed from a news server's list of valid groups. The syntax of its Control header field is:
control-command =/ Rmgroup-command
Rmgroup-command = "rmgroup" Rmgroup-arguments
Rmgroup-arguments = 1*WSP newsgroup-nameThe body of the control message MAY contain anything, usually an explanatory text. Think of it as the little note you leave when you ghost a group 👻
5.2.3. The checkgroups Control Message 📋
The checkgroups control message contains a list of all the valid groups in a hierarchy, with descriptions and moderation status. It requests that a news server update its valid newsgroup list for that hierarchy to include the groups specified, remove any groups not specified, and update group descriptions and moderation status to match those in the checkgroups control message. Full glow-up for the whole hierarchy 🌈 The syntax of its Control header field is:
control-command =/ Checkgroup-command
Checkgroup-command = "checkgroups" Checkgroup-arguments
Checkgroup-arguments= [ chkscope ] [ chksernr ]
chkscope = 1*( 1*WSP ["!"] newsgroup-name )
chksernr = 1*WSP "#" 1*DIGITA checkgroups message is read as an exhaustive list of the valid groups in all hierarchies or sub-hierarchies with a prefix listed in the <chkscope> argument, except for any sub-hierarchy where the <chkscope> argument is prefixed by "!". For complex cases with multiple <chkscope> arguments, start from an empty list of groups, include all groups in the checkgroups control message that match <chkscope> arguments without a "!" prefix, and then exclude all groups matching <chkscope> arguments with a "!" prefix. Follow this method no matter what order the <chkscope> arguments show up in the Control header field 🧮
If no <chkscope> argument is given, it applies to all hierarchies for which group statements appear in the body of the message.
Since a lot of existing software doesn't honor the <chkscope> argument, the body of the checkgroups control message MUST NOT contain group statements for newsgroups outside the intended scope, and SHOULD contain a correct newsgroup list even for sub-hierarchies excluded with "!" <chkscope> terms. News servers, however, MUST honor <chkscope> as specified here. Not everyone reads the manual, so the servers have to be the responsible ones 🫶
The <chksernr> argument may be any positive integer. If present, it MUST increase with every change to the newsgroup list, MUST NOT ever decrease, and MUST be included in all subsequent checkgroups control messages with the same scope. If provided, news servers SHOULD remember the <chksernr> value of the previous checkgroups control message honored for a particular hierarchy or sub-hierarchy, and decline to honor any later checkgroups control message for that same hierarchy or sub-hierarchy with a smaller <chksernr> value or with no <chksernr> value. Serial numbers only go up, iykyk 📈
There is no upper limit on the length of <chksernr>, other than the limit on the length of header fields. Implementations may therefore want to compare values by zero-padding the shorter of two <chksernr> values on the left and then doing a string comparison, rather than assuming <chksernr> can be manipulated as a number. Big numbers are not your friend here 🔢💀
For example, the following Control header field
Control: checkgroups de !de.alt #2009021301indicates that the body of the message will list every newsgroup in the de.* hierarchy, except the de.alt.* sub-hierarchy, and should not be honored if a checkgroups control message with a serial number greater than 2009021301 was previously honored. The serial number in this example was formed from the date in YYYYMMDD format, followed by a two-digit sequence number within that date. So cute, the date IS the serial number 📅🦋
The body of the message is an entity of type application/news-checkgroups. It SHOULD be declared as such with appropriate MIME headers, but news servers SHOULD interpret checkgroups messages that lack the appropriate MIME headers as if the body were of type application/news-checkgroups, for backward compatibility. Old messages still get a seat at the table 🪑💜
5.3. The cancel Control Message 🚫
The cancel control message requests that a target article be withdrawn from circulation and access. The syntax of its Control header field is:
control-command =/ Cancel-command
Cancel-command = "cancel" Cancel-arguments
Cancel-arguments = 1*WSP msg-idThe argument identifies the article to be cancelled by its message identifier. The body of the control message MAY contain anything, usually an explanatory text 📝
A serving agent that elects to honor a cancel message SHOULD make the article unavailable to reading agents (perhaps by deleting it completely). If the cancel control message arrives before the article it targets, news servers that choose to honor it SHOULD remember the message identifier that was cancelled and reject the cancelled article when it arrives. Time-travel-proof cancels, lowkey iconic ⏳✨
To best ensure that it gets relayed to the same news servers as the original message, a cancel control message SHOULD have the same Newsgroups header field as the message it is cancelling. Matching the routing is the whole move 🛣️
Cancel control messages that list moderated newsgroups in their Newsgroups header field MUST contain an Approved header field, like any other article in a moderated newsgroup. This means cancels posted to a moderated newsgroup will normally be sent to the moderator first for approval. Outside of moderated newsgroups, cancel messages aren't required to contain an Approved header field 👑
Contrary to [RFC1036], cancel control messages are not required to contain From and Sender header fields matching the target message. That requirement only encouraged cancel issuers to hide their identity, and it provided no security 🙈 bestie, a hidden identity is not a security feature 💀
5.4. The Supersedes Header Field ✨
okay bestie, so a Supersedes header field in an article is basically saying "please withdraw the message with this ID, the exact same way you'd withdraw a message that's the target of a cancel control message" 💅 It's giving cancel control message energy, just in a different outfit. So news servers SHOULD apply the same authentication and authorization checks to a Supersedes header field that they'd apply to cancel control messages 🔒 And if the Supersedes header field gets honored, the news server SHOULD take the same actions it would take when honoring a cancel control message for that target article 🫶
the article holding the Supersedes header field is a normal article either way, honored or not 🌸 It SHOULD be handled as a normal article and SHOULD NOT get the special control message treatment described in Section 3.7. no special VIP lane for it, periodt 💖
5.5. The ihave and sendme Control Messages 🦋
the ihave and sendme control messages are the OG ancestors of NNTP [RFC3977] 🧚 They're mostly obsolete on the Internet now, but they still get used with some transport protocols like UUCP [UUCP]. News servers are not required to support them, so no pressure 🫶
ihave and sendme share the same vibe when it comes to their Control header field and bodies, so here's the shared syntax:
control-command =/ Ihave-command
Ihave-command = "ihave" Ihave-arguments
Ihave-arguments = 1*WSP *( msg-id 1*WSP ) relayer-name
control-command =/ Sendme-command
Sendme-command = "sendme" Sendme-arguments
Sendme-arguments = Ihave-arguments
relayer-name = path-identity ; see 3.1.5 of [RFC5536]
ihave-body = *( msg-id CRLF )
sendme-body = ihave-bodythe article body is a list of <msg-id>s, one per line 📝 The message identifiers SHOULD go in the body, not in the Control header field. But news servers MAY recognize and process message identifiers in the Control header field for backward compatibility 🕰️ Just don't double-dip: message identifiers MUST NOT be put in the Control header field if they're already in the body of the control message 🙅♀️
so what do these messages actually do? An ihave says "hey, the named relaying agent got articles with these message IDs, which might be of interest to the relaying agents receiving it" 👀 A sendme says "please send me the articles with these message IDs, to the named relaying agent" 📬 Contrary to [RFC1036], the relayer-name MUST be the last argument in the Control header field, so keep it at the end, bestie 💅
once a sendme comes in (and the receiver decides to honor it), the receiving agent sends the requested article or articles 🚀 How that transmission happens isn't specified by this document. It's arranged between the sites involved, so y'all just figure it out 🤝
these control messages are normally sent as point-to-point articles between two sites, and they don't get passed along to other sites 🔒 Newsgroups beginning with "to." are reserved for these private chats. You make one by putting to. in front of a <relayer-name> to form a <newsgroup-name> 🌈 Articles with such a group in their Newsgroups header fields SHOULD NOT be sent to any news server other than the one identified by <relayer-name> 🫣
5.6. Obsolete Control Messages 🥀
these control message types are declared obsolete by this document, so they SHOULD NOT be sent or honored. Bye bye, no cap 👋
sendsys
version
whogets
senduuname6. Security Considerations 🔥
okay, real talk, this part is SERIOUS 😭 Netnews is built for broad dissemination of public messages, and it offers very little in the way of security 💀 Whatever protection it has against abuse and impersonation mostly comes from the underlying transport layer. In big Netnews networks where news servers can't be trusted to enforce authentication and authorization at the transport layer, articles may be trivially forged and widely read, and you can't really be sure who sent an article or that it stays private 🙈
see Section 5 of [RFC5536] for more security stuff about the format of articles 📖
6.1. Compromise of System Integrity 🚨
control messages are a whole thing, because acting on unauthorized ones can get newsgroups removed, articles deleted, and random unwanted newsgroups created 😱 Administrators of news servers SHOULD therefore verify that control messages are authentic, as discussed in Section 5.1 🔍 Articles with Supersedes header fields are effectively cancel control messages, so they SHOULD be subject to the same checks discussed in Section 5.4 🫶 Right now lots of sites are ignoring all cancel control messages and Supersedes header fields, because they're hard to authenticate and get abused like crazy 🥲
cancel control messages don't have to carry the same Newsgroups header field as the messages they're cancelling 🙃 Since they sometimes show up before the original message does, it may not be possible to check that the Newsgroups header fields match. That opens the door for a sneaky poster to inject a cancel control message for an article in a moderated newsgroup without adding an Approved header field to it 😤 They can also hide malicious cancel control messages from some reading agents by using a different Newsgroups header field, so the cancel control message isn't accepted by every news server that accepted the original message 🕵️♀️
all agents should be aware that all article content, especially the content of the Control header field, is potentially untrustworthy and malicious 🚩 Articles may be built in syntactically invalid ways to try to overflow internal buffers, violate hidden assumptions, or exploit implementation weaknesses. For example, some news server implementations have been successfully attacked through Unix shell code in the Control header field 💀 All article contents, and particularly control message contents, SHOULD be handled with care and rigorously verified before any action is taken based on what the article says 🛡️
a malicious poster may add an Approved header field to skip the moderation process of a moderated newsgroup 🙅♀️ Injecting agents SHOULD verify that messages approved for a moderated newsgroup are actually being injected by the moderator, using authentication information from the underlying transport or some other mechanism arranged with the moderator 🔐 There's currently no standardized way to authenticate approval of messages to moderated groups, although some unstandardized methods such as [PGPMOOSE] are in common use 🥴
a malicious news server in a Netnews network may bypass checks done by injecting agents, forge Path header fields and other trace info (like Injection-Info header fields), and otherwise mess with the authorization requirements of the network 😈 News servers SHOULD use the facilities of the underlying transport to authenticate their peers, and reject articles from injecting and relaying agents that don't follow the requirements of this protocol or the Netnews network 🚫
6.2. Denial of Service 🙅♀️
the proper functioning of individual newsgroups can get wrecked by excessive posting of unwanted articles 😩 It can also get wrecked by repeated posting of identical or nearly identical articles, by followups that are unrelated to their precursors or that quote their precursors in full with just a tiny bit of extra stuff added (especially if that gets iterated), by crossposting to or requesting followups in totally unrelated newsgroups, and by abusing control messages and the Supersedes header field to delete articles or newsgroups 🗑️
articles meant to deny service, or other inflammatory articles, may also have their From or Reply-To addresses set to valid but incorrect email addresses. That can flood the true owners of those addresses with huge volumes of email 📧 Users and agents should always be aware that identifying information in articles may be forged 🫠
a malicious poster can stop an article from being seen at a particular site by including that site's <path-identity> in the Path header field of the proto-article 🙅 Using the <diag-keyword> POSTED on injecting agents to mark the point of injection can prevent this attack 🛑
the main responsibility for stopping these attacks sits with injecting agents, which can apply authentication and authorization checks via the underlying transport and stop denial-of-service folks from posting at all 🧱 If specific injecting agents don't live up to their responsibilities, they may be kicked out of the Netnews network by configuring relaying agents to reject articles coming from them 👋
a malicious complainer may send a modified copy of an article (say, one with an altered Injection-Info header field) to the administrator of an injecting agent 😒 The goal is to make the author look bad and maybe even get their posting privileges removed. Administrators SHOULD therefore get a genuine copy of the article from their own serving agent before acting on such a complaint 🔎
6.3. Leakage 🫣
articles meant to have restricted distribution depend entirely on the goodwill of every site that receives them 🥺 You can request limits on dissemination and retention via the Archive and Distribution header fields, but the protocol can't enforce those requests 🙃
the flooding algorithm used by Netnews transports like NNTP [RFC3977] is extremely good at finding any path out of a subnet with supposedly restrictive boundaries 🌊 Avoiding that takes a lot of administrative effort. Organizations that want to control leakage are advised to designate a small number of gateways to handle all news exchanges with the outside world 🚪
the sendme control message (Section 5.5), as long as it's still used, can be used to request articles the requester shouldn't have access to 👀
7. IANA Considerations 🌈
IANA has registered the following media types, described elsewhere in this document, for use with the Content-Type header field, in the IETF tree according to the procedures in [RFC4288] 🎀
application/news-transmission (4.1)
application/news-groupinfo (4.2)
application/news-checkgroups (4.3)application/news-transmission is a change to a previous registration ✨
IANA has registered the new header field, Original-Sender, in the Permanent Message Header Field Repository, using the template in Section 3.10.3 💖
IANA has changed the status of the message/news media type to "OBSOLETE". message/rfc822 should be used instead 🥀 An updated template is included in Section 4 🦋
8. References 📚
8.1. Normative References ✨
okay so these are the must-read docs 📚 if you want the full picture, you need these first, no cap 💅
[ASCII]American National Standard for Information Systems, "Coded Character Sets - 7-Bit American National Standard Code for Information Interchange (7-Bit ASCII)", ANSI X3.4, 1986.[RFC2046]Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, November 1996.[RFC2119]Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.[RFC3629]Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003.[RFC4288]Freed, N. and J. Klensin, "Media Type Specifications and Registration Procedures", BCP 13, RFC 4288, December 2005.[RFC5234]Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008.[RFC5322]Resnick, P., Ed., "Internet Message Format", RFC 5322, October 2008.[RFC5536]Murchison, K., Ed., Lindsey, C., and D. Kohn, "Netnews Article Format", RFC 5536, November 2009.
8.2. Informative References 🌸
these ones are the extra reading, the "ooh tell me more" pile 🧚 they're not required for the core stuff, but they give you so much context 🫶
[PGPMOOSE]Rose, G., "PGP Moose", November 1998.[PGPVERIFY]Lawrence, D., "Signing Control Messages", August 2001,<ftp://ftp.isc.org/pub/pgpcontrol/FORMAT>.[RFC1036]Horton, M. and R. Adams, "Standard for interchange of USENET messages", RFC 1036, December 1987.[RFC2045]Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996.[RFC2606]Eastlake, D. and A. Panitz, "Reserved Top Level DNS Names", BCP 32, RFC 2606, June 1999.[RFC3798]Hansen, T. and G. Vaudreuil, "Message Disposition Notification", RFC 3798, May 2004.[RFC3977]Feather, C., "Network News Transfer Protocol (NNTP)", RFC 3977, October 2006.[SON-OF-1036]Spencer, H., "News Article Format and Transmission", Work in Progress, May 2009.[USEAGE]Lindsey, C., "Usenet Best Practice", Work in Progress, March 2005.[UUCP]O'Reilly, T. and G. Todino, "Managing UUCP and Usenet", O'Reilly & Associates Ltd., January 1992.
Appendix A. Changes to the Existing Protocols 🎀
okay this is the glow-up recap 💖 this is where the doc lists everything that's changed since the old [RFC1036] days, aka the before-and-after moment ✨
this document prescribes many changes, clarifications, and new features since the protocol described in [RFC1036]. most notably:
- a new, backward-compatible Path header field format that permits standardized embedding of additional trace and authentication information is now RECOMMENDED 🦋 see Section 3.2. folding of the Path header is permitted 🫶
- trimming of the References header field is REQUIRED 🔥 and a mechanism for doing so is defined 💅
- addition of the new Injection-Date header field is required in some circumstances for posting agents (Section 3.4.2) and injecting agents (Section 3.5), and MUST be used by news servers for date checks (Section 3.6) 😭 injecting agents are also strongly encouraged to put all local trace information in the new Injection-Info header field 🍓
- a new media type is defined for transmitting Netnews articles through other media (Section 4.1), and moderators SHOULD prepare to receive submissions in that format (Section 3.5.1) 🌈
- certain control messages (Section 5.6) are declared obsolete, and the special significance of "cmsg" at the start of a Subject header field is removed 💀
- additional media types are defined for improved structuring, specification, and automated processing of control messages (Sections 4.2 and 4.3) ⭐
- two new optional parameters are added to the checkgroups control message 🧚
- the message/news media type is declared obsolete 🙈
- cancel control messages are no longer required to have From and Sender header fields matching those of the target message, as this requirement added no real security 💜
- the relayer-name parameter in the Control header field of ihave and sendme control messages is now required 👑
in addition, many protocol steps and article verification requirements that are unmentioned or left ambiguous by [RFC1036] but are widely implemented by Netnews servers have been standardized and specified in detail ✨ that's giving "finally, it's written down" energy 🌸
Appendix B. Acknowledgements 🫶
this doc is the result of a twelve-year effort 😭 and the number of people that have contributed to its content are too numerous to mention individually. huge love to all past and present members of the USEFOR Working Group of the Internet Engineering Task Force (IETF) and the accompanying mailing list 💖
special thanks are due to Henry Spencer, whose [SON-OF-1036] draft served as the initial basis for this document 👑 iconic origin story 🦋
Authors' Addresses 💌
these are the people who made it happen 🧚 contact info is below, so you can say hi 🌈
Russ Allbery (editor)
Stanford University
P.O. Box 20066
Stanford, CA 94309
US
EMail: rra@stanford.edu
URI: http://www.eyrie.org/~eagle/
Charles H. Lindsey
5 Clerewood Avenue
Heald Green
Cheadle
Cheshire SK8 3JU
United Kingdom
Phone: +44 161 436 6131
EMail: chl@clerew.man.ac.uk📦 where this page comes from
- ietf: rfc5537 2009-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms