RFC 468
FTP data compression
RFC 468 🎀 FTP DATA COMPRESSION
Okay bestie, here's the header tea ☕ This is RFC 468 from the Network Working Group, written by R. Braden of UCLA/CCN, with NIC number 14742, dated March 8, 1973. The title is FTP DATA COMPRESSION. The header doesn't list a category, and it doesn't say it obsoletes or updates anything, so there's nothing more to add there 💅
I. INTRODUCTION ✨
APOLOGIA
Okay so the big goals for the proposed File Transfer Protocol (FTP) are reliability and efficiency when moving big files 🚚 Efficiency has two faces: how hard the host CPUs have to work, and how well you use the network bandwidth. Block mode is meant to keep CPU overhead low for bandwidth efficiency, and a mode called "HASP" shows up in RFC 454. The HASP mode of FTP is basically transmission with data compression, which means an encoding scheme that cuts down the information redundancy in messages 🗜️
RFC 454 doesn't actually define the "HASP" or compressed mode. It just says a future RFC by yours truly will define it. FTP people might find that hard to believe, but you're reading the promised RFC right now 😭 It took way longer to arrive than any of us expected. Mea Culpa 🙈
GENERAL CONSIDERATIONS
Early on, the Network was mostly remote terminal stuff and small-to-medium file transfers like remote job entry 💻 But as facilities like the Illiac IV and the Data Machine come online, and users with heavy data needs join, big file transfers are going to be a major way people use the Network. For example, one CCN user expects to send 2 x 10*8 bits of data *each day over the Network 📈
Byte compression like the kind proposed here is especially good at shrinking "printer" files, like the ones sent under the Network RJE protocol 🖨️ CCN's RJS service has seen print files compress by a factor of between two and three. Since FTP was meant to include the data transfer part of the Network RJE protocol as a subset, it makes sense to include a print file compression mechanism in FTP too. That's why the FTP committee put a compressed mode in FTP 🫶
The two main arguments for data compression are economics and convenience (usability). Economics is basically a trade-off between CPU time and transmission costs. As long as Network use is a free commodity, the economics of data compression are all bad 💸 That happy state won't last forever. So what does data compression cost?
Let's only consider simple linear compression schemes, like the one proposed here. "Linear" means the CPU time to examine a source record is proportional to the number of bytes in the record. A simple linear scheme could detect repeated single characters, for example. You could imagine quadratic schemes that detect repeated substrings, but unless the source strings have some structure the compression algorithm knows about, the CPU economics don't favor quadratic compression 🤓
Assuming a reasonable figure for large-scale CPU costs in the generation of CCN's 360/91, we concluded that an upper bound on CPU costs for total compression and decompression would be 5 cents per megabit. That's based on very loose coding of a simple linear algorithm. Compare that with projected Network transmission costs of over 30 cents per megabit (possibly a lot over) 😳
So the CPU time to conserve bandwidth costs significantly less than the bandwidth saved. Both CPU costs and bandwidth costs are trending downward, but it seems exceedingly unlikely that the ratio of CPU cost to bandwidth cost for linear compression will reverse in the next few years. On the other hand, this calculation clearly discourages quadratic compression 🚫
WHY HASP
CCN's batch remote job entry protocol NETRJS (see RFC #189, July 15, 1971) was designed to include two data transfer modes, truncated and compressed 🧩 The NETRJS truncated mode is essentially identical to the current FTP block mode record structure, except for minor bit format differences. The compressed mode of NETRJS uses an adaptation of the compression scheme in the "Multileaving protocol" of the binary synchronous rje support in IBM's HASP system.
It isn't really necessary for defining a compression scheme in FTP, but I included an appendix summarizing very briefly what HASP and its rje package are. That appendix counts as cultural enrichment for those in the Network Community who have been denied the privilege of being an IBM customer 🎓 I know a lot of HASP experts who never heard of TENEX! More seriously, HASP is widely used on IBM machines, so the HASP compression scheme is widely used too. When designing NETRJS, we picked the HASP scheme for its ubiquity and plausibility 🌈
However, certain details of the HASP bit formats are inappropriate or sub-optimal for FTP. So our proposal for compressed mode of FTP is only an adaptation of the HASP compression scheme 🛠️
It should be clear from Appendix A that the compression scheme of HASP, even if used literally, is a very minor and incidental part of that system. Although we ought to properly credit the intellectual origin of FTP's compressed mode, it seems a little strange to call the compressed mode in FTP the "HASP mode". I trust this will be rectified by the forthcoming FTP meeting.
II. PROPOSED FTP COMPRESSED DATA MODE 🦋
Byte size is B bits. Figures above boxes are field lengths in bits.
n bytes of data
/--------/\--------\
1 B-1 / B B \
+---+------+ +--------+ +--------+
Byte String: | 0 | n | | d |. . .| d |
| | | | 1 | | n |
+---+------+ +--------+ +--------+
String of n data bytes d(1),...,d(n)
Count n must be positive 2 B-2 B
+----+------+ +---------+
Replicated Byte: | 1 0| n | | d |
+----+------+ +---------+
String consisting of n replications of the data byte d 2 B-2
+----+------+
Filler String: | 1 1| n |
+----+------+
String of n filler bytes. The filler byte is a "space"
character for ASCII or EBCDIC type, or a binary zero
byte for Image or Local Byte Type. B B
+----------+ +----------+
Control Escape Sequence: | 0......0 | | C | (see below)
+----------+ +----------+The control byte "C", which is the second byte of a control escape sequence, is to have the same coding as the descriptor byte in Block Mode. This includes end-of-file and end-of-record indications. I won't specify this further, because there's some question at present about the exact coding of the Block Mode descriptor byte 🤷♀️
Following the example of APL*, we let the meaning of the filler (blank or 0) be determined by the type: character (ASCII|EBCDIC) vs. binary (Image|Local Byte). If the byte size is equal to the word size of the transmitting host, the compressed mode allows one to send sparse notices with reasonable efficiency ✨
* Compare 1 (take) 0 1\`A' with 1 (take) 0 1\2APPENDIX A: HASP MULTILEAVING ✨
okay so HASP (Houston Automatic Spooling Program) is a subsystem that basically runs inside OS/360 as a job, but it takes over the batch processing management stuff from the operating system itself 💅 HASP handles spooling of card input and printer and punch output, queueing and scheduling of job execution, and the operator control interface. it's a super tight, efficient system for running a big, varied job load through a large-scale machine 🖥️🔥 the name comes from a historical fact: HASP was originally written by a local IBM group for one particular customer, NASA Houston 🚀
HASP has always been kind of an anomaly in the IBM world 🦋 the system was written around 1965 by two programmers, and the HASP group has probably averaged three programmers for most of its life. the leader of the group was "Mr. HASP", Tom Simpson 👑 HASP spread fast through (more or less) underground channels to a bunch of the medium and large scale 360's. at least once, only intense customer pressure stopped IBM from killing the HASP effort 😭 HASP gave its users an astonishing emotional mystique, like the HASP sessions at SHARE Meetings were giving revival meetings 🙈 for years every SHARE Meeting has had HASP song sessions around the piano during the nightly open bar 🎹 honestly, HASP is a fascinating chapter in the history of IBM's large machine business 📖
The core concepts in HASP are pseudo-devices, plus the general technique of intercepting supervisor calls to augment operating system functions without changing the operating system itself 🧚 a whole generation of OS/360 system programmers learned these techniques from HASP. (these important techniques are hardly ever described in the literature, and "practical" programmers don't read the literature anyway 💀)
When HASP starts up (in supervisor state), it overlays an instruction in the I/O Supervisor with a branch to its own code. a user program is written as if it were doing real I/O to card readers and printers. HASP intercepts and interprets these I/O operations to handle job I/O in a manner transparent to OS/360. it does the same thing for operator console I/O 💖
HASP includes batch remote job entry using binary synchronous communication. the HASP communication protocol and message formats use a scheme developed by Simpson's group called "Multileaving Protocol" 🎀 the HASP rje system, by far the best rje package IBM has produced, finally replaced two competitive IBM packages and has effectively become the IBM standard for rje 👑 CCN's RJS system not only adopted the Multileaving Protocol but basically copied its binary synchronous communication line handler directly from HASP. iconic copy behavior 💅
The Multileaving Protocol is described in the HASP manual(1) as the "fully synchronized, pseudo-simultaneous, bidirectional transmission of a variable number of data streams between two or more computers using binary synchronous communications facilities". it lets a remote batch terminal operate a variable number of card readers and printers simultaneously, at different speeds, over one communication line 🌈 it's not surprising that HASP Multileaving contains in miniature many of the features of IMP-IMP Protocol and a little host-host protocol 🦋 Specifically, Multileaving includes the following general features:
- "Conversational" transmission line protocol using transparency (DLE STX, etc.).
- "Strong" error control and retransmission using a 16-bit CRC and a modulo-16 block sequence number.
- Flow control for multiple streams in both directions. This includes the interchanging of matching control records ("RFC's") to open a stream, and a set of flow control bits in each block. Each flow control bit is logically equivalent to an ALLOcate command for one "message" (buffer) for a particular stream.
- Optional Special Control Information for remote devices. This includes printer carriage control, switching card reader hoppers, etc.
- Multiplexing ("multileaving") multiple streams into a single block for transmission.
- Marking end of file and ends of records within each stream.
- Compressing transmitted text by encoding repeated blanks and repeated single characters.
Finally, we've reached the (only) aspect of HASP relevant to FTP: its compression scheme 🫶 HASP uses the following encoding:
8
+---------+
End of Record: | 0 ... 0 |
+---------+
2 6 8 8
+---+---------+ +-------+ +--------+
Data String: |1 1| N | | d | ... | d |
| | | | 1 | | N |
+---+---------+ +-------+ +--------+
3 5
+---+--------+
N Duplicate Blanks |100| N |
+---+--------+
3 5 8
+---+---------+ +---------+
N Replicated Characters D |101| N | | D |
+---+---------+ +---------+HASP is concerned only with 8-bit bytes 🤓 however, there's a provision (which was never implemented) in the Multileaving Protocol to set the unit of the counts N as 1 byte, 2 bytes, or 4 bytes. ngl the "never implemented" part is so relatable 😭
Reference:
(1) HASP II System Manual, IBM Corporation (February 26, 1971)
[ This RFC was put into machine readable form for entry ]
[ into the online RFC archives by Via Genie 4/00]📦 where this page comes from
- ietf: rfc468 1973-03
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms