RFC 803
Dacom 450/500 facsimile data transcoding
RFC 803 📄
Dacom 450/500 Facsimile Data Transcoding ✨
Written by A. Agarwal, M. J. O'Connor and D. L. Mills 🧚
Dated 2 November 1981 🎀
This part of the text doesn't include a status line, so the category and any obsoletes or updates info aren't listed here 🫶
1. Introduction 🌸
okay so as part of our work for the DARPA Internet Program 💖, the team built software modules that encode and decode facsimile data for the Dacom 450 and 500 models of Computerfax facsimile machines. these modules map the data those machines use to and from bit-map and run-length representations, so you can edit, transmit and archive fax images 📠. they're written in PDP-11 MACRO-11 assembly language and can be dropped into programs for, among others, the RT-11 operating system and the DCNET BOS or VOS operating systems 💅.
the first part of this report is the Dacom 450 data compression algorithm, described in detail 🔥. it's an update and correction to an earlier memo [2] 🙈. after that we get the encoding and decoding algorithms, plus the supporting programs and file formats. [3] describes another implementation of the decoding algorithm. big thank you to E. A. Poe of Rapicom for helping out 🫶.
the second part is the Dacom 500 data compression algorithm, as used by the INTELPOST electronic-mail network 🌍. it's being developed by the US Postal Service and several foreign administrations. these machines conform to the CCITT T.4 Draft Recommendation, described in [5]. the supporting programs and file formats are described too ✨.
2. Dacom 450 Data Compression Principles 💜
the encoding algorithm for the Dacom 450 processes the lines the machine scans and produces a two-dimensional run-length code described by Weber [1]. however, that article has a bunch of errors and omissions, and many were only found after a lot of analysis and experimenting [2,3] 😭. the machine works over a coordinate space of l726 by about 2200 pels in high-resolution (detail) mode. in normal (quality) mode the vertical resolution is halved, so about 1100 lines get transmitted. in express mode about 733 lines get transmitted, and the missed lines are filled in on playback by copying previous lines 🦋.
data is encoded two rows at a time using a two-dimensional run-length code 💅. each row-pair is scanned left-to-right, and the line-pairs are processed top-to-bottom of the document. Figure 1 shows how the pels are represented 🍓.
| | |
----+----------+----------+----
... | x(1,j) | x(1,j+1) | ...
----+----------+----------+----
... | x(2,j) | x(2,j+1) | ...
----+----------+----------+----
| | |
Direction of scan ->Figure 1. Data Representation
for each j, the vector (x(1,j),x(2,j)) represents the contents of the jth column, where x(i,j) can be zero (white) or one (black) 🖤. each of the four possible vectors over these values is called a state (Dacom calls these "modes") 👑. the transitions between these states depend on the picture content of that line-pair. scanning goes one line-pair after the other, with no special end-of-line code in the data itself. to make later discussion and comparison with the published data easier, these conventions are used (note: the pels read top-bottom):
Pels Vector State
---------------------
W-W (0,0) 0
B-W (1,0) 1
W-B (0,1) 2
B-B (1,1) 3the algorithm Dacom uses to generate the transmitted data as the columns get scanned can be described as the non-deterministic finite-state automaton (nfsa) in Figure 2 🧚. conceptually, the nfsa starts at the beginning of a page in a designated state, at a point just after scanning the jth column in the jth state. then it scans the (j + 1)th column, enters that state, and emits the string of bits shown in the figure.
0100
.--------------------->----------------------------------.
| |
| .-----------------<------------------------------. |
| | 1 | |
| V | |
.--------------. .---------------. | |
| | | | | |
| | 010 | | | |
.->| 1 |-------------------->| 2 |->. | |
| | | | | | | |
0| | B-W | 101 | W-B | |1 | |
\<-| |<--------------------| |<-' | |
| | | | | |
| | .---->| | | |
\--------------' | \---------------' | |
| A | | | A | |
| | .--------->------' | | | | |
| | | 1 | | | | |
| | | | | | A V
| | | | | | | |
0111| |1 | | 1000| |1 | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | 1011 | | | | |
| | | .-------<----------' | | | |
V | | | V | | |
.--------------. | .---------------. | |
| |<--' | | | |
| | 0 | | | |
| 3 |<--------------------| 0 |-----' |
| | | | |
| B-B | | W-W | |
| |-------------------->| |<--------'
| | 0 | |
| | | |
\--------------' \---------------'
| A | A
| | | |
\----' \----'
run runFigure 2. NFSA Model of Encoding
in the states for W-W (0) and B-B (3) a special run-length encoding technique is used 🌈. each of these two states has two state variables: one is a run-length counter, and the other is the field length (in bits) of that counter. every time either state is entered, the counter starts at zero and counts up for each additional column of the same state. at the end of the run, the counter's value is sent, with high-order zeros added if needed to fill the field length. but if the counter equals 2**n - 1, where n is the field length, then a sequence of n one-bits is emitted, and the counter restarts at zero with a field length of n + 1. so if n = 3, a run length of three is sent as {010}, a run length of seven is sent as {110}, and a run length of eight is sent as two words, {111} followed by {0000} 🎀. the field-length variables are kept separately for the W-W and B-B states, and each time either state is re-entered, the previous values get used again 💖.
Field-length values are not allowed to go above seven 🙅♀️. so runs longer than l27 with n = 7 get encoded as a separate 7-bit word of one-bits for each run of l27, except the last one, which must always contain at least one zero-bit. the field length n goes down by one under these circumstances: the current run has been encoded as a single n-bit field, and either for n in the range four through seven the two high-order bits are zero, or for n equal to three the single high-order bit is zero. the field length can't drop below two bits 🫶.
the encoding algorithm starts in state 0 with both field lengths set to 7 ✨.
2.1. Dacom 450 Decoding Algorithm 🔥
for reasons of speed and simplicity, the Dacom 450 decoding algorithm should be modeled as a deterministic finite-state automaton (dfsa) 🧠. using straightforward formal procedures, you can build the dfsa of Figure 3. this machine makes one state transition for every bit, except in the W-W (0) and B-B (3) states, which must be treated specially in any case 💅. the states are labeled so that they match the states in Figure 2 for states numbered zero through three.
the decoded output symbols, which are the columns for each state, are represented by the states themselves 🦋. when entering state B-W (1) or W-B (2), a run-length counter is set to one. each trip around a loop back to the same state increases this counter. when leaving for any other state, the counter's value is the number of columns to produce. when entering W-W (0) or B-B (3), the run-length counter is set to zero and the field-length state variable n is set up. for each successive n bits of all ones, the counter goes up by 2**n - 1, and then n itself goes up by one, but never above seven. if the next n bits are not all ones, the counter goes up by the value of the n-bit field plus one. finally, if the next n bits are not all ones when entering either state, n goes down by one according to the
rule mentioned in the preceding section. 🧠✨
.-----------. .-----------.
.-----| | | |-----.
| | 9 | | 6 | |
| .-| |<--. .-->| |-. |
| | \-----------' \ / \-----------' | |
1| 0| \ / |1 |0
| | .->Error \ / Error<-. | |
| | 0| \ / |1 | |
| | .-----------. \ / .-----------. | |
| 1 | | | \ / | | | 0 |
| .---| 7 | \ | 10 |---. |
| | | | | / \ | | | | |
| | | \-----------' / \ \-----------' | | |
| | | A / \ A | | |
| | | | / \ | | | |
| | | 1| / \ |0 | | |
| | | .-----------. 0 / \ 1 .-----------. | | |
| | | | |---' \---| | | | |
| | | | 5 | | 8 | | | |
| | | | | | | | | |
| | | \-----------' \-----------' | | |
| | | A A | | |
| | | | | | | |
| | | 1| |0 | | |
| | | .-----------. .-----------. | | |
| | ->| | | |<- | |
| | | 1 | | 2 | | |
| | | B-W |<-----. .----->| W-B | | |
| | \-----------' | | \-----------' | |
| | | A | | A | | |
| | | | |0 1| | | | |
| | \-----' | | \-----' | |
| | 0 .-----------. 0 | |
| | | | | |
| | | 4 | | |
| | RUN | | RUN | |
| | .-----. \-----------' .-----. | |
| | | | A A | | | |
| | | V | | V | | |
| | .-----------. 1 | | 1 .-----------. | |
| \-->| |------' 0 \------| |<--' |
| | 3 |<--------------------| 0 | |
\---->| B-B |-------------------->| W-W |<----'
\-----------' 0 \-----------'Figure 3. DFSA Model of Encoding 🦋
2.2. Formatting Considerations 💖
Okay so, the Dacom 450 sends data in 585-bit frames 📦. Each frame is made up of a 24-bit synchronization code, a 37-bit leader, a 512-bit information area, and an l2-bit checksum. There are two kinds of frames, and you can tell them apart by their leader format: one is for setup or initialization, and the other is for the actual data 🎀. Serial binary image data gets placed in the data area of the frames that come after the setup ones.
The header of each frame is shown in Figure 4, and the fields are defined in Table 1 right after it 🌸.
+-----------+--------+-------------------+----------+
| Sync Code | Leader | Data | CRC Code |
+-----------+--------+-------------------+----------+
24 / 37 \ 512 12
.-------' \----------------------.
/ \
+-------+-------+-------+-------+-------+-------+
| Flags | Count | X Pos | Black | White | State |
+-------+-------+-------+-------+-------+-------+
| 7 \ 10 12 3 3 2
| \--------------------------.
| \
+-----+-----+------+-----+-------+-----+
| Seq | RUN | COFB | RPT | Spare | SUB |
+-----+-----+------+-----+-------+-----+
2 1 1 1 1 1Figure 4. Frame Format ✨
Table 1. Header Field Definitions
Field Width Function Setup Data
(bits) Block Block
-----------------------------------------------------
Sync Code 24 Synchronization 30474730 (octal)
Seq 2 Sequence number 00 00,01,10,11
RUN 1 Initialize-start 0 1
COFB 1 Unknown 0 0
RPT 1 Unknown 1 0
Spare 1 Unknown 0 0
SUB 1 Indicates setup frame 1 0
Count 10 Number of bits in data All 1's
field (0 - 512)
X Pos 12 Current position on All 1's
scan line (0 - 1725)
Black 3 Current black field All 1's
length (2 - 7)
White 3 Current white field All 1's
length (2 - 7)
State 2 Current state (0 - 3) All 1's
Data 512 Data (0 - 512 bits)
CRC Code 12 CRC checksum. Uses polynomial
x**12 + x**8 + x**7 + x**5 + x**3 + 1Setup frames carry some extra goodies in their data field 🎁. Those fields and what they do are listed in Table 2.
Table 2. Field Definitions for Setup Frame.
Field Width Function
--------------------------------
Start bit 1 Always zero
Speed bit 1 Set if express mode
Detail bit 1 Set if detail mode (speed and detail
bits both zero for quality mode)
14 inch 1 Set if 14-inch paper
5 inch 1 Set if 5-inch inch paper (14-inch
and 5-inch inch paper bits both zero
for 11-inch paper)
Paper present 1 Set if paper present in scanner
Spare 5 Can have any value
Multi-page 1 Set if multi-page mode
20 All 0's
480 Alternate 1's and 0'sThe tailing setup frames are basically the same as the leading ones 👯♀️. The only differences are the bits that say whether the system is running in single or multiple page mode, and whether there's paper in the scanner.
All n-bit numeric fields (except Seq) get sent by the Dacom 450 least-significant-bit (LSB) first 🔢. That covers Count, X Pos, Black, White, State, CRC, and the run length words in the data field (i.e. the ones I just named). Everything else goes left-most bit first 👈.
Now, the header stuff is a whole vibe 🥹. The header has enough info about the decoding algorithm's state to get correct decoding going again if a data frame gets lost or mangled. The decoder resets its state variables to the ones in the header every time it starts a new data frame. Honestly, one of the hardest parts of building the decoding algorithm was keeping it synchronized as it crosses the frame boundary with respect to the header info 😭. To keep things in sync, the algorithm has to behave exactly like what's described in the previous section.
This "every data frame has to be self-synchronizing" requirement leads to a few subtleties in the encoding algorithm 🌈. They feel super natural once you see them, but they weren't obvious at the start:
- Transition bits(s) labeling the arcs on the state transition diagram in Figure 2 are not broken across frames. Similarly, individual run-length words are not broken across frames.
- If a frame ends with a transition, the header of the next frame contains the state to which the transition is made.
- If a frame ends with a transition out of state 0 or 3, then the transition bit (0 or 1) is inserted at the end of the current frame (not at the beginning of the next frame).
- The field lengths for black and white runs in the header include changes that may have been caused at the end of the previous frame.
- If a frame begins with a white or black run, then this run is treated (for purpose of decreasing its field length) as if it were the beginning of a new run, since there is no information in the header to indicate otherwise.
The decoding algorithm gets initialized at the first data frame received after the sequence of setup frames at the start of transmission 🚀. That first data frame has a count of zero, meaning there are no data bits in it. The second data frame is where the actual document begins 📄. Its X position looks irrelevant, though 🤷♀️. So instead, we assume the initial X position at this time is one pel to the left of the right margin (-l mod l726). With those assumptions, the algorithm's succeeding X positions and the frame headers agree 💅.
2.3. The Decoding Program 🧚
The decoding algorithm described above has been implemented in PDP-11 MACRO-11 assembly language for the RT-11 operating system 💻. The program has a lot of features for selectively dumping frames and tracing how the algorithm runs 🔍. It's designed to operate on a file that holds the raw data the machine generated, and it doesn't need any prior reformatting of that data first. It will also work on files in the so-called UCL format [4], which has been adopted as the standard for use in the Internet Program 🌐. The existing DCNET supporting software for the Dacom 450 uses the UCL format and works normally to copy data directly between the machine and the file, with decoding done at a later time. But honestly, there's no built-in reason it couldn't decode data straight from the machine, apart from processing-rate limits 🏎️.
In operation, the program scans the input data one bit at a time and looks for the synchronization pattern 👀. Note that all data processed are inverted from the natural interface conventions 🙃. When a synchronization pattern is found, the header and data portions get extracted, and the various state variables are checked and reset, if necessary. Checksum verification is performed according to the polynomial 1 + x**3 + x**5 + x**7 + x**8 + x**12. For setup frames, the format (detail, quality, express), page length (14, 8-l/2, 5-l/4), and multiple-page indicators are pulled out of the data area 📏. Finally, under control of specified options, the header and data portions of the frame are printed with the right headings 🖨️.
The decoding algorithm itself gets called for each data frame 💫. It spits out a sequence of run-length pairs, which can be used to form bit maps and other representations of the data 🗺️. Optionally, it can also produce a printed trace of the operations the algorithm performed 📝.
2.4. The Encoding Program 🎀
The encoding algorithm has been implemented in PDP-11 MACRO-11 assembly language for the RT-11 operating system too 💜. The program takes facsimile data in 16-bit run-length format or bit-map format 📠. The input would normally come from a file, possibly made by translating some other representation (e.g., T.4 format) into run-length or bit-map format 🔁. The program outputs data compressed in Dacom 450 format and packed into 585-bit frames, along with the matching header and checksum information 🎁. The encoding program needs to be careful about how to break data
across frames and how many bits to stuff into each one. The rules from section 2.2. help with the first problem 🎀. The second problem is sneakier ngl 👀. The printer eats data bits at a constant rate, but frames zoom by at line speed and can carry totally different amounts of decoded info, so in extreme cases the buffer overruns 😭.
To handle that rate mismatch, a simple set of empirical rules turns out to be good enough, and they give results pretty similar to the scanner itself ✨. Under these rules a frame counts as "full" when it contains more than 500 bits of data, or when the data represents more than 4800*X pels (or columns) of information,
where X = 2 for transmission rate 2.4 kbs,
X = 1 for transmission rate 4.8 kbs,
X = 1/2 for transmission rate 9.6 kbs.2.5. Dacom 450 File Formats 🌸
Dacom 450 facsimile data is usually stored as an RT-11 file in the so-called UCL format [4]. In this format, each 585-bit frame lives in a 76-byte record. The first byte is the length of the record, the second is a command, and the remaining 72 bytes hold the 585 bits of the original Dacom 450 frame, zero-filled at the end. The command byte works like this:
- a. 56 (70 octal): The data field contains a setup frame (the first record of the file). The length byte is 76 (114 octal).
- b. 57 (71 octal): The data field contains a data frame (the remaining records in the file except the last one). The length byte is 76 (114 octal).
- c. 58 (72 octal): End of file (the last frame of the file). There is no data field and the length byte is 2.
2.6. Run-Length and Bit-Map File Formats 💜
The decode program outputs 16-bit run length words. Each run is encoded in its own 16-bit word, with white runs as positive values and black runs as negative values, in two's complement 🦋. A zero word ends each line, and any trailing white run is suppressed if it's there. An all-white line is encoded as a single run of length one followed by a zero word. The file wraps up with a line of length zero, which is just a single zero word.
Bit-map files start with a four-byte header, then the data. The header is two 2-byte quantities: the first is the number of pels in a line, and the second is the number of lines in the page. Each scanning line is stored in a whole number of bytes, with the last byte of a line zero-filled if necessary 🍓.
3. Dacom 500 Data Compression Principles 🌈
The Dacom 500 machines are the high-speed versions of the Dacom 450 machines. They operate in the 50-Kbps range and use the T.4 compression algorithm. That algorithm is described in [5]. It's one-dimensional, not the two-dimensional one used in the Dacom 450 and described in the earlier sections. Since it's well known and the subject of an international standard, we won't go into more detail here 🙈.
3.1. Dacom 500 Decoding Algorithm 🧠
The decoding program is written in PDP-11 MACRO-11 assembly language for the DCNET and RT-11 operating systems. It takes a file of facsimile data encoded with the T.4 algorithm and produces a file in bit-map format.
It scans the input data bit-by-bit and picks out sequences of bits that form valid run-length codes (see the tables in [5]). Picture the Huffman codes as a binary tree: run lengths like 1, 2, 64, 1728, and so on sit at the terminal nodes, and each branch is labelled 0 or 1. The code for any run length is the sequence of branch labels on the path from the root to the terminal node for that length 🌳.
Black and white run-length codes are stored as separate binary trees. The decoding algorithm starts by setting an accumulator to zero. It begins at the root of the matching tree and walks down it, consuming bits one by one from the input. When it reaches a terminal node, the value stored there is added to the accumulator. If it reaches a make-up node, that value is added to the accumulator and the search resumes in the same tree to find the terminating value. Otherwise, the accumulator now holds the current run length, and the search switches to the alternate tree.
3.2. Dacom 500 Encoding Program ✨
The encoding program is also written in PDP-11 MACRO-11 assembly language for the DCNET and RT-11 operating systems. It scans the bit-map input and encodes each run of black or white pels with a simple table lookup of the Huffman codes. It works on a file of facsimile data in bit-map format and produces a file in T.4 format. The T.4 specifications [5] require a minimum transmission time per scan line of 4.3 milliseconds. At 50-Kbps, that works out to 242 bits (DATA bits plus any required FILL bits plus the EOL bits equal 242 bits minimum) 🧚.
3.3. Dacom 500 File Formats 🎀
The file is made up of a number of 512-byte blocks, and the first one is the header. The header contains a list of two-byte entries. The first entry is the number of pages, and the remaining entries are the lengths (in blocks) of each page, in turn. The remaining blocks of the file hold the data for each page in T.4 format. The data for each page is preceded by a page-setup command and followed by a page-end-of-record command, as transmitted by the Dacom 500. Both commands are the 12-bit T.4 EOL code (000000000001) repeated six times, followed by a special 4-bit code word, also repeated six times. The special code word is made of bits B1 through B4, defined below:
B1: VERTICAL RESOLUTION
0 = 7.7 lines per millimeter
1 = future option, not implemented
B2: OUTPUT PAPER LENGTH
0 = short length (Letter size)
1 = long length (Legal size)
B3: DOCUMENT IN SCANNER
0 = no document present (end of page)
1 = document present (beginning of page)
B4: ODD PARITY OVER B1-B43.4. Comparison of Facsimile Formats and Transcoding Speeds 📊
Four different file formats are used in our system for facsimile data storage: Dacom 450, Dacom 500 (T.4), 16-bit run-length, and bit-map. The sizes of typical files (in megabits) in these formats are shown below for comparison.
File Dacom Dacom 16-bit
450 500 run-length
----------------------------------
PNGUIN 0.22 0.5 0.27
INTELP 0.62 0.77 3.3
PANDA 1.02 2.03 6.41The file called PNGUIN is a block drawing of dancing penguins and represents a "small" file. The file called INTELP is a composite INTELPOST test image with text and graphics and represents a "medium" file. Finally, the file called PANDA is a half-tone newspaper photograph of a giant panda and represents a "monster" file (this file was recorded on the Dacom 450 in quality mode and is therefore about half the size it would be in detail mode). The size of the bit-map file for all these images is 3.8 megabits.
Figure 5 shows the file sizes (in 512-byte blocks) and transcoding times (in seconds) for the INTELPOST test page. The file sizes are indicated on the boxes, while the transcoding times are indicated on the arrows. All times were obtained on the LSI-11/23 processor.
193 925
+-----------+ 95 +-----------+
| |----------->| |
| T.4 | | Bit-map |
| |<-----------| |
+-----------+ 165 +-----------+
A |
60 | |
.----------------------' |110
| |
| V
+-----------+ 89 +-----------+
| |----------->| |
|Run-length | | Dacom 450 |
| |<-----------| |
+-----------+ 153 +-----------+
413 155Figure 5. File Sizes and Transcoding Times
4. References 📚
- Weber, D.R. An adaptive run-length encoding algorithm. ICC-75, IEEE, San Francisco, California, June 1975.
- Palmer, L.C. Final Report, COMSAT Participation in the DARPA Packet Satellite Internetworking and Speech Applications Program. COMSAT Laboratories, July 1980.
- Katz, A. Decoding Facsimile Data from the Rapicom 450. DARPA Network Working Group Report RFC-798, USC/Information Sciences Institute, September 1981.
- Postel, J. Rapicom 450 Facsimile File Formats. DARPA Network Working Group Report RFC-769, USC/Information Sciences Institute, September 1980.
- Draft Recommendation T.4 - Standardization of Group 3 Facsimile for Document Transmission. CCITT Study Group XIV Contribution #25-E, December 1977. (Also in RFC-804).
📦 where this page comes from
- ietf: rfc803 1981-11
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms