RFC 3072

Structured Data Exchange Format (SDXF)

📅 published March 2001, from ietf

Structured Data Exchange Format (SDXF) 🎀

Okay bestie, the title block is giving: this is RFC 3072 📜 from the Network Working Group, written by M. Wildgrube, dated March 2001. It's Informational 🧚, so it's info for the Internet community and not a standard. It doesn't say it obsoletes or updates anything. Low-drama, high-vibes 💅

Status of this Memo 📌

Okay so this memo just provides information for the Internet community 🦋 It does not specify an Internet standard of any kind, and distribution of it is unlimited 💖

Copyright (C) The Internet Society (2001). All Rights Reserved. 🌸

IESG Note 🚨

Okay this part is important, bestie 😭 This doc specifies a data exchange format and, partially, an API for creating and parsing one. The IESG points out that the same problem space can be handled with formats the IETF normally uses, like ASN.1 and XML, so readers are strongly encouraged to carefully read section 13 before choosing SDXF over ASN.1 or XML 🙈 Also, when you're storing text in SDXF, you're encouraged to use the UTF-8 datatype specified in section 2.5 ✨

Abstract 🌈

This spec describes an all-purpose interchange format that can be used as a file format or for networking. Data is organized into chunks, and those chunks can be ordered in hierarchical structures. The format is self-describing and CPU-independent 👑

Table of Contents 📚

1.  Introduction ................................................. 2
2.  Description of the SDXF data format .......................... 3
3.  Introduction to the SDXF functions ........................... 5
3.1 General remarks .............................................. 5
3.2 Writing a SDXF buffer ........................................ 5
3.3 Reading a SDXF buffer ........................................ 6
3.4 Example ...................................................... 6
4.  Platform independence ........................................ 8
5.  Compression .................................................. 9
6.  Encryption ...................................................11
7.  Arrays........................................................11
8.  Description of the SDXF functions ............................12
8.1 Introduction .................................................12
8.2 Basic definitions ............................................13
8.3 Definitions for C++ ..........................................15
8.4 Common Definitions ...........................................16
8.5 Special functions ............................................17
9.  'Support' of UTF-8 ...........................................19
10.  Security Considerations .....................................19
11.  Some general hints ..........................................20
12.  IANA Considerations .........................................20
13.  Discussion ..................................................21
13.1 SDXF vs. ASN.1 ..............................................21
13.2 SDXF vs. XML ................................................22
14.  Author's Address ............................................24
15.  Acknowledgements ............................................24
16.  References ..................................................24
17.  Full Copyright Statement ....................................26

1. Introduction ✨

The whole point of the Structured Data eXchange Format (SDXF) 💅 is to let you swap an arbitrary structured data block that mixes different kinds of data: numbers, text, and bitstrings. Because the data gets normalized into an abstract, computer-architecture-independent "network format", SDXF works as a network interchange data format 🌐

It isn't tied to one app either. The goal is for it to work as a text format for word processing, a picture format, a sound format, for remote procedure calls with complex parameters, for document formats, for swapping business data, and so on 🎨

SDXF is self-describing, so every program can unpack every SDXF data block without knowing what the individual data elements mean 🧠

There's also a set of functions that come with the data format. With them, you can create and access the data elements of SDXF. The idea is that a programmer should only use these functions, instead of managing the structure by hand at the level of bits and bytes. (In object-oriented speak, these functions are methods of an object that acts as a handle for a given SDXF data block.) 🧚

SDXF isn't limited to a specific platform. As long as the SDXF functions are prepared correctly, SDXF data can be exchanged over a network or a data carrier, across the boundaries of different architectures. Those differences include the character code (like ASCII, ANSI, or EBCDIC) and the byte order for binary data 🍓

And SDXF is also set up to compress and encrypt parts or the whole block of SDXF data 🔐

2. Description of SDXF data format. 💖

2.1 Chunk 🧩

First, let's introduce the term "chunk". A chunk is a data structure with a fixed set of components. A chunk may be "elementary" or "structured". The latter one contains one or more other chunks inside it 🪆

A chunk consists of a header and the data body (content):

+----------+-----+-------+-----------------------------------+
| Name     | Pos.| Length| Description                       |
+----------+-----+-------+-----------------------------------+
| chunk-ID |  1  |   2   | ID of the chunk (unsigned short)  |
| flags    |  3  |   1   | type and properties of this chunk |
| length   |  4  |   3   | length  of the following data     |
| content  |  7  |   *)  | net data or a list of of chunks   |
+----------+-----+-------+-----------------------------------+

(* as stated in "length". total length of chunk is length+6.  The
chunk ID is a non-zero positive number.

Or, more visually:

+----+----+----+----+----+----+----+----+----+-...
| chunkID | fl | length       |  content
+----+----+----+----+----+----+----+----+----+-...

Or in ASN.1 syntax:

chunk  ::=  SEQUENCE
{
  chunkID INTEGER (1..65535),
  flags   BIT STRING,
  length  OCTET STRING SIZE 3, -- or: INTEGER (0..16777215)
  content OCTET STRING
}

2.2 Structured chunk 🪆

A structured chunk gets marked as one by the flag byte (see 2.5). Unlike an elementary chunk, its content is a list of chunks (elementary or structured):

+----+-+---+-------+-------+-------+-----+-------+
| id |f|len| chunk | chunk | chunk | ... | chunk |
+----+-+---+-------+-------+-------+-----+-------+

With this concept, you can turn any hierarchically structured data into an SDXF chunk 🌳

2.3 Internal representation of the chunk's elements 🔢

Binary values are always in high-order-first (big endian) format, like the binary values in the IP header (network format). A length of 300 (=256 + 32 + 12) is stored as

+----+----+----+----+----+----+----+----+----+--
|         |    | 00   01   2C |  content
+----+----+----+----+----+----+----+----+----+--

in hexadecimal notation.

The chunk-ID works the same way ✨

2.4 Character values 🔤

Character values in the content portion are also an object of adaptation. See chapter 4 for that 🌸

2.5 Meaning of the flag-bits 🚩

Let's represent the flag byte like this:

  +-+-+-+-+-+-+-+-+
  |0|1|2|3|4|5|6|7|
  +-+-+-+-+-+-+-+-+
   | | | | | | | |
   | | | | | | | +-- reserved
   | | | | | | +---- array
   | | | | | +------ short chunk
   | | | | +-------- encrypted chunk
   | | | +---------- compressed chunk
   | | |
   +-+-+------------ data type (0..7)

The data types are:

0 -- pending structure (chunk is inconsistent, see also 11.1)
1 -- structure
2 -- bit string
3 -- numeric
4 -- character
5 -- float (ANSI/IEEE 754-1985)
6 -- UTF-8
7 -- reserved

2.6 Short chunk 🦋

A short chunk has no data body. The 3-byte Length field gets used as data bytes instead. This saves space when there are lots of small chunks 🧃

2.7 Compressed and encrypted chunks 🗜️🔒

Compressed and encrypted chunks are explained in chapters 5 and 6 💜

2.8 Arrays 📦

Arrays are explained in chapter 7 🎀

2.9 UTF-8 handling 🌍

Handling of UTF-8 is explained in chapter 9 🫶

2.10 Combinations of bits ⚠️

Not all combinations of bits are allowed or reasonable:

-  the flags 'array' and 'short' are mutually exclusive.
-  'short' is not applicable for data type 'structure' and 'float'.
-  'array' is not applicable for data type 'structure'.

3. Introduction to the SDXF functions 🧚

3.1 General remarks 💫

The SDXF concept isn't bound to any programming language. Of course, the functions themselves have to be coded in a particular language, and this document discusses them in C and C++, since by now those languages are available on almost all platforms 💻

All the functions for reading and writing SDXF chunks take just one parameter: a parameter structure. In C++, this parameter structure is part of the "SDXF class", and the SDXF functions are methods of that class 🧁

An exact description of the interface is given in chapter 8 📖

3.2 Writing a SDXF buffer ✍️

To write SDXF chunks, you've got these functions:

init    -- initialize the parameter structure
create  -- create a new chunk
leave   -- "close" a structured chunk

3.3 Reading a SDXF buffer 👀

To read SDXF chunks, you've got these functions:

init    -- initialize the parameter structure
enter   -- "go into" a structured chunk
next    -- "go to" the next chunk inside a structured chunk
extract -- extract the content of an elementary chunk into
           user's data area
leave   -- "go out" off a structured chunk

3.4 Example 🌟

3.4.1 Writing ✍️

For demonstration, we use a reduced (outlined) C++ form of these functions with polymorphic definitions:

void create (short chunkID); // opens a new structure,
void create (short chunkID, char *string);
     // creates a new chunk with dataType character, etc.)

The sequence:

SDXF x(new); // create the SDXF object "x" for a new chunk
             // includes the "init"
x.create (3301);   // opens a new structure
x.create (3302, "first chunk");
x.create (3303, "second chunk");
x.create (3304);   // opens a new structure
x.create (3305, "chunk in a structure");
x.create (3306, "next chunk in a structure");
x.leave ();        // closes the inner structure
x.create (3307, "third chunk");
x.leave ();        // closes the outer structure

This creates a chunk that we can show graphically like this:

3301
 |
 +--- 3302 = "first chunk"
 |
 +--- 3303 = "second chunk"
 |
 +--- 3304
 |      |
 |      +--- 3305 = "chunk in a structure"
 |      |
 |      +--- 3306 = "next chunk in a structure"
 |
 +--- 3307 = "last chunk"

3.4.2 Reading ✨

okay so the typical way to get data out of a structured SDXF chunk is a selection inside a loop 🔁 here's the vibe:

SDXF x(old); // defines a SDXF object "x" for an old chunk
x.enter ();  // enters the structure

while (x.rc == 0) // 0 == ok, rc will set by the SDXF functions
{
  switch (x.chunkID)
  {
    case 3302:
      x.extract (data1, maxLength1);
                // extr. 1st chunk into data1
      break;

    case 3303:
      x.extract (data2, maxLength2);
                // extr. 2nd chunk into data2
      break;

    case 3304:  // we know this is a structure
      x.enter (); // enters the inner structure

      while (x.rc == 0) // inner loop
      {
        switch (x.chunkID)
        {
          case 3305:
            x.extract (data3, maxLength3);
                      // extr. the chunk inside struct.

            break;
          case 3306:
            x.extract (data4, maxLength4);
                      // extr. 2nd chunk inside struct.
            break;
        }
        x.next (); // returns x.rc == 1 at end of structure
      } // end-while
      break;

    case 3307:
      x.extract (data5, maxLength5);
                // extract last chunk into data
      break;
    // default: none - ignore unknown chunks !!!

  } // end-switch
  x.next (); // returns x.rc = 1 at end of structure
} // end-while

4. Platform independence 🌍

the vast majority of computer platforms today have an 8-bits-in-a-byte architecture, and that's what makes data exchange between them possible 🫶 but platforms can still be different in two big ways:

  • (a) The representation of binary numerical values (the short and long int and floats).
  • (b) The representation of characters (ASCII/ANSI vs. EBCDIC)

point (a) is the whole "byte swapping" drama 💀 like, how is a short int value 259 = 0x0103 = X'0103' stored at address 4402? there are two flavours:

4402 4403
01   03    the big-endian, and
03   01    the little-endian.

point (b) is a table that assigns the 256 possible values of a Byte to printable or control characters. (in ASCII the letter "A" sits at value, or position, 0x41 = 65, while in EBCDIC it's 0xC1 = 193 🙈)

the fix is to normalize the data, so we lock in:

  • (a) The internal representation of binary numerals are 2-complements in big-endian order.
  • (b) The internal representation of characters is ISO 8859-1 (also known as Latin 1).

fixing point (b) should be seen as a first strike 🔥 in some environments 8859-1 isn't the best pick. in a greek or russian environment, 8859-5 or 8859-7 are appropriate.

still, within one specific group (or world) of applications, meaning every application that wants to swap data using a defined protocol (over the network, a diskette, or something else), this internal character table has to be unique 💅

so there should be a way to define a translation table (and its inversion) 🎀

important: you don't build a SDXF chunk for one specific addressee. you adapt your data into a normalized format (aka network format) ✨ and this adaption isn't done by the programmer. the create and extract functions handle it, and an administrator has to take care of defining the correct translation tables 🧚

5. Compression 🗜️

as we said in 2.5, there's a flag bit that declares the following data (elementary or structured) as compressed. that data can't be interpreted until it's decompressed. compression happens transparently through the SDXF functions: "create" does the compression for elementary chunks, "leave" for structured chunks, "extract" does the decompression for elementary chunks, and "enter" for structured chunks.

transparently means the programmer only has to tell the SDXF functions that they want the following chunk(s) compressed 💖

for choosing between different compression methods and for controlling the decompressed (original) length, there's one more definition. after the chunk header for a compressed chunk, a compression header follows:

+-----------------------+---------------+---------------->
|      chunk header     | compr. header | compressed data
+---+---+---+---+---+---+---+---+---+---+---------------->
|chunkID|flg|   length  |md | orglength |
+---+---+---+---+---+---+---+---+---+---+---------------->
  • orglength is the original (decompressed) length of the data.
  • md is the "compression method". two methods are described here:
    • method 01 for a simple (fast but not very effective) "Run Length 1" or "Byte Run 1" algorithm. (more than two consecutive identical characters get replaced by the number of those characters and the character itself.)

more precisely: the compressed data consists of several sections of various lengths. every section starts with a "counter" byte, a signed "tiny" (8 bit) integer, which holds length information.

if this byte contains the value "n", with n >= 0 (and n <128), the next n+1 bytes are taken unchanged; with n < 0 (and n > -128), the next byte is replicated -n+1 times; n = -128 is ignored.

appending blanks gets cut in general. if they're necessary, they can be reconstructed while "extract"ing, using the parameter field "filler" (see 8.2.1) set to the space character.

  • method 02 for the wonderful "deflate" algorithm that comes from the "zip"-people 🤩 the authors are Jean-loup Gailly (deflate routine), Mark Adler (inflate routine), and others.

the deflate format is described in [DEFLATE].

the values for the compression method number are maintained by IANA, see chap. 12.1 📋

6. Encryption 🔐

as we said in 2.5, there's a flag bit that declares the following data (elementary or structured) as encrypted. this data isn't interpretable until it's decrypted. en/decryption also happens transparently through the SDXF functions: "create" does the encryption for elementary chunks, "leave" for structured chunks, "extract" does the decryption for elementary chunks, and "enter" for structured chunks. (yes it sounds very similar to chapter 5 😭)

more than one encryption method for a given range of applications isn't very reasonable. some encryption algorithms work with block ciphering, which means the length of the data to encrypt must be rounded up to the next multiple of the block length. this blocksize (zero means non-blocking) is reported by the encryption interface routine (addressed by the option field *encryptProc, see chapter 8.5) with mode=3. if blocking is used, at least one byte is added, and the last byte of the lengthening data holds the number of added bytes minus one. with that, the decryption interface routine can calculate the real data length 🧮

if an application (or a network connect handshaking protocol) needs to negotiate an encryption method, it should use a method number maintained by IANA, see chap. 12.2.

even though en/decryption is transparent, an encryption key (password) still has to be given to the SDXF functions 🗝️ encryption happens after character data gets translated, and decryption happens before translation from the internal ("network-") format.

if both encryption and compression are applied to the same chunk, compression goes first. compressing good encrypted data is a waste of time, since identical strings look different after encryption and the compression rates tend to zero 💀

7. Arrays 🧺

an array is a sequence of chunks with an identical chunk-ID, length and data type.

first, a hint: in principle, SDXF doesn't really need a special definition for an array 🙌

it isn't forbidden to have more than one chunk with the same chunk-ID inside the same structured chunk.

so with a sequence of SDX_next / SDX_extract calls, you can fill the destination array step by step.

if there are lots of chunks with the same chunk-ID (and a comparatively small length), the overhead of the chunk packages can be significant 😩

that's why the array flag exists. an array chunk has only one chunk header for the complete sequence of elementary chunks. after the chunk header for an array chunk, an array header follows:

this is a short integer (big endian!) that holds the number of array elements (CT). every element has a fixed length (EL), so the chunk length (CL) is CL = EL * CT + 2 ✨

the data elements follow immediately after the array header.

the complete array gets constructed by SDX_create, and the complete array gets read by SDX_extract.

the parameter fields (see 8.2.1) 'dataLength' and 'count' are used for the SDXF functions 'extract' and 'create':

  • Field 'dataLength' is the common length of the array elements.
  • 'count' is the actual dimension of the array for 'create' (input).

for the 'extract' function, 'count' acts as both an input and an output parameter:

  • Input: the maximum dimension
  • Output: the actual array dimension.

(if the output count is greater than the input count, the 'data cutted' warning gets responded and the destination array is filled up to the maximum dimension 🫠)

8. Description of the SDXF functions 🛠️

8.1 Introduction 💜

following the principles of Object Oriented Programming, it's not enough to describe the data. you also need the functions that manipulate it, the "methods" 🦋

for the programmer, knowing the methods matters more than knowing the data structure. the methods have to know the exact specifications of the data, and they guarantee the consistency of the data while creating it.

a SDXF object is an instance of a parameter structure that acts as a programming interface. specifically, it points to an actual SDXF data chunk, and while processing that data, there's a pointer to the actual inner chunk that will be the focus of the next operation.

the benefit of an exact interface description is the same as using standard C library functions, for example: by using standard interfaces, your code stays platform independent 👑

8.2 Basic definitions 🎀

8.2.1 The SDXF Parameter structure 💖

okay so every SDXF access function only needs ONE thing to work, and that's a pointer to the SDXF parameter structure 🫶 (lowercase, it's fine, we're just vibing). first, three prerequisite definitions:

typedef short int      ChunkID;
typedef unsigned char  Byte;

typedef struct Chunk
{
  ChunkID    chunkID;
  Byte       flags;
  char       length [3];
  Byte       data;
} Chunk;

and now the main event, the parameter structure ✨:

typedef struct
{
  ChunkID  chunkID;       // name (ID) of Chunk
  Byte    *container;     // pointer to the whole Chunk
  long     bufferSize;    // size of container
  Chunk   *currChunk;     // pointer to actual Chunk
  long     dataLength;    // length of data in Chunk
  long     maxLength;     // max. length of Chunk for SDX_extract
  long     remainingSize; // rem. size in cont. after SDX_create
  long     value;         // for data type numeric
  double   fvalue;        // for data type float
  char    *function;      // name of the executed SDXF function
  Byte    *data;          // pointer to Data
  Byte    *cryptkey;      // pointer to Crypt Key
  short    count;         // (max.) number of elements in an array
  short    dataType;      // Chunk data type / init open type
  short    ec;            // extended return-code
  short    rc;            // return-code
  short    level;         // level of hierarchy
  char     filler;        // filler char for SDX_extract
  Byte     encrypt;       // Indication if data to encrypt (0 / 1)
  Byte     compression;   // compression method
                          //   (00=none, 01=RL1, 02=zip/deflate)
} SDX_obj, *SDX_handle;

Only the "public" fields of the parameter structure are described here, since those are the ones that act as input and output for the SDXF functions 🌸. A given implementation may add some "private" fields to this structure, so don't be shocked if you see extra stuff in there 🙈

8.2.2 Basic Functions 🎀

bestie, all of these functions take a SDX_handle as their only formal parameter 💅. every one of them hands back ec and rc as a report of success. for the values of ec, rc and dataType, see chap. 8.4 👑

  1. SDX_init : Initialize the parameter structure ✨
input : container, dataType, bufferSize (for dataType =
        SDX_NEW only)
output: currChunk, dataLength (for dataType = SDX_OLD only),
        ec, rc,
        the other fields of the parameter structure will be
        initialized.
  1. SDX_enter : Enter a structured chunk. You can access the first chunk inside this structured chunk 🧚
input : none
output: currChunk, chunkID, dataLength, level, dataType,
        ec, rc
  1. SDX_leave : Leave the actual entered structured chunk 🦋
input : none
output: currChunk, chunkID, dataLength, level, dataType,
        ec, rc
  1. SDX_next : Go to the next chunk inside a structured chunk 💜
input : none
output: currChunk, chunkID, dataLength, dataType, count, ec, rc

at the end of a structured chunk, SDX_next returns rc = SDX_RC_failed and ec = SDX_EC_eoc (end of chunk) 💀. the actual structured chunk gets SDX_leave'd automatically, no thinking required 🫶

  1. SDX_extract : Extract data of the actual chunk 🔥. (if the actual chunk is structured, only a copy is done, otherwise the data gets converted to host format.) the input and output depend on the dataType:
if dataType is structured, binary or char:
  input : data, maxLength, count, filler
  output: dataLength, count, ec, rc

if dataType is numeric (float resp.):
  input : none
  output: value (fvalue resp.), ec, rc
  1. SDX_select : Go to the (next) chunk with a given chunkID 🍓
input : chunkID
output: currChunk, dataLength, dataType, ec, rc
  1. SDX_create : Creating a new chunk (at the end of the actual structured chunk) 🌈
input : chunkID, dataLength, data, (f)value, dataType,
        compression, encrypt, count
update: remainingSize, level
output: currChunk, dataLength, ec, rc
  1. SDX_append : Append a complete chunk at the end of the actual structured chunk) ⭐
input : data, maxLength, currChunk
update: remainingSize, level
output: chunkID, chunkLength, maxLength, dataType, ec, rc

8.3 Definitions for C++ 🌸

this is the specification of the SDXF class in C++ 💖 (quick aside: the type Byte is defined as "unsigned char" for bitstrings, as opposed to "signed char" for character strings, so don't mix them up 😭)

class C_SDXF
{
  public:

  // constructors and destructor:
  C_SDXF  ();                          // dummy
  C_SDXF  (Byte *cont);                // old container
  C_SDXF  (Byte *cont, long size);     // new container
  C_SDXF  (long size);                 // new container
  ~C_SDXF ();
  // methods:

  void init  (void);                   // old container
  void init  (Byte *cont);             // old container
  void init  (Byte *cont, long size);  // new container
  void init  (long size);              // new container

  void enter   (void);
  void leave   (void);
  void next    (void);
  long extract (Byte *data, long length);    // chars, bits
  long extract (void);                       // numeric data
  void create  (ChunkID);                    // structured
  void create  (ChunkID, long value);        // numeric
  void create  (ChunkID, double fvalue);     // float
  void create  (ChunkID, Byte *data, long length);// binary
  void create  (ChunkID, char *data);             // chars
  void set_compression (Byte compression_method);
  void set_encryption  (Byte *encryption_key);

  // interface:

  ChunkID  id;        // see 8.4.1
  short    dataType;  // see 8.4.2
  long     length;    // length of data or chunk

  long     value;
  double   fvalue;
  short    rc;  // the raw return code       see 8.4.3
  short    ec;  // the extended return code  see 8.4.4

  protected:
  // implementation dependent ...

};

8.4 Common Definitions: 🎀

8.4.1 Definition of ChunkID: 🧚

typedef short ChunkID;

8.4.2 Values for dataType: 🦋

SDX_DT_inconsistent     = 0
SDX_DT_structured       = 1
SDX_DT_binary           = 2
SDX_DT_numeric          = 3
SDX_DT_char             = 4
SDX_DT_float            = 5
SDX_DT_UTF8             = 6

 data types for SDX_init:
SDX_OLD                 = 1
SDX_NEW                 = 2

8.4.3 Values for rc: 💜

SDX_RC_ok               = 0
SDX_RC_failed           = 1
SDX_RC_warning          = 1
SDX_RC_illegalOperation = 2
SDX_RC_dataError        = 3
SDX_RC_parameterError   = 4
SDX_RC_programError     = 5
SDX_RC_noMemory         = 6

8.4.4 Values for ec: 🍓

SDX_EC_ok              =  0
SDX_EC_eoc             =  1 // end of chunk
SDX_EC_notFound        =  2
SDX_EC_dataCutted      =  3
SDX_EC_overflow        =  4
SDX_EC_wrongInitType   =  5
SDX_EC_comprerr        =  6 // compression error
SDX_EC_forbidden       =  7
SDX_EC_unknown         =  8
SDX_EC_levelOvflw      =  9
SDX_EC_paramMissing    = 10
SDX_EC_magicError      = 11
SDX_EC_not_consistent  = 12
SDX_EC_wrongDataType   = 13
SDX_EC_noMemory        = 14
SDX_EC_error           = 99 // rc is sufficiently

8.5 Special functions 🌈

besides the basic definitions there's a global function, SDX_getOptions, which returns a pointer to a global table of options ✨

with the help of these options you can adapt the behaviour of SDXF 🔥. especially, you can define an alternative pair of translation tables, or an alternative function that reads these tables from an external resource (p.e. from disk) 💅

there's also a pointer inside this table of options to the function used for encryption and decryption 🔐. you can install your own encryption algorithm by setting this pointer 🫶

the options pointer is received by:

SDX_TOptions *opt = SDX_getOptions ();

with:

typedef struct
{
 Byte            *toHost;        // Trans tab net -> host
 Byte            *toNet;         // Trans tab host -> net
 int              maxlevel;      // highest possible level
 int              translation;   // translation net <-> host
                                 // is in effect=1 or not=0
 TEncryptProc    *encryptProc;   // alternate encryption routine
 TGetTablesProc  *getTablesProc; // alternate routine defining
                                 // translation Tables
 TcvtUTF8Proc    *convertUTF8;   // routine to convert to/from UTF-8
}  SDX_TOptions;

typedef long TencryptProc (
  int   mode,   // 1= to encrypt, 2= to decrypt, 3= encrypted length
  Byte *buffer, // data to en/decrypt
  long  len,    // len: length of buffer
  char *passw); // Password

 // returns length of en/de-crypted data
 // (parameter buffer and passw are ignored for mode=3)
 // returns blocksize for mode=3 and len=0.
 // blocksize is zero for non-blocking algorithms

typedef int TGetTablesProc (Byte **toNet, Byte **toHost);
 // toNet, toHost: pointer to output params.  Both params
 // points to translation tables of 256 Bytes.
 // returns success: 1 = ok, 0 = error.

typedef int TcvtUTF8Proc
( int   mode,     // 1 = to UTF-8, 2 = from UTF-8
  Byte *target, int *targetlength,  // output
  Byte *source, int sourcelength);  // input
// targetlength contains maximal size as input param.
// returns success: 1 = ok, 0 = no conversion

9. 'Support' of UTF-8. 🌍

many systems support [UTF-8] as a character format for transferred data 💖. the benefit is that no fixing of a specific character set for an application is needed, because the set of 'all' characters is used, represented by the 'Universal Character Set' UCS-2 [UCS], a double byte coding for characters ✨

SDXF doesn't really deal with UTF-8 by itself 🙈. there are many possibilities to interpret a UTF-8 sequence, and the application may:

  • reconstruct the UCS-2 sequence,
  • accept only the pure ASCII characters and map non-ASCII to a special 'non-printable' character,
  • target pure ASCII, with non-ASCII replaced in a sensible manner (French accented vowels replaced by vowels without accents, etc.),
  • target a specific ANSI character set, with the non-ASCII chars mapped where possible and the others replaced by a 'non-printable' character,
  • etc.

but SDXF does offer an interface for the 'extract' and 'create' functions 🎀

a function pointer may be specified in the options table to maintain this possibility (see 8.5) 🦋. the default for this pointer is NULL: no further conversions are done by SDXF, the data are copied 'as is', and it's treated as a bit string, same as for data type 'binary' 💜

if this function is specified, it's used by the 'create' function with the 'toUTF8' mode, and by the 'extract' function with the 'fromUTF8' mode 🧚. SDXF invokes these functions transparently, so you don't have to do anything 🫶

if the function returns zero (no conversion), SDXF copies the data without conversion 🍓

10. Security Considerations ⚠️

any corruption of data in the chunk headers denounces the complete SDXF structure 💀
any corruption of data in an encrypted or compressed SDXF structure makes that chunk unusable 😭. an integrity check after decryption or decompression should be done by the "enter" function 🔥

while using TCP/IP (more precisely: IP) as a transmission medium, we can trust its CRC check on the transport layer 🌸

11. Some general hints ✨

okay so here are some vibes-check tips for building SDXF structures 💅

  1. A structure is only consistent if every "create" on a structured chunk gets closed by a matching "leave" 🧚 While a structured chunk is being built, its data type is set to zero, which means that chunk is inconsistent (not it, bestie). The SDX_leave function flips the datatype to "structured" ✨
  2. When you create an elementary chunk, the data gets converted from a platform dependent format to a platform independent one 🫶 By the time construction ends, the buffer is ready to ship to another site with no extra translation needed. ✈️
  3. You don't need a data definition in your programming language to build a specific SDXF structure, which is so freeing 🦋 The data gets created on the fly through function calls.
  4. SDXF can be the base for client / server protocols 💜 You can extend those protocols in a downward compatible way by following two rules:
    • Rule 1: Ignore unknown chunkIDs.
    • Rule 2: The sequence of chunks should not be significant.

12. IANA Considerations 🎀

okay so the compression and encryption algorithms for SDXF aren't locked in, because SDXF is open to all kinds of algorithms 🌈 That means everyone needs an agreement on what the compression and encryption algorithm method numbers actually mean. (Encryption isn't a semantic part of SDXF, but a connection protocol can use it to negotiate which encryption method to use 🔐)

IANA registers the following two items 👑

12.1 COMPRESSION METHODS FOR SDXF

The compressed SDXF chunk begins with a "compression header" 🗜️ That header holds the compression method as an unsigned 1-Byte integer (1-255). IANA assigns these numbers, and they're listed here:

compression
 method     Description                     Hints
---------   ------------------------------- -------------
      01    RUN-LENGTH algorithm            see chap. 5
      02    DEFLATE (ZIP)                   see [DEFLATE]
  03-239    IANA to assign
 240-255    private or application specific

12.2 ENCRYPTION METHODS FOR SDXF

A unique encryption method is either fixed or negotiated through handshaking 🤝 For the negotiated option, each encryption method needs its own number. These are unsigned 1-Byte integers (1-255), and IANA assigns them and lists them here:

encryption
  method    Description
 ---------  ------------------------------
  01-239    IANA to assign
 240-255    private or application specific

12.3 Hints for assigning a number:

if you're a developer who wants to register a compression or encryption method for SDXF, you should contact IANA to get a method number 📬 The ASSIGNED NUMBERS document is where you'll find the current list of METHOD numbers and their matching protocols, see [IANA]. The new method SHOULD be a standard published as an RFC or by an established standardization organization (such as OSI) 📜

13. Discussion 🌸

so SDXF is entering a world with other data exchange standards, and the IETF prefers ASN.1 and XML 🙈 That means the reasons for creating a new data format should be discussed, and this section is where we do that.

13.1 SDXF vs. ASN.1

ASN.1 (see [ASN.1]) exists to give you a programming-language-independent way to define data structures 🧠 It doesn't define the real data format used to send data over the wire. For that you usually get BER or PER (or derivatives like CER and DER), see [BER] and [PER].

The idea behind ASN.1 is that on every platform where an application is developed, the data structures get described in ASN.1 notation. An ASN.1 compiler then turns those notations into the real language-dependent definitions 🪄

That same compiler also generates transform functions for those data structures, so they can be packed and unpacked to and from BER (or whatever format you're using) 📦

Comparing ASN.1 and SDXF directly is kind of awkward, honestly 😭 SDXF's data format is more closely related to BER (and its relatives). Using ASN.1 to define data structures doesn't contradict SDXF, but SDXF doesn't need a complete data structure to build the message you're sending, and it doesn't generate a complete data structure from the message it receives 🫶

The big difference is how the message gets built and interpreted, which I'm calling the "static" and "dynamic" concepts:

  • ASN.1 takes a "static" approach: the whole data structure has to exist before the message can be created.
  • SDXF builds and interprets the message "dynamically": the message gets packed and unpacked step by step through SDXF functions.

Static structures can work well for a bunch of applications, but for complex tasks it's often impossible to define the whole message at once. Try writing an ASN.1 description for a complex structured text document presented in XML, with sections, paragraphs, and text elements that can recursively contain sections with specific text attributes, and you'll see what I mean 📝

13.2 SDXF vs. XML

SDXF and XML have a lot in common, since they can both handle any recursive, complex data stream 💕 The main difference is the kind of data they're meant to carry:

  • XML works with pure text data (though XML doesn't standardize the character representation). An XML document with all its tags is readable by humans. Binary data like graphics isn't included directly, but it can be referenced through an external link, as in HTML.

In XML there's no strong separation between informational and control data. Escape characters (like "<" and "&") and the <![CDATA[...]]> construction are used to tell those two types of data apart.

  • SDXF maintains machine-readable data. It isn't designed to be read by humans or edited in a text editor, especially once compression and encryption are involved 🤖 The SDXF functions give you quick, easy access to every data element. The standard parser for an SDXF data structure always follows a simple template, the "while - switch -case ID - enter/extract" pattern outlined in chap. 3.4.2.

XML and SDXF (and even ASN.1) come from completely different philosophies, so a direct comparison may not be very sensible. XML has its own right to exist alongside ASN.1 (and even SDXF) 💅

That said, there's a chance to convert an XML data stream into an SDXF structure ✨ As a first step, every XML tag becomes an SDXF chunk ID. An elementary sequence like <tag>pure text</tag> can be transformed into an elementary (non-structured) chunk with data type "character". Tags with attributes and sequences with nested tags become structured chunks. Since XML allows a tag sequence anywhere in a text stream, you have to introduce an artificial "elementary text" tag, so let's say <t> is the tag for text elements. The sequence:

<t>this is a text <attr value='bold'>with</attr> attributes</t>

is, in thought, replaced by:

<t><et>this is a text </et><attr value='bold'><et>with</et></attr>
<et> attributes</et></t>

(with "et" as the "elementary text" tag) 🎀 That results in this SDXF structure:

ID_t
|
+-- ID_et = " this is a text "
|
+-- ID_attr
|   |
|   +-- ID_value = "bold"
|   |
|   +-- ID_et = "with"
|
+-- ID_et = " attributes"

ID_t and ID_et can be represented by the same chunk ID, and only the data type tells them apart: "structured" for <t> and "character" for <et> 🧩

Binary data like pictures can be embedded directly into an SDXF structure, instead of being referenced by an external link like in HTML 🖼️

14. Author's Address 🦋

Max Wildgrube, Schlossstrasse 120, 60486 Frankfurt, Germany

EMail: max@wildgrube.com

15. Acknowledgements 🫶

Max would like to thank Michael J. Slifcak (mslifcak@iss.net) for the supporting discussions 💖

16. References 📚

  • [ASN.1] Information processing systems - Open Systems Interconnection, "Specification of Abstract Syntax Notation One (ASN.1)", International Organization for Standardization, International Standard 8824, December 1987.
  • [BER] Information Processing Systems - Open Systems Interconnection - "Specification of Basic Encoding Rules for Abstract Notation One (ASN.1)", International Organization for Standardization, International Standard 8825-1, December 1987.
  • [DEFLATE] Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, May 1996.
  • [IANA] Internet Assigned Numbers Authority, http://www.iana.org/numbers.htm
  • [PER] Information Processing Systems - Open Systems Interconnection -"Specification of Packed Encoding Rules for Abstract Syntax Notation One (ASN.1)", International Organization for Standardization, International Standard 8825-2.
  • [UCS] ISO/IEC 10646-1:1993. International Standard -- Information technology -- Universal Multiple-Octet Coded Character Set (UCS)
  • [UTF8] Yergeau, F., "UTF-8, a transformation format of ISO 10646", RFC 2279, January 1998.

Copyright (C) The Internet Society (2001). All Rights Reserved.

This document and translations of it may be copied and furnished to others. Derivative works that comment on or otherwise explain it, or help with its implementation, may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, as long as the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations. The exceptions are when it's needed for developing Internet standards, in which case the procedures for copyrights defined in the Internet Standards process must be followed, or when it's translated into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Acknowledgement 🌈

Funding for the RFC Editor function is currently provided by the Internet Society.

📦 where this page comes from