RFC 6873

Format for the Session Initiation Protocol (SIP) Common Log Format (CLF)

📅 published February 2013, from ietf

RFC 6873 📜✨

Format for the Session Initiation Protocol (SIP) Common Log Format (CLF)

  • Series: Internet Engineering Task Force (IETF), Request for Comments: 6873
  • Authors: G. Salgueiro (Cisco Systems), V. Gurbani (Bell Labs, Alcatel-Lucent), A. B. Roach (Mozilla)
  • Category: Standards Track 🌸
  • ISSN: 2070-1721
  • Date: February 2013 📅
  • Obsoletes/Updates: Nothing listed. This title block doesn't name any RFCs it obsoletes or updates. 🙌

Abstract 🦋

Okay so the SIPCLF working group put together a Common Log Format (CLF) framework for Session Initiation Protocol (SIP) servers 💅. It's basically copying the iconic event logging energy of well-known web servers like Apache and web proxies like Squid 🔥. This doc proposes an indexed text encoding format for the SIP CLF that keeps the big perks of a text-based format but runs way, way faster than a purely text-based setup ⚡. This file format follows the SIP CLF information model and gives an effective encoding scheme for all mandatory and optional fields that show up in a SIP CLF record 🎀.

Status of This Memo 🧚

This is an Internet Standards Track document 💖.

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community, and it got public review and was approved for publication by the Internet Engineering Steering Group (IESG) 👑. If you want more info on Internet Standards, check out Section 2 of RFC 5741 📚.

Info about the current status of this document, any errata, and how to give feedback on it lives at http://www.rfc-editor.org/info/rfc6873 🔗.

Copyright (c) 2013 IETF Trust and the people identified as the document authors. All rights reserved 🌈.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) as they stand on the date this document was published. Please read these carefully, bestie, because they spell out your rights and restrictions for this document 🫶. Code Components pulled out of this document must include Simplified BSD License text, as described in Section 4.e of the Trust Legal Provisions, and they come without warranty, as described in the Simplified BSD License ⚠️.

Table of Contents 📖

1. Introduction ....................................................3
2. Terminology .....................................................3
3. Document Conventions ............................................4
4. Format ..........................................................5
   4.1. Index Pointers .............................................8
   4.2. Mandatory Fields ..........................................10
   4.3. SIP CLF Encoding and Character Escaping Requirements ......13
   4.4. Optional Fields ...........................................14
5. Example SIP CLF Record .........................................22
6. Text Tool Considerations .......................................24
7. Security Considerations ........................................24
8. Operational Guidance ...........................................25
9. IANA Considerations ............................................25
   9.1. SIP CLF Version ...........................................25
   9.2. SIP CLF Transport Flag ....................................26
10. Acknowledgments ...............................................26
11. References ....................................................27
   11.1. Normative References .....................................27
   11.2. Informative References ...................................27

1. Introduction ✨

The Apache Common Log Format (CLF) has so many benefits and got so widely adopted that people wanted a similar event logging mechanism for the Session Initiation Protocol (SIP) [RFC3261] 😍. Logging for SIP is a real challenge 😤. Part of that is because a SIP entity behaves in a more complex way than an HTTP entity 🤯. On top of that, the purely text-based HTTP CLF has shortcomings that need fixing so you can do real-time inspection of SIP log files [RFC6872] 👀. Experience with Apache CLF has shown that working with huge piles of log data is super processor-intensive, because you have to read and parse every single byte in the log file(s) you're looking at 💀.

An implementation-independent framework for the SIP CLF was defined in [RFC6872] 📋. This memo describes an indexed text file format for logging SIP messages received and sent by SIP clients, servers, and proxies, and it follows the information model in Section 8 of [RFC6872] 🧚. This format is no harder for logging entities to generate than standard (aka non-indexed) text log formats, and it's radically faster to process 🚀. It's optimized for two things: rapidly scanning through log records and quickly finding the data fields people access most 💅.

The format proposed here also keeps the key advantage of being human readable and playing nicely with Unix text processing tools like sed, awk, perl, cut, and grep 🌸.

2. Terminology 💅

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] 🎀.

"SHOULD", "SHOULD NOT", "RECOMMENDED", and "NOT RECOMMENDED" are the right move when valid exceptions to a general requirement are known to exist (or seem to), and it's infeasible or impractical to list every single one 🙈. Still, they should not be read as permission for implementers to skip the general requirement when skipping it would cause an interoperability failure 🚫. Don't be that person 😭.

[RFC3261] defines more terms used in this document that are specific to the SIP domain 📚. These include "proxy"; "registrar"; "redirect server"; "user agent server" or "UAS"; "user agent client" or "UAC"; "back-to-back user agent" or "B2BUA"; "dialog"; "transaction"; and "server transaction" 🦋.

This document uses the term "SIP Server" to include these SIP entities: user agent server, registrar, redirect server, a SIP proxy in the role of user agent server, and a B2BUA in the role of a user agent server 🧚.

The reader is expected to be familiar with the terminology and concepts defined in [RFC6872] 💖.

3. Document Conventions 🎀

This document defines the logging syntax for the SIP CLF 📝. That syntax gets shown off through a bunch of examples 🌈. The formatting rules for RFCs mean those examples can't be unambiguously rendered as-is 😩. So to avoid ambiguity and meet the RFC layout requirements, this document uses the <allOneLine/> markup convention established in [RFC4475] ✨.

For clarity and completeness, the entire text defining this markup convention from Section 2.1 of [RFC4475] is quoted below 🫶:

Several of these examples contain unfolded lines longer than 72 characters. These are captured between <allOneLine/> tags. The single unfolded line is reconstructed by directly concatenating all lines appearing between the tags (discarding any line feeds or carriage returns). There will be no whitespace at the end of lines. Any whitespace appearing at a fold-point will appear at the beginning of a line. The following represent the same string of bits:
Header-name: first value, reallylongsecondvalue, third value
<allOneLine>
Header-name: first value,
 reallylongsecondvalue
, third value
</allOneLine>
<allOneLine>
Header-name: first value,
 reallylong
second
value,
 third value
</allOneLine>
Note that this is NOT SIP header-line folding, where different strings of bits have equivalent meaning.

The IP addresses used in the examples in this document correspond to the documentation address block 192.0.2.0/24 (TEST-NET-1) as described in [RFC5737] 🦋.

Part 1 of 6 is done, bestie. Keep your eyes peeled for Section 4 next 👀💜

4. Format ✨

okay bestie, this is where we get into the actual layout 🫶 The CLF for the Session Initiation Protocol [RFC6872] defines an information model that this logging format sticks to, and Section 8.1 of that document lists all the mandatory information model elements 💅

here's the format for SIP CLF records:

 0          7 8        15 16       23 24         31
 +-----------+-----------+-----------+-----------+
 |  Version  |           Record Length           | 0 - 3
 +-----------+-----------+-----------+-----------+
 |       Record Length (cont)        |    0x2C   | 4 - 7
 +-----------+-----------+-----------+-----------+
 |              CSeq Pointer (Hex)               | 8 - 11
 +-----------+-----------+-----------+-----------+
 |      Response Status-Code Pointer (Hex)       | 12 - 15
 +-----------+-----------+-----------+-----------+
 |              R-URI Pointer (Hex)              | 16 - 19
 +-----------+-----------+-----------+-----------+
 |   Destination IP address:port Pointer (Hex)   | 20 - 23
 +-----------+-----------+-----------+-----------+
 |     Source IP address:port Pointer (Hex)      | 24 - 27
 +-----------+-----------+-----------+-----------+
 |             To URI Pointer (Hex)              | 28 - 31
 +-----------+-----------+-----------+-----------+
 |             To Tag Pointer (Hex)              | 32 - 35
 +-----------+-----------+-----------+-----------+
 |            From URI Pointer (Hex)             | 36 - 39
 +-----------+-----------+-----------+-----------+
 |            From Tag Pointer (Hex)             | 40 - 43
 +-----------+-----------+-----------+-----------+
 |             Call-Id Pointer (Hex)             | 44 - 47
 +-----------+-----------+-----------+-----------+
 |           Server-Txn Pointer (Hex)            | 48 - 51
 +-----------+-----------+-----------+-----------+
 |           Client-Txn Pointer (Hex)            | 52 - 55
 +-----------+-----------+-----------+-----------+
 |      Optional Fields Start Pointer (Hex)      | 56 - 59
 +-----------+-----------+-----------+-----------+
 |    0x0A   |                                   | 60 - 63
 +-----------+                                   +
 |                   Timestamp                   | 64 - 67
 +                                   +-----------+
 |                                   |    0x2E   | 68 - 71
 +-----------+-----------+-----------+-----------+
 |         Fractional Seconds        |    0x09   | 72 - 75
 +-----------+-----------+-----------+-----------+
 |                  Flags Field                  | 76 - 79
 +-----------+-----------+-----------+-----------+
 |Flag (cont)|    0x09   |                       | 80 - 83
 |-----------+-----------+                       |
 |                                               |
 |                                               |
 |      Mandatory Fields (variable length)       |
 |                                               |
 |                                               |
 +-----------+-----------+-----------+-----------+
 |    0x09   |          Tag          |    0x40   |\
 +-----------+-----------+-----------+-----------+ \
 |                   Vendor-ID                   |  \
 +-----------+-----------+-----------+-----------+   \
 |                Vendor-ID (cont)               |    \   Repeated
 +-----------+-----------+-----------+-----------+     \  as many
 |     0x2C  |            Length (Hex)           |      > times as
 +-----------+-----------+-----------+-----------+     /  necessary
 | Len (cont)|    0x2C   |    BEB    |    0x2C   |    /
 +-----------+-----------+-----------------------|   /
 |                                               |  /
 |            Value (variable length)            | /
 |                                               |/
 +-----------+-----------+-----------+-----------+
 |    0x0A   |
 +-----------+

Figure 1: SIP Common Log Format

the format in Figure 1 is for ONE single SIP CLF log entry 📝 there's no actual subdivision in practice, but it can be logically split into three distinct components, and honestly this is the part that makes it click for me:

  1. Index Pointers: the first 60 bytes of the format 🧚 This part is metadata, mostly a list of pointers that mark where the variable-length mandatory and optional fields start. The pointers exist to make processing faster and let a reader skip straight to the field they want without slogging through the whole record 🏃‍♀️ This logical piece gets referred to in this document as <IndexPointers>. A 0x0A (LF character) separates <IndexPointers> from the next grouping.
  2. Mandatory Fields: the next grouping is a Tab-delimited (0x09) list of the mandatory fields described in Section 8.1 of [RFC6872], in the order they're listed in <IndexPointers> 💖 This piece gets referred to as <MandatoryFields>.
  3. Optional Fields: the last component MAY be present, since it's an OPTIONAL extension to the SIP CLF format 🌈 Its whole purpose is flexibility, so a developer can log any fields that aren't in <MandatoryFields>, including SIP bodies and vendor-specific extensions. This piece gets referred to as <OptionalFields>.

this logical structure can be shown like this:

                             <IndexPointers>
                             <MandatoryFields>
                             <OptionalFields>

Figure 2: Logical Structure of the SIP CLF Record

ngl, Figures 1 and 2 plus the terminating line-feed (0x0A) at the end of the record are two different ways of showing the SAME format, and they're functionally equivalent 🦋 In practice the record is two lines: the <IndexPointers> metadata sits on one line, and the actual data (<MandatoryFields> and <OptionalFields> if present) sits on the other 🎀

in the sections that follow, "hexadecimal encoded" means the value is always unsigned and gets written out as human-readable base-16 numbers, using the UTF-8 characters 0x30 through 0x39 ('0' through '9') and 0x41 through 0x46 ('A' through 'F') 💜 "Decimal encoded" means the value gets written out as human-readable base-10 numbers, using the UTF-8 characters 0x30 through 0x39 ('0' through '9') 🌸 In both encodings, numbers always take up the number of bytes indicated and get padded on the left with UTF-8 '0' (zero) characters to fill the whole space ✨

4.1. Index Pointers

the <IndexPointers> portion of the SIP CLF record (shown in Figure 3) is a 60-byte header holding metadata about the record 🍓

            0          7 8        15 16       23 24         31
            +-----------+-----------+-----------+-----------+
            |  Version  |           Record Length           | 0 - 3
            +-----------+-----------+-----------+-----------+
            |       Record Length (cont)        |    0x2C   | 4 - 7
            +-----------+-----------+-----------+-----------+
            |              CSeq Pointer (Hex)               | 8 - 11
            +-----------+-----------+-----------+-----------+
            |      Response Status-Code Pointer (Hex)       | 12 - 15
            +-----------+-----------+-----------+-----------+
            |              R-URI Pointer (Hex)              | 16 - 19
            +-----------+-----------+-----------+-----------+
            |   Destination IP address:port Pointer (Hex)   | 20 - 23
            +-----------+-----------+-----------+-----------+
            |     Source IP address:port Pointer (Hex)      | 24 - 27
            +-----------+-----------+-----------+-----------+
            |             To URI Pointer (Hex)              | 28 - 31
            +-----------+-----------+-----------+-----------+
            |             To Tag Pointer (Hex)              | 32 - 35
            +-----------+-----------+-----------+-----------+
            |            From URI Pointer (Hex)             | 36 - 39
            +-----------+-----------+-----------+-----------+
            |            From Tag Pointer (Hex)             | 40 - 43
            +-----------+-----------+-----------+-----------+
            |             Call-Id Pointer (Hex)             | 44 - 47
            +-----------+-----------+-----------+-----------+
            |           Server-Txn Pointer (Hex)            | 48 - 51
            +-----------+-----------+-----------+-----------+
            |           Client-Txn Pointer (Hex)            | 52 - 55
            +-----------+-----------+-----------+-----------+
            |      Optional Fields Start Pointer (Hex)      | 56 - 59
            +-----------+-----------+-----------+-----------+

Figure 3: Index Pointers

here's what makes up <IndexPointers> 🫧

Version (1 byte): this is the UTF-8 encoded version for the SIP CLF record 💅 Valid values run from 'A' (0x41) to 'Z' (0x5A). This document uses a Version value of "0x41" ('A').

The value of the SIP CLF Version MUST be incremented for any new SIP CLF specification that changes any part of the SIP CLF record format. The SIP CLF Version values are IANA-assigned (Section 9.1) via the Standards Action method described in [RFC5226].

since the version is set per record, a SIP CLF log file COULD end up with records that have different versions 🙈 Under normal operating conditions that's pretty unlikely, and it SHOULD be avoided if possible.

Record Length (6 bytes): this is the hexadecimal encoded total length of the log record, starting at the "Version" octet and ending with the terminating line-feed 📏

bytes 8 through 55 hold hexadecimal encoded pointers that mark the starting location of each variable-length mandatory field. Bytes 56 through 59 hold a hexadecimal encoded pointer to where the optional fields portion of the record starts 🧚 No delimiters go between these pointer values. They're packed together as one single 52-character hexadecimal encoded string, which is very cute and very compact 🌈 The "Pointer" fields are absolute byte values within the record, so they're always >=82 🔥 They point to the start of the matching value in the <MandatoryFields> portion. Section 4.2 describes each of the mandatory fields these pointers point to.

Optional Fields Start Pointer: this final pointer shows where the OPTIONAL group of <OptionalFields> begins, if it's present 👑 The "Optional Fields Start Pointer" points to the UTF-8 Tab (0x09) character for the first entry in the <OptionalFields> portion.

If the OPTIONAL group of <OptionalFields> is not implemented, then the "Optional Fields Start Pointer" field MUST point to the terminating line-feed (0x0A) at the end of the SIP CLF record. 🙅‍♀️

4.2. Mandatory Fields ✨

okay so the <MandatoryFields> part of the SIP CLF record looks like this 👀

            0          7 8        15 16       23 24         31
            +-----------+-----------+-----------+-----------+
            |    0x0A   |                                   | 60 - 63
            +-----------+                                   +
            |                   Timestamp                   | 64 - 67
            +                                   +-----------+
            |                                   |    0x2E   | 68 - 71
            +-----------+-----------+-----------+-----------+
            |         Fractional Seconds        |    0x09   | 72 - 75
            +-----------+-----------+-----------+-----------+
            |                  Flags Field                  | 76 - 79
            +-----------+-----------+-----------+-----------+
            |Flag (cont)|    0x09   |                       | 80 - 83
            |-----------+-----------+                       |
            |                                               |
            |                                               |
            |      Mandatory Fields (variable length)       |
            |                                               |
            |                                               |
            +-----------+-----------+-----------+-----------+

                        Figure 4: Mandatory Fields

Following the pointers in <IndexPointers>, two fixed-length fields get encoded to pin down the exact time of the log entry. Same as before, every field is completely filled, with '0' characters pre-pended to the values as needed 🫶

  • Timestamp (10 bytes): the decimal encoded date and time of the request or response, written as the number of seconds since the Unix epoch (aka seconds since midnight, January 1st, 1970, GMT) 🕰️
  • Fractional Seconds (3 bytes): the decimal encoded fractional seconds part of the Timestamp field, to millisecond accuracy ✨

The combined Timestamp and Fractional Seconds fields show up in the log file as a UTF-8 encoded string. It's giving "seconds plus milliseconds since the Unix epoch" 💖 The milliseconds get separated from the seconds by a "." (UTF-8 character 0x2E).

  • Flags Field (5 bytes): 🚩
    • byte 1 - Request/Response Flag
      • R = Request
      • r = Response
    • byte 2 - Retransmission Flag
      • O = Original transmission
      • D = Duplicate transmission
      • S = Server is stateless [i.e., retransmissions are not detected]
    • byte 3 - Sent/Received Flag
      • S = Sent message
      • R = Received message
    • byte 4 - Transport Flag
      • The Transport Flag values are IANA-assigned (Section 9.2) via the IETF Review method described in [RFC5226]. Currently, the registered values are:
        • U = UDP
        • T = TCP
        • S = SCTP
    • byte 5 - Encryption Flag
      • E = Encrypted message (TLS, DTLS, etc.)
      • U = Unencrypted message

After the "Timestamp", "Fractional Seconds", and "Flags" fields come the values for the mandatory fields from Section 8.1 of [RFC6872]. Here's the tea on each one 🍵

  • CSeq: the Command Sequence header field, including the CSeq number and method name.
  • Response Status-Code: set to the value of the SIP response status code for responses. Set to a single UTF-8 dash (0x2D) for requests.
  • R-URI: the Request-URI in the start line (mandatory in request), including any URI parameters.
  • Destination IP address:port: the IP address of the downstream server and the port number, separated by a single ':'. IPv4 addresses are written in "dotted decimal" notation as per [RFC1166]. IPv6 addresses use the hexadecimal notation from Section 4 of [RFC5952] (or the special-case mixed hexadecimal and decimal notation from Section 5 of [RFC5952]) and get wrapped in square brackets ('[' and ']').
  • Source IP address:port: the IP address of the upstream client and the port number the SIP message came in on, separated by a single ':'. IPv4 addresses are written in "dotted decimal" notation as per [RFC1166]. IPv6 addresses use the hexadecimal notation from Section 4 of [RFC5952] (or the special-case mixed hexadecimal and decimal notation from Section 5 of [RFC5952]) and get wrapped in square brackets ('[' and ']').
  • To URI: the value of the URI in the To header field.
  • To Tag: the value of the tag parameter (if present) in the To header field.
  • From URI: the value of the URI in the From header field.
  • From Tag: the value of the tag parameter (if present) in the From header field.
  • Call-Id: the value of the Call-ID header field.
  • Server transaction identification code (Server-Txn): the transaction identifier associated with the server transaction. Implementations can reuse the server transaction identifier (the topmost branch-id of the incoming request, with or without the magic cookie), or they could generate a unique identification string for a server transaction (this identifier needs to be locally unique to the server only). This identifier is used to correlate ACKs and CANCELs to an INVITE transaction, and it also helps track forking. (See Section 9 of [RFC6872] for usage.)
  • Client transaction identification code (Client-Txn): this field associates client transactions with a server transaction for forking proxies or B2BUAs. Upon forking, implementations can reuse the value they inserted into the topmost Via header's branch parameter, or they can generate a unique identification string for the client transaction. (See Section 9 of [RFC6872] for usage.)
Note: The definitions of the Server-Txn and Client-Txn are taken directly from [RFC6872] and are provided here only as a convenience to the implementer. The definitions specified in [RFC6872] should be considered authoritative in the event of a conflict. 🙈

This data MUST appear in the order listed in <IndexPointers>, and each field MUST be present 📋 Fields are subject to the maximum SIP CLF field size of 4096 bytes, as detailed in Section 8 of [RFC6872] 💜

4.3. SIP CLF Encoding and Character Escaping Requirements

The mandatory fields in a SIP CLF record are separated by a single UTF-8 Tab character (0x09). Any Tab characters present in the data to be written will be replaced by a UTF-8 space character (0x20) before being logged 🎀

The decision to swap tabs for spaces was based on there being no standardized use of tabs in SIP headers to convey any other meaning than whitespace. Tabs may appear in message bodies, though, and if those bodies get logged, the conversion to space may cause problems when rebuilding the body from the matching log entry 😭 Two consequences of replacing Tab with a space are: (a) it becomes impossible to reconstruct a signature over the logged field that matches the signature over fields in the original SIP message, and (b) any future SIP header fields that include tabs with a different semantic meaning than simply whitespace will lose that meaning when logged. Finally, the tabs-to-spaces substitution MUST occur when logging mandatory fields and optional SIP Header Field or Reason-Phrase (Tag=00). It MUST also occur when optionally logging either the entire message (Tag=02) or just a SIP body (Tag=01), as described in Section 4.4.

An element won't always have an appropriate value for one of these fields, even when the field is required to appear in the SIP CLF record. In those cases, when a given mandatory field from Section 4.2 (specified in Section 8.1 of [RFC6872]) is not present, that empty field MUST be encoded as a single horizontal dash ("-"). If a field failed to parse, it MUST be encoded as a single question mark ("?"). When these characters are part of a sequence of other characters, there is no ambiguity 🌸 If the field being logged contains only one character, and that character is the literal "-", the implementation SHOULD insert an escaped %2D for that field in the SIP CLF record. Similarly, if the field contains only one character, and that character is the literal "?", the implementation SHOULD insert an escaped %3F for that field in the SIP CLF record.

The terminating carriage return line feed (CRLF) after a given header field value MUST NOT be logged. A bare CRLF sequence isn't permitted within a SIP header field value, so mandatory fields MUST NOT contain a CRLF when logged, and no escaping mechanism is required for it 🦋

Clearly, a SIP parser could not possibly parse a SIP CLF record in its entirety, given the SIP CLF format described in this document. 💀 It is possible to parse individual fields in the SIP CLF record if they're extracted and handed to a SIP parser that would normally parse that sequence of strings. It should be noted that any field value modified by the escaping mechanisms defined in this document before logging ('-', '?', and CRLF) is likely no longer well-formed SIP and will fail when given to such a parser.

The intent of logging using SIP CLF is not to faithfully recreate the bit-exact SIP message being logged. The formatting rules, encoding, and character escaping requirements actually preclude this, and they may introduce information loss relative to the original SIP message 🧚 A log reader should never unescape anything in the SIP CLF record, since these records are meant to be machine processed with text tools like grep and awk. The human user behind the log reader may need to infer more semantics about any differences between the original SIP message and its SIP CLF representation. ✨

4.4. Optional Fields

The <OptionalFields> portion of the SIP CLF record is shown below 🌈

     0          7 8        15 16       23 24         31
     +-----------+-----------+-----------+-----------+
     |    0x09   |          Tag          |    0x40   |\
     +-----------+-----------+-----------+-----------+ \
     |                   Vendor-ID                   |  \
     +-----------+-----------+-----------+-----------+   \
     |                Vendor-ID (cont)               |    \   Repeated
     +-----------+-----------+-----------+-----------+     \  as many
     |     0x2C  |            Length (Hex)           |      > times as
     +-----------+-----------+-----------+-----------+     /  necessary
     | Len (cont)|    0x2C   |    BEB    |    0x2C   |    /
     +-----------+-----------+-----------------------|   /
     |                                               |  /
     |            Value (variable length)            | /
     |                                               |/
     +-----------+-----------+-----------+-----------+

                         Figure 5: Optional Fields

Okay so optional fields are basically any SIP message bits that are NOT on the mandatory list from Section 8.1 of [RFC6872] 🎀 After the <MandatoryFields> part, there's an OPTIONAL <OptionalFields> group (check out Figure 5 💅) that MAY show up zero or more times ✨ This is the extensibility magic of SIP CLF 🦋 Implementers get to log way more than just the mandatory log elements from Section 8.1 of [RFC6872] 🌈

Logging any optional SIP elements MUST be done according to the format in Figure 5, no exceptions 🫶 The start of <OptionalFields> inside a SIP CLF record is marked by the "Optional Fields Start Pointer" field in <IndexPointers> 📍 After the initial Tab delimiter byte (0x09) from Figure 5, the optional field being logged is usually written like this:

Tag@Vendor-ID,Length,BEB,Value

The optional field identifier (Tag@Vendor-ID) is a two-byte Tag plus an eight-byte Vendor-ID (both decimal encoded), separated by an "@" character (0x40) 💜 That combo uniquely identifies the optional field being logged 🔥 The format is loosely modeled on the private use option from syslog [RFC5424] (fyi, this is the second format in Section 6.3.2 of [RFC5424] 🤓). It uses the Private Enterprise Number (PEN), which gives you an identifier from a globally unique name space [PEN] 🌸 This syntax lets SIP CLF log any SIP header, any body, and any vendor-specified SIP element. Literally anything 💀✨

The Base64 Encoded Byte (BEB) is a boolean that says whether the optional element is Base64 encoded 🍓 The Value field MUST be Base64 encoded if it has any "unprintable" characters. For this doc, "unprintable" means a string of octets that: (a) has an octet in the range 0 to 31, inclusive; (b) has an octet with the value 127; or (c) has any series of octets greater than or equal to 128 that does NOT form a valid UTF-8 sequence, as specified by [UNICODE] 😭 If the element is Base64 encoded, then BEB=0x01. If it isn't, then BEB=0x00 🧚

Optional fields are logged using these two syntax rules 👑

(1) Vendor-ID = 00000000

A Vendor-ID of zero is used to log the entire SIP message, message body, Reason-Phrase, or any SIP header fields that are NOT part of the mandatory fields list from Section 8.1 of [RFC6872] 🙈 The following Tag values tell you which optional element is being logged:

Tag = 00 - Log SIP Header Field or Reason-Phrase

When logging a SIP Header Field (Tag=00), the "Value" field MUST be populated with the entire header field: the field-name, the colon (":"), and the field-value 💖 This lets you optionally log any SIP header field by naming it inside the "Value" field.

The Reason-Phrase in a response lives in the Status-Line and doesn't have a field-name, so it's a special case 🦋 Here, the "Value" field MUST be populated with the name "Reason-Phrase", then a colon (":"), then a single space (SP) before the logged Reason-Phrase value 🎀

The corresponding "Length" field counts the whole "Value" field, including the field-name, the colon, and any linear whitespace (LWS) separator ✨ For Tag=00, the BEB depends on whether the SIP Header Field value has any 'unprintable' characters. If it doesn't, BEB=00. If it does, BEB=01. When BEB=01, ONLY the field-value MUST be Base64 encoded. The field-name, the colon, and any LWS separator MUST keep their original encoding 🫶

If an optional field shows up more than once in a SIP message (like Contact, Route, Record-Route, etc.), each occurrence MUST be logged with the same Tag value (Tag=00) as its own distinct optional field entry in the SIP CLF record 🌈 These repeated header fields MUST keep their ordinal position from the SIP header. For example, say a SIP header has two Via header fields in ordinal positions V1,V2. If you log them optionally, they'd show up as these entries in the SIP CLF record. (Note: to keep things short, this example only shows how these optional header fields get logged and skips the rest of the SIP CLF record 🙏)

00@00000000,len_V1,00,Via: V1     00@00000000,len_V2,00,Via: V2

The terminating carriage return line feed (CRLF) after a header field value MUST NOT be logged 🚫 A bare CRLF isn't allowed inside a SIP header field value anyway, so optional SIP header fields logged with Tag=00 MUST NOT contain a CRLF when logged. That means no escaping mechanism is needed for it 💅

Tag = 01 - Log message body

SIP message bodies of all types can be optionally logged with Tag=01 🌸 If the body IS logged, it MUST follow the 4096-byte maximum size limit for a SIP CLF field, from Section 8 of [RFC6872] 😭 Unlike Tag=00, there can only be ONE entry with Tag=01 in the SIP CLF record. If the body goes over the 4096-byte max when you're logging it, it MUST be truncated to fit those size limits ✂️

When logging a message body (Tag=01), the "Value" field holds the Content-Type itself, plus the SIP message body, separated by a space 🍓 This way everything about the body is self-described with one tag, instead of needing a separate tag for each body type 🧠 The "Length" field counts the SIP message body, the length of the embedded Content-Type, and the space separator between the MIME type and the body content 💜

For an optionally logged message body (Tag=01), the BEB depends on whether the body has any 'unprintable' characters. If not, BEB=00. If so, BEB=01. When BEB=01, the body that follows is entirely Base64 encoded EXCEPT for the prepended Content-Type we just talked about 🦋

If an optionally logged SIP message body has any CRLFs, they MUST be escaped using the URI encoded equivalent "%0D%0A" 🔥 This escaping applies to all body types. We don't treat the body types differently 👑 If a logged message body has BEB=01, it MUST be Base64 encoded BEFORE any character escaping. So if a binary body (like an image) gets logged, it's Base64 encoded first, and that Base64 stream can never include the CRLF escape sequence "%0D%0A", because "%" is not a valid Base64 character 🧚✨

Tag = 02 - Log entire SIP message

The entire SIP message (SIP header and message body) can be optionally logged with Tag=02 💖 Logging the entire message MUST conform to the 4096-byte maximum size limit for a SIP CLF field, from Section 8 of [RFC6872] 😭 Unlike Tag=00, there can only be ONE entry with Tag=02 in the SIP CLF record. If the entire SIP message goes over 4096 bytes, it MUST be truncated to meet those size limits 🌈

When optionally logging an entire SIP message (Tag=02), the BEB depends on whether the message body portion has any 'unprintable' characters. If not, BEB=00. If so, BEB=01. When BEB=01, the entire SIP message is Base64 encoded (not just the body) 🍓 Here's the plot twist: unlike Tag=01, when you log an entire SIP message (Tag=02) with 'unprintable' characters (BEB=01), the Content-Type would NOT be known before decoding 🙈

All CRLFs, whether they're in the SIP headers or the SIP message body, MUST be escaped using the URI encoded equivalent "%0D%0A". If a logged SIP message has BEB=01, it MUST be Base64 encoded BEFORE any character escaping 🫶

(2) Vendor-ID = PEN

A Vendor-ID set to a vendor's own private enterprise number, taken from the complete current list of private enterprise numbers that IANA maintains [PEN], is used to log any other vendor-specified optional element of a SIP header or body 🌸 The value of the Tag is up to the implementer 🎀

Tag = Vendor-specified tag

The different values of the optional field identifier (Tag@Vendor-ID) are the basis for how optional elements get logged in SIP CLF 🔥 For completeness, here are the other fields in the Figure 5 format 👑

Length Field (4 bytes): Shows the length of only the "Value" field of this optionally logged element (see Figure 5), hexadecimal encoded. This length matches the "Value" field ONLY and MUST NOT include any of the other elements shown in Figure 5 🙏

Base64 Encoded Byte (BEB) Field (1 byte): Shows whether the subsequent Value Field of the optionally logged element is Base64 encoded. The Value field MUST be Base64 encoded if it contains any character that is 'unprintable' under the definition given earlier in this section. If the element is Base64 encoded, BEB=0x01. If not, BEB=0x00 💜

Value Field (0 to 4096 bytes): Holds the actual value of this optional field. Just like the mandatory fields, UTF-8 Tab characters (0x09) get replaced with UTF-8 space characters (0x20) 🍓

Here come the examples of optionally logged SIP elements using the syntax from this section 🧚 They only show the <OptionalFields> portion of the SIP CLF record. The mandatory <IndexPointers> and <MandatoryFields> portions are intentionally left out for brevity 🌈 Also, every one of these examples starts with a leading Tab delimiter byte (0x09) that you can't really see here 👀

(1) Contact header field logged as an optional field:

Consider the SIP response: ✨

SIP/2.0 180 Ringing
<allOneLine>
Via: SIP/2.0/UDP host.example.com;
branch=z9hG4bKnashds8;received=192.0.2.1
</allOneLine>
To: Bob <sip:bob@example.com>;tag=a6c85cf
From: Alice <sip:alice@example.com>;tag=1928301774
Call-ID: a84b4c76e66710
Contact: <sip:bob@192.0.2.4>
CSeq: 314159 INVITE
Content-Length: 0

Okay so the Contact header field gets logged as an optional field like this 📞✨:

00@00000000,001C,00,Contact: <sip:bob@192.0.2.4>

(2) Reason-Phrase logged as an optional field: For that same SIP response, the Reason-Phrase gets logged as an optional field like so 💅:

00@00000000,0016,00,Reason-Phrase: Ringing

(3) SDP body logged as an optional field: Here's a body you'd want to log 🎀:

v=0
o=alice 2890844526 2890844526 IN IP4 host.example.com
s=-
c=IN IP4 host.example.com
t=0 0
m=audio 49170 RTP/AVP 0 8 97

This body has a Content-Type of application/sdp and is 123 bytes long, line-feeds included. When you log it, the "Value" field is the Content-Type and the body joined together with a space, so the combined length comes out to 139 (0x008B) bytes 🧮. Logged as an optional field, it looks like this:

<allOneLine>
01@00000000,008B,00,application/sdp v=0%0D%0Ao=alice 2890844526
2890844526 IN IP4 host.example.com%0D%0As=-%0D%0A
c=IN IP4 host.example.com%0D%0At=0 0%0D%0A
m=audio 49170 RTP/AVP 0 8 97%0D%0A
</allOneLine>

The body is actually logged on one single line, so it's wrapped between <allOneLine/> tags. The line-feeds get escaped as %0D%0A to separate the lines of the message body 🫶.

(4) Binary body logged as an optional field: The second body part of the multipart/mime SIP message in Section 3.1.1.11 of RFC 4475 is a binary encoded body (shown in hex). If you logged it, it would have BEB=01 and would need Base64 encoding 🤓. Once Base64 encoded, that binary body turns into six lines of output. Then you escape the CRLF characters, and you end up with an optionally logged body that looks like this:

<allOneLine>
01@00000000,0216,01,multipart/mixed;boundary=7a9cbec02ceef655 MI
IBUgYJKoZIhvcNAQcCoIIBQzCCAT8CAQExCTAHBgUrDgMCGjALBgkqhkiG9w0BBw
ExggEgMIIB%0D%0AHAIBATB8MHAxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxp
Zm9ybmlhMREwDwYDVQQHEwhTYW4g%0D%0ASm9zZTEOMAwGA1UEChMFc2lwaXQxKT
AnBgNVBAsTIFNpcGl0IFRlc3QgQ2VydGlmaWNhdGUgQXV0%0D%0AaG9yaXR5AggB
lQBxAjMBEzAHBgUrDgMCGjANBgkqhkiG9w0BAQEFAASBgI70ZvlI8FIt0uWXjp2V
%0D%0Aquny/hWgZllxYpLo2iqo2DUKaM7/rjy9K/8Wdd3VZI5ZPdZHKPJiIPfpQX
SeMw2aFe2r25PRDEIQ%0D%0ALntyidKcwMmuLvvHwM/5Fy87An5PwCfhVG3ktqo6
uz5mzMtd1sZLg4MUnLjm/xgtlE/le2W8mdAF%0D%0A
</allOneLine>

Again, the body is logged on one single line, which is why it sits between <allOneLine/> tags. The line-feeds are escaped with %0D%0A to separate the lines of the Base64 encoded binary body 🔐✨.

(5) Codec information from the SDP body logged as an optional field: Let's look at this SIP message 👀:

INVITE sip:bob@example.com SIP/2.0
Via: SIP/2.0/UDP host.example.com;branch=z9hG4bKnashds8
To: Bob <bob@example.com>
From: Alice <alice@example.com>;tag=1928301774
Call-ID: a84b4c76e66710
CSeq: 314159 INVITE
Max-Forwards: 70
Date: Thu, 21 Feb 2002 13:02:03 GMT
Contact: <sip:alice@host.example.com>
Content-Type: application/sdp
Content-Length: 147

v=0
o=UserA 2890844526 2890844526 IN IP4 example.com
s=Session SDP
c=IN IP4 host.example.com
t=0 0
m=audio 49172 RTP/AVP 0
a=rtpmap:0 PCMU/8000

A vendor might want to log a SIP message element like the codec information from the SDP body, and that's totally allowed 🦋. This vendor-specified SIP element gets logged as an optional field like this:

03@00032473,0014,00,a=rtpmap:0 PCMU/8000

(6) N-th message received from a particular peer logged as an optional field: Maybe a vendor wants to log that this message is the n-th message received from a peering partner 🧚. To do that for the SIP message above, the vendor would log this information as:

07@00032473,0016,00,1877 example.com

Which means this is the 1,877th message from the peering partner example.com 📬. Also, the previous two examples of a vendor-specified SIP element used a Vendor-ID with a Private Enterprise Number of 32473. That value has been reserved by IANA for use as an example PEN in documentation, according to [RFC5612] 🌸.

5. Example SIP CLF Record 🎀

The following SIP message is an INVITE request sent by a SIP client 📨:

INVITE sip:192.0.2.10 SIP/2.0
To: <sip:192.0.2.10>
Call-ID: DL70dff590c1-1079051554@example.com
<allOneLine>
From: "Alice" <sip:1001@example.com:5060>;
tag=DL88360fa5fc;epid=0x34619b0
</allOneLine>
CSeq: 1 INVITE
Max-Forwards: 70
Date: Thu, 21 Feb 2012 15:02:03 GMT
<allOneLine>
Via: SIP/2.0/TCP 192.0.2.200:5060;
branch=z9hG4bK-1f6be070c4-DL
</allOneLine>
Contact: "1001" <sip:1001@192.0.2.200:5060>
Content-Type: application/sdp
Content-Length: 418

v=0
o=1001 1456139204 0 IN IP4 192.0.2.200
s=Session SDP
c=IN IP4 192.0.2.200
b=AS:2048
t=0 0
m=audio 13756 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000

Here's roughly how that message would look as a single record in a SIP CLF logging file, if it's encoded using the syntax from this document 🍓. Because of RFC formatting conventions, this log entry is split across five lines, when it actually lives on two lines in a real log file. The Tab characters are padded out with spaces so they look right in a text terminal 💖.

A000100,0053005C005E006D007D008F009E00A000BA00C700EB00F70100
<allOneLine>
1328821153.010    RORUU    1 INVITE    -    sip:192.0.2.10
192.0.2.10:5060    192.0.2.200:56485    sip:192.0.2.10    -
sip:1001@example.com:5060    DL88360fa5fc
DL70dff590c1-1079051554@example.com    S1781761-88    C67651-11
</allOneLine>

And here's the bit-exact version of the actual log entry, Base64 encoded 🔐:

begin-base64 644 clf_record
QTAwMDEwMCwwMDUzMDA1QzAwNUUwMDZEMDA3RDAwOEYwMDlFMDBBMDAwQkEwMEM3MDBF
QjAwRjcwMTAwCjEzMjg4MjExNTMuMDEwCVJPUlVVCTEgSU5WSVRFCS0Jc2lwOjE5Mi4w
LjIuMTAJMTkyLjAuMi4xMDo1MDYwCTE5Mi4wLjIuMjAwOjU2NDg1CXNpcDoxOTIuMC4y
LjEwCS0Jc2lwOjEwMDFAZXhhbXBsZS5jb206NTA2MAlETDg4MzYwZmE1ZmMJREw3MGRm
ZjU5MGMxLTEwNzkwNTE1NTRAZXhhbXBsZS5jb20JUzE3ODE3NjEtODgJQzY3NjUxLTEx
Cg==
====

To recover the unencoded file, you can pass the Base64 text above as input to the following perl script. Redirect the output to a file 🧁:

<CODE BEGINS>

#!/usr/bin/perl
use strict;
my $bdata = "";
use MIME::Base64;
while(<>)
{
      if (/begin-base64 644 clf_record/ .. /-- ==== --/)
      {
          if ( m/^\s*[^\s]+\s*$/)
          {
            $bdata = $bdata . $_;
          }
       }
}
print decode_base64($bdata);

<CODE ENDS>

6. Text Tool Considerations 🛠️

This format was designed so text tools can process logs easily, without needing to understand the indexing format 🧠. You can throw away index lines super fast by checking the first character of each line: index lines always start with an alphabetical character, while field lines always start with a numerical character.

Within a field line, script tools can quickly split the fields at the Tab characters. The first 12 fields are positional. The meaning of any subsequent fields can be figured out by checking the first four characters of the field. Or, if you prefer, you can locate these non-positional fields with a regular expression 💜. For example, the "Contact value" in a request can be found by searching with the perl regex /\t0000,....,([^\t]*)/.

7. Security Considerations 🔒

This document does not introduce any new security considerations beyond those discussed in [RFC6872]. Not me being dramatic, but this part matters 😭.

In the interest of protecting the sensitive information in a SIP CLF file, [RFC6872] notes that values might need to be obfuscated for privacy reasons when SIP CLF files are exchanged between domains 🕵️‍♀️. If a Base64 encoded string contains the non-obfuscated value, then that would also need to be obfuscated before Base64 encoding. Iykyk, and if you skip this, your private data is literally out in the open 💀.

8. Operational Guidance 📦

SIP CLF log files can take up a substantial amount of disk space, depending on the traffic volume at a processing entity and how much information is being logged 📈. So any enterprise using SIP CLF should set up operational procedures for file rollovers, as appropriate to what the organization needs 🗂️.

Listing those operational guidelines in this document is out of scope for this work, so you'll need to handle that part yourself 🙈.

9. IANA Considerations 🌈

This specification establishes a new "Session Initiation Protocol (SIP) Common Log Format (CLF) Parameters" registry 📋. That registry contains two new sub-registries: "SIP CLF Version Values" and "SIP CLF Transport Flag Values". Both sub-registries get initial entries defined by this specification ✨. Adding any new sub-registry to the "Session Initiation Protocol (SIP) Common Log Format (CLF) Parameters" registry is done using the IETF Review registration policy detailed in [RFC5226] 👑.

9.1. SIP CLF Version

This document defines the SIP CLF "Version" field in Section 4.1. IANA has created a registry of Version values called "SIP CLF Version Values" 🦋. Version numbers MUST be incremented for any new SIP CLF protocol specification that changes any part of the SIP CLF record format. Changes include adding or removing fields, or changing the syntax or semantics of existing fields. No cap, this one's a must-do 🔥.

Version numbers must be registered via the Standards Action method described in [RFC5226]. IANA has registered the Versions shown in Table 1 below 💅.

          +------------+----------------------+-----------+
          |   Version  |        FORMAT        | Reference |
          +------------+----------------------+-----------+
          | 0x41 ('A') | Defined in [RFC6873] | [RFC6873] |
          +------------+----------------------+-----------+

           Table 1: IANA-Registered SIP CLF Version Values

9.2. SIP CLF Transport Flag 🚀

Okay so bestie, this part is all about the Transport Flag 🦋. It's the fourth byte in the Flags field of the SIP CLF record, and the way it's formatted and what values it can hold is explained back in Section 4.2. IANA made a registry for these values called "SIP CLF Transport Flag Values" 📋✨

New Transport Flag values must be registered through the IETF Review method from [RFC5226]. Right now IANA has registered the three values in Table 2 below 💅

                +-------+--------------------+-----------+
                | Value | Transport Protocol | Reference |
                +-------+--------------------+-----------+
                |   U   |         UDP        | [RFC6873] |
                |   T   |         TCP        | [RFC6873] |
                |   S   |        SCTP        | [RFC6873] |
                +-------+--------------------+-----------+

              Table 2: IANA-Registered SIP CLF Transport Flag

10. Acknowledgments 🫶

The authors want to give a huge shoutout to Peter Musgrave (the chair of the SIPCLF working group) and Robert Sparks (the assigned Area Director) 👑 Thank you for the support, the guidance, and all the feedback that kept things going 💖

This work also got a major glow-up from the discussions and really valuable input of the SIPCLF working group members 🌸 That includes Brian Trammell, Eric Burger, Cullen Jennings, Benoit Claise, Saverio Niccolini, and Dan Burnett. Special thanks to Hadriel Kaplan, Chris Lonvick, Paul E. Jones, John Elwell, Claudio Allocchio, and Joe Clarke, whose constructive comments, suggestions, and reviews were critical to shaping and polishing this document 🔥

Thanks also to Anders Nygren for the early implementation, the insight, and the reviews of the SIP CLF format 🧚

11. References 📚

11.1. Normative References

  • [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997.
  • [RFC3261] Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson, J., Sparks, R., Handley, M., and E. Schooler, "SIP: Session Initiation Protocol", RFC 3261, June 2002.
  • [RFC6872] Gurbani, V., Burger, E., Anjali, T., Abdelnur, H., and O. Festor, "The Common Log Format (CLF) for the Session Initiation Protocol (SIP): Framework and Information Model", RFC 6872, February 2013.

11.2. Informative References

  • [PEN] IANA, "Private Enterprise Numbers", 2009, <http://www.iana.org/assignments/enterprise-numbers>.
  • [RFC1166] Kirkpatrick, S., Stahl, M., and M. Recker, "Internet numbers", RFC 1166, July 1990.
  • [RFC4475] Sparks, R., Hawrylyshen, A., Johnston, A., Rosenberg, J., and H. Schulzrinne, "Session Initiation Protocol (SIP) Torture Test Messages", RFC 4475, May 2006.
  • [RFC5226] Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 5226, May 2008.
  • [RFC5424] Gerhards, R., "The Syslog Protocol", RFC 5424, March 2009.
  • [RFC5612] Eronen, P. and D. Harrington, "Enterprise Number for Documentation Use", RFC 5612, August 2009.
  • [RFC5737] Arkko, J., Cotton, M., and L. Vegoda, "IPv4 Address Blocks Reserved for Documentation", RFC 5737, January 2010.
  • [RFC5952] Kawamura, S. and M. Kawashima, "A Recommendation for IPv6 Address Text Representation", RFC 5952, August 2010.
  • [UNICODE] The Unicode Consortium, "The Unicode Standard, Version 6.2.0", (Mountain View, CA: ISBN 978-1-936213-07-8), 2012, <http://www.unicode.org/versions/Unicode6.2.0/>.

Authors' Addresses 🎀

   Gonzalo Salgueiro
   Cisco Systems
   7200-12 Kit Creek Road
   Research Triangle Park, NC  27709
   US

   EMail: gsalguei@cisco.com


   Vijay Gurbani
   Bell Labs, Alcatel-Lucent
   1960 Lucent Lane
   Rm 9C-533
   Naperville, IL  60563
   US

   EMail: vkg@bell-labs.com


   Adam Roach
   Mozilla
   Dallas, TX
   US

   EMail: adam@nostrum.com

That's the whole RFC, bestie!! 🌈💅 Every single detail is still in here, and all the Transport Flag values are still wild 😭✨

📦 where this page comes from