RFC 1045
VMTP: Versatile Message Transaction Protocol: Protocol specification
VMTP: Versatile Message Transaction Protocol 🦋
okay bestie, here's the header tea ☕ this is RFC 1045, and the title is VMTP: Versatile Message Transaction Protocol (the subtitle is Protocol Specification) 🎀. It was written by David Cheriton from Stanford University, and it's dated February 1988 📅. The Status section says it's a protocol proposed as a standard for the Internet community 💅, so it's a proposal, not a finished standard. The header doesn't list any RFCs that it obsoletes or updates 🙈.
Status of This Memo ✨
this RFC describes a protocol that's been proposed as a standard for the Internet community. Comments are encouraged 💌 so if you have thoughts, share them! And distribution of this document is unlimited, so go forth and share 🌈
Overview 🌸
okay so this memo specifies the Versatile Message Transaction Protocol (VMTP), Version 0.7 of 19-Feb-88 💖 It's a transport protocol built specifically for the transaction model of communication, the kind you see in remote procedure call (RPC). The full function of VMTP includes support for security, real-time, asynchronous message exchanges, streaming, multicast and idempotency, so the VMTP user level gets a rich selection to pick from ✨
Subsettability is the lowkey iconic part 💅 It lets the VMTP module for a particular client or server get specialized and simplified down to just the services it actually needs. Examples of these simple clients and servers include PROM network bootload programs, network boot servers, data sensors and simple controllers, to name just a few 🧚
Table of Contents 📚
Table of Contents
1. Introduction 1
1.1. Motivation 2
1.1.1. Poor RPC Performance 2
1.1.2. Weak Naming 3
1.1.3. Function Poor 3
1.2. Relation to Other Protocols 4
1.3. Document Overview 5
2. Protocol Overview 6
2.1. Entities, Processes and Principals 7
2.2. Entity Domains 9
2.3. Message Transactions 10
2.4. Request and Response Messages 11
2.5. Reliability 12
2.5.1. Transaction Identifiers 13
2.5.2. Checksum 14
2.5.3. Request and Response Acknowledgment 14
2.5.4. Retransmissions 15
2.5.5. Timeouts 15
2.5.6. Rate Control 18
2.6. Security 19
2.7. Multicast 21
2.8. Real-time Communication 22
2.9. Forwarded Message Transactions 24
2.10. VMTP Management 25
2.11. Streamed Message Transactions 25
2.12. Fault-Tolerant Applications 28
2.13. Packet Groups 29
2.14. Runs of Packet Groups 31
2.15. Byte Order 32
2.16. Minimal VMTP Implementation 33
2.17. Message vs. Procedural Request Handling 33
2.18. Bibliography 34
3. VMTP Packet Formats 37
3.1. Entity Identifier Format 37
3.2. Packet Fields 38
3.3. Request Packet 45
3.4. Response Packet 47
4. Client Protocol Operation 49
4.1. Client State Record Fields 49
4.2. Client Protocol States 51
4.3. State Transition Diagrams 51
4.4. User Interface 52
4.5. Event Processing 53
4.6. Client User-invoked Events 54
4.6.1. Send 54
4.6.2. GetResponse 56
4.7. Packet Arrival 56
4.7.1. Response 58
4.8. Management Operations 61
4.8.1. HandleNoCSR 62
4.9. Timeouts 64
5. Server Protocol Operation 66
5.1. Remote Client State Record Fields 66
5.2. Remote Client Protocol States 66
5.3. State Transition Diagrams 67
5.4. User Interface 69
5.5. Event Processing 70
5.6. Server User-invoked Events 71
5.6.1. Receive 71
5.6.2. Respond 72
5.6.3. Forward 73
5.6.4. Other Functions 74
5.7. Request Packet Arrival 74
5.8. Management Operations 78
5.8.1. HandleRequestNoCSR 79
5.9. Timeouts 82
6. Concluding Remarks 84
I. Standard VMTP Response Codes 85
II. VMTP RPC Presentation Protocol 87
II.1. Request Code Management 87
III. VMTP Management Procedures 89
III.1. Entity Group Management 100
III.2. VMTP Management Digital Signatures 101
IV. VMTP Entity Identifier Domains 102
IV.1. Domain 1 102
IV.2. Domain 3 104
IV.3. Other Domains 105
IV.4. Decentralized Entity Identifier Allocation 105
V. Authentication Domains 107
V.1. Authentication Domain 1 107
V.2. Other Authentication Domains 107
VI. IP Implementation 108
VII. Implementation Notes 109
VII.1. Mapping Data Structures 109
VII.2. Client Data Structures 111
VII.3. Server Data Structures 111
VII.4. Packet Group transmission 112
VII.5. VMTP Management Module 113
VII.6. Timeout Handling 114
VII.7. Timeout Values 114
VII.8. Packet Reception 115
VII.9. Streaming 116
VII.10. Implementation Experience 117
VIII. UNIX 4.3 BSD Kernel Interface for VMTP 118
Index 120
List of Figures
Figure 1-1: Relation to Other Protocols 4
Figure 3-1: Request Packet Format 45
Figure 3-2: Response Packet Format 47
Figure 4-1: Client State Transitions 52
Figure 5-1: Remote Client State Transitions 68
Figure III-1: Authenticator Format 92
Figure VII-1: Mapping Client Identifier to CSR 109
Figure VII-2: Mapping Server Identifiers 110
Figure VII-3: Mapping Group Identifiers 1111. Introduction ✨
the Versatile Message Transaction Protocol (VMTP) is a transport protocol designed to support remote procedure call (RPC) and general transaction-oriented communication 🦋 When we say transaction-oriented, we mean three things:
- Communication is request-response 💌 A client sends a request for a service to a server, the server processes it, and then the server responds. For example, a client might ask for the next page of a file. The transaction ends when the server responds with that next page.
- A transaction starts when a client sends a request to a server, and it ends when the server responds. There are no separate operations for setting up or terminating associations between clients and servers at the transport level 🙅♀️
- The server is free to throw away communication state about a client between transactions, and that won't cause incorrect behavior or failures. Totally chill 🫶
From here on out, the term message transaction (or just transaction) refers to a request-response exchange in the sense described above 💜
VMTP takes care of error detection, retransmission and duplicate suppression. It can also optionally handle security, all to get transport-level end-to-end reliability 🔥
The protocol is built to give you a range of behaviors within the transaction model, including:
- Minimal two packet exchanges for short, simple transactions ⚡
- Streaming of multi-packet requests and responses for efficient data transfer 🌊
- Datagram and multicast communication as an extension of the transaction model 🧚
Example Uses:
- Page-level file access: VMTP is meant to be the transport level for file access, allowing simple, efficient operation on a local network. It's especially good for diskless workstations that access shared network file servers 🖥️💖
- Distributed programming: VMTP is meant to be an efficient transport level protocol for remote procedure call implementations, distributed object-oriented systems, and message-based systems that follow the request-response model 🎀
- Multicast communication with groups of servers to locate a specific object within the group, update a replicated object, synchronize the commitment of a distributed transaction, and so on 🦋
- Distributed real-time control with prioritized message handling, including datagrams, multicast and asynchronous calls 👑
The protocol is designed to run on top of a simple unreliable datagram service, such as the one IP provides 🌈
1.1. Motivation 💅
VMTP was designed to tackle three categories of deficiencies in existing transport protocols within the Internet architecture. For comparison, the key current transport protocol they look at is TCP 🍓
1.1.1. Poor RPC Performance 🐢
First of all, current protocols give poor performance for remote procedure call (RPC) and network file access 😭 This is attributable to three key
1.1.1. Poor RPC Performance (continued) ✨
okay so the bugs come in three flavors 💅
- TCP needs way too many packets for RPC, especially for one-off calls. The connection setup and clear steps add extra packets on top of what VMTP needs to support RPC. the audacity 😭
- TCP is hard to implement, and that's from ten years of real-world experience, not just vibes 🙈 VMTP was designed right alongside its implementation, so it's easy to build and it comes with sensible subsets of its features 🫶
- TCP handles packet loss from overruns badly 💀 VMTP's authors say overruns are the main cause of packet loss in high-performance RPC, and as networks get faster, they'll keep being the main cause. (Older machines and network interfaces can't keep up with newer ones. Also, cheap network interfaces for high-speed networks have limited receive buffering.) it's giving bottleneck energy 🌊
VMTP is built for easy implementation and efficient RPC ✨ On top of that, it has selective retransmission and rate-based flow control, so it handles every one of these issues. slay 👑
1.1.2. Weak Naming 🎀
Next up, current protocols name transport-level endpoints badly, because the names are based on IP addresses 🥲 A TCP endpoint is named by an Internet address and a port identifier. The problem is that this ties the endpoint to one specific host interface, instead of the process-level state that's actually attached to the transport-level endpoint. This naming causes trouble for process migration, mobile hosts and multi-homed hosts. VMTP uses host-address independent names, so those problems are solved 💖
TCP also gives no security or reliability guarantees for its dynamically allocated names 😩 Apart from well-known ports, a (host-addr, port-id)-tuple can change meaning after a reboot following a crash. VMTP uses large identifiers with a guarantee of stability. That means an identifier either never changes meaning, or it stays invalid for a significant time before becoming valid again 🧚
1.1.3. Function Poor 🦋
TCP doesn't support multicast, real-time datagrams or security. Honestly, it only does pair-wise, long-term, streamed reliable exchanges, which is kind of a lot of missing 😭 Multicast matters more and more and is being developed for the Internet (see RFC 966 and 988). A datagram facility that uses the same naming, transmission and reception facilities as the normal transport level is also a huge asset for real-time and parallel applications 🔥 And security is a basic requirement in more and more environments. Security is natural to build at the transport level to get end-to-end security, as opposed to (inter)network level security 🫶 Without transport-level security, a transport protocol can't guarantee its standard service definition when an intruder is around. An intruder can inject packets or modify them while updating the checksum, which makes the "reliable delivery" claim totally meaningless 💀
VMTP provides multicast, real-time datagrams and security, which targets exactly these weaknesses ✨
VMTP is designed with the next generation of communication systems in mind 🌈 These systems are characterized as follows. RPC, page-level file access and other request-response behavior dominate. The communication substrate, both local and wide-area, provides high data rates, low error rates and relatively low delay. Intelligent, high-performance network interfaces are common, and they're actually required to reach performance that comes close to the network's capability. VMTP is also designed to work acceptably with existing networks and network interfaces, so it's not only for the future 💅
1.2. Relation to Other Protocols 🌸
VMTP is a transport protocol that fits into the layered Internet protocol environment. Figure 1-1 shows where VMTP sits in the protocol hierarchy 🦋
+-----------+ +----+ +-----------------+ +------+
|File Access| |Time| |Program Execution| |Naming|... Application
+-----------+ +----+ +-----------------+ +------+ Layer
| | | | |
+-----------+-----------+-------------+------+
|
+------------------+
| RPC Presentation | Presentation
+------------------+ Layer
|
+------+ +--------+
| TCP | | VMTP | Transport
+------+ +--------+ Layer
| |
+-----------------------------------+
| Internet Protocol & ICMP | Internetwork
+-----------------------------------+ Layer
Figure 1-1: Relation to Other ProtocolsThe RPC presentation level isn't currently defined in the Internet protocol suite 🙈 Appendix II defines a proposed RPC presentation level for use with VMTP, and it's assumed for defining the VMTP management procedures. The Application layer protocols listed above also still need to be defined ✨
If internetwork services aren't required, VMTP can be used without the IP layer, sitting directly on top of the network or data link layers 🧚
1.3. Document Overview 📖
The next chapter gives an overview of the protocol, covering naming, message structure, reliability, flow control, streaming, real-time, security, byte-ordering and management 🎀 Chapter 3 describes the VMTP packet formats. Chapter 4 describes the client VMTP protocol operation in terms of pseudo-code for event handling. Chapter 5 describes the server VMTP protocol operation in terms of pseudo-code for event handling. Chapter 6 summarizes the state of the protocol, some remaining issues and expected directions for the future. Appendix I lists some standard Response codes. Appendix II describes the RPC presentation protocol proposed for VMTP and used with the VMTP management procedures. Appendix III lists the VMTP management procedures. Appendix IV proposes initial approaches for handling entity identification for VMTP. Appendix V proposes initial authentication domains for VMTP. Appendix VI provides some details for implementing VMTP on top of IP. Appendix VII provides some suggestions on host implementation of VMTP, focusing on data structures and support functions. Appendix VIII describes a proposed program interface for UNIX 4.3 BSD and its descendants and related systems 💜
2. Protocol Overview 🌈
VMTP provides an efficient, reliable, optionally secure transport service in the message transaction or request-response model 💖 It has these features:
- Host address-independent naming, with support for multiple forms of names for endpoints as well as associated (security) principals. (See Sections 2.1, 2.2, 3.1 and Appendix IV.)
- Multi-packet request and response messages, with a maximum size of 4 megaoctets per message. (Sections 2.3 and 2.14.)
- Selective retransmission. (Section 2.13.) and rate-based flow control to reduce overrun and the cost of overruns. (Section 2.5.6.)
- Secure message transactions with provision for a variety of encryption schemes. (Section 2.6.)
- Multicast message transactions with multiple response messages per request message. (Section 2.7.)
- Support for real-time communication with idempotent message transactions that have minimal server overhead and state (Section 2.5.3), datagram request message transactions with no response, an optional header-only checksum, priority processing of transactions, conditional delivery and preemptive handling of requests (Section 2.8)
- Forwarded message transactions as an optimization for certain forms of nested remote procedure calls or message transactions. (Section 2.9.)
- Multiple outstanding (asynchronous) message transactions per client. (Section 2.11.)
- An integrated management module, defined with a remote procedure call interface on top of VMTP that provides a variety of communication services (Section 2.10.)
- Simple subset implementation for simple clients and simple servers. (Section 2.16.)
This chapter gives an overview of the protocol as an introduction to the basic ideas, and as preparation for the later chapters that describe the packet formats and event processing procedures in detail 🧚
In overview, VMTP provides transport communication between network-visible entities via message transactions ✨ A message transaction consists of a request message sent by the client, or requestor, to a group of server entities, followed by zero or more response messages to the client, with at most one from each server entity. A message has a message control portion and a segment data portion. A message is transmitted as one or more packet groups. A packet group is one or more packets (up to a maximum of 32 packets) grouped by the protocol for acknowledgment, sequencing, selective retransmission and rate control 🔥
Entities and VMTP operations are managed by a VMTP management mechanism, which is accessed through a procedural interface (RPC) implemented on top of VMTP. Specifically, information about a remote entity is obtained and maintained using the Probe VMTP management operation. Acknowledgment information and requests for retransmission are also sent as notify requests to the management module. (In the following description, a reference to an "acknowledgment" of a request or a response refers to a management-level notify operation that is acknowledging the request or response.) 🫶
2.1. Entities, Processes and Principals 🦋
VMTP defines and uses three main types of identifiers: entity identifiers, process identifiers and principal identifiers, each 64-bits in length 💖 Communication happens between network-visible entities, which typically map to, or represent, a message port or procedure invocation. So entities are the VMTP communication endpoints. The process associated with each entity designates the agent behind the communication activity for purposes of resource allocation and management. For example, when a lock is requested on a file, the lock is associated with the process, not the requesting entity. This lets a process use multiple entity identifiers to perform operations without lock conflicts between those entities 🌸 The principal associated with an entity specifies the permissions, security and accounting designation for that entity. The process and principal identifiers are included in VMTP only to make these values available to VMTP users, with the security and efficiency that VMTP provides. Only the entity identifiers are actively used by the protocol 👑
Entity identifiers are required to have three properties:
Uniqueness Each entity identifier is uniquely defined at any given
time. (An entity identifier may be reused over time.)
Stability An entity identifier does not change between valid
meanings without suitable provision for removing
references to the entity identifier. Certain entity
identifiers are strictly stable, (i.e. never changing
meaning), typically being administratively assigned
(although they need not be bound to a valid entity at
all times), often called well-known identifiers. All
other entity identifiers are required to be T-stable,
not change meaning without having remained invalid for
at least a time interval T.The third property is Host address independent 🏠✨ An entity identifier stays unique no matter what host address its current host has. It's also not tied to one Internet host address, which is so iconic. An entity can migrate between hosts, live on a mobile host that keeps changing its Internet addresses, or live on a multi-homed host. It's up to the VMTP implementation to figure out and keep up to date the host addresses of the entities it's talking to 💅
The stability of entity identifiers means an entity identifier stays the same logical communication entity and principal (in the security sense) for as long as it's valid 🔒 So if an entity identifier is authenticated as having the privileges of a given user account, it keeps those privileges as long as it's continuously valid (unless some explicit notice says otherwise). That means a file server doesn't need to fully authenticate the entity on every single file access request, which is a major efficiency moment 🙌 With T-stable identifiers, checking the validity of an entity identifier every period less than T seconds catches a change in its validity.
A group of entities can form an entity group, which is a set of zero or more entities identified by a single entity identifier 👯 For example, you could have one entity identifier that stands for the group of name servers. An entity identifier for an entity group comes from the same name space as regular entity identifiers. Single entity identifiers are flagged with a bit saying they're known to identify at most one entity. Each entity identifier also includes other standard type flags. One flag shows whether the identifier is an alias for an entity in another domain (See Section 2.2 below.). Another flag shows, for an entity group identifier, whether the group is restricted or not. A restricted group is one where an entity can only be added by another entity with group management authorization 🚫 In an unrestricted group, an entity is allowed to add itself 🙋♀️ If an entity identifier doesn't represent a group, a type bit shows whether the entity uses big-endian or little-endian data representation (matching Motorola 680X0 and VAX byte orders, respectively) 🤓 More on the format of entity identifiers is in Section 3.1 and Appendix IV.
An entity identifier identifies a Client, a Server, or a group of Servers <1> 🎯 A Client is always identified by a T-stable identifier. A server or group of servers may be identified by a T-stable identifier (group or single entity) or by a strictly stable (statically assigned) entity group identifier. The same T-stable identifier can identify a Client and a Server at the same time, as long as both are logically associated with the same entity 🪞 The state needed for reliable, secure communication between entities is kept in client state records (CSRs), which hold the entity identifier of the Client, its principal, its current or next transaction identifier, and so on 📝
<1> Terms such as Client, Server, Request, Response, etc. are capitalized in this document when they refer to their specific meaning in VMTP. 💖
2.2. Entity Domains ✨
An entity domain is an administration (or an administration mechanism) that guarantees the three required entity identifier properties of uniqueness, stability and host address independence for the entities it administers 👑 Basically, entity identifiers are only guaranteed to be unique and stable inside one entity domain. For example, the set of all Internet hosts could work as one domain. Separately, the set of hosts local to one autonomous network could work as a different domain 🌐 Each entity domain is identified by an entity domain identifier, Domain. Only entities in the same domain can communicate directly via VMTP. Hosts and entities can participate in multiple entity domains at the same time, possibly with different entity identifiers 🦋 For example, a file server might participate in multiple entity domains so it can provide file service to each one. Every entity domain specifies the algorithms for allocating, interpreting and mapping entity identifiers.
Domains are needed because it doesn't seem feasible to specify one universal VMTP entity identification administration that covers all entities for all time 😭 Domains limit how many entities need managing to keep the uniqueness and stability of the entity name space. Domains can also separate entities of different security levels 🔐 For instance, allocating an unclassified entity identifier can't conflict with secret-level entity identifiers, because the former is only interpreted in the unclassified domain, which is disjoint from the secret domain.
The plan is to have a small number of domains. In particular, there should be one (or a few) domains per installation "type", rather than per installation 🍓 For example, the Internet is expected to use one domain per security level, which means at most 8 different domains. Cluster-based internetwork architectures (those with a local cluster protocol distinct from the wide-area protocol) may use one domain for local use and one for wide-area use 🧚
Additional details on the specification of specific domains are in Appendix IV.
2.3. Message Transactions 💌
The message transaction is the unit of interaction between a Client that starts the transaction and one or more Servers. A message transaction starts with a request message generated by a Client 🚀 At the service interface, a Server gets involved with a transaction by receiving and accepting the request. A Server stops being involved with a transaction by sending a response message. In a group message transaction, the server entity the Client designated corresponds to a group of entities. In this case, each server in the group receives a copy of the request 👯♀️ From the Client's point of view, the transaction ends when it receives the response message or, in the case of a group message transaction, when it receives the last response message. Since it's normally impractical to figure out when the last response message has arrived, VMTP terminates the current transaction when the next transaction is initiated 🙈
Within an entity domain, a transaction is uniquely identified by the tuple (Client, Transaction, ForwardCount), where Transaction is a 32-bit number and ForwardCount is a 4-bit value 🔢 A Client uses monotonically increasing Transaction identifiers for new message transactions. Normally, the next higher transaction number, modulo 2**32, is used for the next message transaction, although there are cases where it skips a small range of Transaction identifiers (see the description of the STI control flag). ForwardCount is used when a message transaction is forwarded, and it's zero otherwise.
A Client generates a stream of message transactions with increasing transaction identifiers, aimed at a variety of Servers 🌈 A Client has a transaction outstanding if it has invoked a message transaction but hasn't received the last Response (or possibly any Response). Normally, a Client only has one transaction outstanding at a time. However, VMTP lets a Client have multiple message transactions outstanding at once, which supports streamed, asynchronous remote procedure call invocations 🎀 VMTP also supports nested calls, where for example procedure A calls procedure B, which calls procedure C, each on a separate host. Each call has a different client entity identifier, but they're all identified with the same process and principal. Main character energy 👑
2.4. Request and Response Messages 📦
A message transaction consists of a request message and one or more Response messages. A message is structured as a message control block (MCB) and segment data, passed as parameters, as shown below.
+-----------------------+
| Message Control Block |
+-----------------------+
+-----------------------------------+
| segment data |
+-----------------------------------+In the request message, the MCB specifies control information about the request plus an optional data segment. The MCB has the following format:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ ServerEntityId (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Flags | RequestCode |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ CoresidentEntity (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> User Data (12 octets) <
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MsgDelivery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SegmentSize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The ServerEntityId is the entity the Request MCB is sent to (or was sent to, when it's being received) 📬 The Flags show various options for request and response handling, plus whether the CoresidentEntity, MsgDelivery and SegmentSize fields are in use. The RequestCode field specifies the type of Request. It's basically like the packet type field in Ethernet, acting as a switch for higher-level protocols 🔀 The CoresidentEntity field, if used, picks out a subgroup of the ServerEntityId group that the Request should be routed to: the members co-resident with the specified entity (or entity group). The main intended use is to name the manager for a particular service that's co-resident with a particular entity. You do this by putting the well-known entity group identifier for the service manager in the ServerEntityId field, and the identifier for the entity in the CoresidentEntity field 🧩 The next 12 octets are user- or application-specified ✨
The MsgDelivery field is optionally used by the RPC or user level to specify which portions of the segment data to transmit and, on reception, which portions were received. It gives the client and server optional access to, and responsibility for, a simple selective transmission and reception facility 🎯 For example, a client can request retransmission of just the portions of the segment it failed to receive as part of the original Response. The primary intended use is to support highly efficient multi-packet reading from a file server. Using user-level selective retransmission through the MsgDelivery field, the file server VMTP module doesn't need to save multi-packet Responses for retransmission. Retransmissions, when needed, are handled directly from the file server buffers 🗂️
The SegmentSize field indicates the size of the data segment, if one is present. The CoresidentEntity, MsgDelivery and SegmentSize fields can be used as additional user data if they aren't otherwise used 🎁
The Flags field gives the user level a simple way to tell the VMTP module which VMTP options it's using, and lets VMTP modules tell each other the same thing 🤝 Usually the options are fixed for each remote procedure, so an RPC mechanism using VMTP can treat the Flags as an integral part of the RequestCode field, for demultiplexing to the correct stub 🧚♀️
A Response message control block follows the same format, except the Response goes from the Server to the Client and there is no Coresident Entity field 💅 (which means it carries 20 octets of user data, bestie 🧚).
2.5. Reliability 🛡️
VMTP gives you reliable, sequenced delivery of request and response messages, plus a few variants like unreliable datagram requests ✨. The reliability toolkit includes transaction identifiers, checksums, positive acknowledgment of messages, and timeouts with retransmission of lost packets 🔁.
2.5.1. Transaction Identifiers 🆔
Each message transaction is uniquely identified by the pair (Client, Transaction) 🎀. (We'll get to the ForwardCount field in Section 2.9, so hold that thought 🙈.) The 32-bit transaction identifier starts at a random value when the Client entity is created or gets its entity identifier, and it goes up by one at the end of each message transaction 📈. Every Response with the same (Client, Transaction) pair is linked to that Request 💜.
The transaction identifier is also how the Server spots duplicates 👀. The Server keeps a state record for each Client it's currently processing a Request for, keyed by (Client, Transaction) 🗂️. A Request with the same (Client, Transaction) pair gets thrown out as a duplicate 🗑️. (The ForwardCount field has to match too.) Usually the record sticks around for a while after the Response goes out, so the Server can filter any late duplicates of that Request 🧹. When a Request shows up and the Server has no state record for the sending Client, the Server has three options 🤔:
- The Server may send a Probe request, which is a simple query, to the VMTP management module associated with the requesting Client. That tells it the Client's current Transaction identifier (plus other info). The Server can then set up a new state record from that and process the Request like normal 🔍.
- The Server may decide the Request must be new, because it has no state record for this Client, as long as it keeps these state records for the maximum packet lifetime in the network (plus the maximum VMTP retransmission time) and it hasn't rebooted during that window ⏳. The logic is that if the Request weren't new, either it would have outlived the maximum packet lifetime, or the Server would still have a state record for the Client 🧠.
- The Server may know the Request is idempotent, or can be safely redone, so it doesn't need to care whether it's a duplicate 🙌. For example, a request for the current time can just get the current time back, no worries about duplicates 🕰️. The Response gets dropped at the Client if it's no longer of interest 🫶.
2.5.2. Checksum 🧮
Every VMTP packet has a checksum so the receiver can catch corrupted packets, separate from the lower-level checks 🔍. The checksum field is 32 bits, which is way stronger than the standard 16-bit IP checksum (especially paired with an improved checksum algorithm) 💪. Big packets, high packet rates, and general network characteristics we expect down the road make a beefier checksum a must 🚀.
The checksum normally covers both the VMTP header and the segment data 📦. Optionally (for real-time applications), the checksum can cover only the packet header, which is signaled by the HCO control bit being set in the header ⚡. The checksum field goes at the end of the packet so it can be calculated as part of a software copy, or as part of a hardware transmission or reception pipeline, which is where next-gen network interfaces are headed 🏎️. Heads up: the number of header and data octets is an integral multiple of 8, because VMTP requires the segment data to be padded to a multiple of 64 bits 🧩. The checksum field gets appended after any padding 🎀. The actual algorithm is in Section 3.2 📖.
A zero checksum field means no checksum was sent with the packet 0️⃣. VMTP can go without a checksum only when there's a host-to-host error detection mechanism in place and the VMTP security facility isn't being used 🔐. For instance, you could lean on the Ethernet CRC if communication is limited to hosts on the same Ethernet and you trust the network interfaces enough 🌈.
2.5.3. Request and Response Acknowledgment 💌
VMTP assumes an unreliable datagram network and internetwork interface 🌐. To guarantee that Requests and Responses get delivered, VMTP uses positive acknowledgments, retransmissions, and timeouts ⏰.
A Request is normally acknowledged when you receive a Response associated with it, meaning one with the same (Client, Transaction) 🙋. With streamed message transactions, a Request can also be acknowledged by a later Response that acknowledges earlier Requests along with the transaction it names directly 🌊. A Response can be explicitly acknowledged by a NotifyVmtpServer operation requested of the manager for the Server 📣. When streaming, this is a cumulative acknowledgment, so it also acknowledges every Response with a lower transaction identifier 📚. With non-streamed communication, a later Request from the same Client acknowledges Responses to all previous message transactions 🧾. The reason is that either the client got a Response, or it's no longer interested in Responses to those earlier transactions 🫠. Finally, a client response timeout at the server acknowledges a Response, at least in the sense that the server doesn't have to be ready to retransmit that Response later 🕊️. Just so you know, there's no end-to-end guarantee that the Response actually reached the client at the application level 😬.
2.5.4. Retransmissions 🔁
Generally, a Request or Response gets retransmitted periodically until it's acknowledged as described above, up to some maximum number of retransmissions 🎯. VMTP uses the parameters RequestRetries(Server) and ResponseRetries(Client) to set how many retransmissions the server and client each try before giving up 🙅♀️. The authors suggest a value of 5 for both parameters, based on their VMTP experience and Internet packet loss 📊. Smaller values (like 3) can work in low loss environments where you need to spot failed hosts or channels fast 🏃♀️. Larger values should be used in high loss environments where transport-level persistence matters 🧗♀️.
In a low loss environment, a retransmission only includes the MCB and not the segment data of the Request or Response, so it's just one short packet 📦✨. The intended recipient of the retransmission can ask for selective retransmission of all or part of the segment data whenever it's needed 🎯. The selective retransmission mechanism is covered in Section 2.13 📖.
If a Response is specified as idempotent, the Response is neither retransmitted nor stored for retransmission 🚫. Instead, the Client has to retransmit the Request to effectively get the Response retransmitted 🔄. The server VMTP module handles retransmissions of the Request by passing the Request on to the server again, so it can regenerate the Response by redoing the operation, rather than saving a copy of the Response 🪄. Only Request packets for the last transaction from this client get passed on this way; older Request packets from this client are discarded as delayed duplicates 🗑️. If a Response is not idempotent, the VMTP module must make sure it has a copy of the Response for retransmission 💾. It can do that either by making a copy of the Response (physically or copy-on-write), or by stopping the Server from continuing until the Response is acknowledged 🛑.
2.5.5. Timeouts ⏲️
Each Client with an outstanding transaction gets one client timer 🎀. Likewise, each Client transaction that's "active" at the server gets one server timer, meaning there's a transaction record for a Request from that Client 💜.
When the client sends a new Request (without streaming), the client timer gets set to roughly the time the Response is expected back 🕐. On timeout, the Request is retransmitted with the APG (Acknowledge Packet Group) bit set 🚩. The timeout then resets to the expected roundtrip time to the Server, since an acknowledgment should come back right away unless a Response has already been sent ⚡. The Request can also be retransmitted when the client gets a VMTP management operation saying that selected portions of the Request's message segment need retransmitting 🧩. With streaming, the timeout applies to the oldest outstanding message transaction in the run of outstanding transactions 🌊. Without streaming, there's only one transaction in the run, so it goes back to the previous situation 🔙. After the first packet of a Response arrives, the Client resets the timeout to the time it expects before the next packet in the Response packet group arrives, assuming it's a multi-packet Response 📬. If it's not a multi-packet Response, the timer stops ⏹️. Last but not least, the client timer also handles timing out while waiting for the second and later Responses to a multicast Request 📡.
The client timer gets set to four different values at different times 🕰️:
TC1(Server) The expected time required to receive a Response from
the Server. Set on initial Request transmission plus
after its management module receives a NotifyVmtpClient
operation, acknowledging the Request.
TC2(Server) The estimated round trip delay between the client and
the server. Set when retransmitting after receiving no
Response for TC1(Server) time and retransmitting the
Request with the APG bit set.
TC3(Server) The estimated maximum expected interpacket time for
multi-packet Responses from the Server. Set when
waiting for subsequent Response packets within a packet
group before timing out.
TC4 The time to wait for additional Responses to a group
Request after the first Response is received. This is
specified by the user level.Here's how those values get picked 🧁. TC1 can be set to TC2 plus a constant, which reflects the time within which most servers respond to most requests 💅. For example, various measurements of VMTP usage at Stanford showed that 90 percent of the servers respond in less than 200 milliseconds 📈. Setting TC1 to TC2 + 200 means most Requests get a Response before timing out, and also that the retransmission overhead for long running transactions is negligible 🙌. A sophisticated implementation can make the estimate of TC1 more specific to the Server 🧠.
TC2 can be estimated by measuring the time from when a Probe request is sent to the Server until a response comes back ⏱️. TC2 can also be measured as the time between sending a Request with the APG bit set and receiving a management operation that acknowledges receipt of the Request 🎯.
When the Server is an entity group, TC1 and TC2 should be the largest of the values for the group members that are expected to respond 👯♀️. You can find this out by probing the group on first use (and using the values from the last responses to arrive) 🔎. Or, you can fall back on default values 🫶.
TC3 starts out at 10 times the transmission time for the maximum transmission unit (MTU) to be used for the Response 🧮. A sophisticated implementation can record TC3 per Server and refine the estimate based on measurements of actual interpacket gaps 📏. A tighter estimate of TC3 only speeds up the reaction when a packet is lost in a packet group, but it costs some unnecessary retransmissions if the estimate gets too tight ⚠️.
The server timer, one per active Client, takes on the following values:
TS1(Client) The estimated maximum expected interpacket time. Set
when waiting for subsequent Request packets within a
packet group before timing out.
TS2(Client) The time to wait to hear from a client before
terminating the server processing of a Request. This
limits the time spent processing orphan calls, as well
as limiting how out of date the server's record of the
Client state can be. In particular, TS2 should be
significantly less than the minimum time within which it
is reasonable to reuse a transaction identifier.
TS3(Client) Estimated roundtrip time to the Client,
TS4(Client) The time to wait after sending a Response (or last
hearing from a client) before discarding the state
associated with the Request which allows it to filter
duplicate Request packets and regenerate the Response.
TS5(Client) The time to wait for an acknowledgment after sending a
Response before retransmitting the Response, or giving
up (after some number of retransmissions).Okay, so the server has five timers per Client, and they're basically the server's version of a stopwatch. ⏱️💖 TS1 is the one that watches for the next Request packet inside a packet group. TS2 is the "okay, I'm done waiting on you" timer, and it stops the server from wasting time on orphan calls. TS3 is the estimated roundtrip time to the Client. TS4 is how long the server remembers the Client's state after a Response, so duplicate Requests can get filtered and the Response can be regenerated. TS5 is the wait for an acknowledgment before the Response gets resent, or the server gives up after some number of retransmissions. 🫶
TS1 is set the same as TC3. 🎀
The suggested value for TS2 is TC1 + 3*TC2 for this server, giving the Client time to timeout waiting for a Response and retransmit 3 Request packets, asking for acknowledgments. Bestie, this is giving "enough time for the Client to panic, retry three times, and ask for acks" energy. 😭✨
TS3 is estimated the same as TC1 except that refinements to the estimate use measurements of the Response-to-acknowledgment times. 🦋
In the general case, TS4 is set large enough so that a Client issuing a series of closely-spaced Requests to the same Server reuses the same state record at the Server end and thus does not incur the overhead of recreating this state. (The Server can recreate the state for a Client by performing a Probe on the Client to get the needed information.) It should also be set low enough so that the transaction identifier cannot wrap around and so that the Server does not run out of CSR's. We suggest a value in the range of 500 milliseconds. However, if the Server accepts non-idempotent Requests from this Client without doing a Probe on the Client, the TS4 value for this CSR is set to at least 4 times the maximum packet lifetime. 🌸 Main character energy: the state should stick around long enough to be reused, but not so long that the transaction identifier wraps around or the Server runs out of CSR's. 💅
TS5 is TS3 plus the expected time for transmission and reception of the Response. We suggest that the latter be calculated as 3 times the transmission time for the Response data, allowing time for reception, processing and transmission of an acknowledgment at the Client end. A sophisticated implementation may refine this estimate further over time by timing acknowledgments to Responses. 🧚
2.5.6. Rate Control
VMTP is designed to deal with the present and future problem of packet overruns. We expect overruns to be the major cause of dropped packets in the future. A client is expected to estimate and adjust the interpacket gap times so as to not overrun a server or intermediate nodes. The selective retransmission mechanism allows the server to indicate that it is being overrun (or some intermediate point is being overrun). For example, if the server requests retransmission of every Kth block, the client should assume overrun is taking place and increase the interpacket gap times. The client passes the server an indication of the interpacket gap desired for a response. The client may have to increase the interval because packets are being dropped by an intermediate gateway or bridge, even though it can handle a higher rate. A conservative policy is to increase the interpacket gap whenever a packet is lost as part of a multi-packet packet group. 🚦✨
The provision of selective retransmission allows the rate of the client and the server to "push up" against the maximum rate (and thus lose packets) without significant penalty. That is, every time that packet transmission exceeds the rate of the channel or receiver, the recovery cost to retransmit the dropped packets is generally far less than retransmitting from the first dropped packet. 🔥
The interpacket gap is expressed in 1/32nd's of the MTU packet transmission time. The minimum interpacket gap is 0 and the maximum gap that can be described in the protocol is 8 packet times. This places a limit on the slowest receivers that can be efficiently used on a network, at least those handling multi-packet Requests and Responses. This scheme also limits the granularity of adjustment. However, the granularity is relative to the speed of the network, as opposed to an absolute time. For entities on different networks of significantly different speed, we assume the interconnecting gateways can buffer packets to compensate<2>. With different network speeds and intermediary nodes subject to packet loss, a node must adjust the interpacket gap based on packet loss. The interpacket gap parameter may be of limited use. 🦋
<2> Gateways must also employ techniques to preserve or intelligently modify (if appropriate) the interpacket gaps. In particular, they must be sure not to arbitrarily remove interpacket gaps as a result of their forwarding of packets.
2.6. Security
VMTP provides an (optional) secure mode that protects against the usual security threats of peeking, impostoring, message tampering and replays. Secure VMTP must be used to guarantee any of the transport-level reliability properties unless it is guaranteed that there are no intruders or agents that can modify packets and update the packet checksums. That is, non-secure VMTP provides no guarantees in the presence of an intelligent intruder. 🔐💀
The design closely follows that described by Birrell [1]. Authenticated information about a remote entity, including an encryption/decryption key, is obtained and maintained using a VMTP management operation, the authenticated Probe operation, which is executed as a non-secure VMTP message transaction. If a server receives a secure Request for which the server has no entity state, it sends a Probe request to the VMTP management module of the client, "challenging" it to provide an authenticator that both authenticates the client as being associated with a particular principal as well as providing a key for encryption/decryption. The principal can include a real and effective principal, as used in UNIX <3>. Namely, the real principal is the principal on whose behalf the Request is being performed whereas the effective principal is the principal of the module invoking the request or remote procedure call. 🎀
<3> Principal group membership must be obtained, if needed, by a higher level protocol.
Peeking is prevented by encrypting every Request and Response packet with a working Key that is shared between Client and Server. Impostoring and replays are detected by comparing the Transaction identifier with that stored in the corresponding entity state record (which is created and updated by VMTP as needed). Message tampering is detected by encryption of the packet including the Checksum field. An intruder cannot update the checksum after modifying the packet without knowing the Key. The cost of fully encrypting a packet is close to the cost of generating a cryptographic checksum (and of course, encryption is needed in the general case), so there is no explicit provision for cryptographic checksum without packet encryption. 🙈🔒
A Client determines the Principal of the Server and acquires an authenticator for this Server and Principal using a higher level protocol. The Server cannot decrypt the authenticator or the Request packets unless it is in fact the Principal expected by the Client. 🧚
An encrypted VMTP packet is flagged by the EPG bit in the VMTP packet header. Thus, encrypted packets are easily detected and demultiplexed from unencrypted packets. An encrypted VMTP packet is entirely encrypted except for the Client, Version, Domain, Length and Packet Flags fields at the beginning of the packet. Client identifiers can be assigned, changed and used to have no real meaning to an intruder or to only communicate public information (such as the host Internet address). They are otherwise just a random means of identification and demultiplexing and do not therefore divulge any sensitive information. Further secure measures must be taken at the network or data link levels if this information or traffic behavior is considered sensitive. 👑
VMTP provides multiple authentication domains as well as an encryption qualifier to accommodate different encryption algorithms and their corresponding security/performance trade-offs. (See Appendix V.) A separate key distribution and authentication protocol is required to handle generation and distribution of authenticators and keys. This protocol can be implemented on top of VMTP and can closely follow the Birrell design as well. 🌈
Security is optional in the sense that messages may be secure or non-secure, even between consecutive message transactions from the same client. It is also optional in that VMTP clients and servers are not required to implement secure VMTP (although they are required to respond intelligently to attempts to use secure VMTP). At worst, a Client may fail to communicate with a Server if the Server insists on secure communication and the Client does not implement security or vice versa. However, a failure to communicate in this case is necessary from a security standpoint. 🫶
2.7. Multicast
The Server entity identifier in a message transaction can identify an entity group, in which case the Request is multicast to every Entity in this group (on a best-efforts basis). The Request is retransmitted until at least one Response is received (or an error timeout occurs) unless it is a datagram Request. The Client can receive multiple Responses to the Request. 📣
The VMTP service interface does not directly provide reliable multicast because it is expensive to provide, rarely needed by applications, and can be implemented by applications using the multiple Response feature. However, the protocol itself is adequate for reliable multicast using positive acknowledgments. In particular, a sophisticated Client implementation could maintain a list of members for each entity group of interest and retransmit the Request until acknowledged by all members. No modifications are required to the Server implementations. 🦋
VMTP supports a simple form of subgroup addressing. If the CRE bit is set in a Request, the Request is delivered to the subgroup of entities in the Server group that are co-resident with one or more entities in the group (or individual entity) identified by the CoresidentEntity field of the Request. This is commonly used to send to the manager entity for a particular entity, where Server specifies the group of such managers. Co-resident means "using the same VMTP module", and logically on the same network host. In particular, a Probe request can be sent to the particular VMTP management module for an entity by specifying the VMTP management group as the Server and the entity in question as the CoResidentEntity. 🌸
As an experimental aspect of the protocol, VMTP supports the Server sending a group Response which is sent to the Client as well as members of the destination group of Servers to which the original Request was sent. The MDG bit indicates whether the Client is a member of this group, allowing the Server module to determine whether separately addressed packet groups are required to send the Response to both the Client and the Server group. Normally, a Server accepts a group Response only if it has received the Request and not yet responded to the Client. Also, the Server must explicitly indicate it wants to accept group Responses. Logically, this facility is analogous to responding to a mail message sent to a distribution list by sending a copy of the Response to the distribution list. 💌✨
2.8. Real-time Communication 🎀
Okay bestie, VMTP has three ways to handle real-time stuff on top of its normal features 💅, which is why it works for a whole bunch of real-time apps. First, every Request and Response carries a priority that decides how urgently it gets handled ✨ The levels are meant to roughly line up with:
- urgent/emergency.
- important
- normal
- background.
Each level can have extra gradations inside it. What each priority actually does is up to the host itself, like how it gets assigned host processing priorities 🧚
Second, datagram Requests let the Client send to an entity or entity group using the usual VMTP naming, transmission and delivery stuff, but with no blocking, no retransmissions and no acknowledgment 🦋 (The Client can still ask for acknowledgment with the APG bit, though the Server doesn't expect missing pieces of a multi-packet datagram Request to be resent, even if some never show up.) A datagram Request in non-streamed mode supersedes all previous Requests from the same Client 👑. In stream mode it gets queued (if needed) after the earlier datagram Requests on the same stream. (See Section 2.11.)
Finally, VMTP has a few control bit flags that tweak how Requests and Responses get handled for real-time needs 🌈 First up is the conditional message delivery (CMD) flag, which makes a Request get discarded if the recipient isn't waiting for it when it arrives. The same goes for the Response. This lets a client send a Request that only works if the server can process it right away 🫶 Next is the header checksum only (HCO) flag, which means the checksum was calculated only on the VMTP header and not on the data segment. Apps like voice and video can skip the checksum overhead on data that doesn't care much about typical bit errors, and they still keep protection on the header 🎤 Last is the No Retransmission (NRT) flag, which tells the recipient of a message not to ask for a retransmission if part of it is missing. Instead it should either use whatever came in or throw it away 💀
None of these facilities introduce new protocol states, and that's the best part 🙌 In the normal case, the total processing overhead is just a bit flag test for CMD, HCO or NRT, plus assigning priority when packets are sent and received. (In fact, CMD and NRT are not tested in the normal case.) The extra code complexity is minimal. The authors feel the overhead for these real-time facilities is minimal too, and that they're both important and good enough for a wide class of real-time apps 😤✨
Several of VMTP's normal facilities are also useful for real-time apps 💜 First, multicast is great for distributed, replicated (fault-tolerant) real-time apps, since it allows efficient state query and update for things like sensors and control state. Second, the DGM or idempotent flag for Responses has some real-time perks: a Request gets redone to fetch the latest values when the Response is lost, instead of just returning the old values again ⏰ Think about asking for the current time of day. Handling that request idempotently gives you a more accurate time when a retransmission is needed. Finally, the request-response semantics (when there's no streaming) mean each new Request from a Client ends the previous message transactions from that Client, if any. That gives you the "most recent is most important" handling that most real-time apps need 🔥
Overall, a key design goal of VMTP was to provide an efficient general-purpose transport protocol with the features real-time communication needs 🦋 Whether that goal was actually achieved still needs more experience to figure out 🙈
2.9. Forwarded Message Transactions 🎁
A Server can invoke another Server to handle a Request 💌 It's pretty common for that second Server call to be the last thing the first Server does while handling the Request. For example, the original Server might mostly just pick a process to handle the Request. Or it might only check the authorization on the Request. In RPC terms, a nested remote procedure call can be the last action in the remote procedure, with the return parameters exactly matching the nested call's. (This is kind of like tail recursion 🌸)
To optimize this case, VMTP has a Forward operation that lets the server send the nested Request to the other server and have that other server respond directly to the Client ✨
If the message transaction being forwarded wasn't multicast, wasn't secure, or the two Servers are the same principal, and the ForwardCount of the Request is less than the maximum forward count of 15, then the Forward operation works like this: the Server sends a Request onto the next Server. The forwarded Request keeps the same Client and Transaction as the original Request, but with a ForwardCount one greater than what the Request had when the Client sent it 🧚 In this case the new Server responds directly to the Client. Here's what a forwarded Request looks like:
+---------+ Request +----------+
| Client +---------------->| Server 1 |
+---------+ +----------+
^ |
| | forwarded Request
| V
| Response +----------+
+----------------------| Server 2 |
+----------+If the message transaction doesn't meet those requirements, the Server's VMTP module issues a nested call and just maps the returned Response to a Response to the original Request, with no more Server-level processing 🙃 In this case the only optimization over a user-level nested call is one fewer VMTP service operation, since the VMTP module handles the return to the invoking call directly. The Server can also use this form of forwarding when the Request is part of a stream of message transactions. Otherwise it must wait until the forwarded message transaction completes before moving on to the next message transactions in the stream.
Implementing the user-level Forward operation is optional, depending on whether the server modules need this facility 🌈 Handling an incoming forwarded Request is a small tweak on handling a normal incoming Request. In particular, the ForwardCount field only needs to be examined when the Transaction of the Request matches that of the last message transaction received from the Client. So the extra complexity in the VMTP module for the required forwarding support is minimal. The complexity lives in the highly optimized user-level Forward primitive, and that part is optional 💖
2.10. VMTP Management 🛠️
VMTP management covers operations for creating, deleting, modifying and querying VMTP entities and entity groups 🧸 It's logically implemented by a VMTP management server module, which gets invoked using a message transaction addressed to the Server, VMTP_MANAGER_GROUP, a well-known group entity identifier. This works together with the Coresident Entity mechanism introduced in Section 2.7. A particular Request can address the local module, the module managing a particular entity, the set of modules managing the entities in a specific group, or all management modules, whichever fits best 🎯
The VMTP management procedures are specified in Appendix III 📚
2.11. Streamed Message Transactions 🌊
Streamed message transactions are when a Client starts two or more message transactions before it has received the response to the first one 🚀 Each transaction still gets processed and responded to in order, but asynchronously relative to when they were started 💅 A Client streams by sending them as one run of message transactions, which is a sequence with the same Client and Server and consecutive Transaction identifiers. All but the first and last Requests and Responses are flagged with the NSR (Not Start Run) and NER (Not End Run) control bits. (So the first Request and Response don't have the NSR set, and the last Request and Response don't have the NER bit set ✨) The message transactions in a run use consecutive transaction identifiers, except when the STI bit <4> is used in one. In that case, the transaction identifier for the next message transaction is 256 greater, rather than 1 🔢
<4> The STI bit is used by the Client to effectively allocate 255 transaction identifiers for use by the Server in returning a large Response or stream of Responses.The Client keeps a record for each outstanding transaction until it gets a Response or times out in error 📝 That record has the info needed to retransmit the Request. On a retransmission timeout, the client retransmits the last Request it hasn't received a Response for, the same way it does with non-streamed communication. (I.e. there only needs to be one timeout for all the outstanding message transactions associated with a single client ⏱️)
The consecutive transaction identifiers in a run are used as sequence numbers for error control 🧠 The Server handles each message transaction in the sequence set by its transaction identifier. When it receives a message transaction that isn't marked as the beginning of a run, it checks that it previously received a message transaction with the predecessor transaction identifier. That's either 1 less than the current one, or 256 less if the previous one had the STI bit set. If not, the Server sends a NotifyVmtpClient operation to the Client's manager, indicating either (1) the first message transaction wasn't fully received, or else (2) it has no record of the last one received 🚨 If the NRT control flag is set, the Server doesn't wait for or expect retransmission and just goes ahead and handles this Request. This flag is mainly used when datagram Requests are part of a stream of message transactions. If NRT wasn't specified, the Client must retransmit from the first message transaction not fully received (either entirely or in part) before the Server can continue handling this run of Requests, or else restart the run of message transactions 🔁
The Client expects to get the Responses in a consecutive sequence, and it uses the Transaction identifier to spot missing Responses 👀 So the Server must return Responses in sequence, except possibly for some gaps. Here's how gaps work. The Server can use the PGcount field in a Response to say how many consecutive previous Responses this one corresponds to, up to a maximum of 255 previous Responses <5>. So a Response with Transaction identifier 46 and PGcount 3 represents Responses 43, 44, 45 and 46 🎀 This lets the Server skip sending Responses to Requests that don't need one, batching everything into one. It also helps the Server keep strictly consecutive sequencing when the Client has skipped 256 Transaction identifiers using the STI bit and the Server doesn't have that many Responses to return 💖
If the Client gets a Response that isn't consecutive, it retransmits the Request(s) for which the Response(s) are missing, unless of course the corresponding Requests were sent as datagrams 🙏 Also, the Client should wait at the end of a run of message transactions for the last one to complete 🫶
When a Server gets a Request with the NSR bit clear and a higher
transaction identifier than it currently has for the Client, it terminates all processing and discards Responses associated with the previous Requests. 💅 So basically, starting a new run literally aborts the old stream, even if the Server was still in the middle of handling the previous run. Iconic, but a lot. 😭
Mixing datagram and normal Requests in one stream of message transactions can get messy under packet loss, especially with the NRT bit in play. 🫠 It is recommended that a run be all one type, so either all normal or all datagrams, to avoid problems. Also, when a Server forwards a Request that is part of a run, it must suspend further processing of the later Requests until the forwarded Request is handled, so the order of processing stays intact. 🎀 The simplest way to do this is a real nested call when forwarding with streamed message transactions.
Flow control for streamed message transactions works through rate control at the Client, plus whether the management notify operations that signal overrunning arrive or don't. 🌸 A Client must reduce the number of outstanding message transactions at the Server when it receives a NotifyVmtpServer operation with the MSGTRANS_OVERFLOW ResponseCode. The transact parameter tells you the last packet group that was accepted. 🦋
<5> PGcount actually corresponds to packet groups which are described in Section 2.13. This (simplified) description is accurate when there is one Request or Response per packet group.
Implementing multiple outstanding message transactions means being able to record, time out and buffer them at both the Client end and the Server end. 🧚 This facility is optional for both the Client and the Server. Client systems with heavy-weight processes and high network access cost are the most likely to benefit from it. 💜 Servers that serve a wide variety of client machines should implement streaming to accommodate these types of clients.
2.12. Fault-Tolerant Applications
One approach to fault-tolerant systems is to keep a log of every message sent at each node, then replay those messages at a node when it fails, after restarting it from the last checkpoint <6>. 🔥 As an experimental facility, VMTP provides a Receive Sequence Number field in the NotifyVmtpClient and NotifyVmtpServer operations, plus the Next Receive Sequence (NRS) flag in the Response packet. This lets a sender log a receive sequence number with each message it sends, so the packets can be replayed at a recovering node in the same order they were originally received, and the node gets back to the same state as before. 👑
Here's the vibe: each sending node keeps a receive sequence number for each receiving node. When it sends a Request to a node, it presumes the receive sequence number is one greater than the one it has recorded for that node. If that's not the case, the receiving node sends a notify operation saying which receive sequence number it assigned to the Request. ✨ The NRS in the Response confirms that the Request message had the next receive sequence number, so the sender can tell if it missed the notify operation in the previous case. With Responses, the packets are ordered by the Transaction identifier, except for multicast message transactions, where there may be multiple Responses with the same identification. In that case, NotifyVmtpServer operations are used to provide receive sequence numbers. 🌈
This experimental extension is focused on supporting fault-tolerant real-time distributed systems, which are needed in various critical applications. 🥺 It may be removed or extended, depending on further investigations.
<6> The sender-based logging is being investigated by Willy Zwaenepoel of Rice University.
2.13. Packet Groups
A message, whether it's a Request or a Response, is sent as one or more packet groups. 🍓 A packet group is one or more packets, each containing the same transaction identification and message control block. Each packet is formatted as shown below, with the message control block logically embedded in the VMTP header.
+------------------------------------++---------------------+
| VMTP Header || |
+------------+-----------------------|| segment data |
|VMTP Control| Message Control Block || |
+------------+-----------------------++---------------------+Some fields of the VMTP control portion of the packet and the data segment portion can differ between packets within the same packet group. 🎀
The segment data portion of a packet group represents up to 16 kilooctets of the segment specified in the message control block. Which portion goes in each packet is indicated by the PacketDelivery field in the VMTP header. The PacketDelivery field, as a bit mask, works much like the MsgDelivery field, in that each bit corresponds to a segment data block of 512 octets. The PacketDelivery field limits a packet group to 16 kilooctets and a maximum of 32 VMTP packets, with a minimum of 1 packet. Data can be sent in fewer packets by sending multiple data blocks per packet. We require that the underlying datagram service support delivery of (at minimum) the basic 580 octet VMTP packet <7>. 💖 For example, take Ethernet, which has an MTU of 1536 octets, so you'd send 2 512-octet segment data blocks per packet. (Okay but here's the fun part: if a third block is last in the segment, is less than 512 octets, and fits in the packet without making it too big, an Ethernet packet could hold three data blocks. 🤯) So an Ethernet packet group for a segment of size 0x1D00 octets (14.5 blocks) and MsgDelivery 0x000074FF consists of 6 packets, shown as follows <8>.
Packet
Delivery 1 1 1 1 1 1 1 1 0 0 1 0 1 0 1 0 0 0 0 0 0 . . .
0000 0400 0800 0C00 1000 1400 1800 1C00
+----+----+----+----+----+----+----+-+
Segment |....|....|....|....|....|....|....|.|
+----+----+----+----+----+----+----+-+
: : : : : : : / / :
v v v v v v v /| v
+----+----+----+----+ +----+ +---+
Packets | 1 | 2 | 3 | 4 | | 5 | | 6 |
+----+----+----+----+ +----+ +---+Each '.' is 256 octets of data. The PacketDelivery masks for the 6 packets are 0x00000003, 0x0000000C, 0x00000030, 0x000000C0, 0x00001400 and 0x00006000, which indicate the segment blocks contained in each packet. (Note that the delivery bits are in little endian order. 🧠)
A packet group is sent as a single "blast" of packets with no explicit flow control. 🚀 However, the sender should estimate and transmit at a packet rate that avoids congesting the network or overwhelming the receiver, as described in Section 2.5.6. Packets in a packet group can be sent in any order with no change in semantics.
When the first packet of a packet group is received (assuming the Server doesn't decide to discard the packet group), the Server saves a copy of the VMTP packet header, marks that it is currently receiving a packet group, sets a "current delivery mask" (showing the data in the segment received so far) to 0, accepts this packet (updating the current delivery mask), and starts the timer for the packet group. Subsequent packets in the packet group update the current delivery mask. 💅
Reception of a packet group ends when either the current delivery mask shows that all the packets in the group have been received, or the packet group reception timer expires (set to TC3 or TS1). If the timer expires, and the NRT bit is set in the Control flags, then the packet group is discarded if it's incomplete, unless MDM is set. In that case, the MsgDelivery field in the message control block is set to show the segment data blocks actually received, and the message control block and segment data received are delivered to the application level. 🙈
If NRT is not set and not all data blocks have been received, a NotifyVmtpClient (for a Request) or NotifyVmtpServer (for a Response) is sent back with a PacketDelivery field showing the blocks received. The source of the packet group is then expected to retransmit the missing blocks. If not all blocks of a Request are received after RequestAckRetries(Client) retransmissions, the Request is discarded, and a NotifyVmtpClient operation with an error response code is sent to the client's manager, unless MDM is set. With a Response, there are ResponseAckRetries(Server) retransmissions, and then, if MDM is not set, the requesting entity gets the message control block with an indication of how much segment data was received, counting contiguously from the start of the segment. For example, if the sender sent 6 512-octet blocks and only the first two and the last two arrived, the receiver would be told that 1024 octets were received. The ResponseCode field is set to BAD_REPLY_SEGMENT. (Note that VMTP can only indicate the specific segment blocks received if MDM is set. 😤)
The parameters RequestAckRetries(Client) and ResponseAckRetries(Server) could be set on a per-client and per-server basis in a sophisticated implementation, based on knowledge of packet loss. 🦋
If the APG flag is set, a NotifyVmtpClient or NotifyVmtpServer operation is sent back at the end of the packet group reception, depending on whether it's a Request or a Response.
At minimum, a Server should check that each packet in the packet group contains the same Client, Server, Transaction identifier and SegmentSize fields. It is a protocol error for any field other than the Checksum, the packet group control flags, Length and PacketDelivery in the VMTP header to differ between any two packets in one packet group. A packet group with a protocol error of this nature should be discarded. 🚫
Notify operations should be sent (or invoked) in the manager whenever there is a problem with a unicast packet. In other words, negative acknowledgments are always sent in this case. 💌 For problems with multicast packets, the default is to send nothing in response to an error condition, unless there is some clear reason why no other node can respond positively. For example, the packet might be a Probe for an entity that is known to have recently existed on the receiving host but is now invalid and could not have migrated. In that case, the receiving host responds to the Probe indicating the entity is nonexistent, knowing that no other host can respond to the Probe. For packets and packet groups that are received and processed without problems, a Notify operation is invoked only if the APG bit is set. ✨
2.14. Runs of Packet Groups
A run of packet groups is a sequence of packet groups, all Request packets or all Response packets, with the same Client and consecutive transaction identifiers. All but the first and last packets are flagged with the NSR (Not Start Run) and NER (Not End Run) control bits. 🌸 When each packet group in the run corresponds to a single Request or Response, it is identical to a run of message transactions. (See Section 2.11) However, a Request message or a Response message may consist of up to 256 packet groups within a run, for a maximum of 4 megaoctets of segment data. A message that continues in the next packet group in the run is flagged in the current packet group by the CMG flag. Otherwise, the next packet group in the run (if any) is treated as a separate Request or Response. 🎀
Okay so the "normally one packet group" thing is literally the default vibe, bestie 🫶 Single packet group per run, and neither NSR nor NER gets set 💅 But when a message is bigger, the PacketDelivery mask in the i-th packet group lines up with the segment offset by (i-1) × 16 kilooctets, and the first packet group counts as i = 1 🎀
2.15. Byte Order
Okay, byte order is a whole thing 🌈 For sending and receiving, the MCB is treated as 8 32-bit fields, and the segment is just a bunch of bytes. VMTP sends the MCB in big-endian order, swapping bytes first if it has to. A little-endian host has to swap the MCB when it receives it 🔁 The data segment is sent as plain bytes with no reordering, so it doesn't get touched. 🙈
The sender's byte order is shown by the LEE bit in its entity identifier. That's the Client field for a Request and the Server field for a Response. The sender and receiver also have to agree, in some higher-level protocol (like an RPC presentation protocol), on who handles any extra swapping of the MCB and the data segment. Like, the segment might hold a record with 8-bit, 16-bit and 32-bit fields, so it needs extra transformation to move between hosts with different byte orders 💖
VMTP has historically used a higher-level presentation protocol where the segment data goes out in the sending host's native order, and the receiver does the byte-swapping if needed ✨ This keeps swapping overhead low between machines with the same byte order (even when the communication is local to one host), avoids favoring one byte order, and lets a sender talk to a group of hosts with different byte orders. 🦋 (The MCB swapping overhead is minimal, btw.) The presentation-level overhead is also small, because common stuff like file access operations has parameters that fit the MCB and data segment types exactly 🍓
2.16. Minimal VMTP Implementation
A minimal VMTP client needs to send a Request packet group and receive a Response packet group. It also has to accept and respond to Requests sent to its management module, including Probe and NotifyClient operations 🧚 It might also need to invoke Probe and Notify operations to locate a Server and acknowledge responses. (The latter only if it's in transactions that are not idempotent or datagram message transactions. A simple sensor, for example, can send VMTP datagram Requests with its current state using even less mechanism 🌸) So the minimal client needs very little code and works great as a base for something like a network boot loader 👑
A minimal VMTP server handles idempotent, non-encrypted message transactions, and it might not support segment data at all. It should use an entity state record for each Request, but only needs to keep it while processing that Request 💅 Without segment data bigger than one packet, it doesn't need timers, buffering (beyond immediate request processing), or queuing. It only needs as many records as the message transactions it handles at the same time (so maybe just 1) 🎀 The entity state record is required to recognize and respond to Request retransmissions during request processing.
The minimal server only needs to receive Requests and be able to send Response packets. Its management module only needs Probe operation support. (NotifyVmtpClient support is only required if it doesn't respond immediately to a Request.) So VMTP for something like a time server, sensor, or actuator can be super simple 🌷 The server never has to issue a Probe operation if it uses the host address from the Request for the Response and doesn't need the Client information that a Probe returns. The minimal server should also support receiving forwarded Requests 🫶
2.17. Message vs. Procedural Request Handling
A request-response protocol can give you two kinds of semantics on reception 🦋 With procedural handling, a Request gets handled by a process tied to the Server, which basically takes on the identity of the calling process. The Request message acts like a procedure call, and the process gives up that association when it returns. VMTP supports multiple nested calls across multiple machines. In that case, the distributed call stack is tied to one process for authentication and resource management, using the ProcessId field that VMTP supports 👑 The entity identifiers link those call frames together. The Client field in a Request is basically the return link to the previous call frame 🔗
With message handling, a Request message gets queued for a server process. The server process dequeues it, reads it, processes it and responds to it, running as a separate process. Later Requests to the same server wait in the queue until the server asks for the next one 📬
Procedural semantics let each Request run at the same time at the Server (up to its resource limits), with synchronization specific to each Request ⚡ Message semantics mean Requests are serialized at the Server, and processing logically runs with the priority, protection and independent execution of a separate process 🔐 The client can't tell the difference between procedural and message handling, apart from possibly performance 🌈
We see the two approaches as right for different situations, and VMTP supports both models 💜
2.18. Bibliography
The basic protocol is similar to the original form of the V kernel [3, 4], and to the transport protocol in Birrell and Nelson's [2] remote procedure call mechanism 🌸 An earlier version of the protocol was described in SIGCOMM'86 [6]. The rate-based flow control is similar to the techniques of Netblt [9]. The support for idempotency draws, in part, on the favorable experience with idempotency in the V distributed system. Its use was originally inspired by the Woodstock File Server [11]. The multicast support draws on the multicast facilities in V [5], and it's designed to work with (and is now implemented using) the multicast extensions to the Internet [8] described in RFC 966 and 988 🎀 The secure version of the protocol is similar to what Birrell [1] described for secure RPC. The use of runs of packet groups is similar to Fletcher and Watson's delta-T protocol [10]. Using "management" operations implemented with VMTP in place of specialized packet types is part of a general strategy of using recursion to simplify protocol architectures [7] 🧠
Finally, this protocol was designed, in part, to respond to the requirements Braden identified in RFC 955. We believe VMTP satisfies the requirements stated in RFC 955 💖
[1] A.D. Birrell, "Secure Communication using Remote Procedure Calls", ACM. Trans. on Computer Systems 3(1), February, 1985.
[2] A. Birrell and B. Nelson, "Implementing Remote Procedure Calls", ACM Trans. on Computer Systems 2(1), February, 1984.
[3] D.R. Cheriton and W. Zwaenepoel, "The Distributed V Kernel and its Performance for Diskless Workstations", In Proceedings of the 9th Symposium on Operating System Principles, ACM, 1983.
[4] D.R. Cheriton, "The V Kernel: A Software Base for Distributed Systems", IEEE Software 1(2), April, 1984.
[5] D.R. Cheriton and W. Zwaenepoel, "Distributed Process Groups in the V Kernel", ACM Trans. on Computer Systems 3(2), May, 1985.
[6] D.R. Cheriton, "VMTP: A Transport Protocol for the Next Generation of Communication Systems", In Proceedings of SIGCOMM'86, ACM, Aug 5-7, 1986.
[7] D.R. Cheriton, "Exploiting Recursion to Simplify an RPC Communication Architecture", in preparation, 1988.
[8] D.R. Cheriton and S.E. Deering, "Host Groups: A Multicast Extension for Datagram Internetworks", In 9th Data Communication Symposium, IEEE Computer Society and ACM SIGCOMM, September, 1985.
[9] D.D. Clark and M. Lambert and L. Zhang, "NETBLT: A Bulk Data Transfer Protocol", Technical Report RFC 969, Defense Advanced Research Projects Agency, 1985.
[10] J.G. Fletcher and R.W. Watson, "Mechanism for a Reliable Timer- based Protocol", Computer Networks 2:271-290, 1978.
[11] D. Swinehart and G. McDaniel and D. Boggs, "WFS: A Simple File System for a Distributed Environment", In Proc. 7th Symp. Operating Systems Principles, 1979.
3. VMTP Packet Formats
VMTP uses 2 basic packet formats, one for Request packets and one for Response packets 📦 They're nearly identical in most fields, which keeps implementation simple ✨
We'll start with the entity identifier format and the packet fields used in general, then go through each packet format in detail. The individual packet formats are in the subsections below. If you're a reader or implementor, you might want to check Chapters 4 and 5 for how VMTP handles events, and only come back to this detailed stuff as needed 💜
3.1. Entity Identifier Format
The 64-bit non-group entity identifiers have this substructure 🧚
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R| |L|R|
|A|0|E|E| Domain-specific structure
|E| |E|S|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Domain-specific structure |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The field meanings are:
RAE: Remote Alias Entity. The entity identifier names an entity that is acting as an alias for some entity outside this entity domain. This bit is used by higher-level protocols. For instance, servers may take extra security and protection measures with aliases. 🕵️GRP: Group. 0, for non-group entity identifiers.LEE: Little-Endian Entity. The entity transmits data in little-endian (VAX) order. 🌊RES: Reserved. Must be 0.
The 64-bit entity group identifiers have this substructure 💖
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R| |U|R|
|A|1|G|E| Domain-specific structure
|E| |P|S|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Domain-specific structure |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+The field meanings are:
RAE: Remote Alias Entity. Same as for non-group entity identifiers.GRP: Group. 1, for entity group identifiers.UGP: Unrestricted Group. No restrictions are placed on joining this group. In other words, any entity can join, limited only by implementation resources. 🎉RES: Reserved. Must be 0.
The all-zero entity identifier is reserved and guaranteed to be unallocated in all domains 🚫 A domain may also reserve part of the entity identifier space for statically allocated identifiers, but that part is domain-specific 🦋
Okay so the full rundown of every entity identifier domain that's currently defined lives in Appendix IV 📚✨
3.2. Packet Fields 🎀
Client is the 64-bit identifier for the client entity tied to this packet 🧚 Its structure, allocation and binding are specific to the Domain it's in. And bestie, an entity identifier always includes 4 types bits, as Section 3.1 explains 💜
Version is a 3-bit identifier for the protocol version. The current version is version 0 🌸
Domain is a 13-bit identifier for the naming and administration domain of the client and server named in the packet 🌈
Packet Flags are 3 bits total, and the normal case has none of them set 🦋
HCOis Header checksum only 🫶 The checksum is only calculated on the header. Some real-time applications use this because they don't need strict data correctness 🍓EPGis Encrypted packet group, which is part of a secure message transaction 🔐MPGis Multicast packet group, meaning the packet was multicast when it was sent 📡
Length is a 13-bit field for the number of 32-bit words in the segment data part of the packet (if there is any), not counting the checksum field 🍭 Every VMTP packet is required to be a multiple of 64 bits, which can be done by padding out the segment data 🧁 The minimum legal Length is 0, the maximum is 4096, and it must be an even number 💅
Control Flags are 9 bits total, and the normal case has none of them set 🎀
NRSis Next Receive Sequence. The associated Request (in a Response), or the previous Response (if it's a Request), was received consecutively with the last Request from this entity. In other words, no interfering messages were received 🙈APGis Acknowledge Packet Group, meaning acknowledge the packet group on receipt. If it's a Request, send a Request back to the client's manager giving an update on the transaction's state as soon as the request packet group is received, whether or not the response is ready yet. If it's a Response, send an update to the server's manager as soon as possible after the response packet group is received, giving an update on the transaction's state at the client ✨NSRis Not Start Run. It's 1 if this packet isn't part of the first packet group of a run of packet groups 🏃♀️NERis Not End Run. It's 1 if this packet isn't part of the last packet group of a run of packet groups 🏁NRTis No Retransmission. Don't ask for retransmissions of this packet group if it isn't all received within the timeout period. Just deliver or discard it 🗑️MDGis Member of Destination Group. The packet is sent to a group and the client is a member of that group 👑CMGis Continued Message. The message (Request or Response) continues in the next packet group, and that next packet group has to be part of the same run of packet groups 🔁STIis Skip Transaction Identifiers. The next transaction identifier the Client plans to use is the current transaction plus 256, if it's part of the same run, and at least this big if it isn't. In a Request, this authorizes the Server to send back up to 256 packet groups containing the Response 🚀DRTis Delay Response Transmission. The request sender sets it if multiple responses are expected (as shown by the MRD flag in the RequestCode), and it might get overrun by multiple responses. The responder(s) should then add a short random delay before sending the Response, to lower the chance of overrunning the Client 😭 This is normally only used for responding to multicast Requests, where the Client might be getting a huge number of Responses (as shown by the MRD flag in the Request flags). Otherwise, the Response is sent right away ⚡
RetransmitCount is a 3-bit field. It's the ordinal number of transmissions of this packet group before this one, modulo 8 🔢 It's used to estimate roundtrip times. This count may wrap around during a message transaction, but it should still be enough to match acknowledgments and responses to a particular transmission 🔄
ForwardCount is a 4-bit field showing how many times this Request has been forwarded. The original Request is always sent with a ForwardCount of 0 🌟
Interpacket Gap is an 8-bit field 🫧 It shows the recommended time to leave between packet transmissions inside a multi-packet packet group transmission. The Interpacket Gap is measured in 1/32nd of a network packet transmission time, for a packet of size MTU for the node. So the maximum gap is 8 packet times 🕰️
PGcount is an 8-bit field for the number of packet groups this packet group represents, on top of the one given in the Transaction field. It's used to acknowledge multiple packet groups in streamed communication 🌊
Priority is a 4-bit identifier for how urgent it is to process this request, both when it's sent and when it's received. Here's what the values mean:
1100 urgent/emergency
1000 important
0000 normal
0100 backgroundTreat the higher-order bit as a sign bit (1 means negative). Then low values are high priority and high values are low priority. The low-order 2 bits add extra (lower) levels within each level 🎚️
Function Code is 1 bit, and it covers the types of VMTP packets. If the low-order bit of the function code is 0, the packet goes to the Server. If it's 1, it goes to the Client 📬
0 Request
1 ResponseTransaction is a 32-bit identifier for this message transaction 🪪
PacketDelivery is 32 bits. It tells you which segment blocks are in this packet. Each bit stands for one 512-octet block of segment data. A 1 in the i-th bit position (counting the LSB as 0) means the i-th segment block is present 🧩
Server is 64 bits, and it's the entity identifier for the server or server group tied to this transaction. It's the receiver in a Request packet and the sender in a Response packet 💌
Code is 32 bits. It holds the Request Code and Response Code, set either at the user level or the VMTP level depending on how it's used and what kind of packet it is. Both codes include 8 high-order bits from this set of control bits 🎛️
CMDis Conditional Message Delivery. It only delivers the request or response if the receiving entity is waiting for it at the time of delivery. Otherwise, the message gets dropped 🙅♀️DGMis DataGram Message. It means the message is being sent as a datagram. For a Request, don't wait for a reply or retransmit. For a Response, treat this message transaction as idempotent 🦄MDMis Message Delivery Mask. It means the MsgDelivery field is in use. Otherwise, the MsgDelivery field is free for general use 🎨SDAis Segment Data Appended. Segment data is appended to the message control block, and the total size of the segment is given by the SegmentSize field. Otherwise, the segment data is null, and the SegmentSize field isn't used by VMTP. It's available for user- or RPC-level uses instead 🧃CREis CoResident Entity. It means the CoResidentEntity field in the message should be read by VMTP. Otherwise, that field is available for extra user data 🏠MRDis Multiple Responses Desired. Multiple Responses are wanted for this Request if it's multicast. Otherwise, the VMTP module can throw away any Responses after the first one 🗂️PICis Public Interface Code. Values for Code with this bit set are reserved for the VMTP specification and other standard protocols built on top of VMTP 🏛️RESis reserved for future use, and it must be 0 🔒
CoResidentEntity is a 64-bit identifier for an entity or group of entities that the Server entity or entities must be co-resident with. In other words, route only to entities (identified by Server) on the same host(s) as the one given by CoResidentEntity. It only matters if CRE is set in the Code field 🏡
User Data is 12 octets of space in the header for the VMTP user to put in their own control and data 🎁
MsgDelivery is 32 bits. It holds the segment blocks being sent (in total) in this packet group, following the conventions of the PacketDelivery field. The protocol ignores this field and treats it as an extra user data field if MDM is 0. When sending, the user level sets MsgDelivery to show which parts of the segment should be sent. When receiving, the VMTP module changes MsgDelivery to show which segment data blocks were actually received before the message control block gets passed up to the user or RPC level. Importantly, the kernel does not throw away the packet group if segment data blocks are missing. A Server or Client entity that receives a message with MsgDelivery in use must check the field to make sure delivery was good enough, and retry the operation if needed 🥺
SegmentSize is 32 bits. It's the size of the segment in octets, up to a maximum of 16 kilooctets without streaming, and 4 megaoctets with streaming, if SDA is set. Otherwise, the protocol ignores this field and treats it as an extra user data field 📏
Segment Data is 0-16 kilooctets. It's 0 octets if SDA is 0. Otherwise, it's the part of the segment that matches the Delivery Mask, limited by the SegmentSize and the MTU, and padded out to a multiple of 64 bits 🧸
Checksum is 32 bits, and it's the 32-bit checksum for the header and segment data 🛡️
The VMTP checksum algorithm <9> develops a 32-bit checksum by computing
<9> This algorithm and description are largely due to Steve Deering of Stanford University.
two 16-bit, ones-complement sums (like IP) 💖, each covering different parts of the packet. So the packet gets chopped into clusters of 16 16-bit words 🧩. The first, third, fifth, and so on clusters get added into the first sum, and the second, fourth, sixth, and so on go into the second sum. Adding stops at the end of the packet, and you don't have to pad out to a cluster boundary 🙈. BUT the packet has to be an integral multiple of 64 bits, so padding octets are needed. They can have any value, and they count toward the checksum and get sent along with the packet 💅. If either resulting sum comes out as zero, it gets switched to 0xFFFF ✨. The two sums get tacked onto the end of the transmitted packet, with the first sum going first 🎀. And four bytes of zero in place of the checksum can be used to say that no checksum was computed 🥺.
The 16-bit, ones-complement addition here is the same one IP uses, so it gets the same optimizations 🔥. Specifically, you can add the words up 32 bits at a time, as long as the carry-out of each addition gets added into the sum on the next addition, using an "add-with-carry" type of instruction 💜. (64-bit or 128-bit additions would also work on machines with registers that big 🤩.)
Okay bestie, real talk ⚠️: this algorithm (shared by IP) does NOT detect the erroneous swapping of 16-bit words, and that can easily happen thanks to software errors 😭. A future version of VMTP is expected to include a more secure algorithm, but that one seems to need hardware support to run efficiently 💀.
Not every field gets used in every packet 🦋. The specific packet formats are coming up below. If a field isn't mentioned in the description of a packet type, you can just assume its use is clear from what we covered above 🌸.
3.3. Request Packet 📨
This is the Request packet (or packet group) 🍓. The client sends it to the server (or group of servers) to ask for processing and get back zero or more responses. You can spot one by the 0 in the LSB of the fourth 32-bit word in the packet 🔍.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Client (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Ver | |H|E|M| |
|sion | Domain |C|P|P| Length |
| | |O|G|G| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|N|A|N|N|N|M|C|S|D|Retra|Forward| Inter- | |R|R|R| |
|R|P|S|E|R|D|M|T|R|nsmit| Count | Packet | Prior |E|E|E|0|
|S|G|R|R|T|G|G|I|T|Count| | Gap | -ity |S|S|S| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Transaction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PacketDelivery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Server (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|C|D|M|S|R|C|M|P| |
|M|G|D|D|E|R|R|I| RequestCode |
|D|M|M|A|S|E|D|C| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ CoResidentEntity (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> User Data (12 octets) <
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MsgDelivery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SegmentSize |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> segment data, if any <
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3-1: Request Packet Format 🖼️
The Request packet fields follow the semantics from Section 3.2, with a few extra twists 💅:
InterPacketGap: the estimated interpacket gap time the client would love for the Server to use when it sends the Response packet group back for this Request ⏱️Transaction: the transaction identifier, which has to be at least one greater than the last Request this Client issued 🔢Server: the Server this Request is destined for 🎯RequestCode: the request code, telling the Server which operation to perform 🛠️
3.4. Response Packet 💌
The Response packet is sent from the Server back to the Client in response to a Request. You can spot it by the 1 in the LSB of the fourth 32-bit word in the packet 🦄.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Client (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Ver | |H|E|M| |
|sion | Domain |C|P|P| Length |
| | |O|G|G| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|N|A|N|N|N|R|C|S|R|Retra|Forward| | |R|R|R| |
|R|P|S|E|R|E|M|T|E|nsmit| Count | PGcount | Prior |E|E|E|1|
|S|G|R|R|T|S|G|I|S|Count| | | -ity |S|S|S| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Transaction |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PacketDelivery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+ Server (8 octets) +
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|C|D|M|S|R|R|R|R| |
|M|G|D|D|E|E|E|E| ResponseCode |
|D|M|M|A|S|S|S|S| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> UserData (20 octets) <
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| MsgDelivery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Segment Size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> segment data, if any <
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+Figure 3-2: Response Packet Format 🖼️
The Response packet fields follow the semantics from Section 3.2, with these qualifications 🪞:
Client,Version,Domain,Transaction: these have to match the ones in the Request packet group this is a response to 🫶STI: this is 1 if the Response is using one or more of the transaction identifiers the Client skipped after the Request it answers 🎀. Basically,STIin the Request allocates up to 256 transaction identifiers for the Server to use in a run of Response packet groups 🧁RetransmitCount: the retransmit count from the last Request packet received that this Response answers 🔁ForwardCount: how many times the corresponding Request got forwarded before this Response was generated 📬PGcount: how many consecutively previous packet groups this Response is acknowledging, on top of the one named byTransaction🙌Server: the Server sending this Response 🔀. It can differ from the one in the original Request if that Server was a server group, or if the Request got forwarded
Next up are two chapters explaining how the protocol actually runs with these packet formats, with the Client and Server halves covered separately 🎬.
4. Client Protocol Operation 👑
This chapter walks through the client side of VMTP 💖. It covers how the client handles VMTP user events, packet reception events, management operations, and timeout events. Heads up: the client portion is separable from the server portion 🫶, so it's totally feasible for a node to implement only the client end of VMTP.
To keep things simple, we define a client state record (CSR) plus some standard utility routines 🧠.
4.1. Client State Record Fields 📋
There's exactly one client state record (CSR) per (client,transaction) outstanding message transaction 🗂️. Here's a suggested set of fields, bestie:
Link: link to the next CSR when it's queued in the transmission, timeout, or message queues 🔗QueuePtr: pointer to the head of whichever queue this CSR is in, orNULLif it's in none 🚦. That queue could be the transmission queue, timeout queue, server queue, or response queue.ProcessIdentification: the process identification and address space 🪪Priority: the priority for processing, network service, and so on 🏃♀️State: one of the client states we'll cover below 🌀FinishupFunc: the procedure to run on the CSR once it finishes processing in the transmission or timeout queues 🏁TimeoutCount: how long to remain in the timeout queue ⏳TimeoutLimit: the user-specified time after which the message transaction gets aborted ✂️. Set it to zero and the timeout is infinite ♾️.RetransCount: how many retransmissions have happened since we last heard from the Server 📣LastTransmitTime: the time the last packet was sent 🕰️. This is used to work out round-trip times, andRetransmitCounthelps match the responding packet to a particular transmission. In other words, a Response or the managementNotifyVmtpClientoperation gets matched to the Request, and a managementNotifyVmtpServeroperation gets matched to a Response.TimetoLive: the time to live value used when transmitting IP packets 🌐TransmissionMask: a bit mask showing which portions of the segment to transmit 🎭. It's set before the CSR enters the transmission queue, and it gets cleared incrementally as the 512-byte segment blocks of the segment are transmitted.LocalClientLink: link to the next CSR that hashes to the same hash index in theClientMap🧵LocalClient: the entity identifier for the client when this CSR is used to send a Request packet 🪄LocalTransaction: the transaction identifier for the current message transaction the local client still has outstanding ⏰
LocalPrincipal, bestie 💖, is the account identification. It can also include the key and the key timeout 🔑.
LocalDelivery: a bit mask of segment blocks that have not been acknowledged in the Request or have been received in the Response, depending on the state 📦ResponseQueue: a queue of CSRs representing the queued Responses for this entity 🎀VMTP Header: the prototype VMTP header, used to generate and store the header portion of a Request for transmission and retransmission on timeout 🧚SegmentDesc: a description of the segment data associated with the CSR. It's either the area storing the original Request data, the area for receiving Request data, or the area storing the Response data that is returned 🌸HostAddr: the network or internetwork host address the Client last transmitted to. This field also indicates the type of the address, e.g. IP, Ethernet, etc. 🦋
Note: the CSR can be combined with a light-weight process descriptor, and it's lowkey a huge win if the process is designed to block when it issues a message transaction 💅. By combining the two descriptors, the implementation saves time because it only needs to locate and queue one descriptor with various operations, instead of locating two descriptors 🔥. It also saves space, since the VMTP header prototype provides room (like the user data field) that can store processor state for when the process is preempted 🫶. Non-preemptive blocking can use the process stack to store the processor state, so only a program counter and stack pointer may be needed in the process descriptor beyond what we've described. (This is the approach used in the V kernel 👑.)
4.2. Client Protocol States ✨
A Client State Record records the state of a message transaction generated by this host, identified by the (Client, Transaction) values in the CSR. As a client originating a transaction, it is in one of the following states 🌈.
AwaitingResponse: waiting for a Response packet group to arrive with the same (Client,Transaction) identification 👀.ReceivingResponse: waiting for additional packets in the Response packet group it is currently receiving 📨."Other": not waiting for a response. This can be Processing or some other operating system state, or one of the Server states if it also acts as a server 💤.
That covers all the states for a client, periodt 💅.
4.3. State Transition Diagrams 🦋
The client state transitions are illustrated in Figure 4-1. The client goes into the state AwaitingResponse on sending a request, unless it is a datagram request 🚀. In the AwaitingResponse state, it can time out and retry, and eventually give up and return to the processing state unless it receives a Response. (A NotifyVmtpClient operation resets the timeout but does not change the state ⏰.) On receipt of a single packet response, it returns to the processing state. Otherwise, it goes to the ReceivingResponse state. After a timeout or the final response packet is received, the client returns to the processing state. The processing state also includes any other state besides those associated with issuing a message transaction 🧁.
+------------+
| Processing |<--------------------|
| |<-------------| |
| |<---| | |
+|------^--^-+ Single Last |
Transmit | | Packet Response |
| | | Response Packet |
| | | | | |
+-DGM->+ Timeout | | Final timeout
| | | | |
+V-----------+ | +-----------+
| Awaiting |----+ | Receiving |->Response-+
| Response |->Response->| Response | |
| | (multi- | |<----------+
+-|--------^-+ packet) +----------^+
V | | |
+-Timeout+ +>Timeout+
Figure 4-1: Client State Transitions4.4. User Interface 🎀
The RPC or user interface to VMTP is implementation-dependent, and it may use system calls, functions, or some other mechanism 🛠️. The list of requests that follows is intended to suggest the basic functionality that should be available ✨.
Send( mcb, timeout, segptr, segsize ): initiates a message transaction to the server and request message specified by mcb, and returns a response in mcb if it is received within the specified timeout period. Otherwise it returns USER_TIMEOUT in the Code field ⏳. The segptr parameter specifies the location from which the segment data is sent and the location into which the response data is delivered. The segsize field indicates the maximum length of this area 📏.GetResponse( responsemcb, timeout, segptr, segsize ): gets the next response sent to this client as part of the current message transaction, returning the segment data, if any, into the memory specified by segptr and segsize 📬.
This interface assumes that there is a client entity associated with the invoking process that is to be used with these operations. Otherwise, the client entity must be specified as an additional parameter 🧚.
4.5. Event Processing 🔥
The following events may occur in the VMTP client:
- User Requests
- Send
- GetResponse
- Packet Arrival
- Response Packet
- Request
The minimal Client implementation handles Request packets for its VMTP management (server) module and sends NotifyVmtpClient requests in response to others, indicating the specified server does not exist 🙈.
- Management Operation - NotifyVmtpClient
- Timeouts
- Client Retransmission Timeout
The handling of these events is described in detail in the following subsections 📚.
We first describe some conventions and procedures used in the description. A field of the received packet is indicated as (for example) p.Transaction, for the Transaction field. Optional portions of the code, such as the streaming handling code, are prefixed with a "|" in the first column 💜.
MapClient( client ): returns a pointer to the CSR for the client with the specified clientId, else NULL 🗺️.
SendPacketGroup( csr ): sends the packet group (Request, Response) according to that specified by the CSR 📤.
NotifyClient( csr, p, code ): invokes the NotifyVmtpClient operation with the parameters csr.RemoteClient, p.control, csr.ReceiveSeqNumber, csr.RemoteTransaction and csr.RemoteDelivery, and code. If csr is NULL, it uses p.Client, p.Transaction and p.PacketDelivery instead, and the global ReceiveSequenceNumber, if supported. This function simplifies the description over calling NotifyVmtpClient directly in the procedural specification below (See Appendix III) 🍓.
NotifyServer( csr, p, code ): invokes the NotifyVmtpServer operation with the parameters p.Server, csr.LocalClient, csr.LocalTransaction, csr.LocalDelivery and code. It uses p.Client, P.Transaction and 0 for the clientId, transact and delivery parameters if csr is NULL. This function simplifies the description over calling NotifyVmtpServer directly in the procedural specification below (See Appendix III) 🌟.
DGMset(p): True if the DGM bit is set in the packet (or csr), else False. Similar functions are used for other bits 🧩.
Timeout( csr, timeperiod, func ): sets or resets the timer on the csr record for timeperiod later, and invokes func if the timeout expires ⏲️.
4.6. Client User-invoked Events 💖
A user event occurs when a VMTP user application invokes one of the VMTP interface procedures 🎉.
4.6.1. Send 🚀
Send( mcb, timeout, segptr, segsize )
map to main CSR for this client.
increment csr.LocalTransaction
Init csr and check parameters and segment if any.
Set SDA if sending appended data.
Flush queued replies from previous transaction, if any.
if local non-group server then
deliver locally
await response
return
if GroupId(server) then
Check for and deliver to local members.
if CRE request and non-group local CR entity then
await response
return
endif
set MDG if member of this group.
endif
clear csr.RetransCount
set csr.TransmissionMask
set csr.TimeLimit to timeout
set csr.HostAddr for csr.Server
SendPacketGroup( csr )
if DGMset(csr) then
return
endif
set csr.State to AwaitingResponse
Timeout( rootcsr, TC1(csr.Server), LocalClientTimeout )
return
end SendNotes:
- Normally, the HostAddr is extracted from the ServerHost cache, which maps server entity identifiers to host addresses. However, on a cache miss, the client first queries the network using the ProbeEntity operation, as specified in Appendix III, determining the host address from the Response. The ProbeEntity operation is handled as a separate message transaction by the Client 🔍.
The stream interface incorporates a parameter to pass a responseHandler procedure that is invoked when the message transaction completes 🌊.
StreamSend( mcb, timeout, segptr, segsize, responseHandler )
map to main CSR for this client.
| Allocate a new csr if root in use.
| lastcsr := First csr for last request.
| if STIset(lastcsr)
| csr.LocalTransaction := lastcsr.LocalTransaction + 256
| else
| csr.LocalTransaction := lastcsr.LocalTransaction + 1
Init csr and check parameters and segment if any.
. . . ( rest is the same as for the normal Send)Notes:
- Each outstanding message transaction is represented by a CSR queued on the root CSR for this client entity. The root CSR is used to handle timeouts, etc. On timeout, the last packet from the last packet group is retransmitted (with or without the segment data) 🔁.
4.6.2. GetResponse 🧁
GetResponse( req, timeout, segptr, segsize )
csr := CurrentCSR;
if responses queued then return next response
(in req, segptr to max of segsize )
if timeout is zero then return KERNEL_TIMEOUT error
set state to AWAITING_RESPONSE
Timeout( csr, timeout, ReturnKernelTimeout );
end GetResponseNotes:
- GetResponse is only used with multicast Requests, which is the only case in which multiple (different) Responses should be received 📣.
- A response must remain queued until the next message transaction is invoked to filter out duplicates of this response 🧹.
- If the response is incomplete (only relevant if a multi-packet response), then the client may wait for the response to be fully received, including issuing requests for retransmission (using NotifyVmtpServer operations) before returning the response 🧾.
- As an optimization, a response may be stored in the CSR of the client. In this case, the response must be transferred to a separate buffer (for duplicate suppression) before waiting for another response. Using this optimization, a response buffer is not allocated in the common case of the client receiving only one response 💫.
4.7. Packet Arrival 📦
In general, on packet reception, a packet is mapped to the client state record and decrypted if necessary using the key in the CSR. It then has its checksum verified, and then it is transformed to the right byte order 🔐.
The packet is then processed fully relative to its packet function code 🎀 But first, if it's addressed to a domain the receiving host doesn't participate in, it's discarded immediately 🙅♀️ no cap.
We're assuming there's a procedure for each of the 2 packet types. It gets a pointer p to the VMTP packet plus psize, which is the size of the packet in octets. So generic packet reception looks like this 🫶:
if not LocalDomain(p.Domain) then return;
csr := MapClient( p.Client )
if csr is NULL then
HandleNoCsr( p, psize )
return
if Secure(p) then
if SecureVMTP not supported then
{ Assume a Request. }
if not Multicast(p) then
NotifyClient(NULL, p, SECURITY_NOT_SUPPORTED )
return
endif
| Decrypt( csr.Key, p, psize )
if p.Checksum not null then
if not VerifyChecksum(p, psize) then return;
if OppositeByteOrder(p) then ByteSwap( p, psize )
if psize not equal sizeof(VmtpHeader) + 4*p.Length then
NotifyClient(NULL, p, VMTP_ERROR )
return
Invoke Procedure[p.FuncCode]( csr, p, psize )
Discard packet and returnNotes:
- The Procedure[p.FuncCode] refers to one of the 2 procedures corresponding to the two different packet types of VMTP, Requests and Responses. 💜
- In all the following descriptions, a packet is discarded on "return" unless otherwise stated. 🗑️
- The procedure HandleNoCSR is a management routine that allocates and initializes a CSR and processes the packet or else sends an error indication to the sender of the packet. This procedure is described in greater detail in Section 4.8.1. ✨
4.7.1. Response
Okay, this procedure is where incoming Response packets get handled 🦋
HandleResponse( csr, p, psize )
if not LocalClient( csr ) then
if Multicast then return
| if Migrated( p.Client ) then
| NotifyServer(csr, p ENTITY_MIGRATED )
| else
NotifyServer(csr, p, ENTITY_NOT_HERE )
return
endif
if NSRset(p) then
if Streaming not supported then
NotifyServer(csr, p, STREAMING_NOT_SUPPORTED )
return STREAMED_RESPONSE
| Find csr corresponding to p.Transaction
| if none found then
| NotifyServer(csr, p, BAD_TRANSACTION_ID )
| return
else
if csr.LocalTransaction not equal p.Transaction then
NotifyServer(csr, p, BAD_TRANSACTION_ID )
return
endif
Locate reply buffer rb for this p.Server
if found then
if rb.State is not ReceivingResponse then
{ Duplicate }
if APGset(p) or NERset(p) then
{ Send Response to stop response packets. }
NotifyServer(csr, p, RESPONSE_DISCARDED )
endif
return
endif
{ rb.State is ReceivingRequest}
if new segment data then retain in CSR segment area.
if packetgroup not complete then
Timeout( rb, TC3(p.Server), LocalClientTimeout )
return;
endif
goto EndPacketGroup
endif
{ Otherwise, a new response message. }
if (NSRset(p) or NERset(p)) and NoStreaming then
NotifyServer(csr, p, VMTP_ERROR )
return
| if NSRset(p) then
| { Check consecutive with previous packet group }
| Find last packet group CSR from p.Server.
| if p.Transaction not
| lastcsr.RemoteTransaction+1 mod 2**32 then
| { Out of order packet group }
| NotifyServer(csr, p, BAD_TRANSACTION_ID)
| return
| endif
| if lastcsr not completed then
| NotifyServer(lastcsr, p, RETRY )
| endif
| if CMG(lastcsr) then
| Add segment data to lastcsr Response
| Notify lastcsr with new packet group.
| Clear lastcsr.VerifyInterval
| else
| if lastcsr available then
| use it for this packet group
| else allocate and initialize new CSR
| Save message and segment data in new CSR area.
| endif
| else { First packet group }
Allocate and init reply buffer rb for this response.
if allocation fails then
NotifyServer(csr, p, BUSY )
return
Set rb.State to ReceivingResponse
Copy message and segment data to rb's segment area
and set rb.PacketDelivery to that delivered.
Save p.Server host address in ServerHost cache.
endif
if packetgroup not complete then
Timeout( rb, TS1(p.Client), LocalClientTimeout )
return;
endif
endPacketGroup:
{ We have received last packet in packet group. }
if APGset(p) then NotifyServer(csr, p, OK )
| if NERset(p) and CMGset(p) then
| Queue waiting for continuation packet group.
| Timeout( rb, TC2(rb.Server), LocalClientTimeout )
| return
| endif
{ Deliver response message. }
Deliver response to Client, or queue as appropriate.
end HandleResponseNotes:
- The mechanism for handling streaming is optional and can be replaced with the tests for use of streaming. Note that the server should never stream at the Client unless the Client has streamed at the Server or has used the STI control bit. Otherwise, streamed Responses are a protocol error. 🌊
- As an optimization, a Response can be stored into the CSR for the Client rather than allocating a separate CSR for a response buffer. However, if multiple responses are handled, the code must be careful to perform duplicate detection on the Response stored there as well as those queued. In addition, GetResponse must create a queued version of this Response before allowing it to be overwritten. 🧚
- The handling of Group Responses has been omitted for brevity. Basically, a Response is accepted if there has been a Request received locally from the same Client and same Transaction that has not been responded to. In this case, the Response is delivered to the Server or queued. 🍓
4.8. Management Operations
VMTP uses management operations (invoked as remote procedure calls) to effectively acknowledge packet groups and request retransmissions. 🔥 The following routine is invoked by the Client's management module on request from the Server.
NotifyVmtpClient( clientId,ctrl,receiveSeqNumber,transact,delivery,code)
Get csr for clientId
if none then return
if RemoteClient( csr ) and not NotifyVmtpRemoteClient then
return
| else (for streaming)
| Find csr with same LocalTransaction as transact
| if csr is NULL then return
if csr.State not AwaitingResponse then return
if ctrl.PGcount then ack previous packet groups.
select on code
case OK:
Notify ack'ed segment blocks from delivery
Clear csr.RetransCount;
Timeout( csr, TC1(csr.Server), LocalClientTimeout )
return
case RETRY:
Set csr.TransmissionMask to missing segment blocks,
as specified by delivery
SendPacketGroup( csr )
Timeout( csr, TC1(csr.Server), LocalClientTimeout )
case RETRY_ALL
Set csr.TransmissionMask to retransmit all blocks.
SendPacketGroup( csr )
Timeout( csr, TC1(csr.Server), LocalClientTimeout )
| if streaming then
| Restart transmission of packet groups,
| starting from transact+1
return
case BUSY:
if csr.TimeLimit exceeded then
Set csr.Code to USER_TIMEOUT
return Response to application
return;
Set csr.TransmissionMask for full retransmission
Clear csr.RetransCount
Timeout( csr, TC1(csr.Server), LocalClientTimeout )
return
case ENTITY_MIGRATED:
Get new host address for entity
Set csr.TransmissionMask for full retransmission
Clear csr.RetransCount
SendPacketGroup( csr )
Timeout( csr, TC1(csr.Server), LocalClientTimeout )
return
case STREAMING_NOT_SUPPORTED:
Record that server does not support streaming
if CMG(csr) then forget this packet group
else resend Request as separate packet group.
return
default:
Set csr.Code to code
return Response to application
return;
endselect
end NotifyVmtpClientNotes:
- The delivery parameter indicates the segment blocks received by the Server. That is, a 1 bit in the i-th position indicates that the i-th segment block in the segment data of the Request was received. All subsequent NotifyVmtpClient operations for this transaction should be set to acknowledge a superset of the segment blocks in this packet. In particular, the Client need not be prepared to retransmit the segment data once it has been acknowledged by a Notify operation. 🎀
4.8.1. HandleNoCSR
HandleNoCSR gets called when a packet shows up for which there's no CSR matching the client field of the packet 👀
HandleNoCSR( p, psize )
if Secure(p) then
if SecureVMTP not supported then
{ Assume a Request }
if not Multicast(p) then
NotifyClient(NULL,p,SECURITY_NOT_SUPPORTED)
return
endif
HandleRequestNoCSR( p, psize )
return
endif
if p.Checksum not null then
if not VerifyChecksum(p, psize) then return;
if OppositeByteOrder(p) then ByteSwap( p, psize )
if psize not equal sizeof(VmtpHeader) + 4*p.Length then
NotifyClient(NULL, p, VMTP_ERROR )
return
if p.FuncCode is Response then
| if Migrated( p.Client ) then
| NotifyServer(csr, p ENTITY_MIGRATED )
| else
NotifyServer(csr, p, NONEXISTENT_ENTITY )
return
endif
if p.FuncCode is Request then
HandleRequestNoCSR( p, psize )
return
end HandleNoCSRNotes:
- The node need only check to see if the client entity has migrated if in fact it supports migration of entities. 🧳
- The procedure HandleRequestNoCSR is specified in Section 5.8.1. In the minimal client version, it need only handle Probe requests and can do so directly without allocating a new CSR. 💅
4.9. Timeouts
A client with a message transaction in progress has a single timer corresponding to the first unacknowledged request message. (In the absence of streaming, this request is also the last request sent.) This timeout is handled as follows ⏰
LocalClientTimeout( csr )
select on csr.State
case AwaitingResponse:
if csr.RetransCount > MaxRetrans(csr.Server) then
terminate Client's message transactions up to
and including the current message transaction.
set return code to KERNEL_TIMEOUT
return
increment csr.RetransCount
Resend current packet group with APG set.
Timeout( csr, TC2(csr.Server), LocalClientTimeout )
return
case ReceivingResponse:
if DGMset(csr) or csr.RetransCount > Max then
if MDMset(csr) then
Set MCB.MsgDeliveryMask to blocks received.
else
Set csr.Code to BAD_REPLY_SEGMENT
return to user Client
endif
increment csr.RetransCount
NotifyServer with RETRY
Timeout( csr, TC3(csr.Server), LocalClientTimeout )
return
end select
end LocalClientTimeoutOkay bestie, let's get into the notes 📝🎀
- Okay so a Client can only ask for a Response to be retransmitted if that Response is not idempotent 🙅♀️. If it IS idempotent, the Client must retransmit the Request instead. The Server should generally support the
MsgDeliveryMaskfor Requests it treats as idempotent and that need multi-packet Responses. Otherwise, there's no selective retransmission for idempotent message transactions. 💅 - The current packet group is the last one transmitted, so with streaming there may be several packet groups still outstanding that come before the current one 🌊
- The Request packet group should be retransmitted without the segment data, which leaves you with one short packet in the retransmission 🦋 The Server must then send a
NotifyVmtpClientwith aRETRYorRETRY_ALLcode so the segment data gets sent when it's actually needed. This keeps the overhead low on the network and on the server(s) for retransmissions. Iconic, honestly 😭
5. Server Protocol Operation ✨
This section explains the server side of the protocol by walking through how it handles VMTP user events, packet reception events, and timeout events. Every server is assumed to implement the client procedures from the previous chapter. 🫶 That isn't strictly required, but it makes everything way easier to follow.
5.1. Remote Client State Record Fields 🎀
The CSR for a server gets extra fields on top of the ones listed for the client version. Here they are:
RemoteClient Identifier for remote client that sent the Request that
this CSR is handling.
RemoteClientLink
Link to next CSR hashing to same hash index in the
ClientMap.
RemoteTransaction
Transaction identifier for Request from remote client.
RemoteDelivery The segment blocks received so far as part of a Request
or yet to be acknowledged as part of a Response.
VerifyInterval Time interval since there was confirmation that the
remote Client was still valid.
RemotePrincipal Account identification, possibly including key and key
timeout for secure communication.5.2. Remote Client Protocol States 🦋
A CSR on the server end is in one of these states:
AwaitingRequest Waiting for a Request packet group. It may be marked as
waiting on a specific Client, or on any Client.
ReceivingRequest
Waiting to receive additional Request packets in a
multi-packet group Request.
Responded The Response has been sent and the CSR is timing out,
providing duplicate suppression and retransmission (if
the Response was not idempotent).
ResponseDiscarded
Response has been acknowledged or has timed out so
cannot be retransmitted. However, duplicates are still
filtered and CSR can be reused for new message
transaction.
Processing Executing on behalf of the Client.
Forwarded The message transaction has been forwarded to another
Server that is to respond directly to the Client.5.3. State Transition Diagrams 🌈
The CSR state transitions on the server are shown in Figure 5-1. ✨ A CSR usually starts out in the AwaitingRequest state. When a Request comes in, the Server either already has an up-to-date CSR for that Client, or it sends a Probe request (as its own separate VMTP message transaction) to the VMTP management module that's linked to the Client. In that second case, processing of the Request gets held up until a Response to the Probe comes back. Then the CSR info is brought up to date and the Request actually gets processed 🙌
If the Request is a single-packet request, the CSR gets set to the Processing state to handle it. If it's a multi-packet Request instead, the CSR is put into the ReceivingResponse state, where it waits for the rest of the Request packets that make up the message. It leaves the ReceivingRequest state either on timeout or once it gets the last Request packet. If it timed out, the request is delivered with an indication of which portion was received, using the MsgDelivery field if MDM is set 🥺 Once request processing is done, the Response either gets sent and the CSR enters the Responded state, or the message transaction gets forwarded and the CSR enters the Forwarded state.
Now, in the Responded state, if the Response is not marked as idempotent, it gets retransmitted in a few situations. The first is when a retransmission of the corresponding Request arrives. The second is when a NotifyVmtpServer operation asks for retransmission. The third is on timeout, when APG is set so the Client gets asked for an acknowledgment. The Response gets retransmitted up to some maximum number of times. After that, it's discarded and the CSR is marked to match 💀 If a Request or a NotifyVmtpServer operation expects a retransmission of the Response after the CSR has already entered the ResponseDiscarded state, a NotifyVmtpClient operation gets sent back (or invoked in the Client management module) to say the response was discarded. The one exception is when the Request was multicast. Then nothing happens. After that, once the timeout for filtering out duplicates runs out, the CSR goes back to either the AwaitingRequest state or the Processing state. Also, "Ack" here means an acknowledgment that came through a Notify operation.
(Retransmit Forwarded Request and NotifyVmtpClient)
Request/
Ack/
+Timeout+
V |
+-|-------^-+
| |
+-Time-| Forwarded |<-------------+
| out +-----------+ |
| |
| (Retransmit Response) |
| Request |
V Ack |
| +-Timeout-+ |
| V | |
+---------+ Ack/ +|---------^+ |
+-Time-|Response |<-Timeout--| Responded | |
| out |Discarded| +----^------+ |
| +---------+ | |
| +------------+ | |
| | |->-Send Response-+ |
| | |->-forward Request--------+
+->| Processing |<----------------------+
| | |<----------------+ |
| | |<---| | |
| +-|--------^-+ | Last |
| Receive | | Request |
| | Timeout Single Packet |
| | | Packet | Timeout
| | | Request ^ ^
| | | ^ +|-----|--+
| +-V--------|-+ | |Receiving|<-+Time
+->| Awaiting |->--+->Request->| Request |--+ out
| Request | | (multi- +---------+
+------|-----+ ^ packet)
Request |
| Response
Send Probe to
| Probe
+---V----+ |
|Awaiting| ^
|Response|-->--+
|to Probe|
+--------+
Figure 5-1: Remote Client State TransitionsNext, a Request that gets forwarded leaves the CSR in the Forwarded state. In that state, the forwarded Request is retransmitted now and then, while the CSR waits for NotifyRemoteClient operations back from the Server it was forwarded to. This works a lot like the Client behavior in the AwaitingResponse state 🔁 Here, a NotifyRemoteClient from this Server either acknowledges the Request, asks for it to be retransmitted, or reports an error. If the Client sends the Request again, a NotifyVmtpClient gets returned to it when APG is set. The CSR leaves the Forwarded state after timing out with no NotifyRemoteClient operations coming from the forward Server. It also leaves when a NotifyRemoteClient shows up saying the forward Server has sent a Response and gotten an acknowledgement for it. Then it enters the ResponseDiscarded state.
One more thing: if a new Request arrives from the same Client, it aborts the current transaction no matter what state that transaction is in, and starts a new transaction. The exception is when the new Request is part of a run of message transactions. In that case, the handling follows the state diagram, except the new Request doesn't get Processed until a response has been sent for the previous transaction 🫶
5.4. User Interface 💅
How the RPC or user interface to VMTP works is up to each implementation. It might use system calls, functions, or some other mechanism. The list below is meant to suggest the basic functionality that should be available ✨
AcceptMessage( reqmcb, segptr, segsize, client, transid, timeout )AcceptMessage accepts a new Request message into the specified reqmcb area, putting the segment data, if there is any, into the area described by segptr and segsize. It returns the Server in the entityId field of the reqmcb, and the actual segment size in the segsize parameters. It also returns the Client and Transaction for this message transaction in the matching parameters. This procedure supports message semantics for request processing. When a server process runs this call, it blocks until a Request message has been queued for the server. AcceptMessage returns after the specified timeout period if no message has come in by then.
RespondMessage( responsemcb, client, transid, segptr )RespondMessage responds to the client with the specified response message and segment, again using message semantics.
RespondCall( responsemcb, segptr )RespondCall responds to the client with the specified response message and segment, using remote procedure call semantics. This procedure does not return. The lightweight process that runs this procedure is matched to a stack, program counter, segment area and priority, using the information provided in a ModifyService call, as specified in Appendix III.
ForwardMessage( requestmcb, transid, segptr, segsize, forwardserver )ForwardMessage forwards the client to the specified forwardserver with the request specified in mcb.
ForwardCall( requestmcb, segptr, segsize, forwardserver )ForwardCall forwards the client transaction to the specified forwardserver with the request specified by requestmcb. This procedure does not return.
GetRemoteClientId()GetRemoteClientId returns the entityId for the remote client on whose behave the process is executing. This is only applicable in the procedure call model of request handling.
GetForwarder( client )GetForwarder returns the entity that forwarded this Request, if there is one.
GetProcess( client )GetProcess returns an identifier for the process associated with this client entity-id.
GetPrincipal( client )GetPrincipal returns the principal associated with this client entity-id.
5.5. Event Processing 🔥
These are the events that may occur in VMTP servers:
- User Requests
- Receive
- Respond
- Forward
- GetForwarder
- GetProcess
- GetPrincipal
- Packet Arrival
- Request Packet
- Management Operations
- NotifyVmtpServer
- Timeouts
- Client State Record Timeout
The handling of each of these events is covered in detail in the subsections that follow. The conventions from the previous chapter still apply, including the use of the various subroutines in the descriptions. 🧚
5.6. Server User-invoked Events 💜
5.6.1. Receive 📬
Okay so this is the server's "any new requests for me?" moment 💌 Here's the pseudocode:
AcceptMessage(reqmcb, segptr, segsize, client, transid, timeout)
Locate server's request queue.
if request is queued then
Remember CSR associated with this Request.
return Request in reqmcb, segptr and segsize
and client and transaction id.
Wait on server's request queue for next request
up time timeout seconds.
end ReceiveCallNotes:
- If a multi-packet Request is partially received at the time of the AcceptMessage, the process waits until it completes. ⏳
- A process that accepts a Request as a lightweight thread acts pretty much the same, except it runs using the Request data as if it were logically part of the requesting Client process. 🧵✨
5.6.2. Respond 💌
RespondCall is one case of the Respond transmission procedure, and RespondMessage is basically its cousin. Let's see the main act 💅
RespondCall( responsemcb, responsesegptr )
Locate csr for this client.
Check segment data accessible, if any
if local client then
Handle locally
return
endif
if responsemcb.Code is RESPONSE_DISCARDED then
Mark as RESPONSE_DISCARDED
return
SendPacketGroup( csr )
set csr.State to Responded.
if DGM reply then { Idempotent }
release segment data
Timeout( csr, TS4(csr.Client), FreeCsr );
else { Await acknowledgement or new Request else ask for ack. }
Timeout( csr, TS5(csr.Client), RemoteClientTimeout )
end RespondCallNotes:
- RespondMessage is similar, except the Server process has to stay synchronized with the release of the segment data (if there is any) 🔒
- A non-idempotent Response with segment data goes out first without asking for an acknowledgement. If no acknowledgement or new Request shows up from the client in the meantime, the Response gets retransmitted after TS5(client) time. That's the point where the APG bit gets sent. 🔁
- The MCB of the Response gets buffered in the client CSR, which hangs around for TS4 seconds. That's long enough to filter out old duplicates 👀 The segment data (if any) has to be kept fully intact until either (1) after transmission, if it's idempotent, or (2) after it's acknowledged or a timeout has happened, if it's not idempotent. Techniques like copy-on-write might be used to keep a copy of the Response segment data without paying the cost of an actual copy. 🧠
5.6.3. Forward 🦋
Forwarding is logically starting a brand new message transaction between the Server (who is now acting as a Client) and the server the Request gets forwarded to. When that second server sends back a Response, the very same Response is immediately handed back to the Client. VMTP keeps these semantics intact and also throws in some performance upgrades in certain cases 🚀
ForwardCall( req, segptr, segsize, forwardserver )
Locate csr for this client.
Check segment data accessible, if any
if local client or Request was multicast or secure
or csr.ForwardCount == 15 then
Handle as a new Send operation
return
if forwardserver is local then
Handle locally
return
Set csr.funccode to Request
Increment csr.ForwardCount
Set csr.State to Responded
SendPacketGroup( csr ) { To ForwardServer }
Timeout( csr, TS4(csr.Client), FreeAlien )
end ForwardCallNotes:
- A Forward is logically a new call or message transaction. It has to be really implemented as a new message transaction if the original Request was multicast or secure. There's an optional extra refinement too: it can be used with a secure message transaction when the Server and ForwardServer are the same principal and the Request was not multicast. 🔐
- A Forward operation is never handled as an idempotent operation, because that would require knowing the ForwardServer will treat the forwarded operation as idempotent too 🙈 So a Forward operation that includes a segment should set APG on the first transmission of the forwarded Request, so it gets an acknowledgement for that data. Once the acknowledgement arrives, the forwarding Server can toss the segment data and keep only the basic CSR, which handles any retransmissions from the Client. 🗑️
5.6.4. Other Functions 🧚
GetRemoteClient is just a simple local query of the CSR. GetProcess and GetPrincipal pull their info from the CSR too. A server module may put off the Probe callback to the Client and wait to get that information until the Server actually requests it. This works as long as it isn't using secure communication and duplicate suppression is good enough without the callback. GetForwarder is implemented as a callback to the Client, using a GetRequestForwarder VMTP management operation. More management procedures for VMTP are described in Appendix III 📚
5.7. Request Packet Arrival 📦
The basic packet reception works just like what was described for the Client routines. A Request packet gets handled by the procedure HandleRequest, so let's look at that one 👑
HandleRequest( csr, p, psize )
if LocalClient(csr) then
{ Forwarded Request on local Client }
if csr.LocalTransaction != p.Transaction then return
if csr.State != AwaitingResponse then return
if p.ForwardCount < csr.ForwardCount then
Discard Request and return.
Find a CSR for Client as a remote Client.
if not found then
if packet group complete then
handle as a local message transaction
return
Allocate and init CSR
goto newTransaction
{ Otherwise part of current transaction }
{ Handle directly below. }n
if csr.RemoteTransaction = p.Transaction then
{ Matches current transaction }
if OldForward(p.ForwardCount,csr.ForwardCount) then
return
if p.ForwardCount > csr.ForwardCount then
{ New forwarded transaction }
goto newTransaction
{ Otherwise part of current transaction }
if csr.State = ReceivingRequest then
if new segment data then retain in CSR segment area.
if Request not complete then
Timeout( csr, TS1(p.Client), RemoteClientTimeout )
return;
endif
goto endPacketGroup
endif
if csr.State is Responded then
{ Duplicate }
if csr.Code is RESPONSE_DISCARDED
and Multicast(p) then
return
endif
if not DGM(csr) then { Not idempotent }
if SegmentData(csr) then set APG
{ Resend Response or Request, if Forwarded }
SendPacketGroup( csr )
timeout=if SegmentData(csr) then TS5(csr.Client)
else TS4(csr.Client)
Timeout( csr, timeout, RemoteClientTimeout )
return
{ Else idempotent - fall thru to newTransaction }
else { Presume it is a retransmission }
NotifyClient( csr, p, OK )
return
else if OldTransaction(csr.RemoteTransact,p.Transaction) then
return
{ Otherwise, a new message transaction. }
newTransaction:
Abort handling of previous transactions for this Client.
if (NSRset(p) or NERset(p)) and NoStreaming then
NotifyClient( csr, p, STREAMING_NOT_SUPPORTED )
return
| if NSRset(p) then { Streaming }
| { Check that consecutive with previous packet group }
| Find last packet group CSR from this client.
| if p.Transaction not lastcsr.RemoteTransaction+1 mod 2**32
| and not STIset(lastcsr) or
| p.Transaction not lastcsr.RemoteTransaction+256 mod **32
| then
| { Out of order packet group }
| NotifyClient(csr, p, BAD_TRANSACTION_ID )
| return
| endif
| if lastcsr not completed then
| NotifyClient( lastcsr, p, RETRY )
| endif
| if lastcsr available then use it for this packet group
| else allocate and initialize new CSR
| if CMG(lastcsr) then
| Add segment data to lastcsr Request
| Keep csr as record of this packet group.
| Clear lastcsr.VerifyInterval
| endif
| else { First packet group }
if MultipleRemoteClients(csr) then ScavengeCsrs(p.Client)
Set csr.RemoteTransaction, csr.Priority
Copy message and segment data to csr's segment area
and set csr.PacketDelivery to that delivered.
Clear csr.PacketDelivery
Clear csr.VerifyInterval
SaveNetworkAddress( csr, p )
endif
if packetgroup not complete then
Timeout( csr, TS3(p.Client), RemoteClientTimeout )
return;
endif
endPacketGroup:
{ We have received complete packet group. }
if APG(p) then NotifyClient( csr, p, OK )
endif
| if NERset(p) and CMG(p) then
| Queue waiting for continuation packet group.
| Timeout( csr, TS3(csr.Client), RemoteClientTimeout )
| return
| endif
{ Deliver request message. }
if GroupId(csr.Server) then
For each server identified by csr.Server
Replicate csr and associated data segment.
if CMDset(csr) and Server busy then
Discard csr and data
else
Deliver or invoke csr for each Server.
if not DGMset(csr) then queue for Response
else Timeout( csr, TS4(csr.Client), FreeCsr )
endfor
else
if CMDset(csr) and Server busy then
Discard csr and data
else
Deliver or invoke csr for this server.
if not DGMset(csr) then queue for Response
else Timeout( csr, TS4(csr.Client), FreeCsr )
endif
end HandleRequestNotes:
- A Request that specifies a Client which is a local entity should be a Request that a remote server forwarded to a local Server. 🏠
- Another way to handle a Request sent to a group, when there are multiple local group members, is to create a remote CSR for each group member when the first packet arrives. Then you deliver a copy of each packet to each of those remote CSRs as the packets come in. 👯
5.8. Management Operations 🫶
VMTP uses management operations, which get invoked as remote procedure calls, to effectively acknowledge packet groups and request retransmissions. The routine below is invoked by the Server's management module when the Client asks for it.
NotifyVmtpServer(server,clientId,transact,delivery,code)
Find csr with same RemoteTransaction and RemoteClient
as clientId and transact.
if not found or csr.State not Responded then return
if DGMset(csr) then
if transmission of Response in progress then
Abort transmission
if code is migrated then
restart transmission with new host addr.
if Retry then Report protocol error
return
endif
select on code
case RETRY:
if csr.RetransCount > MaxRetrans(clientId) then
if response data segment then
Discard data and mark as RESPONSE_DISCARDED
| if NERset(csr) and subsequent csr then
| Deallocate csr and use later csr for
| future duplicate suppression
| endif
return
endif
increment csr.RetransCount
Set csr.TransmissionMask to missing segment blocks,
as specified by delivery
SendPacketGroup( csr )
Timeout( csr, TS3(csr.Client), RemoteClientTimeout )
case BUSY:
if csr.TimeLimit exceeded then
if response data segment then
Discard data and mark as RESPONSE_DISCARDED
| if NERset(csr) and subsequent csr then
| Deallocate csr and use later csr for
| future duplicate suppression
| endif
endif
endif
Set csr.TransmissionMask for full retransmission
Clear csr.RetransCount
Timeout( csr, TS3(csr.Server), RemoteClientTimeout )
return
case ENTITY_MIGRATED:
Get new host address for entity
Set csr.TransmissionMask for full retransmission
Clear csr.RetransCount
SendPacketGroup( csr )
Timeout( csr, TS3(csr.Server), RemoteClientTimeout )
return
case default:
Abort transmission of Response if in progress.
if response data segment then
Discard data and mark as RESPONSE_DISCARDED
if NERset(csr) and subsequent csr then
Deallocate csr and use later csr for
future duplicate suppression
endif
return
endselect
end NotifyVmtpServerNotes:
- A NotifyVmtpServer operation requesting retransmission of the Response is acceptable only if the Response was not idempotent. When the Response is idempotent, the Client must be prepared to retransmit the Request to effectively request retransmission of the Response. 🔁✨
- A NotifyVmtpServer operation may be received while the Response is being transmitted. If an error return, as an efficiency, the transmission should be aborted, as suggested when the Response is a datagram. 🦋
- A NotifyVmtpServer operation indicating OK or an error allows the Server to discard segment data and not provide for subsequent retransmission of the Response. 💅
5.8.1. HandleRequestNoCSR
okay so, when a Request comes in from a Client the node has zero CSR for 😭, the node makes a brand new CSR for them and then calls back to the Client's VMTP management module to grab the Principal, Process and other info that goes with them 🫶. It also double-checks that the TransactionId is correct so duplicates get filtered out 💅✨.
HandleRequestNoCSR( p, psize )
| if Secure(p) then
| Allocate and init CSR
| SaveSourceHostAddr( csr, p )
| ProbeRemoteClient( csr, p, AUTH_PROBE )
| if no response or error then
| delete CSR
| return
| Decrypt( csr.Key, p, psize )
| if p.Checksum not null then
| if not VerifyChecksum(p, psize) then return;
| if OppositeByteOrder(p) then ByteSwap( p, psize )
| if psize not equal sizeof(VmtpHeader) + 4*p.Length then
| NotifyClient(NULL, p, VMTP_ERROR )
| return
| HandleRequest( csr, p, psize )
| return
if Server does not exist then
NotifyClient( csr, p, NONEXISTENT_ENTITY )
return
endif
if security required by server then
NotifyClient(csr, p, SECURITY_REQUIRED )
return
endif
Allocate and init CSR
SaveSourceHostAddr( csr, p );
if server requires Authentication then
ProbeRemoteClient( csr, p, AUTH_PROBE )
if no response or error then
delete CSR
return
endif
{ Setup immediately as a new message transaction }
set csr.Server to p.Server
set csr.RemoteTransaction to p.Transaction-1
HandleRequest( csr, p, psize )
endifNotes:
- A Probe request is always handled as a Request not requiring authentication so it never generates a callback Probe to the Client. 🙈
- If the Server host retains remote client CSR's for longer than the maximum packet lifetime and the Request retransmission time, and the host has been running for at least that long, then it is not necessary to do a Probe callback unless the Request is secure. A Probe callback can take place when the Server asks for the Process or PrincipalId associated with the Client. 🧚
5.9. Timeouts
the server must implement a timeout for remote client CSRs. There is a timeout for each CSR in the server. ⏰💜
RemoteClientTimeout( csr )
select on csr.State
case Responded:
if RESPONSE_DISCARDED then
mark as timed out
Make a candidate for reuse.
return
if csr.RetransCount > MaxRetrans(Client) then
discard Response
mark CSR as RESPONSE_DISCARDED
Timeout(csr, TS4(Client), RemoteClientTimeout)
return
increment csr.RetransCount
{ Retransmit Response or forwarded Request }
Set APG to get acknowledgement.
SendPacketGroup( csr )
Timeout( csr, TS3(Client), RemoteClientTimeout )
return
case ReceivingRequest:
if csr.RetransCount > MaxRetrans(csr.Client)
or DGMset(csr) or NRTset(csr) then
Modify csr.segmentSize and csr.MsgDelivery
to indicate packets received.
if MDMset(csr) then
Invoke processing on Request
return
else
discard Request and reuse CSR
(Note: Need not remember Request discarded.)
return
increment csr.RetransCount
NotifyClient( csr, p, RETRY )
Timeout( csr, TS3(Client), RemoteClientTimeout )
return
default:
Report error - invalid state for RemoteClientTimeout
endselect
end RemoteClientTimeoutNotes:
- When a CSR in the Responded state times out after discarding the Response, it can be made available for reuse, either by the same Client or a different one. The CSR should be kept available for reuse by the Client for as long as possible to avoid unnecessary callback Probes. 🌸
6. Concluding Remarks
this document is basically a snapshot of where the VMTP design is right now 📸. We're still running a bunch of experimental implementations to poke at and refine every part of the protocol 🧪. Preliminary implementations are already running in the UNIX 4.3BSD kernel and in the V kernel 🔥.
Lots of stuff is still being discussed and explored, and honestly it's giving work-in-progress 🙈. First up, the size of the checksum field and the algorithm used to calculate it are still up for debate. The author thinks the conventional 16-bit checksum that TCP and IP use is too weak for future high-speed networks, and argues for at least a 32-bit checksum 😤. The catch is that there's limited theory on checksum algorithms that are actually suitable for calculating in software 💀.
The streaming facilities of VMTP are still being built, and this one is expected to be a big deal for wide-area, long-delay communication 🌈✨.
I. Standard VMTP Response Codes
the numeric values of the response codes used in VMTP are listed below, no cap 🧾💖.
0 OK
1 RETRY
2 RETRY_ALL
3 BUSY
4 NONEXISTENT_ENTITY
5 ENTITY_MIGRATED
6 NO_PERMISSION
7 NOT_AWAITING_MSG
8 VMTP_ERROR
9 MSGTRANS_OVERFLOW
10 BAD_TRANSACTION_ID
11 STREAMING_NOT_SUPPORTED
12 NO_RUN_RECORD
13 RETRANS_TIMEOUT
14 USER_TIMEOUT
15 RESPONSE_DISCARDED
16 SECURITY_NOT_SUPPORTED
17 BAD_REPLY_SEGMENT
18 SECURITY_REQUIRED
19 STREAMED_RESPONSE
20 TOO_MANY_RETRIES
21 NO_PRINCIPAL
22 NO_KEY
23 ENCRYPTION_NOT_SUPPORTED
24 NO_AUTHENTICATOR
25-63 Reserved for future VMTP assignment.Other values of the codes are available for use by higher level protocols. Separate protocol documents will specify further standard values. 📚
Applications are free to use values starting at 0x00800000 (hex) for application-specific return values. 🎀
II. VMTP RPC Presentation Protocol
for complete generality, the mapping of the procedures and the parameters onto VMTP messages should be defined by a RPC presentation protocol. In the absence of an accepted standard protocol, we define an RPC presentation protocol for VMTP as follows. 🧩
Each procedure is assigned an identifying Request Code. The Request code serves effectively the same as a tag field of variant record, identifying the format of the Request and associated Response as a variant of the possible message formats. 🏷️
The format of the Request for a procedure is its Request Code followed by its parameters sequentially in the message control block until it is full. 📦
The remaining parameters are sent as part of the message segment data formatted according to the XDR protocol (RFC ??). In this case, the size of the segment is specified in the SegmentSize field. 🍓
The Response for a procedure consists of a ResponseCode field followed by the return parameters sequentially in the message control block, except if there is a parameter returned that must be transmitted as segment data, its size is specified in the SegmentSize field and the parameter is stored in the SegmentData field. 💌
Attributes associated with procedure definitions should indicate the Flags to be used in the RequestCode. Request Codes are assigned as described below. 👑
II.1. Request Code Management
Request codes are divided into Public Interface Codes and application-specific, according to whether the PIC value is set. An interface is a set of request codes representing one service or module function. A public interface is one that is to be used in multiple independently developed modules. In VMTP, public interface codes are allocated in units of 256 structured as 🧱
+-------------+----------------+-------------------+
| ControlFlags| Interface | Version/Procedure |
+-------------+----------------+-------------------+
8 bits 16 bits 8 bitsAn interface is free to allocate the 8 bits for version and procedure as desired. For example, all 8 bits can be used for procedures. A module requiring more than 256 Version/Procedure values can be allocated multiple Interface values. They need not be consecutive Interface values. 🦋
III. VMTP Management Procedures
Standard procedures are defined for VMTP management, including creation, deletion and query of entities and entity groups, probing to get information about entities, and updating message transaction information at the client or the server. 🛠️✨
The procedures are implemented by the VMTP manager that constitutes a portion of every complete VMTP module. Each procedure is invoked by sending a Request to the VMTP manager that handles the entity specified in the operation or the local manager. The Request sent using the
So the request goes out as a totally normal Send operation, but the Server is the well-known entity group VMTP_MANGER_GROUP ✨. The CoResident Entity mechanism is what steers it to the exact manager that should handle the Request 🎯. (The ProbeEntity operation gets multicast to VMTP_MANAGER_GROUP if the host address for the entity isn't known locally, and that host address is taken as the host address of the responder 🦋. For every other operation, a ProbeEntity is used to find the host address if it isn't already known 💅.) Specifying co-resident entity 0 means "the co-resident with the invoking process" 🧚. The co-resident entity identifier can also name a group, and then the Request goes to all managers with members in that group 👑.
Here are the standard procedures, with their RequestCode and parameters and what each one means 💜. The RequestCode range 0xVV000100 to 0xVV0001FF is reserved for the VMTP management routines, where VV is any choice of control flags with the PIC bit set. Each procedure sets the flags below as it needs them 🌸.
0x05000101 - ProbeEntity(CREntity, entityId, authDomain) -> (code,
<staterec>)This one asks for and returns info on the specified entity in the specified authDomain 🔍. The Request goes to the VMTP management module coresident with CREntity. If the info can't be provided in that authDomain, you get an error return 🙅♀️. The <staterec> that comes back has these fields:
- Transaction identifier: The current or next transaction identifier being used by the probed entity.
- ProcessId: 64 bits: Identifier for the client process. Its meaning is specified as part of the Domain definition.
- PrincipalId: The identifier for the principal or account associated with the process specified by ProcessId. Its meaning is specified as part of the Domain definition.
- EffectivePrincipalId: The identifier for the principal or account associated with the Client port. It may be different from the PrincipalId, especially in a nested call. Its meaning is specified as part of the Domain definition.
The code field tells you whether this is an error response or not 🚦. The codes and what they mean:
OK: No error. Probe was completed OK. ✅NONEXISTENT_ENTITY: Specified entity does not exist. 👻ENTITY_MIGRATED: The entity has migrated and is no longer at the host to which the request was sent. 🚚NO_PERMISSION: Entity has refused to provide ProbeResponse. 🔒VMTP_ERROR: The Request packet group was in error relative to the VMTP protocol specification. 💀"default": Some type of error. Discard the ProbeResponse. 🗑️
0x0D000102 - AuthProbeEntity(CREntity,entityId,authDomain,randomId) ->
(code,ProbeAuthenticator,EncryptType,EntityAuthenticator)This one requests authentication of the entity specified by entityId from the VMTP manager coresident with CREntity, in the authDomain authentication domain 🛡️. It returns the information in the return parameters. The fields are set the same as the basic ProbeResponse, except as noted below 🎀.
- ProbeAuthenticator: 20 bytes made up of the EntityId, the randomId, and the probed Entity's current Transaction value, plus a 32-bit checksum for these two fields (checksummed using the standard packet Checksum algorithm). All of it is encrypted with the Key supplied in the Authenticator 🔐.
- EncryptType: An identifier for the variant of encryption method the probed Entity is using for packets it transmits and packets it is able to receive. (See Appendix V.) The high-order 8 bits of the EncryptType hold the XOR of the 8 octets of the PrincipalId associated with the private key used to encrypt the EntityAuthenticator. The requestor or Client uses this value as a hint for finding the key to decrypt the authenticator 🔑.
- EntityAuthenticator (returned as segment data): The ProcessId, PrincipalId, and EffectivePrincipal associated with the ProbedEntity, plus the private encryption/decryption key and its lifetime limit to be used for communication with the Entity. The authenticator is encrypted with a private key associated with the Client entity, so that it can be neither read nor forged by a party the Client Entity doesn't trust 🕵️♀️. The format of the Authenticator in the message segment is shown in detail in Figure III-1 📐.
- Key: 64 bits: The encryption key used for encrypting and decrypting packets sent to and received from the probed Entity. This is the "working" key for packet transmissions. VMTP only uses private
+-----------------------------------------------+
| ProcessId (8 octets) |
+-----------------------------------------------+
| PrincipalId (8 octets) |
+-----------------------------------------------+
| EffectivePrincipalId (8 octets) |
+-----------------------------------------------+
| Key (8 octets) |
+-----------------------------------------------+
| KeyTimeLimit |
+-----------------------------------------------+
| AuthDomain |
+-----------------------------------------------+
| AuthChecksum |
+-----------------------------------------------+
Figure III-1: Authenticator Formatkey encryption for data transmission. 🎟️
- KeyTimeLimit: 32 bits: The time in seconds since Dec. 31st, 1969 GMT at which one should stop using the Key ⏳.
- AuthDomain: 32 bits: The authentication domain in which to interpret the principal identifiers. This may differ from the authDomain specified in the call if the Server can't provide the authentication information in the request domain 🌍.
- AuthChecksum: 32 bits: Contains the checksum (using the same Checksum algorithm as for packets) of KeyTimeLimit, Key, PrincipalId, and EffectivePrincipalId ✅.
Notes:
- An authentication Probe Request and Response are generally sent unencrypted, because they're used before there's a secure channel 😬. So specific fields, or groups of fields, are checksummed and encrypted to prevent unauthorized modification or forgery. In particular, the
ProbeAutenticatoris checksummed and encrypted with the Key. - The
ProbeAutenticatorauthenticates the Response as responding to the Request when its EntityId, randomId, and Transaction values match those in the Probe request 🤝. TheProbeAutenticatoris bound to theEntityAutenticatorby being encrypted with the private Key contained in that authenticator. - The authenticator is encrypted so that it can be decrypted by a private key known to the Client. This authenticator is presumably obtained from a key distribution center that the Client trusts 🏰. The AuthChecksum prevents undetected modifications to the authenticator.
0x05000103 - ProbeEntityBlock( entityId ) -> ( code, entityId )This checks whether the block of 256 entity identifiers associated with this entityId is in use 🧱. The entityId returned should match the one being queried. If it doesn't, ignore the return value and redo the operation 🔁.
0x05000104 - QueryVMTPNode( entityId ) -> (code, MTU, flags, authdomain,
domains, authdomains, domainlist)This queries the VMTP management module for entityId to get various module- or node-wide parameters 📊. These include:
- MTU: The maximum transmission unit, or packet size, handled by this node. 📦
- flags: Zero or more of the following bit fields 🎛️:
1: Handles streamed Requests.2: Can issue streamed message transactions for clients.4: Handles secure Requests.8: Can issue secure message transactions.
The authdomain indicates the primary authentication domain supported 🪪. The domains and authdomains parameters give the number of entity domains and authentication domains supported by this node. Those domains are listed in the data segment parameter domainlist if either parameter is non-zero. (All the entity domains come before the authentication domains in the data segment.) 🧁
Okay bestie, now we're getting into the management procedures, and these are giving main character energy 💅✨. Every procedure gets its RequestCode and its parameters, and I'll keep the exact signatures in code so nothing gets lost in translation 🎀.
0x05000105 - GetRequestForwarder( CREntity, entityId1 ) -> (code,
entityId2, principal, authDomain)So GetRequestForwarder is the one that asks for the forwarding server's entity identifier and principal, the stuff for whoever forwards requests for entityId1 🦋. And if you want the local VMTP management module, CREntity should be zero 🧚.
0x05000106 - CreateEntity( entityId1 ) -> ( code, entityId2 )CreateEntity makes a brand new entity and hands you its identifier back in entityId2 🌸. The new entity is created local to whatever entityId1 points at, or local to you (the requestor) if entityId1 is 0 🫶.
0x05000107 - DeleteEntity( entityId ) -> ( code )DeleteEntity deletes the entity named by entityId, and yes, that can be a group 💀. If it's a group, the deletion is best efforts only, so the client must take additional measures to ensure complete deletion if required. Not gonna lie, that's a whole vibe check 🙈.
0x0D000108 -QueryEntity( entityId ) -> ( code, descriptor )QueryEntity returns a descriptor of entityId inside arg, and arg can be up to segmentSize bytes 📦✨.
0x05000109 - SignalEntity( entityId, arg )->( code )SignalEntity sends the signal in arg to the entity named by entityId 🔥. The arg is 32 bits, so keep that in mind 💜.
0x0500010A - CreateGroup(CREntity,entityGroupId,entityId,perms)->(code)CreateGroup asks the VMTP manager local to CREntity to make a new entity group using entityGroupId, with entityId as the first member and perms as the permissions 👑. Perms is a 32-bit field that gets explained later. The invoker gets registered as a manager of the new group, which means it can add or remove members. (Normally CREntity is 0, meaning the VMTP manager local to the requestor 🍓.)
0x0500010B - AddToGroup(CREntity, entityGroupId, entityId,
perms)->(code)AddToGroup asks the VMTP manager local to CREntity to add entityId to entityGroupId with the given permissions 🎀. If entityGroupId is a restricted group, the invoker has to have permission to add members. That means either it's a manager of the group, or it was added with the required permissions 🔒. If CREntity is 0, the local VMTP manager checks permissions, then forwards the request with CREntity set to entityId, and the entityId field set to a digital signature of the Request by the VMTP manager 🖋️. That signature certifies the Client has the permissions the Request needs. Lowkey a clever loop 🧠. (If entityGroupId is an unrestricted group, the Request can go straight to the handling VMTP manager by setting CREntity to entityId 🦋.)
0x0500010C - RemoveFromGroup(CREntity, entityGroupId, entityId)->(code)RemoveFromGroup asks the VMTP manager local to CREntity to remove entityId from the group named by entityGroupId 🙅♀️. Normally CREntity is 0, meaning the VMTP manager local to the requestor. If CREntity is 0, the local VMTP manager checks permissions and forwards the request with CREntity set to entityId, and the entityId field set to a digital signature of the Request by the VMTP manager, certifying the Client has the permissions required 🫶.
0x0500010D - QueryGroup( entityId )->( code, record )...QueryGroup returns info on the specified entity 🔍. Each responding VMTP manager sends back a (code, record) Response. The record is formatted as (memberCount, member1, member2, ...) 📝. These Responses come back on a best efforts basis, so there's no guarantee you'll hear from every manager that has members in the group 😭.
0x0500010E - ModifyService(entityId,flags,count,pc,threadlist)->(code,
count)ModifyService changes the service attached to the entity named by entityId 🔧. The flags can pick a message service model, in which case the count parameter is the max number of queued messages you want, and the returned count tells you how many queued messages are actually allowed 💌. Or the flags can pick the RPC thread service model, in which case count threads are requested. Each thread gets an initial program counter (pc), plus a stack, priority, and message receive area, all taken from the threadlist 🧵. The threadlist is made of count records shaped like this:
(priority,stack,stacksize,segment,segmentsize)Each record gets assigned to one of the threads 🌈. Here are the flags you can use for the flags parameter:
1 THREAD_SERVICE - otherwise the message
model.
2 AUTHENTICATION_REQUIRED - Sent a Probe
request to determine principal
associated with the Client, if not
known.
4 SECURITY_REQUIRED - Request must be
encrypted or else reject.
8 INCREMENTAL - treat the count value as
an increment (or decrement) relative to
the current value rather than an
absolute value for the maximum number of
queued messages or threads.Quick flag breakdown, no cap 💅. Flag 1 picks the thread model, and if it's not set you get the message model 🦋. Flag 2 means you need authentication, so a Probe request goes out to figure out which principal the Client is, if it's not already known 🔐. Flag 4 means the Request must be encrypted or it gets rejected 🚫. Flag 8 makes count an increment or decrement relative to the current value, instead of an absolute number for the max queued messages or threads 📈.
In the thread model, count must be a positive increment or else 0, and 0 disables the service. Only a count of 0 terminates currently queued requests or in-progress request handling. Be careful with that one, bestie, it's serious 😭🚨.
0x4500010F - NotifyVmtpClient(client,cntrl,recSeq,transact,delivery,code)->()
okay so this one updates the state for a transaction 💖 It's picked out by the client entity identifier and the transact transaction identifier, and it's normally only used by another VMTP management module 🫶 Also, it's a datagram operation 📦
Here's what the parameters are giving:
ctrl: a 32-bit value that matches the 4th 32-bit word of the VMTP header of the Response packet that would be sent back for the Request being answered. So the control flags, ForwardCount, RetransmitCount and Priority fields all line up with the Request ✨ (the NRS flag is set if the receiveSeqNumber field is used). The PGCount subfield says how many earlier Request packet groups this Notify is acknowledging. And the bit fields that are reserved in this word in the header stay reserved here too, and must be zero 🙅♀️recSeq: the sequence number of reception at the Server if the NRS flag is set inctrl. Otherwise it's reserved and zero. This is for sender-based logging of message activity so things can be replayed if something fails. It's an optional facility, so no pressure 🧚delivery: shows which segment blocks of the packet group have been received at the Server 🎀code: tells the client what action to take, which we'll get into below 💅
The VMTP management module should take action on this operation based on the code. Here's the tea:
OK: do nothing right now, keep waiting for the response, but with a reset timer ⏰RETRY: retransmit the Request packet group immediately, with at least the segment blocks the Server failed to receive (the complement of the ones indelivery) 🔁RETRY_ALL: retransmit immediately with at least the segment blocks the Server failed to receive, as shown by thedeliveryfield, plus every packet sent after that which is part of this packet run. This one only applies to streamed message transactions 🌊BUSY: the Server couldn't accept the Request right now. Retry later if you still want the message transaction to happen 🕰️NONEXISTENT_ENTITY: the Server entity you asked for doesn't exist 💀ENTITY_MIGRATED: the Server entity has moved and isn't at the host the request went to anymore. The Server should try to figure out the new host address of the Client using the VMTP management ProbeEntity operation (described earlier) 🧭NO_PERMISSION: the Server hasn't authorized messages from this client 🚫NOT_AWAITING_MSG: the conditional message delivery bit was set on the Request packet group, but the Server wasn't waiting for it, so the Request packet group got discarded 🗑️VMTP_ERROR: the Request packet group was in error compared to the VMTP protocol specification 😭BAD_TRANSACTION_ID: the transaction identifier is old compared to the one the Server is holding for the Client 📜STREAMING_NOT_SUPPORTED: the Server doesn't support multiple outstanding message transactions from the same Client (aka streamed message transactions) 🙅♀️SECURITY_NOT_SUPPORTED: the Request was secure, but this Server doesn't do security 🔓SECURITY_REQUIRED: the Server is refusing the Request because it wasn't encrypted 🔐NO_RUN_RECORD: the Server has no record of previous packets in this run of packet groups. This can happen if the first packet group got lost, or if the current one was sent way later than the last one and the Server already threw out its client state record 🫠
0x45000110 - NotifyVmtpServer(server,client,transact,delivery,code)->()
this is the mirror image of the one above 🪞 It updates the server state for the transaction picked out by client and transact (an entity identifier and a transaction identifier). Again, it's normally only used by another VMTP management module, and it's a datagram operation too 📦
delivery: shows which segment blocks of the Response packet group have been received at the Client 🎀code: tells the Server what action to take, listed below 💅
The VMTP management module should act on this operation based on the code:
OK: the Client is happy with the Response data, so the Server can toss the response data if there is any 🥳RETRY: retransmit the Response packet group immediately with at least the segment blocks the Client failed to receive, as shown by thedeliveryparameter. (Thedeliveryparameter tells you which segment blocks the Client did receive) 🔁RETRY_ALL: retransmit immediately with at least the segment blocks the Client failed to receive, going by the complement ofdelivery. Also retransmit every Response packet group that was sent after the specified packet group 🌊NONEXISTENT_ENTITY: the Client entity you named doesn't exist 💀ENTITY_MIGRATED: the Client entity has moved and isn't at the host the response was sent to anymore 🧭RESPONSE_DISCARDED: the Response was thrown out and the Client doesn't care about it anymore. This can happen if the conditional message delivery bit was set on the Response packet group and the Client wasn't waiting for it, so the group got discarded 🗑️VMTP_ERROR: the Response packet group was in error compared to the VMTP protocol specification 😭
0x41000111 - NotifyRemoteVmtpClient(client,ctrl,recSeq,transact,delivery,code->()
this one is basically NotifyVmtpClient, just without the co-resident addressing 🏡➡️🌍 It's for updating client state that lives remotely when a Request gets forwarded 🛰️
Note the use of the CRE bit in the RequestCodes to route the request to the correct VMTP management module(s) to handle the request 🧚
III.1. Entity Group Management 🎀
okay so an entity in a group has a set of permissions that come with its membership 🫶 They decide whether it can add or remove others, whether it can remove itself, and whether others can remove it from the group. Here's the full permissions list, copied exactly as it's written:
VMTP_GRP_MANAGER 0x00000001 { Manager of group. }
VMTP_REM_BY_SELF 0x00000002 { Can be removed self. }
VMTP_REM_BY_PRIN 0x00000004 { Can be rem'ed by same principal}
VMTP_REM_BY_OTHE 0x00000008 { Can be removed any others. }
VMTP_ADD_PRIN 0x00000010 { Can add by same principal. }
VMTP_ADD_OTHE 0x00000020 { Can add any others. }
VMTP_REM_PRIN 0x00000040 { Can remove same principal. }
VMTP_REM_OTHE 0x00000080 { Can remove any others. }Removing someone from a restricted group is a two-part check 🕵️♀️ The invoker needs permission to remove that entity, and the entity needs permissions that let that entity remove it. In an unrestricted group, only the second condition counts ✨
In a restricted group, a member can only be added by another entity that has permission to add other entities 🚪 The creator of a group gets full permissions on it 👑 And when an entity adds another entity to a group, it can only hand over a subset of its own permissions 🍰
With unrestricted groups, any entity can add itself to the group 🙌 It can also add other entities, as long as the entity being added isn't marked as immune to those requests. (This is an implementation restriction that individual entities can put in place) 🛡️
III.2. VMTP Management Digital Signatures 🔏
As mentioned above, the entityId field of AddToGroup and RemoveFromGroup is used to carry a digital signature. That signature shows the sending kernel already checked that the operation is allowed 💅 The digital signature procedures haven't been defined yet, so for now this field should be set to 0 to mean "no signature" 🫠 That goes after the CREntity parameter is set to the entity the operation is being performed on 🧚
IV. VMTP Entity Identifier Domains 🌈
okay and now for the next big idea 🦋 VMTP lets its endpoints live in several disjoint naming domains, and the
64-bit entity identifier is only unique and meaningful within its domain, bestie 💖 Each domain can pick its own algorithm or mechanism for handing out entity identifiers, but every domain mechanism must ensure uniqueness, stability of identifiers and host independence ✨
IV.1. Domain 1 🎀
For initial use of VMTP, we define the domain with Domain identifier 1 like this 💅
+-----------+----------------+------------------------+
| TypeFlags | Discriminator | Internet Address |
+-----------+----------------+------------------------+
4 bits 28 bits 32 bitsThe Internet address is the Internet address of the host where this entity-id was originally allocated. The Discriminator is an arbitrary value that is unique relative to that Internet host address. The host also has to make sure this identifier doesn't get reused for a long period of time after it becomes invalid. ("Invalid" means that no VMTP module considers it bound to an entity 🙈.) One technique is to use the lower order bits of a 1 second clock. The clock doesn't need to represent real-time, but it must never be set back after a crash. In a simple implementation that uses the low order bits of a clock as the time stamp, generating unique identifiers is limited to no more than 1 per second on average 😭. The type flags were described in Section 3.1.
An entity may migrate between hosts. So an implementation can heuristically use the embedded Internet address to locate an entity, but it should be prepared to maintain a cache of redirects for migrated entities, plus accept Notify operations indicating that migration has occurred 🦋.
Entity group identifiers in Domain 1 come in one of two forms, depending on whether they are well-known or dynamically allocated identifiers. A well-known entity identifier is structured as:
+-----------+----------------+------------------------+
| TypeFlags | Discriminator |Internet Host Group Addr|
+-----------+----------------+------------------------+
4 bits 28 bits 32 bitswith the second high-order bit (GRP) set to 1. This form of entity identifier is mapped to the Internet host group address specified in the low-order 32 bits. The Discriminator distinguishes group identifiers using the same Internet host group. Well-known entity group identifiers should be allocated to correspond to the basic services provided by hosts that are members of the group, not specifically because that service is provided by VMTP 🌸. For example, the well-known entity group identifier for the domain name service should contain as its embedded Internet host group address the host group for Domain Name servers.
A dynamically allocated entity identifier is structured as:
+-----------+----------------+------------------------+
| TypeFlags | Discriminator | Internet Host Addr |
+-----------+----------------+------------------------+
4 bits 28 bits 32 bitswith the second high-order bit (GRP) set to 1. The Internet address in the low-order 32 bits is an Internet address assigned to the host that dynamically allocates this entity group identifier. A dynamically allocated entity group identifier is mapped to Internet host group address 232.X.X.X, where X.X.X are the low-order 24 bits of the Discriminator subfield of the entity group identifier.
We use the following notation for Domain 1 entity identifiers <10> and propose it as a standard convention ✨
<flags>-<discriminator>-<Internet address>where <flags> are [X]{BE,LE,RG,UG}[A]:
X= reservedBE= big-endian entityLE= little-endian entityRG= restricted groupUG= unrestricted groupA= alias
and <discriminator> is a decimal integer and <Internet address> is in standard dotted decimal IP address notation.
Examples:
BE-25593-36.8.0.49 is big-endian entity #25593 created on host
36.8.0.49.
RG-1-224.0.1.0 is the well-known restricted VMTP managers group.
UG-565338-36.8.0.77 is unrestricted entity group #565338 created on host
36.8.0.77.
LEA-7823-36.8.0.77 is a little-endian alias entity #7823 created on host
36.8.0.77.This notation makes it easy to communicate and understand entity identifiers for Domain 1 🍓
_______________
<10> This notation was developed by Steve Deering.The well-known entity identifiers specified to date are:
VMTP_MANAGER_GROUPisRG-1-224.0.1.0: Managers for VMTP operations.VMTP_DEFAULT_BECLIENTisBE-1-224.0.1.0: Client entity identifier to use when a (big-endian) host has not determined or been allocated any client entity identifiers.VMTP_DEFAULT_LECLIENTisLE-1-224.0.1.0: Client entity identifier to use when a (little-endian) host has not determined or been allocated any client entity identifiers.
Note that 224.0.1.0 is the host group address assigned to VMTP and to which all VMTP hosts belong 💜. Other well-known entity group identifiers will be specified in subsequent extensions to VMTP and in higher-level protocols that use VMTP.
IV.2. Domain 3 🧚
Domain 3 is reserved for embedded systems that are restricted to a single network and are independent of IP. Entity identifiers are allocated using the decentralized approach described below. The mapping of entity group identifiers is specific to the type of network being used and is not defined here. In general, there should be a simple algorithmic mapping from entity group identifier to multicast address, similar to the one described for Domain 1. Similarly, the values for default client identifier are specific to the type of network and not defined here.
IV.3. Other Domains
Definition of additional VMTP domains is planned for the future. Requests for allocation of VMTP Domains should be addressed to the Internet protocol administrator 🌈.
IV.4. Decentralized Entity Identifier Allocation
The ProbeEntityBlock operation may be used to determine whether a block of entity identifiers is in use. ("In use" means valid or reserved by a host for allocation.) This mechanism is used to detect collisions in allocation of blocks of entity identifiers as part of the implementation of decentralized allocation of entity identifiers. (Decentralized allocation is used in local domain use of VMTP, such as in embedded systems, see Domain 3.)
Basically, a group of hosts can form a Domain or sub-Domain, a group of hosts managing their own entity identifier space or subspace, respectively. As an example of a sub-Domain, a group of hosts in Domain 1 all identified with a particular host group address can manage the sub-Domain corresponding to all entity identifiers that contain that host group address. The ProbeEntityBlock operation is used to allocate the random bits of these identifiers as follows.
When a host requires a new block of entity identifiers, it selects a new block (randomly or by some choice algorithm) and then multicasts a ProbeEntityBlock request to the members of the (sub-)Domain some R times. If no response is received after R (re)transmissions, the host concludes that it is free to use this block of identifiers. Otherwise, it picks another block and tries again 🔥.
Notes:
- A block of 256 identifiers is specified by an entity identifier with the low-order 8 bits all zero.
- When a host allocates an initial block of entity identifiers (and therefore does not yet have a specified entity identifier to use), it uses
VMTP_DEFAULT_BECLIENT(if big-endian, elseVMTP_DEFAULT_LECLIENTif little-endian) as its client identifier in the ProbeEntityBlock Request and a transaction identifier of 0. As soon as it has allocated a block of entity identifiers, it should use these identifiers for all subsequent communication. The default client identifier values are defined for each Domain. - The set of hosts using this decentralized allocation must not be subject to network partitioning. That is, the R transmissions must be sufficient to ensure that every host sees the ProbeEntityBlock request and (reliably) sends a response. (A host that detects a collision can retransmit the response multiple times until it sees a new ProbeEntityBlock operation from the same host/Client up to a maximum number of times.) For instance, a set of machines connected by a single local network may be able to use this type of allocation.
- To guarantee T-stability, a host must prevent reuse of a block of identifiers if any of the identifiers in the block are currently valid or have been valid less than T seconds previously. To this end, a host must remember recently used identifiers and object to their reuse in response to a ProbeEntityBlock operation.
- Care is required in a VMTP implementation to ensure that Probe operations cannot be discarded due to lack of buffer space or queued or delayed so that a response is not generated quickly. This is required not only to detect collisions but also to provide accurate roundtrip estimates as part of ProbeEntity operations 🫶.
V. Authentication Domains 👑
A VMTP authentication domain defines the format and interpretation for principal identifiers and encryption keys. In particular, an authentication domain must specify a means by which principal identifiers are allocated and guaranteed unique and stable. The currently defined authentication domains are as follows (0 is reserved).
Ideally, all entities within one entity domain are also associated with one authentication domain. However, authentication domains are orthogonal to entity domains. Entities within one domain may have different authentication domains. (In this case, it is generally necessary to have some correspondence between principals in the different domains.) Also, one entity identifier may be associated with multiple authentication domains. Finally, one authentication domain may be used across multiple entity domains.
V.1. Authentication Domain 1
A principal identifier is structured as follows.
+---------------------------+------------------------+
| Internet Address | Local User Identifier |
+---------------------------+------------------------+
32 bits 32 bitsThe Internet Address may specify an individual host (such as a UNIX machine) or may specify a host group address corresponding to a cluster of machines operating under a single administration. In both cases, there is assumed to be an administration associated with the embedded Internet address that guarantees the uniqueness and stability of the User Identifier relative to the Internet address. In particular, that administration is the only one authorized to allocate principal identifiers with that Internet address prefix, and it may allocate any of these identifiers 🦋.
In authentication domain 1, the standard EncryptionQualifiers are:
0 Clear text - no encryption.
1 use 64-bit CBC DES for encryption and decryption.V.2. Other Authentication Domains
Other authentication domains will be defined in the future as needed.
VI. IP Implementation ✨
VMTP is designed to be implemented on the DoD IP Internet Datagram Protocol (although it may also be implemented as a local network protocol directly in "raw" network packets.)
VMTP is assigned the protocol number 81.
With a 20 octet IP header and one segment block, a VMTP packet is 600 octets. By convention, any host implementing VMTP implicitly agrees to accept VMTP/IP packets of at least 600 octets 💅.
VMTP multicast facilities are designed to work with, and have been implemented using, the multicast extensions to the Internet [8] described in RFC 966 and 988. The wide-scale use of full VMTP/IP
depends on the availability of IP multicast in this form 💜
VII. Implementation Notes ✨
okay so now we're in implementation land, bestie 💅 A protocol can look perfect on paper and still be a total disaster in real life, because how well it actually runs depends HEAVILY on how it's coded, not just on the design 😭 One of the big goals of the VMTP effort was to make something that's actually efficient to implement. These notes come from building VMTP in the V distributed system and in the UNIX 4.3 BSD kernel 🧚 Also, heads up: everything here describes a client and server handling just one domain. A multi-domain client or server would just repeat these structures for each domain, though buffer space can be shared 🫶
VII.1. Mapping Data Structures 🗺️
The ClientMap procedure uses a hash table that points to the Client State Record, whether the entity is local or remote 🦋 Check out Figure VII-1:
+---+---+--------------------------+
ClientMap | | x | |
+---+-|-+--------------------------+
| +--------------+ +--------------+
+-->| LocalClient |--->| LocalClient |
+--------------+ +--------------+
| RemoteClient | | RemoteClient |-> ...
+--------------+ +--------------+
| | | |
| | | |
+--------------+ +--------------+
Figure VII-1: Mapping Client Identifier to CSRLocal clients get chained together through the LocalClientLink, and remote ones through the RemoteClientLink, same energy 🔗 Once you find a CSR with the right Entity Id, some field or flag tells you whether it's identifying a local or remote Entity. Hash collisions get handled with the overflow pointers LocalClientLink and RemoteClientLink (not shown) in the CSR for the LocalClient and RemoteClient fields, respectively. Also, plot twist: a CSR representing an RPC request has both a local and a remote entity identifier mapping to the same CSR 👀
The Server named in a Request gets mapped to a server descriptor using the ServerMap (with collisions handled by the overflow pointer.) 💖 The server descriptor is the root of a queue of CSR's for handling requests, plus flags that change how the Request gets handled. Here's the layout:
+-------+---+-------------------------+
ServerMap | | x | |
+-------+-|-+-------------------------+
| +--------------+
| | OverflowLink |
| +--------------+
+-->| Server |
+--------------+
| Flags | Lock |
+--------------+
| Head Pointer |
+--------------+
| Tail Pointer |
+--------------+
Figure VII-2: Mapping Server IdentifiersAnd here are the flags, bestie 🎀
THREAD_QUEUE: the Request gets invoked directly as a remote procedure invocation, instead of being picked up by a server process in the message model ✨AUTHENTICATION_REQUIRED: a Probe request gets sent to figure out which principal goes with the Client, if we don't already know 🕵️SECURITY_REQUIRED: the Request has to be encrypted, or it gets rejected 🔒REQUESTS_QUEUED: the queue has waiting requests in it, instead of free CSR's. This request gets queued up too 🫠SERVER_WAITING: the server is sitting there ready to handle an incoming Request right away, as CMD requires 🚀
Alternatively, you can map Server identifiers to a CSR using the MapToClient mechanism, with a pointer in the CSR referring to the server descriptor (if there is one). This is super attractive if you have client CSR's tied to a service, so it can talk to other services as a client using VMTP 🦋
Finally, a similar structure expands entity group identifiers into the local membership, as shown in Figure VII-3. A group identifier gets hashed to an index in the GroupMap. The list of group descriptors rooted at that index contains a group descriptor for each local member of the group. The flags are the group permissions defined in Appendix III 💅
+-------+---+----------------------------------+
GroupMap | | x | |
+-------+-|-+----------------------------------+
| +--------------+
| | OverflowLink |
| +--------------+
+-->|EntityGroupId |
+--------------+
| Flags |
+--------------+
| Member Entity|
+--------------+
Figure VII-3: Mapping Group IdentifiersAlso, big brain move: the same pool of descriptors could be reused for both server and group descriptors, since they're basically the same size 🧠✨
VII.2. Client Data Structures 📦
Each client entity is represented as a client state record. The CSR holds a VMTP header plus other bookkeeping fields, like the timeout count and the retransmission count (see Section 4.1). There's also a timeout queue, a transmission queue and a reception queue 🌸 Last but not least, there's a ServerHost cache that maps server entity-id records to host address, estimated round trip time, interpacket gap, MTU size and (optimally) estimated processing time for that server entity. Iconic cache behaviour 💜
VII.3. Server Data Structures 🏰
The server keeps a heap of client state records (CSR), one for each (Client, Transaction) pair. (If streams aren't supported, there's, at worst, a CSR per Client that the server has talked to recently.) Each CSR has a VMTP header plus bookkeeping fields, including the timeout count and the retransmission count. The server keeps a hash table mapping Client to CSR, along with the transmission, timeout and reception queues. In a VMTP module that does both the client and server jobs, the same timeout queue and transmission queue get used for both 🫶
VII.4. Packet Group transmission 📤
The procedure SendPacketGroup( csr ) transmits the packet group named by the record CSR. It does the following, in order:
- Chops the segment data, if there is any, into packets. (Note, segment data flagged by SDA bit.)
- Tweaks the VMTP header for each packet as needed, e.g. changing the delivery mask.
- Computes the VMTP checksum.
- Encrypts the right portion of the packet, if required.
- Wraps the packet with the network-level header and trailer, using the network address from the ServerHost cache, or from the responding CSR.
- Transmits the packet with the interpacket gap from the cache. This might involve round-robin scheduling between hosts, plus delaying transmissions slightly.
- Invokes the finish-up procedure the CSR record specifies, which wraps up the processing. Usually this finish-up procedure adds the record to the timeout queue with the appropriate timeout queue.
The CSR has a 32-bit transmission mask that says which portions of the segment to send 🎯 The SendPacketGroup procedure is assumed to handle queuing at the network transmission queue, in priority order based on the priority field in the CSR record. (This priority might show up in network transmission behaviour on networks that support priority.)
The SendPacketGroup procedure only looks at these fields of a CSR:
- Transmission mask
- FuncCode
- SDA
- Client
- Server
- CoResidentEntity
- Key
And it modifies these ones:
- Length
- Delivery
- Checksum
If encrypted transmission is happening, it encrypts the entire packet, excluding the Client field and the following 32-bits 🔐
If the packet group is a Response (i.e. the lower-order bit of the function code is 1), the destination network address is determined from the Client. Otherwise, it's determined from the Server. The HostAddr field is set either from the ServerHost cache (for a Request) or from the original Request (for a Response), before SendPacketGroup gets called.
The CSR has timeout and TTL fields, which show the maximum time to finish the processing and the time-to-live for the packets being sent ⏳
SendPacketGroup is seen as the right functionality to implement for transmission in an "intelligent" network interface 🧚
Finally, it's kinda preferable to be able to assume that all portions of the segment stay memory-resident (no page faults) during transmission. In a demand-paged system, some form of locking is needed to keep the segment data in memory 🙈
VII.5. VMTP Management Module 🛠️
The implementation should put the management operations in a separate module that gets invoked from inside the VMTP module. When a Request comes in, whether from the local user level or from the network, for the VMTP management module, the management module is invoked as a remote or local procedure call. It handles the request and returns a response (if it's not a datagram request) 💅 By registering as a local server, the management module can keep the special-case code needed for its invocation to a minimum. Basically, the management module is a case statement that picks the operation based on the RequestCode, and then invokes that management operation. The procedures implementing the management operations, especially operations like NotifyVmtpClient and NotifyVmtpServer, are logically part of the VMTP module, because they need full access to the basic data structures of the VMTP implementation 🔥
The management module should be built so it can respond quickly to all requests, especially since the timing of management interactions is used to estimate round trip time ⚡ To date, every implementation of the management module has been done at the kernel level, right alongside VMTP proper 🌈
VII.6. Timeout Handling ⏰
The timeout queue is a queue of CSR records, ordered by timeout count, as specified in the CSR record. When a CSR enters the timeout queue, its timeout field gets set to the time it should stay in the queue (ideally in milliseconds or a similar unit), and its finishup field gets set to the procedure to run when it's removed from the queue on timeout 💖 The timeout field for a CSR in the queue is the time relative to the record just before it in the queue (if there is one) at which it should be removed. Some system-specific mechanism decrements the time for the record at the front of the queue, and invokes the finishup procedure when the count hits zero.
With this scheme, a special CSR is used to time out and scan CSR's that haven't been pinged recently. That is, this CSR times out and invokes a finishup procedure that scans for non-recently pinged CSR's that are AwaitingResponse, signals the request processing entity, and deletes the CSR. Then it goes back to the timeout queue 🔁
The timeout mechanism tends to be specific to an operating system, so the scheme described here may have to be adapted to whatever operating system VMTP is being implemented on. ⚠️
This mechanism handles both client request timeout and client response timeout. It is not meant to handle interpacket gaps, since those times are expected to be under 1 millisecond in general, and possibly only a few microseconds 🦋
VII.7. Timeout Values 🎀
Round trip timeout values are estimated by matching Responses or NotifyVmtpClient Requests to the transmission of the Request, relying on the retransmitCount to pin down which particular transmission of the Request
that generated the response 💖 A similar trick works for Responses and NotifyVmtpServer Requests too. The retransmitCount gets bumped every single time the Response goes out, whether that's because of a timeout or because the Request got retransmitted 🔥
The ProbeEntity request is recommended as a basic way of getting up-to-date info about a Client, plus a predictable turnaround time from the host machine when it's processing a request 🦋 (VMTP assumes and requires an efficient, bounded response time implementation of the ProbeEntity operation 💅.)
Once you're measuring RTT this way, it's recommended that you adapt the estimation and smoothing techniques that were developed for TCP RTT estimation and use them here. Iconic, honestly 👑
VII.8. Packet Reception ✨
Logically, a network packet carrying a VMTP packet has 5 portions 🎀
- network header, possibly including lower-level headers
- VMTP header
- data segment
- VMTP checksum
- network trailer, etc.
It can be really advantageous to receive a packet split up into these portions, as long as the network module supports it 🧚 In that case, ideally the VMTP header lands directly in a CSR, the data segment goes into a page that can be mapped (not copied) to its final destination, and the VMTP checksum and network header sit in a separate area that's used to pull out the network address of the sender. Main character energy 💖
Packet reception is described in detail by the pseudo-code in Section 4.7 📜
For a Response, the CSR usually has an associated segment area already ready to go, so segment data gets delivered right away 🌸 Server entities should be "armed" with CSR's that have segment areas too, so Requests can be delivered immediately. It's totally reasonable to throw away segment data that can't be delivered right away like this, as long as clients and servers can preallocate CSR's with segment areas for Requests and Responses. Also, a client should be able to provide some extra CSR's for receiving multiple Responses to a multicast Request, because that's a whole vibe 🦋
The CSR data structure is meant to be the interface data structure for an intelligent network interface 🤖 For reception, the interface is "armed" with CSR's that can point to segment areas in main memory, where it can deliver a packet group. Ideally, the interface handles all the processing for every packet, and only talks to the host after a complete Request or Response packet group has arrived. An implementation should use an interface built on SendPacketGroup(CSR) and ReceivePacketGroup(CSR) so it's easy to bring in an intelligent network interface later 🫶
ReceivePacketGroup(csr) gives the interface a CSR descriptor plus zero or more bytes of main memory to receive segment data into. The CSR says whether it's receiving Responses (and if so, for which client) or Requests (and if so, for which server) 💜
The procedure ReclaimCSR(CSR) reclaims the specified record from the interface before it has been returned, even though the specified packet group has already come in 😭
A finishup procedure is set in the CSR and gets invoked when the CSR is returned to the host through the normal processing sequence in the interface ✨ Similarly, the timeout parameter is set to show the maximum time the host is giving the routine to perform the specified function ⏰ The CSR and its associated segment memory are returned to the host after the timeout period, with an indication of progress after that timeout period. It is not returned earlier. Not early, not ever, not me 🙈
VII.9. Streaming 🌊
Streaming is optional in both VMTP clients and servers 💖 Ideally, all performance-critical servers should implement streaming. Clients that have high context switch overhead, high network access overhead, or expect to be talking over long delay links should also implement streaming, honestly iykyk 🧚
A client stream is implemented by allocating a CSR for each outstanding message transaction. A stream of transactions gets handled a lot like multiple outstanding transactions from separate clients, except for the interaction between consecutive numbered transactions in a stream 🦋
On the server VMTP module, streamed message transactions headed to a server get queued (if they're accepted) underneath the first unprocessed CSR for this Client. So streamed transactions from a given Client are always performed in the order set by the transaction identifiers. Order is everything, bestie 👑
If a server doesn't implement streaming, it must refuse streamed message transactions using the NotifyVmtpClient operation. Also, every client VMTP that supports streaming must support the streamed interface to a server that doesn't support streaming. In other words, it must perform the message transactions one at a time. So a program using the streaming interface with a non-streaming server gets degraded performance, but not failure 🍓 It's slower, but it still works, which is honestly the most important thing 🫶
VII.10. Implementation Experience 📈
The implementation experience so far includes a partial implementation (minus streaming and full security) in the V kernel, plus a similar preliminary implementation in the 4.3 BSD Unix kernel 🌈 In the V kernel implementation, the CSR's are part of the (lightweight) process descriptor.
The V kernel implementation can do a VMTP message transaction with no data segment between two Sun-3/75's connected by 10 Mb Ethernet in 2.25 milliseconds ⚡ It can also transfer data at 4.7 megabits per second using 16 kilobyte Requests (but with null checksums 🙈). The UNIX kernel implementation, running on Microvax II's, gets a basic message transaction time of 9 milliseconds and a data rate of 1.9 megabits per second using 16 kilobyte Responses. This implementation uses the standard VMTP checksum 💅
We hope to report more extensive implementation experience in future revisions of this document 🌸
VIII. UNIX 4.3 BSD Kernel Interface for VMTP 🧚
UNIX 4.3 BSD has a socket-based design for program interfaces to a variety of protocol families and types of protocols (streams, datagrams) ✨ In this appendix, we sketch an extension to that design to support a transaction-style protocol. (Some familiarity with UNIX 4.2/3 IPC is assumed 💜.) Several extensions are needed for the system interface, rather than just adding a protocol, because the original design made no provision for transaction protocols 😭 These extensions include a new "transaction" type of socket plus new system calls: invoke, getreply, probeentity, recreq, sendreply and forward 🎀
A socket of type transaction bound to the VMTP protocol type IPPROTO_VMTP is created by the call
s = socket(AF_INET, SOCK_TRANSACT, VMTP);This socket gets bound to an entity identifier by
bind(s, &entityid, sizeof(entityid));The first address/port bound to a socket is considered its primary name, and it's the one used when packets go out 📮 A message transaction gets invoked between the socket named by s and the Server specified by mcb, like this:
invoke(s, mcb, segptr, seglen, timeout );The mcb is a message control block whose format was described in Section 2.4 🧩 The message control block specifies the request to send plus the destination Server. The response message control block the server sends back gets stored in mcb when invoke returns. The invoking process is blocked until a response arrives or the message transaction times out, unless the request is a datagram request 💤 (Non-blocking versions with signals on completion could also be provided, especially with a streaming implementation 🦋.)
For multicast message transactions (sent to an entity group), the next response to the current message transaction (if it arrives in less than timeout milliseconds) gets returned by
getreply( s, mcb, segptr, maxseglen, timeout );The invoke operation sent to an entity group completes as soon as the first response comes in. A request keeps getting retransmitted until the first reply arrives (assuming the request isn't a datagram). So the system doesn't retransmit while getreply is timing out, even if no replies are available 🙃
The state of an entity associated with entityId gets probed using
probeentity( entityId, state );A UNIX process acting as a VMTP server accepts a Request with the operation
recvreq(s, mcb, segptr, maxseglen );The request message for the next queued transaction request is returned in mcb, plus segment data of maximum length maxseglen, starting at segptr in the address space 📦 When it returns, the message control block has the values set by invoke, except for: (1) the Client field, which indicates the Client that sent the received Request message; (2) the Code field, which indicates the type of request; and (3) the MsgDelivery field, which indicates the portions of the segment actually received within the specified segment size, if MDM is 1 in the Code field. A segment block is marked as missing (i.e. the corresponding bit in the MsgDelivery field is 0) unless it's received in its entirety, or it's all of the data in the last segment contained in the segment 🥺
To complete a transaction, the reply specified by mcb gets sent to the client specified by the MCB using
sendreply(s, mcb, segptr );The Client field of the MCB indicates which client to respond to 💌
Finally, a message transaction specified by mcb gets forwarded to newserver as though its original invoker had sent it there, using
forward(s, mcb, segptr, timeout );Index ✨
okay the index is basically the cheat sheet for this whole RFC, so here it is in all its glory 🧚💖
Acknowledgment 14
APG 16, 31, 39
Authentication domain 20
Big-endian 9
Checksum 14, 43
Checksum, not set 44
Client 7, 10, 38
Client timer 16
CMD 42, 110
CMG 32, 40
Co-resident entity 25
Code 42
CoResidentEntity 42, 43
CRE 21, 42
DGM 42
Digital signature, VMTP management 95, 101
Diskless workstations 2
Domain 9, 38
Domain 1 102
Domain 3 104
Entity 7
Entity domain 9
Entity group 8
Entity identifier 37
Entity identifier allocation 105
Entity identifier, all-zero 38
EPG 20, 39
Features 6
ForwardCount 24
Forwarding 24
FunctionCode 41
Group 8
Group message transaction 10
Group timeouts 16
GRP 37
HandleNoCSR 62
HandleRequestNoCSR 79
HCO 14, 23, 39
Host independence 8
Idempotent 15
Interpacket gap 18, 40
IP 108
Key 91
LEE 32, 37
Little-endian 9
MCB 118
MDG 22, 40
MDM 30, 42
Message control block 118
Message size 6
Message transaction 7, 10
MPG 39
MsgDelivery 43
MSGTRANS_OVERFLOW 27
Multicast 4, 21, 120
Multicast, reliable 21
Naming 6
Negative acknowledgment 31
NER 25, 31, 39
NRT 26, 30, 39
NSR 25, 27, 31, 39
Object-oriented 2
Overrun 18
Packet group 7, 29, 39
Packet group run 31
PacketDelivery 29, 31, 41
PGcount 26, 41
PIC 42
Principal 11
Priority 41
Process 11
ProcessId 89
Protocol number,IP 108
RAE 37
Rate control 18
Real-time 2, 4
Realtime 22
Reliability 12
Request message 10
RequestAckRetries 30
RequestRetries 15
Response message 10
ResponseAckRetries 31
ResponseRetries 15
Restricted group 8
Retransmission 15
RetransmitCount 17
Roundtrip time 17
RPC 2
Run 31, 39
Run, message transactions 25
SDA 42
Security 4, 19
Segment block 41
Segment data 43
SegmentSize 42, 43
Selective retransmission 18
Server 7, 10, 41
Server group 8
Sockets, VMTP 118
STI 26, 40
Streaming 25, 55
Strictly stable 8
Subgroups 21
T-stable 8
TC1(Server) 16
TC2(Server) 16
TC3(Server) 16
TC4 16
TCP 2
Timeouts 15
Transaction 10, 41
Transaction identification 10
TS1(Client) 17
TS2(Client) 17
TS3(Client) 17
TS4(Client) 17
TS5(Client) 17
Type flags 8
UNIX interface 118
Unrestricted group 8, 38
NotifyVmtpClient 7, 26, 27, 30
NotifyVmtpServer 7, 14, 30
User Data 43
Version 38
VMTP Management digital signature 95, 101that's the whole index, bestie 🫶 every term and page number is exactly where it lives in the original, so you can flip straight to whatever you're hunting for 🦋
📦 where this page comes from
- ietf: rfc1045 1988-02
- 💖 girlified by claude-haiku-5-5 on : a language model's rewrite of the original, under its terms