RFC 684

RFC 684: A Commentary on Procedure Calling as a Network Protocol 💌

Network Working Group 🌸 RFC #684 ✨ NIC #32252 April 15,1975 🦋

Author: Richard Schantz, BBN-TENEX 💜

Okay bestie, the title block doesn't list a category or an "obsoletes" or "updates" line, so I'm not making any up. 🙈

---

Preface 🎀

Okay so this RFC is the first step in a plan to get people talking about how to design a distributed computing system 🗣️✨ It's looking at the approach in RFC #674, which everyone calls the "Procedure Call Protocol" (PCP) 💅 The main goal here is to point out what we think are the shortcomings of that design approach.

Not the first time we've critiqued PCP, btw 🙈 During the early PCP design stages we met the PCP designers for a short while and suggested a bunch of changes, and a lot of them ended up in PCP Version 2 🎉 But the vibe of those suggestions was totally different from this note. The original ones, and some later ones, were mostly about implementation details. This note is more about the concepts underneath the PCP design than the PCP implementation 🧠

We're sharing this because it raises issues that haven't been properly dealt with yet 👀 The PCP designers deserve major props for writing up their ideas in detail, which gives everyone a natural starting point for talking about distributed system design concepts 👑 The whole point is to get people interacting who are into distributed computing, so that maybe we end up with systems whose designs don't block them from being used in projects other than the one they were originally made for 🌈

Huge thanks to Bob Thomas of BBN-TENEX, who has had tons of discussions with us about this and shares the same point of view 🫶

Introduction 🦋

Okay, so, the Procedure Call Protocol (PCP) and how it's used inside the National Software Works (NSW) project actually tackle a lot of the problems you run into when you try to get independent computing systems to work together on one distributed computation 🔥 BUT we think its design has flaws that should stop it from being used everywhere, and those flaws limit how useful it is overall 💀

We're not objecting to PCP, as it's currently defined, being used as the base-level implementation for the NSW project 🙅‍♀️ It's already too late for that kind of objection anyway 😭 PCP might honestly work really well for NSW, since both projects were moving forward at the same time and have probably influenced each other 🧚 What we're actually objecting to is the "PCP philosophy" 💅 We really don't want this type of protocol to become the de facto network standard for distributed computation, and we want to influence where this effort and similar ones go next 🎯

Some of the things we don't like about PCP might just be differences in personal taste, and philosophers have long said you can't argue about taste 🍓 We've tried to avoid that kind of argument in this document. Instead, we look at PCP through what we've actually learned from building distributed systems 🛠️ Looked at that way, we think PCP and its underlying philosophy have flaws that make it a bad fit as a general purpose protocol and virtual programming system for building distributed software 🙈

Here's our take: PCP is probably complete, meaning you can probably do anything you need with its primitives ✨ So the key question isn't whether this function or that function can be supported 🙄 The question that matters to us is how easy it is to do the things that experience has shown are important in distributed computing 💖 On top of that, a programming discipline built for network applications should pay special attention to steering its users away from things that systems programming in general, and network programming in particular, have shown are pitfalls in system implementation ⚠️

A Point of View 💜

Let's be clear, we're fully behind the parts of the PCP design effort that systematize how distributed elements interact and agree with each other to support inter-machine computing 🫶 That includes defining the different types of replies, standardizing the data structure format for exchanges between machines, and the process creation primitives that stretch across machine boundaries 🌈 These ideas are basic, and any distributed system definition has to include them 💅 Our main concern is NOT with any of that stuff 😌

What we DO take issue with is PCP's underlying premise: that the procedure calling discipline is the starting point for building multi-computer systems 🙅‍♀️ That premise gives you a model with a central point controlling the entire algorithm, when a more natural approach for networks is distributed control, where cooperating independent entities interact through shared communication paths 🦋 Procedure calls might be a good fit for some applications, sure ✨ But we don't think they can directly or accurately model the interactions and control structures that show up in lots of distributed multi-computer systems 🫠

Now, PCP supporters might say, "who cares what you call it, it's doing the same thing" 🙄 Our response: it really matters to have a clear, concise model of distributed computation 🧠 The PCP model doesn't force "poor implementation" of distributed systems, but it also doesn't make "good implementation" any easier, and it doesn't stop ill-advised programming practices either 😬 A model that focuses on the dynamic interconnection of somewhat independent computing entities fits better with defensive programming, and we've found defensive programming is fundamental to building usable multi-machine implementations 💖

The rest of this RFC goes through some of the shortcomings we see in a procedure call protocol 📝

Limitations of Procedure Calling Across Machines 🚧

Okay, first and foremost, our argument is that procedure calling should NOT be the basis for multi-machine interactions 🙅‍♀️ We think a request and reply protocol, along with suitably manipulated communication paths between processes, is a model that fits the situation a network puts us in much better 🌸 In a network you've got autonomous computing entities that have agreed to cooperate, not a master process forcing a certain body of code to run to get its computing done 👑 So in that setup, it's best to ask a process to do something indirectly (by request) rather than directly (by procedure call), so the integrity of each process stays intact 🔒

Procedure calling is usually a super primitive operation 🤏 Its implementation often needs only a single machine instruction ⚡ And it's generally not part of the operating system's domain 💅 [The Multics intersegment procedure calling mechanism may be an exception to this, until linkage is complete. In the remote PCP case, however, linkage can never be complete in the sense of supporting a fast transfer of control between modules]. Processes and communication paths between processes, on the other hand, are undeniably operating system constructs 🖥️ In an environment where local procedure calling is "cheap," blurring the line between a local call (cheap in time and effort) and a remote call would be ill-advised 🙈 A remote call obviously takes a huge amount of effort from the "PCP system," if not from the PCP user too 😭 We also think blurring the local/remote distinction at the procedure call level will show up as more frequent use of a less efficient local procedure calling mechanism 📉

Interprocess communication is a different story 🦋 At least for stream or message oriented channels (not just interrupt signals), it's generally seen as having a significant cost attached 💸 Sending a message is always an interprocess action, and it always needs system intervention 🫠 The gap between the IPC of local processes and the IPC of remote processes isn't nearly as big as the gap between local and remote procedure calling ✨ PCP kind of suggests a model where processes exist that span machine boundaries to provide inter-machine subroutine calling 🤔 But the PCP documentation hasn't pushed the idea of a process spanning machine boundaries, and honestly, that's the right call, because creating one would cause endless problems 💀 Since procedure calling is more of an intra-process idea, it makes more sense to take the interprocess communication framework and extend it so it has one uniform interpretation both locally and remotely 🌈 instead of stretching the procedure calling model 🧚 We also believe a model that relies on procedure calling as its basis doesn't account for the special nature of the network environment, and that a message passing model can handle that environment way better 💖

We also think programming as a whole, even purely local computing, would benefit from paying more attention to reliability and robustness 🛡️ Those things have come to the forefront because of experience with an often unreliable network and collection of hosts 😭 An IPC model puts the spotlight on the connections between disjoint processes, which reinforces the idea that distributed computing happens by joining separate entities together 🫶 That makes defensive programming and error handling techniques feel totally appropriate 💅 PCP is, we think, aimed at distributed system builders, not end users (for example, an RSEXEC user) 👩‍💻 So avoiding the network, interconnection issues, and relative costs could backfire if the goal is to build usable network systems 🙃

Along similar lines, the whole idea of inter-machine procedure calling basically extends the address space of a single process 🤯 In effect, there is a single locus of algorithm control (though maybe not a single locus of execution) 👑 That might work great for a "local" computation where the parts are strongly bound together 🔗 But our experience building working distributed systems has shown us the value of a model with multiple loci of control and execution 🌸 In that kind of model, a computation gets carried out through agreements on the method and type of information exchange and synchronization, rather than at the direction of one central entity 🦋 We think a process with distributed control and execution is in a much better position to naturally cope with the many vagaries that come up in a network environment 💪

The unmistakable trend in systems programming is toward inviolable (protected) process structures with external synchronization 🔐 That's how people cope with complex debugging tasks and with how hard it is to make system changes 🛠️ We think that trend is better supported by a message passing model than a procedural

model of computation. Plus, okay, we feel that network programming techniques should be applied to local computation, not the other way around 💅🎀

Some Particulars ✨

Okay, so in the list below, we get specific about the situations where we think the PCP concept might be weak as the basis for a network programming system 🧚 For some of these examples to make sense, you should be pretty familiar with the PCP documents issued as RFC 674 📚

1. Recovery from component malfunction can be super hard for a process that isn't the central control (aka a process that's getting manipulated by having its procedures executed) 😭 Is network trouble, for example, supposed to be modeled as a forced procedure call to some error recovery routine? Honestly, these are exactly the moments where distributed control is the better model 💖 Think about introducing an inferior to another acquaintance, then handing the new handle over as a parameter to a later procedure call in the inferior. The inferior just blindly uses that parameter to talk to the other process, which shows how the superior is basically in control of everything 👑 The inferior never actually finds out about a new communication path to a new process. The inferior environment (as kept up by the PCP "system") got changed by the superior, with no active notification to the inferior. Weird, right? This makes user coded error recovery kinda awkward, ngl 🙈
2. Process manipulation can sometimes break the principles of modular programming 🍓 It seems way better to be able to debug the pieces of a computation separately, and then only worry about how they synchronize, so you end up with a totally debugged system ✨ With PCP in its fullest sense, the danger of error propagation seems bigger, because a process has the power to make an arbitrary procedure run and to read and write remote data stores without the remote process actively taking part 😬
3. Initialization sequencing is a big question mark 🤔 Can we assume a proper initialization sequence if our procedures are called remotely? Does every procedure have to contain code that checks whether the call is proper and correctly sequenced? A model where each remote process is an active computing element seems much better at conveniently applying protective standards to the code and data it wraps up 🫶
4. Long term parallel activity isn't modeled very conveniently in PCP 🦋 That's a problem, because we need it to handle all sorts of asynchronous producer/consumer process relationships. The synchronization is geared more toward a one-to-one call and return, instead of the asynchronous nature and multiple returns for a single request that you see in lots of network services. On top of that, low priority, preemptable background tasks are hard (impossible?) to model in a procedure call environment 💀
5. Communication paths aren't treated as abstract objects that are independent from the actual entities they connect, so they can't be used in some useful ways (e.g. to carry non PCP messages) 🌈 Also, since paths aren't objects, there's no concept of passing a communication path to an inferior (or an acquaintance) without having to create a new "connection" (whether or not that turns out to be a physical channel). Passing paths is often super useful for subcontracting requests to inferior processes. Doing that within PCP requires the calling process to cooperate (i.e. to use the new connection handle), which again seems to violate the concepts of modular programming 🙅‍♀️ The other option in PCP is to have the superior relay the later communications to its created inferior, but the effort involved would probably stop anyone from using this technique for subcontracting 😭
6. Periodic but short interactions (i.e. a few words exchanged) are too much for PCP, which feels way too complicated for this kind of processing 🧁 An example is the way the TIP uses the TENEX accounting servers (see RFC #672). And honestly, PCP is probably much too complex for implementation on a small host 💅 There also doesn't seem to be a definition of what a minimum implementation for a host/process could look like if it didn't, or couldn't, handle everything that's been developed.
7. Persistence and priority servicing get awkward in the PCP model 🌸 It may become awkward or resource consuming for a service program to do things like queue operations for execution at a later time (persistence), or at a more opportune time (priority servicing mechanism). Those implementations may need dummy returns and changes to the controlling fork concept, or they may need processing forks kept alive over long periods of inactivity 🫠
8. Splicing a process to a service shouldn't always let that process influence the service process environment in any direct way 🚨 How can a service process in PCP stop a malicious user from splicing to it and then introducing it to an arbitrary number of processes, which overflows the table space in that process? All of that could happen without ever executing a single instruction of user written code. This difficulty is a consequence of the PCP notion of having one process manipulate the environment of another without its active participation in those actions. Yikes, bestie 😱
9. Doesn't it say something that the network PCP process implementation is so much neater than the TENEX PCP process implementation (since TENEX doesn't have a general IPC facility)? 🦋 It suggests that message passing and communication facilities supplied by the "system" are a sound basis for multi-process implementations. Maybe such facilities should be primitively available to the distributed system builders who will use PCP 💜
10. There's a question about whether PCP is an implementation virtual machine (language), or an application virtual machine (language) 🤨 In other words, is PCP meant to be used to build systems that manage distributed resources, or is it meant to be an end product that makes network resources easier for the everyday, ordinary programmer to use (e.g. makes the network itself transparent to users)? One gets the feeling the designers had both goals, and neither one is totally satisfied 🥲 If we take the former goal, we believe most of the complexities (e.g. network trouble, broken connections, etc.) and possibilities (e.g. redundant implementation, broadcast request, etc.) of network implementations aren't handled adequately. In this view, the NSW framework (Works manager, FE) is the distributed system that uses the PCP implementation language. We don't see how using PCP in this context gives you either an extra-reliable system through component redundancy, or a persistent system that can tolerate temporary malfunctions. If you go with this view, then it doesn't seem right that the objects running under the created system (i.e. the tools that run under the PCP implemented Front End, Works Manager, and TBH monitor) should also be aware of or use PCP. If we consider the latter goal, that PCP implements a virtual machine to be presented to all programmers for making distributed resources easy to use, then it's clear that PCP, with its manifest concern for object location, doesn't provide the desirable properties of network transparency 😭✨

---

Our conclusion is that procedure calling is not the appropriate basis for distributed multi-computer systems, because it can neither directly nor accurately model the network environment 🎀 The PCP virtual programming system may be inadequate for implementing many distributed systems, because the complexities and possibilities unique to the network environment aren't provided for at this basic level 💖🌈