Remote Procedure Call (RPC)
In this lesson, we look at a programming language abstraction, RPC, to provide the basis for building distributed systems.
We'll cover the following
While OS abstractions turned out to be a poor choice for building distributed systems, programming language (PL) abstractions make much more sense. The most dominant abstraction is based on the idea of a
Remote procedure call packages all have a simple goal: to make the process of executing code on a remote machine as simple and straightforward as calling a local function. Thus, to a client, a procedure call is made, and sometime later, the results are returned. The server simply defines some routines that it wishes to export. The rest of the magic is handled by the RPC system, which in general has two pieces: a stub generator (sometimes called a protocol compiler), and the run-time library. We’ll now take a look at each of these pieces in more detail.
Stub generator
The stub generator’s job is simple: to remove some of the pain of packing function arguments and results in messages by automating it. Numerous benefits arise: one avoids, by design, the simple mistakes that occur in writing such code by hand. Further, a stub compiler can perhaps optimize such code and thus improve performance.
The input to such a compiler is simply the set of calls a server wishes to export to clients. Conceptually, it could be something as simple as this:
interface {int func1(int arg1);int func2(int arg1, int arg2);};
The stub generator takes an interface like this and generates a few different pieces of code. For the client, a client stub is generated, which contains each of the functions specified in the interface; a client program wishing to use this RPC service would link with this client stub and call into it in order to make RPCs.
Internally, each of these functions in the client stub does all of the work needed to perform the remote procedure call. To the client, the code just appears as a function call (e.g., the client calls func1(x)); internally, the code in the client stub for func1() does this:
-
Create a message buffer. A message buffer is usually just a contiguous array of bytes of some size.
-
Pack the needed information into the message buffer. This information includes some kind of identifier for the function to be called, as well as all of the arguments that the function needs (e.g., in our example above, one integer for
func1). The process of putting all of this information into a single contiguous buffer is sometimes referred to as the marshaling of arguments or the serialization of the message. -
Send the message to the destination RPC server. The communication with the RPC server, and all of the details required to make it operate correctly, are handled by the RPC run-time library, described further below.
-
Wait for the reply. Because function calls are usually synchronous, the call will wait for its completion.
-
Unpack return code and other arguments. If the function just returns a single return code, this process is straightforward. However, more complex functions might return more complex results (e.g., a list), and thus the stub might need to unpack those as well. This step is also known as unmarshaling or deserialization.
-
Return to the caller. Finally, just return from the client stub back into the client code.
For the server, code is also generated. The steps taken on the server are as follows:
-
Unpack the message. This step, called unmarshaling or deserialization, takes the information out of the incoming message. The function identifier and arguments are extracted.
-
Call into the actual function. Finally! We have reached the point where the remote function is actually executed. The RPC runtime calls into the function specified by the ID and passes in the desired arguments.
-
Package the results. The return argument(s) are marshaled back into a single reply buffer.
-
Send the reply. The reply is finally sent to the caller.
There are a few other important issues to consider in a stub compiler. The first is complex arguments, i.e., how does one package and send a complex data structure? For example, when one calls the write() system call, one passes in three arguments: an integer file descriptor, a pointer to a buffer, and a size indicating how many bytes (starting at the pointer) are to be written. If an RPC package is passed a pointer, it needs to be able to figure out how to interpret that pointer and perform the correct action. Usually, this is accomplished through either well-known types (e.g., a buffer_t that is used to pass chunks of data given a size, which the RPC compiler understands), or by annotating the data structures with more information, enabling the compiler to know which bytes need to be serialized.
Another important issue is the organization of the server with regards to concurrency. A simple server just waits for requests in a simple loop and handles each request one at a time. However, as you might have guessed, this can be grossly inefficient; if one RPC call blocks (e.g., on I/O), server resources are wasted. Thus, most servers are constructed in some sort of concurrent fashion. A common organization is a thread pool. In this organization, a finite set of threads are created when the server starts. When a message arrives, it is dispatched to one of these worker threads, which then does the work of the RPC call, eventually replying. During this time, the main thread keeps receiving other requests, and perhaps dispatching them to other workers. Such an organization enables concurrent execution within the server, thus increasing its utilization. The standard costs arise as well, mostly in programming complexity, as the RPC calls may now need to use locks and other synchronization primitives in order to ensure their correct operation.
Run-time library
The run-time library handles much of the heavy lifting in an RPC system; most performance and reliability issues are handled herein. We’ll now discuss some of the major challenges in building such a run-time layer.
One of the first challenges we must overcome is how to locate a remote service. This problem, of naming, is a common one in distributed systems, and in some sense goes beyond the scope of our current discussion. The simplest of approaches build on existing naming systems, e.g., hostnames and port numbers provided by current internet protocols. In such a system, the client must know the hostname or IP address of the machine running the desired RPC service, as well as the port number it is using (a port number is just a way of identifying a particular communication activity taking place on a machine, allowing multiple communication channels at once). The protocol suite must then provide a mechanism to route packets to a particular address from any other machine in the system. For a good discussion of naming, you’ll have to look elsewhere, e.g., read about DNS and name resolution on the Internet, or better yet just read the excellent chapter in
Once a client knows which server it should talk to for a particular remote service, the next question is which transport-level protocol should RPC be built upon. Specifically, should the RPC system use a reliable protocol such as TCP/IP, or be built upon an unreliable communication layer such as UDP/IP?
Naively the choice would seem easy: clearly we would like for a request to be reliably delivered to the remote server, and clearly we would like to reliably receive a reply. Thus we should choose reliable transport protocols such as TCP, right?
Unfortunately, building RPC on top of a reliable communication layer can lead to a major inefficiency in performance. Recall from the discussion above how reliable communication layers work: with acknowledgments plus timeout/retry. Thus, when the client sends an RPC request to the server, the server responds with an acknowledgment so that the caller knows the request was received. Similarly, when the server sends the reply to the client, the client acks it so that the server knows it was received. By building a request/response protocol (such as RPC) on top of a reliable communication layer, two “extra” messages are sent.
For this reason, many RPC packages are built on top of unreliable communication layers, such as UDP. Doing so enables a more efficient RPC layer, but does add the responsibility of providing reliability to the RPC system. The RPC layer achieves the desired level of responsibility by using timeout/retry and acknowledgments much as we described above. By using some form of sequence numbering, the communication layer can guarantee that each RPC takes place exactly once (in the case of no failure), or at most once (in the case where failure arises).
Other issues
There are some other issues an RPC run-time must handle as well. For example, what happens when a remote call takes a long time to complete? Given our timeout machinery, a long-running remote call might appear as a failure to a client, thus triggering a retry, and thus the need for some care here. One solution is to use an explicit acknowledgment (from the receiver to sender) when the reply isn’t immediately generated; this lets the client know the server received the request. Then, after some time has passed, the client can periodically ask whether the server is still working on the request; if the server keeps saying “yes”, the client should be happy and continue to wait (after all, sometimes a procedure call can take a long time to finish executing).
The run-time must also handle procedure calls with large arguments, larger than what can fit into a single packet. Some lower-level network protocols provide such sender-side fragmentation (of larger packets into a set of smaller ones) and receiver-side reassembly (of smaller parts into one larger logical whole); if not, the RPC run-time may have to implement such functionality itself. See
One issue that many systems handle is that of byte ordering. As you may know, some machines store values in what is known as big endian ordering, whereas others use little endian ordering. Big endian stores bytes (say, of an integer) from most significant to least significant bits, much like Arabic numerals; little endian does the opposite. Both are equally valid ways of storing numeric information; the question here is how to communicate between machines of different endianness.
RPC packages often handle this by providing a well-defined endianness within their message formats. In Sun’s RPC package, the XDR (eXternal Data Representation) layer provides this functionality. If the machine sending or receiving a message matches the endianness of XDR, messages are just sent and received as expected. If however, the machine communicating has a different endianness, each piece of information in the message must be converted. Thus, the difference in endianness can have a small performance cost.
A final issue is whether to expose the asynchronous nature of communication to clients, thus enabling some performance optimizations. Specifically, typical RPCs are made synchronously, i.e., when a client issues the procedure call, it must wait for the procedure call to return before continuing. Because this wait can be long, and because the client may have other work it could be doing, some RPC packages enable you to invoke an RPC asynchronously. When an asynchronous RPC is issued, the RPC package sends the request and returns immediately; the client is then free to do other work, such as call other RPCs or other useful computations. The client at some point will want to see the results of the asynchronous RPC; it thus calls back into the RPC layer, telling it to wait for outstanding RPCs to complete, at which point return arguments can be accessed.
Aside: THE END-TO-END ARGUMENT
The end-to-end argument makes the case that the highest level in a system, i.e., usually the application at “the end”, is ultimately the only locale within a layered system where certain functionality can truly be implemented. In their
argue this through an excellent example: reliable file transfer between two machines. If you want to transfer a file from machine A to machine B, and make sure that the bytes that end up on B are exactly the same as those that began on A, you must have an “end-to-end” check of this; lower-level reliable machinery, e.g., in the network or disk, provides no such guarantee. landmark paper, Saltzer et al. “End-To-End Arguments in System Design” by Jerome H. Saltzer, David P. Reed, David D. Clark. ACM TOCS, 2:4, November 1984. A beautiful discussion of layering, abstraction, and where functionality must ultimately reside in computer systems. The contrast is an approach that tries to solve the reliable-file-transfer problem by adding reliability to lower layers of the system. For example, say we build a reliable communication protocol and use it to build our reliable file transfer. The communication protocol guarantees that every byte sent by a sender will be received in order by the receiver, say using timeout/retry, acknowledgments, and sequence numbers. Unfortunately, using such a protocol does not a reliable file transfer make; imagine the bytes getting corrupted in sender memory before the communication even takes place, or something bad happening when the receiver writes the data to disk. In those cases, even though the bytes were delivered reliably across the network, our file transfer was ultimately not reliable. To build a reliable file transfer, one must include end-to-end checks of reliability, e.g., after the entire transfer is complete, read back the file on the receiver disk, compute a checksum, and compare that checksum to that of the file on the sender.
The corollary to this maxim is that sometimes having lower layers provide extra functionality can indeed improve system performance or otherwise optimize a system. Thus, you should not rule out having such machinery at a lower-level in a system. Rather, you should carefully consider the utility of such machinery, given its eventual usage in an overall system or application.
Get hands-on with 1400+ tech skills courses.