1. A Problem 19 Years in the Making

   SMTP [1] is close to being the perfect application protocol: it
   solves a large, important problem in a minimalist way.  It's simple
   enough for an entry-level implementation to fit on one or two screens
   of code, and flexible enough to form the basis of very powerful
   product offerings in a robust and competitive market.  Modulo a few
   oddities (e.g., SAML), the design is well conceived and the resulting
   specification is well-written and largely self-contained.  There is
   very little about good application protocol design that you can't
   learn by reading the SMTP specification.

   Unfortunately, there's one little problem: SMTP was originally
   published in 1981 and since that time, a lot of application protocols
   have been designed for the Internet, but there hasn't been a lot of
   reuse going on.  You might expect this if the application protocols
   were all radically different, but this isn't the case: most are
   surprisingly similar in their functional behavior, even though the
   actual details vary considerably.

   In late 1998, as Carl Malamud and I were sitting down to review the
   Blocks architecture, we realized that we needed to have a protocol
   for exchanging Blocks.  The conventional wisdom is that when you need
   an application protocol, there are four ways to proceed:

   1. find an existing exchange protocol that (more or less) does what
      you want;

   2. define an exchange model on top of the world-wide web
      infrastructure that (more or less) does what you want;

   3. define an exchange model on top of the electronic mail
      infrastructure that (more or less) does what you want; or,

   4. define a new protocol from scratch that does exactly what you
      want.

   An engineer can make reasoned arguments about the merits of each of
   the these approaches.  Here's the process we followed...

   The most appealing option is to find an existing protocol and use
   that.  (In other words, we'd rather "buy" than "make".) So, we did a
   survey of many existing application protocols and found that none of
   them were a good match for the semantics of the protocol we needed.

   For example, most application protocols are oriented toward
   client/server behavior, and emphasize the client pulling data from
   the server; in contrast with Blocks, a client usually pulls data from
   the server, but it also may request the server to asynchronously push
   (new) data to it.  Clearly, we could mutate a protocol such as FTP
   [2] or SMTP into what we wanted, but by the time we did all that, the
   base protocol and our protocol would have more differences than
   similarities.  In other words, the cost of modifying an off-the-shelf
   implementation becomes comparable with starting from scratch.

   Another approach is to use HTTP [3] as the exchange protocol and
   define the rules for data exchange over that.  For example, IPP [4]
   (the Internet Printing Protocol) uses this approach.  The basic idea
   is that HTTP defines the rules for exchanging data and then you
   define the data's syntax and semantics.  Because you inherit the
   entire HTTP infrastructure (e.g., HTTP's authentication mechanisms,
   caching proxies, and so on), there's less for you to have to invent
   (and code!).  Or, conversely, you might view the HTTP infrastructure
   as too helpful.  As an added bonus, if you decide that your protocol
   runs over port 80, you may be able to sneak your traffic past older
   firewalls, at the cost of port 80 saturation.

   HTTP has many strengths: it's ubiquitous, it's familiar, and there
   are a lot of tools available for developing HTTP-based systems.
   Another good thing about HTTP is that it uses MIME [5] for encoding
   data.

   Unfortunately for us, even with HTTP 1.1 [6], there still wasn't a
   good fit.  As a consequence of the highly-desirable goal of
   maintaining compatibility with the original HTTP, HTTP's framing
   mechanism isn't flexible enough to support server-side asynchronous
   behavior and its authentication model isn't similar to other Internet
   applications.

   Mapping IPP onto HTTP 1.1 illustrates the former issue.  For example,
   the IPP server is supposed to signal its client when a job completes.
   Since the HTTP client must originate all requests and since the
   decision to close a persistent connection in HTTP is unilateral, the
   best that the IPP specification can do is specify this functionality
   in a non-deterministic fashion.

   Further, the IPP mapping onto HTTP shows that even subtle shifts in
   behavior have unintended consequences.  For example, requests in IPP
   are typically much larger than those seen by many HTTP server
   implementations -- resulting in oddities in many HTTP servers (e.g.,
   requests are sometimes silently truncated).  The lesson is that
   HTTP's framing mechanism is very rigid with respect to its view of
   the request/response model.

   Lastly, given our belief that the port field of the TCP header isn't
   a constant 80, we were immune to the seductive allure of wanting to
   sneak our traffic past unwary site administrators.

   The third choice, layering the protocol on top of email, was
   attractive.  Unfortunately, the nature of our application includes a
   lot of interactivity with relatively small response times.  So, this
   left us the final alternative: defining a protocol from scratch.

   To begin, we figured that our requirements, while a little more
   stringent than most, could fit inside a framework suitable for a
   large number of future application protocols.  The trick is to avoid
   the kitchen-sink approach.  (Dave Clark has a saying: "One of the
   roles of architecture is to tell you what you can't do.")