Why Is REST So Hard to Understand?

I know, I know. Not another article about REST – hasn’t this horse been flogged enough? Perhaps – but recently I’ve been watching the GIS industry finally discover REST and unsurprisingly there is bit of misunderstanding and a bit of resistance. Kudos to Sean, and others, for their efforts in spreading the word.

But it raises an interesting question – why is it so difficult for most people, including myself, to understand REST at first? Its not technical complexity – REST is significantly simpler than CORBA, or COM, or just about any programming language. And its not for a lack of an example – the web is one of the most successful distributed computer system ever built.

Instead, I think its because we are looking at the problem the wrong way. To borrow a metaphor from physics, we are using the wrong frame of reference. I finally figured out how important using the correct frame of reference is during a junior year thermodynamics class (my favorite engineering subject, I have to say). Our task was to calculate various properties (temperature, wind speed, etc) in front of and behind a shock wave. If you try to do it using a stationary frame of reference, the math is well nigh impossible. But if you change you frame of reference to the shock wave itself, then the math falls nicely out and you can easily solve the problem.

So instead of focusing on the nitty-gritty of REST, including URIs, resources, HTTP verbs, etc., let’s take a look at the big picture.

The Service Oriented Architecture (SOA) Frame of Reference

Let’s start by boiling a program down to its most fundamental parts:

Client Server

A client uses a service to access or process some data. Note I am using a very broad definition of client and service – this applies to a object calling another object in a program, a program calling a function in a dll, a browser accessing a web site, etc.

When learning to program, we are taught that the most important part of a system is the services and APIs they provide.

Client Server

In fact, a large part of object oriented design is teasing out the right objects (which are services under my broad definition) and designing their APIs. Design patterns are no more than precanned services that have been discovered via trial and error.

Notice three fundamental characteristics of this approach:

  • It’s a design goal to hide data from clients via encapsulation
  • Every service has its own API
  • Clients are tightly bound to services – if the service API changes the client also has to change

How well does the SOA approach work in the real world? Object oriented design has become the accepted approach for building large scale applications. It works to a degree – although the failure rate for creating large applications makes one wonder if there is a better way.

It also works, albeit barely, for creating in-process libraries (shared libraries or DLLs). The barely bit has to do with the difficulties of versioning service APIs, as witnessed by DLL hell on the Windows platform.

However, this approach fails when building distributed systems. It fails because clients are too tightly bound to services, it ignores the unreliability of networks and its not scaleable because of the inherent state held by services. Take a look around – when is the last time you saw a large distributed computer system working across multiple organizational boundaries that is based on COM, CORBA, RMI, SOAP web services, etc?

The Resource Oriented Architecture (ROA) Frame of Reference

Now lets change our frame of reference to a ROA. In ROA, the most important part of the system is the data.

Client Server

Anything of interest gets a globally unique identifier, which is done via URIs.

Next, the service becomes uninteresting because every service has the exact same API. On the Web, the service API is defined by HTTP verb’s GET, POST, PUT, DELETE, etc.

So what are the benefits of these changes?

  • Since all interesting data has a globally unique identifier, it becomes possible to create a web of links
  • Since there is only one service API, it becomes possible to create a single client (think of a browser) that can access an infinite number of services (think of websites)
  • Versioning issues between clients and services are greatly reduced (although they still happen when HTTP changes)
  • State can be taken out of the system and made into its own resource, thereby making the system much more scaleable

How does interoperability happen with ROA given that the service API is always the same? It’s done through globally agreed upon data formats, known as representations. This might seem to be an impossible task, but in practice its has worked out quite well:

  • xhtml/html/css are used for content, layout and presentation
  • png/jpeg/gif are used for raster images
  • atom/rss are used to model collections of items
  • pdf is used to preserve print versus online fidelity
  • mp3 and others are used for audio data

How well does the ROA approach work in the real world? Well, the web is one of the most successful distributed computer systems ever built, so I would say quite well.

Next Steps

I’m curious to see if understanding the different frames of reference used by SOA and ROA helps clear up some of the mystery surrounding REST – so I’d love to hear your feedback.

For a similar take on the differences between SOA and ROA, there are a number of excellent articles including ones by Stuart Charlton, Alex Bunardzic, Stefan Tilkov and Sanjiva Weerawarana.

And for a more in-depth look at ROA, read the W3C’s Architecture of the Web document and O’Reilly’s new book RESTful Web Services.

Update: In response to a post by Alex Bunardzic, I’ve written a follow up article that talks more about the data part of the system.

  1. Peter Williams
    August 13, 2007

    The term “ROA” is a bit of a problem in the context you are trying to use it. Namely, it seems to be already used in the REST community to describe applications that are “resource” oriented but have missed the fact that the most single most important thing about REST is the hypertext.

    Reply
  2. Charlie Savage –
    August 13, 2007

    Hey Peter,

    Its hard to say that hypertext is the most important part of REST since without resources and URI’s you couldn’t have hypertext.

    I think my use of ROA is appropriate – the focus of the architecture is on the resources and not the services.

    Reply
  3. August 13, 2007

    Like S3?

    Reply
  4. Jeremy McAnally
    August 13, 2007

    I think it’s hard to understand because people use hard words to describe it. A lot of the terminology I see in “introductory” articles would (and should) put the REST newbie off.

    Reply
  5. Peter Williams
    August 13, 2007

    I say that hypertext is the most important thing because if you have
    “hypertext” you MUST have global name (URIs) and resources. You can
    conceive of global names without hypertext but an architecture
    comprised of just resources with global names does not hold nearly the
    same potential as a fully REST based system with hypertext.

    Reply
  6. Charlie Savage –
    August 13, 2007

    Hey Sean,

    What do you mean, like S3? Are you using it as an example?

    Reply
  7. Charlie Savage –
    August 13, 2007

    Jeremy,

    I agree – people tend to dive right into resources, uris, http verbs, idempotent, GET vs PUT, etc. Unfortunately, the recent article on Directions magazine (link is in first paragraph above) is a good example of causing more confusion than enlightenment.

    Reply
  8. Charlie Savage –
    August 13, 2007

    Peter,

    I’d agree with the statement that hypertext brings together all the pieces of REST (resources, single API, URIs, hyperlinked representations) and shows its full power. So I think we are in agreements – REST without hypertext would negate many of the benefits.

    Reply
  9. dstar
    August 13, 2007

    A _large_ part of the problem is that I have yet to see an article that explains

    a) Why I, as someone with an existing rails app, would want to refactor it to be RESTful
    b) How to do so via simple examples

    but doesn’t

    c) Assume I already know the terminology

    A related issue is that all the articles I’v seen so far seem to think that because I’m building a RESTful app, I want to be serving XML and HTML and perhaps other formats via the same action, which is orthagonal to REST and does nothing more than complicate the issue (indeed, one article I looked at seemed to thing that moving to REST was the only way to do so).

    I’d really _like_ an article that fit criteria a, b, and c — I don’t suppose you know of one?

    Reply
  10. August 13, 2007

    Yes, I meant S3 as an example of a services that eschews links.

    Reply
  11. Charlie Savage –
    August 13, 2007

    Dstar,

    I took a stab at a) and b) last year – see my article about a [REST Controller for Rails](http://cfis.savagexi.com/articles/2006/03/26/rest-controller-for-rails).

    As for c), I like Joe Gregorio’s [articles](http://www.xml.com/pub/au/225) on xml.com

    And for your last point, it is part of REST since a resource can be represented using any number of formats (HTML, XML, JSON, ATOM, etc.). But whether you want to do such a thing or not is up to you (I find it helpful, see my [article ](http://cfis.savagexi.com/articles/2007/08/06/making-rails-better-content-negotiation) about content negotiation).

    Reply
  12. Charlie Savage –
    August 13, 2007

    Sean,

    Ah. I wasn’t eschewing links though, did I come across that way?

    Reply
  13. dstar
    August 13, 2007

    Charlie,

    Thanks for the pointer — that helped a fair amount. I do still have some questions, though.

    For example, in my app, I have Stories, which have Chapters. I have two seperate actions which correspond to show — show, which shows the chapter in its normal form, and show_draft, which shows comments and includes links to edit the individual paragraphs. How could I map both of those to a GET operation? It seems to me that I’ll have to include the action in the URL after all, to disambiguate.

    Reply
  14. August 13, 2007

    I don’t think you did, Charlie. I was just offering a concrete example to help people understand comment #1.

    Reply
  15. Charlie
    August 13, 2007

    Dstar,

    There are two things here. First, you have “nested” resources. From the outside, the url could look like this:

    /stories/1/chapter/2

    As far as editing and showing a draft, what we do is:

    /stories/1
    /stories/editor/1

    Thus a story editor is separate resource.

    There is a no clear “right” way to do this, what we do is different than what the Rails core teams suggests. The link above talk a bit about this issue. Also, Peter recently wrote an [article](http://pezra.barelyenough.org/blog/2007/07/hierarchical-resources-in-rails/) about nested resources you might want to look at.

    Reply
  16. Charlie
    August 13, 2007

    Sean,

    Doh – now I get it! Sorry for my confusion.

    Reply
  17. dstar
    August 13, 2007

    Okay, I think I get it now — instead of ‘show’ and ‘show_draft’ actions, I would treat them as separate things:
    /stories/1/chapter/draft/1
    /stories/1/chapter/1/

    Reply
  18. Charlie
    August 13, 2007

    dstar,

    Yup, that’s what I would do.

    Reply
  19. Jeremy McAnally
    August 13, 2007

    dstar: It would actually be /stories/1/chapter/1/draft

    Reply
  20. Espen
    August 14, 2007

    Your article did at least give me a new perspective on a subject I am struggling to understand.

    Reply
  21. August 14, 2007

    Thanks for the interesting take. I need to learn more about REST.

    But I’ll provide some feedback anyway. I think REST proponents tend to hand wave away the complexity of large scale distributed systems by saying, “look, it’s just URLs and a few simple operations.”

    Well, yes, but the innate complexity of the system is still there, it’s just expressed differently. Suppose you need to do 100 different things with data. You can write an API with a 100 different service methods, or you can have 4 methods and move a lot of the complexity into the data itself.

    And the various manifestations of SOA spend a lot of time and complexity on describing data in generic, cross-platform ways because it is HARD to make things interoperable. I believe those difficulties still exist in making REST resources interoperable, but the culture allows for a certain amount of “we’ll make it work” rather than nailing down all the details.

    Finally, “How well does the ROA approach work in the real world? Well, the web is the most success distributed computer system in the world, so I would say quite well.”

    I don’t think that is proof at all. The web is VERY forgiving of errors, missing resources, and inadequately structured data because navigation of the web is primarily driven by human intelligence, which can handle sloppiness better than automated distributed systems.

    Reply
  22. Charlie Savage –
    August 14, 2007

    Hi Dennis,

    Thanks for your thoughts.

    On your first point, remember in SOA that interoperability can only happen when a client and server can find each other (via a URI on the web), agree on an API and agree on a data format.

    In ROA the API barrier is removed – a client and server have to find each other and agree on a data format. So now you only have to solve the data format issue.

    So I agree the data format (representation) is very important, but in practice it has turned out be solvable since ROA makes the representation a point of extensibility (for example, in the last few years we’ve seen RSS and Atom emerge as new agreed-upon representation formats).

    As far as the existence of the Web not being proof that it works – I have to disagree with you there. Sure the Web is forgiving – but maybe that means you need to have a forgiving system to scale globally. Also navigation is driven by links – machines can follow them just as well – otherwise Google wouldn’t exist.

    I do agree for automated systems, HTML/XHTML leave a lot be desired as formats – but that’s ok because for automated systems they’ll be replaced by Atom and the Atom Publishing Protocol as I’ve talked about [before](http://cfis.savagexi.com/articles/2007/04/26/atom-will-change-the-world).

    Reply
  23. August 15, 2007

    Hi Charlie,

    Thanks for the articles and explanations concerning REST and ROA. Hopefully we’ll be able to go back to putting the services/software cart after the data/format horse instead of the other way around.

    My focus has been on format with ObsKML http://nautilus.baruch.sc.edu/twiki_dmcc/bin/view/Main/ObsKML for my work with observations systems, in utilizing KML’s Metadata tag to help with data reuse beyond the limits of the single string description tag only.

    Reply
  24. Samuel A. Falvo II
    August 15, 2007

    > Take a look around – when is the last
    > time you saw a large distributed
    > computer system working across multiple
    > organizational boundaries that is based
    > on COM, CORBA, RMI, SOAP web services,
    > etc?

    Every single day of my life. In fact, I help maintain software which is based on proprietary RPC calls. Previous lifetimes has seen me developing user management systems for ISPs using CORBA. And somewhere in the middle, I worked for a company that used SOAP pretty well exclusively to make its internal software work.

    So, yeah, I’ve seen these packages running and the benefits to society that they offer VERY often indeed. And I continue to use and support them today.

    Reply
  25. Samuel A. Falvo II
    August 15, 2007

    > Anything of interest gets a globally
    > unique identifier, which is done via
    > URIs.

    CORBA objects each receive an IOR, guaranteed to be unique in the known universe.

    > Next, the service becomes uninteresting
    > because every service has the exact same
    > API. On the Web, the service API is
    > defined by HTTP verb’s GET, POST, PUT,
    > DELETE, etc.

    This is not an API. This is a networking layer. Sure, it’s an API between web browser and web server, but this “transport” (as it’s called) is completely transparent to the higher-level software.

    Likewise, CORBA also implements its own transport that is demonstrably superior to HTTP. It’s stateful, thus eliminating outright the need for the megabytes of software that exist PURELY to manage cookies, sessions, and other bizarre concepts.

    > Since all interesting data has a
    > globally unique identifier, it becomes
    > possible to create a web of links

    Since all CORBA objects refer to other objects via their IORs, CORBA also supports creating a web of object links.

    > Since there is only one service API,
    > it becomes possible to create a single
    > client (think of a browser) that can
    > access an infinite number of services
    > (think of websites)

    This is actually false on the global scale — that is, it is true ONLY for a certain subset of applications (namely, those applications whose data is formatted conveniently for human consumption). For all others, you end up base-64 encoding binary data, thus resulting in 25% additional bandwidth overheads, or worse, using XML to encapsulate data structures which are well known to both sides of the connection, thus increasing bandwidth requirements to an amazing 200% to 500% in the process.

    EA-IFF-85a files are binary files with recursive descent characteristics — they solve the exact same problem as XML does, but it does so without the bloat of XML. It allows for textual or binary data (indeed, IFF files are designed specifically with binary files in mind). But, alas, it died out, having been seen ONLY as a “file format for that Amiga platform that nobody SERIOUS uses.”

    I’ll admit that there is SOME coupling that occurs with CORBA’s IIOP protocol, but let’s not forget that both the client and the server have to agree on an interface to communicate to begin with. Since both interfaces are already known in advance on both sides of the connection, IN PRACTICE, tight coupling at the IIOP level has not been a problem.

    Versioning of interfaces happens by changing the name of the interface. If I release an object that implements Foo, and suddenly I find I need new functionality in Foo, I might re-release the object as Foo2, which inherits Foo as a base interface to support backward compatibility. IIOP allows this, and even encourages this.

    > Versioning issues between clients and
    > services are greatly reduced (although
    > they still happen when HTTP changes)

    They still happen even when HTTP _doesn’t_ change. Remember, HTTP is used only as a transport.

    See above point on how IIOP/CORBA handles versioning of interfaces.

    > State can be taken out of the system
    > and made into its own resource,
    > thereby making the system much more
    > scaleable

    IIOP connections are stateful in nature, and that is true. But that doesn’t mean the objects behind them have to be stateful. Indeed, architecturally, there is NO difference between a CORBA ORB and an HTTP server. None. The difference lies in how the connections are managed. As soon as an HTTP server is done spewing porn, spam, and advertisements down the pipe, it closes the connection. As soon as IIOP is done spewing, the connection remains live.

    In order for a service built on top of HTTP to remain stateful, complex mechanisms employing cookies and “sessions” are required. An incredible amount of software goes into maintaining sessions — so much so, in fact, that most languages now have specialized web design toolkits (Ruby on Rails, Zope, etc) which abstract away much of this duldrum.

    CORBA does not need this kind of complexity, since the objects on the other end CAN reliably be stateful. And, if the connection is lost for some reason, you can reconnect to that object and pick up where you left off (unless it’s a transactional object, in which case you’ll need to start your transaction over).

    In retrospect, it is patently clear that there is _nothing_ new under the sun here. Just as those who are unaware of Unix are doomed to repeat it, poorly, so to is the case with distributed services: those who don’t know CORBA are doomed to repeat CORBA … poorly.

    There is a silver lining to this tornado cell though. For those who are able to stretch their imaginations a bit, and are even half-way familiar with Haskell, you can think of the session management functionalities as implementing a state monad. Therefore, the HTTP approach tends to be look “functional” in nature; it has to, as the protocol is a stateless protocol. The only way to reliably maintain state from page to page, therefore, is to pass your state around as hidden form entries or as parts of the URL (both of which are considered “parameters” to the web-based functions you’re hitting).

    Unfortunately, this state monad is muddied by the fact that you’re now maintaining state in a database, which just screams for an imperative method of programming. If we’re going to be using state monads to model state in a program, then there is certainly nothing preventing the use of CORBA at the level of a functional program either, since socket I/O is just another case of using a state monad.

    We’re back once more to where CORBA shines and HTTP is simply inadequate.

    Reply
  26. Charlie Savage –
    August 15, 2007

    Hi Jeremy,

    Thanks for the note and link. Funny for me to say this – but have you looked at the observation functionality built into GML? I haven’t used it myself, so no comments on its suitability for what you need.

    Reply
  27. Jeremy Cothran
    August 15, 2007

    I hadn’t taken a look at GML’s observation functionality, but based on the description(fields are time,location,resultOf,using,subject) at http://geoweb.blog.com/313930/ it doesn’t provide the level of detail that I’m looking for directly although the ObsKML schema which provides the necessary detail(an observation list bound by the same time/location with each observation having a observation type, unit of measure type and measurement value(and optional elevation) could be linked/nested within GML,etc similar to what I’m doing with KML.

    Reply
  28. Oisin Hurley
    August 15, 2007

    Samuel said:

    “This is not an API. This is a networking layer. Sure,
    it’s an API between web browser and web server, but
    this “transport” (as it’s called) is completely
    transparent to the higher-level software.”

    One of the core issues here is whether you approach solutions as a ‘layerist’ – that is, you isolate the protocol from the programming, as CORBA does, or an ‘applicationist’ – the protocol directly maps to the agent that is to process the request.

    Your statement above indicates ‘layerist’ thinking, however most REST aficionados (and certainly ones that use HTTP as the basis for examples), will be part of the ‘applicationist’ camp.

    This fork in thinking is a vital and fundamental differentiator – for the REST-with-HTTP guy, those POST/PUT/GET methods _are the API_, the API to a resource.

    Reply
  29. Charlie Savage –
    August 15, 2007

    Hi Samuel,

    Wow – you win the contest for submitting the longest comment in the history of this blog. Let me respond to a couple of things you said, although maybe the better idea would be to write another post.

    To start, I’m not anti-CORBA or anti-JMI or anti-DCOM, etc. Such systems provide a workable solution within organizations – however I do not think they work inter-organizationally. I would be interested in hearing more about the examples you mentioned in your first comment.

    Next, I’ll disagree with you on HTTP – its a transfer, or application, level protocol and thus its purpose in life is to provide a single API for clients and servers that support it. No doubt HTTP has been abused by various protocols that simply use it as a bit stream to avoid firewalls, but that’s not HTTP’s fault.

    Now is IIOP better than HTTP? Is a non-question since they are such different beasts. HTTP is stateless, text-based protocol designed to provide a single application level API used to tranfer chunks of content. IIOP is stateful, binary protocol designed to support distributed objects.

    As far as versioning in CORBA, its sounds similar to COM (I’ve done a lot of COM/DCOM programming, but not CORBA), where a published interface is immutable. Its sounds good in practice, but reality has shown it doesn’t work very well. Perhaps you can dismiss this as Microsoft’s incompetence, but I think that’s a simplistic viewpoint. Versioning is a deep, hard problem which I won’t pretend to know anything about. But any distributed system based on interfaces will run right into it.

    So I stick to my point that CORBA and REST are fundamentally different – CORBA focuses on services and REST on resources. In my opinion, the focus on resources is in fact something new under the sun.

    To finish up, your musing on Haskell/monads/state/HTTP are interesting. HTTP’s statelessness is a good thing (leads to scalability), but dealing with application state (versus resource state) is a pain. Part of the problem is that browsers haven’t provided any infrastructure for managing application state (it belongs on the client), until very recently with the WHATG spec (though kudos to IE for offering some local storage for a long time now). And part of the problem is the cookie spec isn’t very good. But neither are the fault of HTTP.

    Anyway, thanks for your thoughts, and I enjoy reading your blog.

    Reply
  30. Samuel A. Falvo II
    August 16, 2007

    Wow. I must have had a REALLY bad day yesterday. Although the content wasn’t intended to be offensive, and I still don’t take it as such, I still think I could have worded things a whole lot better. Not only that, but, I completely missed half of what you wrote.

    Yes, I am a layerist, because at some point, SOME kind of interface has to exist. Each layer means more software to maintain, which means more bugs. Collapsing the stack is critical to me; my brain can only handle so much.

    That being said, I do have some other issues with REST — many compare REST to “what OO should have been” (to paraphrase a page on the restWiki — I can’t remember the link though). I see this as a flawed analogy; OO has always been about not just the data, but what you can do with it. Hence, data and functionality together.

    REST is an extension of the bus interface of the CPU — a CPU can read from memory, and write to it. That’s it. Ditto for the Commodore 8-bit Kernals — they were file oriented long before Unix became a household name, and implements an interface that some argue is purer than Plan-9’s. You had only four functions: open, read, write, close. That’s it; literally everything else was done either through tunneling through an overtly documented command channel, or through reading/writing state. Almost RESTful. It only lacked representational standards. Moving one step further, we have the WWW, where RESTful applications are essentially “active RAM.”

    I would like to see how security is addressed though. Object capabilities and ACLs both have been demonstrated to be equivalent, but object capabilities are much more scalable (control over individual methods for individual programs for individual users, something which ACLs are absolutely cumbersome at achieving). Yet, except for .htaccess-style ACLs, I can’t see how to map object capabilities to this model. As a result, I see security in a RESTful environment as being a significant challenge — something SOAP and its ilk can implement much more easily.

    Thanks.

    Reply
  31. William Tanksley
    August 17, 2007

    Object capabilities could be implemented with REST by naming the URI path components according to the HMAC hashes of the names rather than the pure English names.

    Reply

Leave a Reply

Your email address will not be published.

Top