Here at Emergent One, we love meta things. A lot.

Our control panel communicates with our private REST API to define and configure yours. This is a little meta, when you think about it - an API that creates other APIs? It’s actually a lot more meta than you might guess.

Our private API is generated from the exact same framework as the APIs our customers generate.

Definitions of resources, which tables their data comes from, and their relationships to each all use the same data structures that our UI generates. Even the queries against our own database are generated by the the same code that creates queries for the APIs we host. In fact, the only handwritten queries in our codebase are the ones that download database schemas - everything else is generated from the API’s configuration.

The only difference is instead of reading that configuration from our database (where we store the configuration for other APIs), it.s defined in code. But these are all declarative statements - configuration, not logic.

There’s a huge advantage to eating our own dogfood like this. As we need to do more complex things with our own API, we are also building out the framework to enable those complex things for our customers. If we can’t figure out how to make it work for us, how are we ever going to make it work for them?

Our API is also a place where we can validate and test features before we launch them. We currently take advantage of a lot more features than we include in our hosted APIs. Things like write access, execution of custom code for non-data-driven resources, and pre- and post-request hooks. There’s a ton of UI design and usability work we still need to do to make those features available, but those features do exist, deep in meta-land. In this way, our control panel reflects the future complexity of apps you’ll be able to build around an API you make with us.

Our vision is to be able to use our own platform to create an API that duplicates our private API. We want to be able to change one line of configuration to have our front-end talk to that new API, and have everything still work. And then we could do it again… and again…

That’s the kind of technology we’re building: the kind that can produce itself as output.

Like I said: Here at Emergent One, we love meta things. A lot.

P.S. If that excites you, we’re hiring.

When we talk to people about the benefits of hypermedia in APIs, we often get asked for an example of how it is useful. I’ve always used the Web as a whole to show how the concept works: hypermedia clearly drives the Web, and the Web is clearly useful, dynamic, and successful. But some people have expressed frustration with that example - they’re not building the Web, of course. So in this post I’m going to explain how we use hypermedia in our API, and show how it has granted us increased flexibility with just a little extra up-front work.

Our control panel is a single-page web application built on Backbone.js. When someone logs in to our app, our back-end bootstraps the page with some models and collections. This information includes the full account information, and a few different entities owned by the account. We get this information by making calls to our API during the bootstraping process. This ensures that changes to these resources always make it to the control panel with no changes to the latter.

We’ve extended Backbone to support nested resources, and specially handle links. All of our resources have the same basic structure, which lets Backbone process all of them the same way. Here’s an example of a resource in our system that represents a database host, with some information stripped for readability, and some minor changes to illustrate some points.

{
  "@metadata": {
    "created": 1356545474,
    "id": "1",
    "links": [
      {
        "href": "http://.../accounts/items/1/children/hosts/items/1",
        "rel": "self",
        "type": "application/vnd.api-host.host"
      }
    ],
    "modified": 1356619856
  },
  "account": {
    "href": "http://.../accounts/items/1",
    "rel": "describedby",
    "type": "application/vnd.api-host.account"
  },
  "tables": {
    "href": "http://.../accounts/items/1/children/hosts/items/1/children/tables/items",
    "rel": "index",
    "type": "application/vnd.api-host.tables"
  }
}

You can see that:

• Resources have metadata in the @metadata field.
• All links are hashes that contain an href, a rel (relation) and a type (the media type of the target resource).
• The resource has a link with relation self in its metadata. This is the URL that was requested to get this resource. We tell Backbone to always use this URL when getting or saving resources.
• The account field is a link with relation describedby indicating that the URL provided contains the desired information.
• The tables field is an link with relation index meaning the link points to a collection.

What we’ve done with Backbone is write a parser for our resources that finds any field outside @metadata that looks like a link, and replaces it with a model or collection. We do this with a factory that inspects the type attribute of that link to create the object using the correct class. This factory basically acts as a media type registrar.

We’ve also extended the get method for models to do a little more work. For example:
var table_collection = host.get('tables');
gives us a collection of models, not the link itself. If we want to populate that collection immediately:
var table_collection = host.get('tables', {fill: true});

At no point in our application is a URL or Backbone collection constructed by hand, and models are only constructed by hand for new resources we need to persist.

This makes our application completely insensitive to URL changes. Only resource changes affect us now. Since our API is still young, things are still evolving. Using hypermedia has given us incredible flexibility to make these changes without altering our front-end.

Additionally, as we add new resources to our API, we only need to register the media type with our model factory, and we can immediately start using that new resource in our application. Since our Backbone model classes are generated dynamically by our factory, we don’t even have to write a new class.

The bottom line is that we focused on defining resources, not URLs. To our application, the URLs are incidental, and opaque - a sidenote. By building a consistent resource structure, focusing on hypermedia, using standardized link relations, and defining our own media types, we’ve been able to make changes more quickly and much more easily than otherwise. The initial cost of setting things up for this approach has easily paid itself back in ease at which we can evolve things in the face of changing requirements.

DHH posted a rant on hypermedia APIs earlier today, and we feel like he got a lot of things wrong. We’ve seen that the view he’s expressing comes from an incomplete understanding of the ideas that both defined the web in general, and that are pushing the current movement in API design.

In general, hypermedia in APIs is when resources provide links to other resources as part of its representation. “Proper” use of a REST or hypermedia API generally requires you only ever initiate communications with the API from the root (https://my.api.example/ versus https://my.api.example/some/other/url). From there, every response from the API will then contains links to other resources, often with a relation to explain how each resource relates to the next. In this way, you never guess a URL - you are always given it.

In theory, this lets the API structure evolve over time without breaking older applications, and allows applications to do new things without being changed. In design pattern language, we would say that this allows client and server to vary independently.

DHH seems to be arguing the following:

• Hypermedia doesn’t improve discoverability, documentation does, as there’s no guarantee every piece of data can be found.
• Bookmarking URLs destroys the benefit of using hypermedia, as you’d be saving a URL that might change.
• A single API client for many APIs is a myth we should stop chasing after.

Discovery in Hypermedia

For most data APIs, discovery is the core benefit of hypermedia - to communicate all related data points in your representations. These data points need to include not only individual data points but also collections of data points, or any other related resources. APIs that use hypermedia but don’t link to all possible locations are broken.

Let’s use search engines and crawlers as an example. If you have a page in your website that nothing links to, how can they find it? With no link to it, they can’t. The solution is a sitemap. This is an API, with a standard, using hypermedia, solely to enable discovery.

Secondly, I believe developers should need to read little or no documentation to start playing with an API. We’re inquisitive folk by nature, and enjoy experimenting with technology. Hypermedia enables this experimentation in a lightweight, convenient way. DHH suggests good API docs explain all possible attributes of a resource. The API should too. An API that someone can dive right into without documentation just to play is a good API.

Bookmarks

Saving API URLs is something I’ve thought a lot about over last few years. People are definitely going to do it, and DHH does make a good point that it can break down the benefits of hypermedia.

But there’s a simple and easy solution to the whole problem. There’s a ‘bookmark’ relation. A link with this relation could be a URL like https://my.api.example/find?type=project&id=5 or https://my.api.example/urn_url_converter?urn=urn:api.example:project:5, providing a universal way to locate an individual resource in an API where URLs can change.

The query string could really be whatever you want to uniquely and permanently identify the resource. So when I move the project resource from being at /projects to being under a User at /users/1/projects, that bookmark URL can still work. And when I rename Contact to Friend, I can still get you where you want to be.

And bookmarks will break, as they do on the web. When they do, hypermedia enables us to get back to where we want from a fixed, known starting point. Without it, code needs to be changed.

Universal API Client

Is it a myth?

It seems to work extremely well for the web - the largest and most successful application of RESTful concepts in the world. Hint: Your browser is an API client, HTML resources are hypermedia/HATEOAS in action, and javascript is code-on-demand to enhance the experience.

The movement towards REST/hypermedia in API design is an attempt to apply the concepts and techniques used to achieve the massive success of the web to the API world. Dismiss the hype, sure, but not the concepts. We, as a community, are still trying to figure out the right way to apply some of those.

As an example of where we actually do have some answers, well documented and consistently used custom media types can allow an API to easily understand “the difference between Flickr photos and Basecamp projects”. Media types help drive the web, so why not APIs?

Final Thoughts

I can’t help but wonder if DHH’s post is a sign that we may be moving into a Trough of Disillusionment with respect to REST/hypermedia APIs. That’s actually very exciting to me, because it means we’re one step closer to sitting on that shiny plateau. That’s probably where the mythical universal non-browser API client has the potential to rear its beautiful head, and once again change the way we interact with information forever.

If you visit our homepage, you’ll see a fun little effect in our headline. A lot of sites are doing similar things these days, so when I set out to build it, I was surprised that I couldn’t find some code anywhere online. Maybe I was searching for the wrong thing. What do you call that anyways? It’s not a carousel - those pretty universally go side-to-side. Searching for slot machine turned up quite a few actual slot machines. Text cycle got pretty close, but none of the results made me happy.

So, not being able to steal borrow code from someone else, I built it. It turned out to be a lot tricker than I expected, and getting the styles to behave was fairly frustrating at times. After trying a lot of different approaches, I was able to get things to a point where the required css classes were few, the markup was clean, and it worked for all text alignments.

You can check out the code in this jsfiddle.

Feel free to steal borrow this for yourself! I’ve tested it in the latest versions of Firefox, Chrome, Safari, Opera, and IE. Yep, it even works in IE.

Negotiating content-type is just one kind of content negotiation that can be performed over HTTP. There are other preferences that the client can communicate to the server in a request.

Accept-Language can contain a list of language codes. This is especially useful in an API for translating error messages, to ensure that the error comes back in a form that’s presentable to the end-user.

Accept-Charset communicates which character sets the client supports. This information can also be included as a parameter for text-based media types listed in the Accept header, but should be provided as it’s own header to be fully compliant with the HTTP standard.

Accept-Encoding lets the server know what sorts of compression the client can handle. Compressing HTTP responses is a huge performance boost.

Accept-Datetime is a new header, still in a draft state, that lets a client request the content from a specific date. If this header gains traction, it would add native support for revisioning content on the web.

This array of headers makes HTTP a highly flexible mechanism for transmitting information in various forms and ensuring the requestor can handle the response in the most efficient way possible.