jonm.dev

The Art of Writing Software

An Introduction to JSON-LD

Tags [ JSON-LD ]

I’ve been spending some time recently thinking about ways to maintain loose coupling between lots of different teams. I suspect this involves thinking about shared protocols between the teams (i.e. having well-defined interactions) more than worrying about internal “implementations” like what flavor of Agile you use or what programming language you write in.

The World Wide Web is a pretty obvious example of a system that did this well, and its architectural style was described in Roy Fielding’s thesis. I’m not going to refer to this style by its well-known acronym, because I’ve done a lot of writing about it in the past, and that’s not really the point. For this article, I just want to highlight that one of the key principles of the Web’s architecture is that of self-descriptive messages (which is, in turn, part of the uniform interface architectural constraint). In other words, a message should carry enough information that you can understand what it means and what to do with it. For example, the HTTP response containing the words you’re reading now came marked with a header declaring Content-Type: text/html, which in turn tells your browser how to process the stream of bytes: by parsing and rendering the markup.

Now, many APIs today use JSON as a message format. It is simple, has parsers available for many programming languages, and is easy for humans to read and write. However, JSON only gets us part of the way to what I really would think of as self-describing messages. If an API response is marked as Content-Type: application/json, it tells me how to parse it, but not what I can do with it or what it means. Even this web page has more semantics available than JSON, as your browser at least knows to render <a> tags as clickable links, even if it doesn’t know what the links mean, or what the content on this page is about.

Actually, though, that’s not quite true. The <head> of this page includes some metadata that allows programmatic clients to make some sense of the page content:

<head>
  <meta charset="utf-8" />
  <meta name="description" content="..." />
  <meta name="author" content="Jon Moore" />
  <meta property="og:title" content="An Introduction to JSON-LD" />
  <meta property="og:description" content="..." />
  <meta property="og:type" content="article" />
  <meta property="og:url" content="https://blog.jonm.dev/posts/intro-json-ld/" />
  <meta property="article:published_time" content="2019-11-30T10:53:43-05:00" />
  <meta property="article:modified_time" content="2019-11-30T10:53:43-05:00" />
  <meta itemprop="name" content="An Introduction to JSON-LD">
  <meta itemprop="description" content="...">
  <meta itemprop="datePublished" content="2010-08-11T21:34:00-05:00" />
  <meta itemprop="dateModified" content="2010-08-11T21:34:00-05:00" />
  <meta itemprop="wordCount" content="...">

  <meta itemprop="keywords" content="architecture,hypermedia,REST,REST API,RESTful web services,XHTML," />
  <meta name="twitter:card" content="summary"/>
  <meta name="twitter:title" content="An Introduction to JSON-LD"/>
  <meta name="twitter:description" content="..."/>
</head>

From here various services know how to make reasonable previews of this page, for example, although some duplication is required. Facebook uses OpenGraph, while Twitter uses Twitter Cards. Sigh.

But JSON, by itself, doesn’t give us that. If you were an API client, and you got the following blob of JSON (marked solely as application/json), would you know what it means?

{
	"vq": 6253282,
	"vq_fge": "6253282",
	"anzr": "Gjvggre NCV",
	"fperra_anzr": "GjvggreNCV",
	"ybpngvba": "Fna Senapvfpb, PN",
	"qrfpevcgvba": "Gur Erny Gjvggre NCV. Gjrrgf nobhg NCV punatrf, freivpr vffhrf naq bhe Qrirybcre Cyngsbez. Qba'g trg na nafjre? Vg'f ba zl jrofvgr.",
	"hey": "uggcf:\/\/g.pb\/8VxPmPQe19",
	"ragvgvrf": {
		"hey": {
			"heyf": [{
				"hey": "uggcf:\/\/g.pb\/8VxPmPQe19",
				"rkcnaqrq_hey": "uggcf:\/\/qrirybcre.gjvggre.pbz",
				"qvfcynl_hey": "qrirybcre.gjvggre.pbz",
				"vaqvprf": [
					0,
					23
				]
			}]
		},
		"qrfpevcgvba": {
			"heyf": []
		}
	},
	"cebgrpgrq": false,
	"sbyybjref_pbhag": 6133636,
	"sevraqf_pbhag": 12,
	"yvfgrq_pbhag": 12936,
	"perngrq_ng": "Jrq Znl 23 06:01:13 +0000 2007",
	"snibhevgrf_pbhag": 31,
	"irevsvrq": true,
	"fgnghfrf_pbhag": 3656,
	"cebsvyr_vzntr_hey": "uggcf:\/\/cof.gjvzt.pbz\/cebsvyr_vzntrf\/942858479592554497\/OonmYB9Y_abezny.wct",
	"qrsnhyg_cebsvyr": false,
	"qrsnhyg_cebsvyr_vzntr": false,
}

(This is a user object from the Twitter API, but with all the strings ROT13‘ed). But this might as well be what a programmatic client sees when you hand it raw JSON. There’s structure here (scalars, arrays, maps) but no meaning. What does the snibhevgrf_pbhag field mean? In fact, there are not even any hints that this came from the Twitter API!

Enter JSON-LD

JSON-LD is a specification being developed in the World Wide Web Consortium (W3C) as a media type, application/ld+json. This means you can still parse it as a plain-ol’ JSON document, but there are some additional conventions that add meaning to the content. It turns out there are a bunch of semantically-equivalent ways to annotate the relationships in JSON-LD. For example, we can use IRIs in place of field names, as a way of annotating a specific concept, like so:

{
	"http://purl.org/dc/terms/identifier": [ 6253282, "6253282" ],
	"http://purl.org/dc/terms/title": "Twitter API",
	"http://xmlns.com/foaf/spec/#term_nick": "TwitterAPI",
	"http://xmlns.com/foaf/spec/#term_based_near": "San Francisco, CA",
	"http://schema.org/description": "The Real Twitter API. Tweets about API changes, service issues and our Developer Platform. Don't get an answer? It's on my website.",
	"http://schema.org/url": "https:\/\/t.co\/8IkCzCDr19",
    ...
}

The interesting thing is that there exist a lot of standardized ontologies for the Semantic Web that already have standard names for certain concepts. For example, the notion of a “screen name” maps pretty cleanly onto the FOAF term nick. In fact, there is a whole list of ontologies in the recommended JSON-LD context.

But on the other hand, some of the terms in the Twitter user object are specific to Twitter, like default_profile_image (whether the user still has the default profile image or not). Now, it’d really be nice to be able to have a clear identifier for that field and its semantics; I’d love to be able to use an IRI like https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/user-object#default_profile_image for that, but (alas!) the field names on the Twitter documentation web page don’t have id attributes, so that anchor is not really a valid target. (If someone at Twitter reads this, please mark these!).

But in any event, the problem with going back around and modifying the underlying JSON in this way is that it would break all the existing clients. So one option instead is to add a @context field:

{
    "@context" : {
      "id" : "http://purl.org/dc/terms/identifier",
      "id_str" : "http://purl.org/dc/terms/identifier",
      "name" : "http://purl.org/dc/terms/title",
      "screen_name" : "http://xmlns.com/foaf/spec/#term_nick",
      "location" : "http://xmlns.com/foaf/spec/#term_based_near",
      "description" : "http://schema.org/description",
      "url" : {
        "@id" : "http://schema.org/url",
        "@type" : "@id"
      },
    },
	"id": 6253282,
	"id_str": "6253282",
	"name": "Twitter API",
	"screen_name": "TwitterAPI",
	"location": "San Francisco, CA",
	"description": "The Real Twitter API. Tweets about API changes, service issues and our Developer Platform. Don't get an answer? It's on my website.",
	"url": "https:\/\/t.co\/8IkCzCDr19",
    ...
}

You can think of this as a “dictionary” explaining what each of the other JSON fields mean. But of course, we don’t really want to include that with each of our API responses. So another option is to locate the context itself at another URL and just refer to it there; for example, this might be perfectly reasonable:

{
    "@context" : "https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/user-object.jsonld",
	"id": 6253282,
	"id_str": "6253282",
	"name": "Twitter API",
	"screen_name": "TwitterAPI",
	"location": "San Francisco, CA",
	"description": "The Real Twitter API. Tweets about API changes, service issues and our Developer Platform. Don't get an answer? It's on my website.",
	"url": "https:\/\/t.co\/8IkCzCDr19",
    ...
}

assuming there was a JSON-LD context describing the user object at that location. Finally, if we can’t actually modify the response itself, we could potentially add an HTTP Link header like so:

Link: <https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/user-object.jsonld>;
  rel="http://www.w3.org/ns/json-ld#context";
  type="application/ld+json"

So this gives me a smooth upgrade path to start annotating the responses with semantic information. Of course, if we were designing a new API from scratch, instead of screen_name would could use the recommended JSON-LD context and use the field name foaf:nick, since JSON-LD supports compacted/shortened IRIs like this.

I expect this type of semantic annotation might be an important enabler for serendipitous reuse of APIs within an enterprise, and relying on existing public ontologies can cut down on the amount of documentation that might need to be written for an API.