Non-trivial data often has structures that cannot be well-defined with normal linear, acyclic data descriptions. Data that consists of cycles, many-to-one relationships, and non-local references often requires custom strategies for serialization and transfer of the data over JSON. Dojo 1.2 has added support for JSON referencing with the dojox.json.ref module. This module provides support for several forms of referencing including circular, multiple, inter-message, and lazy referencing using id, path, and combined referencing forms.
These references can be automatically generated on serialization and resolved on deserialization. JSON referencing is designed to decouple the topology of data structures from the meaning of the data structures in a way that is consistent with runtime object models of modern OO languages (foremost JavaScript). Separating graph connectedness concerns allows referencing and dereferencing to be handled by dojox.json.ref with minimal or no knowledge of and integration with higher level JSON consumers and producers, and permits more extensive JavaScript serialization capabilities.
dojox.json.ref API
First we will look at how to use the new module to serialize JavaScript objects. Perhaps the simplest data structure that requires referencing is a circular reference. We can easily create an object with a circular reference in JavaScript:
var obj = {};
obj.me = obj;
Most JSON libraries cannot serialize such an object, and will generally result in a stack overflow error. However, we can serialize this with dojox.json.ref:
var jsonWithCircularRef = dojox.json.ref.toJson(obj);
And de-serialize the JSON back to an object:
obj = dojox.json.ref.fromJson(jsonWithCircularRef);
obj.me == obj // -> true, the reproduced object will
//have a property named "me" with a value of itself.
Another common data topology is multiple references:
var me = {
name:"Kris",
father:{name:"Bill"},
mother:{name:"Karen"}};
me.father.wife = me.mother;
var jsonMe = dojox.json.ref.toJson(me); // serialize me
var newMe = dojox.json.ref.fromJson(jsonMe); // de-serialize me
newMe.father.wife == newMe.mother // -> true, the reproduced object will
// properly reproduce both references to the same object.
JSON Referencing Convention
Now we will look at how to create JSON with references that can be parsed and resolved by dojox.json.ref. The JSON referencing format is primarily based on a reference object which is a special JSON object with a single property named $ref with a value indicating the target of the reference. The simplest way to create references is with id references. This is done by adding ids to objects, and then referencing the object by id. A JSON object with a circular reference as in the first example would look like:
{"id":"1","me":{"$ref":"1"}}
This could be loaded from a file, and then de-serialized by calling:
dojox.json.ref.fromJson(jsonFromFile)
We could also reproduce the second example with id-based referencing:
{"name":"Kris",
"father":{"name":"Bill","wife":{"$ref":"2"}},
"mother":{"name":"Karen","id":"2"}}
Path-Based Referencing
Id-based referencing is convenient and readable in situations where objects already have an existing identity property. However, when objects do not already have an identity property, generating ids for every object that might be referenced can be difficult, verbose, time-consuming, and may generate extra properties that could interfere with the intended structure of the data. In these situations, path-based referencing is a cleaner approach. Path-based referencing is inspired by JSONPath, which is a language agnostic convention for querying JSON data, with a syntax that is similar to JavaScript (and supported by Dojo). With a path-based reference, # denotes the root of the JSON object, and .property and [‘property’] syntax are used to denote property values in the JSON object. Therefore, to create the circular reference example with path-based referencing:
{"me":{"$ref":"#"}}
And to create the multiple reference example with path-based referencing:
{"name":"Kris",
"father":{"name":"Bill","wife":{"$ref":"#mother"}},
"mother":{"name":"Karen"}}
Combined Path/Id Referencing
Id and path based referencing can be combined. Rather than starting a path reference with the # root reference, a path reference may begin with an id reference. For example one could have a reference:
{"$ref":"5.foo.bar"}
This would indicate that this reference should be resolved by finding the object with an id of “5”, and finding the value of the foo property of that object and then finding the value of bar property of that value. Combined path/id referencing can be very valuable in situations where not all objects have ids. In particular, it is impossible to give keyed properties (and consequently ids) to arrays with JSON, so combining path/id referencing can be essential in referencing arrays and values in arrays when id referencing is necessary (inter-message referencing).
dojox.json.ref.toJson will automatically determine the optimum referencing technique to use when serializing JavaScript objects. If ids are present on objects, than id-based referencing (or combined) will be used, otherwise path-based referencing is used. In the first example where a JavaScript object was serialized, path-based referencing would be used since there were no ids on the objects. dojox.json.ref.fromJson can also automatically handle all three forms of referencing.
Inter-Message Referencing
Referencing is not limited to within a single JSON string/message. References can be made to other objects that have been de-serialized in previous (or future) messages. This can be very important for preserving identity of referenced objects as well as reducing the amount of data that must be transmitted. For example if we loaded an object describing me from a server:
{"name":"Kris",
"father":{
"id":"bill",
"name":"Bill"}
}And in the next message, we loaded an object describing my sister, we would the object to reference the same father object (and not have to download it again):
{"name":"Kari",
"father":{"$ref":"bill"}
}When dojox.json.ref.fromJson() parses the second JSON message, it should resolve the reference to the object that it created for my father in the first message. firstObject.father should be equal to secondObject.father and resolve references from the other. Not only does this preserve object identity, it can significantly reduce the size of the transferred data by eliminating the need to retransmit large objects that referenced multiple times.
Inter-message references are dependent on providing an index map object as a parameter to fromJson() (see the API description below). If the same index object is provided, than the inter-message references can be resolved. Otherwise references will be limited to that message. The new JsonRestStore module uses the Rest service with the referencing module, and a single index is used across Rest requests with automatic contextualization.
Lazy Referencing
dojox.json.ref also allows objects to reference other objects that have not been loaded and parsed yet. For example, if this object comes first:
{"name":"Kari",
"father":{"$ref":"bill"}
}Then the father property will be a lazy reference to an object that has not been loaded yet. Lazy references are denoted with a special dojo.Deferred value. Lazy referencing can be used in conjunction with the dojox.rpc.Rest service to automatically load referenced values when they are needed. To load a lazy value, simply a callback listener using addCallback to the Deferred object. As soon as the addCallback is called, the referenced value will be loaded, if it has not already been loaded. Lazy referencing can be valuable for maintaining the topology of JSON object structures, while deferring the loading of large sets of data until necessary, using the standard Dojo concepts of Deferred values.
JSON Referencing + Relative URLs
Using ids within a single logical domain can limit the scope of referencing. Data stores may wish to reference data accessible from different paths, and decentralized cross-site data structures may be advantageous. dojox.json.ref supports standard ids along with references outside of origin id domain using relative URLs for references. By equating references to relative URLs, references have a natural correspondence to resource location. JSON referencing leverages the ubiquity and flexibility of the standard relative URL convention to extend the scope of potential referents without compromising the simplicity of simple intra-message references.
To illustrate with an example, suppose we retrieve the resource from http://mysite.com/Person/1 and receive:
{"id":"1","name":"Kris", "father":{"$ref":"2"}}
This is a simple id-based lazy reference, with the father property referencing the object with an id of “2” within our context. However, using the concept of relative URLs in conjunction with our knowledge of where the resource was requested we can infer absolute resource locations for the objects and their references. We can determine that the target of the reference can be found at a location of http://mysite.com/Person/2. Note that we didn’t have to change the syntax of the JSON referencing at all with references within the message and context.
Now imagine that we want to load an object that does make a reference to a different context. Suppose, we load a purchase order object from http://mysite.com/PO/1:
{"id":"1",
"amount":5.99,
"purchasedBy":{"$ref":"/Person/1"}
}Now we can use relative URLs to determine that the purchasedBy property is referring to the object from http://mysite.com/Person/1. If this object has already been loaded, the property can be directly resolved by dojox.json.ref.fromJson, and if not, it can create a lazy reference that will load the object from the correct URL as needed. It is important to note that the message and client can remain agnostic of how this reference was formed. The reference may be a result of a relationship in a RDBMS, a direct reference in non-relational databases, or manually created. The client does not need to understand the details of how the reference was formed.
We can even go a step further and use cross-site references. Suppose we are a reseller, and we want to actually link directly to the manufacturer’s data for the products in our purchase order. We might have a purchase order like:
{"id":"1",
"amount":5.99,
"purchasedBy":{"$ref":"/Person/1"},
"product":{"$ref":"https://www.sitepen.com/t-shirt"}
}
This is allows us to maintain a much more DRY data storage system. Once again, dojox.json.ref can handle resolving references and client code can deal directly with the data without concern for the underlying details of recreating the topology of the data structure even if connections span domains.
Additional dojox.json.ref Options
In order for dojox.json.ref to properly handle references that cross messages, domains, and contexts, it is necessary to provide additional information to the functions. dojox.json.ref.fromJson takes the following parameters:
dojox.json.ref.fromJson(str, args)- str – The JSON string to parse
- args – A keyword set of arguments that can have the following properties (all are optional):
- index – The index object (map) to use to store an index of all the objects. If you are using inter-message referencing, you must provide the same object for each call.
- defaultId – The default id to use for the root object (if it doesn’t define it’s own id)
- idPrefix – The prefix to use for the ids as they enter the index. This allows multiple tables to use ids (that might otherwise collide) that enter the same global index. For example, if you request an object from the /Person/1, then you should provide an idPrefix of “/Person/”.
- idAttribute – Indicates what property is the identity property. This defaults to “id”. It is recommended that you use “id” as the identity property for simplicity and consistency.
dojox.json.ref.resolveJson(object, args)resolveJson works the same as fromJson, but it operates on a pre-parsed JSON object. Therefore dojox.json.ref.fromJson(str) is roughly the same as dojox.json.ref.resolveJson(eval('('+str+')'))
dojox.json.ref.toJson(it, prettyPrint, idPrefix)- it – The value to serialize.
- prettyPrint – Indicates whether to use a pretty printing of the JSON.
- idPrefix – Indicates what context to use for serializing the JSON.
dojox.json.ref is used for reference resolution, indexing, and serialization by dojox.data.JsonRestStore. JsonRestStore combines the REST service module with JSON referencing and automatically handles the interaction with dojox.json.ref to provide the proper context and index information to the serializer and de-serializer.
dojox.json.ref.fromJson also supports a non-JSON, JavaScript syntax, referencing convention. Rather than using:
...
property:{"$ref":"reference-target"}
...
to define a reference, you can alternately create a reference:
...
property:{ref("reference-target")}
...
Performance
JSON referencing adds additional processing to the serialization and de-serialization process, so it inevitably is slower than standard dojo.toJson and dojo.fromJson. The dojox.json.ref.fromJson is approximately 2-3 times slower than standard dojo.fromJson. However, fromJson is usually used for parsing data sent from a server, and dojox.json.ref.fromJson is generally still faster than most download speeds. For example, on IE8, dojox.json.ref.fromJson can parse JSON at about 1MB per second (of course, this is highly dependent on the structure of the data). When referencing can be used to eliminate redundancy in JSON messages, the performance gains due to the reduced data transfer can easily outweigh the extra client side processing costs. dojox.json.ref also simultaneously indexes objects as it is parsing, and this capability is leveraged by the new JsonRestStore to avoid any extra steps of indexing, generally canceling any client side processing performance hits in such situations.
dojox.json.ref.toJson has a negligible performance hit in comparison to the standard dojo.toJson (less than 5%).
Conclusion
Dojo’s new support for JSON referencing allows a wide range of referencing possibilities using a consistent simple convention, and permits much more flexibility in describing complex data structures and serializing JavaScript data.