RESTful JSON + Dojo Data

NOTE: This post is very out of date. For more up to date information about RESTful JSON in Dojo applications there are a number of more recent tutorials are available, such as the Dojo Store introduction, as well as tutorials demonstrating store-driven grids and trees, among others. You should also take a look at dstore, the next generation Dojo store architecture, for an even more modern take on RESTful JSON. We have a series of tutorials introducing the concepts.

Dojo 1.2 now includes an infrastructure for interacting with JSON REST services and data storage systems. JsonRestStore is a new Dojo Data store for interacting with the RESTful JSON data sources. This new data store allows you to communicate with server side database/persistent data storage using the Dojo Data API with JavaScript and efficiently handles create, read, update, and delete (CRUD) operations. This can greatly simplify client server communication, interacting can be done simply using straightforward JavaScript instead of having to build your communication for CRUD actions. In addition, Dojo data stores can plugin to many Dojo widgets (Dijits).

Complex database driven widgets can be utilized with minimal coding effort. RESTful JSON is an increasingly popular database interface, and in later posts we will look at how JsonRestStore can be used with Amazon S3, CouchDB, Persevere, and Ruby on Rails. The JsonRestStore fully implements the Dojo Data read, write, notification, and identity interfaces. Therefore it can be used with any widget that can utilize these data stores including widgets with editing and live auto-updating features.

Also new to Dojo 1.2 is the ServiceStore. JsonRestStore is an extension of ServiceStore. ServiceStore provides Dojo Data read and identity interface for remote web services. However, I will primarily focus on the JsonRestStore in this article.

Using a JsonRestStore

The JsonRestStore can be created with a REST service function, or it can be instantiated directly with a target URL. To get a JsonRestStore instance with a target URL:

recipeStore = new dojox.data.JsonRestStore({target:"/recipeStore"});

A JsonRestStore can be created from a REST service function which can be generated with dojox.rpc.Service:

services = dojox.rpc.Service("/mySMD");
recipeStore = new dojox.data.JsonRestStore({service:services.myRestService});

Fast and Compact Syntax

While you can use JsonRestStore and ServiceStore as you would another data store, they have been designed to support a more compact syntax for data interaction as well. This can be very beneficial if you have extensive code that interacts with the data store. First, several of the commonly used store methods are available as “static” functions, including getValue and setValue. Therefore we can easily define a convenient global or local functions for common operations.

The get, set, and save functions can be used with multiple JsonRestStore stores; it is not limited to the JsonRestStore from which it was retrieved.

With a few simple guidelines, you can actually interact with JsonRestStore items using standard JavaScript property syntax. JsonRestStore items are actually simple JavaScript objects, therefore you can always directly read properties from items.

get = recipeStore.getValue;
set = recipeStore.setValue;
save = recipeStore.save;
... // fetch an item
get(item,"foo"); // instead of using recipeStore.getValue(item,"foo");
set(item,"foo","bar"); // instead of using // recipeStore.setValue(item,"foo","bar");
save();

value = item.foo;
// instead of calling value = recipeStore.getValue(item,"foo");
var prop = "baz";
anotherValue = item[prop];
// instead of calling value = recipeStore.getValue(item,prop);

You can also modify properties of items using standard JavaScript syntax. However, in order for the JsonRestStore to know that an item has been modified, you must call the changing() method prior to modifying properties. Once you have called changing(), the object will be denoted as being dirty and you can make as many changes to the object (the object’s properties) as you want until you call the save() method. Once you call the save() method, you need to call changing() again before modifying an item again. Note, that it is safe to call changing() multiple times.

recipeStore.changing(item);
// mark item as dirty
item.foo = "bar";
// instead of calling value = recipeStore.setValue(item,"foo","bar");
item[prop] = 4;
// instead of calling value = recipeStore.setValue(item,prop,4);
recipeStore.save();
// this always must be called to commit the changes,
// regardless of whether you use setValue or not.

If you are writing extensive code based on a data store, this can make your code much more compact, readable, and maintainable. In addition, there are significant performance benefits from direct property access. Property access is 10-50 times faster than using getValue. For large or computationally intense use of data stores, direct property access may be very important for optimal performance.

There are a couple of things to remember with direct property access. First, setting properties directly does not trigger notifications. Since setting properties is usually much less frequent than reading properties, it is recommended that in most situations that you only use direct property access for reading, and use the standard simple setValue (or a convenience copy) for writing.

Also, the JsonRestStore architecture supports lazy loading using JSON referencing. When a value is accessed that is referenced, but not yet loaded, the property value can be tested with the isItemLoaded function. You can load and access the value by calling the loadItem function. You can also use the standard getValue method (or a copy of it as demonstrated above) in situations where the property may be lazy, and the getValue method will automatically download the value on demand. If you are not using JSON referencing and lazy loading you can always directly access properties using normal JavaScript syntax. However, if you are using lazy loading, when a property value is a lazy loaded you can test for values that have not been loaded and load them asynchronously:

myValue = item.foo;
if(recipeStore.isItemLoaded(myValue)){
	recipeStore.loadItem({item:myValue,onItem:function(result){
		... resume with result ...
	}});
}

or simply using synchronous loading (for lazy loaded properties, the Dojo Data getValue function is the easiest form of access):

myValue = recipeStore.getValue(item,"foo");

Remember, this is only necessary if you are using lazy loading via JSON referencing. Also, using direct property access is purely optional, you can use the JsonRestStore with the same API as any other Dojo Data store.

JsonRestStore also provides a constructor for aesthetic creation of new objects. This constructor is accessible with the getConstructor method:

Recipe = recipeStore.getConstructor();
// create a new recipe object instead of recipeStore.newItem();
var recipe = new Recipe();

The constructor also includes a load function for convenient access to the Rest service querying and fetching items by id.

Now we can put this all together, If we first create aliases for getValue, setValue, save, and changing and then create the Recipe constructor we can write:

// make Apple Pie
var query = Recipe.load("?type='Pie'"); // query for the recipes for pie
query.addCallback(function(queryResults){ // when the results are returned
	var recipe = queryResults[0]; // get the first result
	if(recipe){
		if(recipe.name!="Apple Pie"){ // get the name property
			changing(recipe); // indicate we are changing properties
			recipe.name = "Apple Pie"; // rename this recipe to apple pie
		}
	}else{
		// create a new Recipe for apple pie
		recipe = new Recipe({name:"Apple Pie"});
	}
	save(); // save our changes to the database
});

JsonRestStore and ServiceStore also support synchronous mode. Synchronous requests can create a poor user experience since they generally lock up the browser while the browser waits for a response from the server. However, synchronous mode can also simplify programming. Since it is not necessary to use nested callbacks for handling responses, they can be accessed directly after making a request inducing call. Firefox 3 has also eliminated the browser lock-up associated with synchronous calls, making it a more attractive mechanism. To use synchronous mode, you include the syncMode option when instantiating a data store:

recipeStore = new dojox.data.JsonRestStore({target:"/data",syncMode:true});

In synchronous mode, one can fetch without providing a callback, by directly accessing the results property from the request object that is returned from the fetch operation:

var queryResults = recipeStore.fetch({query:"?tastes='good'"}).results;
var firstItem = queryResults[0];

Implementing a RESTful JSON Server

The JsonRestStore can be used with a number of server storage systems that support HTTP JSON/REST interface without any server-side coding. Several JsonRestStore extensions designed to easily connect with Persevere, CouchDB, and Amazon S3 with minimal configuration are included with Dojo 1.2. You can also easily create your JSON/REST interface for an existing web application and database. This is essentially done by implementing the HTTP methods GET, PUT, POST, and DELETE according to the HTTP specifications and providing the data in JSON format. While the JsonRestStore can be configured (using dojox.rpc.Service) to support other configurations, generally rows/items/objects should be accessible using URLs of the form /table/id. GET is used to retrieve objects and perform queries, POST is used to create new objects (by POSTing to /table/), PUT is used to modify objects, and DELETE deletes objects. You can also follow the example used by other JSON/REST interfaces like Persevere and CouchDB.

Transactions

JsonRestStore provides transaction state information so that servers can implement transactions that correspond to the Dojo Data it saves if desired (this is not necessary for a server to implement in order to support REST). Transactions are indicated by a X-Transaction header in the modifications requests. If the X-Transaction header has a value of open, this means that further requests will be delivered that should be included in the current transaction. Once a request is received without an X-Transaction header of open, the server can commit all the changes from the current request and the previous requests that indicated an open transaction. It is recommended that you utilize deterministic request ordering and page sessions if you implement JsonRestStore directed transactions on the server.

JsonRestStore also features a shared repository of transactional data between all JsonRestStore instances. Therefore, if you save a change, all the JsonRestStore data store’s unsaved data will be committed. This means that you don’t have to track which data stores have modified data, and it also means that you transactions can involve modifications across multiple data stores and corresponding server tables.

Build with, not on JsonRestStore

The JSON REST infrastructure is composed of a layer of modules that can easily be utilized and extended on their own, following the Dojo philosophy of extensibility. First, JsonRestStore is an extension of the ServiceStore data store and is designed for provider/store separation. The ServiceStore is a read-only data store built such that various different remote data providers can plugin to the ServiceStore. Thus the ServiceStore provides an adapter between the remote communication provider and widgets. Within dojox.rpc, several commonly used remote communication plugins are provided including REST, JSON-RPC (version 1 and 2), JSONP, and direct POST and GET. You can also easily create custom remote data providers, by creating a service function that takes parameters and returns a dojo.Deferred object that will receive the result of the query. These can all be used with the ServiceStore. The demonstration of using ServiceStore with a Yahoo search web service shows this in action. You should use the ServiceStore whenever you are working with a read-only web service (that doesn’t support full bi-directional REST), as it is lighter than JsonRestStore.

ServiceStore only provides read capabilities (the Dojo Data Read API), however, writable remote communication providers (REST services) can plugin to the JsonRestStore for full read-write capability. A writable provider is one that provides put, post, and delete functions as properties of the main retrieval function. Once again, you can use the HTTP/REST compliant provider included with Dojo (dojox.rpc.Rest) or you can create your own.

Build with REST Infrastructure

dojox.rpc.Rest can also be used directly. A REST service can be constructed with dojox.rpc.Rest:

var restService = dojo.rpc.Rest("/myRestTarget",true);

This indicates that the rest service can be found at /myRestTarget and it is a JSON service, which enables additional caching and optimization capabilities. You can retrieve data from this service:

restService("10") // retrieve the object with an id/resource location of 10

This would result in a fetch from the local resource /myRestTarget/10. You can modify this resource with a PUT:

restService.put("10",{foo:"bar"});

This will store the provided JSON object into the target resource. You can also use restService.post and restService.delete to further directly modify resources. JsonRestStore uses these access points to persist all data changes in the data store. New objects are added with a service.post, deleted objects are removed with a service.delete and modifying objects are updated with a service.put.

The dojox.rpc.JsonRest module is the core engine behind JsonRestStore. JsonRest can be used directly, providing a compact API with the core features of JsonRestStore, including transactional data interaction and query and object loading. dojox.rpc.JsonRest is a singleton object, and provides functions: fetch, commit, revert, changing, deleteObject, getConstructor, and isDirty. These correspond to the same functions on JsonRestStore, but the single JsonRest object can be used without requiring separate data stores to be used to access the API. If you do not need the Dojo Data API, and you only need JSON/REST interaction, you can go lighter, and directly interact with dojox.rpc.JsonRest. However, because the Dojo Data API enables you to plug stores into numerous widgets with minimal effort, generally using JsonRestStore is preferable to directly using JsonRest.

JsonRestStore also has comprehensive referencing capabilities, including circular, multiple, and cross-store, and cross-site references. This feature is also provided by a separate JSON referencing module, dojox.json.ref for modularity and extensibility. Therefore, you can use the JSON referencing module to resolve and serialize references for data exchange outside of JsonRestStore. For example, you could easily use referencing for JSON-RPC data, and even allow RPC parameters and return values that reference data from REST stores. This modular structure also allows you to extend or use alternate referencing conventions for JsonRestStore.

Live and Offline Data Stores

Because the JsonRestStore uses the extensible Rest service, JsonRestStore also works with HttpChannels, the new cometd transport/protocol module, to provide a Comet-powered real time view of data on a server. This does not require writing any additional event handlers. If you are using a server that supports HTTP Channels (like Persevere), you can simply add the HttpChannels and you can will have a live view of your data. Be sure to check out the JsonRestStore in action with live updates.

Likewise, JsonRestStore will also run in offline mode by simply adding the forthcoming dojox.rpc.OfflineRest module. The OfflineRest module augments the REST service. As a result, a JsonRestStore running off of a REST service will automatically be able to run in offline mode, with data changes saved locally when offline and automatically re-synced when connectivity is restored.

Dojo 1.2 now has a comprehensive infrastructure for RESTful interaction with servers. JsonRestStore is a modular data store with built in capabilities for interacting with servers in standards-based manner that already works with a number of existing server technologies, and provides a widely usable data store for server side storage that is easy for servers to interact with. Stay tuned for more posts on using JsonRestStore with various storage servers.