Dojo 1.6 introduces a new data store API called Dojo Object Store. This new store API is based on the HTML5 IndexedDB object store API and is designed to greatly simplify and ease the interaction and construction of Dojo stores. Based on HTML5 IndexedDB

Update: dstore is developing as a successor to Dojo object stores, so if you are starting a new application, you may want to look into this project. However, this blog post still describes an important API that is widely used, and is foundational for dstore, sharing much of the same philosophy and even much of the interface.

Dojo 1.10 also includes new stores in dojox/store for local database support. Check the latest blog posts for more information about this.

There are several key philosophies behind the Dojo object store API:

  • Separate UI from data concerns – This has long been the motivation for our data API. This API helps us to separate concerns and permits independent evolution of widgets and data providers.
  • Keep it simple – You can literally create a usable store by simply creating an object instance with query() and get() methods that return objects. As you add functionality like creating new objects, you add an add() method, and for updating objects you include a put() method.
  • Plain JavaScript Objects – Rather than using opaque items, the new object store API uses plain JavaScript objects. Once a get() method has returned an object, or a query() method has returned an array of objects, you can access and enumerate properties using standard JavaScript properties and for-in loops. Saving objects is a matter of passing a plain object to a put() method.
  • Promise-based – The interface is the same for sync and async except sync methods directly return values and async methods return promises. Any method can choose to be sync or async (promise-returning). This greatly simplifies interaction with known synchronous stores, since callbacks aren’t needed. This also greatly simplifies the API since asynchronous concerns are separated from the interface.
  • Simple feature discovery – No extra methods are needed to determine the feature set of a store. If you want to know if you can add new objects to a store, just check if it has an add() method; if you want to see if it is queryable, check if it has a query() method. If you want to create a read-only store, only implement the read functions (get, query, getIdentity).
  • Layered functionality – By layering functionality, it is possible to start with a lightweight, simple store and then add functionality as needed. Dojo’s core comes with a caching wrapper (dojo/store/Cache) for improving performance and a wrapper for notification of data changes (dojo/store/Observable). These store wrapper/layer modules are optional and can be added to any other store. This makes it possible for us to keep our core stores — dojo/store/Memory and dojo/store/JsonRest — very simple and small, and makes it possible for you to easily create new stores.

Dojo Object Store API is an interface between different data consumers to different data producers. This interface can have any implementation, and Dojo core comes with two key implementations that are commonly needed: dojo/store/Memory and dojo/store/JsonRest.

Simple: dojo/store/Memory

This is a very simple in-memory store. This is highly useful for quickly creating a store, particularly for smaller datasets. A Memory store can be created by simply providing a plain array of objects as the data source for this store, and then you can start querying and interacting with the store (see the Memory object store documentation for more usage details).

The Memory store is a synchronous store, which means it directly returns values, making it very easy to use. For example, to retrieve an object by id:

var product = productStore.get("slinky");

And again, the new object store returns plain objects, so we can get properties with simple property access:

var name = product.name;

Using plain objects makes updates easy as well:

product.name = "New name";
productStore.put(product);

One of the exciting new features in Dojo object stores is improved querying. Queries are made using the query() method and the returned result set provides convenient iterative methods — much like dojo.query — that can be consistently used regardless of whether the store is sync or async. Therefore, we can rely on using forEach, map, or filter methods. The memory store supports several forms of querying. First, we can query by name-value matches (like Dojo Data’s ItemFileReadStore). Here we query by category:

store.query({category:"shoe"}).forEach(function(shoe){
  // called for each match
});

Name-value matching provides a simple, easy query mechanism, but sometimes more complex queries are needed. The Memory store also accepts functions for filtering, thus allowing arbitrarily complex queries. For example, to query by products with a price less than 10:

store.query(function(product){
  return product.price < 10;
}).forEach(function(shoe){
  // called for each match
});

We can also refer to functions by name, and they will be looked up as a method on the store:

store.lessThanTen = function(product){
  return product.price < 10;
});
store.query("lessThanTen").forEach(function(shoe){
  // called for each match
});

JSON and REST: dojo/store/JsonRest

JsonRest assumes the existence of a server-side API, and is designed for interaction with a store. This implements a solid, standards-compliant HTTP/REST client interface to a server. The dojo/store/JsonRest follows REST principles for high scalability and is well-suited for large datasets. JsonRest is an asynchronous store, and returns promises from all the operational methods that are asynchronous (the exception: getIdentity should always be synchronous).

The JsonRest object store follows much the same approach to HTTP-compliant server interaction as dojox.data.JsonRestStore. However, JsonRest has been greatly simplified in the redesign for the new store API. A JsonRest store can be created by simply providing a URL to connect to a server (see the JsonRest object store documentation for more usage details):

require(['dojo/store/JsonRest'], function (JsonRest) {
  store = new JsonRest({target:"/Data/"});
  ...

Store methods intuitively map to HTTP methods. Calling store.get(“some-id”) would result in a GET request sent to /Data/some-id and return a promise/Deferred for the result. For example:

store.get("some-id").then(function(someObject){
  // use someObject
});

Calling store.remove(id) will similarly result in a DELETE request. The add(object) and put(object) methods will trigger requests too. If the object passed to put(object) (or add(object)) includes the identity property, a PUT request will be used. If the object does not include an id or the second argument (options) includes an incremental property set to true, a POST request will be used.

The request will include an If-Match: * header if options.overwrite is true or an If-None-Match: * header if options.overwrite is false or if add(object) is used. This communicates to the server when object creation or modification is expected.

Normalizing sync and async

If you are writing a widget that may be used with either a sync or an async store, we recommend leveraging dojo/when. Rather than relying on a then() method, which is only available if the store method is async, we can use dojo/when and any return value will be properly handled. For example if we were doing a get() with an unknown store, we could write:

require(['dojo/when'], function (when) {
  when(store.get(id), function(object){
    // this function will be called with object once get() is complete, 
    // regardless of whether it returns a promise, or directly and 
    // immediately returns a value
  });
  ...

The dojo.when() function can be used with any store method.

Client-Side Data Caching: dojo/store/Cache

In addition to the core store implementations, Dojo comes with two store wrappers. The first wrapper is dojo/store/Cache. This wrapper is called with two stores: a caching store and a master store. A typical scenario for using the Cache wrapper is using JsonRest as a master store and a Memory store as a client-side caching store. This allows you to leverage the JsonRest store for server communication of changes, while using the Memory store for caching and avoiding unnecessary HTTP requests for data. Here is an example of how we could set this up:

require(['dojo/store/Memory', 'dojo/store/JsonRest', 'dojo/store/Cache'],
    function (Memory, JsonRest, Cache) {
  memoryStore = new Memory({});
  restStore = new JsonRest({target:"/Data/"});
  store = new Cache(restStore, memoryStore);
  ...

Now we could perform a query with our aggregate store. Here we will just query for all the objects (we can query for everything by omitting a query):

var results = store.query();

This will result in the response being cached in the memory store. Later we can get() an object without incurring an HTTP request:

object = store.get("some-id");

Changes to data through put(), add(), and remove() are all reflected in the cached data. Querying generally requires finer grained application control over exactly what should be cached and what should not. Consequently, the Cache store does not attempt to automagically query against the cache. However, if you choose to query using the cache, it is trivial. Simply query the caching store, the memoryStore in our example:

memoryStore.query({category:"shoe"}).forEach(...);

Observing Data Updates: dojo/store/Observable

Dojo also comes with a store wrapper/layer for adding support for notification events of data changes. The Dojo object store API takes a very different approach to notifications than the legacy Dojo Data API. The old API was problematic in that notifications were store-wide, and it was impossible to determine exactly how an event should really affect a rendered result set.

The Dojo object store solves this problem by connecting the observation of notification events to query result sets instead of the store. With the dojo/store/Observable module, you can wrap a store and the resulting store will always have “observable” result sets returned from queries. That is, the object/array/promise returned from the query() method will have an observe() method that can be called to monitor the result set for changes. See the Observable store wrapper documentation for the exact signature of the observe() method and callback.

The Observable module makes it extremely easy to render a result set and then respond to any changes in the underlying data with real-time UI updates. Let’s look at an example. We will create an unordered list (<ul>) of items corresponding to objects in storage. We will first create the list, and then respond to changes:

require(['dojo/store/Observable', 'dojo/dom'],
    function (Observable, dom) {
  // first add observable functionality to our store
  store = new Observable(store); 

  var listNode = dom.byId("list");
  var itemNodes = [];
  // now we will do the query for data
  var shoes = store.query({category:"shoe"});

  // and then render each item returned
  shoes.forEach(function(shoe){
    // and we render each node
    insertRow(shoe, itemNodes.length);
  });

  // now we monitor the result set for any changes
  shoes.observe(function(object, removedFrom, insertedInto){
    if(removedFrom > -1){ // existing object removed
      dojo.destroy(itemNodes[removedFrom]);
      itemNodes.splice(removedFrom, 1);
    }
    if(insertedInto > -1){ // new or updated object inserted
      insertRow(object, insertedInto);
    }
  });
  function insertRow(product, index){
    return itemNodes.splice(index, 0, dojo.create("li",
      {innerHTML: product.name + ": " + product.price}, listNode));
  }

By setting up an observing function that can remove rows and insert rows, we are properly set to respond to any changes, including additions, deletions, and object updates. The Observable module even sends index updates such that updated objects will properly be moved to a new index, if there are sort order changes within the result set. Also note that by observing the shoes result set in the example, we will only get updates that match the criteria for this particular result set. If an object that is updated, deleted, or added, and it does not have a category of “shoe”, no notification events will be posted for this listener. If an object was not a “shoe” before and is updated to be a “shoe”, this will trigger a notification of an addition to the result set. If an object was a “shoe” before and is updated to not be a “shoe”, this will trigger a notification of a removal from the result set.

The Observable module also adds a notify() method to stores. This is extremely useful for Comet-driven real-time applications that asynchronously receive updates from a server and wish to notify the store (and all the store’s result set listeners) of the change. Be sure to check out Dojo Socket for more details on Comet and real-time apps with Dojo 1.6.

Working with Existing Widgets and Stores

Most of the Dijit widgets are still based on the legacy Dojo Data API. However, Dojo comes with an adapter for using a new object store with a Dojo Data-based widget. The dojo/data/ObjectStore module is an adapter that accepts an object store and returns a data store. Dojo also comes with an adapter for using a legacy data store with widgets that expect an object store. The dojo/store/DataStore module is an adapter that accepts a data store and returns an object store.

Hierarchy

The object store API defines a method for hierarchy, created by providing a getChildren(object, options) method. getChildren should be called with a parent object, and returns a set of children. The implementation of getChildren is generally application-specific, but there are a couple of common ways to implement hierarchy:

  • Objects include a “children” property with an array of children – With this approach, each object defines its children in an array, thus specifically preserving the order of children. This works well for smaller ordered data sets.
  • Objects include a parent reference – With this approach, each object defines its parent, and we retrieve children by querying for all objects with a given parent id. This works well for larger data sets that may involve additional handling like paging or sorting.

Adapting to Other Stores

Update: the dojox.storage are very old, legacy modules, and should probably be avoided. This section is included for historic/informative purposes. Because the store API is a common pattern, there are a number of libraries that can easily be adapted to the store API. The dojox.storage providers are a close match, except the put() signature is slightly different. This can easily be adapted:

var storage = dojo.delegate(dojox.storage);
var storage.put = function(object, options){
  var deferred = dojo.Deferred();
  dojox.storage.put(options.id || object.id, object, function(status){
    if(status == dojox.storage.FAILED){
      deferred.reject(status);
    }else if(status == dojox.storage.SUCCESS){
      deferred.resolve(status);
    }
  });
  return deferred;
};

Or we could adapt to Jens Arps StorageJS library, which uses set() instead of put():

var store = dojo.delegate(storage);
var store.put = function(object, options){
  return storage.set(options.id || object.id, object);
};

The StorageJS API also has an allKeys() method that could be converted to a query() method.

Object Stores

The new Dojo object store infrastructure has been redesigned from the ground up to incorporate the best ideas from dojo.data, standardize around HTML5 IndexedDB, simplify usage, and layer functionality. We are excited to start building applications with this new approach, and we are looking forward to the Dojo community’s feedback on this design.