Overview
Every week we will be sending out a status report on the Dojo Offline Toolkit project to keep folks abreast of what we have accomplished the last week and what is planned for the week ahead.
Last Week
Last week the focus was on syncing and local data storage. Just as in previous big tasks we grounded things by focusing on real world applications, we began work on the JavaScript API for syncing and local data storage with a real application: Moxie.
Moxie is a demo application created for Dojo Storage in the past, and is a web-based word editor and word-processor. At the beginning of last week, Moxie only did persistent local storage using Dojo Storage with no server-side component. To make sure I’m not wandering off into architecture astronautics, and to also always have a real-world demo application for Dojo Offline, I wanted to turn Moxie into the kinds of applications that the Dojo Offline Toolkit will be embedded into; one’s with a server-side system, some kind of web-servicey way for the client to interact with the server (REST, XML-RPC, etc.), and a relational database for storing data.
I modified Moxie to use a MySQL database for storing it’s documents on the server-side. Then, I created a simple data layer for accessing these documents in Java using JDBC; I didn’t necessarily need to use a relational database or a simple facade when dealing with the database (I could have done everything as a serialized hashtable into the filesystem, for example), but I imagine that most applications that will embed the Dojo Offline Toolkit will be constructed in a similar way, and I want to catch any synchronization or offline issues that might be common. All of this was done in Java.
Next, I exposed a simple RESTian API for Moxie to get documents, post new documents, delete documents, get a list of documents, etc. using a Java servlet . REST is a simple alternative for web-based services based on HTTP itself, as opposed to XML-RPC, SOAP, etc.
The next step was to modify Moxie’s client-side to now interact with this server-side, using the RESTian APIs to get documents, post new ones and updated ones, etc.. At this point, Moxie was now a traditional web application — it did not save it’s data into Dojo Storage, but rather loaded and saved its data and list through the server-side’s RESTian API.
Jumping back to the server, I created a full synchronization framework, loosely based on the API that was in our Dojo Offline API. I got this sync framework to take in the JSON client sync info like we laid out in that blog post, and got the server to correctly process this JSON and produce it’s own JSON response. I hand fed the client-side’s JSON into this server-side sync system, and the server would correctly respond with the right sync responses. I also made sure that this sync infrastructure is generic, so that any Java server-side apps that need sychronization with the Dojo Offline Toolkit — it works generically with different item types, such as “documents”, “tasks”, etc. I won’t describe it in detail, because, as you’ll see in our next section, I found something better.
This Week
After building the server-side sync framework, I realized some important facts: this was just too hard to build, and there’s a simpler way to do things. While I was able to roll a generic Java-based server-side syncing framework, what about someone working in PHP or Ruby? Are they going to roll custom support for Dojo Offline’s JSON-based syncing protocol? It’s just too difficult.
I then realized that we don’t need any real support from the server-side for syncing — in fact, the JSON-based Dojo Offline sync protocol is actually a premature optimization that’s not needed. When I created the server-side Moxie, I created RESTian based APIs that allowed a traditional web-client to communicate with the server, then I created the SyncServlet which allowed a web-client to do syncing through a completely different avenue. If you crack open our Dojo Offline sync protocol, you will basicly just see a list of commands, such as “create this item”, “delete this item”, etc.
This lead me to realize that I can ditch our sync protocol and the need for server-side support, and instead have the client-side be a little more involved. This is great, since it lowers the bar even more for folks to be able to use Dojo Offline in their applications without heavy re-engineering.
On the client-side, in the Dojo Offline Blog post, I described some of the data structures that a programmer writes into so that their data is persisted on the client-side and is synced:
// called when the page is finished loading and the offline toolkit
// is ready to be used
offline.onLoad = function(){
// define where to put our offline user data; these
// data structures will automatically be kept in sync
// so we can use them in our application, and will
// automatically be persisted in local client-side
// storage
contacts = offline.getDataStore("contact");
}
function addContact(contact){
contacts.newItem(contact);
}
When you use a DataStore, such as the ‘contacts’ data structure, you use methods like newItem(), deleteItem(), etc. to work with it — this causes a command log to be updated in the background transparently. In the old sync design we would essentially serialize this into JSON and send it to the server, where the server would deserialize and replay the log, running each of the commands, such as “update” and “create” on the server-side. Why should we do this, though? Why not just allow the application programmer to place their own hooks on the client-side, and simply replay the command log on the client-side? The command log consists of a bunch of CREATE, UPDATE, DELETE, etc commands — the client-side can just have a callback to know when a create command has occurred, for example, and simply call a pre-existing server-side API that exists for creating that kind of item, updating that kind of item, etc. Most application’s will already have these kinds of APIs, since they are needed by Ajax application’s anyway, so why not leverage them?
For example, in the above ‘contacts’ DataStore, the client-side can place a function on to contacts.sync.onCreateCommand that will be called whenever a create command is inside the command log. This method could then take the command data, which would be a new contact, and call a pre-existing server API for creating a contact.
This week I will build the client-side using the approach described above. Changing our sync approach won’t create any new work for me, since I hadn’t tied the server-side syncing into the client-side yet — plus, this sync approach is simpler, so it should be simpler to implement.