CommonJS (formerly known as ServerJS) has become the essential hub around the development of server side JavaScript (SSJS). SSJS for years has suffered from fragmentation, but the CommonJS project has provided the momentum to bring different frameworks together and start building interoperable modules. SSJS is ripe with potential; JavaScript has been soaring in popularity and the ECMAScript 5 specification was recently accepted. JavaScript has proven itself as the language of the web’s client-side (even ActionScript is a derivative of JavaScript). The opportunity to use the same language on both client and server is certainly most realistic and viable with JavaScript as it avoids the need for translation.

CommonJS

CommonJS has focused on developing critical APIs for building reusable modules, particularly for server-side JavaScript environment. The server-side is generally based around database interaction, file I/O, HTTP serving, and the generation of data formats and HTML, whereas the client-side is based around DOM manipulation and the browser object model. There are certainly APIs that can be used on both sides, and JavaScript on the client and server invites the reuse of APIs where possible. The WebWorker, Indexed Database, and XHR APIs are promising to be enormously beneficial on the server side, and with excellent client server consistency. But still the server side requires special attention, and CommonJS is bringing the needed standards and conventions.

There have been a number of different areas that CommonJS has invested in for creating the necessary foundation for SSJS. The primary initial effort was in defining the module system, which provides an easy to use system for loading modules and their dependencies, while ensuring that each module is loaded only once. This module system is now very broadly implemented and extensively used by the other subsequent module APIs defined by CommonJS. The module system provides the essential basis for modular library and application development.

CommonJS is working on building up the low-level I/O components, defining the necessary binary data types, the APIs for file interaction, and encoding. There are proposals in place for unit testing, and a number of modules to provide concurrency support. And finally there is a HTTP interface specification known as JSGI.

Concurrency

One of the most central characteristics of a runtime is the concurrency approach. The inherent dangers of shared memory threading are widely understood, and the browsers wisely avoid exposing developers to the low level complications of shared-memory threads, alternately providing shared nothing event-loop concurrency. With the latter approach, data structures are not exposed to preemptive threads that could cause race conditions. Instead, events are queued up, and deterministically executed. Concurrent operations are handled with WebWorkers, in separate memory spaces, with the ability to communicate to other workers with message passing.

The CommonJS group defines APIs, so it is beyond the scope of CommonJS to dictate the concurrency model used by servers, but there is general consensus recommending that JavaScript-based servers utilize shared-nothing event loop concurrency, and CommonJS APIs are built to support this model. The CommonJS wiki includes proposals for exposing the WebWorker API through a worker module, and for exposing controls for the event loop with the event-queue module.

Promises

One key abstraction that is extremely useful for event-based systems is a promise. A promise is an abstraction that encapsulates the execution of an asynchronous action and its subsequent value that it returns. A promise can be thought of as representing the result of an asynchronous computation. The promise will receive the value returned by the call when it is finished, and acts as a placeholder for the returned value. Promises are critical to developing large scale applications that make heavy use of asynchronicity. Using callback-passing for asynchronous actions does not compose well. It is impossible to properly encapsulate functionality, switching an implementation from synchronous to asynchronous breaks APIs and creates complex flows of passing callbacks around to handle return values. With promises, values can be returned just like synchronous functions, and callers can listen for asynchronous results without requiring additional callback-passing APIs.

The promise API in CommonJS is an asynchronous handling system based on the collective experience of the promises in Dojo, Twisted, and ref_send. The ref_send promises focus on secure immutable promises, but lacks a reasonable API for promise implementors. Dojo provides a promise API (called Deferreds) with convenient chaining support, but has suffered from the use of mutable promises, with side-effect inducing callbacks complicating the data flow. The promise API combines the best of both, providing secure immutable promises with a very simple, easy to implement API for implementors, and full chaining support. The promise API in CommonJS provides a foundation that implementors can easily extend to provide the convenience functions from Dojo’s Deferred API as well as ref_send’s static operators.

An example of using CommonJS promise might look like:

requestSomeData("http://example.com/foo") // returns a promise for the response
    .then(function(response){ // 'then' is used to provide a promise handler 
        return JSON.parse(response.body); // parse the body
    }) // returns a promise for the parsed body
    .then(function(data){
        return data.price; // get the price
    }) // returns a promise for the price
    .then(function(price){ // print out the price when it is fulfilled
        print("The price is " + price);
    });

JSGI: HTTP Gateway

The JavaScript gate interface (JSGI) API has been asymptotically reaching consensus in the CommonJS group, as a standard API for serving HTTP requests through JavaScript application servers. The JSGI provides a very easy to use API for responding to HTTP requests, influenced by the design of WSGI and Rack. A basic JSGI application looks like:

app = function(request){
    return {
        status: 200,
        headers: {},
        body: ["Hello World"]
    };
};

JSGI has been intentionally designed to make it easy to write “middleware”, modules that exist at different places in the request handling chain to add various forms of functionality. Middleware can include support for cross-domain requests, logging, error handling, special method handling, mapping URLs to different applications, and much more.

Most existing HTTP interfaces were designed around a synchronous model, and have had an extremely difficult time adapting to the asynchronous model, which is essential for efficient Comet support. JSGI on the other hand, has been developed from the ground up with asynchronicity in mind, both to support Comet, and to work properly in an event-based environment. JSGI achieves support for asynchronicity while still maintaining a simple function call and return style by leveraging promises. Here we can see the power of promises, JSGI applications can return a promise to indicate that the response is not yet complete, and no additional callback parameter need to be passed around. With promise support throughout, an asynchronous JSGI app can be as simple as:

app = function(request){
    return doSomethingAsynchronous().then(function(result){
        return {
            status: 200,
            headers: {},
            body: ["Result ", result]
        };
    });
};

Since JSGI effectively provides a JSON/JavaScript representation of the HTTP protocol, JSGI has powerful opportunities for use in communication outside of just HTTP server handling. JSGI can be used as an API for making HTTP requests as a client (as a promise based alternative to XHR), and can be used as an inter-process/inter-worker protocol. HTTP is almost certainly the most commonly used communication protocol, and the RESTful power of HTTP can easily be leveraged for worker communication with JSGI’s JSON-based format (that doesn’t require HTTP parsing).

Projects

CommonJS and JSGI are purely API specifications, not implementations. Lets look at some projects that implement these specifications.

Narwhal/Jack

Probably the most advanced and mature CommonJS implementation is Narwhal plus Jack. Basically all of the CommonJS APIs are implemented in Narwhal/Jack and/or their forks. Narwhal provides extensive I/O support, a large collection of useful library modules, and a great package management system called tusk with package resource handling built into its module loader. Narwhal is designed for running on various JavaScript platforms and has an intelligently architected layer of abstractions for dealing with different engines.

Narwhal has the most complete engine support for Rhino. Rhino is almost certainly the most popular JavaScript engine for server side JavaScript due to its seamless integration with Java. With Rhino, the virtually limitless expanse of Java libraries are available and easily accessible. While great efforts are being made to important libraries in other engines, the breadth of libraries that are available through the Rhino’s Java bridge is not likely to be matched anytime soon. It also comes with a good visual debugger. Rhino is also perhaps the most feature-rich JavaScript implementation available, as it implements most of the ECMAScript 5 specification and Mozilla’s JavaScript 1.8 features.

Narwhal also runs on JSCore, the JavaScript engine used by Safari. JSCore is a very high-performance engine (second only to V8, perhaps), and consequently there is great potential for high-performance applications running on Narwhal/JSCore.

Jack is a JSGI implementation that has been developed to run on Narwhal, and is also built to run on various web server technologies. Jack has been somewhat of the de facto reference implementation for JSGI, and runs on Simple, CGI, servlet containers (including Jetty and GAE), and others. Older versions of Jack relied on shared memory threading, but with the latest work on Jack, requests are handled with full event loop based shared-nothing concurrency, with request delegation across multiple workers for efficient multi-core utilization.

Node

Node is an exciting project in the SSJS realm, and has received a lot of publicity recently, for good reason. Node builds on the amazingly fast V8 engine and provides important low-level functionality. V8 is one of the fastest dynamic language engines on the planet. Node uses libev for an event-loop based asynchronous I/O model, providing the same type of event-loop shared-nothing concurrency used in Narwhal. However, Node has had a stronger emphasis on trying to provide asynchronous APIs for all I/O operations. By exposing almost all I/O operations as asynchronous events, Node rarely has to block while waiting for I/O, and is able to execute much more efficiently than spreading workloads across multiple blocking threads. The performance advantages to event-loops over threads are well-documented. This focus, in combination with the blazing fast V8 engine, promises to place Node as the performance king of dynamic language based application servers.

There are certainly gaps that need to be filled with Node. Node implements CommonJS’s module system, but does not implement other CommonJS APIs, and has been slow to add better support for interoperability. Node runs its HTTP server and handling on a single process/thread, which means it can’t utilize multi-cores yet (support for that is coming). Node does not support the JSGI specification; however, it is possible to run JSGI on Node with the JSGI-Node adapter that I built.

There are other important things like package support and module reloading that are definitely needed. From my experience, Node does not feel as complete as Narwhal. However, despite these concerns, Node has a lot of potential, and is an exciting project. Node primarily fulfills a role as a low-level JavaScript platform, on which higher level functionality can be built. The Node community is responsive. There is an effort to run Narwhal on top of Node, which would be a powerful combination.

Ejscript

Ejscript is a JavaScript platform that extends the JavaScript language with many new constructs. Many of these additions are based on the ECMAScript 4 specification that was eventually abandoned because many implementors felt it was too extensive of a revision of the language. However, many of the features included in ECMAScript 4 that Ejscript implements were technically excellent additions, and may be included in ECMAScript 6. Beyond extra language features, Ejscript boasts a very high performance engine with remarkably low memory footprints, claiming performance that rivals Node with less than a tenth of the memory consumption.

Persevere

Persevere was one of the first projects to implement the CommonJS module specification. However, the primary goal of the Persevere project is to provide consistent end-to-end data semantics leveraging RESTful architecture. Persevere 1.0 provided CommonJS module support to provide interoperability with other CommonJS libraries, but being a JavaScript platform is beyond the primary scope of the project. With the new array of CommonJS platforms, future Persevere development will now focus on building on top of these new platforms and providing its data persistence, security, REST architecture, Comet notifications, and advanced web communication capabilities for multiple platforms.

Others

Mozilla’s SpiderMonkey engine is also a capable engine for server side JavaScript. Flusspferd is a project to add low-level system functionality to SpiderMonkey and couples with Zest as a JSGI server and Juice as a web framework on top.

There are number of other important server-side projects including HelmaNG, with high-quality technology that predates CommonJS and now are adapting to work in harmony with the CommonJS ecosystem, but I won’t cover all of them.

Web Frameworks

Some exciting new web frameworks are building on these technologies such as Nitro, Bogart, and JuiceJS. Also some platforms come with their own MVC framework such as HelmaNG, and EJScript. However, for many the move to JavaScript has been motivated by investment in rich client interfaces that already have presentation capabilities (the V and C in MVC), so a more modern REST-style, “thin server architecture” may be a better fit than traditional server-side MVC frameworks for many next generation JavaScript-based web applications. Pintura is the core of the next generation of Persevere, a cross-platform REST-style architecture web framework that is designed specifically for Ajax-style applications that maintain presentation logic on the client separate from the server-side storage concerns. We will cover Pintura in the next article of this series.

Conclusion

The combination of improvements and maturation of ECMAScript standards, with the new line of astonishingly fast JavaScript, the Ajax driven interest in JavaScript, the collaborative efforts of CommonJS standards, and the emerging array of server-side JavaScript projects are all coming together to propel server-side JavaScript to the forefront of web application development. It is not unreasonable to expect that the CommonJS APIs implemented with a web stack of the new line of SSJS projects could be the most important new web platform of the next decade.