Test early, test often, and test some more. Why put our heart and soul into our web applications only to be let down because we are not completely testing them. Let’s explore how the different web frameworks approach testing.

Testing is so essential to the success of any project that we have dedicated an entire post to this topic. We want to explore several questions in this post. Does our web framework hold itself to high testing standards? Does it do more than just play lip service to testing? Is there any secret sauce that our framework can add to the mix to make our testing better or more efficient?

This is part of a series on web frameworks. If you’ve missed our previous posts, you may want to start with If we chose our JavaScript framework like we chose our music…

Internal testing approach

One of the biggest insights we can gain about the quality of a web framework is, how does it approach testing its own source code. How does it ensure what we are using from it is well tested? Their approach to tests often gives insight into how we might test our code as well. We also want to look at how transparent and easy it is to see the output of that testing, to help give us a level of confidence that we are building on good foundations.

How does the framework recommend testing our source code? How much focus does that framework have on helping us ensure the code we write is tested? Some frameworks have strong opinions while others leave it up to the end developers. Where does each framework fit on this spectrum?

Testing considerations

Especially when it comes to testing web applications, the days of sitting a tester in front of a computer and interacting with an application have gone the way of the dodo. While this sort of exploratory testing still has its place, the ability to test has gone well beyond the early days of manual testing. Not only can we automate our unit and end-to-end testing, we can do many other types of testing, including items like accessibility, visual regression, and performance. We need to see if the frameworks provide any specific considerations for testing and how the framework might go beyond the basics.

Jump to:

  • logo
  • logo
  • logo
  • logo
  • logo
  • logo

Angular 2+

Internal testing approach

Angular 2+ uses Karma as its test runner and Jasmine as a Behavior Driven Development (BDD) framework for authoring its tests. Angular 2+ utilizes Travis CI and CircleCI to run the tests against two different on-demand testing grid providers, BrowserStack and SauceLabs, against a spectrum of browsers and devices.

For some of their testing, Angular 2+ use Protractor, an end-to-end test framework for Angular 2+ applications.

There is no apparent code coverage information provided out of the box.

Angular 2+ documents their recommended testing approach. They have four key recommended parts:

  • Karma as the recommended test runner
  • Jasmine as the recommended Behavior Driven Development approach to author unit tests
  • Protractor to provide the end-to-end testing
  • Angular testing utilities, a set of APIs that make it easier to test Angular applications

Testing considerations

Angular 2+ recommends two different styles of unit tests. There are the isolated unit tests (e.g. testing each class or module on its own). This is recommended for pipes and services. For components, where two-way data binding can make it more difficult to isolate the unit, the recommendation is to use the Angular testing utilities to allow integration tests to be created where the component under test is allowed to interact with other components as it would within the actual application.

The command line tool has a test command (ng test) which will run the tests via Karma as configured in the project.

Angular 2+ provides a benchmarking suite, Benchpress, which has some limited documentation on how to benchmark and profile your applications.

Code coverage and accessibility testing are not directly covered by the Angular project, but there are references available on how to achieve this via third-party libraries on the web.

React + Redux

Internal testing approach

React is tested internally using Jest, Facebook’s official JavaScript test runner, along with React’s own Test Utils package. Jest provides out of the box code coverage information and the project’s core package posts its coverage information to Coveralls. The main package currently maintains a 90% test coverage. Continuous integration/build is provided by CircleCI.

Redux also uses Jest, but uses Travis CI for its continuous integration/build processing. It does not appear to publish its code coverage information.

React and Redux both suggest authoring tests with Jest. Since testing React applications can be difficult given the intricacies of testing and asserting properties of virtual DOM instead of a traditional, tangible DOM. To make for a better developer experience and to alleviate many common testing pains encountered when asserting deeply-rendered components, React offers an official set of testing utilities. React also explicitly points to Enzyme, an Airbnb-developed abstraction that wraps React’s test-utils package with additional functionality and more advanced assertion APIs.

Because most of the Redux part of an application should be pure reducer functions, they are often testable without any need for mocking inputs. There is an official guide to writing tests for Redux based applications.

Testing considerations

As mentioned in previous sections, React acknowledges that their Test Utilities package is limited in scope, and points to community-driven initiatives like Enzyme to provide a more robust testing API. React makes no suggestions for accessibility, visual, or performance testing tools, though it should be mentioned that the React Developer Tools can greatly facilitate the diagnosis of rendering performance issues.

Because Jest supports code coverage analysis out of the box, it is possible to integrate a solution to manage the code coverage in line with the way that code coverage is integrated into the main React package.

Jest focuses on testing under Node.js, including the browser functionality. Jest does not come with out of the box integrations to make it easy to test in actual browsers. Without testing in browsers, many projects would lull themselves into a sense of overconfidence that their solutions would work in all target browsers.

Vue.js

Internal testing approach

Vue.js uses Karma as a test runner and Jasmine for authoring BDD tests. Vue.js uses Nightwatch for their end-to-end functional tests, testing against Chrome and PhantomJS. They utilize CircleCI for continuous integration and Codecov.io for code coverage analysis, which currently sits at 100% coverage for the main Vue.js package.

Vue’s self-tests are configured to be able to be run on SauceLabs though it does not appear that is a standard part of their continuous integration approach.

The Vue.js documentation points out that any unit testing framework that supports modular code will work. The Vue.js project does not recommend any particular framework, but they suggest using the Karma test runner because of its many plugins, particularly its support for bundling tools like Webpack and Browserify. The example tests in the docs use Jasmine.

Testing considerations

The Vue docs do not mention anything beyond simple unit testing. They do not recommend nor supply any tools for making functional or end-to-end testing easier. Their own test suite appears to only consider performance benchmarking for the server-side rendering features and that animations transitions complete in an expected time frame. Vue.js does not provide out of the box considerations for visual regression or accessibility testing.

Dojo 2

Internal testing approach

Dojo 2 uses Intern as its test runner and for all of its internal unit and functional (end-to-end) testing. Dojo 2 uses Travis CI for continuous integration and have configuration options for both running tests against BrowserStack and SauceLabs where some of the tested browsers vary by package, but generally include IE 11, Edge, Chrome, Firefox, Safari, and iOS. The project posts their code coverage information to Codecov.io for analysis. The average across packages appears to be about 90% code coverage. Packages that need to ultimately interact with the DOM run their tests in Node.js using jsdom to provide browser-like APIs.

For command-line packages, Dojo 2 also leverages Appveyor for running continuous integration and testing on Windows 10 platforms. This helps find subtle Windows-specific issues in projects where most developers use macOS or Linux.

Some packages use the dedicated testing utilities of @dojo/test-extras which provides tools to make testing of Dojo 2 and its virtual DOM based widgets easier.

The standard Dojo CLI command @dojo/cli-test-intern integrates Intern into testing Dojo 2 projects and is recommended by the project for testing. Projects scaffolded with the @dojo/cli-create-app will automatically configure a directory for both unit and functional test cases. The CLI command provides authored (TypeScript) code coverage information out of the box as well as configuration to run the tests with Intern locally in a Chrome browser or under Node.js.

The CLI command also supports easy integration with BrowserStack and SauceLabs for running tests against a range of browsers.

As mentioned above, there is a package of testing utilities which can also be used by end developers called @dojo/test-extras. It provides automatic configuration of jsdom under a Node.js environment to allow full testing of widgets as well as utilities for interacting with widgets without having to perform extensive functional tests.

Testing considerations

Dojo 2 provides a solid approach to automated testing. Intern supports accessibility testing, visual regression testing, and performance testing. It is likely that these features will be incorporated into the out of the box testing experience as well as additional advice and direction are given to end developers on how to leverage these types of tests.

Ember

Internal testing approach

Ember.js uses QUnit as its testing framework, which provides a test runner, unit testing, and functional testing. They use Travis-CI for continuous integration. They use SauceLabs to run their tests under IE 9, 10, 11, Edge, and Safari. They utilize Code Climate for code quality analysis. For testing under Node.js, Ember.js utilizes PhantomJS for headless testing.

Using the Ember CLI, testing defaults to QUnit. The project provides extensive guidelines on what type of testing to do, and how to accomplish it. They provide advice on how to test each part of an Ember application.

Testing considerations

The official accessibility project for Ember provides a set of a11y tests to make it easier to test the accessibility of your Ember.js applications. While they don’t have built in performance testing, rendering performance is a consideration of their Ember Inspector browser add-on.

Aurelia

Internal testing approach

Aurelia uses Karma as its test runner and Jasmine as a Behavior Driven Development (BDD) framework for authoring its tests. It uses CircleCI for its continuous integration. It appears to test code against Chrome as part of the CI process, and appears to collect coverage information, but that is not made easily accessible (using a code quality reporting service).

Aurelia UX seems to have, at the time of this writing, only recently added tests to the package and that testing does not yet appear to be integrated into the CI for the package.

Aurelia officially is agnostic when it comes to a recommended framework, though its CLI assumes Karma and Jasmine as well as its testing tools package. Aurelia provides specific guidance on testing components and end-to-end testing where they recommend using Protractor.

Testing considerations

While Aurelia provides a benchmarking library, there is no documentation on how to integrate performance testing into your test strategy. Also, as mentioned above, there is a dedicated package to make it easier to test components of an Aurelia application.

Summary

Angular 2+

Angular 2+ has a robust level of testing on their own code and have extensive documentation and guidelines on how to test applications built with Angular 2+. Self-testing and promotion of good test practices seem to be a focus of Angular 2+. The complexity of two-way data binding does make it more difficult to perform isolated unit testing, but Angular 2+ attempts to provide tooling to make that easier to accomplish. Angular 2+ has a strong focus on end-to-end application testing.

React + Redux

Because of the nature of React and Redux, it is likely that other libraries will be included in your project which may make setting up a robust testing strategy slightly more challenging. The challenge of testing has given rise to the community developed solutions like Airbnb’s Enzyme. This is great, but none of the official solutions actually test React and Redux in real browsers and instead is heavily focused on unit tests without a good solution for end-to-end testing.

Vue.js

Vue.js as a project is very well tested and they seem to have a good testing ethos within the project. While Vue.js mentions testing in their documentation, they leave most of it up to the end developer to deal with and do not really provide much help, advice, or additional tooling.

Dojo 2

Dojo 2 has a strong focus on testing, both its own code, and in enabling end developers to be able to test their code efficiently and completely. There is a high level of assistance with the tooling.

Ember.js

As you might have come to expect from Ember.js, just as they have strong opinions on how to build web applications, they also have strong opinions on how to test applications. Ember provides out of the box scaffolding of tests along with code, as well as strong guidance on what sort of tests should be performed on an application. They also have a strong testing ethos within the project as a whole.

Aurelia

For a framework that indicates code quality as a major benefit of the solution, the execution on that promise could be dramatically improved. In particular, shipping a UX framework without tests from when it was announced in November 2016 until August 2017 seems counterproductive to claiming a focus on testing.

For end developers, Aurelia gives a reasonable amount of guidance on how to efficiently perform testing and use tooling.

Up next

One of the biggest challenges building web applications is helping ensure that the code you author actually behaves properly at run-time. Over the past few years, there have been several strategies to accomplish this and to make JavaScript more predictable. In the next post, we will take a look at how the different frameworks try to accomplish that goal and how they can help you as a downstream developer build solid applications from the start.