Thanks to a generous Mozilla Open Source Support program award, Intern has expanded its testing ecosystem to include new and robust sets of tools. Engineers can now easily write tests to benchmark sections of code, test for accessibility (a11y) support, and test for visual differences. Today we’ll show you how to use Intern’s Visual Regression plugin to test for visual changes using a simple web page.

Visual Regression testing

Visual changes are notoriously difficult to test. Even the best functional test suite cannot tell you if a page looks correct. Instead, Visual Regression testing takes a screenshot of a web page (or state within a webapp) and compares it to a previously taken screenshot (referred to as a baseline) to see if the contents have changed, creating a report immediately revealing what is different. This helps engineers to:

  • know when a change to CSS results in undesired visual changes elsewhere in the application
  • ensure a page is rendered consistently across browsers
  • put a quick set of tests around a legacy system to identify regressions

Visual Regression testing is provided via a plugin to Intern. To get started, first use npm install intern-visual to install the plugin. Second, configure the plugin as a package with your module loader.

loaderOptions: {
	packages: [
		{ name: 'src', location: `src` },
		{ name: 'tests', location: `tests` }
		{ name: 'visual', location: `node_modules/intern-visual` }
	]
};

Creating a Visual Regression test

Visual Regression tests are written for the WebDriver engine (the one typically used with functional tests) and can be written programmatically or via configuration.

Configuration-driven tests are ideal for situations where the goal is simply to verify that a web page hasn’t visually changed. These tests are very simple to author and are a great way to quickly create tests.

import visualTest from 'visual/test';

registerSuite({
    test: visualTest({
        url: 'https://www.sitepen.com',
        width: 1024,
        height: 768,
        missingBaseline: 'snapshot',
    });
});

The above example instructs intern-visual to create a test that visits the provided url, resize the browser to the provided width and height value, and create a baseline image snapshot if one does not already exist with the missingBaseline property.

However, there are times when we need more control over the page, the application we are testing, or when additional steps need to be taken during a test. These additional steps could include inputting and responding to text prompts, hiding parts of the DOM that change (e.g. ads), or interacting with the page. In these cases, the programmatic API provides a command helper that expects a screenshot as a parameter.

import assertVisuals from 'visual/assert';

registerSuite({
    test() {
        this.remote()
            .get('https://www.sitepen.com')
            .setWindowSize(1024, 768)  // set the window size
            .takeScreenshot()
            .then(assertVisuals(this, {
                missingBaseline: 'snapshot'
            }));
    }
});

In the above example, assertVisuals is a command helper that accepts a reference to the current test and various options, returning a function that expects a screenshot from the takeScreenshot command.

The programmatic code shown above does virtually the same thing as the configuration-based visual test, except the WebDriver commands are defined explicitly. Both of the above tests search for a baseline image on the local machine, generate one if necessary, and provide a pass/fail to Intern along with some additional information about the visual comparison.

The VisualRegression Reporter

Reporters are Intern’s mechanism for providing test results from a variety of sources in a variety of formats. The intern-visual test and assert modules are part of the test and assertion layer of Intern, providing a pass or a fail notification. To get a visual HTML report, simply add the VisualRegression report to Intern’s list of reporters.

reporters: [
    'Runner',
    'intern-visual/reporters/VisualRegression'
]

The VisualRegression report will write an HTML report showing any visual regressions in the visual-test directory. It will not output any failures to the console, so be sure to include your preferred console reporter if you would like to see pass/fail information in the console. When a visual regression test fails, it will appear in the report with the differences overlaid in red, as shown in the below example.

intern-visual-failing-report

Going further

We plan to continue to add features to the visual regression plugin to make it easier to use and better at distinguishing visual differences. The most important of these is to create good documentation around the project and fully detail the various configuration options. We also want to continue to improve the configuration-based visual testing to make it easier and faster to write good visual tests using less code. Finally, the current image comparison algorithms are best for comparing two identically sized images due to how image differences are identified. We want to improve these algorithms to be more tolerant of minor differences caused by anti-aliasing allowing for minor differences between browsers or between different window sizes, ignoring all but relevant visual differences.

Visual regression tests provide a quick and easy way to establish baselines for how your application should appear. They are especially valuable when you need to put tests around a legacy application or to ensure that a CSS change has not caused undesired changes. For more information, take a look at intern-visual on Github. Visual regression test examples are available under the tests/visual directory.

Be heard

We want to make sure Intern is the very best JavaScript testing ecosystem. To do this, we need feedback from engineers at all levels and experience. If you have something to say about difficulties with Intern or other testing frameworks, let us know on Stack Overflow or make a recommendation for a feature using Intern’s issue tracker. Building a community and building solutions go hand in hand. We look forward to hearing from you!