Setting Up an End-to-End Testing Workflow with Gulp, Mocha, and WebdriverIO

Introduction

Manual testing is usually slow, tedious and error-prone, which is why end-to-end testing is important — we need a way to automate testing across different browsers and platforms.

In this tutorial, we will learn how to set up end-to-end testing with Selenium and write a simple test checking if the page title of our HTML page is what we expected. We will discuss writing real application tests in a future article.

Prerequisites

For this tutorial you will need to have Node.js installed, v4 or higher.

We’ll use Mocha because it’s a popular choice for unit testing. If you’re not familiar with Mocha, you can read this tutorial.

Directory Structure

This is what the directory structure of our project will look like:

e2e-testing
├── gulpfile.js
├── package.json
└── test
    ├── fixtures
    │   └── index.html
    └── specs
        └── page-title.js

Fixtures

To get an idea of what we’re doing, let’s first create a simple fixture at test/fixtures/index.html:

<!-- test/fixtures/index.html -->
<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <title>End-to-End Testing</title>
  </head>
  <body>
    <h1>Hello World</h1>
  </body>
</html>

We will run our tests on this HTML page. More specifically, we’re going to check and confirm if the page title is End-to-End Testing.

Let’s start building our workflow.

Installing Node.js

If you’re not familiar with npm, you can read our Node.js Package Manager tutorial.

It’s easy to install Node.js using nvm. If you’re on macOS and have Homebrew installed, you can install nvm by running:

brew install nvm

Otherwise, run:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash

With nvm, we can have as many versions of Node.js installed as we want, and switch between them using nvm use. To install Node v6, run:

nvm install 6

This will automatically install npm as well. We can set v6 as the default version by running:

nvm alias default 6

To test if Node.js and npm are installed, run:

node -v && npm -v

This should output their version numbers.

Now let’s create an empty package.json file. Because we’re not going to publish this module, we’ll make it private:

{
  "private": true
}

Gulp

If you’re new to Gulp, you can learn the basics in this tutorial.

Let’s install it, the library locally and the CLI globally:

npm install --save-dev gulp
npm install --global gulp-cli

The CLI provides us with the gulp command, which allows us to call Gulp tasks. Now we create a file called gulpfile.js and define the test task:

// gulpfile.js
const gulp = require('gulp');

gulp.task('test', () => {
  console.log('It works!');
});

To check if the setup is working correctly, run:

gulp test

As we can see, the arguments to the gulp command are names of tasks to run. In the generated output we should see the following:

[14:44:19] Using gulpfile ~/Code/test/e2e-testing/gulpfile.js
[14:44:19] Starting 'test'...
It works!
[14:44:19] Finished 'test' after 178 μs

HTTP Server

We are going to run tests against real HTML pages, so for testing locally we need to start an HTTP server. There are many ways to do this, but in this case we just need something fast and simple, so we’re going to use connect and serve-static:

npm install --save-dev connect serve-static

In case you need something more advanced and don’t feel like installing a lot of Express plugins, you can check out BrowserSync.

Next, we’ll add the server task to the gulpfile:

// gulpfile.js
const gulp = require('gulp');
const http = require('http');
const connect = require('connect');
const serveStatic = require('serve-static');

gulp.task('http', (done) => {
  const app = connect().use(serveStatic('test/fixtures'));
  http.createServer(app).listen(9000, done);
});

the URL will be localhost:9000
it will be serving the test/fixtures directory
done will be called when the server starts — this tells Gulp when the task is complete

To make sure the task works, we can run:

gulp http

and navigate to localhost:9000. We should see our fixture, i.e. a heading that says “Hello World”. Now that we are sure the task works, we can close the Gulp process.

WebdriverIO

WebdriverIO is a collection of Node.js bindings for Selenium. Let’s install it in order to run the task runner and create a configuration file:

npm install --global webdriverio

WebdriverIO comes with the wdio executable that’s available in node_modules/.bin. We could install WebdriverIO globally, but we’re going to use the executable only once, to generate the initial WebdriverIO configuration, so let’s run it from node_modules:

./node_modules/.bin/wdio config

Leave all options to their defaults, except the following:

select the “spec” reporter in order to get a more verbose output
select selenium-standalone and phantomjs services
set the base url to http://localhost:9000, that’s the port our server is listening on

Our generated wdio.conf.js file should look similar to this:

// wdio.conf.js
exports.config = {

    //
    // ==================
    // Specify Test Files
    // ==================
    // Define which test specs should run. The pattern is relative to the directory
    // from which wdio was called. Notice that, if you are calling wdio from an
    // NPM script (see https://docs.npmjs.com/cli/run-script) then the current working
    // directory is where your package.json resides, so wdio will be called from there.
    //
    specs: [
        './test/specs/**/*.js'
    ],
    // Patterns to exclude.
    exclude: [
        // 'path/to/excluded/files'
    ],
    //
    // ============
    // Capabilities
    // ============
    // Define your capabilities here. WebdriverIO can run multiple capabilities at the same
    // time. Depending on the number of capabilities, WebdriverIO launches several test
    // sessions. Within your capabilities you can overwrite the spec and exclude options in
    // order to group specific specs to a specific capability.
    //
    // First, you can define how many instances should be started at the same time. Let's
    // say you have 3 different capabilities (Chrome, Firefox, and Safari) and you have
    // set maxInstances to 1; wdio will spawn 3 processes. Therefore, if you have 10 spec
    // files and you set maxInstances to 10, all spec files will get tested at the same time
    // and 30 processes will get spawned. The property handles how many capabilities
    // from the same test should run tests.
    //
    maxInstances: 10,
    //
    // If you have trouble getting all important capabilities together, check out the
    // Sauce Labs platform configurator - a great tool to configure your capabilities:
    // https://docs.saucelabs.com/reference/platforms-configurator
    //
    capabilities: [{
        // maxInstances can get overwritten per capability. So if you have an in-house Selenium
        // grid with only 5 firefox instances available you can make sure that not more than
        // 5 instances get started at a time.
        maxInstances: 5,
        //
        browserName: 'firefox'
    }],
    //
    // ===================
    // Test Configurations
    // ===================
    // Define all options that are relevant for the WebdriverIO instance here
    //
    // By default WebdriverIO commands are executed in a synchronous way using
    // the wdio-sync package. If you still want to run your tests in an async way
    // e.g. using promises you can set the sync option to false.
    sync: true,
    //
    // Level of logging verbosity: silent | verbose | command | data | result | error
    logLevel: 'silent',
    //
    // Enables colors for log output.
    coloredLogs: true,
    //
    // Saves a screenshot to a given path if a command fails.
    screenshotPath: './errorShots/',
    //
    // Set a base URL in order to shorten url command calls. If your url parameter starts
    // with "/", then the base url gets prepended.
    baseUrl: 'http://localhost:9000',
    //
    // Default timeout for all waitFor* commands.
    waitforTimeout: 10000,
    //
    // Default timeout in milliseconds for request
    // if Selenium Grid doesn't send response
    connectionRetryTimeout: 90000,
    //
    // Default request retries count
    connectionRetryCount: 3,
    //
    // Initialize the browser instance with a WebdriverIO plugin. The object should have the
    // plugin name as key and the desired plugin options as properties. Make sure you have
    // the plugin installed before running any tests. The following plugins are currently
    // available:
    // WebdriverCSS: https://github.com/webdriverio/webdrivercss
    // WebdriverRTC: https://github.com/webdriverio/webdriverrtc
    // Browserevent: https://github.com/webdriverio/browserevent
    // plugins: {
    //     webdrivercss: {
    //         screenshotRoot: 'my-shots',
    //         failedComparisonsRoot: 'diffs',
    //         misMatchTolerance: 0.05,
    //         screenWidth: [320,480,640,1024]
    //     },
    //     webdriverrtc: {},
    //     browserevent: {}
    // },
    //
    // Test runner services
    // Services take over a specific job you don't want to take care of. They enhance
    // your test setup with almost no effort. Unlike plugins, they don't add new
    // commands. Instead, they hook themselves up into the test process.
    services: ['selenium-standalone','phantomjs'],
    //
    // Framework you want to run your specs with.
    // The following are supported: Mocha, Jasmine, and Cucumber
    // see also: http://webdriver.io/guide/testrunner/frameworks.html
    //
    // Make sure you have the wdio adapter package for the specific framework installed
    // before running any tests.
    framework: 'mocha',
    //
    // Test reporter for stdout.
    // The only one supported by default is 'dot'
    // see also: http://webdriver.io/guide/testrunner/reporters.html
    reporters: ['spec'],

    //
    // Options to be passed to Mocha.
    // See the full list at http://mochajs.org/
    mochaOpts: {
        ui: 'bdd'
    },
    //
    // =====
    // Hooks
    // =====
    // WebdriverIO provides several hooks you can use to interfere with the test process in order to enhance
    // it and to build services around it. You can either apply a single function or an array of
    // methods to it. If one of them returns with a promise, WebdriverIO will wait until that promise got
    // resolved to continue.
    //
    // Gets executed once before all workers get launched.
    // onPrepare: function (config, capabilities) {
    // },
    //
    // Gets executed before test execution begins. At this point you can access all global
    // variables, such as browser. It is the perfect place to define custom commands.
    // before: function (capabilities, specs) {
    // },
    //
    // Hook that gets executed before the suite starts
    // beforeSuite: function (suite) {
    // },
    //
    // Hook that gets executed _before_ a hook within the suite starts (e.g. runs before calling
    // beforeEach in Mocha)
    // beforeHook: function () {
    // },
    //
    // Hook that gets executed _after_ a hook within the suite starts (e.g. runs after calling
    // afterEach in Mocha)
    // afterHook: function () {
    // },
    //
    // Function to be executed before a test (in Mocha/Jasmine) or a step (in Cucumber) starts.
    // beforeTest: function (test) {
    // },
    //
    // Runs before a WebdriverIO command gets executed.
    // beforeCommand: function (commandName, args) {
    // },
    //
    // Runs after a WebdriverIO command gets executed
    // afterCommand: function (commandName, args, result, error) {
    // },
    //
    // Function to be executed after a test (in Mocha/Jasmine) or a step (in Cucumber) starts.
    // afterTest: function (test) {
    // },
    //
    // Hook that gets executed after the suite has ended
    // afterSuite: function (suite) {
    // },
    //
    // Gets executed after all tests are done. You still have access to all global variables from
    // the test.
    // after: function (result, capabilities, specs) {
    // },
    //
    // Gets executed after all workers got shut down and the process is about to exit. It is not
    // possible to defer the end of the process using a promise.
    // onComplete: function(exitCode) {
    // }
}

For this tutorial we’ll change browserName in the capabilities from firefox to phantomjs, this will tell WebdriverIO to run tests in PhantomJS, which is faster.

Now we can add the end-to-end testing task, specifying http as the dependency and pointing the launcher to our configuration file:

// gulpfile.js
const gulp = require('gulp');
const http = require('http');
const connect = require('connect');
const serveStatic = require('serve-static');
const Launcher = require('webdriverio/build/lib/launcher');
const path = require('path');
const wdio = new Launcher(path.join(__dirname, 'wdio.conf.js'));

gulp.task('http', done => {
  const app = connect().use(serveStatic('test/fixtures'));
  http.createServer(app).listen(9000, done);
});

gulp.task('e2e', ['http'], () => {
  return wdio.run(code => {
    process.exit(code);
  }, error => {
    console.error('Launcher failed to start the test', error.stacktrace);
    process.exit(1);
  });
});

Here we’re using WebdriverIO’s test runner programmatically. We are returning the promise to Gulp, which means that once it resolves or rejects, the task will be finished.

When the e2e task finishes, we need to close the HTTP server. We can create a main task for that called test, which will run e2e and any other testing tasks you might want to add. We will store a reference to the HTTP server and use it in the test task to close it:

const gulp = require('gulp');
const http = require('http');
const connect = require('connect');
const serveStatic = require('serve-static');
const Launcher = require('webdriverio/build/lib/launcher');
const path = require('path');
const wdio = new Launcher(path.join(__dirname, 'wdio.conf.js'));

let httpServer;

gulp.task('http', done => {
  const app = connect().use(serveStatic('test/fixtures'));
  httpServer = http.createServer(app).listen(9000, done);
});

gulp.task('e2e', ['http'], () => {
  return wdio.run(code => {
    process.exit(code);
  }, error => {
    console.error('Launcher failed to start the test', error.stacktrace);
    process.exit(1);
  });
});

gulp.task('test', ['e2e'], () => {
  httpServer.close();
});

Before running our tests, we have to actually write some!

Writing Tests

Now, we are ready to start writing tests. Let’s create a file named test/specs/page-title.js which will check if the page title of our fixture is correct:

import assert from 'assert';

describe('fixture', () => {
  it('has the expected page title', () => {
    browser.url('/');
    assert.equal(browser.getTitle(), 'End-to-End Testing');
  });
});

we’ve navigated to /, which is actually http://localhost:9000/ because of the baseUrl option in our wdio.conf.js
we called the getTitle method which returnes a promise
using the built-in assert module we’ve checked whether the page title is correct

If you are wondering why can we write the tests synchronously, that’s because WebdriverIO uses wdio-sync by default. If you’d like to use promises instead, set sync to false in wdio.conf.js.

Now we can run our test:

gulp test

We should see something similar to the following output:

[14:13:23] Using gulpfile ~/Code/test/e2e-testing/gulpfile.js
[14:13:23] Starting 'http'...
[14:13:23] Finished 'http' after 5.15 ms
[14:13:23] Starting 'e2e'...
------------------------------------------------------------------
[phantomjs #0-0] Session ID: a169d6f0-a810-11e6-a380-87b5b0ceaf33
[phantomjs #0-0] Spec: /Users/matija/Code/test/e2e-testing/test/specs/page-title.js
[phantomjs #0-0] Running: phantomjs
[phantomjs #0-0]
[phantomjs #0-0]   fixture
[phantomjs #0-0]       ✓ has the expected page title
[phantomjs #0-0]
[phantomjs #0-0]
[phantomjs #0-0] 1 passing (1s)
[phantomjs #0-0]

[14:13:26] Finished 'e2e' after 3.25 s
[14:13:26] Starting 'test'...
[14:13:26] Finished 'test' after 323 μs

Yay, our test is passing!

The Test Command

Currently, our testing command is gulp test. In a Node.js workflow it’s generally a good practice to make the npm test command run the tests, since:

this gives you and others a single command to remember, regardless of your toolchain
CI services default to running npm test in a Node.js environment
scripts use local executables from node_modules, which means that we don’t have to have Gulp globally installed to run the tests

We can make the npm test map to gulp test in our package.json using scripts:

{
  "private": true,
  "scripts": {
    "test": "gulp test"
  },
  "devDependencies": {
    // ...
  }
}

To make sure this works, we can run:

npm test

We should see the same output as the one generated by the gulp test command.

Troubleshooting

WebdriverIO runs the Selenium Standalone server in the background, but it’s a tricky piece of software that can sometimes continue running in the background after WebdriverIO is done, which prevents us from starting another one. We can shut it down by killing that process:

pkill -f selenium-standalone

Conclusion

This is just an example of a workflow and, depending on the stack, your workflow could be very different. In any case, it is important to automate as much as we can in our workflow, because it saves us precious time. Even if we completely change our underlying stack, we should still be able to run all our tests with a single command.