I have been working on first-draft, an opensource (currently) canvas-based collaborative online drawing/annotation tool for a while now. I have been meaning to add UI testing to it, but I could never find something that would allow me to test in multiple browsers, do complex interactions and work with the web socket communications first-draft uses.

I recently decided to look at UI testing libraries again and while I was looking back over WebdriverIO, a node-based testing framework using the WebDriver API, I managed to come across the function performActions. "The Perform Actions command is used to execute complex user actions." This had been the thing I had been looking for for months and never found it!

I looked into the spec to see what the "actions" were and they were exactly what I needed - the ability to do complex pointer (mouse and touch) and keyboard interactions. The spec includes an example of pinch and zoom interaction

[
  {
    "type": "pointer",
    "id": "finger1",
    "parameters": {"pointerType": "touch"},
    "actions": [
      {"type": "pointerMove", "duration": 0, "x": 100, "y": 100},
      {"type": "pointerDown", "button": 0},
      {"type": "pause", "duration": 500},
      {"type": "pointerMove", "duration": 1000, "origin": "pointer", "x": -50, "y": 0},
      {"type": "pointerUp", "button": 0}
    ]
  }, {
    "type": "pointer",
    "id": "finger2",
    "parameters": {"pointerType": "touch"},
    "actions": [
      {"type": "pointerMove", "duration": 0, "x": 100, "y": 100},
      {"type": "pointerDown", "button": 0},
      {"type": "pause", "duration": 500},
      {"type": "pointerMove", "duration": 1000, "origin": "pointer", "x": 50, "y": 0},
      {"type": "pointerUp", "button": 0}
    ]
  }
]

I was instantly in love. Wondering if this could be the test framework for me, I tried to find a way I could deal with the Web Socket communications and I came across browser.execute, which allows you send javascript to run in the browser.

The stars had aligned. I had everything I needed! Now I just needed to write some code.

Mocking the Web Socket Client

Once I have installed webdriverio using the getting started instructions and configured it to work with typescript files, I decided how I was going to run things.

With the ability to run javascript on the page under test, I decided to create a mock web socket client for the app to use when running tests. The mock allows the test to see all the events that have been emitted by the app, send events to the app and add handler functions to respond to events emitted by the app.

const socket = {
  sent: [],
  queue: [],
  mockers: {},
  addMocker: (event, handler) => {
    if (socket.mockers[event]) {
      if (socket.mockers[event].indexOf(handler) === -1) {
        socket.mockers[event].push(handler);
      }
    } else {
      socket.mockers[event] = [handler];
      let i = 0;
      while (i < socket.queue.length) {
        if (event === '*') {
          handler(socket.queue[i].event, socket.queue[i].data);
        } else if (socket.queue[i].event === event) {
          handler(socket.queue[i].data);
          socket.queue.splice(i, 1);
        } else {
          i++;
        }
      }
    }
  },
  removeMocker: (event, handler) => {
    if (socket.mockers[event]) {
      const index = socket.mockers[event].indexOf(handler);
      if (index !== -1) {
        socket.mockers[event].splice(index, 1);
      }
    }
  },
  handlers: {},
  on: (event, handler) => {
    if (socket.handlers[event]) {
      if (socket.handlers[event].indexOf(handler) === -1) {
        socket.handlers[event].push(handler);
      }
    } else {
      socket.handlers[event] = [handler];
    }
  },
  off: (event, handler) => {
    if (socket.handlers[event]) {
      const index = socket.handlers[event].indexOf(handler);
      if (index !== -1) {
        socket.handlers[event].splice(index, 1);
      }
    }
  },
  emit: (event, data) => {
    console.debug('%c<<< Emitted event', 'color: #860', event, data);
    socket.sent.push({ event, data });
    if (socket.mockers[event]?.length) {
      socket.mockers[event].map((handler) => handler(data));
    } else {
      socket.queue.push({ event, data });
    }
    (socket.mockers['*'] || []).map((handler) => handler(event, data));
  },
  emitMock: (event, data) => {
    console.debug('%c>>> Received event', 'color: #090', event, data);
    (socket.handlers[event] || []).map((handler) => handler(data));
  },
  clear: () => {
    socket.sent = [];
  }
};

Writing Tests

With the mock socket client set up, I set about writing my first test. I wanted to make sure the app was redirecting to a blank draft and loading the toolbar. When first-draft initially loads, it send a connect event to the server requesting the draft from the server, so I had to use my mock socket client to respond with a null value (no exist) response. I then checked that the toolbar was displayed.

import { blankDraft } from '../helpers/utils';

describe('first-draft', () => {
  before(async () => {
    // Go to first-draft
    await browser.url('/');
    // Attach function to return with a blank draft when requested
    await browser.execute(blankDraft);
  });

  it('redirect to a random draft', async () => {
    expect(browser).toHaveUrlContaining('/d/');
  });

  it('requests the draft', async () => {
    // Get the draftId from the url
    const draftId = (await browser.getUrl()).replace(/^.*\/d\/(.*)\//, '$1');
    const connectEvent = await browser.execute(() =>
      window['socketEvents'].sent.find(
        (event) => event.event === 'draft:connect'
      )
    );

    expect(connectEvent).toBeDefined();
    expect(connectEvent.data.name).toEqual(draftId);
    expect(connectEvent.data.getDraft).toEqual(true);
  });
});

I saved the test file and ran the test runner

$ yarn run test:ui
wdio wdio.conf.js
...
 "spec" Reporter:
------------------------------------------------------------------
[Chrome 84.0.4147.125 linux #0-0] Spec: first-draft/test/specs/redirect.ts
[Chrome 84.0.4147.125 linux #0-0] Running: Chrome (v84.0.4147.125) on linux
[Chrome 84.0.4147.125 linux #0-0] Session ID: 10f20b22-88dd-4297-965f-3b71302af99e
[Chrome 84.0.4147.125 linux #0-0]
[Chrome 84.0.4147.125 linux #0-0] first-draft
[Chrome 84.0.4147.125 linux #0-0]    ✓ redirect to a random draft
[Chrome 84.0.4147.125 linux #0-0]    ✓ requests the draft
[Chrome 84.0.4147.125 linux #0-0]
[Chrome 84.0.4147.125 linux #0-0] 2 passing (2s)


Spec Files:      1 passed, 1 total (100% completed) in 00:00:16

Excited by the green ticks, I decided to get a little more complex and started on a test to test the pen tool.

The test had to load a blank draft, select to pen tool and then click and drag to draw a line. Once the line was drawn, I then checked the events that had been emitted by the app to ensure they were as expected.

import { asyncFind, blankDraft, getCoordinate } from '../helpers/utils';

describe('first-draft pen tool', () => {
  it('becomes selected when clicked', async () => {
    await browser.url('/');
    await browser.execute(blankDraft);
    const toolbar = await $('#toolbar');
    expect(toolbar).toExist();
    const toolButton = await asyncFind(
      await toolbar.$$('button'),
      async (button) => (await button.getAttribute('title')) === 'Pen tool'
    );
    expect(toolButton).toExist();

    await toolButton.click();
    expect(toolButton).toHaveAttribute('aria-selected', 'true');
    const changeEvent = await browser.execute(() =>
      window['socketEvents'].sent.find(
        (event) => event.event === 'draft:changeTool'
      )
    );

    expect(changeEvent).toBeDefined();
    expect(changeEvent.data.tool).toEqual('pen');
  });

  it('draws a line with a mouse', async () => {
    await browser.execute(() => window['socketEvents'].clear());

    // Draw line
    const actions = [
      {
        type: 'pointer',
        id: 'mouse',
        parmeters: {
          pointerType: 'mouse'
        },
        actions: [
          { type: 'pointerMove', duration: 0, x: 20, y: 100 },
          { type: 'pointerDown', button: 0 },
          { type: 'pause', duration: 500 },
          {
            type: 'pointerMove',
            duration: 500,
            origin: 'pointer',
            x: 50,
            y: 50
          },
          { type: 'pause', duration: 500 },
          {
            type: 'pointerMove',
            duration: 500,
            origin: 'pointer',
            x: 50,
            y: 50
          },
          { type: 'pause', duration: 500 },
          { type: 'pointerUp', button: 0 },
          {
            type: 'pointerMove',
            duration: 500,
            origin: 'pointer',
            x: 50,
            y: 50
          },
          { type: 'pause', duration: 500 }
        ]
      }
    ];

    await browser.performActions(actions);

    const drawEvents = await browser.execute(() =>
      window['socketEvents'].sent.filter(
        (event) => event.event === 'draft:draw'
      )
    );

    const lastEvent = drawEvents.pop();
    const elementName = lastEvent.data.element.name;

    expect(lastEvent.data.element.type).toBe('Path');
    expect(lastEvent.data.active).toBe(false);

    for (let i = 0; i < drawEvents.length; i++) {
      expect(drawEvents[i].data.active).toBe(true);
      expect(drawEvents[i].data.element.name).toBe(elementName);
      expect(drawEvents[i].data.element.type).toBe('Path');
    }

    expect(lastEvent.data.element.segments.length).toBeLessThan(
      drawEvents[drawEvents.length - 1].data.element.segments.length
    );

    // Check didn't take in last pointer move
    const first = getCoordinate(lastEvent.data.element.segments, 0);
    const last = getCoordinate(lastEvent.data.element.segments, -1);
    expect(last[0] - first[0]).toBe(100);
    expect(last[1] - first[1]).toBe(100);
  });
});

I saved the test and ran the test runner again.

For some reason it wasn't finding any events. I ran it again and looked to see if it was drawing a line on the canvas and it wasn't. Puzzled, I remembered having seen something that seemed to suggest that Chrome (the default browser) didn't currently support actions, so I tried switching to Firefox.

[0-2]  Error:  Only Nightly release channel is supported in Devtools/Puppeteer for Firefox. Refer to the following issue:
        https://bugzilla.mozilla.org/show_bug.cgi?id=1606604

        You can use the following link to download Firefox Nightly edition:
        https://www.mozilla.org/en-US/firefox/channel/desktop/

        Adding the following binary capability in Firefox Options is mandatory to run with Nightly edition:

        'moz:firefoxOptions': {
            binary: '/path/to/firefox'
        }

        Note: "Nightly" as a term should be present in the "Firefox Application Name" across all OS's in binary path mentioned above for this to work.

Drats. After downloading pointing the tests to use firefox nightly, I got successful test run.

It turns out the giberish at the bottom of the error means you don't need to include the path to the binary option if a command like firefox-nightly exists (just needs to contain nightly).

Running the Tests on Gitlab CI

Having written and got the tests running, I now needed to commit them and get them running on Gitlab CI.

Having found an article about getting webdriver running on gitlab CI, I added a similar job and script to my projects .gitlab-ci.com and package.json files.

.gitlab-ci.yml addition

ui-test:
   stage: test
   image: node:lts-alpine
   services:
     - name: selenium/standalone-firefox
       alias: selenium
   before_script:
     - yarn
   script:
     - yarn run test:ui --host=selenium

package.json addition

  "scripts": {
    "test:ui": "NODE_ENV=test npx wdio",

Commiting and pushing I was greeted with same need Firefox Nightly error. Noooooooo.

After a bunch of trial and error, including building custom versions of the standalone-firefox docker image, I found, by running npx wdio run --help, that the --host argument used to select the driver host should have actually been --hostname.

Fixing the config files and pushing to gitlab, I finally got the tests passing.

Success.

Other Gotchas

While WebdriverIO is incredibly powerful, it can be quite finicky to get right. Below are some of issues I have had so far.

Actions Not Running

Mouse interactions seem to be very flakey unless you give them a lot of time to run (by specifying duration values that would be expected if a user was actually doing the tests). My problems with Chrome not running my actions in the first tests I wrote was actually caused by this. Changing the action (pointerDown, pointerMove etc) times to 1000ms and adding 500ms pauses between the actions solved this issue.

Returning Complex Objects

If try to return data when using browser.execute that can't be JSON.stringifyed, the test execution will fail. This can be avoided cleaning objects before returning them, for example returning only the properties that you need.

Failing Tests Due to Still Loading

If you are loading things such as images after the page has loaded and trying to test things that should happen after those things have loaded, you need to wait for those things to load. On testing the initial pan/zoom ws working correctly, the tests ran fine on my local machine, but always failed running on the Gitlab runner. Adding a large pause using browser.pause before the tests proved that it was an loading issue. I ended up adding the emitting of a ready event once all the images had loaded.