VISUAL COMPONENT TESTING WITH CYPRESS.IO AND APPLITOOLS

Gil Tayar

Sr. Architect, Applitools

@giltayar

Gleb Bahmutov

VP of Engineering, Cypress.io

@bahmutov

WEBCAST

Today's webinar

  • End-to-end visual tests

  • Component tests for React

    • with visual assertions

  • Tips & Tricks

  • Questions & Answers

code at https://github.com/bahmutov/sudoku-applitools

live at https://glebbahmutov.com/sudoku-applitools/

code at https://github.com/bahmutov/sudoku-applitools

live at https://glebbahmutov.com/sudoku-applitools/

First cypress test

describe('Sudoku', () => {
  context('on mobile', () => {
    beforeEach(() => {
      cy.viewport(300, 600)
      cy.visit('/')
    })

    it('plays on mobile', () => {
      // on easy setting there are 45 filled cells at the start
      cy.get('.game__cell--filled').should('have.length', 45)
      cy.contains('.status__time', '00:00')
      cy.contains('.status__difficulty-select', 'Easy')
    })
  })
})

https://www.cypress.io/

describe('Sudoku', () => {
  context('on mobile', () => {
    beforeEach(() => {
      cy.viewport(300, 600)
      cy.visit('/')
    })

    it('plays on mobile', () => {
      // on easy setting there are 45 filled cells at the start
      cy.get('.game__cell--filled').should('have.length', 45)
      cy.contains('.status__time', '00:00')
      cy.contains('.status__difficulty-select', 'Easy')
    })
  })
})

First cypress test

https://www.cypress.io/

I think this CSS change makes sense, right?!

.status__difficulty {
  /* position: relative; */
  top: 39px;
  left: 20px;
}

???

https://github.com/bahmutov/sudoku-applitools/pull/6

Does it look right?

👩‍💻 ✅

If I change this CSS (or class name or layout) just a little bit ...

🤖 🛑

@bahmutov

desktop

tablet

mobile

At every resolution?

Let's take visual snapshots

https://applitools.com/cypress/

npm i --save-dev @applitools/eyes-cypress
npx eyes-setup
beforeEach(() => {
  cy.eyesOpen({
    appName: 'Sudoku',
    browser: [
      { width: 1024, height: 768 }, // desktop
      { width: 600, height: 750 }, // tablet
      { width: 450, height: 650 }, // mobile
      { width: 1024, height: 768, name: 'firefox' }, // Firefox
      { width: 1024, height: 768, name: 'safari' }, // Safari
      { width: 1024, height: 768, name: 'ie11' }, // 😱
      { iosDeviceInfo: {  // iOS devices
          deviceName: 'iPhone XR', 
      	}
      },
      { mobile: true, // Chrome device emulation
        width: 800, height: 600, deviceScaleFactor: 3,
      }
    ],
  })
})

Let's take visual snapshots

https://applitools.com/cypress/

cy.visit('/')
// on easy setting there are 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00')
cy.contains('.status__difficulty-select', 'Easy')
cy.eyesCheckWindow({ tag: 'App' })

functional assertions

visual assertions

Let's take visual snapshots

https://applitools.com/cypress/

cy.visit('/')
// on easy setting there are 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
// cy.contains('.status__time', '00:00')
// cy.contains('.status__difficulty-select', 'Easy')
cy.eyesCheckWindow({ tag: 'App' })

functional assertions

visual assertions

Applitools creates screenshots with multiple resolutions, devices, browsers at the same time

Applitools Test Results

Does it look right?

If I change this CSS (or class name or layout) just a little bit ...

👩‍💻 ✅ 🕰

🤖 🛑

Does it look the same?

🤖 ✅ ⏱

👩‍💻 🛑 🕰

If I change this CSS (or class name or layout) just a little bit ...

So what does this change?

.status__difficulty {
  /* position: relative; */
  top: 39px;
  left: 20px;
}

???

https://github.com/bahmutov/sudoku-applitools/pull/6

Visual Differences

Root Cause Analysis

Ignore Grid Cells

// ignore the contents of the board's cells
cy.eyesCheckWindow({
  tag: 'App',
  layout: {
    selector: 'game__board',
  },
})

https://www.npmjs.com/package/@applitools/eyes-cypress#Check-window

Ignore Grid Cells

// ignore the contents of the board's cells
cy.eyesCheckWindow({
  tag: 'App',
  layout: {
    selector: 'game__board',
  },
})

True Visual Diff

dynamic (random) data

clocks

animations and transitions

visual testing: potential problems

Missing in E2E test:

playing the actual game

(because of the random board)

testing the components that build the game

in all possible states

Built from React Components

import React from 'react'
import { render } from 'react-dom'
import { App } from './App'

render(<App />, document.getElementById('root'))

index.js

import React from 'react'
import { Game } from './Game'
import './App.css'
import { SudokuProvider } from './context/SudokuContext'

export const App = () => {
  return (
    <SudokuProvider>
      <Game />
    </SudokuProvider>
  )
}

App.js

Top level component App

import React, { useState, useEffect } from 'react'
import moment from 'moment'
import { Header } from './components/layout/Header'
import { GameSection } from './components/layout/GameSection'
import { StatusSection } from './components/layout/StatusSection'
import { Footer } from './components/layout/Footer'
import { getUniqueSudoku } from './solver/UniqueSudoku'
import { useSudokuContext } from './context/SudokuContext'

export const Game = () => {
  ...
}

Game.js

Game component

return (
  <>
    <div className={overlay?"container blur":"container"}>
      <Header onClick={onClickNewGame}/>
      <div className="innercontainer">
        <GameSection
          onClick={(indexOfArray) => onClickCell(indexOfArray)}
        />
        <StatusSection
          onClickNumber={(number) => onClickNumber(number)}
          onChange={(e) => onChangeDifficulty(e)}
          onClickUndo={onClickUndo}
          onClickErase={onClickErase}
          onClickHint={onClickHint}
          onClickMistakesMode={onClickMistakesMode}
          onClickFastMode={onClickFastMode}
        />
      </div>
      <Footer />
    </div>
  </>
)

Game.js

Game component

Component Inputs

import React from 'react';
import { useSudokuContext } from '../context/SudokuContext';

/**
 * React component for the Number Selector in the Status Section.
 */
export const Numbers = (props) => {
  let { numberSelected } = useSudokuContext();
  return (
    <div className="status__numbers">
      {
        [1, 2, 3, 4, 5, 6, 7, 8, 9].map((number) => {
          if (numberSelected === number.toString()) {
            return (
              <div className="status__number status__number--selected"
                key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          } else {
            return (
              <div className="status__number" key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          }
        })
      }
    </div>
  )
}

Numbers.js

<Numbers onClickNumber={(number) => props.onClickNumber(number)} />

StatusSection.js

<Numbers .../>

props

context

user clicks

DOM

prop calls

How does the component look and behave under different inputs?

@bahmutov

check how component renders with different props / data

check how component behaves when you interact with it

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

yarn add -D cypress-react-unit-test

@bahmutov

React Component Tests

https://github.com/bahmutov/cypress-react-unit-test

// cypress/support/index.js
require('cypress-react-unit-test/support')
// cypress/plugins/index.js
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  return config
}
// cypress.json
{
  "experimentalComponentTesting": true,
  "componentFolder": "src"
}

@bahmutov

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />); // instead of cy.visit()
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

@bahmutov

test Numbers component

Numbers.spec.js

@bahmutov

test Numbers component

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />); // instead of cy.visit()
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

apply global styles

import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
import '../App.css'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Numbers.spec.js

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

set the right structure

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  )
  // confirm numbers
})

Numbers.spec.js

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

click a number

it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

Numbers.spec.js

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4')
      .should('have.class', 'status__number--selected')
  })
})

Numbers.spec.js

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>,
  )
  // use a single image snapshot after making sure
  // the component has been rendered into the DOM
  cy.get('.status__number').should('have.length', 9)
  cy.eyesCheckWindow({ tag: 'all numbers' })
})

Numbers.spec.js

@bahmutov

visual component test

Assert the UI has updated before taking the snapshot

it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>,
  )
  // use a single image snapshot after making sure
  // the component has been rendered into the DOM
  cy.get('.status__number').should('have.length', 9)
  cy.eyesCheckWindow({ tag: 'all numbers' })
})

Numbers.spec.js

@bahmutov

visual component test

Assert the UI has updated before taking the snapshot

Applitools screenshot

it('shows selected number', () => {
  mount(
    <SudokuContext.Provider value={{ numberSelected: '4' }}>
      <div className="innercontainer">
        <section className="status">
          <Numbers />
        </section>
      </div>
    </SudokuContext.Provider>,
  )
  cy.contains('.status__number', '4').should(
    'have.class',
    'status__number--selected',
  )
  cy.eyesCheckWindow({ tag: 'selected 4' })
})

@bahmutov

selected number visual test

Numbers.spec.js

it('shows selected number', () => {
  mount(
    <SudokuContext.Provider value={{ numberSelected: '4' }}>
      <div className="innercontainer">
        <section className="status">
          <Numbers />
        </section>
      </div>
    </SudokuContext.Provider>,
  )
  cy.contains('.status__number', '4').should(
    'have.class',
    'status__number--selected',
  )
  cy.eyesCheckWindow({ tag: 'selected 4' })
})

@bahmutov

Numbers.spec.js

selected number visual test

Applitools screenshot

Numbers component screenshots

Applitools Dashboard: results, insights

Applitools is fast enough to work interactively or you could skip diff comparisons when running locally. See code in https://github.com/bahmutov/sudoku-applitools

component tests: any level

?

<App />

<Game />

<Header />

<GameSection />

<StatusSection />

<Footer />

<Timer />

<Difficulty />

<Numbers />

tested

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.eyesCheckWindow({ tag: 'created board' })
})

App.spec.js

@bahmutov

Why not the entire game?

component tests: any level

import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  cy.eyesCheckWindow({ tag: 'created board' })
})

App.spec.js

Because every time test runs, a new random board will be generated

component tests: any level

// App.js uses Game.js
// Game.js
import { getUniqueSudoku } from './solver/UniqueSudoku'
...
function _createNewGame(e) {
  let [temporaryInitArray, temporarySolvedArray] = getUniqueSudoku(difficulty, e);
  ...
}

// cypress/fixtures/init-array.json
["0", "0", "9", "0", "2", "0", "0", ...]
// cypress/fixtures/solved-array.json
["6", "7", "9", "3", "2", "8", "4", ...]

@bahmutov

mock component methods

mock ES6 import from test

import initArray from '../cypress/fixtures/init-array.json'
import solvedArray from '../cypress/fixtures/solved-array.json'
import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('plays one move', () => {
  cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.eyesCheckWindow({ tag: 'same board' })
})

mock component methods

mock ES6 import

import initArray from '../cypress/fixtures/init-array.json'
import solvedArray from '../cypress/fixtures/solved-array.json'
import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('plays one move', () => {
  cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.eyesCheckWindow({ tag: 'same board' })
})

Same board every time

import initArray from '../cypress/fixtures/init-array.json'
import solvedArray from '../cypress/fixtures/solved-array.json'
it('plays one move', () => {
  cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.eyesCheckWindow({ tag: 'one move' })
})

play a game move

import initArray from '../cypress/fixtures/init-array.json'
import solvedArray from '../cypress/fixtures/solved-array.json'
it('plays one move', () => {
  cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.eyesCheckWindow({ tag: 'one move' })
})

play a game move

Plays one move screenshots

it('plays to win', () => {
  // start with all but the first cell filled with solved array
  const almostSolved = [...solvedArray]
  // by setting entry to "0" we effectively clear the cell
  almostSolved[0] = '0'
  cy.stub(UniqueSudoku, 'getUniqueSudoku')
    .returns([almostSolved, solvedArray])
    .as('getUniqueSudoku')
  cy.clock()
  mount(<App />)
  cy.eyesCheckWindow({ tag: '1 game is almost solved' })

  // win the game
  cy.get('.game__cell').first().click()
  // use the known number to fill the first cell
  cy.contains('.status__number', solvedArray[0]).click()
  
  // winning message displayed
  cy.get('.overlay__text').should('be.visible')
  cy.eyesCheckWindow({ tag: '2 game is solved' })

  // clicking the overlay starts the new game
  cy.get('@getUniqueSudoku').should('have.been.calledOnce')
  cy.get('.overlay__text').click()
  cy.get('.overlay').should('not.be.visible')
  cy.get('@getUniqueSudoku').should('have.been.calledTwice')

  cy.eyesCheckWindow({ tag: '3 start new game after solved game' })
})

play the full game via component test

play the full game via component test

it('plays to win', () => {
  // start with all but the first cell filled with solved array
  const almostSolved = [...solvedArray]
  // by setting entry to "0" we effectively clear the cell
  almostSolved[0] = '0'
  cy.stub(UniqueSudoku, 'getUniqueSudoku')
    .returns([almostSolved, solvedArray])
    .as('getUniqueSudoku')
  cy.clock()
  mount(<App />)
  cy.eyesCheckWindow({ tag: '1 game is almost solved' })

  // win the game
  cy.get('.game__cell').first().click()
  // use the known number to fill the first cell
  cy.contains('.status__number', solvedArray[0]).click()
  
  // winning message displayed
  cy.get('.overlay__text').should('be.visible')
  cy.eyesCheckWindow({ tag: '2 game is solved' })

  // clicking the overlay starts the new game
  cy.get('@getUniqueSudoku').should('have.been.calledOnce')
  cy.get('.overlay__text').click()
  cy.get('.overlay').should('not.be.visible')
  cy.get('@getUniqueSudoku').should('have.been.calledTwice')

  cy.eyesCheckWindow({ tag: '3 start new game after solved game' })
})

Observe the test inside Cypress

Desktop

Mobile

Applitools checks the screenshots

visual testing tips

Always use an assertion before the visual snapshot command

// use a single image snapshot after making sure
// the component has been rendered into the DOM
cy.get('.status__number').should('have.length', 9)
cy.eyesCheckWindow(...)
cy.get('.overlay__text').should('be.visible')
cy.eyesCheckWindow(...)

visual testing tips

Always use a single assertion before the visual snapshot command

// on easy setting there are 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.contains('.status__time', '00:00')
cy.contains('.status__difficulty-select', 'Easy')
cy.eyesCheckWindow(...)

Unnecessary

visual testing tips

Always use a single assertion before the visual snapshot command

// on easy setting there are 45 filled cells at the start
cy.get('.game__cell--filled').should('have.length', 45)
cy.eyesCheckWindow(...)

Validates the entire page in a single shot!

Configure IntelliSense

https://www.npmjs.com/package/@applitools/eyes-cypress#intelligent-code-completion

visual testing tips

Use tags

cy.eyesCheckWindow({tag: 'solved board'})

// several screenshots in the test
cy.eyesCheckWindow({tag: '1 game is almost solved'})

cy.eyesCheckWindow({tag: '2 game is solved'})

visual testing tips

Test components at different resolutions, devices, and browsers

cy.eyesOpen({
  appName: 'Sudoku',
  batchName: 'Sudoku',
  browser: [
    { width: 800, height: 600, name: 'chrome' },
    { width: 1024, height: 768, name: 'chrome' },
    { width: 1920, height: 1080, name: 'chrome' },
    { width: 800, height: 600, name: 'firefox' },
    { deviceName: 'iPhone X' },
    { deviceName: 'iPad' },
  ],
})

End-to-end test visits the URL and verifies the deployed application works similar to user scenario

Component test mounts the component and verifies that it works in a variety of situations

Real browsers, real interactions

Functional + visual assertions

Thank you 👏

Gleb Bahmutov @bahmutov

Gil Tayar @giltayar

Contact