End to End Testing
Using Cypress

Gleb Bahmutov

VP of Engineering

use up/down and left/right keys to navigate these slides

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end.

we have to act today

ME

you

you

you

you

if I can help I will

@bahmutov

gleb.bahmutov@gmail.com

If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.

Take a deep breath

fight apathy before it kills us all

Testing is a drag

Testing does not pay

Testing is boring

This presentation

Cypress Dev Experience

Code Coverage is a 🗺

GitHub Actions are awesome

Testing smarter

30 people. Atlanta and the World

Fast, easy and reliable testing for anything that runs in a browser

Dr. Gleb Bahmutov, PhD

these slides

🦉 @bahmutov

$ npm install -D cypress
// ui-spec.js
it('loads the app', () => {
  cy.visit('http://localhost:3000')
  cy.get('.todoapp').should('be.visible')
})

Mocha BDD syntax

Chai assertions

it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

Tutorials

Api

Examples

DOM

storage

location

cookies

Cypress tests run in the same browser

DOM

storage

location

cookies

Cypress acts as a proxy for your app

Cypress vs X

People testing with Selenium / WebDriver

Cypress users?

No E2E tests

$ npx cypress open

"Interactive Mode"

  • Runs one or all specs
  • File watcher
  • DOM snapshots and time-traveling debugger

$ npx cypress run

"Headless Mode"

  • Runs single spec at a time
  • No file watching
  • No DOM snapshots
  • automatic screenshot on failure
  • automatic video

What should I test?

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

What should I test?

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

typical Todo application http://todomvc.com/

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Ohhh, we don't have a test for Feature C

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track manually

1. Carefully collect user stories and feature requirments

2. Write matching tests

Keep updating user stories and tests to keep them in sync

HARD

  • first test
  • two more tests
  • hundreds of tests!!!

Keep updating user stories and tests to keep them in sync

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

green: lines executed during the tests

red: lines NOT executed during the tests

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Code coverage from tests indirectly

measures implemented features tested

Keeping track via code coverage

⚠️ 100% Code Coverage ≠ 0🐞

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Unrealistic tests; subset of inputs

code does not implement the feature correctly

Code Coverage with @cypress/code-coverage

  1. Instrument code (YOU)

    1. Using Istanbul / nyc library

  2. Cypress does the rest

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

We have tested "add todo"

Need tests

Write more end-to-end tests

Ughh, missed it

Can we always write an E2E test?

What is this code doing?

Can I write an E2E test to hit this code?

Sometimes you cannot

This code should be unreachable from the user interface

Write a Unit Test!

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

@cypress/code-coverage plugin

We got this line from the unit test

@cypress/code-coverage plugin

Nice job, @cypress/code-coverage plugin

combines e2e and unit test coverage automatically

But wait, there is more ...

If you add Istanbul (nyc) to your Node.js server code

{
  "scripts": {
    "start": "node server",
    "start:coverage": "nyc --silent node server",
  }
}

But wait, there is more ...

And add a GET coverage route

if (global.__coverage__) {
  // Express / Hapi / HTTP / Next.js available
  require('@cypress/code-coverage/middleware/express')(app)
}

Full stack code coverage

Use Code Coverage

  1. Guide test writing
  2. Stop build if coverage drops below X%
  3. Look beyond LoC coverage

How long should a test be?

Most unit tests are very short

  1. Arrange

  2. Act

  3. Assert

Tests are kept short because they run in the terminal. Their shortness helps to debug a failed test.

Write longer tests

Go ahead

Meaningful Long Test

When a test runs for too long...

Multi-page form example

The test is too long

  1. Split into 3 tests

  2. End each test with a "checkpoint"

  3. Starts next test from a "checkpoint"

class MasterForm extends React.Component {
  constructor (props) {
    super(props)
    if (window.Cypress) {
      window.app = this
    }
  }
  ...
}

Inside your app code

Now tests can control the application directly

expose app reference

cy.contains('Next').click()

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
  .its('app.state')
  .should('deep.equal', startOfSecondPageState)

End of the first test

cy.window()
  .its('app')
  .invoke('setState', startOfSecondPageState)

cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)

Start of the second test

Start a test right from the state at the end of the previous test

How to run tests on CI?

# work locally
git add .
git commit -m "feature A + tests"
git push
# CI runs all tests on each commit
cypress run

CI should test every commit

CI without tests is like never changing the oil in your car. It is just a matter of time before it blows up

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

apt-get install xvfb libgtk-3-dev \
  libnotify-dev libgconf-2-4 \
  libnss3 libxss1 libasound2

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

Use one of our prepared images from
https://github.com/cypress-io/cypress-docker-images

  • cypress/base (OS deps + Node)
  • cypress/browsers (cypress/base + Chrome)
  • cypress/included (cypress/browsers + global Cypress module)

Verify Cypress can run

npx cypress verify
# either exits fine
# or shows the OS / install error

"cypress run" calls "verify" when it runs for the very first time on the machine

Run Cypress binary by itself

Linux

npx cypress cache path
/root/.cache/Cypress
/root/.cache/Cypress/3.3.1/Cypress/Cypress --smoke-test --ping=101
101
ldd /home/person/.cache/Cypress/3.3.1/Cypress/Cypress
  linux-vdso.so.1 (0x00007ffe9eda0000)
  libnode.so => /home/person/.cache/Cypress/3.3.1/Cypress/libnode.so (0x00007fecb43c8000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fecb41ab000)
  libgtk-3.so.0 => not found
  libgdk-3.so.0 => not found
  ...

this is what "cypress verify" does

Cache NPM and Cypress

  • git checkout
  • npm install
  • cypress run

Performance

(on CI server)

Cache NPM and Cypress

  • git checkout
  • npm install
    • cypress download
  • cypress run

Performance

(on CI server)

Cache NPM and Cypress

  • git checkout
  • npm install
    • cypress download
  • ​cache ~/.npm and ~/.cache
  • cypress run

Performance

npx cypress cache path

(on CI server)

Cache NPM and Cypress

Performance

  • git checkout
  • restore cache
  • npm install
    • cypress download
  • ​cache ~/.npm and ~/.cache
  • cypress run

(on CI server)

version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record

workflows:
  version: 2
  build:
    jobs:
      - test

Docker image

Caching

Caching

typical CI config file

Install

run tests

maybe start app

Making it easy for users is not easy

Cypress continuous intgegration guide https://on.cypress.io/ci

Copy / paste / tweak until CI passes 🙁

I have 100s of tests ...

$ npx cypress run --record --parallel

Cypress v3.1.0

Spin N CI machines and

cypress-dashboard parallelization

# machines run duration time savings
1 22:50 ~
2 11:47 48%
3 7:51 65%
4 5:56 74%
6 3:50 83%
8 3:00 87%
10 2:19 90%

cypress-dashboard parallelization

10 machines = 10x speed up

defaults: &defaults
  working_directory: ~/app
  docker:
    - image: cypress/browsers:chrome67

version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # find compatible cache from previous build,
      # it should have same dependencies installed from package.json checksum
      - restore_cache:
          keys:
            - cache-{{ .Branch }}-{{ checksum "package.json" }}
      - run:
          name: Install Dependencies
          command: npm ci
      # run verify and then save cache.
      # this ensures that the Cypress verified status is cached too
      - run: npm run cy:verify
      # save new cache folder if needed
      - save_cache:
          key: cache-{{ .Branch }}-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      - run: npm run types
      - run: npm run stop-only
      # all other test jobs will run AFTER this build job finishes
      # to avoid reinstalling dependencies, we persist the source folder "app"
      # and the Cypress binary to workspace, which is the fastest way
      # for Circle jobs to pass files
      - persist_to_workspace:
          root: ~/
          paths:
            - app
            - .cache/Cypress

  4x-electron:
    <<: *defaults
    # tell CircleCI to execute this job on 4 machines simultaneously
    parallelism: 4
    steps:
      - attach_workspace:
          at: ~/
      - run:
          command: npm start
          background: true
      # runs Cypress test in load balancing (parallel) mode
      # and groups them in Cypress Dashboard under name "4x-electron"
      - run: npm run e2e:record -- --parallel --group $CIRCLE_JOB

workflows:
  version: 2
  # this workflow has 4 jobs to show case Cypress --parallel and --group flags
  # "build" installs NPM dependencies so other jobs don't have to
  #   └ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
  #   └ "4x-electron" job load balances all specs across 4 CI machines
  #   └ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
  build_and_test:
    jobs:
      - build
      # this group "4x-electron" will load balance all specs
      # across 4 CI machines
      - 4x-electron:
          requires:
            - build

Parallel config is ... more complicated

typical CI config file

- install + run jobs

- workspace

- parallel flags

# first, install Cypress, then run all tests (in parallel)
stages:
  - build
  - test

# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm"
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"

# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .npm
    - cache/Cypress
    - node_modules

# this job installs NPM dependencies and Cypress
install:
  image: cypress/base:10
  stage: build

  script:
    - npm ci
    - $(npm bin)/print-env CI
    - npm run cy:verify

# all jobs that actually run tests can use the same definition
.job_template: &job
  image: cypress/base:10
  stage: test
  script:
    # print CI environment variables for reference
    - $(npm bin)/print-env CI
    # start the server in the background
    - npm run start:ci &
    # run Cypress test in load balancing mode, pass id to tie jobs together
    - npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons

# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
  <<: *job
electrons-2:
  <<: *job
electrons-3:
  <<: *job
electrons-4:
  <<: *job
electrons-5:
  <<: *job
pipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
    }
  }

  stages {
    // first stage installs node dependencies and Cypress binary
    stage('build') {
      steps {
        // there a few default environment variables on Jenkins
        // on local Jenkins machine (assuming port 8080) see
        // http://localhost:8080/pipeline-syntax/globals#env
        echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
        sh 'npm ci'
        sh 'npm run cy:verify'
      }
    }

    stage('start local server') {
      steps {
        // start local server in the background
        // we will shut it down in "post" command block
        sh 'nohup npm start &'
      }
    }

    // this tage runs end-to-end tests, and each agent uses the workspace
    // from the previous stage
    stage('cypress parallel tests') {
      environment {
        // we will be recordint test results and video on Cypress dashboard
        // to record we need to set an environment variable
        // we can load the record key variable from credentials store
        // see https://jenkins.io/doc/book/using/using-credentials/
        CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
        // because parallel steps share the workspace they might race to delete
        // screenshots and videos folders. Tell Cypress not to delete these folders
        CYPRESS_trashAssetsBeforeRuns = 'false'
      }

      // https://jenkins.io/doc/book/pipeline/syntax/#parallel
      parallel {
        // start several test jobs in parallel, and they all
        // will use Cypress Dashboard to load balance any found spec files
        stage('tester A') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }

        // second tester runs the same command
        stage('tester B') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }
      }

    }
  }

  post {
    // shutdown the server running in the background
    always {
      echo 'Stopping local server'
      sh 'pkill -f http-server'
    }
  }
}
language: node_js

node_js:
  # Node 10.3+ includes npm@6 which has good "npm ci" command
  - 10.8

cache:
  # cache both npm modules and Cypress binary
  directories:
    - ~/.npm
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify

defaults: &defaults
  script:
    #   ## print all Travis environment variables for debugging
    - $(npm bin)/print-env TRAVIS
    - npm start -- --silent &
    - npm run cy:run -- --record --parallel --group $STAGE_NAME
    # after all tests finish running we need
    # to kill all background jobs (like "npm start &")
    - kill $(jobs -p) || true

jobs:
  include:
    # we have multiple jobs to execute using just a single stage
    # but we can pass group name via environment variable to Cypress test runner
    - stage: test
      env:
        - STAGE_NAME=1x-electron
      <<: *defaults
    # run tests in parallel by including several test jobs with same name variable
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults

Different CIs

Copy / paste / tweak until CI passes 😡

GitHub Actions v2

 🎉 GitHub Actions v2  🎉

GitHub Actions are a way to write CI code using JavaScript / TypeScript

You

Tool's author

- uses: actions/checkout@v1
- name: Cache node modules
  uses: actions/cache@v1
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: ${{ runner.os }}-node-
- run: npm ci
- run: npm test

NPM or Yarn install

writer: You

ughh, I never remember these

NPM or Yarn install

const hasha = require('hasha')
// we are using dependency
// "cache": "github:cypress-io/github-actions-cache#8bec6cc"
const { restoreCache, saveCache } = require('cache/lib/index')

const useYarn = fs.existsSync('yarn.lock')
const lockFilename = useYarn ? 'yarn.lock' : 'package-lock.json'
const lockHash = hasha.fromFileSync(lockFilename)
const platformAndArch = `${process.platform}-${process.arch}`

// this is simplified code for clarity
// see action file index.js for full details
const NPM_CACHE = {
  inputPath: '~/.npm', // or '~/.cache/yarn'
  primaryKey: `npm-${platformAndArch}-${lockHash}`,
  restoreKeys: `npm-${platformAndArch}-`
}

const restoreCachedNpm = () => {
  console.log('trying to restore cached NPM modules')
  return restoreCache(
    NPM_CACHE.inputPath,
    NPM_CACHE.primaryKey,
    NPM_CACHE.restoreKeys
  )
}

const saveCachedNpm = () => {
  console.log('saving NPM modules')
  return saveCache(NPM_CACHE.inputPath, NPM_CACHE.primaryKey)
}
restoreCachedNpm()
.then(npmCacheHit => {
  console.log('npm cache hit', npmCacheHit)

  return install().then(() => {
    if (npmCacheHit) {
      return
    }

    return saveCachedNpm()
  })
})
.catch(error => {
  console.log(error)
  core.setFailed(error.message)
})
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm test

NPM or Yarn install

writer: Me

user: You

Cypress GitHub Action

Cypress GitHub Action

name: End-to-end tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1

(does npm install and caching, runs the tests correctly, can build app and start server and more)

Cypress GitHub Action

name: Chome tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1
        with:
          browser: chrome

Use parameters to control behavior

Cypress GitHub Action

name: Record test results
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1
        with:
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}

Cypress GitHub Action

name: Parallel tests
on: [push]
jobs:
  cypress-run:
    strategy:
      matrix:
        machines: [1, 2, 3]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v1
      - uses: cypress-io/github-action@v1
        with:
          record: true
          parallel: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}

Cypress GitHub Action

parallel-runs-across-platforms:
  strategy:
    matrix:
      os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
      machines: [1, 2]
  runs-on: ${{ matrix.os }}
  steps:
    - uses: actions/checkout@v1
    - uses: cypress-io/github-action@v1
      with:
        record: true
        parallel: true
        group: Parallel 2x on ${{ matrix.os }}
      env:
        CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}

recorded parallel runs

How to run tests quickly?

click DOWN arrow to navigate these slides

Different Environments

OS: Windows, Linux, Darwin

Browser: Electron, Chrome, Firefox, Edge Beta

Env: local, staging, production

Run all tests in each environment

N tests x 3 (OS) x 3 (Browser) x 3 (env)

Run all tests in each environment

N tests x 3 (OS) x 3 (Browser) x 3 (env) x 2 (viewport)

+ nightly tests?

Running all tests in each environment

Will be wasteful

Instead consider a variety of tests:

  • functional

  • visual

  • a11y

  • API

Approach: Smoke test

// smoke.js
export const smokeTest = () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
  // click through main app features
  ...
}

// smoke-spec.js
import { smokeTest } from './smoke'
it('does not smoke', () => {
  smokeTest()
})

1 smoke test = 84% coverage

// viewports-spec.js
import { smokeTest } from './smoke'

Cypress._.each(['macbook-15', 'iphone-6'], 
  viewport => {
    it(`works on ${viewport}`, () => {
      cy.viewport(viewport)
      smokeTest()
    })
})

Cover viewports

# CircleCI config
- cypress/run:
    name: smoke tests
    filters:
      branches:
        only:
        - master
    start: npm start
    command: |
      npx cypress run --config \
      baseUrl=http://localhost:1234,testFiles=smoke-spec.js

Vary settings

# CircleCI config
- cypress/run:
    name: smoke tests
    filters:
      branches:
        only:
        - master
    start: npm start
    command: |
      npx cypress run \
      --config-file cypress-smoke.json
{
  "baseUrl": "http://localhost:1234",
  "testFiles": [
    "smoke-spec.js"
  ]
}

cypress-smoke.json

Vary test behavior

it('works differently', () => {
  if (Cypress.browser.family === 'electron') {
    ...
  }
  if (Cypress.platform === 'darwin') {
    ...
  }
  // pass custom env variables
  // CYPRESS_ENVIRONMENT=... cypress run
  if (Cypress.env('ENVIRONMENT') === 'staging') {
    ...
  }
})

Vary test behavior: OS

import { onlyOn, skipOn } from '@cypress/skip-test'
onlyOn('mac', () => {
  // this callback will only evaluate on Mac
  // thus the tests will be completely hidden from other platforms
  describe('Mac tests', () => {
    it('works', () => {})
  })
})
skipOn('mac', () => {
  // this test will run on every platform but Mac
  it('hides this test on Mac', () => {})
})

Vary test behavior: browser

import { onlyOn, skipOn } from '@cypress/skip-test'
skipOn('firefox', () => {
  it('does not work on FF yet', () => {
    ...
  })
})

Vary test behavior: environment

import {onlyOn} from '@cypress/skip-test'
const stubServer = () => {
  cy.server()
  cy.route('/api/me', 'fx:me.json')
  cy.route('/api/permissions', 'fx:permissions.json')
  // Lots of fixtures ...
}
it('works', () => {
  onlyOn('staging', stubServer)
  cy.visit('/')
  cy.get('.user-profile').should('be.visible')
  ...
})

when testing "staging" should stub network calls

If you remember just 3 things:

Cypress focuses on DX

cypress-io/github-action

end-to-end tests are effective

Thank you 👏

Gleb Bahmutov

@bahmutov