use up/down and left/right keys to navigate these slides
https://lizkeogh.com/2019/07/02/off-the-charts/
+3 degrees Celsius will be the end.
If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.
Example: https://fab.earth
fight apathy before it kills us all
30 people. Atlanta and the World
Fast, easy and reliable testing for anything that runs in a browser
these slides
$ npm install -D cypress
// ui-spec.js
it('loads the app', () => {
cy.visit('http://localhost:3000')
cy.get('.todoapp').should('be.visible')
})
Mocha BDD syntax
Chai assertions
it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})
Tutorials
Api
Examples
DOM
storage
location
cookies
Cypress tests run in the same browser
DOM
storage
location
cookies
Cypress acts as a proxy for your app
People testing with Selenium / WebDriver
Cypress users?
No E2E tests
"Interactive Mode"
"Headless Mode"
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
typical Todo application http://todomvc.com/
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })
it('completes todos', () => {
... })
Ohhh, we don't have a test for Feature C
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
green: lines executed during the tests
red: lines NOT executed during the tests
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Code coverage from tests indirectly
measures implemented features tested
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Unrealistic tests; subset of inputs
code does not implement the feature correctly
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
We have tested "add todo"
Need tests
Write more end-to-end tests
Ughh, missed it
What is this code doing?
Can I write an E2E test to hit this code?
This code should be unreachable from the user interface
import {getVisibleTodos} from '../../src/selectors'
describe('getVisibleTodos', () => {
it('throws an error for unknown visibility filter', () => {
expect(() => {
getVisibleTodos({
todos: [],
visibilityFilter: 'unknown-filter'
})
}).to.throw()
})
})
@cypress/code-coverage plugin
We got this line from the unit test
@cypress/code-coverage plugin
Nice job, @cypress/code-coverage plugin
combines e2e and unit test coverage automatically
If you add Istanbul (nyc) to your Node.js server code
{
"scripts": {
"start": "node server",
"start:coverage": "nyc --silent node server",
}
}
And add a GET coverage route
if (global.__coverage__) {
// Express / Hapi / HTTP / Next.js available
require('@cypress/code-coverage/middleware/express')(app)
}
Tests are kept short because they run in the terminal. Their shortness helps to debug a failed test.
Go ahead
class MasterForm extends React.Component {
constructor (props) {
super(props)
if (window.Cypress) {
window.app = this
}
}
...
}
Now tests can control the application directly
expose app reference
cy.contains('Next').click()
cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.window()
.its('app.state')
.should('deep.equal', startOfSecondPageState)
cy.window()
.its('app')
.invoke('setState', startOfSecondPageState)
cy.log('Second page')
cy.contains('h1', 'Book Hotel 2')
cy.get('#username').type('JoeSmith', typeOptions)
# work locally
git add .
git commit -m "feature A + tests"
git push
# CI runs all tests on each commit
cypress run
CI without tests is like never changing the oil in your car. It is just a matter of time before it blows up
Nothing to install, Cypress should just work
apt-get install xvfb libgtk-3-dev \
libnotify-dev libgconf-2-4 \
libnss3 libxss1 libasound2
Nothing to install, Cypress should just work
Use one of our prepared images from
https://github.com/cypress-io/cypress-docker-images
npx cypress verify
# either exits fine
# or shows the OS / install error
"cypress run" calls "verify" when it runs for the very first time on the machine
npx cypress cache path
/root/.cache/Cypress
/root/.cache/Cypress/3.3.1/Cypress/Cypress --smoke-test --ping=101
101
ldd /home/person/.cache/Cypress/3.3.1/Cypress/Cypress
linux-vdso.so.1 (0x00007ffe9eda0000)
libnode.so => /home/person/.cache/Cypress/3.3.1/Cypress/libnode.so (0x00007fecb43c8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fecb41ab000)
libgtk-3.so.0 => not found
libgdk-3.so.0 => not found
...
this is what "cypress verify" does
Performance
(on CI server)
Performance
(on CI server)
Performance
npx cypress cache path
(on CI server)
Performance
(on CI server)
version: 2
jobs:
test:
docker:
- image: cypress/base:10
steps:
- checkout
# restore folders with npm dependencies and Cypress binary
- restore_cache:
keys:
- cache-{{ checksum "package.json" }}
# install npm dependencies and Cypress binary
# if they were cached, this step is super quick
- run:
name: Install dependencies
command: npm ci
- run: npm run cy:verify
# save npm dependencies and Cypress binary for future runs
- save_cache:
key: cache-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
# start server before starting tests
- run:
command: npm start
background: true
- run: npm run e2e:record
workflows:
version: 2
build:
jobs:
- test
Docker image
Caching
Caching
typical CI config file
Install
run tests
maybe start app
Making it easy for users is not easy
Cypress continuous intgegration guide https://on.cypress.io/ci
I have 100s of tests ...
$ npx cypress run --record --parallel
Cypress v3.1.0
Spin N CI machines and
# machines | run duration | time savings |
---|---|---|
1 | 22:50 | ~ |
2 | 11:47 | 48% |
3 | 7:51 | 65% |
4 | 5:56 | 74% |
6 | 3:50 | 83% |
8 | 3:00 | 87% |
10 | 2:19 | 90% |
10 machines = 10x speed up
defaults: &defaults
working_directory: ~/app
docker:
- image: cypress/browsers:chrome67
version: 2
jobs:
build:
<<: *defaults
steps:
- checkout
# find compatible cache from previous build,
# it should have same dependencies installed from package.json checksum
- restore_cache:
keys:
- cache-{{ .Branch }}-{{ checksum "package.json" }}
- run:
name: Install Dependencies
command: npm ci
# run verify and then save cache.
# this ensures that the Cypress verified status is cached too
- run: npm run cy:verify
# save new cache folder if needed
- save_cache:
key: cache-{{ .Branch }}-{{ checksum "package.json" }}
paths:
- ~/.npm
- ~/.cache
- run: npm run types
- run: npm run stop-only
# all other test jobs will run AFTER this build job finishes
# to avoid reinstalling dependencies, we persist the source folder "app"
# and the Cypress binary to workspace, which is the fastest way
# for Circle jobs to pass files
- persist_to_workspace:
root: ~/
paths:
- app
- .cache/Cypress
4x-electron:
<<: *defaults
# tell CircleCI to execute this job on 4 machines simultaneously
parallelism: 4
steps:
- attach_workspace:
at: ~/
- run:
command: npm start
background: true
# runs Cypress test in load balancing (parallel) mode
# and groups them in Cypress Dashboard under name "4x-electron"
- run: npm run e2e:record -- --parallel --group $CIRCLE_JOB
workflows:
version: 2
# this workflow has 4 jobs to show case Cypress --parallel and --group flags
# "build" installs NPM dependencies so other jobs don't have to
# └ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
# └ "4x-electron" job load balances all specs across 4 CI machines
# └ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
build_and_test:
jobs:
- build
# this group "4x-electron" will load balance all specs
# across 4 CI machines
- 4x-electron:
requires:
- build
Parallel config is ... more complicated
typical CI config file
- install + run jobs
- workspace
- parallel flags
# first, install Cypress, then run all tests (in parallel)
stages:
- build
- test
# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
npm_config_cache: "$CI_PROJECT_DIR/.npm"
CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"
# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm
- cache/Cypress
- node_modules
# this job installs NPM dependencies and Cypress
install:
image: cypress/base:10
stage: build
script:
- npm ci
- $(npm bin)/print-env CI
- npm run cy:verify
# all jobs that actually run tests can use the same definition
.job_template: &job
image: cypress/base:10
stage: test
script:
# print CI environment variables for reference
- $(npm bin)/print-env CI
# start the server in the background
- npm run start:ci &
# run Cypress test in load balancing mode, pass id to tie jobs together
- npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons
# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
<<: *job
electrons-2:
<<: *job
electrons-3:
<<: *job
electrons-4:
<<: *job
electrons-5:
<<: *job
pipeline {
agent {
// this image provides everything needed to run Cypress
docker {
image 'cypress/base:10'
}
}
stages {
// first stage installs node dependencies and Cypress binary
stage('build') {
steps {
// there a few default environment variables on Jenkins
// on local Jenkins machine (assuming port 8080) see
// http://localhost:8080/pipeline-syntax/globals#env
echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
sh 'npm ci'
sh 'npm run cy:verify'
}
}
stage('start local server') {
steps {
// start local server in the background
// we will shut it down in "post" command block
sh 'nohup npm start &'
}
}
// this tage runs end-to-end tests, and each agent uses the workspace
// from the previous stage
stage('cypress parallel tests') {
environment {
// we will be recordint test results and video on Cypress dashboard
// to record we need to set an environment variable
// we can load the record key variable from credentials store
// see https://jenkins.io/doc/book/using/using-credentials/
CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
// because parallel steps share the workspace they might race to delete
// screenshots and videos folders. Tell Cypress not to delete these folders
CYPRESS_trashAssetsBeforeRuns = 'false'
}
// https://jenkins.io/doc/book/pipeline/syntax/#parallel
parallel {
// start several test jobs in parallel, and they all
// will use Cypress Dashboard to load balance any found spec files
stage('tester A') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
// second tester runs the same command
stage('tester B') {
steps {
echo "Running build ${env.BUILD_ID}"
sh "npm run e2e:record:parallel"
}
}
}
}
}
post {
// shutdown the server running in the background
always {
echo 'Stopping local server'
sh 'pkill -f http-server'
}
}
}
language: node_js
node_js:
# Node 10.3+ includes npm@6 which has good "npm ci" command
- 10.8
cache:
# cache both npm modules and Cypress binary
directories:
- ~/.npm
- ~/.cache
override:
- npm ci
- npm run cy:verify
defaults: &defaults
script:
# ## print all Travis environment variables for debugging
- $(npm bin)/print-env TRAVIS
- npm start -- --silent &
- npm run cy:run -- --record --parallel --group $STAGE_NAME
# after all tests finish running we need
# to kill all background jobs (like "npm start &")
- kill $(jobs -p) || true
jobs:
include:
# we have multiple jobs to execute using just a single stage
# but we can pass group name via environment variable to Cypress test runner
- stage: test
env:
- STAGE_NAME=1x-electron
<<: *defaults
# run tests in parallel by including several test jobs with same name variable
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
- stage: test
env:
- STAGE_NAME=4x-electron
<<: *defaults
GitHub Actions are a way to write CI code using JavaScript / TypeScript
You
Tool's author
- uses: actions/checkout@v1
- name: Cache node modules
uses: actions/cache@v1
with:
path: ~/.npm
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: ${{ runner.os }}-node-
- run: npm ci
- run: npm test
ughh, I never remember these
const hasha = require('hasha')
// we are using dependency
// "cache": "github:cypress-io/github-actions-cache#8bec6cc"
const { restoreCache, saveCache } = require('cache/lib/index')
const useYarn = fs.existsSync('yarn.lock')
const lockFilename = useYarn ? 'yarn.lock' : 'package-lock.json'
const lockHash = hasha.fromFileSync(lockFilename)
const platformAndArch = `${process.platform}-${process.arch}`
// this is simplified code for clarity
// see action file index.js for full details
const NPM_CACHE = {
inputPath: '~/.npm', // or '~/.cache/yarn'
primaryKey: `npm-${platformAndArch}-${lockHash}`,
restoreKeys: `npm-${platformAndArch}-`
}
const restoreCachedNpm = () => {
console.log('trying to restore cached NPM modules')
return restoreCache(
NPM_CACHE.inputPath,
NPM_CACHE.primaryKey,
NPM_CACHE.restoreKeys
)
}
const saveCachedNpm = () => {
console.log('saving NPM modules')
return saveCache(NPM_CACHE.inputPath, NPM_CACHE.primaryKey)
}
restoreCachedNpm()
.then(npmCacheHit => {
console.log('npm cache hit', npmCacheHit)
return install().then(() => {
if (npmCacheHit) {
return
}
return saveCachedNpm()
})
})
.catch(error => {
console.log(error)
core.setFailed(error.message)
})
- uses: actions/checkout@v1
- uses: bahmutov/npm-install@v1
- run: npm test
name: End-to-end tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
(does npm install and caching, runs the tests correctly, can build app and start server and more)
name: Chome tests
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
browser: chrome
Use parameters to control behavior
name: Record test results
on: [push]
jobs:
cypress-run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
name: Parallel tests
on: [push]
jobs:
cypress-run:
strategy:
matrix:
machines: [1, 2, 3]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
parallel-runs-across-platforms:
strategy:
matrix:
os: ['ubuntu-latest', 'windows-latest', 'macos-latest']
machines: [1, 2]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v1
- uses: cypress-io/github-action@v1
with:
record: true
parallel: true
group: Parallel 2x on ${{ matrix.os }}
env:
CYPRESS_RECORD_KEY: ${{ secrets.cy_token }}
recorded parallel runs
click DOWN arrow to navigate these slides
// smoke.js
export const smokeTest = () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
// click through main app features
...
}
// smoke-spec.js
import { smokeTest } from './smoke'
it('does not smoke', () => {
smokeTest()
})
// viewports-spec.js
import { smokeTest } from './smoke'
Cypress._.each(['macbook-15', 'iphone-6'],
viewport => {
it(`works on ${viewport}`, () => {
cy.viewport(viewport)
smokeTest()
})
})
# CircleCI config
- cypress/run:
name: smoke tests
filters:
branches:
only:
- master
start: npm start
command: |
npx cypress run --config \
baseUrl=http://localhost:1234,testFiles=smoke-spec.js
CircleCI config using https://github.com/cypress-io/circleci-orb
# CircleCI config
- cypress/run:
name: smoke tests
filters:
branches:
only:
- master
start: npm start
command: |
npx cypress run \
--config-file cypress-smoke.json
CircleCI config using https://github.com/cypress-io/circleci-orb
{
"baseUrl": "http://localhost:1234",
"testFiles": [
"smoke-spec.js"
]
}
cypress-smoke.json
it('works differently', () => {
if (Cypress.browser.family === 'electron') {
...
}
if (Cypress.platform === 'darwin') {
...
}
// pass custom env variables
// CYPRESS_ENVIRONMENT=... cypress run
if (Cypress.env('ENVIRONMENT') === 'staging') {
...
}
})
import { onlyOn, skipOn } from '@cypress/skip-test'
onlyOn('mac', () => {
// this callback will only evaluate on Mac
// thus the tests will be completely hidden from other platforms
describe('Mac tests', () => {
it('works', () => {})
})
})
skipOn('mac', () => {
// this test will run on every platform but Mac
it('hides this test on Mac', () => {})
})
import { onlyOn, skipOn } from '@cypress/skip-test'
skipOn('firefox', () => {
it('does not work on FF yet', () => {
...
})
})
import {onlyOn} from '@cypress/skip-test'
const stubServer = () => {
cy.server()
cy.route('/api/me', 'fx:me.json')
cy.route('/api/permissions', 'fx:permissions.json')
// Lots of fixtures ...
}
it('works', () => {
onlyOn('staging', stubServer)
cy.visit('/')
cy.get('.user-profile').should('be.visible')
...
})
when testing "staging" should stub network calls