Cypress.io
When testing is easy, developers build better things faster and with confidence.
@bahmutov
@amirrustam
Ask your question and up-vote questions at: https://www.sli.do/
Because writing tests is easy
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
typical Todo application http://todomvc.com/
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })
it('completes todos', () => {
... })
Ohhh, we don't have a test for Feature C
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Requires really good and up-to-date test case management system
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
green: lines executed during the tests
red: lines NOT executed during the tests
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Code coverage from tests indirectly
measures implemented features tested
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })
it('completes todos', () => {
... })
it('deletes todos', () => {
... })
Unrealistic tests; subset of inputs
code does not implement the feature correctly
console.log('a')
if (true) {
console.log('b')
} else {
console.log('c')
}
console.log('d')
$ node .
a
b
d
run it
what does this code do?
console.log('a')
if (true) {
console.log('b')
} else {
console.log('c')
}
console.log('d')
$ node .
a
b
d
which statements did the code execute?
run it
// 5 statements
const s = [0, 0, 0, 0, 0]
s[0]++; console.log('a')
s[1]++; if (true) {
s[2]++; console.log('b')
} else {
s[3]++; console.log('c')
}
s[4]++; console.log('d')
console.log(s)
$ node ./instrumented.js
a
b
d
[ 1, 1, 1, 0, 1 ]
run it
Can you prove it?
$ node ./instrumented.js
a
b
d
[ 1, 1, 1, 0, 1 ]
coverage report:
Statements 80% (4/5)
Branches 50% (1/2)
Function N/A
$ node ./instrumented.js
a
b
d
[ 1, 1, 1, 0, 1 ]
<div class="covered">console.log('a')</div>
<div class="covered">if (true) {</div>
<div class="covered"> console.log('b')</div>
<div>} else {</div>
<div class="not-covered"> console.log('c')</div>
<div>}</div>
<div class="covered">console.log('d')</div>
0
1
1
1
1
console.log('a')
if (true) {
console.log('b')
} else {
console.log('c')
}
console.log('d')
$ npm install -D nyc
$ open coverage/index.html
console.log('a')
if (true) {
console.log('b')
} else {
console.log('c')
}
console.log('d')
$ npx nyc instrument index.js
var cov_afcvt8rcn=function(){var path="/Users/gleb/git/training/javascript/instrument-code/index.js";var hash="bc50734d55b896e9ec221181b179e6ded88a3baa";var global=new Function("return this")();var gcv="__coverage__";var coverageData={path:"/Users/gleb/git/training/javascript/instrument-code/index.js",statementMap:{"0":{start:{line:1,column:0},end:{line:1,column:16}},"1":{start:{line:2,column:0},end:{line:6,column:1}},"2":{start:{line:3,column:2},end:{line:3,column:18}},"3":{start:{line:5,column:2},end:{line:5,column:18}},"4":{start:{line:7,column:0},end:{line:7,column:16}}},fnMap:{},branchMap:{"0":{loc:{start:{line:2,column:0},end:{line:6,column:1}},type:"if",locations:[{start:{line:2,column:0},end:{line:6,column:1}},{start:{line:2,column:0},end:{line:6,column:1}}],line:2}},s:{"0":0,"1":0,"2":0,"3":0,"4":0},f:{},b:{"0":[0,0]},_coverageSchema:"43e27e138ebf9cfc5966b082cf9a028302ed4184",hash:"bc50734d55b896e9ec221181b179e6ded88a3baa"};var coverage=global[gcv]||(global[gcv]={});if(coverage[path]&&coverage[path].hash===hash){return coverage[path];}return coverage[path]=coverageData;}();cov_afcvt8rcn.s[0]++;console.log('a');cov_afcvt8rcn.s[1]++;if(true){cov_afcvt8rcn.b[0][0]++;cov_afcvt8rcn.s[2]++;console.log('b');}else{cov_afcvt8rcn.b[0][1]++;cov_afcvt8rcn.s[3]++;console.log('c');}cov_afcvt8rcn.s[4]++;console.log('d');
Global coverage variable __coverage__.
If you see this variable - the code was instrumented by Istanbul
Q & A sli.do event #cycc
{
"plugins": ["istanbul"]
}
.babelrc
{
"env": {
"test": {
"plugins": ["istanbul"]
}
}
}
.babelrc
NODE_ENV=test npm start
{
"presets": ["@babel/preset-react"],
"env": {
"test": {
"plugins": ["istanbul"]
}
},
"plugins": ["transform-class-properties"]
}
.babelrc
If you use something else - there is probably an Istanbul plugin or loader
follow issue https://github.com/cypress-io/cypress/issues/687
down arrow: slides for TodoMVC demo
Global variable __coverage__.
If you see this variable - the code was instrumented by Istanbul
$ npm run dev
During end-to-end tests these counters will be incremented
$ npm run dev
How do we save the __coverage__ object after the tests and generate reports?
@cypress/code-coverage plugin
$ npm run dev
@cypress/code-coverage plugin
Saves __coverage__ from Cypress
end-to-end and unit tests
$ npm i -D @cypress/code-coverage nyc istanbul-lib-coverage
import '@cypress/code-coverage/support'
cypress/support/index.js
module.exports = (on, config) => {
on('task', require('@cypress/code-coverage/task'))
}
cypress/plugins/index.js
down arrow: slides for coverage commands and reports
@cypress/code-coverage plugin keeps track of the coverage results
Send LCOV or Clover reports to 3rd party code coverage services, like Coveralls, Codecov, etc
Istanbul full JSON report, use to merge with other Istanbul reports https://github.com/bahmutov/cypress-and-jest
Static HTML report. Look at it
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
down arrow: slides for coverage from this one test
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
We have tested "add todo"
Need tests
What is this code doing?
Can I write an E2E test to hit this code?
This code should be unreachable from the user interface
Unit test with (Jest | Ava | Mocha)
Save unit test coverage
End-to-end tests with Cypress
Save E2E test coverage
+ with "nyc merge"
Combined Code Coverage
Unit test and End-to-end tests with Cypress
Combined Code Coverage
module.exports = (on, config) => {
on('task', require('@cypress/code-coverage/task'))
on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'))
}
cypress/plugins/index.js
Instruments your spec and unit test code
down arrow: slides for unit testing TodoMVC demo
import {getVisibleTodos} from '../../src/selectors'
describe('getVisibleTodos', () => {
it('throws an error for unknown visibility filter', () => {
expect(() => {
getVisibleTodos({
todos: [],
visibilityFilter: 'unknown-filter'
})
}).to.throw()
})
})
@cypress/code-coverage plugin
We got this line from the unit test
@cypress/code-coverage plugin
Nice job, @cypress/code-coverage plugin
Q & A sli.do event #cycc
# store HTML coverage folder
- store_artifacts:
path: coverage/lcov-report
# store test run video
- store_artifacts:
path: cypress/videos
- store_artifacts:
path: cypress/screenshots
# store HTML coverage folder
- store_artifacts:
path: coverage/lcov-report
# store test run video
- store_artifacts:
path: cypress/videos
- store_artifacts:
path: cypress/screenshots
# load balance tests on 4 machines
# https://on.cypress.io/parallelization
- cypress/run:
record: true
parallel: true
parallelism: 4
Each of the 4 CI machines only runs a subset of the spec files; generates partial code coverage
npm install -D coveralls
# send coverage report to Coveralls.io
# COVERALLS_REPO_TOKEN=...
cat lcov.info | coveralls
collects and merges results 🤞
shows coverage information see example
adds checks on pull requests
Pull request has same or higher code coverage: excellent!
Why?
{
"config": {
"pre-git": {
"pre-commit": ["npx stop-only --warn --folder cypress"],
"pre-push": ["npx stop-only --folder cypress"]
}
}
}
Front end
NodeJS server
Collect code coverage
Q & A sli.do event #cycc
{
"scripts": {
"start": "node server",
"start:coverage": "nyc --silent node server"
}
}
{
"scripts": {
"start": "node server",
"start:coverage": "nyc --silent node server"
}
}
const express = require('express')
const app = express()
if (global.__coverage__) {
require('@cypress/code-coverage/middleware/express')(app)
}
@cypress/code-coverage plugin
down arrow: slides for full stack realworld demo
server code
Help us with your code coverage examples:
By Cypress.io
Link to webcast: https://www.youtube.com/watch?v=C8g5X4vCZJA As you write more and more end-to-end tests, you probably find yourself wondering - do I need to write more tests? Are there parts of the application still untested? Are you running redundant tests that are wasting resources and time? This presentation shows how to achieve full code coverage using Cypress Test Runner through a combination of end-to-end and unit tests.
When testing is easy, developers build better things faster and with confidence.