Complete Code Coverage

with

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Amir Rustamzadeh

Sr. Engineer

@amirrustam

Contents

  • Why do we need code coverage (CC)?
  • CC in JavaScript world
  • CC for end-to-end tests
    • cypress-io/cypress-example-todomvc-redux
  • ​CC for unit tests
    • bahmutov/cypress-and-jest
    • cypress-io/cypress-example-todomvc-redux
  • Full stack CC
    • cypress-io/cypress-example-realworld
  • ​Q & A sli.do event #cycc

Q & A

Ask your question and up-vote questions at: https://www.sli.do/

Event code: #cycc

Why code coverage?

Because writing tests is easy

  • first Cypress test
  • two more Cypress tests
  • hundreds of Cypress tests!!!

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

typical Todo application http://todomvc.com/

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Ohhh, we don't have a test for Feature C

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Requires really good and up-to-date test case management system

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

green: lines executed during the tests

red: lines NOT executed during the tests

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Code coverage from tests indirectly

measures implemented features tested

Keeping track via code coverage

⚠️ 100% Code Coverage ≠ 0🐞

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Unrealistic tests; subset of inputs

code does not implement the feature correctly

JavaScript Code Coverage

  1. Instrument code

  2. Run tests

  3. Report results

console.log('a')
if (true) {
  console.log('b')
} else {
  console.log('c')
}
console.log('d')
$ node .
a
b
d

run it

what does this code do?

console.log('a')
if (true) {
  console.log('b')
} else {
  console.log('c')
}
console.log('d')
$ node .
a
b
d

which statements did the code execute?

run it 

// 5 statements
const s = [0, 0, 0, 0, 0]
s[0]++; console.log('a')
s[1]++; if (true) {
  s[2]++; console.log('b')
} else {
  s[3]++; console.log('c')
}
s[4]++; console.log('d')
console.log(s)
$ node ./instrumented.js 
a
b
d
[ 1, 1, 1, 0, 1 ]

run it

Can you prove it?

$ node ./instrumented.js 
a
b
d
[ 1, 1, 1, 0, 1 ]
coverage report:
  Statements 80% (4/5)
  Branches 50% (1/2)
  Function N/A

Report results

$ node ./instrumented.js 
a
b
d
[ 1, 1, 1, 0, 1 ]
<div class="covered">console.log('a')</div>
<div class="covered">if (true) {</div>
<div class="covered">  console.log('b')</div>
<div>} else {</div>
<div class="not-covered">  console.log('c')</div>
<div>}</div>
<div class="covered">console.log('d')</div>

0

1

1

1

1

Istanbul.js - code instrumentation library

Istanbul.js - code instrumentation library

nyc - CLI around Istanbul.js

console.log('a')
if (true) {
  console.log('b')
} else {
  console.log('c')
}
console.log('d')
$ npm install -D nyc
$ open coverage/index.html
console.log('a')
if (true) {
  console.log('b')
} else {
  console.log('c')
}
console.log('d')
$ npx nyc instrument index.js
var cov_afcvt8rcn=function(){var path="/Users/gleb/git/training/javascript/instrument-code/index.js";var hash="bc50734d55b896e9ec221181b179e6ded88a3baa";var global=new Function("return this")();var gcv="__coverage__";var coverageData={path:"/Users/gleb/git/training/javascript/instrument-code/index.js",statementMap:{"0":{start:{line:1,column:0},end:{line:1,column:16}},"1":{start:{line:2,column:0},end:{line:6,column:1}},"2":{start:{line:3,column:2},end:{line:3,column:18}},"3":{start:{line:5,column:2},end:{line:5,column:18}},"4":{start:{line:7,column:0},end:{line:7,column:16}}},fnMap:{},branchMap:{"0":{loc:{start:{line:2,column:0},end:{line:6,column:1}},type:"if",locations:[{start:{line:2,column:0},end:{line:6,column:1}},{start:{line:2,column:0},end:{line:6,column:1}}],line:2}},s:{"0":0,"1":0,"2":0,"3":0,"4":0},f:{},b:{"0":[0,0]},_coverageSchema:"43e27e138ebf9cfc5966b082cf9a028302ed4184",hash:"bc50734d55b896e9ec221181b179e6ded88a3baa"};var coverage=global[gcv]||(global[gcv]={});if(coverage[path]&&coverage[path].hash===hash){return coverage[path];}return coverage[path]=coverageData;}();cov_afcvt8rcn.s[0]++;console.log('a');cov_afcvt8rcn.s[1]++;if(true){cov_afcvt8rcn.b[0][0]++;cov_afcvt8rcn.s[2]++;console.log('b');}else{cov_afcvt8rcn.b[0][1]++;cov_afcvt8rcn.s[3]++;console.log('c');}cov_afcvt8rcn.s[4]++;console.log('d');

Global coverage variable __coverage__.

If you see this variable - the code was instrumented by Istanbul

CC from End-to-end Tests

  • Instrument web application code

  • Run Cypress tests

  • Save collected code coverage information

Q & A sli.do event #cycc

Instrument code:

babel-plugin-istanbul

{
  "plugins": ["istanbul"]
}

.babelrc

{
  "env": {
    "test": {
      "plugins": ["istanbul"]
    }
  }
}

.babelrc

Instrument code:

babel-plugin-istanbul

NODE_ENV=test npm start

https://github.com/cypress-io/cypress-example-todomvc-redux

{
  "presets": ["@babel/preset-react"],
  "env": {
    "test": {
      "plugins": ["istanbul"]
    }
  },
  "plugins": ["transform-class-properties"]
}

.babelrc

Instrument code:

babel-plugin-istanbul

Babel + Parcel.js

Babel + Webpack

Babel + Browserify

Babel + Rollup.js

babel-plugin-istanbul 

should work with any Babel

If you use something else - there is probably an Istanbul plugin or loader

https://on.cypress.io/code-coverage#Examples

No-config code instrumentation is coming ...

follow issue https://github.com/cypress-io/cypress/issues/687

Full Network Layer Stubbing

down arrow: slides for TodoMVC demo

Global variable __coverage__.

If you see this variable - the code was instrumented by Istanbul

https://github.com/cypress-io/cypress-example-todomvc-redux

$ npm run dev

During end-to-end tests these counters will be incremented

https://github.com/cypress-io/cypress-example-todomvc-redux

$ npm run dev

How do we save the __coverage__ object after the tests and generate reports?

@cypress/code-coverage plugin

https://github.com/cypress-io/cypress-example-todomvc-redux

$ npm run dev

@cypress/code-coverage plugin

Saves __coverage__ from Cypress

end-to-end and unit tests

$ npm i -D @cypress/code-coverage nyc istanbul-lib-coverage
import '@cypress/code-coverage/support'

cypress/support/index.js

module.exports = (on, config) => {
  on('task', require('@cypress/code-coverage/task'))
}

cypress/plugins/index.js

down arrow: slides for coverage commands and reports

@cypress/code-coverage plugin keeps track of the coverage results

Automatic reports

Send LCOV or Clover reports to 3rd party code coverage services, like Coveralls, Codecov, etc

Istanbul full JSON report, use to merge with other Istanbul reports https://github.com/bahmutov/cypress-and-jest

Static HTML report. Look at it

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

down arrow: slides for coverage from this one test

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

We have tested "add todo"

Need tests

https://github.com/cypress-io/cypress-example-todomvc-redux

Chasing missed lines

What is this code doing?

Can I write an E2E test to hit this code?

Sometimes you cannot

This code should be unreachable from the user interface

https://github.com/cypress-io/cypress-example-todomvc-redux

Write a Unit Test!

Contents

  • Why do we need code coverage (CC)?
  • CC in JavaScript world
  • CC for end-to-end tests
    • cypress-io/cypress-example-todomvc-redux
  • ​CC for unit tests
    • cypress-io/cypress-example-todomvc-redux
    • bahmutov/cypress-and-jest
  • Full stack CC
    • cypress-io/cypress-example-realworld
  • ​Q & A sli.do event #cycc 

Unit + E2E coverage:

The hard way

Unit test with (Jest | Ava | Mocha)

Save unit test coverage

End-to-end tests with Cypress

Save E2E test coverage

+ with "nyc merge"

Combined Code Coverage

https://github.com/bahmutov/cypress-and-jest

Unit + E2E coverage:

The Cypress way

Unit test and End-to-end tests with Cypress

Combined Code Coverage

module.exports = (on, config) => {
  on('task', require('@cypress/code-coverage/task'))
  on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'))
}

cypress/plugins/index.js

Instruments your spec and unit test code

down arrow: slides for unit testing TodoMVC demo

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

@cypress/code-coverage plugin

We got this line from the unit test

@cypress/code-coverage plugin

Nice job, @cypress/code-coverage plugin

Coverage as guard

Fail build if coverage drops below N%

Q & A sli.do event #cycc

Store HTML report on CI

# store HTML coverage folder
- store_artifacts:
   path: coverage/lcov-report
# store test run video 
- store_artifacts:
   path: cypress/videos
- store_artifacts:
   path: cypress/screenshots

https://circleci.com/gh/cypress-io/cypress-example-todomvc-redux/39#artifacts/containers/0

Store HTML report on CI

# store HTML coverage folder
- store_artifacts:
   path: coverage/lcov-report
# store test run video 
- store_artifacts:
   path: cypress/videos
- store_artifacts:
   path: cypress/screenshots

https://circleci.com/gh/cypress-io/cypress-example-todomvc-redux/39#artifacts/containers/0

Coverage as a Service

https://github.com/cypress-io/cypress-example-realworld

# load balance tests on 4 machines
# https://on.cypress.io/parallelization
- cypress/run:
    record: true
    parallel: true
    parallelism: 4

Each of the 4 CI machines only runs a subset of the spec files; generates partial code coverage

🤔

Coveralls, Codecov, etc

https://github.com/cypress-io/cypress-example-realworld

npm install -D coveralls
# send coverage report to Coveralls.io
# COVERALLS_REPO_TOKEN=...
cat lcov.info | coveralls

3rd party coverage service

  • collects and merges results 🤞

  • shows coverage information see example

  • adds checks on pull requests

👍

Coveralls, Codecov, etc

https://github.com/cypress-io/cypress-example-realworld/pull/49

Pull request has same or higher code coverage: excellent!

Coveralls, Codecov, etc

https://github.com/cypress-io/cypress-example-todomvc-redux/pull/14

Why?

Tip: catch exclusive tests

https://github.com/bahmutov/stop-only

{
  "config": {
    "pre-git": {
      "pre-commit": ["npx stop-only --warn --folder cypress"],
      "pre-push": ["npx stop-only --folder cypress"]
    }
  }
}

Badges

Confidence

Full Stack Code Coverage

Front end

NodeJS server

Collect code coverage

Q & A sli.do event #cycc

{
  "scripts": {
    "start": "node server",
    "start:coverage": "nyc --silent node server"
  }
}

Instrument Server Code

{
  "scripts": {
    "start": "node server",
    "start:coverage": "nyc --silent node server"
  }
}

Instrument Server Code

const express = require('express')
const app = express()
if (global.__coverage__) {
  require('@cypress/code-coverage/middleware/express')(app)
}

@cypress/code-coverage plugin

down arrow: slides for full stack realworld demo

server code

https://github.com/cypress-io/cypress-example-realworld

E2E tests are extremely effective at covering a lot of app code

More information

More examples

🔗 on.cypress.io/code-coverage#Examples

Help us with your code coverage examples:

  • Angular + TypeScript
  • Svelte
  • Rollup.js

Q & A

  • Why do we need code coverage (CC)?
  • CC in JavaScript world
  • CC for end-to-end tests
  • ​CC for unit tests
  • Full stack CC
  • ​Q & A use sli.do event #cycc 

🔗 on.cypress.io/code-coverage     @cypress_io

Complete Code Coverage with Cypress

By Cypress.io

Complete Code Coverage with Cypress

Link to webcast: https://www.youtube.com/watch?v=C8g5X4vCZJA As you write more and more end-to-end tests, you probably find yourself wondering - do I need to write more tests? Are there parts of the application still untested? Are you running redundant tests that are wasting resources and time? This presentation shows how to achieve full code coverage using Cypress Test Runner through a combination of end-to-end and unit tests.

  • 6,439

More from Cypress.io