Gleb Bahmutov

VP of Engineering

Cypress.io

Achievement Unlocked: Drive development, increase velocity, and write blissful tests with Cypress

AllThingsOpen 2019 logo

use up/down and left/right keys to navigate these slides

our planet is in imminent danger

we have to act today

ME

you

you

you

you

vote & lobby & rebel

if I can help I will

@bahmutov

gleb.bahmutov@gmail.com

If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.

Take a deep breath

30 people. Atlanta, Philly, Boston, NYC, the World

Fast, easy and reliable testing for anything that runs in a browser

it('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

typical Cypress end-to-end test

$ cypress open

"Interactive Mode"

  • Runs one or all specs
  • File watcher
  • DOM snapshots and time-traveling debugger

$ cypress run

"Headless Mode"

  • Runs single spec at a time
  • No file watching
  • No DOM snapshots
  • automatic screenshot on failure
  • automatic video

Cypress is going open source!

repo made public

Oct 2017

❤️ Hundreds of tweets like this one ❤️

  1. Build the best end-to-end test runner

  2. Teach developers how to write good end-to-end tests

  3. Teach developers how to run end-to-end tests on CI

Cypress.io Goals

This Presentation

  • How I think about testing

  • How we prosper as a company built around testing tools

  • What we are working on

Dr. Gleb Bahmutov, PhD

these slides

I test because

I doubt myself

E2E

integration

unit

Testing Pyramid △

Code

Test

API

Test

Web app

Test

There is only a test appropriate for the thing you doubt works

Thing is a function

const add = (a, b) => a + b
it('adds numbers', () => {
  expect(add(2, 3)).to.equal(5)
})

you write a test

Thing is a component

import { HelloState } from '../src/hello-x.jsx'
import { HelloState } from '../src/hello-x.jsx'
import React from 'react'
it('changes state', () => {
  cy.mount(<HelloState />)
  cy.contains('Hello Spider-man!')
  const stateToSet = { name: 'React' }
  cy.get(HelloState).invoke('setState', stateToSet)
  cy.get(HelloState)
    .its('state')
    .should('deep.equal', stateToSet)
  cy.contains('Hello React!')
})

you write a test

Thing is an API

it('yields result that has log messages', () => {
  cy.api({ url: '/' }, 'hello world')
  .then(({ messages }) => {
    const logs = Cypress._.filter(messages, {
      type: 'console',
      namespace: 'log'
    })
    expect(logs, '1 console.log message').to.have.length(1)
    expect(logs[0]).to.deep.include({
      type: 'console',
      namespace: 'log',
      message: 'processing GET /'
    })
  })
})

you write a test

Thing is an API

you write a test

Thing is a webapp

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

you write a test

Thing is a webapp

you write a test

Thing is a webapp's style

it('draws pizza correctly', function () {
  cy.percySnapshot('Empty Pizza')

  cy.enterDeliveryInformation()
  const toppings = ['Pepperoni', 'Chili', 'Onion']
  cy.pickToppings(...toppings)

  // make sure the web app has updated
  cy.contains('.pizza-summary__total-price', 'Total: $12.06')
  cy.percySnapshot(toppings.join(' - '))
})

you write a test

Thing is a webapp's style

you write a test

<style type="text/css">
  .st0{fill:#FFD8A1;}
  .st1{fill:#E8C08A;}
- .st2{fill:#FFDC71;}
+ .st2{fill:#71FF71;}
  .st3{fill:#DFBA86;}
</style>

changes crust color SVG

Thing is a webapp's style

you write a test

Thing is an Electron app

// main.js
const { app } = require('electron')
const MainBrowserWindow = require('./main_browser_window')
let win
function createWindow () {
  win = MainBrowserWindow()
  win.loadURL('http://localhost:4600')
}

app.on('ready', createWindow)

Thing is an Electron app

you write a test

// cypress/integration/spec.js
it('clicks', () => {
  // window creation and url load
  cy.electronVisitUrl('./main_browser_window.js', 
                       'http://localhost:4600')
  cy.get('button')
    .click()
    .click()
  cy.get('#clicked').should('have.text', '2')
})

Thing is an Electron app

you write a test

Thing is accessability

it('has good contrast', () => {
  cy.visit('/')
  cy.injectAxe()
  cy.checkA11y({
    runOnly: ['cat.color'],
  })
})

you write a test

You can always write a test using the right tool

75 Cypress plugins

  • Framework tooling
  • Custom commands
  • Development tools
  • Visual testing
  • Preprocessors
  • Reporters
  • Component tests
  • Authentication

What should I test?

@cypress/code-coverage plugin

example

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

typical Todo application http://todomvc.com/

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Ohhh, we don't have a test for Feature C

Keeping track manually

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track manually

1. Carefully collect user stories and feature requirments

2. Write matching tests

3. Keep updating user stories and tests to keep them in sync

HARD

  • first test
  • two more tests
  • hundreds of tests!!!

Keep updating user stories and tests to keep them in sync

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })

green: lines executed during the tests

red: lines NOT executed during the tests

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Keeping track via code coverage

Are we testing all features?

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Code coverage from tests indirectly

measures implemented features tested

Keeping track via code coverage

⚠️ 100% Code Coverage ≠ 0🐞

Feature A

User can add todo items

Feature B

User can complete todo items

Feature C

User can delete todo items

❚❚❚❚❚
  ❚❚❚❚❚ ❚❚ ❚❚❚❚
  ❚❚❚
❚❚❚❚

❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
  ❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚
  ❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚

source code

it('adds todos', () => { ... })
it('completes todos', () => { 
  ... })
it('deletes todos', () => { 
  ... })

Unrealistic tests; subset of inputs

code does not implement the feature correctly

Code Coverage with @cypress/code-coverage

  1. Instrument code (YOU)

  2. Run tests

  3. Report results

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

E2E tests are extremely effective at covering a lot of app code

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

Coverage as a guide to writing E2E tests

it('adds todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)
})

We have tested "add todo"

Need tests

Write more end-to-end tests

Ughh, missed it

Can we always write an E2E test?

What is this code doing?

Can I write an E2E test to hit this code?

This code should be unreachable from the user interface

Write a Unit Test!

Can we always write an E2E test?

import {getVisibleTodos} from '../../src/selectors'

describe('getVisibleTodos', () => {
  it('throws an error for unknown visibility filter', () => {
    expect(() => {
      getVisibleTodos({
        todos: [],
        visibilityFilter: 'unknown-filter'
      })
    }).to.throw()
  })
})

@cypress/code-coverage plugin

We got this line from the unit test

@cypress/code-coverage plugin

Nice job, @cypress/code-coverage plugin

combines e2e and unit test coverage automatically

Keep the core small

iterate quickly via plugins

E2E tests are extremely effective at covering your

  • application code

  • configuration

  • entire stack

E2E

integration

unit

Really important to users

Really important to developers

Tutorials

Api

Examples

Let's talk money

How does Cypress.io make money?

Cypress Test Runner - OSS 100% forever

Cypress Dashboard paid SaaS for companies

Paid Features 💵

  • recording test artifacts
  • test parallelization
  • GitHub integration
  • (WIP) historical insights

Modern CI UI is ok for console log

Paid Features 💵

$ npx cypress run --record

Upload test artifacts from any CI

Paid Features 💵: artifacts

test output, video, screenshots

Test results view 1 / 3

Test results view 2 / 3

Test results view 3 / 3

WIP

I have 100s of tests ...

$ npx cypress run --record --parallel

Cypress v3.1.0

Spin N CI machines and

Text

cypress-dashboard parallelization

cypress-dashboard parallelization

# machines run duration time savings
1 22:50 ~
2 11:47 48%
3 7:51 65%
4 5:56 74%
6 3:50 83%
8 3:00 87%
10 2:19 90%

cypress-dashboard parallelization

10 machines = 10x speed up

We need to work as a team

Paid Features 💵: GH integration

https://www.cypress.io/blog/2019/07/29/github-integration-for-the-cypress-dashboard/

GitHub Integration 1 / 6

GitHub Integration 2 / 6

GitHub Integration 3 / 6

GitHub Integration 4 / 6

GitHub Integration 5 / 6

By Spec

By Run Group

GitHub Integration 6 / 6

PR Comments

Cypress & Open Source Community

This dark red color looks familiar ...

Nice contrast on the footer labels

todomvc-app-css@2.3.0

todomvc-app-css@2.2.0

Oh yeah, it was my PR!

cypress-axe plugin

by Andrew van Slaars

@avanslaars

it('has good contrast when empty', () => {
  cy.visit('/')
  cy.injectAxe()
  cy.checkA11y({
    runOnly: ['cat.color'],
  })
})
it('has good contrast with several todos', () => {
  cy.visit('/')
  cy.injectAxe()
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')

  cy.get('.todo-list li').should('have.length', 2)
  cy.checkA11y({
    runOnly: ['cat.color'],
  })
})

uses

Cypress tests

use

Cypress tests

uses

Cypress tests

In 2 years since Cypress went OSS it has become a popular and widely used testing tool

Thank you ❤️

Introducing the Open Source and Full-Featured Seed Dashboard Plans

62 organizations on OSS plan

What we are working on

1. Dashboard

2. Test Runner

Test analytics 1 / 4

Test analytics 2 / 4

Test analytics 3 / 4

Test analytics 4 / 4

Run Tagging

cypress run --tag develop
cypress run --tag staging
cypress run --tag production
cypress run --tag nightly
cypress run --tag hourly
cypress run --group desktop-uk --tag develop,nightly
cypress run --group desktop-de --tag production

Cypress Dashboard Features

What we are working on

1. Dashboard

2. Test Runner

Firefox support

failing 15 of 3000 tests

Better errors

Better errors

Source of command errors unclear

No full diffs

Better errors

App Errors

Assertions

Code Frames

Better errors

  • Error explanation
  • Command doc link
  • Precise source maps

Better errors

Open your text editor right at the error location

PR #4041

PR #3930

Test Retries 1 / 2

Test Retries 2 / 2

PR #3968

Click on the failed test attempt to see the command log

WIP

  • Dashboard
    • Test analytics
    • Better error views
  • Test Runner
    • Firefox support
    • Much better errors
    • Test retries

Conclusions

Give Cypress a try

Conclusions

Free Dashboard plan for OSS projects

Conclusions

Future is bright

Angry or happy? Let me know

@bahmutov     @cypress_io

submit feedback via ATO app

Thank you 👏

Achievement Unlocked: Drive development, increase velocity, and write blissful tests with Cypress

AllThingsOpen 2019 logo

Achievement Unlocked: All Things Open Conference

By Cypress.io

Achievement Unlocked: All Things Open Conference

The demand for developer productivity has never been greater. We are expected to continuously improve and release code, often several times a day. But how do we achieve this kind of velocity without introducing bugs? How can our codebase adapt to changing business requirements, while still keeping technical debt to a minimum? Video at https://www.youtube.com/watch?v=2gP1-TNDzK4

  • 1,092
Loading comments...

More from Cypress.io