Cypress.io
When testing is easy, developers build better things faster and with confidence.
Gleb Bahmutov
VP of Engineering
Cypress.io
Achievement Unlocked: Drive development, increase velocity, and write blissful tests with Cypress
use up/down and left/right keys to navigate these slides
our planet is in imminent danger
https://lizkeogh.com/2019/07/02/off-the-charts/
https://gleb.dev/blog/climate-emergency
+3 degrees Celsius will be the end.
we have to act today
ME
you
you
you
you
vote & lobby & rebel
if I can help I will
@bahmutov
gleb.bahmutov@gmail.com
If there is a company that fights global climate catastrophe and needs JavaScript and testing skills - I will do for free.
Example: https://fab.earth
Take a deep breath
30 people. Atlanta, Philly, Boston, NYC, the World
Fast, easy and reliable testing for anything that runs in a browser
it('adds 2 todos', () => {
cy.visit('http://localhost:3000')
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li')
.should('have.length', 2)
})typical Cypress end-to-end test
$ cypress open
"Interactive Mode"
- Runs one or all specs
- File watcher
- DOM snapshots and time-traveling debugger
$ cypress run
"Headless Mode"
- Runs single spec at a time
- No file watching
- No DOM snapshots
- automatic screenshot on failure
- automatic video
Cypress is going open source!
repo made public
Oct 2017
❤️ Hundreds of tweets like this one ❤️
-
Build the best end-to-end test runner
-
Teach developers how to write good end-to-end tests
-
Teach developers how to run end-to-end tests on CI
Cypress.io Goals
This Presentation
-
How I think about testing
-
How we prosper as a company built around testing tools
-
What we are working on
- Hundreds of OSS projects at gleb.dev
- Hundreds of blog posts at gleb.dev/blog
- @bahmutov
- cypress.slides.com/cypress-io/achievement-unlocked
Dr. Gleb Bahmutov, PhD
these slides
I test because
I doubt myself
E2E
integration
unit
Testing Pyramid △
Code
Test
API
Test
Web app
Test
There is only a test appropriate for the thing you doubt works
Thing is a function
const add = (a, b) => a + bit('adds numbers', () => {
expect(add(2, 3)).to.equal(5)
})you write a test
Thing is a component
import { HelloState } from '../src/hello-x.jsx'import { HelloState } from '../src/hello-x.jsx'
import React from 'react'
it('changes state', () => {
cy.mount(<HelloState />)
cy.contains('Hello Spider-man!')
const stateToSet = { name: 'React' }
cy.get(HelloState).invoke('setState', stateToSet)
cy.get(HelloState)
.its('state')
.should('deep.equal', stateToSet)
cy.contains('Hello React!')
})you write a test
Thing is an API
it('yields result that has log messages', () => {
cy.api({ url: '/' }, 'hello world')
.then(({ messages }) => {
const logs = Cypress._.filter(messages, {
type: 'console',
namespace: 'log'
})
expect(logs, '1 console.log message').to.have.length(1)
expect(logs[0]).to.deep.include({
type: 'console',
namespace: 'log',
message: 'processing GET /'
})
})
})you write a test
Thing is an API
you write a test
Thing is a webapp
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})you write a test
Thing is a webapp
you write a test
Thing is a webapp's style
it('draws pizza correctly', function () {
cy.percySnapshot('Empty Pizza')
cy.enterDeliveryInformation()
const toppings = ['Pepperoni', 'Chili', 'Onion']
cy.pickToppings(...toppings)
// make sure the web app has updated
cy.contains('.pizza-summary__total-price', 'Total: $12.06')
cy.percySnapshot(toppings.join(' - '))
})you write a test
Thing is a webapp's style
you write a test
<style type="text/css">
.st0{fill:#FFD8A1;}
.st1{fill:#E8C08A;}
- .st2{fill:#FFDC71;}
+ .st2{fill:#71FF71;}
.st3{fill:#DFBA86;}
</style>changes crust color SVG
Thing is a webapp's style
you write a test
Thing is an Electron app
// main.js
const { app } = require('electron')
const MainBrowserWindow = require('./main_browser_window')
let win
function createWindow () {
win = MainBrowserWindow()
win.loadURL('http://localhost:4600')
}
app.on('ready', createWindow)Thing is an Electron app
you write a test
// cypress/integration/spec.js
it('clicks', () => {
// window creation and url load
cy.electronVisitUrl('./main_browser_window.js',
'http://localhost:4600')
cy.get('button')
.click()
.click()
cy.get('#clicked').should('have.text', '2')
})Thing is an Electron app
you write a test
Thing is accessability
it('has good contrast', () => {
cy.visit('/')
cy.injectAxe()
cy.checkA11y({
runOnly: ['cat.color'],
})
})you write a test
You can always write a test using the right tool
75 Cypress plugins
- Framework tooling
- Custom commands
- Development tools
- Visual testing
- Preprocessors
- Reporters
- Component tests
- Authentication
What should I test?
@cypress/code-coverage plugin
example
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
typical Todo application http://todomvc.com/
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })it('completes todos', () => {
... })Ohhh, we don't have a test for Feature C
Keeping track manually
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
it('adds todos', () => { ... })it('completes todos', () => {
... })it('deletes todos', () => {
... })Keeping track manually
1. Carefully collect user stories and feature requirments
2. Write matching tests
3. Keep updating user stories and tests to keep them in sync
HARD
- first test
- two more tests
- hundreds of tests!!!
Keep updating user stories and tests to keep them in sync
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
Keeping track via code coverage
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })it('completes todos', () => {
... })Keeping track via code coverage
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })it('completes todos', () => {
... })green: lines executed during the tests
red: lines NOT executed during the tests
Keeping track via code coverage
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })it('completes todos', () => {
... })it('deletes todos', () => {
... })Keeping track via code coverage
Are we testing all features?
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })it('completes todos', () => {
... })it('deletes todos', () => {
... })Code coverage from tests indirectly
measures implemented features tested
Keeping track via code coverage
⚠️ 100% Code Coverage ≠ 0🐞
Feature A
User can add todo items
Feature B
User can complete todo items
Feature C
User can delete todo items
❚❚❚❚❚
❚❚❚❚❚ ❚❚ ❚❚❚❚
❚❚❚
❚❚❚❚
❚❚❚ ❚❚❚❚❚❚❚
❚❚❚❚
❚❚
❚❚❚❚❚ ❚❚
❚❚❚❚❚❚
❚❚❚
❚❚❚❚❚
❚❚❚❚❚
❚❚❚❚
❚❚❚❚
❚❚❚❚❚❚ ❚❚❚ ❚❚
❚
❚❚
source code
it('adds todos', () => { ... })it('completes todos', () => {
... })it('deletes todos', () => {
... })Unrealistic tests; subset of inputs
code does not implement the feature correctly
Code Coverage with @cypress/code-coverage
-
Instrument code (YOU)
-
Run tests
-
Report results
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
E2E tests are extremely effective at covering a lot of app code
Coverage as a guide to writing E2E tests
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})Coverage as a guide to writing E2E tests
it('adds todos', () => {
cy.visit('/')
cy.get('.new-todo')
.type('write code{enter}')
.type('write tests{enter}')
.type('deploy{enter}')
cy.get('.todo').should('have.length', 3)
})
We have tested "add todo"
Need tests
Write more end-to-end tests
Ughh, missed it
Can we always write an E2E test?
What is this code doing?
Can I write an E2E test to hit this code?
This code should be unreachable from the user interface
Write a Unit Test!
Can we always write an E2E test?
import {getVisibleTodos} from '../../src/selectors'
describe('getVisibleTodos', () => {
it('throws an error for unknown visibility filter', () => {
expect(() => {
getVisibleTodos({
todos: [],
visibilityFilter: 'unknown-filter'
})
}).to.throw()
})
})@cypress/code-coverage plugin
We got this line from the unit test
@cypress/code-coverage plugin
Nice job, @cypress/code-coverage plugin
combines e2e and unit test coverage automatically
Keep the core small
iterate quickly via plugins
E2E tests are extremely effective at covering your
-
application code
-
configuration
-
entire stack
E2E
integration
unit
Really important to users
Really important to developers
Tutorials
Api
Examples
Let's talk money
How does Cypress.io make money?
Cypress Test Runner - OSS 100% forever
Cypress Dashboard paid SaaS for companies
Paid Features 💵
- recording test artifacts
- test parallelization
- GitHub integration
- (WIP) historical insights
Modern CI UI is ok for console log
Paid Features 💵
$ npx cypress run --record
Upload test artifacts from any CI
Paid Features 💵: artifacts
test output, video, screenshots
Test results view 1 / 3
Test results view 2 / 3
Test results view 3 / 3
WIP
I have 100s of tests ...
$ npx cypress run --record --parallel
Cypress v3.1.0
Spin N CI machines and
Text
cypress-dashboard parallelization
cypress-dashboard parallelization
| # machines | run duration | time savings |
|---|---|---|
| 1 | 22:50 | ~ |
| 2 | 11:47 | 48% |
| 3 | 7:51 | 65% |
| 4 | 5:56 | 74% |
| 6 | 3:50 | 83% |
| 8 | 3:00 | 87% |
| 10 | 2:19 | 90% |
cypress-dashboard parallelization
10 machines = 10x speed up
We need to work as a team
Paid Features 💵: GH integration
https://www.cypress.io/blog/2019/07/29/github-integration-for-the-cypress-dashboard/
GitHub Integration 1 / 6
GitHub Integration 2 / 6
GitHub Integration 3 / 6
GitHub Integration 4 / 6
GitHub Integration 5 / 6
By Spec
By Run Group
GitHub Integration 6 / 6
PR Comments
Cypress & Open Source Community
This dark red color looks familiar ...
Nice contrast on the footer labels
todomvc-app-css@2.3.0
todomvc-app-css@2.2.0
Oh yeah, it was my PR!
it('has good contrast when empty', () => {
cy.visit('/')
cy.injectAxe()
cy.checkA11y({
runOnly: ['cat.color'],
})
})it('has good contrast with several todos', () => {
cy.visit('/')
cy.injectAxe()
cy.get('.new-todo')
.type('learn testing{enter}')
.type('be cool{enter}')
cy.get('.todo-list li').should('have.length', 2)
cy.checkA11y({
runOnly: ['cat.color'],
})
})
uses
Cypress tests
use
Cypress tests
uses
Cypress tests
In 2 years since Cypress went OSS it has become a popular and widely used testing tool
Thank you ❤️
Introducing the Open Source and Full-Featured Seed Dashboard Plans
62 organizations on OSS plan
What we are working on
1. Dashboard
2. Test Runner
Test analytics 1 / 4
Test analytics 2 / 4
Test analytics 3 / 4
Test analytics 4 / 4
Run Tagging
cypress run --tag develop
cypress run --tag staging
cypress run --tag production
cypress run --tag nightly
cypress run --tag hourlycypress run --group desktop-uk --tag develop,nightly
cypress run --group desktop-de --tag production
Cypress Dashboard Features
What we are working on
1. Dashboard
2. Test Runner
Firefox support
failing 15 of 3000 tests
Better errors
Better errors
Source of command errors unclear
No full diffs
Better errors
App Errors
Assertions
Code Frames
Better errors
- Error explanation
- Command doc link
- Precise source maps
Better errors
Open your text editor right at the error location
PR #4041
PR #3930
Test Retries 1 / 2
Test Retries 2 / 2
PR #3968
Click on the failed test attempt to see the command log
WIP
- Dashboard
- Test analytics
- Better error views
-
Test Runner
- Firefox support
- Much better errors
- Test retries
Conclusions
Give Cypress a try
Conclusions
Free Dashboard plan for OSS projects
Conclusions
Future is bright
Thank you 👏
Achievement Unlocked: Drive development, increase velocity, and write blissful tests with Cypress
Achievement Unlocked: All Things Open Conference
By Cypress.io
Achievement Unlocked: All Things Open Conference
The demand for developer productivity has never been greater. We are expected to continuously improve and release code, often several times a day. But how do we achieve this kind of velocity without introducing bugs? How can our codebase adapt to changing business requirements, while still keeping technical debt to a minimum? Video at https://www.youtube.com/watch?v=2gP1-TNDzK4
