on

Continuous Integration Server

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Justin James

Let Your Nerd Be Heard

@digitaldrummerj

https://cypress.slides.com/cypress-io/cypress-on-ci

"cypress open" vs "cypress run"
Testing on CI
- installing and caching
CI examples
- TeamCity
- Travis
- CircleCI
Common problems & solutions
Q & A sli.do event #cyci

Q & A

Ask your question and up-vote questions at: https://www.sli.do/

Event code: #cyci

$ cypress open

"Interactive Mode"

Runs one or all specs
File watcher
DOM snapshots and time-traveling debugger

$ cypress run

"Headless Mode"

Runs single spec at a time
No file watching
No DOM snapshots
automatic screenshot on failure
automatic video

# work locally
git add .
git commit -m "feature A + tests"
git push
# CI runs all tests on each commit
cypress run

CI should test every commit

http://on.cypress.io/using-cypress-faq#What-are-your-best-practices-for-organizing-tests

CI without tests is like never changing the oil in your car. It is just a matter of time before it blows up

said by me, just now

(Justin)

Team Responsibility

Merge frequently
master branch is always deployable
Only commit working and tested code
Only merge when build is passing
Don't leave right after merging until builds passes

Let's run tests on CI

Set up OS dependencies
Cache NPM and Cypress
How to run tests in parallel
CI examples
- Travis
- TeamCity
- Circle

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

https://on.cypress.io/continuous-integration

apt-get install xvfb libgtk-3-dev \
  libnotify-dev libgconf-2-4 \
  libnss3 libxss1 libasound2

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

Use one of our prepared images from
https://github.com/cypress-io/cypress-docker-images

cypress/base (OS deps + Node)
cypress/browsers (cypress/base + Chrome)
cypress/included (cypress/browsers + global Cypress module)

Verify Cypress can run

https://on.cypress.io/continuous-integration

npx cypress verify
# either exits fine
# or shows the OS / install error

https://on.cypress.io/debugging

"cypress run" calls "verify" when it runs for the very first time on the machine

Run Cypress binary by itself

Linux

npx cypress cache path
/root/.cache/Cypress

/root/.cache/Cypress/3.3.1/Cypress/Cypress --smoke-test --ping=101
101

ldd /home/person/.cache/Cypress/3.3.1/Cypress/Cypress
  linux-vdso.so.1 (0x00007ffe9eda0000)
  libnode.so => /home/person/.cache/Cypress/3.3.1/Cypress/libnode.so (0x00007fecb43c8000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fecb41ab000)
  libgtk-3.so.0 => not found
  libgdk-3.so.0 => not found
  ...

https://on.cypress.io/continuous-integration

https://on.cypress.io/debugging

this is what "cypress verify" does

Cache NPM and Cypress

git checkout
npm install
cypress run

Performance

(on CI server)

Cache NPM and Cypress

git checkout
npm install
- cypress download
cypress run

Performance

(on CI server)

Cache NPM and Cypress

git checkout
npm install
- cypress download
cache ~/.npm and ~/.cache
cypress run

Performance

npx cypress cache path

(on CI server)

Cache NPM and Cypress

Performance

git checkout
restore cache
npm install
- cypress download
cache ~/.npm and ~/.cache
cypress run

(on CI server)

Cache NPM and Cypress

Performance

on.cypress.io/caching

- checkout
- restore_cache:
    keys:
      - cache-{{ checksum "package.json" }}
- run: npm ci
- run: npx cypress verify
- save_cache:
    key: cache-{{ checksum "package.json" }}
    paths:
      - ~/.npm
      - ~/.cache

github.com/cypress-io/cypress-example-kitchensink

typical CI config file

Single install and run

npm install

cypress run

Multiple Workers

npm install

cypress run

npm install

cypress run

npm install

cypress run

Parallel Runs

npm install

cypress run

--record --parallel

cypress run

--record --parallel

cypress run

--record --parallel

Cypress Dashboard

Load balancing spec files

https://on.cypress.io/parallelization

CI Process

Build Code in Production Mode
Start Web Server
Wait for Web Server to Respond
Run Cypress Tests
Stop Web Server

Build Code & Start Server

npm install -D npm-run-all

# run in serial lint:ci and build:prod
npx run-s lint:ci build:prod

Cross Platform Run Tasks in Serial or Parallel

Build Code & Start Server

npm install -D start-server-and-test

# run start server, wait for localhost:4200, and run cy:run
# "ci:start-server": "angular-http-server --path ./dist/ngws -p 4200",
# "cy:run": "cypress run",
npx start-server-and-test ci:start-server 4200 cy:run

Start Server and Wait for Response

Build Code & Start Server

npm install -D angular-http-server

# start server in ./dist/ngws on port 4200
angular-http-server --path ./dist/ngws -p 4200

Angular Server with Deep Linking

Angular CI Process

"ci": "run-s lint:ci build:prod ci:cy-run",
"lint:ci": "ng lint",
"build:prod": "ng build --prod",
"ci:cy-run": "start-server-and-test ci:start-server 4200 cy:run",
"ci:start-server": "angular-http-server --path ./dist/ngws -p 4200",
"cy:run": "cypress run"

Lint, Unit Tests, Prod Build, Cypress Tests

package.json

CI Examples

Travis CI
TeamCity
CircleCI

CI Examples: Travis

Hosted continuous integration service for open-source and private projects on GitHub

Public Repos: https://travis-ci.org/

Private Repos: https://travis-ci.com/

service hook

Add Repo to Travis

when to trigger build

Build Settings

when to cancel queued build

Build Settings

environment config

Build Settings

schedule build

Build Settings

Travis Build Script

language: node_js

node_js:
  - "lts/*"
addons:
  chrome: stable
  apt:
    packages:
    - libgconf-2-4

.travis.yml

Travis Build Script

cache:
  npm: true
  directories:
    - ~/.npm
    - ./node_modules
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify

.travis.yml

Travis Build Script

install:
    - npm ci

script:
  - npm run ci

.travis.yml

git push

git add .travis.yml

git commit .travis.yml

service hook

run tests / builds

fresh environments

Travis Build Process

Build Log

CI Examples: TeamCity

Powerful Continuous Integration
out of the box

https://www.jetbrains.com/teamcity

Reporter

npm install mocha-teamcity-reporter mocha

Build Script

"ci:teamcity-cy-run": "cypress run --reporter mocha-teamcity-reporter"

package.json

Build Configuration

git push

service hook

run tests / builds

TeamCity Build Process

Build Log

CI Examples: CircleCI

give people everywhere the power to build and deliver software at the speed of imagination

https://circleci.com/

Add Repo to CircleCI

Use Cypress CircleCI Orb

https://github.com/cypress-io/circleci-orb

version: 2.1
orbs:
  # our orb will take care of environment
  # install, caching, build, etc
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      # "cypress" is the name of the imported orb
      # "run" is the name of the job defined in Cypress orb
      - cypress/run

1 install job, 4 test jobs

https://github.com/cypress-io/circleci-orb

version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/install:
          build: 'npm run build'
      - cypress/run:
          requires:
            - cypress/install
          record: true # record results on Cypress Dashboard
          parallel: true # split all specs across machines
          parallelism: 4 # use 4 CircleCI machines to finish quickly
          start: 'npm start' # start server before running tests

Cypress + CircleCI Webinar

https://www.youtube.com/watch?v=J-xbNtKgXfY

https://slides.com/bahmutov/circleci-cypress-orb

"cypress open" vs "cypress run"
Testing on CI
CI examples
- TeamCity
- Travis
- CircleCI
Testing different environments
Common problems & solutions
Q & A sli.do event #cyci

Running Cypress tests against multiple environments

Local + staging

Local + staging + production

Local

Vary configuration settings

{
  "baseUrl": "http://localhost:8080",
  "video": false,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}

{
  "baseUrl": "https://staging.server.com",
  "video": true,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}

local

staging

cypress.json

Cypress settings (config)

vs user environment variables (env)

{
  "baseUrl": "http://localhost:8080",
  "video": false,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}

cypress.json

Cypress settings

Locally run tests against "localhost:8000"

{
  "baseUrl": "http://localhost:8080"
}

cypress run --config baseUrl=https://staging.server.com

Testing staging server

export CYPRESS_baseUrl=https://staging.server.com
cypress run

CLI

env

cy.visit('/')

cypress.json

Command timeout

{
  "baseUrl": "http://localhost:8080",
  "defaultCommandTimeout": 1000
}

cypress run --config \
  baseUrl=https://staging.server.com,defaultCommandTimeout=5000

Testing staging server

export CYPRESS_baseUrl=https://staging.server.com
export CYPRESS_defaultCommandTimeout=5000
cypress run

CLI

env

Cypress settings

cy.visit('/')

cypress.json

baseUrl
defaultCommandTimeout
pageLoadTimeout
requestTimeout
blacklistHosts
modifyObstructiveCode
...

https://on.cypress.io/configuration

Cypress settings

https://on.cypress.io/configuration

// cypress/plugins/index.js
module.exports = (on, config) => {
  config.baseUrl = 'https://staging.server.com/app/'
  // change more options ...
  return config
}

Use plugins file

Cypress settings

https://on.cypress.io/configuration

cypress.json 
  < environment variable 
    < CLI parameter 
      < plugin 
        < run-time Cypress.config(...)

Resolution preference

Wins

Cypress settings

https://on.cypress.io/configuration

cypress run --config "/path/to/config.json"

Coming soon (issue #1369)

Cypress settings

Pass env variables

Need test user name and password

const name = Cypress.env('username')
const pass = Cypress.env('password')
cy.get('#name').type(name)
cy.get('#pass').type(pass)
// log in

How do store and use username and password environment variables?

Good practice

{
  "env": {
    "username": "Joe Tester",
    "password": ""
  }
}

cypress.json

const name = Cypress.env('username')
expect(name, 'username').to.be.a('string')
  .and.be.not.empty
const pass = Cypress.env('password')
// password will not appear in the UI
assert(pass, 'forgot password')
cy.get('#name').type(name)
cy.get('#pass').type(pass)
// log in

test

Pass env variables

Password is sensitive, user name is not

{
  "env": {
    "username": "Joe Tester",
    "password": ""
  }
}

cypress.json

cypress run --env password=***

export CYPRESS_password=****
cypress run

CLI

env

⚠️ 🛑

Cypress automatically puts all unknown env variables that start with CYPRESS_ into env vars

✅

Tip: use as-a locally

; create a section for each app settings
[my-app]
CYPRESS_password=test1234

~/.as-a/.as-a.ini

$ npm i -g as-a
$ as-a my-app cypress run

loads variables from section "my-app" and runs Cypress with those environment variables

https://github.com/bahmutov/as-a

Record tests on Dashboard

cypress run --record --key=***

# use CI private variables
# CYPRESS_RECORD_KEY=****
cypress run --record

CLI

env

⚠️ 🛑

✅

How to pass your private record key

Security

Record tests on Dashboard

Be careful with forked pull requests, a hacker might simply do "echo $CYPRESS_RECORD_KEY"

Vary tests to run in diff environments

cypress/
  integration/
    main/
      spec.js
      spec2.js
    admin/
      auth-spec.js
    prod/
      read-only-spec.js

# run tests safe to run in production
$ npx cypress run --spec "cypress/integration/prod/*.js"

Vary test behavior

// cypress/support/index.js
Cypress.isDevelopment = () =>
  // NODE_ENV is set in plugins file
  Cypress.env('NODE_ENV') === 'development'

// cypress/integration/main_spec.js
if (!Cypress.isDevelopment()) {
  it('has robots.txt', () => {
    cy.request('/robots.txt').its('body')
      .should('include', 'Disallow: /ja/')
      .and('include', 'Disallow: /zh-cn/')
      .and('include', 'Disallow: /pt-br/')
  })
}

Example: https://github.com/cypress-io/cypress-documentation

Vary test behavior: stub XHR

// cypress/support/index.js
Cypress.isDevelopment = () =>
  // NODE_ENV is set in plugins file
  Cypress.env('NODE_ENV') === 'development'

// cypress/integration/main_spec.js
it('shows initial users', () => {
  if (!Cypress.isDevelopment()) {
    cy.server()
      .route('/users', 'fixture:ten-users')
  }
  cy.get('.list').should('have.length', 10)
})

Cypress on CI:

Common problems & solutions

Missing Git information
Flaky tests on CI
Video freezes

Help: recorded tests on Dashboard have no Git info

closed issue #777

# set on CI CYPRESS_RECORD_KEY=...
cypress run --record

Cypress gets commit information

using https://github.com/cypress-io/commit-info

1. First, trying to use "git" commands

branch: 'git rev-parse --abbrev-ref HEAD'

author: 'git show -s --pretty=%an'

Cypress gets commit information

2. If Git information is unavailable, check env variables (CI-specific)

branch: BITBUCKET_BRANCH (BitBucket)
branch: CIRCLE_BRANCH (CircleCI)

ci_provider.js

https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/util/ci_provider.js

Cypress gets commit information

3. If env variables are not found, use fallback env variables COMMIT_INFO_*

COMMIT_INFO_BRANCH=$MY_CI_BRANCH_NAME
COMMIT_INFO_SHA=$MY_CI_COMMIT_REF
...
cypress run --record

your CI file

https://github.com/cypress-io/commit-info#fallback-environment-variables

Cypress gets commit information

branch: COMMIT_INFO_BRANCH
message: COMMIT_INFO_MESSAGE
email: COMMIT_INFO_EMAIL
author: COMMIT_INFO_AUTHOR
sha: COMMIT_INFO_SHA
timestamp: COMMIT_INFO_TIMESTAMP
remote: COMMIT_INFO_REMOTE

env vars

https://github.com/cypress-io/commit-info#fallback-environment-variables

$ docker run \
  -e COMMIT_INFO_BRANCH=develop \
  -e COMMIT_INFO_SHA=e5d9eb66474bc0b681da9240aa5a457fe17bc8f3 \
  <container name>

Help: my tests run locally but fail on CI! 1/3

cypress run

uses headless Electron (old version)

cypress run --headed

uses headed Electron just like "cypress open"

📹

🚫

cypress run --browser chrome

uses headed Chrome just like "cypress open"

📹

🚫

Help: my tests run locally but fail on CI! 2/3

There might be race condition between the test and the application. CI with its speed difference catches the problem.

Control the tests more:

add more `should` assertions
add more network route waits

Help: my tests run locally but fail on CI! 3/3

If you mocking XHR calls, make sure the API isn't running locally when running tests

Get more information from CI

https://github.com/bahmutov/cypress-failed-log

Get more information from CI

https://github.com/bahmutov/cypress-failed-log

Automatically saves JSON file in "cypress/logs"

{
  "specName": "failing-spec.js",
  "title": "loads the About tab",
  "suiteName": "Website",
  "testName": "Website loads the About tab",
  "testError": "Timed out retrying: Expected to find content: 'Join Us' but never did.",
  "testCommands": [
    "visit",
    "new url https://www.company.com/#/",
    "contains a.nav-link, About",
    "click",
    "new url https://www.company.com/#/about",
    "hash",
    "assert expected **#/about** to equal **#/about**",
    "contains Join Us",
    "assert expected **body :not(script):contains(**'Join Us'**), [type='submit'][value~='Join Us']** to exist in the DOM"
  ],
  "screenshot": "loads-the-about-tab.png"
}

Get Chrome logs

https://github.com/flotwig/cypress-log-to-output

cypress run --browser=chrome

Zach Bloomquist

@flotwig

See what happens on CI

Point Linux XVFB at your local X11 server

export DISPLAY=<your IP X11 display>
cypress run --headed

https://www.cypress.io/blog/2019/05/02/run-cypress-with-a-single-docker-command/

Help: test video freezes or has gaps!

closed issue #4722

If CI container is low on CPU, then video starts dropping frames or stops completely. Use more powerful CI machine.

Related: browser crashes during tests

https://letyournerdbeheard.com/masterclass

Free Masterclass
Wednesday 11am PST

https://letyournerdbeheard.com/masterclass

Free Masterclass
Wednesday 11am PST

How to overcome fear of public speaking
The 4 key ingredients for effective storytelling
How to home your message
Key design principles to create rock solid slides
Strategies to deliver your talk without feeling anxious and stressed out

on

Continuous Integration Server

Gleb Bahmutov

Vp of Engineering

WEBCAST

Justin James

Let Your Nerd Be Heard

Contents

Q & A

Event code: #cyci

$ cypress open

$ cypress run

CI should test every commit

CI without tests is like never changing the oil in your car. It is just a matter of time before it blows up

Team Responsibility

Merge frequently

master branch is always deployable

Only commit working and tested code

Only merge when build is passing

Don't leave right after merging until builds passes

Let's run tests on CI

Set up OS dependencies

Cache NPM and Cypress

How to run tests in parallel

CI examples

Travis

TeamCity

Circle

What do you need on CI

Windows, Mac

Linux

What do you need on CI

Windows, Mac

Linux

Verify Cypress can run

Run Cypress binary by itself

Linux

Cache NPM and Cypress

Cache NPM and Cypress

Cache NPM and Cypress

Cache NPM and Cypress

Cache NPM and Cypress

Single install and run

Multiple Workers

Parallel Runs

CI Process

Build Code in Production Mode

Start Web Server

Wait for Web Server to Respond

Run Cypress Tests

Stop Web Server

Build Code & Start Server

Build Code & Start Server

Build Code & Start Server

Angular CI Process

CI Examples

CI Examples: Travis

Hosted continuous integration service for open-source and private projects on GitHub

Public Repos: https://travis-ci.org/

Private Repos: https://travis-ci.com/

Add Repo to Travis

when to trigger build

Build Settings

when to cancel queued build

Build Settings

environment config

Build Settings

schedule build

Build Settings

Travis Build Script

Travis Build Script

Travis Build Script

Travis Build Process

Build Log

Build Log

CI Examples: TeamCity

Powerful Continuous Integration out of the box

https://www.jetbrains.com/teamcity

Reporter

Build Script

Powerful Continuous Integration
out of the box

Free Masterclass
Wednesday 11am PST

Free Masterclass
Wednesday 11am PST

Free Masterclass
Wednesday 11am PST