on

Continuous Integration Server

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Justin James

Let Your Nerd Be Heard

@digitaldrummerj

Contents

  • "cypress open" vs "cypress run"
  • Testing on CI
    • installing and caching
  • CI examples
    • TeamCity
    • Travis
    • CircleCI
  • Common problems & solutions
  • ​Q & A sli.do event #cyci

Q & A

Ask your question and up-vote questions at: https://www.sli.do/

 

Event code: #cyci

$ cypress open

"Interactive Mode"

  • Runs one or all specs
  • File watcher
  • DOM snapshots and time-traveling debugger

$ cypress run

"Headless Mode"

  • Runs single spec at a time
  • No file watching
  • No DOM snapshots
  • automatic screenshot on failure
  • automatic video
# work locally
git add .
git commit -m "feature A + tests"
git push
# CI runs all tests on each commit
cypress run

CI should test every commit

CI without tests is like never changing the oil in your car. It is just a matter of time before it blows up

 said by me, just now

(Justin)

Team Responsibility

  • Merge frequently

  • master branch is always deployable

  • Only commit working and tested code 

  • Only merge when build is passing

  • Don't leave right after merging until builds passes

Let's run tests on CI

  • Set up OS dependencies

  • Cache NPM and Cypress

  • How to run tests in parallel

  • CI examples

    • Travis

    • TeamCity

    • Circle

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

apt-get install xvfb libgtk-3-dev \
  libnotify-dev libgconf-2-4 \
  libnss3 libxss1 libasound2

What do you need on CI

Windows, Mac

Nothing to install, Cypress should just work

Linux

Use one of our prepared images from
https://github.com/cypress-io/cypress-docker-images

  • cypress/base (OS deps + Node)
  • cypress/browsers (cypress/base + Chrome)
  • cypress/included (cypress/browsers + global Cypress module)

Verify Cypress can run

npx cypress verify
# either exits fine
# or shows the OS / install error

"cypress run" calls "verify" when it runs for the very first time on the machine

Run Cypress binary by itself

Linux

npx cypress cache path
/root/.cache/Cypress
/root/.cache/Cypress/3.3.1/Cypress/Cypress --smoke-test --ping=101
101
ldd /home/person/.cache/Cypress/3.3.1/Cypress/Cypress
  linux-vdso.so.1 (0x00007ffe9eda0000)
  libnode.so => /home/person/.cache/Cypress/3.3.1/Cypress/libnode.so (0x00007fecb43c8000)
  libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fecb41ab000)
  libgtk-3.so.0 => not found
  libgdk-3.so.0 => not found
  ...

this is what "cypress verify" does

Cache NPM and Cypress

  • git checkout
  • npm install
  • cypress run

Performance

(on CI server)

Cache NPM and Cypress

  • git checkout
  • npm install
    • cypress download
  • cypress run

Performance

(on CI server)

Cache NPM and Cypress

  • git checkout
  • npm install
    • cypress download
  • ​cache ~/.npm and ~/.cache
  • cypress run

Performance

npx cypress cache path

(on CI server)

Cache NPM and Cypress

Performance

  • git checkout
  • restore cache
  • npm install
    • cypress download
  • ​cache ~/.npm and ~/.cache
  • cypress run

(on CI server)

Cache NPM and Cypress

Performance

- checkout
- restore_cache:
    keys:
      - cache-{{ checksum "package.json" }}
- run: npm ci
- run: npx cypress verify
- save_cache:
    key: cache-{{ checksum "package.json" }}
    paths:
      - ~/.npm
      - ~/.cache

typical CI config file

Single install and run

npm install

cypress run

Multiple Workers

npm install

cypress run

npm install

cypress run

npm install

cypress run

Parallel Runs

npm install

cypress run

--record --parallel

cypress run

--record --parallel

cypress run

--record --parallel

Cypress Dashboard

Load balancing spec files

CI Process

  1. Build Code in Production Mode

  2. Start Web Server

  3. Wait for Web Server to Respond

  4. Run Cypress Tests

  5. Stop Web Server

Build Code & Start Server

npm install -D npm-run-all
# run in serial lint:ci and build:prod
npx run-s lint:ci build:prod

Cross Platform Run Tasks in Serial or Parallel

Build Code & Start Server

npm install -D start-server-and-test
# run start server, wait for localhost:4200, and run cy:run
# "ci:start-server": "angular-http-server --path ./dist/ngws -p 4200",
# "cy:run": "cypress run",
npx start-server-and-test ci:start-server 4200 cy:run

Start Server and Wait for Response

Build Code & Start Server

npm install -D angular-http-server
# start server in ./dist/ngws on port 4200
angular-http-server --path ./dist/ngws -p 4200

Angular Server with Deep Linking

Angular CI Process

"ci": "run-s lint:ci build:prod ci:cy-run",
"lint:ci": "ng lint",
"build:prod": "ng build --prod",
"ci:cy-run": "start-server-and-test ci:start-server 4200 cy:run",
"ci:start-server": "angular-http-server --path ./dist/ngws -p 4200",
"cy:run": "cypress run"

Lint, Unit Tests, Prod Build, Cypress Tests

package.json

CI Examples

  • Travis CI
  • TeamCity
  • CircleCI

CI Examples: Travis

Hosted continuous integration service for open-source and private projects on GitHub

Public Repos: https://travis-ci.org/

    Private  Repos: https://travis-ci.com/

service hook

Add Repo to Travis

when to trigger build

Build Settings

when to cancel queued build

Build Settings

environment config

Build Settings

schedule build

Build Settings

Travis Build Script

language: node_js

node_js:
  - "lts/*"
addons:
  chrome: stable
  apt:
    packages:
    - libgconf-2-4

.travis.yml

Travis Build Script

cache:
  npm: true
  directories:
    - ~/.npm
    - ./node_modules
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify

.travis.yml

Travis Build Script

install:
    - npm ci

script:
  - npm run ci

.travis.yml

git push

git add .travis.yml

git commit .travis.yml

service hook

run tests / builds

fresh environments

Travis Build Process

Build Log

Build Log

CI Examples: TeamCity

Powerful Continuous Integration
out of the box

Reporter

npm install mocha-teamcity-reporter mocha

Build Script

"ci:teamcity-cy-run": "cypress run --reporter mocha-teamcity-reporter"

package.json

Build Configuration

Build Configuration

Build Configuration

Build Configuration

Build Configuration

Build Configuration

Build Configuration

Build Configuration

Build Configuration

git push

service hook

run tests / builds

TeamCity Build Process

Build Log

Build Log

Build Log

CI Examples: CircleCI

give people everywhere the power to build and deliver software at the speed of imagination

Add Repo to CircleCI

Use Cypress CircleCI Orb

version: 2.1
orbs:
  # our orb will take care of environment
  # install, caching, build, etc
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      # "cypress" is the name of the imported orb
      # "run" is the name of the job defined in Cypress orb
      - cypress/run

1 install job, 4 test jobs

version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/install:
          build: 'npm run build'
      - cypress/run:
          requires:
            - cypress/install
          record: true # record results on Cypress Dashboard
          parallel: true # split all specs across machines
          parallelism: 4 # use 4 CircleCI machines to finish quickly
          start: 'npm start' # start server before running tests

Cypress + CircleCI Webinar

Contents

  • "cypress open" vs "cypress run"
  • Testing on CI
  • CI examples
    • TeamCity
    • Travis
    • CircleCI
  • Testing different environments
  • Common problems & solutions
  • ​Q & A sli.do event #cyci

Running Cypress tests against multiple environments

Local + staging

Local + staging + production

Local

Vary configuration settings

{
  "baseUrl": "http://localhost:8080",
  "video": false,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}
{
  "baseUrl": "https://staging.server.com",
  "video": true,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}

local

staging

cypress.json

Cypress settings (config)

vs user environment variables (env)

{
  "baseUrl": "http://localhost:8080",
  "video": false,
  "env": {
    "username": "Joe Tester",
    "testAccount": {
      "id": "1123",
      "admin": false
    }
  }
}

cypress.json

Cypress settings

Locally run tests against "localhost:8000"

{
  "baseUrl": "http://localhost:8080"
}
cypress run --config baseUrl=https://staging.server.com

Testing staging server

export CYPRESS_baseUrl=https://staging.server.com
cypress run

CLI

env

cy.visit('/')

cypress.json

Command timeout

{
  "baseUrl": "http://localhost:8080",
  "defaultCommandTimeout": 1000
}
cypress run --config \
  baseUrl=https://staging.server.com,defaultCommandTimeout=5000

Testing staging server

export CYPRESS_baseUrl=https://staging.server.com
export CYPRESS_defaultCommandTimeout=5000
cypress run

CLI

env

Cypress settings

cy.visit('/')

cypress.json

  • baseUrl
  • defaultCommandTimeout
  • pageLoadTimeout
  • requestTimeout
  • blacklistHosts
  • modifyObstructiveCode
  • ...

Cypress settings

// cypress/plugins/index.js
module.exports = (on, config) => {
  config.baseUrl = 'https://staging.server.com/app/'
  // change more options ...
  return config
}

Use plugins file

Cypress settings

cypress.json 
  < environment variable 
    < CLI parameter 
      < plugin 
        < run-time Cypress.config(...)

Resolution preference

Wins

Cypress settings

cypress run --config "/path/to/config.json"

Coming soon (issue #1369)

Cypress settings

Pass env variables

Need test user name and password

const name = Cypress.env('username')
const pass = Cypress.env('password')
cy.get('#name').type(name)
cy.get('#pass').type(pass)
// log in

How do store and use username and password environment variables?

Good practice

{
  "env": {
    "username": "Joe Tester",
    "password": ""
  }
}

cypress.json

const name = Cypress.env('username')
expect(name, 'username').to.be.a('string')
  .and.be.not.empty
const pass = Cypress.env('password')
// password will not appear in the UI
assert(pass, 'forgot password')
cy.get('#name').type(name)
cy.get('#pass').type(pass)
// log in

test

Pass env variables

Password is sensitive, user name is not

{
  "env": {
    "username": "Joe Tester",
    "password": ""
  }
}

cypress.json

cypress run --env password=***
export CYPRESS_password=****
cypress run

CLI

env

⚠️ 🛑

Cypress automatically puts all unknown env variables that start with CYPRESS_ into env vars

Tip: use as-a locally

; create a section for each app settings
[my-app]
CYPRESS_password=test1234

~/.as-a/.as-a.ini

$ npm i -g as-a
$ as-a my-app cypress run

loads variables from section "my-app" and runs Cypress with those environment variables

Record tests on Dashboard

cypress run --record --key=***
# use CI private variables
# CYPRESS_RECORD_KEY=****
cypress run --record

CLI

env

⚠️ 🛑

How to pass your private record key

Security

Record tests on Dashboard

Be careful with forked pull requests, a hacker might simply do "echo $CYPRESS_RECORD_KEY"

Vary tests to run in diff environments

cypress/
  integration/
    main/
      spec.js
      spec2.js
    admin/
      auth-spec.js
    prod/
      read-only-spec.js
# run tests safe to run in production
$ npx cypress run --spec "cypress/integration/prod/*.js"

Vary test behavior

// cypress/support/index.js
Cypress.isDevelopment = () =>
  // NODE_ENV is set in plugins file
  Cypress.env('NODE_ENV') === 'development'

// cypress/integration/main_spec.js
if (!Cypress.isDevelopment()) {
  it('has robots.txt', () => {
    cy.request('/robots.txt').its('body')
      .should('include', 'Disallow: /ja/')
      .and('include', 'Disallow: /zh-cn/')
      .and('include', 'Disallow: /pt-br/')
  })
}

Vary test behavior: stub XHR

// cypress/support/index.js
Cypress.isDevelopment = () =>
  // NODE_ENV is set in plugins file
  Cypress.env('NODE_ENV') === 'development'

// cypress/integration/main_spec.js
it('shows initial users', () => {
  if (!Cypress.isDevelopment()) {
    cy.server()
      .route('/users', 'fixture:ten-users')
  }
  cy.get('.list').should('have.length', 10)
})

Cypress on CI:

Common problems & solutions

  • Missing Git information
  • Flaky tests on CI
  • Video freezes

Help: recorded tests on Dashboard have no Git info

closed issue #777

# set on CI CYPRESS_RECORD_KEY=...
cypress run --record

Cypress gets commit information

1. First, trying to use "git" commands

branch: 'git rev-parse --abbrev-ref HEAD'
author: 'git show -s --pretty=%an'

Cypress gets commit information

2. If Git information is unavailable, check env variables (CI-specific)

branch: BITBUCKET_BRANCH (BitBucket)
branch: CIRCLE_BRANCH (CircleCI)

ci_provider.js

Cypress gets commit information

3. If env variables are not found, use fallback env variables COMMIT_INFO_* 

COMMIT_INFO_BRANCH=$MY_CI_BRANCH_NAME
COMMIT_INFO_SHA=$MY_CI_COMMIT_REF
...
cypress run --record

your CI file

Cypress gets commit information

branch: COMMIT_INFO_BRANCH
message: COMMIT_INFO_MESSAGE
email: COMMIT_INFO_EMAIL
author: COMMIT_INFO_AUTHOR
sha: COMMIT_INFO_SHA
timestamp: COMMIT_INFO_TIMESTAMP
remote: COMMIT_INFO_REMOTE

env vars

$ docker run \
  -e COMMIT_INFO_BRANCH=develop \
  -e COMMIT_INFO_SHA=e5d9eb66474bc0b681da9240aa5a457fe17bc8f3 \
  <container name>

Help: my tests run locally but fail on CI! 1/3

cypress run

uses headless Electron (old version)

cypress run --headed

uses headed Electron just like "cypress open"

📹

📹

🚫

cypress run --browser chrome

uses headed Chrome just like "cypress open"

📹

🚫

Help: my tests run locally but fail on CI! 2/3

There might be race condition between the test and the application. CI with its speed difference catches the problem.

Control the tests more:

  • add more `should` assertions
  • add more network route waits

Help: my tests run locally but fail on CI! 3/3

If you mocking XHR calls, make sure the API isn't running locally when running tests

Get more information from CI

Get more information from CI

Automatically saves JSON file in "cypress/logs"

{
  "specName": "failing-spec.js",
  "title": "loads the About tab",
  "suiteName": "Website",
  "testName": "Website loads the About tab",
  "testError": "Timed out retrying: Expected to find content: 'Join Us' but never did.",
  "testCommands": [
    "visit",
    "new url https://www.company.com/#/",
    "contains a.nav-link, About",
    "click",
    "new url https://www.company.com/#/about",
    "hash",
    "assert expected **#/about** to equal **#/about**",
    "contains Join Us",
    "assert expected **body :not(script):contains(**'Join Us'**), [type='submit'][value~='Join Us']** to exist in the DOM"
  ],
  "screenshot": "loads-the-about-tab.png"
}

Get Chrome logs

cypress run --browser=chrome

Zach Bloomquist

@flotwig

See what happens on CI

Point Linux XVFB at your local X11 server

export DISPLAY=<your IP X11 display>
cypress run --headed

Help: test video freezes or has gaps!

closed issue #4722

If CI container is low on CPU, then video starts dropping frames or stops completely. Use more powerful CI machine.

Related: browser crashes during tests

Free Masterclass
Wednesday 11am PST

Free Masterclass
Wednesday 11am PST

  • How to overcome fear of public speaking

  • The 4 key ingredients for effective storytelling

  • How to home your message 

  • Key design principles to create rock solid slides

  • Strategies to deliver your talk without feeling anxious and stressed out

Free Masterclass
Wednesday 11am PST

Make sure to claim your spot

Seats are limited to 100 people

Q & A

  • Running Cypress on CI
    • TeamCity, Travis, Circle, etc
  • ​Varying tests per environment
  • Common problems
  • Q & A use sli.do event #cyci 

Cypress on Continuous Integration Server

By Cypress.io

Cypress on Continuous Integration Server

Link to webcast: https://www.youtube.com/watch?v=saYovXS9Llk Running end-to-end tests on each commit should be simple. In this presentation, we will review how Cypress should be installed and cached on a variety of CI systems.

  • 2,772
Loading comments...

More from Cypress.io