Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Rick Fleuren

Software Developer

@richyflowers

+

Cypress & DHL

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Rick Fleuren

Software Developer

@richyflowers

Sli.do #cydhl https://app.sli.do/event/vcay1jyw

Docker compose
Test parallelization on Jenkins
Test parallelization via Docker
Cypress pitfalls
Q & A

Q & A: use Sli.do event code

#cydhl

or direct link https://app.sli.do/event/vcay1jyw

The problem was facing

The stack

Docker / docker-compose

API

Kafka

Label

Capabilities

Cypress docker-compose intro

Simple: run web application and Cypress locally

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

Cypress docker-compose intro

Simple: run web application in Docker container and Cypress locally

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

https://github.com/bahmutov/cypress-tests-apache-in-docker

Docker container

Cypress docker-compose intro

Medium: run web application in Docker container and Cypress in another Docker container

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

https://github.com/cypress-io/cypress-example-docker-compose

Docker container

version: '3'
services:
  web:
    image: apache
    build: ./webapp
    container_name: apache
    restart: always
    # we can see the server running at "localhost:8080"
    ports:
      - "8080:80"

  e2e:
    image: cypress
    build: ./e2e
    container_name: cypress
    depends_on:
      - web
    # note: inside e2e container, the network allows accessing
    # "web" host under name "web"
    # so "curl http://web" would return whatever the webserver
    # in the "web" container is cooking
    # see https://docs.docker.com/compose/networking/
    environment:
      - CYPRESS_baseUrl=http://web
    command: npx cypress run
    # mount the host directory e2e/cypress and the file e2e/cypress.json as
    # volumes within the container
    # this means that:
    #  1. anything that Cypress writes to these folders (e.g., screenshots,
    #     videos) appears also on the Docker host's filesystem
    #  2. any change that the developer applies to Cypress files on the host
    #     machine immediately takes effect within the e2e container (no docker
    #     rebuild required).
    volumes:
      - ./e2e/cypress:/app/cypress
      - ./e2e/cypress.json:/app/cypress.json

https://github.com/cypress-io/cypress-example-docker-compose

docker-compose.yml

Cypress docker-compose intro

Medium: Cypress in Docker container and web application anywhere

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

https://github.com/cypress-io/cypress-example-docker-compose-included

Docker container

services:
  zookeeper:
    image: wurstmeister/zookeeper
    ...

  kafka:
    image: wurstmeister/kafka
    ...
    
    
  dhl-cap:
    image: dhlparcel/dhl-parcel-capability-service:${DOCKER_COMPOSE_CAP_VERSION:-latest}
    ...
  
  dhl-lab:
    image: dhlparcel/dhl-parcel-label-service:${DOCKER_COMPOSE_LAB_VERSION:-latest}
    ...
  
  dhl-api:
    image: dhlparcel/dhl-parcel-api:${DOCKER_COMPOSE_API_VERSION:-latest}
    ...

... all sorts of micro services ...

  dhl-load-data:
    image: dhlparcel/dhl-parcel-build-postgres:latest
    ...

The local stack

docker-compose.yml

Overrides

services:
  dhl-e2e-cypress:
    image: dhlparcel/dhl-parcel-cypress:4.2.0
    ipc: host
    command: "sh -c '/var/mdp/scripts/compose/run-cypress.sh'"
    volumes:
      - ./:/var/mdp:rw
    environment:
      - CI_BUILD_ID=${CI_BUILD_ID}
      - KEY=${MY_CYPRESS_KEY}
    links:
      - dhl-api:localhost.dhlparcel.nl
      - dhl-api:localhost.dhlparcel.be
      - dhl-api:localhost.dhlparcel.pt
      - dhl-api:localhost.dhlparcel.es
      - dhl-api:localhost.dhlparcel.eu
      - dhl-api:localhost.dhlparcel.ch
      - dhl-api:localhost.dhlparcel.se
      - dhl-admin-api:localhost.admin.dhlparcel.nl
    depends_on:
      - dhl-load-data

docker-compose.jenkins.yml

#!/usr/bin/env bash

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"
cd /var/mdp/${PROJECT}/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --key ${KEY}

run-cypress.sh

def composeBuild = "-f docker-compose.yml -f docker-compose.jenkins.yml --project-name ${projectBuildNr}"

Jenkinsfile

      try {
          stage("E2E") {
           sh "docker-compose ${composeBuild} run --rm --no-deps dhl-e2e-cypress"
          }
      } finally {
        sh "docker-compose ${composeBuild} logs --tail='all'"
        junit "**/e2e_cypress/test-results.xml"
      }

 docker-compose run --help
 
    --no-deps             Don't start linked services.
    --rm                  Remove container after run. Ignored in detached mode.

Start docker-compose

The challenge

4 different frontend project

(2 React apps, 2 Angular apps)

B2X

~ 6 min

C2C

~ 2 min

PWA

~ 3 min

Admin

~ 1 min

400+ release to production / year

Consecutive run: ~10 mins!

Slow and you had to wait for it twice:

Once for your pull request, once for the master build

Fast delivery, slow builds?

test codebase

Tried our own version of parallelisation

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}


parallel(
  "B2X-1": {
    stage("E2E - B2X - 1") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-1"
    }
  },
  "B2X-2": {
    stage("E2E - B2X - 2") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-2"
    }
  },
  "C2C": {
    stage("E2E - C2C") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c"
    }
  },
  "Admin": {
    stage("E2E - Admin") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-admin"
    }
  },
  "PWA": {
    stage("E2E - PWA") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa"
    }
  },
)

  dhl-e2e-b2x-suite-1:
    image: dhlparcel/dhl-parcel-cypress:4.2.0
    ipc: host
    command: "sh -c '/var/mdp/scripts/compose/run-cypress.sh --spec='cypress/integration/suite-1-*"
    volumes:
      - ./:/var/mdp:rw
    environment:
      - PROJECT=business-site
    links:
      - dhl-api:localhost.dhlparcel.nl
      - dhl-api:localhost.dhlparcel.be
      - dhl-api:localhost.dhlparcel.pt
      - dhl-api:localhost.dhlparcel.es
      - dhl-api:localhost.dhlparcel.eu
      - dhl-api:localhost.dhlparcel.ch
      - mailcatcher:mailcatcher
    depends_on:
      - dhl-load-data

Homebrew test run: ~7 mins!

Listen to friendly neighbourhood Cypress

How to add more 'machines'?

Jenkins parallelisation

Let Jenkins do the work

Jenkins parallelisation

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}


parallel(
  "B2X-1-2": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-1" }
  },
  "B2X-1-1": {
    ...
  },
  "B2X-1-3": {
    ...
  },
  "B2X-1-4": {
    ...
  },
  "B2X-2-1": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-2" }
  },
  "B2X-2-2": {
    ...
  },
  "B2X-2-3": {
    ...
  },
  "C2C": {
    stage("E2E - C2C") {  sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c" }
  },
  "C2C": {
   ...
  },
  "Admin": {
   ...
  },
  "PWA-1": {
    stage("E2E - PWA") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa" }
  },
  "PWA-2": {
    ...
  },
  "PWA-3": {
    ...
  },
  "PWA-4": {
    ...
  },
)

Test run: ~ 6 min

Grouping B2X

parallel(
  "B2X-1": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite" }
  },
  "B2X-2": {
    ...
  },
  "B2X-3": {
    ...
  },
  "B2X-4": {
    ...
  },
  "B2X-5": {
    ...
  },
  "B2X-6": {
    ...
  },
  "B2X-7": {
    ...
  },
  "C2C": {
    stage("E2E - C2C") {  sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c" }
  },
  "C2C": {
   ...
  },
  "Admin": {
   ...
  },
  "PWA-1": {
    stage("E2E - PWA") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa" }
  },
  "PWA-2": {
    ...
  },
  "PWA-3": {
    ...
  },
  "PWA-4": {
    ...
  },
)

Tests run: ~5.5 min

CPU

C2C / Admin end their run

PWA

Time

Load

CPU

Time

Load

Wasted CPU

Grouping the test runs

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"


cd /var/mdp/business-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${B2X_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/pwa-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${PWA_KEY} --ci-build-id ${CI_BUILD_ID} 

cd /var/mdp/admin-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${ADMIN_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${C2C_KEY} --ci-build-id ${CI_BUILD_ID}

run-cypress.sh

Machines

B2X

PWA

C2C

...

B2X

PWA

C2C

...

B2X

PWA

C2C

...

B2X

PWA

C2C

Admin

...

B2X

C2C

...

Grouping all tests

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}

parallel(
  "E2E-1": {
    stage("E2E - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-2": {
    stage("E2E - 2") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-3": {
    stage("E2E - 3") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-4": {
    stage("E2E - 4") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-5": {
    stage("E2E - 5") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-6": {
    stage("E2E - 6") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-7": {
    stage("E2E - 7") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-8": {
    stage("E2E - 8") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
)

CPU

Time

Load

Tests run: ~4.5 min

Problems:

init step needed before run, risk of running twice

quite verbose in syntax

Docker parallelisation

Let Docker do the work

Jenkins

stage("Run E2E") {
  sh "docker-compose ${composeBuild} up --scale dhl-e2e-cypress=8 dhl-e2e-cypress"
}

Tests run: ~3.5 min

We done?

Builds marked successful who were actually failing

Jenkins

Jenkins only looked at the first exit code of the first container that stopped

First container ended successfully?

Build green, carry on

Passing the error codes

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"
err=0
trap '(( err += $? ))' ERR


cd /var/mdp/business-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${B2X_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/pwa-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${PWA_KEY} --ci-build-id ${CI_BUILD_ID} 

cd /var/mdp/admin-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${ADMIN_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${C2C_KEY} --ci-build-id ${CI_BUILD_ID} 

exit "$err"

run-cypress.sh

Trap

err=0
trap '(( err += $? ))' ERR

...

exit err

trap defines and activates handlers to be run when the shell receives signals or other special conditions.

Jenkins

"Run E2E": {
  stage("Run E2E") {
    sh "docker-compose ${composeBuild} up --scale dhl-e2e-cypress=8 dhl-e2e-cypress"
    sh "docker-compose ${composeBuild} ps"
    CYPRESS_ERROR_CODE = sh (
        script: "docker-compose ${composeBuild} ps -q 
                 | xargs docker inspect -f '{{ .State.ExitCode }}' 
                 | grep -v '^0' 
                 | wc -l 
                 | tr -d ' '",
        returnStdout: true
    ).trim()

    echo "Number of Cypress tests failed: ${CYPRESS_ERROR_CODE}"

    if (CYPRESS_ERROR_CODE.toInteger() != 0) {
        error("Cypress tests failed: ${CYPRESS_ERROR_CODE}")
    }
  }
},

Jenkins

sh "docker-compose ${composeBuild} ps"

docker-compose ${composeBuild} ps -q 			# Only print the IDS
| xargs docker inspect -f '{{ .State.ExitCode }}' 	# Get all the exit codes
| grep -v '^0' 						# Invert match anything other than 0
| wc -l 						# Line count those results
| tr -d ' '						# Remove all the spaces

Parse it to Integer, and process in Jenkins

if (CYPRESS_ERROR_CODE.toInteger() != 0) {
  error("Cypress tests failed: ${CYPRESS_ERROR_CODE}")
}

Summary

Test run	Time
Consecutive	~10 mins
Homebrew parallelisation	~7 mins
Jenkins parallelisation	~4.5 mins
Docker parallelisation	~3.5 mins

Cypress Pitfalls

Things we learned using Cypress

.wait('@get-orders')

.wait('@get-settings')

Wrong

.wait(['@get-orders', '@get-settings'])

Right

.wait(1000)

Wrong

.wait('@specific-xhr')

.get('div.class')

Right

https://on.cypress.io/wait

First rule of parallelization:

Make your tests autonomous

Cypress does not know everything you're doing

Think before you add more 'machines'

If you can't spare the resources, your tests will only perform slower

Know when to stop optimizing

Please enjoy the opening tune for the rest of your day!

Gleb Bahmutov

Vp of Engineering

@bahmutov

Rick Fleuren

Software Developer

@richyflowers

https://cypress.slides.com/cypress-io/cypress-and-dhl

Gleb Bahmutov

Vp of Engineering

WEBCAST

Rick Fleuren

Software Developer

+

Cypress & DHL

Gleb Bahmutov

Vp of Engineering

WEBCAST

Rick Fleuren

Software Developer

Contents

Docker compose

Test parallelization on Jenkins

Test parallelization via Docker

Cypress pitfalls

Q & A

The problem was facing

The stack

Docker / docker-compose

Cypress docker-compose intro

Cypress docker-compose intro

Cypress docker-compose intro

Cypress docker-compose intro

The local stack

docker-compose.yml

Overrides

docker-compose.jenkins.yml

run-cypress.sh

Jenkinsfile

The challenge

4 different frontend project

B2X

C2C

PWA

Admin

400+ release to production / year

Consecutive run: ~10 mins!

Fast delivery, slow builds?

test codebase

Tried our own version of parallelisation

Tried our own version of parallelisation

Homebrew test run: ~7 mins!

Other problems

Listen to friendly neighbourhood Cypress

How to add more 'machines'?

Jenkins parallelisation

Let Jenkins do the work

Jenkins parallelisation

Test run: ~ 6 min

Grouping B2X

Tests run: ~5.5 min

CPU

CPU

Grouping the test runs

Machines

Grouping all tests

CPU

Tests run: ~4.5 min

Problems:

Docker parallelisation

Let Docker do the work

Jenkins

Tests run: ~3.5 min

Builds marked successful who were actually failing

Passing the error codes

Trap

Jenkins

Jenkins

Summary

Cypress Pitfalls

Things we learned using Cypress

.wait('@get-orders')

.wait('@get-settings')

Wrong

.wait(['@get-orders', '@get-settings'])

Right

.wait(1000)

Wrong