Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Rick Fleuren

Software Developer

@richyflowers

+

Cypress & DHL

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Rick Fleuren

Software Developer

@richyflowers

Contents

  1. Docker compose

  2. Test parallelization on Jenkins

  3. Test parallelization via Docker

  4. Cypress pitfalls

  5. Q & A

Q & A: use Sli.do event code

#cydhl

or direct link https://app.sli.do/event/vcay1jyw

The problem           was facing

The stack

Docker / docker-compose

API

Kafka

Label

Capabilities

Cypress docker-compose intro

Simple: run web application and Cypress locally

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

Cypress docker-compose intro

Simple: run web application in Docker container and Cypress locally

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

Docker container

Cypress docker-compose intro

Medium: run web application in Docker container and Cypress in another Docker container

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

Docker container

Docker container

version: '3'
services:
  web:
    image: apache
    build: ./webapp
    container_name: apache
    restart: always
    # we can see the server running at "localhost:8080"
    ports:
      - "8080:80"

  e2e:
    image: cypress
    build: ./e2e
    container_name: cypress
    depends_on:
      - web
    # note: inside e2e container, the network allows accessing
    # "web" host under name "web"
    # so "curl http://web" would return whatever the webserver
    # in the "web" container is cooking
    # see https://docs.docker.com/compose/networking/
    environment:
      - CYPRESS_baseUrl=http://web
    command: npx cypress run
    # mount the host directory e2e/cypress and the file e2e/cypress.json as
    # volumes within the container
    # this means that:
    #  1. anything that Cypress writes to these folders (e.g., screenshots,
    #     videos) appears also on the Docker host's filesystem
    #  2. any change that the developer applies to Cypress files on the host
    #     machine immediately takes effect within the e2e container (no docker
    #     rebuild required).
    volumes:
      - ./e2e/cypress:/app/cypress
      - ./e2e/cypress.json:/app/cypress.json

docker-compose.yml

Cypress docker-compose intro

Medium: Cypress in Docker container and web application anywhere

Run Web application

Run Cypress

Host machine (Mac, Windows, Linux)

Docker container

services:
  zookeeper:
    image: wurstmeister/zookeeper
    ...

  kafka:
    image: wurstmeister/kafka
    ...
    
    
  dhl-cap:
    image: dhlparcel/dhl-parcel-capability-service:${DOCKER_COMPOSE_CAP_VERSION:-latest}
    ...
  
  dhl-lab:
    image: dhlparcel/dhl-parcel-label-service:${DOCKER_COMPOSE_LAB_VERSION:-latest}
    ...
  
  dhl-api:
    image: dhlparcel/dhl-parcel-api:${DOCKER_COMPOSE_API_VERSION:-latest}
    ...

... all sorts of micro services ...

  dhl-load-data:
    image: dhlparcel/dhl-parcel-build-postgres:latest
    ...

The local stack

docker-compose.yml

Overrides

services:
  dhl-e2e-cypress:
    image: dhlparcel/dhl-parcel-cypress:4.2.0
    ipc: host
    command: "sh -c '/var/mdp/scripts/compose/run-cypress.sh'"
    volumes:
      - ./:/var/mdp:rw
    environment:
      - CI_BUILD_ID=${CI_BUILD_ID}
      - KEY=${MY_CYPRESS_KEY}
    links:
      - dhl-api:localhost.dhlparcel.nl
      - dhl-api:localhost.dhlparcel.be
      - dhl-api:localhost.dhlparcel.pt
      - dhl-api:localhost.dhlparcel.es
      - dhl-api:localhost.dhlparcel.eu
      - dhl-api:localhost.dhlparcel.ch
      - dhl-api:localhost.dhlparcel.se
      - dhl-admin-api:localhost.admin.dhlparcel.nl
    depends_on:
      - dhl-load-data

docker-compose.jenkins.yml

#!/usr/bin/env bash

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"
cd /var/mdp/${PROJECT}/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --key ${KEY}

run-cypress.sh

def composeBuild = "-f docker-compose.yml -f docker-compose.jenkins.yml --project-name ${projectBuildNr}"


Jenkinsfile

      try {
          stage("E2E") {
           sh "docker-compose ${composeBuild} run --rm --no-deps dhl-e2e-cypress"
          }
      } finally {
        sh "docker-compose ${composeBuild} logs --tail='all'"
        junit "**/e2e_cypress/test-results.xml"
      }
      
 docker-compose run --help
 
    --no-deps             Don't start linked services.
    --rm                  Remove container after run. Ignored in detached mode.
    
    

Start docker-compose

The challenge

4 different frontend project

(2 React apps, 2 Angular apps)

B2X

~ 6 min

C2C

~ 2 min

PWA

~ 3 min

Admin

~ 1 min

400+ release to production / year

Consecutive run: ~10 mins!

Slow and you had to wait for it twice:

Once for your pull request, once for the master build

Fast delivery, slow builds?

test codebase

Tried our own version of parallelisation

Tried our own version of parallelisation

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}


parallel(
  "B2X-1": {
    stage("E2E - B2X - 1") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-1"
    }
  },
  "B2X-2": {
    stage("E2E - B2X - 2") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-2"
    }
  },
  "C2C": {
    stage("E2E - C2C") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c"
    }
  },
  "Admin": {
    stage("E2E - Admin") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-admin"
    }
  },
  "PWA": {
    stage("E2E - PWA") {
      sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa"
    }
  },
)
  dhl-e2e-b2x-suite-1:
    image: dhlparcel/dhl-parcel-cypress:4.2.0
    ipc: host
    command: "sh -c '/var/mdp/scripts/compose/run-cypress.sh --spec='cypress/integration/suite-1-*"
    volumes:
      - ./:/var/mdp:rw
    environment:
      - PROJECT=business-site
    links:
      - dhl-api:localhost.dhlparcel.nl
      - dhl-api:localhost.dhlparcel.be
      - dhl-api:localhost.dhlparcel.pt
      - dhl-api:localhost.dhlparcel.es
      - dhl-api:localhost.dhlparcel.eu
      - dhl-api:localhost.dhlparcel.ch
      - mailcatcher:mailcatcher
    depends_on:
      - dhl-load-data

Homebrew test run: ~7 mins!

 

 

Other problems

 

 

  • B2X: Everything was added in suite-1
  • People needed to be taught
  • Less tests being written

Listen to friendly neighbourhood Cypress

How to add more 'machines'?

Jenkins parallelisation

Let Jenkins do the work

 

Jenkins parallelisation

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}


parallel(
  "B2X-1-2": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-1" }
  },
  "B2X-1-1": {
    ...
  },
  "B2X-1-3": {
    ...
  },
  "B2X-1-4": {
    ...
  },
  "B2X-2-1": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite-2" }
  },
  "B2X-2-2": {
    ...
  },
  "B2X-2-3": {
    ...
  },
  "C2C": {
    stage("E2E - C2C") {  sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c" }
  },
  "C2C": {
   ...
  },
  "Admin": {
   ...
  },
  "PWA-1": {
    stage("E2E - PWA") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa" }
  },
  "PWA-2": {
    ...
  },
  "PWA-3": {
    ...
  },
  "PWA-4": {
    ...
  },
)

Test run: ~ 6 min

Grouping B2X

parallel(
  "B2X-1": {
    stage("E2E - B2X - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-b2x-suite" }
  },
  "B2X-2": {
    ...
  },
  "B2X-3": {
    ...
  },
  "B2X-4": {
    ...
  },
  "B2X-5": {
    ...
  },
  "B2X-6": {
    ...
  },
  "B2X-7": {
    ...
  },
  "C2C": {
    stage("E2E - C2C") {  sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-c2c" }
  },
  "C2C": {
   ...
  },
  "Admin": {
   ...
  },
  "PWA-1": {
    stage("E2E - PWA") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-pwa" }
  },
  "PWA-2": {
    ...
  },
  "PWA-3": {
    ...
  },
  "PWA-4": {
    ...
  },
)

Tests run: ~5.5 min

CPU

C2C / Admin end their run

PWA

4

2

1

Time

Load

CPU

4

2

1

Time

Load

Wasted CPU

Grouping the test runs

 

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"


cd /var/mdp/business-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${B2X_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/pwa-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${PWA_KEY} --ci-build-id ${CI_BUILD_ID} 

cd /var/mdp/admin-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${ADMIN_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${C2C_KEY} --ci-build-id ${CI_BUILD_ID} 

run-cypress.sh

Machines

B2X

B2X

B2X

PWA

PWA

C2C

...

B2X

B2X

PWA

PWA

C2C

C2C

...

B2X

B2X

B2X

PWA

PWA

C2C

...

B2X

B2X

PWA

PWA

C2C

Admin

...

B2X

B2X

B2X

B2X

C2C

C2C

...

Grouping all tests

stage('Init e2e') {
   sh "docker-compose ${composeBuild} up dhl-load-data"
}

parallel(
  "E2E-1": {
    stage("E2E - 1") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-2": {
    stage("E2E - 2") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-3": {
    stage("E2E - 3") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-4": {
    stage("E2E - 4") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-5": {
    stage("E2E - 5") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-6": {
    stage("E2E - 6") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-7": {
    stage("E2E - 7") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
  "E2E-8": {
    stage("E2E - 8") { sh "docker-compose ${composeBuild} ${runOnly} dhl-e2e-cypress" }
  },
)

CPU

Time

Load

Tests run: ~4.5 min

Problems:

init step needed before run, risk of running twice

quite verbose in syntax

 

Docker parallelisation

Let Docker do the work

Jenkins

stage("Run E2E") {
  sh "docker-compose ${composeBuild} up --scale dhl-e2e-cypress=8 dhl-e2e-cypress"
}

Tests run: ~3.5 min

We done?

Builds marked successful who were actually failing

Jenkins

Jenkins only looked at the first exit code of the first container that stopped

First container ended successfully?

Build green, carry on

Passing the error codes

 

while ! timeout 1 bash -c "echo > /dev/tcp/dhl-api/5000" &> /dev/null; do
  echo "Waiting dhl-api to launch on 5000..."
  sleep 1
done

echo "Dhl-api launched"
err=0
trap '(( err += $? ))' ERR


cd /var/mdp/business-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${B2X_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/pwa-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${PWA_KEY} --ci-build-id ${CI_BUILD_ID} 

cd /var/mdp/admin-site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${ADMIN_KEY} --ci-build-id ${CI_BUILD_ID}

cd /var/mdp/site/e2e_cypress && /home/node/node_modules/.bin/cypress run --record --parallel --key ${C2C_KEY} --ci-build-id ${CI_BUILD_ID} 

exit "$err"

run-cypress.sh

Trap

err=0
trap '(( err += $? ))' ERR

...

exit err

trap defines and activates handlers to be run when the shell receives signals or other special conditions.

Jenkins

"Run E2E": {
  stage("Run E2E") {
    sh "docker-compose ${composeBuild} up --scale dhl-e2e-cypress=8 dhl-e2e-cypress"
    sh "docker-compose ${composeBuild} ps"
    CYPRESS_ERROR_CODE = sh (
        script: "docker-compose ${composeBuild} ps -q 
                 | xargs docker inspect -f '{{ .State.ExitCode }}' 
                 | grep -v '^0' 
                 | wc -l 
                 | tr -d ' '",
        returnStdout: true
    ).trim()

    echo "Number of Cypress tests failed: ${CYPRESS_ERROR_CODE}"

    if (CYPRESS_ERROR_CODE.toInteger() != 0) {
        error("Cypress tests failed: ${CYPRESS_ERROR_CODE}")
    }
  }
},

Jenkins

sh "docker-compose ${composeBuild} ps"
docker-compose ${composeBuild} ps -q 			# Only print the IDS
| xargs docker inspect -f '{{ .State.ExitCode }}' 	# Get all the exit codes
| grep -v '^0' 						# Invert match anything other than 0
| wc -l 						# Line count those results
| tr -d ' '						# Remove all the spaces

Parse it to Integer, and process in Jenkins

 

if (CYPRESS_ERROR_CODE.toInteger() != 0) {
  error("Cypress tests failed: ${CYPRESS_ERROR_CODE}")
}

Summary

Test run Time
Consecutive ~10 mins
Homebrew parallelisation ​~7 mins
Jenkins parallelisation ~4.5 mins
Docker parallelisation ~3.5 mins

Cypress Pitfalls

Things we learned using Cypress

.wait('@get-orders')

.wait('@get-settings')

Wrong

.wait(['@get-orders', '@get-settings'])

Right

.wait(1000)

Wrong

.wait('@specific-xhr')

 

or

 

.get('div.class')

Right

First rule of parallelization:

Make your tests autonomous

Cypress does not know everything you're doing

Think before you add more 'machines'

 

 

If you can't spare the resources, your tests will only perform slower

Know when to stop optimizing

Please enjoy the opening tune for the rest of your day!

Gleb Bahmutov

Vp of Engineering

@bahmutov

Rick Fleuren

Software Developer

@richyflowers

Q & A 

time

Cypress and DHL

By Cypress.io

Cypress and DHL

In this webcast, Murat Ozcan, Test Lead for Siemens SW Hub, and Gleb Bahmutov, VP of Engineering at Cypress, will show how the Siemens SW Hub Cloud team uses Cypress to run hundreds of tests across multiple platforms to ensure all systems are working correctly for their Smart Infrastructure cloud product: Building Operator. Video at https://www.youtube.com/watch?v=aMPkaLOpyns

  • 483
Loading comments...

More from Cypress.io