& 

Gleb Bahmutov

VP of Engineering

@bahmutov

WEBCAST

Tobias Mueller

Sr. Software Engineer

Haydeé Muñoz De la Rocha

Sr. Frontend Engineer

@userlaneHQ

@userlaneHQ

cypress.slides.com/cypress-io/userlane

Questions at Slido.com #cyuserlane

Contents

  • About Userlane
  • Before Cypress
  • Moving to Cypress
  • Scaling up and dealing with flaky tests
  • Questions and answers

Questions at Slido.com #cyuserlane

Userlane

Digital Adoption Platform

With Userlane’s interactive step-by-step guides, anyone
can instantly use any software without formal training.

How does it work?

 

No custom development

Userlane runs on top of every browser-based application 

 

Scalable content creation

No-code editor and dashboard allow everyone in an organization to create content

Tech Stack 

  • React + Redux application
  • Microservice backend running Kubernetes
  • Sandbox: A collection of different web technologies for testing

 

User facing app

Team structure

  • Userlane was founded 4 years ago
  • Only 4 full time engineers less than one year ago
  • Now 14 developers
  • New QA team - 3 people

 

Growing team, setting processes

Before Cypress...

  • Testcafe
  • Codeship
  • Production Environment
  • Using testing properties 

Initial setup

Before Cypress...

Difficulty debugging

Before Cypress...

Slow and unreliable tests

  • Flaky and slow tests

  • Difficulty debugging

  • Uphill battle

  • No developer buy-in

Moving to Cypress...

Nice debugging capabilities

Moving to Cypress...

Nice dashboard with analytics

Moving to Cypress...

Easier syntax

Listening to route response in Testcafe

Listening to route response in Cypress

Moving to Cypress...

True parallelization

Running concurrently

Running in parallel

Moving to Cypress...

The ultra fast migration

  • Migrated 37 specs

  • Mini hackathon

  • Done within a week

  • 37 specs to 100 in about a month
  • Still running against production, now in parallel

 

Pipelines were starting to fail more often (up to 50% failures due to flaky tests)

Scaling it up

How can we fix this?

Reducing flaky tests

Using a dedicated environment

Issues:

  • Impacted production performance

  • Risk of impacting customers

Solution:

  • Dedicated testing environment

  • Nightly deployment + reset

Reducing flaky tests

Challenges with parallelization

Issues:

  • Pipelines running in parallel were interfering with each other

  • Required Data was deleted/modified

Solution:

  • Isolate test runs, make tests independent

  • Tests inject all required data

Reducing flaky tests

Challenges with parallelization: Testing concept

  1. Initialization:  

    Create a new test bed (property) with a clean but well defined state 

  2. Setup:

    Add all the data that your test requires

  3. Test & Verify:

    Let the test run and check if it did what it was supposed to do

Reducing flaky tests

Challenges with parallelization: Automated test setup

Reducing flaky tests

Best practices for writing tests

Issues:

  • Many different reasons for flaky tests
  • No internal guidelines for our application

Solution:

  • Understand why tests might fail (race conditions)

  • Create best practices for YOUR application

Reducing flaky tests

Best practices for writing tests

Some examples:

  • Don't use cy.wait(Number) -> wait for aliased route instead
  • Again: wait for aliased routes
  • Don't expect ordered routes -> use cy.wait([route1, route2])
  • ​Check your code -> your test might be perfectly fine 

More information: https://cypress.io/blog/tag/flake/

Reducing flaky tests

Native test retries

Issues:

  • Some flaky tests are really hard to fix, takes too much effort
  • No time for rewriting legacy tests

Solution:

  • Use test retries to keep test runs from failing

  • Native support in Cypress 5.0
  • High number of retries, low timeout values

Reducing flaky tests

Keeping an eye on it

Keeping it easy

Custom commands

Specific value

Click on the element

overlay.e2e.spec.ts

overlayTutorial.json

Keeping it easy

Automated accessibility testing

Keyboard Navigation & Focus

Static checks using Axe

The outcome

Failure rate

  • Dedicated testing environment
  • Isolating tests
  • Dedicated test setup by injecting data 

The outcome

Number of specs

  • Integrating E2E into the development flow
  • Dedicated QA team for writing tests
  • Now over 300 test cases
  • 75% code coverage

The outcome

Pipeline duration

  • Parallelized tests in pipelines
  • Average duration per spec from 24s to 4s

https://cypress.slides.com/cypress-io/userlane

Q & A time

@userlaneHQ

Cypress and Userlane

By Cypress.io

Cypress and Userlane

Userlane's interactive, step-by-step guides help users instantly use any software without formal training. Find out how the team uses Cypress to increase stability, improve performance, and ensure quality for their CI/CD process while maintaining developer happiness.

  • 1,198
Loading comments...

More from Cypress.io