Cypress.io
When testing is easy, developers build better things faster and with confidence.
&
Gleb Bahmutov
VP of Engineering
@bahmutov
WEBCAST
Tobias Mueller
Sr. Software Engineer
Haydeé Muñoz De la Rocha
Sr. Frontend Engineer
@userlaneHQ
@userlaneHQ
Questions at Slido.com #cyuserlane
Contents
- About Userlane
- Before Cypress
- Moving to Cypress
- Scaling up and dealing with flaky tests
- Questions and answers
Questions at Slido.com #cyuserlane
Userlane
Digital Adoption Platform
With Userlane’s interactive step-by-step guides, anyone
can instantly use any software without formal training.
How does it work?
No custom development
Userlane runs on top of every browser-based application
Scalable content creation
No-code editor and dashboard allow everyone in an organization to create content
Tech Stack
- React + Redux application
- Microservice backend running Kubernetes
- Sandbox: A collection of different web technologies for testing
User facing app
Team structure
- Userlane was founded 4 years ago
- Only 4 full time engineers less than one year ago
- Now 14 developers
- New QA team - 3 people
Growing team, setting processes
Before Cypress...
- Testcafe
- Codeship
- Production Environment
- Using testing properties
Initial setup
Before Cypress...
Difficulty debugging
Before Cypress...
Slow and unreliable tests
-
Flaky and slow tests
-
Difficulty debugging
-
Uphill battle
-
No developer buy-in
Moving to Cypress...
Nice debugging capabilities
Moving to Cypress...
Nice dashboard with analytics
Moving to Cypress...
Easier syntax
Listening to route response in Testcafe
Listening to route response in Cypress
Moving to Cypress...
True parallelization
Running concurrently
Running in parallel
Moving to Cypress...
The ultra fast migration
-
Migrated 37 specs
-
Mini hackathon
-
Done within a week
- 37 specs to 100 in about a month
- Still running against production, now in parallel
Pipelines were starting to fail more often (up to 50% failures due to flaky tests)
Scaling it up
How can we fix this?
Reducing flaky tests
Using a dedicated environment
Issues:
-
Impacted production performance
- Risk of impacting customers
Solution:
-
Dedicated testing environment
- Nightly deployment + reset
Reducing flaky tests
Challenges with parallelization
Issues:
-
Pipelines running in parallel were interfering with each other
- Required Data was deleted/modified
Solution:
-
Isolate test runs, make tests independent
-
Tests inject all required data
Reducing flaky tests
Challenges with parallelization: Testing concept
-
Initialization:
Create a new test bed (property) with a clean but well defined state -
Setup:
Add all the data that your test requires -
Test & Verify:
Let the test run and check if it did what it was supposed to do
Reducing flaky tests
Challenges with parallelization: Automated test setup
Reducing flaky tests
Best practices for writing tests
Issues:
- Many different reasons for flaky tests
- No internal guidelines for our application
Solution:
-
Understand why tests might fail (race conditions)
-
Create best practices for YOUR application
Reducing flaky tests
Best practices for writing tests
Some examples:
- Don't use cy.wait(Number) -> wait for aliased route instead
- Again: wait for aliased routes
- Don't expect ordered routes -> use cy.wait([route1, route2])
- Check your code -> your test might be perfectly fine
More information: https://cypress.io/blog/tag/flake/
Reducing flaky tests
Native test retries
Issues:
- Some flaky tests are really hard to fix, takes too much effort
- No time for rewriting legacy tests
Solution:
-
Use test retries to keep test runs from failing
- Native support in Cypress 5.0
- High number of retries, low timeout values
Reducing flaky tests
Keeping an eye on it
Keeping it easy
Custom commands
Specific value
Click on the element
overlay.e2e.spec.ts
overlayTutorial.json
Keeping it easy
Automated accessibility testing
Keyboard Navigation & Focus
Static checks using Axe
The outcome
Failure rate
- Dedicated testing environment
- Isolating tests
- Dedicated test setup by injecting data
The outcome
Number of specs
- Integrating E2E into the development flow
- Dedicated QA team for writing tests
- Now over 300 test cases
- 75% code coverage
The outcome
Pipeline duration
- Parallelized tests in pipelines
- Average duration per spec from 24s to 4s
Q & A time
@userlaneHQ
Cypress and Userlane
By Cypress.io
Cypress and Userlane
Userlane's interactive, step-by-step guides help users instantly use any software without formal training. Find out how the team uses Cypress to increase stability, improve performance, and ensure quality for their CI/CD process while maintaining developer happiness.
