Mike Cataldo

DX Engineer

@mccataldo

WEBCAST

Flaky Test Management with Cypress

Ask Questions at Sli.do

event code #flaky

Cecelia Martinez

Technical Account Mgr

@ceceliacreates

Agenda

  • What is flake?
  • Causes of flake
  • Managing flake
  • Q&A at Sli.do event code #flaky

What is Flake?

What is a flaky test?

A test is considered to be flaky when it can pass and fail across multiple retry attempts without any code changes.

 

For example, a test is executed and fails, then the test is executed again, without any change to the code, but this time it passes.

✅ ? ❌

Flaky Test in Cypress Real World App

github.com/cypress-io/cypress-realworld-app

// Backend source code for "like" functionality
// creates an arbitrary delay on the server to simulate inconsistent response times

   const delay = Math.random() * 5500;
   setTimeout(() => {
     res.sendStatus(200);
   }, delay);

Test case: User A likes a transaction of User B; User B gets notification that User A liked transaction

✅ ? ❌

cy.contains(likesCountSelector, 0);
cy.getBySelLike("like-button").click();
// a successful "like" should disable the button and increment
// the number of likes
cy.getBySelLike("like-button").should("be.disabled");
cy.contains(likesCountSelector, 1);

Test case: User A likes a transaction of User B; User B gets notification that User A liked transaction

✅ ? ❌

Test case: User A likes a transaction of User B; User B gets notification that User A liked transaction

✅ ? ❌

How Flake Impacts a Test Suite

  • Longer deployment process
    • Have to rerun tests or restart CI builds if flake causes a failure
  • Longer debugging process
    • Have to determine if a failure is a "real" failure or flake
  • Reduced confidence
    • Do failures actually represent regressions?
    • Is flake hiding underlying issues in my application or test suite?

Causes of Flake

DOM-related Flake

  • Occurs when there is inconsistency with how elements are rendered to the DOM (or how quickly they are rendered)

DOM-related Flake

Examples:

  • Sometimes an element loads within the cy.get() timeout, other times it doesn't
  • Cypress gets an element, but when it tries to click, the element is disabled because the state hasn't updated yet
  • The DOM re-renders between a cy.get() and an action command, causing the selected element to be detached from the DOM
  • Cypress types in an input field, but the application is slow to process the keypress events and the field value doesn't update completely before clicking 'Submit'

Network-related Flake

  • Occurs when a network request responds inconsistently, either an internal API or a a third-party or serverless endpoint

Network-related Flake

Examples:

  • A slow API response results in a DOM element loading outside of the default timeout (like in our example)
  • A slow API response from a third-party login provider causes tests to fail
  • A microservices endpoint has a cold start/start delay, so it fails the first time but passes on subsequent attempts

Environment-related Flake

Examples:

  • Inaccessible or inaccurate environment variables
  • Running tests across different size machines
  • A failed dependency install in the environment
  • Inconsistent data across staging/QA/dev environments

Managing Flake

DOM-related Flake

Cypress Test Runner

Query Command Retryability

it('adds two items', () => {
  cy.get('.new-todo')
    .type('todo A{enter}')
    .type('todo B{enter}')

  cy.get('.todo-list li')
    .should('have.length', 2)
})

Cypress Test Runner

Query Command Retryability

cy.get('.todo-list li') // command
  .should('have.length', 2) // assertion

✅                         ❌

If the assertion that follows the cy.get() command fails, then the cy.get() command will requery the application's DOM again, and again until the cy.get() command timeout is reached.

If the assertion that follows the cy.get() command passes, then the command finishes successfully.

Cypress Test Runner

Query Command Retryability

Test-writing Best Practices

🧠 Only the last query command is retried

💡Use a single query command instead of chaining

-- cy.get() not retried

cy.get(selector).contains(text)
cy.get(selector).its(selector)

-- Entire query is retried

cy.contains(selector, text)

💡Alternate commands and assertions

-- Test will retry the cy.get() until assertion passes
cy.get(selector)
.should('have.length', 3)

-- Now we know we have the correct element before proceeding
.parent()
.should('contain', text)

Test-writing Best Practices

https://cypress.io/blog/tag/flake/

Network-related Flake

Test-writing Best Practices

🧠 cy.intercept() can spy and stub all network requests

💡Wait for long network requests before proceeding

cy.intercept('GET', '/api/accounts/*').as('getAccount')
cy.visit('/accounts/123')

// Wait up to 30 seconds for request to complete
cy.wait('@getAccount', { timeout: 30000 })
cy.get(selector)
.should('have.text', accountName)

💡Stub inconsistent network requests to control response

// Intercept and stub the response as successful

cy.intercept({ method: 'POST', url: '/inconsistent' }, { success: true })

All types of Flake

Test Retries

Will retry an individual test a specified number of times before failing

// cypress.json
{
  "retries": 1
}

// configure by mode
{
  "retries": {
// Configure retry attempts for `cypress run`
  "runMode": 2,
// Configure retry attempts for `cypress open`
  "openMode": 0
  }
}

Cypress Dashboard Flake Detection

  • Identifies flaky tests AND flake severity based on number of times a test is retried
  • Test retries must be enabled in the Cypress Test Runner to use
  • See artifacts from failed attempts on flaky runs
  • See most common errors, historical flake and fail rates, and test definition for flaky test cases with Analytics

Cypress Dashboard Flake Management

Coming Soon: Test Burn-In

 

  • Identify if new tests are flaky before introducing them to your test suite
  • Cypress Dashboard will identify new tests and automatically retry multiple times to check for flake
  • Fix flaky test cases before they become an issue

GitHub & Slack Flake Alerts

Mike Cataldo

DX Engineer

@mccataldo

Q&A

Ask Questions at Sli.do

event code #flaky

Cecelia Martinez

Technical Account Mgr

@ceceliacreates

Flaky Test Management

By Cecelia Martinez

Flaky Test Management

Flaky Test Management

  • 1,823
Loading comments...

More from Cecelia Martinez