Cypress & Lightstep

Gleb Bahmutov

Vp of Engineering

@bahmutov

WEBCAST

Vp of Engineering

@bahmutov

Ted Pennings

Product Engineer

@thesleepyvegan

Jonah Moses

Product Engineer

@jonahmoses

https://cypress.slides.com/cypress-io/lightstep/

Jonah

Ted

Product Engineer @ Lightstep

Portland, Oregon

12 years engineering experience

3 years experience with automated testing, Cypress and Selenium

Writes a lot of React code

Likes gardening 🌱

Product Engineer @ Lightstep

Portland, Oregon

6 years engineering experience

2 years experience with automated testing and Cypress

Writes a lot of React code

Has a giant swimming pool

true story

Your system is complex and getting more complex

800TB

Processed Daily

Q & A: use Sli.do event code

#cylightstep

or direct link https://app.sli.do/event/jbmiwnpn

  • Previous Process

  • Adopting Cypress

  • See it in action

  • Facets of Testing

  • We ❤️ Metrics

  • What's next and Questions

Agenda

Ask questions at sli.do #cylightstep

A Google Doc 😭

Previous Process

Ask questions at sli.do #cylightstep

Previous QA Process

Previous QA Process

insert screenshot

Previous QA Process

2 hours!

For every deploy!

How long did this take?!

  • Only tested one environment ... and it wasn't a prod environment!

  • Anxiety regarding doing deploys

    • Large # of PRs per deploy

    • Nobody wanted to deploy 

      • once a week, maybe.

  • Tests were error prone (because humans)

Something had to change

Adopting Cypress

Ask questions at sli.do #cylightstep

Where did we start?

First we proved Cypress could work locally.

Write a spec to login and assert one thing.

Figure out where to run it (CI/CD).

Run it often.

Next we added a few tests, in important parts of product

Aim to replace the most tedious manual tests

We had strong support. But not much time allocated.

Deployments setup

We run Cypress tests after every deployment.

We do not block deployment pipelines on Cypress test results

All test failures are reported in a Slack channel

Wait to add PR checks until things are very stable!*

✅ Use test parallelization

✅ Use Cypress.io Dashboard

* More about this later

(Slack webhooks coming to Cypress Dashboard soon)

Knowledge Sharing

Open, frequent communication

Slack channels + automated posts

PR reviews to share knowledge

We're still iterating. Things still fail, which we'll talk more about later.

Iterate

2 weeks: Most painful tests automated

2 months: Manual QA process eliminated

3 months: PR checks added

Examples and Demos

Ask questions at sli.do #cylightstep

Deployments

Deployments

describe('Deployments Tab', () => {
  beforeEach(() => {
    cy.login().visit('/').getByTestId('service-directory');
  });

  it('renders the tab with many chart rows', () => {
    cy.getByTestId('deployments-tab')
      .getByTestId('deployments-tab-chart-row')
      .its('length')
      .should('be.at.least', 2);
  });

  it('renders charts with data (latency, errors, rate)', () => {
    const options = { timeout: CHART_LOADING_TIMEOUT };
    cy.getByTestId('latency-chart').assertChartHasLineData({ options });
    cy.getByTestId('error-chart').assertChartHasLineData({ options });
    cy.getByTestId('rate-chart').assertChartHasLineData({ options });
  });

  it('displays chart tooltips', () => {
    const options = { timeout: CHART_LOADING_TIMEOUT };
    cy.getByTestId('latency-chart', { options })
      .findChartSurfaces()
      .first()
      .trigger('mouseover')
      .getByTestId('deployments-chart-tooltip')
      .should('be.visible');
  });

  it('can change the chart time window', () => {
    const options = { timeout: CHART_LOADING_TIMEOUT };
    cy.getByTestId('time-window-control')
      .click()
      .getByTestId('time-window-menu-item')
      .eq(3) // arbitrary
      .then(($el) => {
        // $el is a jQuery-wrapped element whose attributes we read for later assertions
        const expectedTimeWindowKey = $el.attr('data-test-value');
        cy.wrap($el).click(); // re-wrap with Cypress so we can click it
        cy.getByTestId('time-window-control').should(
          'have.attr',
          'data-time-window-key',
          expectedTimeWindowKey
        );
        cy.getByTestId('latency-chart')
          .should(
            'have.attr',
            'data-chart-time-window-key',
            expectedTimeWindowKey
          )
          .findChartSurfaces()
          .assertChartHasLineData({ options });
      });
  });
});

// These command definitions belong in support/commands.js or similar
Cypress.Commands.add(
  'assertChartHasLineData',
  { prevSubject: 'element' },
  (subject, { selector = '.recharts-line path', ...options }) => {
    return cy
      .get(selector, { withinSubject: subject, ...options })
      .first()
      .invoke('attr', 'd')
      .its('length')
      .should('be.at.least', 2 * 1000);
  }
);

Cypress.Commands.add(
  'findChartSurfaces',
  { prevSubject: 'element' },
  (subject, options) => {
    return cy.get('.recharts-surface:not([data-test-id=chart-placeholder])', {
      withinSubject: subject,
      ...options,
    });
  }
);

Explorer

Explorer

 

  • Explorer - service diagram
  • Mention histogram filtering; code show in next section

(Ted)

TODO: Find code snippets and cypress.io recording links

it('displays a service diagram with nodes and edges', () => {
  cy.awaitServiceDiagram();
  cy.getByTestId('service-diagram')
    .should('be.visible')
    .getByTestId('service-diagram-node')
    .its('length')
    // node count varies based on data source
    .should('be.at.least', 3)
    .getByTestId('service-diagram-edge')
    // at least one edge between nodes
    .should('be.visible');
});

Query Builder

Query Builder

describe('Tags', () => {
  it('Handles tag selection, removal, and querying', () => {
    cy.server();
    cy.route({
      method: 'POST',
      url: '**/api/v1/suggestion_query/query',
      response: {
        done: true,
        suggestions: [
          {
            suggestion_string: 'tag:"client.browser"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:"card.type"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:card.type="Visa"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:card.type="MasterCard"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:card.type="AMEX"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:client.browser="Chrome"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:client.browser="Safari"',
            score: 1020,
            match: [0, 4],
          },
          {
            suggestion_string: 'tag:client.browser="Firefox"',
            score: 1020,
            match: [0, 4],
          },
        ],
      },
    }).as('suggestionsRequest');

    cy.route({
      method: 'POST',
      url: '**/api/v1/explore/create',
      response: {},
    }).as('createRequest');

    cy.getByTestId('query-builder-remove-tag-filter')
      // confirm a tag row hasn't been started
      .should('not.be.visible')
      .getByTestId('query-builder-add-tag-filter')
      .click()
      .getByTestId('query-builder-remove-tag-filter')
      // confirm tag row
      .should('be.visible')
      .getByTestId('query-builder-tag-row')
      .should('be.visible')
      .within(($tagRow) => {
        // confirm both key and value text fields visible
        cy.getByTestId('query-text-field')
          .should(($textField) => {
            expect($textField).to.have.length(2);
          })
          // Find first text field for tag key
          .getByTestId('query-text-field')
          .eq(0)
          .click()
          // Confirm suggestions menu is open
          .getByTestId('suggestions-menu')
          .should('be.visible')
          .wait('@suggestionsRequest')
          .getByTestId('suggestions-menu-item')
          // only the two stubbed out suggestions appear
          .should(($suggestion) => {
            expect($suggestion).to.have.length(2);
          })
          // select first option
          .eq(0)
          .click()
          // Find second text field for tag value
          .getByTestId('query-text-field')
          .eq(1)
          .click()
          // Confirm suggestions menu is open
          .getByTestId('suggestions-menu')
          .should('be.visible')
          .wait('@suggestionsRequest')
          .getByTestId('suggestions-menu-item')
          .should(($suggestion) => {
            expect($suggestion).to.have.length(6);
          })
          // add two suggestions
          .eq(0)
          .click()
          .getByTestId('suggestions-menu-item')
          .eq(3)
          .click()
          // confirm chips were added
          .getByTestId('query-term-chip')
          .should('be.visible')
          // confirm two suggestions were applied
          .should(($chip) => {
            expect($chip).to.have.length(2);
          })
          // delete the last suggestion chip
          .getByTestId('query-term-chip')
          .eq(1)
          .find('svg')
          .click()
          .getByTestId('query-term-chip')
          .should('be.visible')
          // confirm last suggestion chip was deleted
          .should(($chip) => {
            expect($chip).to.have.length(1);
          });
      })

      .getByTestId('query-bar')
      // make sure the query bar has the correct query
      .contains('"card.type" IN ("AMEX")')
      .getByTestId('query-builder-run-query')
      .click()
      .wait('@createRequest')
      .its('request.body')
      .its('raw_query')
      // make sure the request we send is correct
      .should('eq', '"card.type" IN ("AMEX")');
  });
});

www.lightstep.com/sandbox

Facets of Testing

Ask questions at sli.do #cylightstep

Data Density

As an observability product,

data visualization is the core of Lightstep's product, and our data is highly dynamic...

 

Data was our biggest challenge for testing.

Data Density

  • Most difficult (and unreliable) part of our tests
  • Prefer unit tests for data correctness
  • Use stubbed API responses: cy.server()
  • Predictable data generators
  • Fuzzy assertions
    • eg, at least 2 nodes in diagram
  • Limit test assumptions
    • eg, click first item in list, assert something is visible
  • Only modify resources created in the same test
    • For uniqueness, add unix timestamps to resource names​
  • Is data correctness your responsibility? In browser tests?

Basic Interactions

  • Trigger interactions normally with click, type
  • Use data-* tags on elements for statuses
    • data-loading
  • Chain cy.get() assertions for timeouts and error specificity

getByTestId() is a helper command we wrote for cy.get('[data-test-id=$val]')

Drag Interactions

Drag Interactions

  • Use cy.trigger() on an element
    1. mouseup
    2. mousemove
    3. mousedown
  • Try not to make your final assertion too specific
  • Visibility rules may require you to target a different element, as in complex SVGs (like charts)

Commands

We wrote a bunch of commands. Like 25.

  • Repetitive tasks (login, clicking a row in table)
  • Repetitive assertions (table has many rows; see snippet)
  • Centralize logic, like timeouts (await tooltip animation)

Tip: Command Naming

  • await*
  • assert*
  • do*
  • get* and find*

Feature Flags

We use LaunchDarkly for feature flags. A lot.

 

It's possible to test different variants if you add more user logins for Cypress. Target specific Cypress users for your feature flags variants and use that login.

 

Testing feature flag variants can add to test maintenance burden because you have to remove all code that references a flag before deleting it. It can take time (~days) for this code be fully gone if you run Cypress tests on PRs.

 

If you can, avoid testing feature flags because it adds complexity and brittleness.

 

How to Write Automation Tests for Feature Flags with Cypress.io and Selenium

https://blog.optimizely.com/2020/03/26/automation-testing-feature-flags/

Look for Cypress + Optimizely webinar in June

User Roles, Environments

We have a JSON map of configuration for users and environments

  • Four environments
    • URL for each, other metadata
  • Each environment has 3 users
    • admin
    • viewer
    • beta, for feature flags

User Roles, Environments

Run different tests for different environment depending on OS, CI, environment variable: https://github.com/cypress-io/cypress-skip-test

CYPRESS_ENVIRONMENT=staging npx cypress run
import {onlyOn} from '@cypress/skip-test'
const stubServer = () => {
  cy.server()
  cy.route('/api/me', 'fx:me.json')
  cy.route('/api/permissions', 'fx:permissions.json')
  // Lots of fixtures ...
}
it('works', () => {
  onlyOn('staging', stubServer)
  ...
})

User Roles, Environments

https://cypress.slides.com/cypress-io/siemens-case-study

User Roles, Environments

  • Cypress env var from runner (CI/CD) that sets environment
  • Helpers lookup config based on environment (via Cypress.env).
  • Override visit command to specify base URL; tests have cy.visit('/account')

Metrics

Ask questions at sli.do #cylightstep

165 tests in 36 specs

8-12k tests run per day

All frontend engineers have contributed tests

 

80% of engineering org are dashboard.cypress.io users

Number of deployments


Time to validate a deployment


PRs per deployment

0
 Advanced issue found
▲
 

2 hours -> 5 minutes

Less than 1 per week -> 5-8 per week

25 PRs -> 5 PRs

This is how we built the confidence to deploy faster

Ask questions at sli.do #cylightstep

The Future

Future is bright

Visual Testing

cy.visit('/page')
cy.get('...').click()
  // other actions
cy.get('...')
  .should(...)
cy.get('...')
  .should(...)
cy.get('...')
  .should(...)
cy.get('...')
  .should(...)
cy.get('...')
  .should(...)
cy.get('...')
  .should(...)

Visual Testing

cy.visit('/page')
cy.get('...').click()
  // other actions
cy.get('...')
  .should(...)
// the page has updated
cy.visualCheck()

Visual Testing

Happo.io 🦛

https://on.cypress.io/visual-testing

Happo.io codes in 2 days (!)

https://github.com/happo/happo-cypress

Visual Testing

https://github.com/happo/

happo-cypress-example-todomvc

  it('adds 2 todos', function() {
    cy.get('.new-todo')
      .type('learn testing{enter}')
      .type('be cool{enter}')
      .happoScreenshot({ component: 'New todo' });

    cy.get('.todo-list li').should('have.length', 2);
    cy.get('body').happoScreenshot({
      component: 'App',
      variant: 'two items added',
    });
  });

Look for Cypress + Happo webinar in May

Charts & Tooltip Testing

  • More extensive charts UI testing, such as tooltips and cursors
  • Chart data rendering accuracy
  • Visual testing for other visualizations

Stubbing & Test Ownership

  • Once visual testing is setup we will rely more heavily on data stubbing with cy.server + cy.route
  • Setup for distributed test ownership
    • Widen circle of champions

Ted Pennings

Product Engineer

@thesleepyvegan

Jonah Moses

Product Engineer

@jonahmoses

https://cypress.slides.com/cypress-io/lightstep/

Q & A: use Sli.do event code

#cylightstep