&

Amir Rustamzadeh

Director of DX

@amirrustam

WEBCAST

Building an E2E Strategy that Scales

Ask Questions at Sli.do

event code #cy-faire

David Mamujee

Software Engineer

github.com/dmamujee

About me

  • Front-end Engineer @ Faire
  • Located in Waterloo, Ontario, Canada

github.com/dmamujee

Agenda

  • Implementation and structure of our E2E suite
  • Flaky test management
  • Ways we leverage E2E suite

Ask Questions at Sli.do

event code #cy-faire

  • Founded in 2017, Faire is a wholesale marketplace connecting independent brands and retailers. Currently serving over 250,000 retailers and 30,000 brands.
  • Our mission is to empower entrepreneurs to chase their dreams
  • Our aim is to grow and support the wholesale industry and make it easier for local retailer and wholesale brands to find one another and grow their businesses
  • Raised $260M in Series F round at $7B valuation in June 2021

Why invest in E2E testing?

  • Had existing front-end component and unit tests, but no tests interacting with the backend
  • When releasing to production, we had unscalable manual testing script
  • As engineering team scales and product complexity increases, need tool to ensure quality

Why use Cypress?

  • Easy to get started
  • Good documentation and examples, easy for new engineers to ramp-up
  • Easy to debug and manage tests with Cypress Dashboard
  • Easy to scale with parallelization

E2E repo organization

  • Started with mono-repo for FE
  • Migrated to multi-repo for various FE areas
  • Used git submodule to have common implementation of test utilities
  • Planning to migrate to use internal npm package with shared e2e utilities, have test implementation per repo

Test parallelization

Flaky Test Management ❄️

Pros βœ…

  • Simple to setup and configure
  • All latest and greatest changes on staging
  • Closely resembles production
  • Flags breakages to staging

Cons 🚩

  • Prone to transient breakages
  • Configuration values can be modified, breaking tests
  • Staging data can be unstable

Testing Against Staging

Cypress Cucumber Preprocessor πŸ₯’

Feature: Orders

  Background:
    Given a brand exists
    And the brand has 5 products
    And a retailer exists
    And I am logged in as a retailer

  Scenario: Return Order
    Given I order and return items
    Then Test order page reflects returned items

  Scenario: Report Issue
    Given I create a new order
    And I report an issue on the order
    Then I view the order and see the reported issue
  • Creates human-readable definition of the test scenario

Cypress Cucumber Preprocessor πŸ₯’

Feature: Orders

  Background:
    Given a brand exists
    And the brand has 5 products
    And a retailer exists
    And I am logged in as a retailer
  
  @Ignore
  Scenario: Return Order
    Given I order and return items
    Then Test order page reflects returned items

  @Flaky
  Scenario: Report Issue
    Given I create a new order
    And I report an issue on the order
    Then I view the order and see the reported issue
  • Creates human-readable definition of the test scenario
  • Creates infrastructure to tag tests

Using the Cucumber Tags

  • Want to block PRs on E2E test failure
  • Don't want to block PRs on flaky failures
  • Create "@Flaky" tag for unstable tests. Don't block PR merging when flaky tests fails
  • Flaky test demotion/graduation

Current initiatives to reduce flake ❄️

  • Run E2E test suite against a local backend instance
  • Only run relevant E2E tests in a PR
    • ​Create relevancy map using istanbul.js to determine which files an E2E tests uses
    • On PRs, only run E2E tests that use the files being changed

Ways we leverage our E2E test suite

Testing configuration changes βš™οΈ

  • Use setting values for experimentation, feature flag, and configuration values
  • Modifying these values can break E2E test suite
  • Modifying setting values can cause regressions
  • Before changing setting value, can run E2E suite with updated value

Cypress Cucumber Preprocessor πŸ₯’

  Scenario Outline: Sign up with specific countries
    Given I am located in "<country>"
    When I visit the brand page
    Then the correct products should display with the right price: "<price>"
    When I click on the product tile button
    And I sign up with
      | country   | storeZipCode  | language   |
      | <country> | <postal_code> | <language> |
    Then I should be on the product details page
    Examples:
      | country | postal_code | language | price   |
      | USA     | 90210       | ENG      | $100    |
      | DEU     | 51149       | DEU      | 100 €   |
      | FRA     | 75001       | FRA      | 100 €   |
      | DEU     | 51149       | ENG      | €100    |
      | FRA     | 75001       | ENG      | €100    |
      | GBR     | SW1A 2AA    | ENG      | Β£100    |
  • Create data table for easy definition of multiple test scenarios
  • Easily test many different combinations of mobile/desktop with different languages

Testing Emails πŸ“«

  • Using E2E tests, test triggering and content of emails
  • Tests functionality of email infrastructure

Continuous Deployment πŸ”

  • Goal is to create stable infrastructure to deploy frontend and backend releases automatically with confidence

Q & A time

Amir Rustamzadeh

@amirrustam

Building an E2E Strategy that Scales

David Mamujee

Software Engineer

Thank you πŸ‘

github.com/dmamujee

Director of DX

Cypress and Faire

By Cypress.io

Cypress and Faire

  • 2,412