& 

Gleb Bahmutov

VP of Engineering

@bahmutov

WEBCAST

Tyler Monteith

Engineering Team Lead

Carvana

Arick Hanna

Engineer II, Quality

Carvana

How Carvana Transformed their Testing with Cypress Test Analytics

https://cypress.slides.com/cypress-io/carvana

Questions at Slido.com #carvana

Carvana

Online Used Car Retailer

Carvana is a leading e-commerce platform for buying and selling used vehicles, offering as-soon-as-next-day delivery in 262 markets across the U.S. and pick-up at 26 patented signature car vending machines.

Carvana

Continued

Consumers can browse through more than 20,000 vehicles on carvana.com, then purchase and finance a quality used vehicle in as little as five minutes- all from the comfort and safety of home or on the go via a mobile device. Carvana vehicles come with a 7-day return policy, have undergone a rigorous, 150-point inspection, and have never been in a reported accident.

Technologies Used

  • .NET Heavy Backend
  • React Heavy Frontend
  • Multiple Technology stacks to support

We Support Tons of Apps/Features

Continued

Mobile App Too...

Our Team

  • Carvana was founded in 2013, and did not hire the first Quality Engineer until 2014.

 

  • In 2019 the team grew to 15.

 

  • Currently we have a team of 20!

photo 2019

Before Cypress

  • Selenium
  • C#/.NET
  • POM

Framework

  • VSTS - Azure Pipelines
  • Custom Java App to deploy browsers/nodes

Test Environment

Limited Suite, Highly Customized

  • Our Selenium Suite had ~100 test cases
  • Used custom Zalenium + "Moon" k8s pods to deploy browsers
  • 2+ hour execution time
  • limited reporting and NO video playback :(

Challenges

  1. Selenium was very difficult to debug on our long running e2e tests
  2. Using POM + .NET had a high barrier to entry (no jr QA could write automation)
  3. Single Test Environment
  4. Lack of developer support
  5. Difficult constraints, little online support for testing at scale

Why we chose Cypress

Cypress allowed us to rapidly develop tests in a uniform, low barrier to entry environment.

 

Developers love Javascript!

 

The Dashboard took care of all of our parallelization and analytics needs out of the box.

Initial Wins

  • After porting over 100+ test cases we immediately had a clearer idea of why/where things were failing with the easy-to-digest Dashboard.
  • video + screenshots out of the box!
  • Wired up results to Slack for faster feedback.

Scaling Out

  • We quickly began moving into areas of the site that were historically challenging for us.
  • Some devs started learning/participating in writing test cases!

Scaling Up Cypress

Our Challenges...

  • Parallelization required
  • We needed a single platform for multiple teams to use
  • It had to support multiple test environments
  • It had to support multiple Cypress Projects + Video/Screenshot capabilities.
  • And more...

Cypress Made Parallelization Easy

The Solution

Azure Pipelines within Azure DevOps (Formally VSTS) to build and maintain cloud execution steps

We built a set of 10 Linux VMs using Cypress's best practices/recommendations to run the tests + gather results

Slack Integration for reporting results and sharing failed test runs

npx cypress run --browser chrome --config video=true --group qe-regression-tests --record --key $(KEY) --spec 'cypress/integration/$(SPEC)/**/*' --env configFile=$(ENVIRONMENT) --parallel --ci-build-id $(BUILD.BUILDNUMBER)

Followed Cypress Guides For Parallelization

Analytics Powered Decision Making

Test Suite Size Vs Duration Over Time

Text

Slowest Tests

Failures

Test Retries Created New Opportunities

With Test Retries, we were able to move into areas of our site previously avoided.

it('Submitting a purchase creates a loan application', { retries: 4 }, () => {
    cy.route('POST', '/acquisition/api/v1/purchase/place*').as('placeOrder');
    cy.fixture('purchases/API/AZ_Finance_Purchase.json').then((purchase) => {
      cy.makeEmailAddress(
        purchase.firstName,
        purchase.lastName,
        purchase.homeAddress.state
      ).then((email) => {
        cy.makeNewUserSoftpull(purchase, email)
          .then(() => {
            cy.getUserToken(purchase);
            cy.getToken('Token', purchase);
          })
          .then(() => {
            cy.getUnlockedCoreVehicleId(purchase);
          })
          .then(() => {
            cy.startPurchase(purchase);
          })
          .then(() => {
            cy.postAddress(purchase);
          })
          .then(() => {
            cy.tradeInOrNot(purchase);
          })
          .then(() => {
            cy.postPaymentOptions(purchase);
          })
          .then(() => {
            cy.getFinancingTerms(purchase).then(() => {
              cy.postTerms(purchase);
            });
          })
          .then(() => {
            cy.gapWaiverSelection(purchase);
          })
          .then(() => {
            cy.deliveryOptionSelection(purchase);
          })
          .then(() => {
            cy.selectServiceContract(purchase);
          })
          .then(() => {
            cy.uploadDriversLicense(purchase);
          })
          .then(() => {
            cy.postPaymentCollectionSubscription(purchase);
          })
          .then(() => {
            cy.login(email, Cypress.env('userPassword')).then(() => {
              cy.visit('/procurement/review-order');
              cy.getByCv('styledCheckbox', { timeout: 30000 }).click({
                force: true,
              });
              cy.getByCv('continueToVerifications')
                .click({ force: true })
                .wait('@placeOrder', { timeout: 100000 });
              cy.getCase(purchase).then(() => {
                cy.executeSqlQuery({
                  database: 'CarvanaIsTheBest',
                  query: `SELECT [BestEmployees] FROM [Carvana].[OnlyTheBest] 
			WHERE [EmployeeName] = ‘Arick Hanna’ OR [EmployeeName] = ‘Tyler Monteith’`,
                }).then((result) => {
                  expect(result[0].BestEmployees).to.equal(true);
                });
              });
            });
          });
      });
    });
  });

We can retry an entire suite OR individual tests, flexibility is great!

Test Retries + Analytics = WIN

Test retry analytics has allowed us to determine which tests are flaky

This helps tremendously with development efforts to determine why certain areas of the application are more flaky than others

Debugging Efficiently

it('Autopay discount should never exceed $10', { retries: 4 }, () => {
    cy.route('GET', '/acquisition/api/v1/paymentcollection/subscription').as(
      'paymentSub'
    );
    cy.fixture('purchases/API/AZ_Finance_Purchase.json').then((purchase) => {
      cy.makeEmailAddress(
        purchase.firstName,
        purchase.lastName,
        purchase.homeAddress.state
      ).then((email) => {
        cy.makeNewUserSoftpull(purchase, email)
          .then(() => {
            cy.getUserToken(purchase);
          })
          .then(() => {
            cy.getUnlockedCoreVehicleIdByMinMaxPrice(purchase);
          })
          .then(() => {
            cy.startPurchase(purchase);
          })
          .then(() => {
            cy.postAddress(purchase);
          })
          .then(() => {
            cy.tradeInOrNot(purchase);
          })
          .then(() => {
            cy.postPaymentOptions(purchase);
          })
          .then(() => {
            cy.getFinancingTerms(purchase).then(() => {
              cy.postTerms(purchase);
            });
          })
          .then(() => {
            cy.gapWaiverSelection(purchase);
          })
          .then(() => {
            cy.deliveryOptionSelection(purchase);
          })
          .then(() => {
            cy.selectServiceContract(purchase);
          })
          .then(() => {
            cy.uploadDriversLicense(purchase);
          })
          .then(() => {
            cy.login(email, Cypress.env('userPassword')).then(() => {
              cy.visit('/procurement/down-payment');
              cy.wait('@paymentSub', { timeout: 100000 });
              if (purchase.down > 0) {
                cy.getByCv('manual-account-checboxSelectionContainer').click();
                cy.getByCv('routingNumber').type(purchase.routingNumber);
                cy.getByCv('accountNumber').type(purchase.accountNumber);
                cy.getByCv('accountNumberConfirm').type(purchase.accountNumber);
                cy.getByCv('Button').contains('Submit').click();
                cy.getByCv('authorizationCheckboxContainer')
                  .find('[data-cv-test="styledCheckbox"]')
                  .click();
                cy.getByCv('Button')
                  .contains('Save & Continue')
                  .click()
                  .wait(5000);
              }
              cy.get('[class*="panel__with-autopay"]')
                .find('[data-cv-test="monthlyPaymentAmount"]')
                .invoke('text')
                .then((autopay) => {
                  autopay = autopay.replace(/\D+/, '');
                  cy.get('[class*="autopaystyles__Panel"]')
                    .last()
                    .find('[data-cv-test="monthlyPaymentAmount"]')
                    .invoke('text')
                    .then((regularPayment) => {
                      regularPayment = regularPayment.replace(/\D+/, '');
                      expect(parseInt(autopay)).to.equal(
                        parseInt(regularPayment) - 10
                      );
                    });
                });
            });
          });
      });
    });
  });

The Flaky Tests Dashboard allows us to quickly and efficiently debug failing and flaky tests

Conclusions

  • Decreased test execution time by 72%
  • Significantly less failures (less flaky failures and quicker debugging)
  • Increased the number of tests by 140%
  • ~50% of our engineering teams utilize Cypress to test their apps

Gleb Bahmutov

VP of Engineering

@bahmutov

WEBCAST

Tyler Monteith

Engineering Team Lead

Carvana

Arick Hanna

Engineer II, Quality

Carvana

Thank you

https://cypress.slides.com/cypress-io/carvana