How PlanGrid (Autodesk) Achieved Consistency in Automation Across 20 Development Teams

https://cypress.slides.com/cypress-io/quality-in-1-million-projects

Gleb Bahmutov

VP of Engineering @ Cypress.io

Graeme Harvey

Engineering Manager, Automation Platform

@ PlanGrid (Autodesk Construction Solutions)

Brendan Drew

Tech Lead, Automation Platform

@ PlanGrid (Autodesk Construction Solutions)

About PlanGrid

Used on over one million projects around the world, PlanGrid is the first construction productivity software that allows contractors and owners in commercial, heavy civil, and other industries to work and collaborate from anywhere.

About PlanGrid

Established in 2012
Acquired by Autodesk in 2019
4 Platforms providing a consistent user experience
Web, Android, iOS, Windows
112k Monthly Active Users
- >50% Web

Development organization
Five problems we have solved
1. picking the tool
2. breaking old habits
3. don't reinvent the wheel
4. lessen the maintenance
5. everyone writes tests

Q & A: use Sli.do event code #cyplangrid

or link https://app.sli.do/event/sawiafoj

The Platform Team Model

Platform teams belong to
Frameworks & Infrastructure group
Automation Platform - team of 6 engineers
Support:
- automation frameworks
- pipeline integrations
- across web, Android, iOS & Windows + backend

Our (Web) Development

App Shell model - 10+ independent GitHub repositories
- Each repo is a feature of the app
- Independent build/test/deploy via Jenkins pipelines
- Shared libraries help set standards and make this easier

Methodology: Scrum(ish)
Delivery: Varying degrees of CI/CD via Jenkins & Spinnaker
Technology: React, AWS, Postgres, Redis, Istio, LaunchDarkly

The Branching Strategy

Something Had to Change

Back in 2018...

<50 Selenium tests
Largely written and maintained by QA
Not well understood by dev team
- Gherkin layer of abstraction
- Python Behave framework
- Often skipped upon failure

The New Goal

To provide a testing framework familiar and easy to use for frontend developers, allowing them to write, run and maintain tests without increasing their workload significantly.

Our Must-Haves

Javascript / Typescript - same language as frontend development
- We believe dev teams must own their own tests
Quick execution time
- We believe in blocking merges on tests
Easy debugging
- We believe skipping failures should be an exception, not the norm
CI integration
- We believe in removing as much manual intervention as possible

The Contenders

Webdriver.io

TestCafe

Cypress.io

The Vetting

Have engineers review
- After all, they’re the ones expected to write and maintain tests
A strong preference was seen for Cypress’ runner
- DOM state snapshotting
- Included, familiar tools/techniques
  - Mocha
  - Chai
  - Sinon
Cross-browser was determined as not a requirement
- Data-driven decision
- Based on # of P0/P1 issues reported with a browser based root cause

5 Major Challenges We Encountered

Make our tool the obvious choice
Encourage teams to replace existing tests & break old (bad) habits
Avoid "reinventing the wheel"
Limit maintenance burden to unblock progress
Enable anyone to debug, monitor, & own their tests

1. Make our tool the obvious choice

Free market model
- Teams can use whatever tools they want
Teams are empowered to build, test, release however they want
- Suggested tools, but no mandates (with some exceptions)
Tools often chosen based on senior members' previous experience
- Jest vs mocha
- Sinon vs nock
- Etc.

Integrate with shared react-scripts library
- Part of our fork of create-react-app
Integrate with shared Jenkins build library

Make the easy way the right way

Make our tool the obvious choice

Integrate with react-scripts

Detects if this is CI or local and either opens the GUI test runner or runs headlessly
Runs inside or outside of a docker container

// package.json
...
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "lint": "react-scripts lint",
    "tsc": "react-scripts tsc",
    "publish": "react-scripts publish",
    "format": "react-scripts format",
    "serve:build": "react-scripts serve:build",
    "cypress": "react-scripts cypress",
    "cypress:build": "react-scripts cypress:build",
    "inspect": "react-scripts inspect"
  },

Make our tool the obvious choice

> npm run cypress

> npm run cypress:build

Starts the built application using the same script as npm run serve:build
Runs cypress using the same script as npm run cypress
Useful in CI

💡 Cypress team's tip: create CRA apps using our official template

https://github.com/cypress-io/cra-template-cypress

if (process.env.CI === 'true') {
  cypress
    .run(args)
    .then(results => {
      const testsPassed = results.totalFailed === 0;
      console.log(
        `Cypress tests ${testsPassed
          ? chalk.green('PASSED')
          : chalk.red('FAILED')}.  Result details:`
      );
      console.log(results);
      process.exit(testsPassed ? 0 : 1);
    })
    .catch(ex => {
      console.error(
        chalk.red('Cypress encountered an error and was unable to complete:')
      );
      console.error(ex);
      process.exit(1);
    });
} else {
  cypress.open(args);
}

Integrate with react-scripts

Make our tool the obvious choice

Integrate with shared build library

// Jenkinsfile
@Library("web-build-tools") _

stage("Checkout") {
  GitClean()
  checkout scm
}

runFullBuild(
  projectType: "app"
)

// Jenkinsfile
@Library("web-build-tools") _

stage("Checkout") {
  GitClean()
  checkout scm
}

def cypressConfig = [
  additionalRunVars: "BROWSER=chrome"
]

runFullBuild(
  projectType: "app",
  cypressConfig: cypressConfig
)

Cypress runs out of the box with runFullBuild()

Customization via a cypressConfig object

Make our tool the obvious choice

Integrate with shared build library

Cypress tests are important so they block PRs!
- Scheduled runs require someone to monitor results - too slow of a feedback loop
This naturally enforces:

Tests need to run fast
Tests need to be able to run regardless of the target environment
Test fixes must be checked in with the code that broke them

Make our tool the obvious choice

2. Encourage teams to replace existing tests & break old (bad) habits

Teams had tests
The tests worked
Lots of:
- setup in the UI
- long running tests

some

kind of

Prove the value

Start porting old tests
- Run new versions in parallel to existing tests
- Don't block on new tests right away
- Once coverage is at parity, flip which tests block merge
- ...wait...
- Delete old tests
PROVE they are faster, more stable, easier to maintain

Encourage teams to replace existing tests & break old (bad) habits

describe("with feature flag enabled on project", () => {
    before(() => {
      cy.setFeatureFlag(FEATURE_FLAG, projectUid);
      cy.visit(`/projects/${projectUid}/transmittals/${DUMMY_TRANSMITTAL_UID}`);
    });

    after(() => {
      cy.resetFeatureFlag(FEATURE_FLAG, projectUid);
    });

    it("should show project transmittal", () => {
      cy.getByDataQa("NewProjectTransmittal").should("be.visible");

      cy.getByDataQa("NewProjectTransmittalTitle")
        .should("be.visible")
        .should("contain.text", "New Transmittal");
    });
  });

Encourage teams to replace existing tests & break old (bad) habits

Demonstrate Good Patterns

Write smaller, isolated, targeted tests
Use APIs for setup and teardown.
UI test ONLY what’s under test

3. Avoid "reinventing the wheel"

As more and more teams onboard:
- Everyone needs to log in
- Most tests need a project to work with
- Many tests need documents in a project
- Some tests need to manipulate feature flags
Easier to maintain if everyone performs these actions the same way
- Benefit: Broken functionality gets fixed quickly and for everyone

You Aren't Gonna Need It (YAGNI)

Avoid "reinventing the wheel"

a principle of extreme programming (XP) that states a programmer should not add functionality until deemed necessary.

...we started with

createUser() & login()

Create a shared Cypress library

Treat like Open Source Software
- Everyone can contribute
- Useful documentation
Single point of management for shared commands & plugins
An extensions library, not a wrapper of Cypress

> npm install @plangrid-private/acs-cypress

> npx acs-cypress connect

Avoid "reinventing the wheel"

// cypress/plugins/index
const acsPlugin = require("@plangrid-private/acs-cypress/dist/plugins/acsPlugin");

module.exports = (on, config) => {
  return acsPlugin.plugin(on, config);
};

// cypress/support/index
import "@plangrid-private/acs-cypress";

Avoid "reinventing the wheel"

Test Data Management

Tests should run in any environment with no manual setup required

{
  "users": [
    {
      "user_id": "5e0f9cdb154987db154d2c85",
      "first_name": "Test",
      "last_name": "User",
      "email": "testuser@example.com",
      "sheet_count": 0,
      "plan": "inf",
      "password": "--hidden--",
      "apiToken": "--hidden--",
      "stack": "dev",
      "url": "http://localhost:3000",
      "projects": [
        {
          "template": "threesheet",
          "projectUid": "b75900a3",
          "stack": "dev",
          "returnProject": false
        }
      ],
      "projectUid": "b75900a3"
    }
  ],
  "projects": [
    {
      "template": "threesheet",
      "projectUid": "b75900a3",
      "stack": "dev",
      "returnProject": false
    }
  ],
  "orgs": []
}

If fixture file is present, user and project data are used
If no fixture is present, data is created on-demand* and written to fixture file

Locally, fixture persists and user/project data is reused
In CI, fixtures are cleaned so new data is created on every test run

* To prevent time consuming operations projects can be pulled

from pre-populated pool

Launch Darkly

LaunchDarkly integration gives tests control over feature flag settings
Feature flags control is required to move away from static fixture data

Avoid "reinventing the wheel"

describe("with feature flag enabled on project", () => {
  
  const FEATURE_FLAG = "MY_FF";
  
  before(() => {
    cy.setFeatureFlag(FEATURE_FLAG, projectUid);
  });

  after(() => {
    cy.resetFeatureFlag(FEATURE_FLAG, projectUid);
  });
}

Have strong documentation: TypeScript + IntelliSense + TypeDoc

Use TypeScript function comments
Generates HTML docs using TypeDoc
Deployed automatically as part of this library's build process
IntelliSense makes discovering and using custom commands easy

Avoid "reinventing the wheel"

4. Limit maintenance burden to unblock progress

More time spent helping 20 teams debug failures is less time spent adding new functionality to the library for them
Reuse code wherever possible to eliminate duplication
Test the test code

Integrate with our Web API Client

Frontend repos share an internal API client library
We leverage the same library to make backend requests
No need to maintain requests across tests
- they stay in sync with the frontend

Limit maintenance burden to unblock progress

API Client Integration

import { capiClient } from "@plangrid-private/web-api-clients";

// In your test
cy.capi(capiClient.createProject, { name: name }).then(project => {
  // Do something with your new project
});

Limit maintenance burden to unblock progress

💡 Cypress team's tip: for strictly API tests you can use

https://github.com/bahmutov/cy-api plugin

Test the Test Code

Treat test libraries as seriously as production libraries
Unit tests
- prevent introducing breaking changes
Versioned with Semantic Versioning
- Allows rollback if bugs are introduced
- Consumers know when to expect safe upgrades
Release notes/changelog
- Communicate what changed, and why

5. Enable anyone to debug, monitor, & own their tests

Avoid discussions such as "my Cypress tests are failing and I don't know why"
Shift mindset from "the test is probably flaky" to "what has changed to cause this test to fail"

Automatic retries plugin

cypress-plugin-retries
There are pitfalls of automatic retries
- Hide poorly written tests
- Test suites take longer
Avoid these pitfalls
- Only allow retries in CI - Failures are seen during development
- Allow teams to specify the number of retries

Enable anyone to debug, monitor, & own their tests

💡 Cypress team's tip: test retries are coming to the Cypress core, see issue #1313

Datadog Integration

Automatically push results via Cypress hooks and events
Chart metrics and data about the tests
Provides data about stability

Enable anyone to debug, monitor, & own their tests

Dashboard Integration

Keep it easy to set up
Handle collecting the git and Jenkins info in the shared libraries

// Jenkinsfile
@Library("web-build-tools") _

stage("Checkout") {
  GitClean()
  checkout scm
}

def cypressConfig = [
  additionalRunVars: "BROWSER=chrome",
  recordKey: "r3c0rd-k3y-f4k3-r3c0rdk3y"
]

runFullBuild(
  projectType: "app",
  cypressConfig: cypressConfig
)

// cypress.json
{
  "projectId": "f4k31d"
}

Enable anyone to debug, monitor, & own their tests

Dashboard Integration

See all relevant data in one place

Easier than navigating through
Jenkins or Datadog
View screenshots and videos

Enable anyone to debug, monitor, & own their tests

Better logging solutions

Custom commands include clear log output using Cypress' logger

Enable anyone to debug, monitor, & own their tests

Default:

Custom:

Better logging solutions

Matches verbiage and style of default commands

Enable anyone to debug, monitor, & own their tests

cy.capi() Custom Command:

cy.get() Default Command:

Better logging solutions

Environment variable print out
No questions about your test environment

$ npm run cypress
> react-scripts cypress

Loading environment for APP=cypress STACK=dev
Loading environment from .env
Loading environment from .env.stack.dev

CI: false
DEBUG: error
SHARED_SECRET_PASSWORD: -hidden-
MAILOSAUR_SERVER: -hidden-
MAILOSAUR_API_KEY: -hidden-
DATADOG_API_KEY: -hidden-
DATADOG_APP_KEY: -hidden-
SALESFORCE_ACCOUNT_ID: -hidden-
GRIDBOY_ADMIN: -hidden-
LAUNCHDARKLY_ACCESS_TOKEN: -hidden-
..... etc

Cypress-log-to-output plugin
Logs browser errors to console - these are normally hidden in CI

Enable anyone to debug, monitor, & own their tests

💡 Cypress team's tip: find this and similar plugins at https://on.cypress.io/plugins#reporting

Where are we now?

All new projects come with Cypress, the shared library, and CI integration set up out of the box
20+ repositories use our shared Cypress library
20+ shared custom commands
900+ tests run per day
96%+ test pass rate
4 min avg. test run duration (without parallelization)

What’s Next

Parallelization (In Progress)
- Allow a cypressConfig value for numMachines
Mock API server to avoid environment instability issues

// Jenkinsfile
@Library("web-build-tools") _

stage("Checkout") {
  GitClean()
  checkout scm
}

def cypressConfig = [
  additionalRunVars: "BROWSER=chrome",
  recordKey: "a5a19008-e622-b014-d966fdacc667",
  numMachines: 5
]

runFullBuild(
  projectType: "app",
  cypressConfig: cypressConfig
)

Unscalable maintenance is the death of progress

Q & A

Sli.do event code #cyplangrid

or use URL https://app.sli.do/event/sawiafoj

these slides at

https://cypress.slides.com/cypress-io/quality-in-1-million-projects

How PlanGrid (Autodesk) Achieved Consistency in Automation Across 20 Development Teams

About PlanGrid

About PlanGrid

Contents

The Platform Team Model

Our (Web) Development

The Branching Strategy

Something Had to Change

The New Goal

Our Must-Haves

The Contenders

The Vetting

5 Major Challenges We Encountered

1. Make our tool the obvious choice

Make the easy way the right way

Make our tool the obvious choice

Integrate with react-scripts

Make our tool the obvious choice

Integrate with react-scripts

Make our tool the obvious choice

Integrate with shared build library

Make our tool the obvious choice

Integrate with shared build library

Make our tool the obvious choice

2. Encourage teams to replace existing tests & break old (bad) habits

Prove the value

Encourage teams to replace existing tests & break old (bad) habits

Encourage teams to replace existing tests & break old (bad) habits

Demonstrate Good Patterns

3. Avoid "reinventing the wheel"

You Aren't Gonna Need It (YAGNI)

Avoid "reinventing the wheel"

Create a shared Cypress library

Avoid "reinventing the wheel"

Avoid "reinventing the wheel"

Test Data Management

Launch Darkly

Avoid "reinventing the wheel"

Have strong documentation: TypeScript + IntelliSense + TypeDoc

Avoid "reinventing the wheel"

4. Limit maintenance burden to unblock progress

Integrate with our Web API Client

Limit maintenance burden to unblock progress

API Client Integration

Limit maintenance burden to unblock progress

Test the Test Code

5. Enable anyone to debug, monitor, & own their tests

Automatic retries plugin

Enable anyone to debug, monitor, & own their tests

Datadog Integration

Enable anyone to debug, monitor, & own their tests

Dashboard Integration

Enable anyone to debug, monitor, & own their tests

Dashboard Integration

Enable anyone to debug, monitor, & own their tests

Better logging solutions

Enable anyone to debug, monitor, & own their tests

Better logging solutions

Enable anyone to debug, monitor, & own their tests

Better logging solutions

Enable anyone to debug, monitor, & own their tests

Where are we now?

What’s Next

Q & A

Sli.do event code #cyplangrid

or use URL https://app.sli.do/event/sawiafoj