&

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

How Cypress helped Ansible re-write their entire UI from scratch

Ask Questions at Sli.do

event code #cy-ansible

https://cypress.slides.com/cypress-io/cypress-and-ansible

John Hill

Senior Automation

Engineer

github.com/unlikelyzero

About me

  • Senior Automation Engineer @ Ansible
  • Durham, NC
  • UI Testing since 2014

unlikelyzero.com

github.com/unlikelyzero

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Ansible has a UI?

  • Ansible is best known as an automation language with a CLI
  • 3 Ansible UIs are being tested by Cypress!
  • Private Automation Hub
    • On-Prem Ansible Content Management platform
  • Automation Analytics
    • Gives deep insight into execution of Ansible across your Org
  • Ansible Tower (AWX)
    • Manages and Controls execution of Playbooks
    • What we're looking at today

Agenda

  • UI + Test Rewrite
  • Test Framework Comparison
  • What we learned about Cypress
  • What we learned from the rewrite
  • Where we're going next
  • Q/A at Sli.do event code #cy-ansible

https://cypress.slides.com/cypress-io/cypress-and-ansible

Rewrite Ansible Tower UI

New

"Classic"

"Classic" UI

  • 2014
  • “Grew organically over time”
  • Bespoke implementations and inconsistent UX
  • Angular 1.8 LTS
  • Dependency bump problems

 

"Classic" Tests

  • No documented testcases
  • Low unit test coverage
  • ~100 e2e tests with NightwatchJS
  • Flaky and slow
  • Selenium dependency bumps
  • Needed refactor to support ES6
  • "QA Interpretation" of results

Why rewrite?

  • Patternfly standardizes look and feel of Red Hat's UIs
  • Angular 1.x deprecates on Dec 31, 2021
  • Didn't meet accessibility standards (a11y)

Automation Analytics

https://www.patternfly.org

New UI

  • 100% Feature Parity
  • ReactJS
  • Patternfly Design Library
    • Consistent UX
    • a11y

New Tests

  • “What would we do with years of hindsight?"
  • Automatable Tests
  • High unit test coverage
  • Visual Regression testing
  • Fast and reliable
  • Test Architecture up front
  • Self Service

https://www.patternfly.org

UI Rewrite Strategy

  • Start From Scratch. No migration.
  • Build by Dependency instead of Pendo
  • UX Improvements
  • API changes to improve UI performance
  • Partial Implementations

Rewrite Test Strategy

  • Start From Scratch. Too much changing
  • Delete the old codebase
  • Implement Tests and Test Functionality alongside dev
  • Test Codebase as source of truth with TODOs
  • Maintain the API Tests
  • Balance unit / e2e test coverage
  • Patternfly as a source of destabilization
  • “How to build without deleting?”

Testing Wishlist

  • 30 minutes total runtime (Parallelization)
  • 5 minute testsuite runtime (Pre-emptive node $$$)
  • Stability
  • Testability
  • Pre-merge Testing

Based on 2 years with Nightwatch

the-true-cost-of-a-software-bug

Our Test Framework Shootout

  • NightwatchJS (Selenium), Cypress, and puppeteer (Jest)
  • 2 weeks dedicated
  • 3 engineers with "controls"
  • Write the same 4 testcases (CRUD on Organization)
  • Document what we found WITHOUT asking for help

Requirements and Criteria (May 2019)

Nightwatch.js jest-puppeteer Cypress
Multi-browser Selenium! No* No*
OSS Bus Factor Low Medium Medium*
Visual Testing Yes Yes Yes*
Triage-ability Medium* Low Off the Charts
Network Mocks No Yes* Yes
Parallelization Yes No Yes*
Documentation Medium Low Off the Charts
Native .todo() No Yes No

(what we had before)

Who won? 🎁

Requirements and Criteria (May 2019)

Nightwatch.js jest-puppeteer Cypress
Multi-browser Selenium! No* No*
OS Bus Factor Low Medium Medium*
Visual Testing Yes Yes Yes*
Triage-ability Medium* Low Off the Charts
Network Stubs/Mocks No Yes* Yes
Parallelization Yes No Yes*
Documentation Medium Low Off the Charts
Native .todo() No Yes No

(what we had before)

What did we learn about Cypress along the way?

Everything is a "solved problem"

  • No time wasted on experimentation and failure
  • API Documentation
  • Example repos (Percy and Applitools)
  • Plugin ecosystem (a11y)

GitHub Actions (GHA)

  • Jenkins Build/Deploy is 22 minutes
  • Cypress Serial Runtime 3 hours
  • GHA Build/Deploy Time ~5 min
  • Cypress Github Action ~2 min
  • Cypress Dashboard x17
  • Total GHA Runtime of ~14 minutes
  • Zuul Unit tests take ~17 minutes

https://github.com/cypress-io/github-action

We can leverage python SDK from within Cypress

//commands.js
Cypress.Commands.add('createOrReplace', (resource, name) => {
  const options = `--resource="${resource}" --name="${name}"`
  const command = `python akit_client.py create_or_replace ${options} "${Cypress.config().baseUrl}"`
  console.log(`Calling awxkit create_or_replace(): ${options}`)
  cy.exec(command, { timeout: Cypress.config('akit_wait') }).then((ret) => {
    expect(ret.stderr).to.be.empty
    return JSON.parse(ret.stdout)
  })
})

We can leverage python SDK from within Cypress

//commands.js
Cypress.Commands.add('createOrReplace', (resource, name) => {
  const options = `--resource="${resource}" --name="${name}"`
  const command = `python akit_client.py create_or_replace ${options} "${Cypress.config().baseUrl}"`
  console.log(`Calling awxkit create_or_replace(): ${options}`)
  cy.exec(command, { timeout: Cypress.config('akit_wait') }).then((ret) => {
    expect(ret.stderr).to.be.empty
    return JSON.parse(ret.stdout)
  })
})
beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

We can leverage python SDK from within Cypress

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Object was created via Python SDK

No Page Object Model?

No problem if you're not a true SPA

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Visual Tests need Cypress

cy.intercept() on the timestamps

Fixture data for timeseries plots

https://on.cypress.io/visual-testing

Visual Tests Find Bugs

Dashboard breaks down barriers between dev and QE

"When did it break?"

UI is often the bearer of bad news

Run all tests every 3 hours for a baseline

Reduce Triage Time

Embrace Failure

What did we learn about the Rewrite Strategy?

Where are the bugs? 🐞

  • No multi-browser bugs
  • No patternfly design library bugs
  • Patternfly Cypress?
  • Unit tests?

Test Rewrite Strategy

  • Deleting tests? Oops!
  • Test Deduplication
  • Start with data-id test attributes
  • Hard to keep up with Devs

TODOs

  • Dev quickly outpaced QA
  • Testability was the "floor" to Definition of Done
  • Test codebase became source of truth via "stubs"
  • "TBI" or "To Be Implemented"
context('Testsuite Example', function () {
  it('a real test', function () {
    cy.get('#the-thing').click()
  })
  it.skip('a stubbed test', function () {})
  it.skip('a test which cannot be implemented', function () {}) // TBI /awx/issues/9122
})

Lots of functionality

Where are we going?

  • Coverage
  • Speed
  • Re-use of test codebase

Coverage

  • Solid Foundation
  • Pendo!
  • e2e + unit test coverage

https://on.cypress.io/code-coverage

Speed

We have spent 0 time optimizing our runs

Speed (Continuted)

Test the UI, not through it

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Object was created via Python SDK

i18n Testing

Cypress.on('window:before:load', (window) => {
  switch (Cypress.env('BROWSER_LANGUAGE')) {
    case 'japanese':
      console.log('Loading Japanese...')
      Object.defineProperty(
		window.navigator, 'language',
		{ value: 'ja-JP' })
      Object.defineProperty(
		window.navigator,
		'languages', ['jp']
		)
      break
    case 'french':
      console.log('Loading French...')
      Object.defineProperty(
		window.navigator,
		'language', { value: 'fr-FR' }
		)
      Object.defineProperty(
		window.navigator, 
		'languages', ['fr'])
      break
  }
})

Summary

  • Rewrite and Migration
  • Test Framework Comparison
  • What we learned about Cypress
  • What we learned from the rewrite
  • Where we're going next

Gleb Bahmutov

Distinguished Engineer

@bahmutov

How Cypress helped Ansible re-write their entire UI from scratch

https://cypress.slides.com/cypress-io/cypress-and-ansible

John Hill

Senior Automation

Engineer

Thank you 👏

unlikelyzero.com

github.com/unlikelyzero