&

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

How Cypress helped Ansible re-write their entire UI from scratch

Ask Questions at Sli.do

event code #cy-ansible

John Hill

Senior Automation

Engineer

github.com/unlikelyzero

About me

  • Senior Automation Engineer @ Ansible
  • Durham, NC
  • UI Testing since 2014

unlikelyzero.com

github.com/unlikelyzero

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Ansible has a UI?

  • Ansible is best known as an automation language with a CLI
  • 3 Ansible UIs are being tested by Cypress!
  • Private Automation Hub
    • On-Prem Ansible Content Management platform
  • Automation Analytics
    • Gives deep insight into execution of Ansible across your Org
  • Ansible Tower (AWX)
    • Manages and Controls execution of Playbooks
    • What we're looking at today

Agenda

  • UI + Test Rewrite
  • Test Framework Comparison
  • What we learned about Cypress
  • What we learned from the rewrite
  • Where we're going next
  • Q/A at Sli.do event code #cy-ansible

Rewrite Ansible Tower UI

New

"Classic"

"Classic" UI

  • 2014
  • “Grew organically over time”
  • Bespoke implementations and inconsistent UX
  • Angular 1.8 LTS
  • Dependency bump problems

 

"Classic" Tests

  • No documented testcases
  • Low unit test coverage
  • ~100 e2e tests with NightwatchJS
  • Flaky and slow
  • Selenium dependency bumps
  • Needed refactor to support ES6
  • "QA Interpretation" of results

Why rewrite?

  • Patternfly standardizes look and feel of Red Hat's UIs
  • Angular 1.x deprecates on Dec 31, 2021
  • Didn't meet accessibility standards (a11y)

Automation Analytics

New UI

  • 100% Feature Parity
  • ReactJS
  • Patternfly Design Library
    • Consistent UX
    • a11y

New Tests

  • “What would we do with years of hindsight?"
  • Automatable Tests
  • High unit test coverage
  • Visual Regression testing
  • Fast and reliable
  • Test Architecture up front
  • Self Service

UI Rewrite Strategy

  • Start From Scratch. No migration.
  • Build by Dependency instead of Pendo
  • UX Improvements
  • API changes to improve UI performance
  • Partial Implementations

Rewrite Test Strategy

  • Start From Scratch. Too much changing
  • Delete the old codebase
  • Implement Tests and Test Functionality alongside dev
  • Test Codebase as source of truth with TODOs
  • Maintain the API Tests
  • Balance unit / e2e test coverage
  • Patternfly as a source of destabilization
  • “How to build without deleting?”

Testing Wishlist

  • 30 minutes total runtime (Parallelization)
  • 5 minute testsuite runtime (Pre-emptive node $$$)
  • Stability
  • Testability
  • Pre-merge Testing

Based on 2 years with Nightwatch

Our Test Framework Shootout

  • NightwatchJS (Selenium), Cypress, and puppeteer (Jest)
  • 2 weeks dedicated
  • 3 engineers with "controls"
  • Write the same 4 testcases (CRUD on Organization)
  • Document what we found WITHOUT asking for help

Requirements and Criteria (May 2019)

Nightwatch.js jest-puppeteer Cypress
Multi-browser Selenium! No* No*
OSS Bus Factor Low Medium Medium*
Visual Testing Yes Yes Yes*
Triage-ability Medium* Low Off the Charts
Network Mocks No Yes* Yes
Parallelization Yes No Yes*
Documentation Medium Low Off the Charts
Native .todo() No Yes No

(what we had before)

Who won? 🎁

Requirements and Criteria (May 2019)

Nightwatch.js jest-puppeteer Cypress
Multi-browser Selenium! No* No*
OS Bus Factor Low Medium Medium*
Visual Testing Yes Yes Yes*
Triage-ability Medium* Low Off the Charts
Network Stubs/Mocks No Yes* Yes
Parallelization Yes No Yes*
Documentation Medium Low Off the Charts
Native .todo() No Yes No

(what we had before)

What did we learn about Cypress along the way?

Everything is a "solved problem"

  • No time wasted on experimentation and failure
  • API Documentation
  • Example repos (Percy and Applitools)
  • Plugin ecosystem (a11y)

GitHub Actions (GHA)

  • Jenkins Build/Deploy is 22 minutes
  • Cypress Serial Runtime 3 hours
  • GHA Build/Deploy Time ~5 min
  • Cypress Github Action ~2 min
  • Cypress Dashboard x17
  • Total GHA Runtime of ~14 minutes
  • Zuul Unit tests take ~17 minutes

We can leverage python SDK from within Cypress

//commands.js
Cypress.Commands.add('createOrReplace', (resource, name) => {
  const options = `--resource="${resource}" --name="${name}"`
  const command = `python akit_client.py create_or_replace ${options} "${Cypress.config().baseUrl}"`
  console.log(`Calling awxkit create_or_replace(): ${options}`)
  cy.exec(command, { timeout: Cypress.config('akit_wait') }).then((ret) => {
    expect(ret.stderr).to.be.empty
    return JSON.parse(ret.stdout)
  })
})

We can leverage python SDK from within Cypress

//commands.js
Cypress.Commands.add('createOrReplace', (resource, name) => {
  const options = `--resource="${resource}" --name="${name}"`
  const command = `python akit_client.py create_or_replace ${options} "${Cypress.config().baseUrl}"`
  console.log(`Calling awxkit create_or_replace(): ${options}`)
  cy.exec(command, { timeout: Cypress.config('akit_wait') }).then((ret) => {
    expect(ret.stderr).to.be.empty
    return JSON.parse(ret.stdout)
  })
})
beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

We can leverage python SDK from within Cypress

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Object was created via Python SDK

No Page Object Model?

No problem if you're not a true SPA

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Visual Tests need Cypress

cy.intercept() on the timestamps

Fixture data for timeseries plots

Visual Tests Find Bugs

Dashboard breaks down barriers between dev and QE

"When did it break?"

UI is often the bearer of bad news

Run all tests every 3 hours for a baseline

Reduce Triage Time

Embrace Failure

What did we learn about the Rewrite Strategy?

Where are the bugs? 🐞

  • No multi-browser bugs
  • No patternfly design library bugs
  • Patternfly Cypress?
  • Unit tests?

Test Rewrite Strategy

  • Deleting tests? Oops!
  • Test Deduplication
  • Start with data-id test attributes
  • Hard to keep up with Devs

TODOs

  • Dev quickly outpaced QA
  • Testability was the "floor" to Definition of Done
  • Test codebase became source of truth via "stubs"
  • "TBI" or "To Be Implemented"
context('Testsuite Example', function () {
  it('a real test', function () {
    cy.get('#the-thing').click()
  })
  it.skip('a stubbed test', function () {})
  it.skip('a test which cannot be implemented', function () {}) // TBI /awx/issues/9122
})

Lots of functionality

Where are we going?

  • Coverage
  • Speed
  • Re-use of test codebase

Coverage

  • Solid Foundation
  • Pendo!
  • e2e + unit test coverage

Speed

We have spent 0 time optimizing our runs

Speed (Continuted)

Test the UI, not through it

beforeEach(function () {
  cy.createOrReplace('credentials', `credential-name`).as('cred')
})

it('can copy a credential', function () {
  cy.visit(`/#/credentials?credential.name__icontains=${this.cred.name}`)
  cy.server()
  cy.route('POST', `/api/v2/credentials/${this.cred.id}/copy/`).as('copy')
  cy.route('OPTIONS', `/api/v2/credentials/`).as('getCredentials')
  cy.get('#credential-list-copy-button').click()
  cy.wait(['@copy', '@getCredentials'])
  cy.get('tbody').find('tr').should('have.length', 2)
 })

Object was created via Python SDK

i18n Testing

Cypress.on('window:before:load', (window) => {
  switch (Cypress.env('BROWSER_LANGUAGE')) {
    case 'japanese':
      console.log('Loading Japanese...')
      Object.defineProperty(
		window.navigator, 'language',
		{ value: 'ja-JP' })
      Object.defineProperty(
		window.navigator,
		'languages', ['jp']
		)
      break
    case 'french':
      console.log('Loading French...')
      Object.defineProperty(
		window.navigator,
		'language', { value: 'fr-FR' }
		)
      Object.defineProperty(
		window.navigator, 
		'languages', ['fr'])
      break
  }
})

Summary

  • Rewrite and Migration
  • Test Framework Comparison
  • What we learned about Cypress
  • What we learned from the rewrite
  • Where we're going next

Gleb Bahmutov

Distinguished Engineer

@bahmutov

How Cypress helped Ansible re-write their entire UI from scratch

John Hill

Senior Automation

Engineer

Thank you 👏

unlikelyzero.com

github.com/unlikelyzero

Cypress and Ansible

By Cypress.io

Cypress and Ansible

In this webcast, John Hill, Senior Automation Engineer at Ansible, and Gleb Bahmutov, Distinguished Engineer at Cypress, will show how the Ansible team completely re-wrote the Ansible Tower UI and revamped their testing efforts by deleting all of their old tests and starting from scratch with Cypress’s next-generation test tooling.

  • 1,164
Loading comments...

More from Cypress.io