& 

Gleb Bahmutov

VP of Engineering

@bahmutov

WEBCAST

Jeff Sing

QA Manager

@joptimizely

Todd Seller

Software QA Engineer

Questions via Sli.do #cyopt

& 

Gleb Bahmutov

VP of Engineering

@bahmutov

Todd Seller

Software QA Engineer

Jeff Sing

QA Manager

@joptimizely

Contents

  1. About Optimizely
  2. How we use Cypress.io at Optimizely
    1. Why Cypress.io?
    2. Regression Suite & Test Parallelization & CI System
    3. Salesforce Integration Example 
  3. Feature Flags and Automation Strategy
    1. Automation Best Practices
    2. Feature Flag Types & Testing
    3. What's next?
  4. Q&A

Experimentation Platform (A/B/n, Personalization, MVT, Stats Acceleration)

Progressive Delivery (Feature Rollouts, Feature Flag Experiments)

Progressive Delivery (Feature Rollouts, Feature Flag Experiments)

Why Cypress.io?

  1. Very difficult to onboard. Original dev created bespoke framework in Python.
  2. Very difficult to debug.
  3. Pipeline took forever to run (probably our fault)
  1. Very quick to train our team. Able to stand up E2E  within a month.
  2. Easier to debug.
  3. Easy and clear how to test in parallel
defaults: &defaults
  script:
    - npm run cy:run -- --record --parallel --group $STAGE_NAME
    - kill $(jobs -p) || true
jobs:
  include:
    # we have multiple jobs to execute using just a single stage
    # but we can pass group name via environment variable to Cypress test runner
    # run tests in parallel by including several test jobs with same name variable
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=8x-electron
      <<: *defaults

Parallelization in Travis-CI

Cypress gave me the ability, and confidence, to test our integration within Salesforce

Mike London, QA Engineer - Optimizely

...and now, Feature Flags

Are you using feature flags right now?

Feature flags (also known as feature toggles or feature switches) are a software development technique that turns certain functionality on and off during runtime, without deploying new code. This allows for better control and more experimentation over the full lifecycle of features.

What are Feature Flags/Feature Toggles

Feature Flags are being used by almost all major companies...

Has over 100+ active flags in our code.

  • 1000 customers (200 paying customers/ 800 monthly active free accounts)
  • 8000 Feature Flags created last month
Do you have tests for your feature flags?

General Automation Test Strategy

Test Runner + INDETERMINISM = BAD

General Automation Test Strategy

Remove INDETERMINISM

  1. Special Test User
  2. Test Cookie
  3. Test Query Parameter to force a particular variation

General Automation Test Strategy

General Automation Test Strategy

describe('astro_boy Feature Scenarios', function () {
  it('Validate that the astro_boy feature is not enabled with no query parameter', function () {
    // Visit app
    cy.visit('/')

    // Validate feature is disabled
    cy.get('#astronaut')
      .should('not.exist')
  })

  it('Validate that the astro_boy feature is not enabled with incorrect query parameter', function () {
    // Visit app
    cy.visit('/?cypress=off')

    // Validate feature is disabled
    cy.get('#astronaut')
      .should('not.exist')
  })

  it('Validate that the astro_boy feature is enabled', function () {
    // Visit app with audience query parameter
    cy.visit('/?cypress=on')

    // Validate feature is enabled
    cy.get('#astronaut')
      .should('exist')
  })

  it('Validate button is clickable', function () {
    // Visit app with audience query parameter
    cy.visit('/?cypress=on')

    // Validate feature is enabled
    cy.get('button')
      .click()
  })
})

General Automation Test Strategy

Automation Test Strategies for Different Type of Flags

Short Term

Feature Rollout

For deploying a new feature that will be permanent.

Long Term

Permission

Change product experience that certain users receive

Operational

Controls operational aspects of your system; allows control of certain operational features in their production env.

Circuit Breaker

Turn on/off certain features or conditions in your production env.

Automation Test Strategies for Different Type of Flags

Short Term

Feature Rollout

For deploying a new feature that will be permanent.

Long Term

Permission

Change product experience that certain users receive

Operational

Controls operational aspects of your system; allows control of certain operational features in their production env.

Circuit Breaker

Turn on/off certain features or conditions in your production env.

Automation Test Strategies for Different Type of Flags

Short Term

Feature Rollout

For deploying a new feature that will be permanent.

Long Term

Permission

Change product experience that certain users receive

Operational

Controls operational aspects of your system; allows control of certain operational features in their production env.

Circuit Breaker

Turn on/off certain features or conditions in your production env.

Rollouts

What do you do once your feature is rolled out to 100%, but the feature flag has not been removed?

Cypress makes it easy to organize test cases. Once a feature is rolled out we move the legacy test case into a deprecated file.

defaults:
  script: &defaults
  - npm run cy:run -- --record --parallel --group $STAGE_NAME
  - kill $(jobs -p) || true
jobs:
  include:
  # we have multiple jobs to execute using just a single stage
  # but we can pass group name via environment variable to Cypress test runner
  # run tests in parallel by including several test jobs with same name variable
  - stage: deprecated
    env:
    - STAGE_NAME=deprecated
    script: npm run cy:deprecated -- --record --config integrationFolder=cypress/deprecated
  - stage: test
    env:
    - STAGE_NAME=5x-electron
    script: *defaults
  - stage: test
    env:
    - STAGE_NAME=5x-electron
    script: *defaults
  - stage: test
    env:
    - STAGE_NAME=5x-electron
    script: *defaults
  - stage: test
    env:
    - STAGE_NAME=5x-electron
    script: *defaults
  - stage: test
    env:
    - STAGE_NAME=5x-electron
    script: *defaults

Running Specific Folders in Travis-CI

How do you test different account or permission types?

Cypress makes it easy to create custom commands that allow us to take advantage of our create test account handler.

Permission

 Cypress.Commands.add('createAccount', (config) => {
  config += `&email=bdd%2Btest${ Date.now() }@optimizely.com`
  config += `&password1=${ Cypress.env('newAccountPassword') }`

  let email = ''
  let accountId = null
  let projectId = null
  let csrfToken = ''

  const options = {
    url: '/account/test_account_create',
    method: 'POST',
    body: config,
    headers: { 'User-Agent': 'optimizely-e2e-test', 'Cookie': 'green=true' },
    form: true
  }
  cy.log(`Attempting to create account on ${ Cypress.env('baseUrl') }`)

  cy.request(options)
    .then(response => {
      accountId = response.body.account_id
      csrfToken = response.body.csrf_token
      email = response.body.user_email
      projectId = response.body.project_id

      cy.log(`Account ${ accountId } created for ${ email }.`)
      cy.log('Getting personal access token and setting persistent cookies.')
    
      cy.acceptTOS({ email: email, csrfToken: csrfToken })

      cy.getPersonalAccessToken({ email: email, accountId: accountId, csrfToken: csrfToken })
        .then(response => {
          window.sessionStorage.setItem('accessToken', response.body.token_hash)
          persistentStorage.accessToken = response.body.token_hash
        })

      cy.setCookie('csrftoken', csrfToken)
      cy.setCookie('hasSeenABV2FeedbackRecently', 'true')
      window.sessionStorage.setItem('accountId', accountId)
      window.sessionStorage.setItem('projectId', projectId)
      persistentStorage = {
        accountId: accountId,
        projectId: projectId
      }

      cy.getCookies()
        .then(cookies => {
          persistentCookies = cookies
        })

    })
})
describe('Rollouts Experiments Dashboard Tests', function () {
  before(function () {
    cy.createAccount(Cypress.env('free_rollout_account'))
      .saveGeneratedTimeBasedIdAs('featureKey')
      .saveGeneratedTimeBasedIdAs('featureKey1')
      .saveGeneratedTimeBasedIdAs('experimentKey')
      .saveGeneratedTimeBasedIdAs('experimentKey1')
  })
describe('Full Stack Permissions Experiments Dashboard Tests', function () {
  before(function () {
    cy.createAccount(Cypress.env('full_stack_account'))
      .saveGeneratedTimeBasedIdAs('featureKey')
      .saveGeneratedTimeBasedIdAs('featureKey1')
      .saveGeneratedTimeBasedIdAs('experimentKey')
      .saveGeneratedTimeBasedIdAs('experimentKey1')
  })

Rollout Account User

Full Stack Account User

Operational

Requires testing for both the default and operational variables.

Default Variable

Operational

Requires testing for both the default and operational variables.

Operational Variable

Operational

describe('Astro Boy Operational Flag Tests', function () {
  it('Button should have default text when user is not in feature test', () => {
    cy.visit('/')

    cy.get('img#astronaut')
      .should('be.visible')
      .and('have.attr', 'alt')

    cy.get('button')
      .should('have.text', 'Learn More')
      .and('be.enabled')
  })

  it('Button should have updated text when user is in feature test', () => {
    cy.visit('/?cypress=on')

    cy.get('img#astronaut')
      .should('be.visible')
      .and('have.attr', 'alt')

    cy.get('button')
      .should('have.text', 'Blast Off!')
      .and('be.enabled')
  })
})

Circuit Breaker 

describe('Snippet Circuit Breaker Test', () => {
  // Validate the pressence of the Optimizely snippet when the flag is on

  before(() => {
    cy.login()
      .setPersistentCookies()
  })

  it('Contains the Optimizely snippet in the <head> tag with the feature on', () => {

    cy.visit(`${Cypress.env('referer')}?cypress=on`)

    cy.get('head script[src="//cdn.optimizely.com/public/5935064/s/optimizely_app.js"]')
      .should('exist')
  })
  
  it('Does not contain the Optimizely snippet in the <head> tag with the feature off', () => {
    
    cy.visit(Cypress.env('referer'))
    
    cy.get('head script[src="//cdn.optimizely.com/public/5935064/s/optimizely_app.js"]')
      .should('not.exist')
  })
})

What don't we test

Short Term Flags

Experimentation

Allows you to perform A/B tests.

Allows you to make data-driven optimization

Non Critical Variations

Low risk or non important variations

What's next?

Goal: Don't have to write 2 test cases (feature off/feature on)


 

What's next?

Goal: Don't have to write 2 test cases (feature off/feature on)

Solution: Cypress should know which variation (API call to Optimizely) which allows it to know what variation or feature variable

Contact information

Todd: todd.seller@optimizely.com

 Jeff:   jeffrey.sing@optimizely.com

 

✨Free Version of Optimizely Feature Flags/Experiments✨

👉 https://optimize.ly/cypress-webinar

Slides: https://cypress.slides.com/cypress-io/cypress-and-optimizely

Thank you!