&

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

How Pendo Maintains a Culture of Quality with Cypress

Ask Questions at Sli.do

event code #cy-pendo

George Palfreyman

Senior Quality Engineer

@georgepalf

George

  • Senior Quality Engineer @ Pendo
  • Sheffield, United Kingdom
  • 6 years testing experience
  • ~2 years working with automated testing

Ask Questions at Sli.do

event code #cy-pendo

"Pendo combines powerful product usage analytics with user guidance, communication, feedback, and planning tools to offer the most complete integrated platform for digital product teams. With Pendo, product teams can understand product usage, collect feedback, measure sentiment, onboard users, and announce new features in app—all without requiring engineering resources."

7 offices worldwide
1,700 customers

500+ employees

Reminder: Q&A

Ask Questions at Sli.do event code #cy-pendo

Pendo Products

  • Insights
     
  • Sentiment
     
  • Feedback
     
  • Mobile
     
  • Integrations
     
  • Adopt
  • Guidance
     

Pendo Products

  • Insights
     
  • Sentiment
     
  • Feedback
     
  • Mobile
     
  • Integrations
     
  • Adopt
  • Guidance

Guidance

Ask Questions at Sli.do

event code #cy-pendo

Guides provide personalised in-app guides and messaging

Guidance

Highlight new features

User onboarding and tutorials

Guidance

Polls & Surveys

 

Can be used to capture feedback on new products and features

Guidance

Resource Center
 

Houses additional help for users including:

 

  • guides
  • announcements
  • FAQs
  • help article links
  • integrations

Guidance

Guide activation - Automatic

Guidance

Guide activation - Badge

Guidance

Guide activation - Target Element

Guidance

But how do we do this?..

 

Users launch the
Users can then create guides targeting their features and pages where required.

Visual Design Studio

 over their application.

Guidance


 

  • Used for building guides
  • User ability to design many different types of guides (polls, surveys etc)
  • User freedom to implement different types of content (images, video links etc)
  • Users can combine these and build them at different lengths
  • Can be launched over any customer site
     

Visual Design Studio

  • Attracts great customers
     

Bonus point!

 

Who then go on to create...

Guidance

Beautiful walkthroughs...

Stunning polls...

And delightful resource centers...

Guidance

Stunning Polls...

Guidance

And delightful Resource Centers...

Before Cypress...

Ask Questions at Sli.do

event code #cy-pendo

  • Functionality issues found during manual regression testing
  • Manual testing focus of functionality rather than potential use cases

Manual Testing

  • guide types (polls, surveys, walkthroughs)
  • guide content (images, video links, buttons)
  • guide lengths (single step guide... 15 step guide)
  • guide activation (element, badge, automatically)
  • Launched over any customer site (CSS rules)

General pain points potentially leading to false starts using test automation frameworks...

Automated Testing

Lack of maintenance and iterations

Flaky/unstable tests

Inconsistent test patterns - storing of web elements (with POM)

Lack of visibility of tests and results

Onboarding or prior knowledge required

Lack of team adoption

Lack of leadership buy in

  • Drive quality culture within our engineering teams
  • Everyone owns quality of our products
     

All eyes on quality...

  • Revise implemented automation to catch regressions
  • Focus manual testing on customer problems and use cases
  • Needed a tool that works for the many and not just the few
     
  • Good reputation and recommendations from engineers and the wider community
     
  • Easy to get up and running
     
  • Lots of documentation and community driven questions and answers online

And that's where Cypress comes in...

And that's where Cypress comes in...

Problem

Solution

Flaky/unstable tests

Cypress Retries

Inconsistent test patterns

Best practice of using data attributes

Static waits

Waiting on aliases and/or assertions

Lack of visibility

Cypress Dashboard

Onboarding or prior knowledge required

Front end tests with detailed documentation

Team adoption

Leadership buy in

Maintenance

Using Cypress to rebuild our Guides test suite...

Ask Questions at Sli.do

event code #cy-pendo

Using Cypress to rebuild our Guides test suite...

Sanity Suite

Regression Suite

Runs on all branches and blocks merge

Light touch of all areas

Full coverage of the Visual Design Studio

Tests are dependent on environment resources

Tests are independent and create their own resources and can be pointed at any environment

cypress/integration/sanity
cypress/integration/regression

Scheduled to run nightly on the release branch

1. Define test suite coverage...

  • Environment already existed, just needed to configure data
  • Create users
  • Tools to clean up the data

3. Build the Sanity suite

4. Build the Regression suite

5. Document maintenance processes following Cypress best practices, leveraging Slack and      Cypress Dashboard

2. Infrastructure setup and configuration

Using Cypress to rebuild our Guides test suite...

The outcome...

Ask Questions at Sli.do

event code #cy-pendo

The pipeline...

Scheduled run

Regression suite

CI

Test case management system

Sanity suite

Manually triggered

Branch merge

Cypress dashboard

Slack channel

Regression Test example - Manual steps

Regression Suite

Regression suite

Regression Test example - The code

Regression Suite

describe('Alerts', function () {
    before(function () {
        cy.getUserAndLogin();

      	cy.launchGuidesList('iframes-page.html');
     	cy.intercept('GET', '**/route/to/layouts').as('loadLayouts');
      	cy.intercept('GET', '**/route/to/themes').as('loadThemes');

      	cy.createGuide({ layoutName: defaultLayoutNames.lightbox, isDefault: true });
    });

    after(function () {
        cy.returnUser();
    });

    describe('Activation Settings Alerts', function () {
        describe('Element found in multiple frames', function () {
            it('C490 Selects an element, div, that exists in multiple frames and outlines them in red', function () {
                // Adds a second step
                ActionBar.actionBarContainer()
                    .find('.action-bar-container-header')
                    .should('be.visible');
                ActionBar.stepsTrayToggle().click();
                actionBarIframe()
                    .find('.action-bar-container')
                    .should('have.class', 'expanded');
                ActionBar.newStepButton().click();
                cy.wait('@loadThemes')
                    .its('response.statusCode')
                    .should('eq', 200);
                cy.wait('@loadLayouts')
                    .its('response.statusCode')
                    .should('eq', 200);
                LayoutChooser.getLayoutByName(defaultLayoutNames.lightbox, true).click();
                actionBarIframe()
                    .find('.action-bar-container-tray-inner')
                    .find('.pendo-draggable[class*="has-draggable-"]')
                    .children()
                    .should('have.length', 2);

                // Select target
                CommonUFO.locationTab().should('be.visible').click();
                EditContainerLocation.elementLocationChooser()
                    .find('.pendo-button')
                    .click();
                LiveApplication.waitForNavigationControls();
                LiveApplication.customTargetedElement('.title').click({ force: true });
                EditContainerLocation.selectedElementLocation()
                    .should('contain', '.title');

                // Change target type to custom css and input div
                EditContainerLocation.elementLocationRadioButton('customCss').click();
                EditContainerLocation.customCssInput()
                    .clear()
                    .type('div');

                // Assert red outline on all divs targeted
                iFrameTown.iFrameApplication()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame1()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame2()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame3()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame4()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);

                // Change target type to parent
                EditContainerLocation.elementLocationRadioButton('suggestedMatch').click();
                cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 1);

                // Assert prior div selection will not have red outlines
                ActionBar.editStepButton().click();
                cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 0);
                ActionBar.editStep(1).click();
                cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 1);

                // Change target type to custom css and input div
                EditContainerLocation.elementLocationRadioButton('customCss').click();
                EditContainerLocation.customCssInput()
                    .clear()
                    .type('div');

                // Assert red outline on all divs targeted
                iFrameTown.iFrameApplication()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame1()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame2()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame3()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);
                iFrameTown.iFrame4()
                    .get('div[style*="outline: red solid 2px;"]')
                    .should('have.length', 2);

                // Assert alert
                Alerts.verifyAlertText('Step 2 uses an element target, but the target was found in multiple frames ' +
                                       'in your application. This can potentially lead to inconsistent guide display');
                Alerts.errorLink().click();

                // Resolve alert
                LiveApplication.waitForNavigationControls();
                LiveApplication.customTargetedElement('.iframe-container').click({ force: true });
                EditContainerLocation.selectedElementLocation()
                    .should('contain', 'iframe');
                ActionBar.alertButton().click();
                Alerts.verifyNoAlertText('You have no alerts.');
            });
        });
    });
});


// Separate file for the Action Bar
newStepButton () {
  return actionBarIframe().find('[data-cy="guide-step-add"]');
}

Regression suite

Regression Test example - Cypress test

Regression Suite

Regression suite

The pipeline...

Scheduled run

Regression suite

CI

Test case management system

Sanity suite

Manually triggered

Branch merge

Cypress dashboard

Slack channel

Test Retries & Cypress Dashboard

Cypress Dashboard

Cypress dashboard

Test Retries & Cypress Dashboard

 

Cypress Dashboard

Step 1: Enable test retries

 

Step 2: Check Flaky tests in Cypress Dashboard

 

Step 3: ????

 

Step 4: Profit

Cypress dashboard

Step 1: Enable test retries (it's really easy!)

 

Cypress Dashboard

Test retries is a feature where Cypress will automatically retry a failed test prior to marking it as failed.

 

Ensure you are on version >=5.0 and update your cypress.json (or config file):

 

 

Or visit the docs for other approaches

"retries": {
  "runMode": 3,
  "openMode": 0
}

Cypress dashboard

Step 2: Check Flaky tests in Cypress Dashboard

Cypress Dashboard

Cypress dashboard

Step 3: ????

Cypress Dashboard

Cypress dashboard

Step 4: Profit

Cypress Dashboard

Before

After

1. Flaky test fails

 

2. Engineer looks into failure as a priority

 

2. Cypress retries the test (the number of  times specified) and the test passes

 

3. Upon investigation engineer notices a    (potentially unrelated) test has failed

 

4. Engineer either fixes the test or reruns the test to see if it passes the second time

1. Flaky test fails

 

3. The flaky failure is recorded in the test results of Cypress Dashboard

4. The team work through any flaky tests recorded in Cypress Dashboard

Cypress dashboard

So how long does it take to run these tests?..

Cypress Dashboard

Sanity

Cypress dashboard

So how long does it take to run these tests?..

Cypress Dashboard

Regression

Cypress dashboard

The pipeline...

Scheduled run

Regression suite

CI

Test case management system

Sanity suite

Manually triggered

Branch merge

Cypress dashboard

Slack channel

Slack channel reporting
 

Slack channel

  • Provides a clear visibility of test results
     
  • Empowers the whole team to own the tests
     
  • Enables us to build a process around monitoring failures

Slack channel

The pipeline...

Scheduled run

Regression suite

CI

Test case management system

Sanity suite

Manually triggered

Branch merge

Cypress dashboard

Slack channel

Test case management integration
 

Test Case Management System

  • By utilising the Cypress plugins available we were able to report Cypress results to our test case management software
     
  • This gives us one source of truth for all our release testing
     
  • In CI we update the reporter in the config to a Mocha reporter using the plugins file

Test case management system

So, to summarise...

Ask Questions at Sli.do

event code #cy-pendo

We now have...

So, to summarise...

  • An increased product test coverage of ~34%
  • An increased test run speed of ~74% by using parallelisation
  • An increase in NPS scores for Quality
  • A surge of Cypress adoption and progression across our teams
     

We now have...

So, to summarise...

  • Stable tests
  • Visibility of test suites and results
    • Test results and metrics tracked in Cypress Dashboard
    • Manual and automated test results presented in our test case management system
       
  • Team driven maintenance processes for owning and fixing tests

&

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

Thank you!

Ask Questions at Sli.do event code

George Palfreyman

Senior Quality Engineer

@georgepalf

Cypress and Pendo

By Cypress.io

Cypress and Pendo

In this webcast, we'll introduce one of Pendo’s unique products—Guides— and the team’s approach to testing this complex application.

  • 816
Loading comments...

More from Cypress.io