Cypress.io
When testing is easy, developers build better things faster and with confidence.
&
Gleb Bahmutov
Distinguished Engineer
@bahmutov
WEBCAST
How Pendo Maintains a Culture of Quality with Cypress
Ask Questions at Sli.do
event code #cy-pendo
George Palfreyman
Senior Quality Engineer
@georgepalf
George
- Senior Quality Engineer @ Pendo
- Sheffield, United Kingdom
- 6 years testing experience
- ~2 years working with automated testing
Ask Questions at Sli.do
event code #cy-pendo
"Pendo combines powerful product usage analytics with user guidance, communication, feedback, and planning tools to offer the most complete integrated platform for digital product teams. With Pendo, product teams can understand product usage, collect feedback, measure sentiment, onboard users, and announce new features in app—all without requiring engineering resources."
7 offices worldwide
1,700 customers
500+ employees
Reminder: Q&A
Ask Questions at Sli.do event code #cy-pendo
Pendo Products
- Insights
- Sentiment
- Feedback
- Mobile
- Integrations
- Adopt
- Guidance
Pendo Products
- Insights
- Sentiment
- Feedback
- Mobile
- Integrations
- Adopt
- Guidance
Guidance
Ask Questions at Sli.do
event code #cy-pendo
Guides provide personalised in-app guides and messaging
Guidance
Highlight new features
User onboarding and tutorials
Guidance
Polls & Surveys
Can be used to capture feedback on new products and features
Guidance
Resource Center
Houses additional help for users including:
- guides
- announcements
- FAQs
- help article links
- integrations
Guidance
Guide activation - Automatic
Guidance
Guide activation - Badge
Guidance
Guide activation - Target Element
Guidance
But how do we do this?..
Users launch the
Users can then create guides targeting their features and pages where required.
Visual Design Studio
over their application.
Guidance
- Used for building guides
- User ability to design many different types of guides (polls, surveys etc)
- User freedom to implement different types of content (images, video links etc)
- Users can combine these and build them at different lengths
- Can be launched over any customer site
Visual Design Studio
- Attracts great customers
Bonus point!
Who then go on to create...
Guidance
Beautiful walkthroughs...
Stunning polls...
And delightful resource centers...
Guidance
Stunning Polls...
Guidance
And delightful Resource Centers...
Before Cypress...
Ask Questions at Sli.do
event code #cy-pendo
- Functionality issues found during manual regression testing
- Manual testing focus of functionality rather than potential use cases
Manual Testing
- guide types (polls, surveys, walkthroughs)
- guide content (images, video links, buttons)
- guide lengths (single step guide... 15 step guide)
- guide activation (element, badge, automatically)
- Launched over any customer site (CSS rules)
General pain points potentially leading to false starts using test automation frameworks...
Automated Testing
Lack of maintenance and iterations
Flaky/unstable tests
Inconsistent test patterns - storing of web elements (with POM)
Lack of visibility of tests and results
Onboarding or prior knowledge required
Lack of team adoption
Lack of leadership buy in
- Drive quality culture within our engineering teams
- Everyone owns quality of our products
All eyes on quality...
- Revise implemented automation to catch regressions
- Focus manual testing on customer problems and use cases
- Needed a tool that works for the many and not just the few
- Good reputation and recommendations from engineers and the wider community
- Easy to get up and running
- Lots of documentation and community driven questions and answers online
And that's where Cypress comes in...
And that's where Cypress comes in...
Problem
Solution
Flaky/unstable tests
Cypress Retries
Inconsistent test patterns
Best practice of using data attributes
Static waits
Waiting on aliases and/or assertions
Lack of visibility
Cypress Dashboard
Onboarding or prior knowledge required
Front end tests with detailed documentation
Team adoption
Leadership buy in
Maintenance
Using Cypress to rebuild our Guides test suite...
Ask Questions at Sli.do
event code #cy-pendo
Using Cypress to rebuild our Guides test suite...
Sanity Suite
Regression Suite
Runs on all branches and blocks merge
Light touch of all areas
Full coverage of the Visual Design Studio
Tests are dependent on environment resources
Tests are independent and create their own resources and can be pointed at any environment
cypress/integration/sanitycypress/integration/regressionScheduled to run nightly on the release branch
1. Define test suite coverage...
- Environment already existed, just needed to configure data
- Create users
- Tools to clean up the data
3. Build the Sanity suite
4. Build the Regression suite
5. Document maintenance processes following Cypress best practices, leveraging Slack and Cypress Dashboard
2. Infrastructure setup and configuration
Using Cypress to rebuild our Guides test suite...
The outcome...
Ask Questions at Sli.do
event code #cy-pendo
The pipeline...
Scheduled run
Regression suite
CI
Test case management system
Sanity suite
Manually triggered
Branch merge
Cypress dashboard
Slack channel
Regression Test example - Manual steps
Regression Suite
Regression suite
Regression Test example - The code
Regression Suite
describe('Alerts', function () {
before(function () {
cy.getUserAndLogin();
cy.launchGuidesList('iframes-page.html');
cy.intercept('GET', '**/route/to/layouts').as('loadLayouts');
cy.intercept('GET', '**/route/to/themes').as('loadThemes');
cy.createGuide({ layoutName: defaultLayoutNames.lightbox, isDefault: true });
});
after(function () {
cy.returnUser();
});
describe('Activation Settings Alerts', function () {
describe('Element found in multiple frames', function () {
it('C490 Selects an element, div, that exists in multiple frames and outlines them in red', function () {
// Adds a second step
ActionBar.actionBarContainer()
.find('.action-bar-container-header')
.should('be.visible');
ActionBar.stepsTrayToggle().click();
actionBarIframe()
.find('.action-bar-container')
.should('have.class', 'expanded');
ActionBar.newStepButton().click();
cy.wait('@loadThemes')
.its('response.statusCode')
.should('eq', 200);
cy.wait('@loadLayouts')
.its('response.statusCode')
.should('eq', 200);
LayoutChooser.getLayoutByName(defaultLayoutNames.lightbox, true).click();
actionBarIframe()
.find('.action-bar-container-tray-inner')
.find('.pendo-draggable[class*="has-draggable-"]')
.children()
.should('have.length', 2);
// Select target
CommonUFO.locationTab().should('be.visible').click();
EditContainerLocation.elementLocationChooser()
.find('.pendo-button')
.click();
LiveApplication.waitForNavigationControls();
LiveApplication.customTargetedElement('.title').click({ force: true });
EditContainerLocation.selectedElementLocation()
.should('contain', '.title');
// Change target type to custom css and input div
EditContainerLocation.elementLocationRadioButton('customCss').click();
EditContainerLocation.customCssInput()
.clear()
.type('div');
// Assert red outline on all divs targeted
iFrameTown.iFrameApplication()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame1()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame2()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame3()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame4()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
// Change target type to parent
EditContainerLocation.elementLocationRadioButton('suggestedMatch').click();
cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 1);
// Assert prior div selection will not have red outlines
ActionBar.editStepButton().click();
cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 0);
ActionBar.editStep(1).click();
cy.get('div[style*="outline: red solid 2px;"]').should('have.length', 1);
// Change target type to custom css and input div
EditContainerLocation.elementLocationRadioButton('customCss').click();
EditContainerLocation.customCssInput()
.clear()
.type('div');
// Assert red outline on all divs targeted
iFrameTown.iFrameApplication()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame1()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame2()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame3()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
iFrameTown.iFrame4()
.get('div[style*="outline: red solid 2px;"]')
.should('have.length', 2);
// Assert alert
Alerts.verifyAlertText('Step 2 uses an element target, but the target was found in multiple frames ' +
'in your application. This can potentially lead to inconsistent guide display');
Alerts.errorLink().click();
// Resolve alert
LiveApplication.waitForNavigationControls();
LiveApplication.customTargetedElement('.iframe-container').click({ force: true });
EditContainerLocation.selectedElementLocation()
.should('contain', 'iframe');
ActionBar.alertButton().click();
Alerts.verifyNoAlertText('You have no alerts.');
});
});
});
});
// Separate file for the Action Bar
newStepButton () {
return actionBarIframe().find('[data-cy="guide-step-add"]');
}Regression suite
Regression Test example - Cypress test
Regression Suite
Regression suite
The pipeline...
Scheduled run
Regression suite
CI
Test case management system
Sanity suite
Manually triggered
Branch merge
Cypress dashboard
Slack channel
Test Retries & Cypress Dashboard
Cypress Dashboard
Cypress dashboard
Test Retries & Cypress Dashboard
Cypress Dashboard
Step 1: Enable test retries
Step 2: Check Flaky tests in Cypress Dashboard
Step 3: ????
Step 4: Profit
Cypress dashboard
Step 1: Enable test retries (it's really easy!)
Cypress Dashboard
Test retries is a feature where Cypress will automatically retry a failed test prior to marking it as failed.
Ensure you are on version >=5.0 and update your cypress.json (or config file):
Or visit the docs for other approaches
"retries": {
"runMode": 3,
"openMode": 0
}Cypress dashboard
Step 2: Check Flaky tests in Cypress Dashboard
Cypress Dashboard
Cypress dashboard
Step 3: ????
Cypress Dashboard
Cypress dashboard
Step 4: Profit
Cypress Dashboard
Before
After
1. Flaky test fails
2. Engineer looks into failure as a priority
2. Cypress retries the test (the number of times specified) and the test passes
3. Upon investigation engineer notices a (potentially unrelated) test has failed
4. Engineer either fixes the test or reruns the test to see if it passes the second time
1. Flaky test fails
3. The flaky failure is recorded in the test results of Cypress Dashboard
4. The team work through any flaky tests recorded in Cypress Dashboard
Cypress dashboard
So how long does it take to run these tests?..
Cypress Dashboard
Sanity
Cypress dashboard
So how long does it take to run these tests?..
Cypress Dashboard
Regression
Cypress dashboard
The pipeline...
Scheduled run
Regression suite
CI
Test case management system
Sanity suite
Manually triggered
Branch merge
Cypress dashboard
Slack channel
Slack channel reporting
Slack channel
-
Provides a clear visibility of test results
- Empowers the whole team to own the tests
- Enables us to build a process around monitoring failures
Slack channel
The pipeline...
Scheduled run
Regression suite
CI
Test case management system
Sanity suite
Manually triggered
Branch merge
Cypress dashboard
Slack channel
Test case management integration
Test Case Management System
- By utilising the Cypress plugins available we were able to report Cypress results to our test case management software
- This gives us one source of truth for all our release testing
- In CI we update the reporter in the config to a Mocha reporter using the plugins file
Test case management system
So, to summarise...
Ask Questions at Sli.do
event code #cy-pendo
We now have...
So, to summarise...
- An increased product test coverage of ~34%
- An increased test run speed of ~74% by using parallelisation
- An increase in NPS scores for Quality
- A surge of Cypress adoption and progression across our teams
- Sanity tests running on every pull request
- Regression tests running nightly
- independent tests following the best practice of using it.only
- independent tests following the best practice of using it.only
- A unified approach to selecting elements
We now have...
So, to summarise...
- Stable tests
- Less flake by preventing unnecessary waits
- Potential flake controlled by utilising test retries
- Visibility of test suites and results
- Test results and metrics tracked in Cypress Dashboard
- Manual and automated test results presented in our test case management system
- Team driven maintenance processes for owning and fixing tests
&
Gleb Bahmutov
Distinguished Engineer
@bahmutov
WEBCAST
Thank you!
Ask Questions at Sli.do event code
George Palfreyman
Senior Quality Engineer
@georgepalf
Cypress and Pendo
By Cypress.io
Cypress and Pendo
In this webcast, we'll introduce one of Pendo’s unique products—Guides— and the team’s approach to testing this complex application.
