&

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

Covering your code

with end-to-end testing

Tom Hu

Developer Relations Lead

@thomasrockhu

https://cypress.slides.com/cypress-io/codecov

Ask Questions at Sli.do event code #codecov

Agenda

Demo application
Starting with code coverage
Code coverage as a service
Increasing code coverage
1. 97%
2. 99.10%
3. 99.83%
4. 100%
Conclusions
Q&A

Ask Questions at Sli.do event code #codecov

Intro to Code Coverage with Cypress

https://www.cypress.io/blog/2019/08/08/webcast-recording-complete-code-coverage-with-cypress/ covers the basics of collecting and using code coverage in Cypress

Read Cypress guide https://on.cypress.io/code-coverage

RealWorldApp

For this webinar: https://github.com/bahmutov/cypress-realworld-app

Full version: https://github.com/cypress-io/cypress-realworld-app

First test: visitor -> signin

describe("Auth", () => {
  it("redirects visitors to sign in page", () => {
    cy.task("db:seed");
    cy.visit("/personal");
    cy.location("pathname").should("equal", "/signin");
  });
});

What did we just test?

Alternative: What web application code did just run?

Instrument code
Run tests
Report coverage

Ask Questions at Sli.do event code #codecov

Instrument Web App

- react-scripts start
+ react-scripts -r @cypress/instrument-cra start

yarn add -D @cypress/instrument-cra

Our application uses react-scripts

Change the start command

https://github.com/cypress-io/instrument-cra

Instrument Web App

For more instrumentation examples in JavaScript see

https://github.com/cypress-io/code-coverage#examples

Instrument Web App

Check by looking at window.__coverage__

Should have every instrumented JS file*

For every file, a counter for:

every function (f)
every statement (s)
every branch (b)

* lazy loading might surprise you

Report Coverage

yarn add -D @cypress/code-coverage

https://github.com/cypress-io/code-coverage

// add to cypress/support/index.js file
import '@cypress/code-coverage/support'

// add to cypress/plugins/index.js file
module.exports = (on, config) => {
  require('@cypress/code-coverage/task')(on, config)
  return config
}

From the Cypress code coverage plugin

Click on the last "Coverage" command to print report details

Coverage Report: HTML

open coverage/index.html

totals

21% of statements

3% of branches

5% of functions

23% of lines

low coverage

medium coverage

high coverage

eliminate all red first?

low coverage

medium coverage

high coverage

yellow to green?

Do you

Let's set CC limit to 90%!

But right now it is 21% ☹️

Ask Questions at Sli.do event code #codecov

Longterm Improvement

Every pull request should preserve or increase the code coverage

what should it be compared to?

We need:

📈 A place to see the coverage report (as a team)
💾 A place to store coverage numbers
⛔️ A mechanism to fail pull requests that decrease code coverage

Code coverage as a Service!

Dedicated code coverage solution
Always free for open source
Language agnostic
Dedicated to working in the developer workflow

CircleCI test artifacts

run Cypress tests

- store_artifacts:
    path: coverage
- run: npx nyc report --reporter=text || true

circleci.yml

Codecov on CircleCI

orbs:
  codecov: codecov/codecov@1

run Cypress tests

- codecov/upload:
    file: coverage/coverage-final.json
    flags: ui

circleci.yml

https://docs.codecov.io/docs/supported-ci-providers

Codecov on CircleCI

orbs:
  codecov: codecov/codecov@1

run Cypress tests

- codecov/upload:
    file: coverage/coverage-final.json
    flags: ui

circleci.yml

[![codecov](https://codecov.io/gh/bahmutov/cypress-realworld-app/branch/develop/graph/badge.svg?token=<token>)]
(https://codecov.io/gh/bahmutov/cypress-realworld-app)

Codecov GitHub App

Enable to have Codecov set status on every pull request

Ask Questions at Sli.do event code #codecov

Tweak Codecov settings

https://docs.codecov.io/docs/codecov-yaml

codecov:
  require_ci_to_pass: yes
  
coverage:
  status:
    project:
      default:
        target: auto   # the required coverage value
        threshold: 1%  # the leniency in hitting the target
    patch:
      default:
        target: 80%
    	
comment:
  layout: "reach,diff,flags,files,footer"
  behavior: default
  require_changes: no

Let's Add Tests

import { User } from "../../src/models";
import { isMobile } from "../support/utils";

describe("User Sign-up and Login", function () {
  beforeEach(function () {
    cy.task("db:seed");

    cy.server();
    cy.route("POST", "/users").as("signup");
    cy.route("POST", "/bankAccounts").as("createBankAccount");
  });

  it("should redirect unauthenticated user to signin page", function () {
    cy.visit("/personal");
    cy.location("pathname").should("equal", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should remember a user for 30 days after login", function () {
    cy.database("find", "users").then((user: User) => {
      cy.login(user.username, "s3cret", true);
    });

    // Verify Session Cookie
    cy.getCookie("connect.sid").should("have.property", "expiry");

    // Logout User
    if (isMobile()) {
      cy.getBySel("sidenav-toggle").click();
    }
    cy.getBySel("sidenav-signout").click();
    cy.location("pathname").should("eq", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should allow a visitor to sign-up, login, and logout", function () {
    const userInfo = {
      firstName: "Bob",
      lastName: "Ross",
      username: "PainterJoy90",
      password: "s3cret",
    };

    // Sign-up User
    cy.visit("/");

    cy.getBySel("signup").click();
    cy.getBySel("signup-title").should("be.visible").and("contain", "Sign Up");
    cy.visualSnapshot("Sign Up Title");

    cy.getBySel("signup-first-name").type(userInfo.firstName);
    cy.getBySel("signup-last-name").type(userInfo.lastName);
    cy.getBySel("signup-username").type(userInfo.username);
    cy.getBySel("signup-password").type(userInfo.password);
    cy.getBySel("signup-confirmPassword").type(userInfo.password);
    cy.visualSnapshot("About to Sign Up");
    cy.getBySel("signup-submit").click();
    cy.wait("@signup");

    // Login User
    cy.login(userInfo.username, userInfo.password);

    // Onboarding
    cy.getBySel("user-onboarding-dialog").should("be.visible");
    cy.visualSnapshot("User Onboarding Dialog");
    cy.getBySel("user-onboarding-next").click();

    cy.getBySel("user-onboarding-dialog-title").should("contain", "Create Bank Account");

    cy.getBySelLike("bankName-input").type("The Best Bank");
    cy.getBySelLike("accountNumber-input").type("123456789");
    cy.getBySelLike("routingNumber-input").type("987654321");
    cy.visualSnapshot("About to complete User Onboarding");
    cy.getBySelLike("submit").click();

    cy.wait("@createBankAccount");

    cy.getBySel("user-onboarding-dialog-title").should("contain", "Finished");
    cy.getBySel("user-onboarding-dialog-content").should("contain", "You're all set!");
    cy.visualSnapshot("Finished User Onboarding");
    cy.getBySel("user-onboarding-next").click();

    cy.getBySel("transaction-list").should("be.visible");
    cy.visualSnapshot("Transaction List is visible after User Onboarding");

    // Logout User
    if (isMobile()) {
      cy.getBySel("sidenav-toggle").click();
    }
    cy.getBySel("sidenav-signout").click();
    cy.location("pathname").should("eq", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should display login errors", function () {
    cy.visit("/");

    cy.getBySel("signin-username").type("User").find("input").clear().blur();
    cy.get("#username-helper-text").should("be.visible").and("contain", "Username is required");
    cy.visualSnapshot("Display Username is Required Error");

    cy.getBySel("signin-password").type("abc").find("input").blur();
    cy.get("#password-helper-text")
      .should("be.visible")
      .and("contain", "Password must contain at least 4 characters");
    cy.visualSnapshot("Display Password Error");

    cy.getBySel("signin-submit").should("be.disabled");
    cy.visualSnapshot("Sign In Submit Disabled");
  });

  it("should display signup errors", function () {
    cy.visit("/signup");

    cy.getBySel("signup-first-name").type("First").find("input").clear().blur();
    cy.get("#firstName-helper-text").should("be.visible").and("contain", "First Name is required");

    cy.getBySel("signup-last-name").type("Last").find("input").clear().blur();
    cy.get("#lastName-helper-text").should("be.visible").and("contain", "Last Name is required");

    cy.getBySel("signup-username").type("User").find("input").clear().blur();
    cy.get("#username-helper-text").should("be.visible").and("contain", "Username is required");

    cy.getBySel("signup-password").type("password").find("input").clear().blur();
    cy.get("#password-helper-text").should("be.visible").and("contain", "Enter your password");

    cy.getBySel("signup-confirmPassword").type("DIFFERENT PASSWORD").find("input").blur();
    cy.get("#confirmPassword-helper-text")
      .should("be.visible")
      .and("contain", "Password does not match");
    cy.visualSnapshot("Display Sign Up Required Errors");

    cy.getBySel("signup-submit").should("be.disabled");
    cy.visualSnapshot("Sign Up Submit Disabled");
  });

  it("should error for an invalid user", function () {
    cy.login("invalidUserName", "invalidPa$$word");

    cy.getBySel("signin-error")
      .should("be.visible")
      .and("have.text", "Username or password is invalid");
    cy.visualSnapshot("Sign In, Invalid Username and Password, Username or Password is Invalid");
  });

  it("should error for an invalid password for existing user", function () {
    cy.database("find", "users").then((user: User) => {
      cy.login(user.username, "INVALID");
    });

    cy.getBySel("signin-error")
      .should("be.visible")
      .and("have.text", "Username or password is invalid");
    cy.visualSnapshot("Sign In, Invalid Username, Username or Password is Invalid");
  });
});

💡 focus on testing the features

And open a pull request

https://github.com/bahmutov/cypress-realworld-app/pull/1

Sunburst Chart

If we disable a test...

it.skip("should allow a visitor to sign-up, login, and logout", function () {
  ...
})

https://github.com/bahmutov/cypress-realworld-app/pull/2

If we disable a test...

it.skip("should allow a visitor to sign-up, login, and logout", function () {
  ...
})

https://github.com/bahmutov/cypress-realworld-app/pull/2

Write Feature E2E Tests

and watch code coverage 📈

Ask Questions at Sli.do event code #codecov

Test "Bank Accounts" feature

https://github.com/bahmutov/cypress-realworld-app/pull/3

Test "Bank Accounts" feature

https://github.com/bahmutov/cypress-realworld-app/pull/3

Test transactions

https://github.com/bahmutov/cypress-realworld-app/pull/4

Test transactions

https://github.com/bahmutov/cypress-realworld-app/pull/4

Feature	Code coverage Δ
user search	no change
notifications	+7%
user settings	+4%
transaction feed	+6%
transaction view	+1%
Total	97%

Disclaimer: the application code is already written, we are just adding tests

code already covered by other tests

Parallel CI

The tests start to take too long when running on a single CI machine

- cypress/run:
    record: true
    parallel: true
    parallelism: 2

https://github.com/cypress-io/circleci-orb#parallel

circleci.yml

Runs on 2 machines

The coverage remained the same 🎉

97%

Where is the missing 3%?

src/utils/transactionUtils.ts

Our app has backend and frontend

backend/
  app.ts
  auth.ts
  ...
src/
  components/
  containers/
  ...

We completely ignored the backend code

used in

Instrument the Backend

yarn add -D nyc

NYC CLI instruments code https://istanbul.js.org/

- node ./backend
+ nyc --silent node ./backend

Change the start command during tests

https://github.com/cypress-io/code-coverage#instrument-backend-code

if (global.__coverage__) {
  require("@cypress/code-coverage/middleware/express")(app);
}

you can fetch code coverage at /__coverage__ endpoint

"codeCoverage": {
  "url": "http://localhost:3001/__coverage__"
}

cypress.json

Instrument the Backend

code coverage for each visited page

code coverage for the backend

code coverage report

https://github.com/bahmutov/cypress-realworld-app/pull/10

☹️

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

Codecov Flags

Measure coverage based on uploads

Upload separate coverage for frontend and backend to track coverage changes to each part of the product

  report-coverage:
    description: |
      Store coverage report as an artifact and send it to Codecov service.
   parameters:
      flag:
        type: string
        default: ""
    steps:
      - store_artifacts:
          path: coverage
      - run: npx nyc report --reporter=text || true
      - codecov/upload:
          file: coverage/coverage-final.json
          flags: << parameters.flag >>

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the CircleCI Config

- cypress/run:
    name: API tests
    executor: cypress/base-12-18-3
    spec: "cypress/tests/api/*"
    yarn: true
    start: yarn start:ci
    wait-on: "http://localhost:3000"
    requires:
      - cypress/install
    record: true
    group: api
    post-steps:
      - report-coverage:
          flag: backend
    # do not preserve the workspace in this job, since
    # there are no jobs that follow it
    no-workspace: true

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the CircleCI Config

coverage:
  status:
    project:
      default:
        target: 90%
      backend:
      	target: 80%
        flags:
          - backend
flags:
  backend:
    paths:
      - backend/

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the Codecov Config

https://github.com/bahmutov/cypress-realworld-app/pull/10

Viewing Flags

Where is the missing 5%?

The edge cases in the backend

Ask Questions at Sli.do event code #codecov

We could use E2E tests to reach those lines

Or we could just run API tests to directly hit them

Cypress API test

it("gets a list of contacts by username", function () {
  const { username } = ctx.authenticatedUser!;
  cy.request("GET", `${apiContacts}/${username}`)
    .then((response) => {
      expect(response.status).to.eq(200);
      expect(response.body.contacts[0]).to.have.property("userId");
    });
});

https://www.cypress.io/blog/2017/11/07/add-gui-to-your-e2e-api-tests/

https://github.com/bahmutov/cypress-realworld-app/pull/11

Nice 🎉 but why not 100%?

Missed lines

TransactionListAmountRangeFilter.tsx

Depending on the xsBreakpoint we ran the top code twice, and the bottom code zero times

const xsBreakpoint = 
  useMediaQuery(theme.breakpoints.only("xs"));

Application renders differently depending on the viewport

Missed lines solution

TransactionListAmountRangeFilter.tsx

Run the UI tests in desktop and mobule resolutions

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

jobs:
  - cypress/install

  - cypress/run:
      name: API tests
      spec: "cypress/tests/api/*"
      requires:
        - cypress/install
      record: true
      group: api
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E desktop tests
      spec: "cypress/tests/ui/*"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: e2e
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E mobile tests
      spec: "cypress/tests/ui/*"
      config: "viewportWidth=375,viewportHeight=667"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: "e2e mobile"
      post-steps:
        - report-coverage

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

jobs:
  - cypress/install

  - cypress/run:
      name: API tests
      spec: "cypress/tests/api/*"
      requires:
        - cypress/install
      record: true
      group: api
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E desktop tests
      spec: "cypress/tests/ui/*"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: e2e
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E mobile tests
      spec: "cypress/tests/ui/*"
      config: "viewportWidth=375,viewportHeight=667"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: "e2e mobile"
      post-steps:
        - report-coverage

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

Towards 100%

We are missing 1 line!

Towards 100%

Find the file

Towards 100%

setMessage was never called

Towards 100%

setMessage is called if another action fails

Towards 100%

For example if loading bank accounts fails...

Towards 100%

If this line was never covered by our tests, then ... we never tested how the application handles these particular failures!

Towards 100%

it.only("shows loading accounts error", () => {
  cy.intercept("/bankAccounts", { forceNetworkError: true })
    .as("getBankAccounts");
  cy.visit("/bankaccounts");
  cy.wait("@getBankAccounts");
    
  cy.getBySel("empty-list-header")
    .should("contain", "No Bank Accounts")
    .should("be.visible");
});