& 

Gleb Bahmutov

Distinguished Engineer

@bahmutov

WEBCAST

Covering your code

with end-to-end testing

Tom Hu

Developer Relations Lead

@thomasrockhu

https://cypress.slides.com/cypress-io/codecov

Ask Questions at Sli.do event code #codecov

Agenda

  1. Demo application
  2. Starting with code coverage
  3. Code coverage as a service
  4. Increasing code coverage
    1. 97%
    2. 99.10%
    3. 99.83%
    4. 100%
  5. Conclusions
  6. Q&A

Ask Questions at Sli.do event code #codecov

Ask Questions at Sli.do event code #codecov

Intro to Code Coverage with Cypress

https://www.cypress.io/blog/2019/08/08/webcast-recording-complete-code-coverage-with-cypress/ covers the basics of collecting and using code coverage in Cypress

Read Cypress guide https://on.cypress.io/code-coverage

RealWorldApp

For this webinar: https://github.com/bahmutov/cypress-realworld-app

Full version: https://github.com/cypress-io/cypress-realworld-app

First test: visitor -> signin

First test: visitor -> signin

describe("Auth", () => {
  it("redirects visitors to sign in page", () => {
    cy.task("db:seed");
    cy.visit("/personal");
    cy.location("pathname").should("equal", "/signin");
  });
});

What did we just test?

Alternative: What web application code did just run?

  1. Instrument code
  2. Run tests
  3. Report coverage

Ask Questions at Sli.do event code #codecov

Instrument Web App

- react-scripts start
+ react-scripts -r @cypress/instrument-cra start
yarn add -D @cypress/instrument-cra

Our application uses react-scripts

Change the start command

https://github.com/cypress-io/instrument-cra

Instrument Web App

For more instrumentation examples in JavaScript see

https://github.com/cypress-io/code-coverage#examples

Instrument Web App

Check by looking at window.__coverage__

Should have every instrumented JS file*

For every file, a counter for:

  • every function (f)
  • every statement (s)
  • every branch (b)

* lazy loading might surprise you 

Report Coverage

yarn add -D @cypress/code-coverage

https://github.com/cypress-io/code-coverage

// add to cypress/support/index.js file
import '@cypress/code-coverage/support'
// add to cypress/plugins/index.js file
module.exports = (on, config) => {
  require('@cypress/code-coverage/task')(on, config)
  return config
}

From the Cypress code coverage plugin

Click on the last "Coverage" command to print report details

Coverage Report: HTML

open coverage/index.html

totals

21% of statements

3% of branches

5% of functions

23% of lines

low coverage

medium coverage

high coverage

eliminate all red first?

low coverage

medium coverage

high coverage

or 

yellow to green?

Do you

Let's set CC limit to 90%!

But right now it is 21% ☹️

Ask Questions at Sli.do event code #codecov

Ask Questions at Sli.do event code #codecov

Longterm Improvement

Every pull request should preserve or increase the code coverage

what should it be compared to?

We need:

  1. 📈 A place to see the coverage report (as a team)
  2. 💾 A place to store coverage numbers
  3. ⛔️ A mechanism to fail pull requests that decrease code coverage

Code coverage as a Service!

  1. Dedicated code coverage solution

  2. Always free for open source

  3. Language agnostic

  4. Dedicated to working in the developer workflow

CircleCI test artifacts

run Cypress tests

- store_artifacts:
    path: coverage
- run: npx nyc report --reporter=text || true

circleci.yml

Codecov on CircleCI

orbs:
  codecov: codecov/codecov@1

run Cypress tests

- codecov/upload:
    file: coverage/coverage-final.json
    flags: ui

circleci.yml

https://docs.codecov.io/docs/supported-ci-providers

Codecov on CircleCI

orbs:
  codecov: codecov/codecov@1

run Cypress tests

- codecov/upload:
    file: coverage/coverage-final.json
    flags: ui

circleci.yml

[![codecov](https://codecov.io/gh/bahmutov/cypress-realworld-app/branch/develop/graph/badge.svg?token=<token>)]
(https://codecov.io/gh/bahmutov/cypress-realworld-app)

Codecov GitHub App

Enable to have Codecov set status on every pull request

Ask Questions at Sli.do event code #codecov

Tweak Codecov settings

https://docs.codecov.io/docs/codecov-yaml

codecov:
  require_ci_to_pass: yes
  
coverage:
  status:
    project:
      default:
        target: auto   # the required coverage value
        threshold: 1%  # the leniency in hitting the target
    patch:
      default:
        target: 80%
    	
comment:
  layout: "reach,diff,flags,files,footer"
  behavior: default
  require_changes: no

Let's Add Tests

import { User } from "../../src/models";
import { isMobile } from "../support/utils";

describe("User Sign-up and Login", function () {
  beforeEach(function () {
    cy.task("db:seed");

    cy.server();
    cy.route("POST", "/users").as("signup");
    cy.route("POST", "/bankAccounts").as("createBankAccount");
  });

  it("should redirect unauthenticated user to signin page", function () {
    cy.visit("/personal");
    cy.location("pathname").should("equal", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should remember a user for 30 days after login", function () {
    cy.database("find", "users").then((user: User) => {
      cy.login(user.username, "s3cret", true);
    });

    // Verify Session Cookie
    cy.getCookie("connect.sid").should("have.property", "expiry");

    // Logout User
    if (isMobile()) {
      cy.getBySel("sidenav-toggle").click();
    }
    cy.getBySel("sidenav-signout").click();
    cy.location("pathname").should("eq", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should allow a visitor to sign-up, login, and logout", function () {
    const userInfo = {
      firstName: "Bob",
      lastName: "Ross",
      username: "PainterJoy90",
      password: "s3cret",
    };

    // Sign-up User
    cy.visit("/");

    cy.getBySel("signup").click();
    cy.getBySel("signup-title").should("be.visible").and("contain", "Sign Up");
    cy.visualSnapshot("Sign Up Title");

    cy.getBySel("signup-first-name").type(userInfo.firstName);
    cy.getBySel("signup-last-name").type(userInfo.lastName);
    cy.getBySel("signup-username").type(userInfo.username);
    cy.getBySel("signup-password").type(userInfo.password);
    cy.getBySel("signup-confirmPassword").type(userInfo.password);
    cy.visualSnapshot("About to Sign Up");
    cy.getBySel("signup-submit").click();
    cy.wait("@signup");

    // Login User
    cy.login(userInfo.username, userInfo.password);

    // Onboarding
    cy.getBySel("user-onboarding-dialog").should("be.visible");
    cy.visualSnapshot("User Onboarding Dialog");
    cy.getBySel("user-onboarding-next").click();

    cy.getBySel("user-onboarding-dialog-title").should("contain", "Create Bank Account");

    cy.getBySelLike("bankName-input").type("The Best Bank");
    cy.getBySelLike("accountNumber-input").type("123456789");
    cy.getBySelLike("routingNumber-input").type("987654321");
    cy.visualSnapshot("About to complete User Onboarding");
    cy.getBySelLike("submit").click();

    cy.wait("@createBankAccount");

    cy.getBySel("user-onboarding-dialog-title").should("contain", "Finished");
    cy.getBySel("user-onboarding-dialog-content").should("contain", "You're all set!");
    cy.visualSnapshot("Finished User Onboarding");
    cy.getBySel("user-onboarding-next").click();

    cy.getBySel("transaction-list").should("be.visible");
    cy.visualSnapshot("Transaction List is visible after User Onboarding");

    // Logout User
    if (isMobile()) {
      cy.getBySel("sidenav-toggle").click();
    }
    cy.getBySel("sidenav-signout").click();
    cy.location("pathname").should("eq", "/signin");
    cy.visualSnapshot("Redirect to SignIn");
  });

  it("should display login errors", function () {
    cy.visit("/");

    cy.getBySel("signin-username").type("User").find("input").clear().blur();
    cy.get("#username-helper-text").should("be.visible").and("contain", "Username is required");
    cy.visualSnapshot("Display Username is Required Error");

    cy.getBySel("signin-password").type("abc").find("input").blur();
    cy.get("#password-helper-text")
      .should("be.visible")
      .and("contain", "Password must contain at least 4 characters");
    cy.visualSnapshot("Display Password Error");

    cy.getBySel("signin-submit").should("be.disabled");
    cy.visualSnapshot("Sign In Submit Disabled");
  });

  it("should display signup errors", function () {
    cy.visit("/signup");

    cy.getBySel("signup-first-name").type("First").find("input").clear().blur();
    cy.get("#firstName-helper-text").should("be.visible").and("contain", "First Name is required");

    cy.getBySel("signup-last-name").type("Last").find("input").clear().blur();
    cy.get("#lastName-helper-text").should("be.visible").and("contain", "Last Name is required");

    cy.getBySel("signup-username").type("User").find("input").clear().blur();
    cy.get("#username-helper-text").should("be.visible").and("contain", "Username is required");

    cy.getBySel("signup-password").type("password").find("input").clear().blur();
    cy.get("#password-helper-text").should("be.visible").and("contain", "Enter your password");

    cy.getBySel("signup-confirmPassword").type("DIFFERENT PASSWORD").find("input").blur();
    cy.get("#confirmPassword-helper-text")
      .should("be.visible")
      .and("contain", "Password does not match");
    cy.visualSnapshot("Display Sign Up Required Errors");

    cy.getBySel("signup-submit").should("be.disabled");
    cy.visualSnapshot("Sign Up Submit Disabled");
  });

  it("should error for an invalid user", function () {
    cy.login("invalidUserName", "invalidPa$$word");

    cy.getBySel("signin-error")
      .should("be.visible")
      .and("have.text", "Username or password is invalid");
    cy.visualSnapshot("Sign In, Invalid Username and Password, Username or Password is Invalid");
  });

  it("should error for an invalid password for existing user", function () {
    cy.database("find", "users").then((user: User) => {
      cy.login(user.username, "INVALID");
    });

    cy.getBySel("signin-error")
      .should("be.visible")
      .and("have.text", "Username or password is invalid");
    cy.visualSnapshot("Sign In, Invalid Username, Username or Password is Invalid");
  });
});

💡 focus on testing the features

And open a pull request

https://github.com/bahmutov/cypress-realworld-app/pull/1

Sunburst Chart

If we disable a test...

it.skip("should allow a visitor to sign-up, login, and logout", function () {
  ...
})

https://github.com/bahmutov/cypress-realworld-app/pull/2

If we disable a test...

it.skip("should allow a visitor to sign-up, login, and logout", function () {
  ...
})

https://github.com/bahmutov/cypress-realworld-app/pull/2

Write Feature E2E Tests

and watch code coverage 📈

Ask Questions at Sli.do event code #codecov

Test "Bank Accounts" feature 

https://github.com/bahmutov/cypress-realworld-app/pull/3

Test "Bank Accounts" feature 

https://github.com/bahmutov/cypress-realworld-app/pull/3

Test transactions

https://github.com/bahmutov/cypress-realworld-app/pull/4

Test transactions

https://github.com/bahmutov/cypress-realworld-app/pull/4

Feature Code coverage Δ
user search no change
notifications +7%
user settings +4%
transaction feed +6%
transaction view +1%
Total 97%

Disclaimer: the application code is already written, we are just adding tests

code already covered by other tests

Parallel CI

The tests start to take too long when running on a single CI machine

- cypress/run:
    record: true
    parallel: true
    parallelism: 2

https://github.com/cypress-io/circleci-orb#parallel

circleci.yml

Runs on 2 machines

The coverage remained the same 🎉

97%

Where is the missing 3%?

src/utils/transactionUtils.ts

Our app has backend and frontend

 

backend/
  app.ts
  auth.ts
  ...
src/
  components/
  containers/
  ...

We completely ignored the backend code

used in

Instrument the Backend

yarn add -D nyc

NYC CLI instruments code https://istanbul.js.org/

- node ./backend
+ nyc --silent node ./backend

Change the start command during tests

https://github.com/cypress-io/code-coverage#instrument-backend-code

if (global.__coverage__) {
  require("@cypress/code-coverage/middleware/express")(app);
}

you can fetch code coverage at /__coverage__ endpoint

"codeCoverage": {
  "url": "http://localhost:3001/__coverage__"
}

cypress.json

Instrument the Backend

code coverage for each visited page

code coverage for the backend

code coverage report

https://github.com/bahmutov/cypress-realworld-app/pull/10

☹️

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

https://github.com/bahmutov/cypress-realworld-app/pull/10

😄

Codecov Flags

Measure coverage based on uploads

Upload separate coverage for frontend and backend to track coverage changes to each part of the product

  report-coverage:
    description: |
      Store coverage report as an artifact and send it to Codecov service.
   parameters:
      flag:
        type: string
        default: ""
    steps:
      - store_artifacts:
          path: coverage
      - run: npx nyc report --reporter=text || true
      - codecov/upload:
          file: coverage/coverage-final.json
          flags: << parameters.flag >>

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the CircleCI Config

- cypress/run:
    name: API tests
    executor: cypress/base-12-18-3
    spec: "cypress/tests/api/*"
    yarn: true
    start: yarn start:ci
    wait-on: "http://localhost:3000"
    requires:
      - cypress/install
    record: true
    group: api
    post-steps:
      - report-coverage:
          flag: backend
    # do not preserve the workspace in this job, since
    # there are no jobs that follow it
    no-workspace: true

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the CircleCI Config

coverage:
  status:
    project:
      default:
        target: 90%
      backend:
      	target: 80%
        flags:
          - backend
flags:
  backend:
    paths:
      - backend/

https://github.com/bahmutov/cypress-realworld-app/pull/10

Updating the Codecov Config

https://github.com/bahmutov/cypress-realworld-app/pull/10

Viewing Flags

Where is the missing 5%?

The edge cases in the backend

Ask Questions at Sli.do event code #codecov

We could use E2E tests to reach those lines

Or we could just run API tests to directly hit them

Cypress API test

it("gets a list of contacts by username", function () {
  const { username } = ctx.authenticatedUser!;
  cy.request("GET", `${apiContacts}/${username}`)
    .then((response) => {
      expect(response.status).to.eq(200);
      expect(response.body.contacts[0]).to.have.property("userId");
    });
});

https://www.cypress.io/blog/2017/11/07/add-gui-to-your-e2e-api-tests/

https://github.com/bahmutov/cypress-realworld-app/pull/11

Nice 🎉 but why not 100%?

Missed lines

TransactionListAmountRangeFilter.tsx

Depending on the xsBreakpoint we ran the top code twice, and the bottom code zero times

const xsBreakpoint = 
  useMediaQuery(theme.breakpoints.only("xs"));

Application renders differently depending on the viewport

Missed lines solution

TransactionListAmountRangeFilter.tsx

Run the UI tests in desktop and mobule resolutions

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

jobs:
  - cypress/install

  - cypress/run:
      name: API tests
      spec: "cypress/tests/api/*"
      requires:
        - cypress/install
      record: true
      group: api
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E desktop tests
      spec: "cypress/tests/ui/*"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: e2e
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E mobile tests
      spec: "cypress/tests/ui/*"
      config: "viewportWidth=375,viewportHeight=667"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: "e2e mobile"
      post-steps:
        - report-coverage

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

jobs:
  - cypress/install

  - cypress/run:
      name: API tests
      spec: "cypress/tests/api/*"
      requires:
        - cypress/install
      record: true
      group: api
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E desktop tests
      spec: "cypress/tests/ui/*"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: e2e
      post-steps:
        - report-coverage

  - cypress/run:
      name: E2E mobile tests
      spec: "cypress/tests/ui/*"
      config: "viewportWidth=375,viewportHeight=667"
      requires:
        - cypress/install
      record: true
      parallel: true
      parallelism: 2
      group: "e2e mobile"
      post-steps:
        - report-coverage

2x

2x

CI config

https://github.com/bahmutov/cypress-realworld-app/pull/12

Towards 100%

We are missing 1 line!

Towards 100%

Find the file

Towards 100%

setMessage was never called

Towards 100%

setMessage is called if another action fails

Towards 100%

For example if loading bank accounts fails...

Towards 100%

If this line was never covered by our tests, then ... we never tested how the application handles these particular failures!

Towards 100%

it.only("shows loading accounts error", () => {
  cy.intercept("/bankAccounts", { forceNetworkError: true })
    .as("getBankAccounts");
  cy.visit("/bankaccounts");
  cy.wait("@getBankAccounts");
    
  cy.getBySel("empty-list-header")
    .should("contain", "No Bank Accounts")
    .should("be.visible");
});

Local test

Hits this line

On CI

https://github.com/bahmutov/cypress-realworld-app/pull/13

💯

Tip: look at "istanbul ignore"

66 coverage exceptions!

valid exception

Tip: look at "istanbul ignore"

valid exception

Tip: look at "istanbul ignore"

probably needs an API test

Ask Questions at Sli.do event code #codecov

Conclusions

Introduce the code coverage metric early

At first, just write feature tests, the coverage will improve

Ask Questions at Sli.do event code #codecov

Conclusions

Track code coverage and prevent poorly tested code changes with Codecov

Conclusions

Read coverage reports as a map to find the missing tests

Gleb Bahmutov

Distinguished Engineer

@bahmutov

Thank you 👏

Tom Hu

Developer Relations Lead

@thomasrockhu

https://cypress.slides.com/cypress-io/codecov

https://docs.codecov.io/docs/common-recipe-list

Cypress and Codecov

By Cypress.io

Cypress and Codecov

In this webcast, Gleb Bahmutov, Distinguished Engineer at Cypress, and Tom Hu, Developer Relations Lead at Codecov, will show you how to use Cypress tests with code coverage to write highly durable code that allows you to deploy with confidence. We’ll explain how to focus your efforts with code coverage, and what a good coverage value looks like. Next, we’ll discuss how Codecov fits into your workflow and how you can use coverage metrics to inform your testing. We’ll also show you how your Cypress tests can leverage Codecov to ship safer code. Finally, we’ll demonstrate how you can quickly get Cypress and Codecov up and running in your setups.

  • 2,951

More from Cypress.io