Building Test Coverage Momentum

At Jobber, we established 85% as our test coverage target on two of our largest codebases in order to increase our confidence and speed in making code changes. Through a combination of automation, visibility of progress, and establishing the importance of quality, we have maintained an upwards trajectory towards that target organically for several years. Here are the steps we took to build up our test coverage momentum.

Establishing the Importance of Quality

The amount of test coverage is important to us for several reasons:

• It impacts the effectiveness of continuous regression testing
• It is a key factor in gaining confidence in library or framework upgrades
• It enables the possibility of more automation, including automated deployments
• It correlates to how well the tests demonstrate how to use the code

This means that most PRs dont just have code changes - they also include the corresponding changes to tests.

Having a strong culture around pull request reviews is instrumental to this, but the key is that the importance of quality is a foundational part of the beliefs and values of the organization.

An example of this is our Quality is everyones responsibility engineering principle - part of a set of engineering principles endorsed and championed by leadership.

Ratcheting

to proceed by steps or degrees, in one direction only

SVG: Dr. Schorsch Animation: MichaelFrey, Ratchet Drawing Animation, CC BY-SA 3.0

Building test coverage momentum on an existing codebase is more challenging compared to establishing an overall target across a brand new repository. On an existing codebase, we want to use the current coverage (whatever it is) on each file as its goal. New files use the overall goal.

Before: Attempting to set a coverage goal on an existing codebase, without a strategy to deal with the existing files

After: Existing files that are below the overall goal get their own per-file goals - a starting point for ratcheting upwards!

Knowing the goals for both the existing files and new files, we enforce them as a minimum amount of test coverage in CI. A violation of these goals will result in a CI failure with a descriptive message. Actually failing the CI is important:

• It provides early feedback during the lifecycle of a PR (from the first draft) that the tests need further work.
• It ensures that by the time the PR is approved and ready to be released, our coverage goals havent been compromised.
• It helps spread the word about the importance of quality. Learning why something failed and how to fix it is always more useful for building the culture of quality than just mentioning it in leadership presentations.

This approach means that over time, the test coverage has nowhere to go but up - and the tooling will lock-in the gains on each files coverage percentage (up to the target) as the new goal!

Another complexity around establishing code coverage on an existing codebase is that we already have a test suite in place - large enough that parallelism is being leveraged on CI to keep execution times within acceptable limits. In order for ratcheting to work across the set of parallelized test runs, an independent step in the CI collects the detailed test coverage from each run and merges them together before leveraging it for enforcing goals.

Jobber has open-sourced the tool it uses to achieve this: @jobber/jest-a-coverage-slip-detector

This library can be used to ratchet coverage in projects large or small where jest is being used as the test framework, but theres a number of design considerations that make it particularly low-friction to use in larger projects, based on the following assumptions:

• You are unlikely to run the full set of tests locally as part of your usual development workflow, and so CI is the most reliable place to collect code coverage and check it against the per-file goals.
• You want your CI to be read-only. If per-file targets need to be updated, it can propose them and provided a guided experience to making the update, but the tooling shouldnt go so far as to modify files under source control in an automated way. Ultimately you want eyes on the change before it lands, as part of your pull request workflow.
• You are likely leveraging parallelism in your CI (perhaps using --shard) and would appreciate a library that automatically merges together the coverage.

To get started, follow the installation and configuration steps, and then perform a first run to setup the per-file coverage goals. Note that the provided CLI supports a --report-only option in case you want to start out with a soft-launch.

Visibility of Progress and Celebrating Improvements

Any team at Jobber can view a trend of the test coverage over time in our dashboarding tool. This helps teams answer questions such as:

• Is the test coverage trending in the right direction?
• Is it going in the right direction at an acceptable rate?
• What does the test coverage look like compared to last quarter?

Additionally, an automation reports the test coverage of a Pull Request (PR) as a comment right on the PR. The coverage details are available in an expandable summary that celebrates meeting or exceeding the target!

This automation intelligently identifies the set of code to put under test based on what files the PR is modifying - the test execution and code coverage generation part of this step is typically less than 60 seconds.

Conclusion

Its never too late to add code coverage goals to an existing codebase. Invest in tooling and automation to surface successes and failures within your Pull Request workflow, and leverage ratcheting to maintain test coverage momentum.

About Jobber

Our awesome Jobber technology teams span across Payments, Infrastructure, AI/ML, Business Workflows & Communications. We work on cutting edge & modern tech stacks using React, React Native, Ruby on Rails, & GraphQL.

If you want to be a part of a collaborative work culture, help small home service businesses scale and create a positive impact on our communities, then visit our careers site to learn more!