100% Test Coverage is a Bad Metric
Unit tests, developer’s best friends, help the maintainability of a code base. But what makes a unit test good? What makes a test superfluous vs. effective?
Improving you unit test foo.
For the last 3 years I have actively written unit tests for 20-100% of my job. It became clear that my earlier unit tests were not so good as bugs existed in the code, though i had coverage. In fact, more than once, I had 100% coverage and somehow ended up with several bugs. It became clear that coverage does not imply quality
Lets create a simple method,
isPathValue. What is a PathValue?
path and a
value key. Lets just write the simplest version of
The function contains several bugs which we will capture through
In our example, the output space is requires a minimum amount of tests.
Since the output type is boolean, there are only two potential values,
For this example the output space is a great place to start testing. Its limited in scope and hopefully will catch some simple bugs. As a side note, we have achieved 100% coverage, despite very little testing.
In our example, input space is more complex. There are several types
and values that could be passed in. The potential values of
x could be:
- null, undefined
- NaN, Infinity
And the values for
value can be any type from above. Now its
time to narrow down the potential tests since we don’t want to write 7
tests for the value of
x and 7x7 for
x.value. Instead, lets
redefine the input space and group together like values.
Falsey: Simple (skipping the more advanced falseys).
Values that throw errors when accessed with dot separators.
Values that don’t throw errors when accessed with dot separators:
Values that properties can be added to:
With that definition lets define our tests:
Falsy value for
x should be a value that will throw errors if dot separators are used.
One of the requirements is that the object only has
x should have more than just
x should be an
value key to ensure we object check.
Several bugs will be found, including errors that are thrown. So lets
isPathValue function to correctly pass all of our unit tests.
Unit test coverage is a metric that does not indicate quality of tests. One of the most important things to take away is testing functions should never consider internal implementations. Tests should only consider the interface. This way, as the internals change, we ensure that the contract continues to work.
The testing here only considers function testing, Object testing is considerably harder since method behavior can be/is little/largly driven by the state of the Object.