6.8.6 Testability measures

What this heading covers

Testability has three parts: controllability, observability and the cost of the test. A coverage percentage is a partial measure and can be gamed. A better family of measures includes the proportion of requirements with an automated test, the time to set up a test environment, and the proportion of failures that can be diagnosed from the product's own output. The measure should be tied to the risk, because testing everything to the same depth is not a strategy.

Record position: family 31 of 40 in clause 6, under 6.8 Maintainability, which carries 5 families. It is one of the 40 families a programme has to map its measures to, and its elements are listed below. The heading is 6.8.6 in the public contents; the text behind it is not public.

How a measure for this family is built

Count the requirements or risks covered by an executable test as the numerator over those in scope. Add the median setup time and the proportion of failures with enough diagnostic output. Report the gaps by risk, not only as a coverage percentage.

What to put in place before publication

1. Map tests to requirements or risks, not only to code.
2. Measure the setup time and the diagnostic quality as well as coverage.

Evidence to keep

Two questions to ask

• Are tests mapped to requirements or risks, or only to code?
• How long does a test environment take to set up, and can a failure be diagnosed from the output?

Where teams usually fall short

• A line coverage figure presented as testability.

What the number cannot tell you

It cannot tell you whether the tests are good. A test can execute a requirement and assert nothing useful about it.

What happens next

Flexibility is next: the four families about changing context, scale and installation.