6.3.2 Time behaviour measures

What this heading covers

The number is the easy part; the conditions are the measure. Load, data volume, hardware, network and concurrent users all move the result, and the draft has no target values, so the requirement has to supply them. The family covers both response time, which a user feels, and processing time, which a batch or a job consumes. A measure that reports an average without a distribution hides the tail, and the tail is what users complain about.

Record position: family 4 of 40 in clause 6, under 6.3 Performance efficiency, which carries 3 families. It is one of the 40 families a programme has to map its measures to, and its elements are listed below. The heading is 6.3.2 in the public contents; the text behind it is not public.

How a measure for this family is built

Take the count of operations whose time is within the stated limit as the numerator and the count of operations measured as the denominator. Report the percentile that matters for the task as well as the mean, and record the profile. A mean alone is not a defensible time behaviour measure.

What to put in place before publication

1. Record the load profile with every result, and keep the raw timings.
2. State the percentile and the limit the result is judged against.

Evidence to keep

Two questions to ask

• What load, data volume and hardware produced this response time?
• Which percentile is reported, and what is the limit it is judged against?

Where teams usually fall short

• A response time from a warm cache on a developer machine, presented as the product result.

What the number cannot tell you

It cannot tell you how the product behaves at a load you did not test. A fast result on a quiet system says nothing about the peak.

What happens next

Read capacity next, because time behaviour at the limit is a different question from time behaviour at rest.