A mathematical model of the finding of usability problems

TLDR

The model enables planners to determine how many evaluations are needed to reach desired thoroughness or benefit levels. By incorporating evaluation costs and detection values, the model calculates the optimal number of evaluations for cost/benefit maximization and identifies when marginal utility ceases. Across 11 studies, usability problem detection follows a Poisson process; early results estimate remaining problems and needed evaluations, with a medium example suggesting 16 evaluations are cost‑effective and a maximum benefit/cost at four.

Abstract

For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.