Evaluating a user interface with ergonomic criteria

TLDR

A mixed two‑factor experiment compared two groups of usability specialists evaluating a musical database management system, with the criteria group using ergonomic criteria during a second evaluation phase that involved replaying prior exploration, while the control group relied solely on expertise. The ergonomic criteria improved expert performance, yielding more new problems and higher detection rates in the second phase, whereas no differences were observed in the first phase.

Abstract

The usefulness of a set of ergonomic criteria for the evaluation of user interfaces was assessed using a mixed two‐factors experimental design. Two groups of usability specialists (control, criteria) were asked to evaluate the interface of a musical database management system in two phases. In the first phase of the experiment, all the participants relied solely on their expertise; in the second phase they were instructed to evaluate the management system again but this time through the replay of their previous exploration: Participants in the criteria group used a set of ergonomic criteria whereas the participants in the control group did not. In the first phase, the two groups did not differ in terms of: (a) the number of usability problems detected, and (b) the proportions of usability problems uncovered as well as the proportion of usability problems found in common, with respect to the number of single evaluations (one by participant) that were computed into combined evaluations (i.e., aggregates). In the second phase, however, the participants in the criteria group had better performances than those in the control group: They uncovered more new problems, and the proportion of problems uncovered as well as the proportion of problems found in common was greater as a function of the number of evaluators in the aggregates. To sum up, the criteria increased the evaluation performance of the experts.