Concurrent and Retrospective Verbal Protocols in Usability Testing: Is there Value Added in Collecting Both?

Abstract

This study compared the relative sensitivities of the concurrent and retrospective verbal protocol methods in an error analysis of human-computer interaction. Twelve subjects performed bibliographic retrieval tasks in “walk-up-and-use” usability test sessions. Each subject provided concurrent “think aloud” verbal reports throughout task performance, and then provided retrospective verbal reports while viewing their video taped performance without sound. Verbal reports that related specifically to errors were encoded as mistakes or slips, in an error classification schema which distinguished errors of intent (mistakes), from unintentional errors (slips). These data were analyzed in a 2×2 (error type x verbal protocol) repeated measures analysis of variance. In a second design, verbal protocols were encoded by statement type and analyzed in a 3×2 (statement type x verbal protocol) repeated measures analysis of variance. This study demonstrated an interaction between concurrent and retrospective methodologies and the types of error-related statements elicited. In addition, this within-subjects study supported the findings of the previous between-groups study (Bowers and Snyder, 1990). The findings support the practice of collecting concurrent verbal reports and then following up with retrospective verbal reports to collect additional information about mistakes and complex design problems.