Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue

TLDR

Think‑aloud protocols are widely used in usability testing, yet empirical evidence supporting their validity remains scarce. This study compares concurrent and retrospective think‑aloud protocols in a usability test of an online library catalogue. The comparison examined usability problems detected, overall task performance, and participant experiences. Concurrent and retrospective protocols identified comparable sets of problems, but retrospective protocols detected more via verbalization while concurrent protocols relied more on observation; moreover, concurrent protocols impaired task performance, raising concerns about their reactivity in complex tasks.

Abstract

Think-aloud protocols are a dominant method in usability testing. There is, however, only little empirical evidence on the actual validity of the method. This paper describes an experiment that compares concurrent and retrospective think-aloud protocols for a usability test of an online library catalogue. There were three points of comparison: usability problems detected, overall task performance, and participant experiences. Results show that concurrent and retrospective think-aloud protocols reveal comparable sets of usability problems, but that these problems come to light in different ways. In retrospective think-aloud protocols, more problems were detected by means of verbalisation, while in concurrent think-aloud protocols, more problems were detected by means of observation. Moreover, in the concurrent think-aloud protocols, the requirement to think aloud while working had a negative effect on the task performance. This raises questions about the reactivity of concurrent think-aloud protocols, especially in the case of high task complexity.