Criteria For Evaluating Usability Evaluation Methods

TLDR

There is a wide range of usability evaluation methods for software systems, yet practitioners lack clear understanding of each method’s capabilities and limitations, and reliable comparison is hindered by the absence of standard evaluation criteria. The article aims to discuss factors, comparison criteria, and performance measures for usability evaluation methods, providing operational definitions and potential metrics to guide studies comparing UEMs. The authors propose a practical framework that identifies key factors, establishes comparison criteria, and defines performance measures for evaluating and comparing usability methods. The study identifies challenges in comparing UEMs and offers a starting point for refining evaluation principles and techniques.

Abstract

The current variety of alternative approaches to usability evaluation methods (UEMs) designed to assess and improve usability in software systems is offset by a general lack of understanding of the capabilities and limitations of each. Practitioners need to know which methods are more effective and in what ways and for what purposes. However, UEMs cannot be evaluated and compared reliably because of the lack of standard criteria for comparison. In this article, we present a practical discussion of factors, comparison criteria, and UEM performance measures useful in studies comparing UEMs. In demonstrating the importance of developing appropriate UEM evaluation criteria, we offer operational definitions and possible measures of UEM performance. We highlight specific challenges that researchers and practitioners face in comparing UEMs and provide a point of departure for further discussion and refinement of the principles and techniques used to approach UEM evaluation and comparison.