Evaluation Strategies for HCI Toolkit Research

TLDR

Toolkit research is crucial in HCI for shaping interactive systems, yet its evaluation is problematic because the field lacks clear definitions of what constitutes evaluating a toolkit and which methods are suitable. This study aims to survey, reflect on, and discuss evaluation methods for toolkit contributions in HCI. The authors analyzed 68 published toolkit papers to investigate evaluation practices. They identified four evaluation strategies, categorized them, and discussed each strategy’s value, techniques, limitations, and trade-offs.

Abstract

Toolkit research plays an important role in the field of HCI, as it can heavily influence both the design and implementa-tion of interactive systems. For publication, the HCI commu-nity typically expects toolkit research to include an evalua-tion component. The problem is that toolkit evaluation is challenging, as it is often unclear what 'evaluating' a toolkit means and what methods are appropriate. To address this problem, we analyzed 68 published toolkit papers. From our analysis, we provide an overview of, reflection on, and dis-cussion of evaluation methods for toolkit contributions. We identify and discuss the value of four toolkit evaluation strat-egies, including the associated techniques that each employs. We offer a categorization of evaluation strategies for toolkit researchers, along with a discussion of the value, potential limitations, and trade-offs associated with each strategy.