A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

TLDR

Visual analytics systems combine interactive visualizations and machine learning to help experts solve complex tasks, yet their multidisciplinary nature makes evaluation difficult. This survey provides a comprehensive overview of evaluations in human‑centered machine learning, focusing on human factors such as trust, interpretability, and explainability, and aims to distill design dimensions, identify gaps, and suggest future research. The authors systematically review evaluation studies from leading information‑visualization and human‑computer‑interaction conferences and journals, analyzing their experimental setups and findings. They find that current evaluations lack structure and comparability, and they propose design dimensions, highlight gaps, and outline future research opportunities.

Abstract

Abstract Visual analytics systems integrate interactive visualizations and machine learning to enable expert users to solve complex analysis tasks. Applications combine techniques from various fields of research and are consequently not trivial to evaluate. The result is a lack of structure and comparability between evaluations. In this survey, we provide a comprehensive overview of evaluations in the field of human‐centered machine learning. We particularly focus on human‐related factors that influence trust, interpretability, and explainability. We analyze the evaluations presented in papers from top conferences and journals in information visualization and human‐computer interaction to provide a systematic review of their setup and findings. From this survey, we distill design dimensions for structured evaluations, identify evaluation gaps, and derive future research opportunities.