Evidence-Centered Design of Epistemic Games: Measurement Principles for Complex Learning Environments.

TLDR

Assessing 21st‑century skills requires innovative yet reliable methods, and epistemic games—games that immerse learners in professional practice—promote domain expertise through collaborative learning, distributed expertise, and complex problem‑solving. The paper outlines a comprehensive research program to investigate methodological challenges in rigorously studying epistemic games. The authors employ the evidence‑centered design framework and contemporary reliability‑validity theory to guide epistemic game development and illustrate, via the Urban Science game, the design decisions and evidence‑gathering implications for assessing learners’ expertise. The study demonstrates that careful design decisions in epistemic games, guided by evidence‑centered design, enable the collection of both qualitative and quantitative evidence of learners’ developing expertise.

Abstract

We are currently at an exciting juncture in developing effective means for assessing so-called 21st-century skills in an innovative yet reliable fashion. One of these avenues leads through the world of epistemic games (Shaffer, 2006a), which are games designed to give learners the rich experience of professional practica within a discipline. They serve to develop domain-specific expertise based on principles of collaborative learning, distributed expertise, and complex problem-solving. In this paper, we describe a comprehensive research program for investigating the methodological challenges that await rigorous inquiry within the epistemic games context. We specifically demonstrate how the evidence-centered design framework (Mislevy, Almond, & Steinberg, 2003) as well as current conceptualizations of reliability and validity theory can be used to structure the development of epistemic games as well as empirical research into their functioning. Using the epistemic game Urban Science (Bagley & Shaffer, 2009), we illustrate the numerous decisions that need to be made during game development and their implications for amassing qualitative and quantitative evidence about learners’ developing expertise within epistemic games.