Computing Response Metrics for Online Panels

TLDR

Online panel research is growing, yet metric calculation suffers from inconsistent terminology and methods, and only completion rates are computable for opt‑in panels. The study aims to establish standardized formulas and terminology for computing response, refusal, and other rates in online panel research by outlining the stages of panel construction. It distinguishes probability‑based from volunteer opt‑in panels, defines stage‑specific metrics, and combines them into cumulative response‑rate formulas applicable to probability‑based panels. The authors conclude by interpreting the proposed metrics and recommending which ones should be reported for each panel type.

Abstract

Abstract As more researchers use online panels for studies, the need for standardized rates to evaluate these studies becomes paramount. There are currently many different ways and conflicting terminology used to compute various metrics for online panels. This paper discusses the sparse literature on how to compute response, refusal, and other rates and proposes a set of formulas and a standardized terminology that can be used to calculate and interpret these metrics for online panel studies. A description of and distinction between probability-based and volunteer opt-in panels is made since not all metrics apply to both types. A review of the existing discussion and recommendations, mostly from international organizations, is presented for background and context. In order to propose response and other metrics, the different stages involved in building an online panel are delineated. Metrics associated with these stages contribute to cumulative response rate formulas that can be used to evaluate studies using online probability-based panels. (Only completion rates can be calculated with opt-in panels.) We conclude with a discussion of the meaning of the different metrics proposed and what we think should be reported for which type of panel.