Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomes

TLDR

Meta‑analysis of binary outcomes relies on weighted averages of trial‑level summary statistics, yet choosing the appropriate statistic is contentious because different measures imply distinct patterns of absolute benefit across baseline risks. The study aims to identify a summary statistic that remains consistent across trials with varying control event rates, arguing that selection should be based on empirical evidence and clinical judgment. Four common statistics—odds ratios, risk differences, risk ratios for beneficial and harmful outcomes—were evaluated, and the authors note that choosing a statistic solely on heterogeneity tests is unreliable when few trials are available. Empirical analysis of 551 reviews showed that risk ratios and odds ratios were more consistent than risk differences, with no difference between RR and OR, while a second set of 114 meta‑analyses indicated that for preventive interventions the constant risk‑ratio model best captures absolute benefit in high‑baseline‑risk trials; nevertheless, choice may still be uncertain without understanding control‑group variation. © 2002 John Wiley & Sons, Ltd.

Abstract

Abstract Meta‐analysis of binary data involves the computation of a weighted average of summary statistics calculated for each trial. The selection of the appropriate summary statistic is a subject of debate due to conflicts in the relative importance of mathematical properties and the ability to intuitively interpret results. This paper explores the process of identifying a summary statistic most likely to be consistent across trials when there is variation in control group event rates. Four summary statistics are considered: odds ratios (OR); risk differences (RD) and risk ratios of beneficial (RR(B)); and harmful outcomes (RR(H)). Each summary statistic corresponds to a different pattern of predicted absolute benefit of treatment with variation in baseline risk, the greatest difference in patterns of prediction being between RR(B) and RR(H). Selection of a summary statistic solely based on identification of the best‐fitting model by comparing tests of heterogeneity is problematic, principally due to low numbers of trials. It is proposed that choice of a summary statistic should be guided by both empirical evidence and clinically informed debate as to which model is likely to be closest to the expected pattern of treatment benefit across baseline risks. Empirical investigations comparing the four summary statistics on a sample of 551 systematic reviews provide evidence that the RR and OR models are on average more consistent than RD, there being no difference on average between RR and OR. From a second sample of 114 meta‐analyses evidence indicates that for interventions aimed at preventing an undesirable event, greatest absolute benefits are observed in trials with the highest baseline event rates, corresponding to the model of constant RR(H). The appropriate selection for a particular meta‐analysis may depend on understanding reasons for variation in control group event rates; in some situations uncertainty about the choice of summary statistic will remain. Copyright © 2002 John Wiley & Sons, Ltd.