The Problems in Using Fixed-Effects Models of Meta-Analysis on Real-World Data

TLDR

Meta‑analysis can be conducted under a fixed‑effects assumption of a common effect size or a random‑effects assumption of varying effect sizes across studies. The article investigates the problems of applying fixed‑effects models to random‑effects data through two Monte Carlo simulations. The authors use two Monte Carlo simulations to illustrate these problems. The simulations reveal that fixed‑effects models inflate significance tests of mean effect sizes, highlighting serious implications for existing and future meta‑analytic reviews.

Abstract

There are 2 approaches to meta-analysis: One assumes that studies in a meta-analysis are sampled from populations with the same effect size (the fixed-effects case), the other assumes that studies are taken from populations that have varying effect sizes (the random-effects case). As such, 2 corresponding meta-analytic frameworks have been developed: fixed- and random-effects methods. Recent evidence suggests that the assumption of fixed population effect sizes is not tenable for virtually all real-world data (e.g., Hunter & Schmidt, 2000), and yet fixed-effects methods of meta-analysis are routinely applied to real-world data (see National Research Council, 1992). This article describes some of the problems in using fixed-effects models on random-effects data by presenting 2 Monte Carlo simulations. In keeping with statistical theory (e.g., Hunter & Schmidt, 2000) results show a radical inflation of the significance tests of the mean effect sizes (above and beyond theoretical predictions). These results are discussed in terms of the implications for previously published meta-analytic reviews and those yet to be done.