Psychological Methods · 2001 · 251 citations · 65 references
Public HealthBiostatisticsPsychologyEvidence-based PracticeImplementation ScienceMeta-analysisEfficacy EvaluationEvaluation DesignEvidence-based InterventionsExperimental EvaluationSystematic ReviewTreatment EffectivenessMeasurement Of Treatment EfficacyMeta-analysis Of Treatment EffectsPooled Treatment Effect
The authors synthesized 319 meta‑analyses and used an index to quantify how much of the effect‑size variance is attributable to study‑method features. They found that methodological factors explained almost as much outcome variability as intervention characteristics, with study design and dependent‑variable operationalization contributing most, and sampling error comparable, highlighting the challenge of detecting treatment effects and the value of meta‑analysis.
A synthesis of 319 meta-analyses of psychological, behavioral, and educational treatment research was conducted to assess the influence of study method on observed effect sizes relative to that of substantive features of the interventions. An index was used to estimate the proportion of effect size variance associated with various study features. Study methods accounted for nearly as much variability in study outcomes as characteristics of the interventions. Type of research design and operationalization of the dependent variable were the method features associated with the largest proportion of variance. The variance as a result of sampling error was about as large as that associated with the features of the interventions studied. These results underscore the difficulty of detecting treatment outcomes, the importance of cautiously interpreting findings from a single study, and the importance of meta-analysis in summarizing results across studies.
65
Statistical Power Analysis for the Behavioral Sciences
Keith E. Muller · Technometrics · 1989 · 16.5K citations