The Relative Trustworthiness of Inferential Tests of the Indirect Effect in Statistical Mediation Analysis

TLDR

The analysis of 2 years of Psychological Science articles revealed inconsistencies in how researchers infer indirect effects in mediation analysis. The study aimed to determine whether different indirect‑effect tests disagree, how the choice of method influences conclusions, and to identify a trustworthy test. The authors compared commonly used indirect‑effect tests to assess agreement and recommend the most reliable approach. They found that tests agree most of the time, but disagreements are more frequent when an effect exists; the bias‑corrected bootstrap confidence interval is most trustworthy when power is paramount, while the Monte Carlo or distribution‑of‑product intervals best control Type I error, and the percentile bootstrap offers a reasonable compromise.

Abstract

A content analysis of 2 years of Psychological Science articles reveals inconsistencies in how researchers make inferences about indirect effects when conducting a statistical mediation analysis. In this study, we examined the frequency with which popularly used tests disagree, whether the method an investigator uses makes a difference in the conclusion he or she will reach, and whether there is a most trustworthy test that can be recommended to balance practical and performance considerations. We found that tests agree much more frequently than they disagree, but disagreements are more common when an indirect effect exists than when it does not. We recommend the bias-corrected bootstrap confidence interval as the most trustworthy test if power is of utmost concern, although it can be slightly liberal in some circumstances. Investigators concerned about Type I errors should choose the Monte Carlo confidence interval or the distribution-of-the-product approach, which rarely disagree. The percentile bootstrap confidence interval is a good compromise test.