Reliability estimation in a multilevel confirmatory factor analysis framework.

TLDR

Scales with varying reliability are commonly used in multistage sampling, where variance occurs at individual and group levels. The study highlights the need to assess level‑specific reliability due to a lack of existing methodological guidance. The authors use simulation and an applied example to compare multilevel reliability estimation methods via multilevel confirmatory factor analysis, offering Mplus and R code for credible and Monte Carlo confidence intervals. They find that single‑level estimates misrepresent reliability unless it is identical across levels, that two‑level alpha and omega perform well, that maximal reliability H is more biased, and that small cluster sizes inflate between‑level reliability; additionally, Monte Carlo and Bayesian intervals accurately reflect the sampling distribution.

Abstract

Scales with varying degrees of measurement reliability are often used in the context of multistage sampling, where variance exists at multiple levels of analysis (e.g., individual and group). Because methodological guidance on assessing and reporting reliability at multiple levels of analysis is currently lacking, we discuss the importance of examining level-specific reliability. We present a simulation study and an applied example showing different methods for estimating multilevel reliability using multilevel confirmatory factor analysis and provide supporting Mplus program code. We conclude that (a) single-level estimates will not reflect a scale's actual reliability unless reliability is identical at each level of analysis, (b) 2-level alpha and composite reliability (omega) perform relatively well in most settings, (c) estimates of maximal reliability (H) were more biased when estimated using multilevel data than either alpha or omega, and (d) small cluster size can lead to overestimates of reliability at the between level of analysis. We also show that Monte Carlo confidence intervals and Bayesian credible intervals closely reflect the sampling distribution of reliability estimates under most conditions. We discuss the estimation of credible intervals using Mplus and provide R code for computing Monte Carlo confidence intervals.