Reliability and omega hierarchical in multidimensional data: A comparison of various estimators.

Abstract

The current guidelines for estimating reliability recommend using two omega combinations in multidimensional data. One omega is for factor analysis (FA) reliability estimators, and the other omega is for omega hierarchical estimators (i.e., ω<i>_h</i>). This study challenges these guidelines. Specifically, the following three questions are asked: (a) Do FA reliability estimators outperform non-FA reliability estimators? (b) Is it always desirable to estimate ω<i>_h</i>? (c) What are the best reliability and ω<i>_h</i> estimators? This study addresses these issues through a Monte Carlo simulation of reliability and ω<i>_h</i> estimators. The conclusions are given as follows. First, the performance differences among most reliability estimators are small, and the performance of FA estimators is comparable to that of non-FA estimators. However, the current, most-recommended estimators, that is, estimators based on the bifactor model and exploratory factor analysis, tend to overestimate reliability. Second, the accuracy of ω<i>_h</i> estimators is much lower than that of reliability estimators, so we should perform ω<i>_h</i> estimation selectively only on data that meet several requirements. Third, exploratory bifactor analysis is more accurate than confirmatory bifactor analysis only in the presence of cross-loading; otherwise, exploratory bifactor analysis is less accurate than confirmatory bifactor analysis. Fourth, techniques known to improve the Schmid-Leiman (SL) transformation are not superior to SL transformation but have different advantages. This study provides an R Shiny app that allows users to obtain multidimensional reliability and ω<i>_h</i> estimates with a few mouse clicks. (PsycInfo Database Record (c) 2025 APA, all rights reserved).