Detecting Unit of Analysis Problems in Nested Designs: Statistical Power and Type I Error Rates of the F Test for Groups-within-Treatments Effects

Abstract

The nested analysis of variance (ANOVA) model is often recommended for analysis of educational data in which students receive treatments within classrooms. The Type I error rate and statistical power of the F test for groups-within-treatments effects associated with such nested ANOVA designs were evaluated in a Monte Carlo study. Data were generated for ANOVA models comprising two and three levels of the treatment variable; two, three, and five groups nested within each treatment; and 3, 10, and 30 observations within each group. The intraclass correlation among scores within groups was controlled at levels ranging from zero (independent observations) to .50, and the F test for groups-within-treatments effects was conducted at nominal alpha levels ranging from .05 to .30. The results indicated that the test for groups within treatments did not indicate sufficient power, even at the most liberal alpha level examined, to detect intraclass correlation in the sample data. The procedure was not effective in supporting decisions to use individual observations rather than group means as the unit of analysis.