Avoiding Type III Errors in Health Education Program Evaluations: A Case Study

TLDR

Monitoring program implementation improves data interpretability and helps avoid Type III errors in evaluations. This article evaluates a fifth‑grade health‑education curriculum by analyzing its implementation, measuring students’ cognitive learning outcomes, and examining the link between classroom implementation and knowledge gains. The study compared 101 students in the curriculum to 84 in a comparison group, using pre‑ and post‑tests of health knowledge, daily classroom implementation monitoring, teacher and principal questionnaires, and descriptive, parametric/nonparametric, and qualitative analyses. Results showed the curriculum enhanced health knowledge, implementation varied across classes, teachers and principals viewed it favorably, and no significant relationship emerged between implementation fidelity and learning gains, highlighting the need to measure implementation in program evaluations.

Abstract

Monitoring the implementation of a program being evaluated can improve the interpretability of data collected and help evaluators to avoid committing a Type III error: evaluating a program that has not been adequately implemented. This article describes an evaluation that analyzed the implementation of a school health education curriculum, assessed cognitive learning outcomes attributable to the curriculum, and examined the relationship between classroom implementation and changes in students' knowledge. Five fifth-grade classes (n = 101) participated in the curriculum, and five classes (n = 84) served as a comparison group. Data collection procedures involved a pretest and posttest of all students' health-related knowledge, daily monitoring of classroom implementation by the five teachers participating, and questionnaires completed by principals and teachers. Analysis methods included descriptive statistics, parametric and nonparametric tests of significance, and qualitative assessment procedures. Results indicated that the curriculum had a positive effect on learning in students; curriculum implementation varied considerably among the five classes participating; teaching/learning activities that were most and least likely to be implemented could be identified and described; both teachers and principals perceived the program favorably; some health instruction was occurring in the comparison classes, so it was not appropriate to consider them as pure controls; and no statistically significant relationship between curriculum implementation and cognitive outcomes was observed. This study provides evidence of the need for and value of measuring implementation of programs being evaluated. Implications for developing implementation measures and the role of formative evaluation in health education practice are considered.