Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice

Abstract

A fundamental question within the field of program evaluation is “Do social programs work?” Although experiments allow us to answer this question with certainty, they have some limitations. Experiments generate mean program impacts and even mean impacts by subgroup, but they often leave unexplored the impacts on subgroups determined by treatment use. This work proposes a methodology for analyzing the impacts of social programs on previously unexamined subgroups. Rather than using a single trait to define subgroups—which is currently the dominant method of subgroup analysis—the proposed approach estimates the impact of programs on subgroups identified by a post-treatment choice while still maintaining the integrity of the experimental research design. Analysis of data from the experimental evaluation of New York State’s Child Assistance Program (CAP) provides an application of the proposed technique.