Comparing Predictive Accuracy

TLDR

Recent economic studies use quasi‑experimental designs that exploit exogenous variation from state laws and other mechanisms to compare treatment and control groups. This paper outlines the benefits of such designs, proposes improvements, and offers guidance for assessing their inferential validity. The authors advocate addressing design complications by incorporating multiple treatment and comparison groups and multiple pre‑ and post‑intervention observations.

Abstract

Using research designs patterned after randomized experiments, many recent economic studies examine outcome measures for treatment groups and comparison groups that are not randomly assigned. By using variation in explanatory variables generated by changes in state laws, government draft mechanisms, or other means, these studies obtain variation that is readily examined and is plausibly exogenous. This paper describes the advantages of these studies and suggests how they can be improved. It also provides aids in judging the validity of inferences they draw. Design complications such as multiple treatment and comparison groups and multiple pre- or post-intervention observations are advocated.