Impact evaluation of job training programmes: Selection bias in multilevel models

Abstract

This paper focuses on the evaluation of a job training programme composed of several different courses. The aim is to evaluate the impact of the programme for the participants with respect to non-participants, paying attention to possible differences in the effectiveness between the courses. The analysis is based on discrete data with a hierarchical structure. Multilevel modelling is the natural choice in this setting, but the results may be severely affected by selection bias. We propose a two-step procedure, which suits both the hierarchical structure and the observational nature of data. The method selects the appropriate control group, using standard results of the propensity score methodology. A suitable multilevel model is formulated, and the dependence of the results on the amount of non-random sample selection is analysed within a likelihood-based framework. As a result, rankings for comparative performances are obtained, adjusted for the amount of plausible selection bias. The procedure is illustrated with reference to a data set about a job training programme organized in Italy in the late 1990s.