Use of retrospective pre/post assessments in faculty development

Abstract

Context and setting Our school’s faculty development (FD) programme annually produces 6 or more FD workshops covering a range of topics. We ordinarily assess these workshops, which each attract 20–60 participants from a wide variety of disciplines, using post-workshop questionnaires. In order to enhance the value of our evaluations, we decided to assess the usefulness of retrospective pre/post assessments. Why the idea was necessary A recent review of the FD literature concluded that assessment of outcome is often poorly carried out, and that measurement of true impact is difficult. It is common practice to elicit participants’ perceptions after an intervention. However, even when combining this with pre-intervention self-assessment, the validity is often limited by the phenomenon of ‘response shift bias’. That is, participants tend to over-rate themselves before the intervention and then rate themselves more accurately afterwards when they are more familiar with the content. Retrospective pre/post self-assessments, which are completed after an intervention, offer promise of greater validity. This method allows participants to rate their perceptions of their knowledge and skills after a workshop has been completed and to simultaneously rate their perceptions of their knowledge and skills beforehand. What was done Our FD series included workshops on ‘Teaching when there is no time to teach’ (TWNTT) and ‘Designing successful workshops’ (DSW), both of which were repeated 1 year later. At the end of each workshop, participants completed a retrospective pre/post self-assessment questionnaire. Participants in the TWNTT workshop rated 15 items on a scale of 1–5, indicating their perceptions of their knowledge and skills after completing the workshop and their perceptions of how their knowledge and skills had been before the workshop. We used the same procedure in the DSW workshop, but the latter included only 5 questionnaire items. We calculated mean differences and standard deviations between pre- and post-workshop scores on the questionnaires and, using a Wilcoxon statistic, evaluated the differences in each pre/post mean item score. Evaluation of results and impact Fifty faculty members participated in 1 TWNTT workshop; 49 different faculty attended the same workshop 1 year later. The DSW workshop was attended by 33 and 40 faculty staff at the same time-points. Self-ratings on all competency items were significantly higher on the retrospective post-workshop items than on the retrospective pre-workshop items for all 4 workshops. In both pairs of workshops, the mean changes in the pre/post ratings for each item were remarkably similar. This self-assessment process consists of only 1 brief questionnaire and is minimally intrusive. It may help to avoid the response shift bias inherent in traditional pre and post self-assessments caused by pre-test over- or underestimation. The remarkable stability and robust nature of our data, and the published evidence indicating that post-workshop interviews support this type of self-assessment, lead us to conclude that retrospective pre/post assessment is a valuable measure of the impact of faculty development.