Evaluation of a Medical Faculty Development Program

TLDR

Pre‑ and post‑intervention self‑assessments are widely used to evaluate educational interventions, but response shift can undermine their validity. We assessed the influence of response shift in a national faculty development program for clinical teachers. We compared changes in traditional pre/post self‑assessment ratings with changes in retrospective pre/post self‑assessment ratings, using faculty self‑assessments and evaluations by housestaff and students. Retrospective pre/post comparisons revealed greater and more consistent improvements in teaching performance and attitudes, aligned better with housestaff and student evaluations, suggesting they may be more sensitive and valid than traditional pre/post ratings.

Abstract

Pre- and postintervention self-assessments are commonly used to evaluate educational interventions. However, when training influences participants'criteria for their self-ratings (response shift), the validity of the traditional prefpost comparisons is suspect. We assessed the influence of this phenomenon in a national faculty development program for clinical teachers. We compared changes in traditional pre/post self-assessment ratings with changes in retrospective pre/post self-assessment ratings. Data included prelpost intervention faculty self-assessments and evaluations of faculty by housestaff and students. On dimensions addressed in the training program, retrospective pre/post comparisons revealed more significant changes in teaching performance and attitudes than traditional pre/post comparisons. Housestaff and student evaluations were more consistent with thefaculty 's retrospective pre/post than with traditional prelpost comparisons. We conclude that, compared to traditional prelpost self-assessment; retrospective pre/post ratings may provide a more sensitive and more valid measure of the effects offaculty development.