A Controlled Study to Determine Measurement Conditions Necessary for a Reliable and Valid Operative Performance Assessment

Abstract

OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.