Validation of an objective structured assessment of technical skill for surgical residents

TLDR

Whether this was a failing of the OSATS or the faculty rankings requires further study. This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS) for general surgery residents. Twelve senior and junior residents were ranked by OSATS scores and faculty, and the correspondence was assessed with Spearman rank-order correlation coefficients. OSATS scores correlated strongly with faculty rankings for senior residents but poorly for junior residents, indicating validity for senior but not junior residents.

Abstract

PURPOSE: This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS), a new test of technical skill for general surgery residents. METHOD: Twelve residents (six in their senior, or fifth, year and six in their junior, or third, year) at the University of Toronto in 1994-95 were ranked within level of training according to their OSATS marks and by surgical faculty. Correspondence between OSATS and faculty rankings was assessed using Spearman rank-order correlation coefficients. RESULTS: The correlations between test scores and faculty rankings were generally high for the senior residents but low for the junior residents. CONCLUSION: Scores on the OSATS accurately reflect the independent opinions of faculty regarding the technical skills of senior residents, suggesting that it is a valid measure of technical skill for these individuals. The scores did not, however, reproduce faculty rankings of the junior residents. Whether this was a failing of the OSATS or the faculty rankings requires further study.