Journal of Continuing Education in the Health Professions · 2015 · 35 citations · 16 references
ManagementEducational AssessmentPatient SafetyPerformance MeasurementTeamworkTeam EffectivenessTeam LearningTeam PerformanceInterrater ReliabilityMeasurement Of Team PerformancePatient Safety In TeamsRater AgreementSimulation-based Team TrainingTeam Performance AssessmentTeam Performance MetricsTraining Program Design
INTRODUCTION: Simulation-based methodologies are increasingly used to assess teamwork and communication skills and provide team training. Formative feedback regarding team performance is an essential component. While effective use of simulation for assessment or training requires accurate rating of team performance, examples of rater-training programs in health care are scarce. We describe our rater training program and report interrater reliability during phases of training and independent rating. METHODS: We selected an assessment tool shown to yield valid and reliable results and developed a rater training protocol with an accompanying rater training handbook. The rater training program was modeled after previously described high-stakes assessments in the setting of 3 facilitated training sessions. Adjacent agreement was used to measure interrater reliability between raters. RESULTS: Nine raters with a background in health care and/or patient safety evaluated team performance of 42 in-situ simulations using post-hoc video review. Adjacent agreement increased from the second training session (83.6%) to the third training session (85.6%) when evaluating the same video segments. Adjacent agreement for the rating of overall team performance was 78.3%, which was added for the third training session. Adjacent agreement was 97% 4 weeks posttraining and 90.6% at the end of independent rating of all simulation videos. DISCUSSION: Rater training is an important element in team performance assessment, and providing examples of rater training programs is essential. Articulating key rating anchors promotes adequate interrater reliability. In addition, using adjacent agreement as a measure allows differentiation between high- and low-performing teams on video review.
16
John Kim, David Neilipovitz, Pierre Cardinal et al. · Critical Care Medicine · 2006 · 324 citations
James F. Malec, Laurence C. Torsher, William F. Dunn et al. · Simulation in Healthcare The Journal of the Society for Simulation in Healthcare · 2007 · 309 citations
Management, Human Resource Management, Crisis Management +12