Advances in Health Sciences Education · 2011 · 37 citations · 10 references
EducationEducational MeasurementGeneralizability TheoryHealth AssessmentPsychometricsHuman ReliabilityMeasurement ErrorReliabilityResident AssessmentTest ReliabilityUniverse Score EstimationAssessmentsAssessor ReliabilityClinical Measurement ReliabilityGeneralizability Study DesignInstrument ReliabilityInter-rater ReliabilityReliability AssessmentReliability Testing
Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent. This is difficult to achieve. Furthermore, the variance caused by the case/patient or by assessor is completely confounded. We have no idea how much each of these factors contribute to the noise in the measurement. The aim of this study was to use a controlled setup that overcomes these difficulties and to estimate the reproducibility of the mini-CEX. Three encounters were videotaped from 21 residents. The patients were the same for all residents. Each encounter was assessed by 3 assessors who assessed all encounters for all residents. This delivered a fully crossed (all random) two-facet generalizability design. A quarter of the total variance was associated with universe score variance (28%). The largest source of variance was the general error term (34%) followed by the main effect of assessors (18%). Generalizability coefficients indicated that an approximate sample of 9 encounters was needed assuming a single different assessor per encounter and assuming different cases per encounter (the usual situation in real practice), 4 encounters when 2 raters were used and 3 encounters when 3 raters are used. Unexplained general error and the leniency/stringency of assessors are the major causes for unreliability in mini-CEX. To optimize reliability rater training might have an effect.
10
The Mini-CEX: A Method for Assessing Clinical Skills
John J. Norcini, Linda L. Blank, F. Daniel Duffy et al. · Annals of Internal Medicine · 2003 · 718 citations
Generalisability: a key to unlock professional assessment
Jim Crossley, Helena Davies, Gerry Humphris et al. · Medical Education · 2002 · 170 citations