Systematic Reviews · 2017 · 65 citations · 17 references
MedicineOrthopaedic SurgeryOrthopedic SurgeryEvidence-based PracticeOutcomes ResearchRisk AssessmentClinical TrialsOutcome AssessmentPain AssessmentRisk Of Bias AssessmentTest ReliabilityBias In Clinical AssessmentBias In Randomized TrialsInterrater ReliabilityKappa StatisticPilot Study DesignPrimary Outcome AnalysisRandomized Clinical TrialsRisk Of Bias Tools
BACKGROUND: The Cochrane risk of bias tool is commonly criticized for having a low reliability. We aimed to investigate whether training of raters, with objective and standardized instructions on how to assess risk of bias, can improve the reliability of the Cochrane risk of bias tool. METHODS: In this pilot study, four raters inexperienced in risk of bias assessment were randomly allocated to minimal or intensive standardized training for risk of bias assessment of randomized trials of physical therapy treatments for patients with knee osteoarthritis pain. Two raters were experienced risk of bias assessors who served as reference. The primary outcome of our study was between-group reliability, defined as the agreement of the risk of bias assessments of inexperienced raters with the reference assessments of experienced raters. Consensus-based assessments were used for this purpose. The secondary outcome was within-group reliability, defined as the agreement of assessments within pairs of inexperienced raters. We calculated the chance-corrected weighted Kappa to quantify agreement within and between groups of raters for each of the domains of the risk of bias tool. RESULTS: A total of 56 trials were included in our analysis. The Kappa for the agreement of inexperienced raters with reference across items of the risk of bias tool ranged from 0.10 to 0.81 for the minimal training group and from 0.41 to 0.90 for the standardized training group. The Kappa values for the agreement within pairs of inexperienced raters across the items of the risk of bias tool ranged from 0 to 0.38 for the minimal training group and from 0.93 to 1 for the standardized training group. Between-group differences in Kappa for the agreement of inexperienced raters with reference always favored the standardized training group and was most pronounced for incomplete outcome data (difference in Kappa 0.52, p < 0.001) and allocation concealment (difference in Kappa 0.30, p = 0.004). CONCLUSIONS: Intensive, standardized training on risk of bias assessment may significantly improve the reliability of the Cochrane risk of bias tool.
17
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Julian P. T. Higgins, Doug Altman, Peter C Gøtzsche et al. · BMJ · 2011 · 34.2K citations · Full text
Chapter 8: Assessing risk of bias in included studies
Julian P. T. Higgins, Douglas G. Altman · 2008 · 3.8K citations
Better Bootstrap Confidence Intervals
Bradley Efron · Journal of the American Statistical Association · 1987 · 3.3K citations
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Peter Jüni · JAMA · 1999 · 2K citations