Reliability of the PEDro Scale for Rating Quality of Randomized Controlled Trials

TLDR

Assessment of RCT quality is common in systematic reviews, yet the reliability of most quality scales remains unestablished. This report investigates the reliability of the PEDro scale for rating RCT quality in physical therapy. Two studies were conducted: one with 11 raters rating 25 RCTs, and another with 2 raters rating 120 RCTs, with disagreements resolved by a third rater and repeated to generate individual and consensus ratings. Reliability ranged from fair to substantial for individual items (kappa .36–.80) and from fair to good for the total score (ICC .56 for individuals, .68 for consensus).

Abstract

Assessment of the quality of randomized controlled trials (RCTs) is common practice in systematic reviews. However, the reliability of data obtained with most quality assessment scales has not been established. This report describes 2 studies designed to investigate the reliability of data obtained with the Physiotherapy Evidence Database (PEDro) scale developed to rate the quality of RCTs evaluating physical therapist interventions.In the first study, 11 raters independently rated 25 RCTs randomly selected from the PEDro database. In the second study, 2 raters rated 120 RCTs randomly selected from the PEDro database, and disagreements were resolved by a third rater; this generated a set of individual rater and consensus ratings. The process was repeated by independent raters to create a second set of individual and consensus ratings. Reliability of ratings of PEDro scale items was calculated using multirater kappas, and reliability of the total (summed) score was calculated using intraclass correlation coefficients (ICC [1,1]).The kappa value for each of the 11 items ranged from.36 to.80 for individual assessors and from.50 to.79 for consensus ratings generated by groups of 2 or 3 raters. The ICC for the total score was.56 (95% confidence interval=.47-.65) for ratings by individuals, and the ICC for consensus ratings was.68 (95% confidence interval=.57-.76).The reliability of ratings of PEDro scale items varied from "fair" to "substantial," and the reliability of the total PEDro score was "fair" to "good."