A checklist for critical appraisal of indirect comparisons

Abstract

The aim of this study was to develop a user-friendly checklist for critical appraisal of indirect comparisons of drugs, considering clinical, methodological/statistical and quality aspects, mainly to be applied in drug evaluation in the decision-making context. After conducting a review of the literature, we used group consensus to establish the key points of the checklist, focusing mainly on indirect comparisons, but including topics related to network meta-analysis or multiple treatment comparisons. The coordinating group elaborated the first draft, which was reviewed by external experts, re-evaluated by the coordinating group and finally assessed by 23 drug evaluation experts trained in indirect comparisons, who applied the checklist to one study. The Kappa index of agreement was calculated and the final checklist was developed by group consensus including the external experts. The checklist has two parts. The first consists of three eliminatory key questions while the second includes 17 items: 5 regarding quality, 5 regarding clinical issues and 7 dealing with methodology/statistics. The median kappa values of the 23 evaluations were 0.83 (range 0.67-0.93), 0.61 (0.54-0.91) and 0.36 (0.22-1) with regard to quality, clinical aspects and methodology/statistics, respectively. A structured checklist was developed to facilitate critical appraisal of key issues in indirect comparisons, including comments for assessing the consequences of its application to drug evaluation in the decision-making context. Agreement between reviewers in clinical and quality items was good, but weaker in methodology/statistics ones.