2007 · 717 citations · 24 references
ComputingComputer ScienceInformation RetrievalPerformance MeasurementTest ConstructionEvaluation Of Retrieval SystemsHypothesis TestingTest ValidityInformation Retrieval EvaluationMean Average PrecisionPaired T-testRetrieval EvaluationSign TestStatistical Significance TestingTrec EvaluationWilcoxon Signed-rank Test
IR researchers typically rely on the Student's paired t‑test, Wilcoxon signed rank test, and sign test, while bootstrap and Fisher's randomization tests have been suggested but are rarely employed. The authors evaluated five significance tests—t, Wilcoxon, sign, bootstrap, and Fisher's randomization—by applying them to ad‑hoc retrieval runs from TREC 3 and 5–8 and measuring the significance of mean average precision differences between each pair of runs. The study found that the randomization, bootstrap, and t tests perform similarly, whereas the Wilcoxon and sign tests are unreliable, prone to false positives, and should be abandoned for measuring mean differences.
Information retrieval (IR) researchers commonly use three tests of statistical significance: the Student's paired t-test, the Wilcoxon signed rank test, and the sign test. Other researchers have previously proposed using both the bootstrap and Fisher's randomization (permutation) test as non-parametric significance tests for IR but these tests have seen little use. For each of these five tests, we took the ad-hoc retrieval runs submitted to TRECs 3 and 5-8, and for each pair of runs, we measured the statistical significance of the difference in their mean average precision. We discovered that there is little practical difference between the randomization, bootstrap, and t tests. Both the Wilcoxon and sign test have a poor ability to detect significance and have the potential to lead to false detections of significance. The Wilcoxon and sign tests are simplified variants of the randomization test and their use should be discontinued for measuring the significance of a difference between means.
24
R: A Language and Environment for Statistical Computing
R Core Team · 2000 · 353K citations · Full text
Natural Sciences, Computer Science, Statistical Computing +10
Individual Comparisons by Ranking Methods
Frank Wilcoxon · Biometrics Bulletin · 1945 · 14.4K citations
Business, Computer Science, Multi-criteria Decision Making +8
Sidney Addelman, G. E. P. Box, W. G. Hunter et al. · Technometrics · 1979 · 6.5K citations
Makoto Matsumoto, Takuji Nishimura · ACM Transactions on Modeling and Computer Simulation · 1998 · 5.7K citations
An Introduction to the Bootstrap
Scott D. Grimshaw, Bradley Efron, Robert Tibshirani · Technometrics · 1995 · 2.2K citations
Natural Sciences, Astronomy, Physics +11