A comparison of statistical significance tests for information retrieval evaluation

Mark D. Smucker, James Allan, Ben Carterette

2007 · 717 citations · 24 references

DOIFull text

Open access

Concepts

TL;DR

IR researchers typically rely on the Student's paired t‑test, Wilcoxon signed rank test, and sign test, while bootstrap and Fisher's randomization tests have been suggested but are rarely employed. The authors evaluated five significance tests—t, Wilcoxon, sign, bootstrap, and Fisher's randomization—by applying them to ad‑hoc retrieval runs from TREC 3 and 5–8 and measuring the significance of mean average precision differences between each pair of runs. The study found that the randomization, bootstrap, and t tests perform similarly, whereas the Wilcoxon and sign tests are unreliable, prone to false positives, and should be abandoned for measuring mean differences.

Abstract

Information retrieval (IR) researchers commonly use three tests of statistical significance: the Student's paired t-test, the Wilcoxon signed rank test, and the sign test. Other researchers have previously proposed using both the bootstrap and Fisher's randomization (permutation) test as non-parametric significance tests for IR but these tests have seen little use. For each of these five tests, we took the ad-hoc retrieval runs submitted to TRECs 3 and 5-8, and for each pair of runs, we measured the statistical significance of the difference in their mean average precision. We discovered that there is little practical difference between the randomization, bootstrap, and t tests. Both the Wilcoxon and sign test have a poor ability to detect significance and have the potential to lead to false detections of significance. The Wilcoxon and sign tests are simplified variants of the randomization test and their use should be discontinued for measuring the significance of a difference between means.

References

24