Statistics, Handle with Care: Detecting Multiple Model Components with the Likelihood Ratio Test

Abstract

The likelihood ratio test (LRT) and the related $F$ test, do not (even\nasymptotically) adhere to their nominal $\\chi^2$ and $F$ distributions in many\nstatistical tests common in astrophysics, thereby casting many marginal line or\nsource detections and non-detections into doubt. Although there are many\nlegitimate uses of these statistics, in some important cases it can be\nimpossible to compute the correct false positive rate. For example, it has\nbecome common practice to use the LRT or the $F$ test for detecting a line in a\nspectral model or a source above background despite the lack of certain\nrequired regularity conditions. In these and other settings that involve\ntesting a hypothesis that is on the boundary of the parameter space, {\\it\ncontrary to common practice, the nominal $\\chi^2$ distribution for the LRT or\nthe $F$ distribution for the $F$ test should not be used}. In this paper, we\ncharacterize an important class of problems where the LRT and the $F$ test fail\nand illustrate this non-standard behavior. We briefly sketch several possible\nacceptable alternatives, focusing on Bayesian posterior predictive\nprobability-values. We present this method in some detail, as it is a simple,\nrobust, and intuitive approach. This alternative method is illustrated using\nthe gamma-ray burst of May 8, 1997 (GRB 970508) to investigate the presence of\nan Fe K emission line during the initial phase of the observation.\n