Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information

TLDR

Normal approximations to the distribution of the maximum likelihood estimator in one‑parameter families traditionally use 1/expected Fisher information, but many authors, including Fisher, advocate using 1/observed Fisher information, a view grounded in Fisher’s foundational work on likelihood estimation. The study aims to provide a frequentist justification for preferring the observed Fisher information 1/I(x) over the traditional 1/expected Fisher information. The authors support this justification with a large number of illustrative examples that supplement a concise theoretical framework. The observed Fisher information 1/I(x) is shown to approximate the conditional variance given an ancillary statistic, and the evidence indicates a preference for the likelihood‑ratio method of constructing confidence limits.

Abstract

This paper concerns normal approximations to the distribution of the maximum likelihood estimator in one-parameter families. The traditional variance approximation is 1/§, where θ is the maximum likelihood estimator and § is the expected total Fisher information. Many writers, including R. A. Fisher, have argued in favour of the variance estimate 1/I(x), where I(x) is the observed information, i.e. minus the second derivative of the log likelihood function at θ given data x. We give a frequentist justification for preferring 1/I(x) to 1/§. The former is shown to approximate the conditional variance of 8 given an appropriate ancillary statistic which to a first approximation is I(x). The theory may be seen to flow naturally from Fisher's pioneering papers on likelihood estimation. A large number of examples are used to supplement a small amount of theory. Our evidence indicates preference for the likelihood ratio method of obtaining confidence limits.