The Cost of Adding Parameters to a Model

Jeremy M. G. Taylor, Arminda Lúcia Siqueira, Robert E. Weiss

Journal of the Royal Statistical Society Series B (Statistical Methodology) · 1996 · 34 citations · 17 references

Concepts

Abstract

SUMMARY For a general regression model with n independent observations we consider the variance of the estimate of a quantity of interest under two scenarios. One scenario is where all the parameters are estimated from the data; the other scenario is where a subset of the parameters is assumed known at their true values and the remaining parameters are estimated. We focus on quantities of interest which are defined on the scale of the response variable. We show that, under certain conditions, the ratio of a weighted sum across the design points of the variance of the quantity of interest is given by q/p, where q and p are the number of free parameters in the two scenarios. Thus, in this average sense, the inflation in variance associated with adding parameters, also interpreted as the cost of adding parameters to a model, is directly proportional to the number of parameters. We study models involving power transformations, non-linear models and exponential family models.

References

17