Improved rates and asymptotic normality for nonparametric neural network estimators

Abstract

We obtain an improved approximation rate (in Sobolev norm) of r/sup -1/2-/spl alpha//(d+1)/ for a large class of single hidden layer feedforward artificial neural networks (ANN) with r hidden units and possibly nonsigmoid activation functions when the target function satisfies certain smoothness conditions. Here, d is the dimension of the domain of the target function, and /spl alpha//spl isin/(0, 1) is related to the smoothness of the activation function. When applying this class of ANNs to nonparametrically estimate (train) a general target function using the method of sieves, we obtain new root-mean-square convergence rates of Op([n/log(n)]/sup -/(1+2/spl alpha//(d+1))/[4(1+/spl alpha//(d+1))])=op(n/sup -1/4/) by letting the number of hidden units /spl tau//sub n/, increase appropriately with the sample size (number of training examples) n. These rates are valid for i.i.d. data as well as for uniform mixing and absolutely regular (/spl beta/-mixing) stationary time series data. In addition, the rates are fast enough to deliver root-n asymptotic normality for plug-in estimates of smooth functionals using general ANN sieve estimators. As interesting applications to nonlinear time series, we establish rates for ANN sieve estimators of four different multivariate target functions: a conditional mean, a conditional quantile, a joint density, and a conditional density. We also obtain root-n asymptotic normality results for semiparametric model coefficient and average derivative estimators.