Selecting training exemplars for neural network learning

Abstract

We examine methods for selecting training exemplars for the purpose of training neural networks. We use statistical methods to obtain techniques based upon an objective criterion, obtaining techniques for both clean and noisy data. The techniques efficiently select concise sets of exemplars. The methods we develop are not restricted to use on neural networks, being applicable to nonlinear regression in general. These techniques are nonparametric in the sense that they automatically adjust the complexity of the network model as appropriate for the learning task at hand. Empirical evidence with the method for clean data indicates that this technique can be extremely beneficial to making learning more efficient and reliable. The method also selects concise training sets, substantiating recent theoretical advancements in understanding the relationship between network complexity and the number of training examples. The algorithm we developed for this method has been rigorously tested, and proves to be highly autonomous and robust. We also prove that a well known estimate of network learning performance is statistically accurate and precise. This result has wide scope, being applicable to nonlinear regression in general. Finally, preliminary results with a method for selecting noisy exemplars indicate that this method has the potential for making classical optimization algorithms more robust and reliable in the presence of noise.