Effects of Noise on Convergence and Generalization in Recurrent Networks

Abstract

We introduce and study methods of inserting synaptic noise into dynamically-driven recurrent neural networks and show that applying a controlled amount of noise during training may improve convergence and generalization. In addition, we analyze the effects of each noise parameter (additive vs. multiplicative, cumulative vs. non-cumulative, per time step vs. per string) and predict that best overall performance can be achieved by injecting additive noise at each time step. Extensive simulations on learning the dual parity grammar from temporal strings substantiate these predictions. 1 INTRODUCTION There has been much research in applying noise to neural networks to improve network performance. It has been shown that using noisy hidden nodes during training can result in error-correcting codes which increase the tolerance of feedforward nets to unreliable nodes (Judd and Munro, 1992). Also, randomly disabling hidden nodes during the training phase increases the tolerance of MLP's to nod...