2009 · 13 citations · 15 references
ComputingComputer EngineeringComputer ScienceAlgorithmsDynamical SystemsOptimizationNeural NetworksOptimization For Machine LearningSystem IdentificationAlgorithmic PruningInternal DynamicsJacobian Matrix AnalysisLayer PruningModel IdentificationPruningPruning HorizonPruning Intervals
This paper presents a pruning algorithm using adaptive pruning interval for system identification with general dynamic neural networks (GDNN). GDNNs are artificial neural networks with internal dynamics. All layers have feedback connections with time delays to the same and to all other layers. The parameters are trained with the Levenberg-Marquardt (LM) optimization algorithm. Therefore the Jacobian matrix is required. The Jacobian is calculated by real time recurrent learning (RTRL). As both LM and OBS need Hessian information, computing time can be saved, if OBS uses the scaled inverse Hessian already calculated for the LM algorithm. This paper discusses the effect of using the scaled Hessian instead of the real Hessian in the OBS pruning approach. In addition to that an adaptive pruning interval is introduced. Due to pruning the structure of the identification model is changed drastically. So the parameter optimization task between the pruning steps becomes more or less complex. To guarantee that the parameter optimization algorithm has enough time to cope with the structural changes in the GDNN-model, it is suggested to adapt the pruning interval during the identification process. The proposed algorithm is verified simulatively for two standard identification examples.
15
Neural networks for pattern recognition
Choice Reviews Online · 1994 · 18.7K citations
Backpropagation through time: what it does and how to do it
Paul J. Werbos · Proceedings of the IEEE · 1990 · 4.9K citations