Machine Learning and Knowledge Extraction · 2020 · 16 citations · 42 references
ComputingComputer ScienceArtificial IntelligenceComputer VisionMachine LearningDeep LearningNetwork RobustnessNeural NetworksOptimization For Machine LearningPattern RecognitionBackpropagationDiscretizationNonlinear TransformationsRegularizationResidual NetworksTransformations
In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power, and allows for control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique, and provide numerical results in support of our findings.
42
Densely Connected Convolutional Networks
Gao Huang, Zhuang Liu, Laurens van der Maaten et al. · 2017 · 44.7K citations
Spectral Normalization for Generative Adversarial Networks
Takeru Miyato, Toshiki Kataoka, Masanori Koyama et al. · arXiv (Cornell University) · 2018 · 2.3K citations · Full text
Neural networks for control systems—A survey
Kate Hunt, Daniel Sbárbaro, R. Żbikowski et al. · Automatica · 1992 · 1.9K citations