Infoscience (Ecole Polytechnique Fédérale de Lausanne) · 2018 · 16 citations · 4 references
Open access
ComputingComputer ScienceMachine LearningOptimizationConstrained OptimizationNeural NetworksBackpropagationOptimization Algorithms
We propose BlockProp, a neural network training algorithm. Unlike backpropagation, it does not rely on direct top-to-bottom propagation of an error signal. Rather, by interpreting backpropagation as a constrained optimization problem we split the neural network model into sets of layers (blocks) that must satisfy a consistency constraint, i.e. the output of one set of layers must be equal to the input of the next. These decoupled blocks are then updated with the gradient of the optimization constraint violation. The main advantage of this formulation is that we decouple the propagation of the error signal on different subparts (blocks) of the network making it particularly relevant for multi-devices applications.
4
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Random synaptic feedback weights support error backpropagation for deep learning
Timothy Lillicrap, Daniel Cownden, Douglas Tweed et al. · Nature Communications · 2016 · 851 citations · Full text
Training Neural Networks Without Gradients: A Scalable ADMM Approach
Gavin Taylor, Ryan Burmeister, Zheng Xu et al. · arXiv (Cornell University) · 2016 · 146 citations · Full text
Distributed optimization of deeply nested systems
Weiran Wang · arXiv (Cornell University) · 2012 · 104 citations · Full text