IEEE Journal of Selected Topics in Quantum Electronics · 2022 · 55 citations · 66 references
ComputingComputer EngineeringComputer ScienceComputer ArchitectureComputer VisionMachine LearningAnalog Integrated CircuitsDeep LearningHardware AccelerationNeural NetworksNoise In Analog CircuitsDeep Learning AccelerationDeep Learning AcceleratorsDynamic QuantizationEnergy-efficient ComputationNeural Network QuantizationOptical Neural NetworksThermal NoiseThermal Noise Reduction
Analog electronic and optical computing exhibit tremendous advantages over digital computing for accelerating deep learning when operations are executed at low precision. Although digital architectures support programmable precision to increase efficiency, analog computing architectures today only support a single, static precision. In this work, we characterize the relationship between the effective number of bits (ENOB) of precision of analog processors, which is limited by noise, and digital bit precision for quantized neural networks. We propose extending analog computing architectures to support dynamic levels of precision by repeating operations and averaging the result, decreasing the impact of noise. To utilize dynamic precision, we propose a method for learning the precision of each layer of a pre-trained model without retraining network weights. We evaluate this method on analog architectures subject to shot noise, thermal noise, and weight noise and find that employing dynamic precision reduces energy consumption by up to 89% for computer vision models such as Resnet50 and by 24% for natural language processing models such as BERT. In one example, we apply dynamic precision to a shot-noise limited homodyne optical neural network and simulate inference at an optical energy consumption of 2.7 aJ/MAC for Resnet50 and 1.6 aJ/MAC for BERT with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${< }2\%$</tex-math></inline-formula> accuracy degradation, implying that the optical energy consumption is unlikely to be the dominant cost.
66
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al. · 2016 · 222.1K citations · Full text
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Going deeper with convolutions
Christian Szegedy, Wei Liu, Yangqing Jia et al. · 2015 · 46.8K citations