arXiv (Cornell University) · 2020 · 24 citations · 20 references
ComputingComputer EngineeringComputer ScienceArtificial IntelligenceMachine LearningNatural Language ProcessingDeep LearningLoss FunctionNeural NetworksPattern RecognitionRepresentation LearningFine-tuningFloating-point ArithmeticMemory Footprint ReductionModel Accuracy AssessmentNeural Network QuantizationPrecision
Training with larger number of parameters while keeping fast iterations is an\nincreasingly adopted strategy and trend for developing better performing Deep\nNeural Network (DNN) models. This necessitates increased memory footprint and\ncomputational requirements for training. Here we introduce a novel methodology\nfor training deep neural networks using 8-bit floating point (FP8) numbers.\nReduced bit precision allows for a larger effective memory and increased\ncomputational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We\nshow that, unlike previous 8-bit precision training methods, the proposed\nmethod works out-of-the-box for representative models: ResNet-50, Transformer\nand NCF. The method can maintain model accuracy without requiring fine-tuning\nloss scaling parameters or keeping certain layers in single precision. We\nintroduce two learnable statistics of the DNN tensors - shifted and squeezed\nfactors that are used to optimally adjust the range of the tensors in 8-bits,\nthus minimizing the loss in information due to quantization.\n
20
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba · UvA-DARE (University of Amsterdam) · 2014 · 84.8K citations · Full text
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher et al. · 2009 IEEE Conference on Computer Vision and Pattern Recognition · 2009 · 61.8K citations
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Kishore Papineni, Salim Roukos, Todd J. Ward et al. · 2001 · 21.5K citations · Full text
Neural Collaborative Filtering
Xiangnan He, Lizi Liao, Hanwang Zhang et al. · 2017 · 6.6K citations · Full text