Shifted and Squeezed 8-bit Floating Point format for Low-Precision\n Training of Deep Neural Networks

Léopold Cambier, Anahita Bhiwandiwalla, Ting Gong, Mehran Nekuii, Oğuz H. Elibol, Hanlin Tang

arXiv (Cornell University) · 2020 · 24 citations · 20 references

DOIFull text

Open access

Concepts

Abstract

Training with larger number of parameters while keeping fast iterations is an\nincreasingly adopted strategy and trend for developing better performing Deep\nNeural Network (DNN) models. This necessitates increased memory footprint and\ncomputational requirements for training. Here we introduce a novel methodology\nfor training deep neural networks using 8-bit floating point (FP8) numbers.\nReduced bit precision allows for a larger effective memory and increased\ncomputational speed. We name this method Shifted and Squeezed FP8 (S2FP8). We\nshow that, unlike previous 8-bit precision training methods, the proposed\nmethod works out-of-the-box for representative models: ResNet-50, Transformer\nand NCF. The method can maintain model accuracy without requiring fine-tuning\nloss scaling parameters or keeping certain layers in single precision. We\nintroduce two learnable statistics of the DNN tensors - shifted and squeezed\nfactors that are used to optimally adjust the range of the tensors in 8-bits,\nthus minimizing the loss in information due to quantization.\n

References

20