EIE

TLDR

State‑of‑the‑art deep neural networks contain hundreds of millions of connections, making them computationally and memory intensive; fetching weights from DRAM dominates power consumption, and prior compression techniques enable fitting large models into on‑chip SRAM. The authors introduce EIE, an energy‑efficient inference engine that operates directly on a compressed network model and accelerates sparse matrix‑vector multiplication with weight sharing. Compression is achieved by pruning redundant connections and sharing weights, and EIE exploits this sparsity and weight sharing to perform inference efficiently on embedded hardware. EIE delivers 120× energy savings from SRAM, 10× from sparsity, 8× from weight sharing, and 3× from skipping zero activations, achieving 189× and 13× speedups over CPU and GPU, 102 GOPS (3 TOPS uncompressed) at 600 mW, and outperforming DaDianNao in throughput, energy, and area efficiency.

Abstract

State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120× energy saving; Exploiting sparsity saves 10×; Weight sharing gives 8×; Skipping zero activations from ReLU saves another 3×. Evaluated on nine DNN benchmarks, EIE is 189× and 13× faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102 GOPS working directly on a compressed network, corresponding to 3 TOPS on an uncompressed network, and processes FC layers of AlexNet at 1.88×10 4 frames/sec with a power dissipation of only 600mW. It is 24,000× and 3,400× more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9×, 19× and 3× better throughput, energy efficiency and area efficiency.