ACM SIGPLAN Notices · 2018 · 103 citations · 28 references
ComputingCloud ComputingComputer ScienceArtificial IntelligenceEmbedded SystemsMachine LearningDeep LearningEnergy-efficient Embedded SystemsNeural NetworksOptimization For Machine LearningPattern RecognitionLatency MinimizationOn-device InferenceOn-device Machine LearningPrivacyResource-constrained Devices
The recent ground-breaking advances in deep learning networks (DNNs) make them attractive for embedded systems. However, it can take a long time for DNNs to make an inference on resource-limited embedded devices. Offloading the computation into the cloud is often infeasible due to privacy concerns, high latency, or the lack of connectivity. As such, there is a critical need to find a way to effectively execute the DNN models locally on the devices. This paper presents an adaptive scheme to determine which DNN model to use for a given input, by considering the desired accuracy and inference time. Our approach employs machine learning to develop a predictive model to quickly select a pre-trained DNN to use for a given input and the optimization constraint. We achieve this by first training off-line a predictive model, and then use the learnt model to select a DNN model to use for new, unseen inputs. We apply our approach to the image classification task and evaluate it on a Jetson TX2 embedded deep learning platform using the ImageNet ILSVRC 2012 validation dataset. We consider a range of influential DNN models. Experimental results show that our approach achieves a 7.52% improvement in inference accuracy, and a 1.8x reduction in inference time over the most-capable single DNN model.
28
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su et al. · International Journal of Computer Vision · 2015 · 40.2K citations
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman · 2015 · 5.1K citations · Full text
EIE: Efficient Inference Engine on Compressed Deep Neural Network
Song Han, Xingyu Liu, Huizi Mao et al. · 2016 · 967 citations
Christina Delimitrou, Christos Kozyrakis · 2014 · 698 citations