Single-Path Mobile AutoML: Efficient ConvNet Design and NAS\n Hyperparameter Optimization

Abstract

Can we reduce the search cost of Neural Architecture Search (NAS) from days\ndown to only few hours? NAS methods automate the design of Convolutional\nNetworks (ConvNets) under hardware constraints and they have emerged as key\ncomponents of AutoML frameworks. However, the NAS problem remains challenging\ndue to the combinatorially large design space and the significant search time\n(at least 200 GPU-hours). In this work, we alleviate the NAS search cost down\nto less than 3 hours, while achieving state-of-the-art image classification\nresults under mobile latency constraints. We propose a novel differentiable NAS\nformulation, namely Single-Path NAS, that uses one single-path\nover-parameterized ConvNet to encode all architectural decisions based on\nshared convolutional kernel parameters, hence drastically decreasing the search\noverhead. Single-Path NAS achieves state-of-the-art top-1 ImageNet accuracy\n(75.62%), hence outperforming existing mobile NAS methods in similar latency\nsettings (~80ms). In particular, we enhance the accuracy-runtime trade-off in\ndifferentiable NAS by treating the Squeeze-and-Excitation path as a fully\nsearchable operation with our novel single-path encoding. Our method has an\noverall cost of only 8 epochs (24 TPU-hours), which is up to 5,000x faster\ncompared to prior work. Moreover, we study how different NAS formulation\nchoices affect the performance of the designed ConvNets. Furthermore, we\nexploit the efficiency of our method to answer an interesting question: instead\nof empirically tuning the hyperparameters of the NAS solver (as in prior work),\ncan we automatically find the hyperparameter values that yield the desired\naccuracy-runtime trade-off? We open-source our entire codebase at:\nhttps://github.com/dstamoulis/single-path-nas.\n