Adaptive Stochastic Natural Gradient Method for One-Shot Neural\n Architecture Search

Youhei Akimoto, Shinichi Shirakawa, Nozomu Yoshinari, Kento Uchida, Shota Saito, Kouhei Nishida

arXiv (Cornell University) · 2019 · 24 citations · 20 references

DOIFull text

Open access

Concepts

Abstract

High sensitivity of neural architecture search (NAS) methods against their\ninput such as step-size (i.e., learning rate) and search space prevents\npractitioners from applying them out-of-the-box to their own problems, albeit\nits purpose is to automate a part of tuning process. Aiming at a fast, robust,\nand widely-applicable NAS, we develop a generic optimization framework for NAS.\nWe turn a coupled optimization of connection weights and neural architecture\ninto a differentiable optimization by means of stochastic relaxation. It\naccepts arbitrary search space (widely-applicable) and enables to employ a\ngradient-based simultaneous optimization of weights and architecture (fast). We\npropose a stochastic natural gradient method with an adaptive step-size\nmechanism built upon our theoretical investigation (robust). Despite its\nsimplicity and no problem-dependent parameter tuning, our method exhibited near\nstate-of-the-art performances with low computational budgets both on image\nclassification and inpainting tasks.\n

References

20