Initialization of ReLUs for Dynamical Isometry

Rebekka Burkholz, Alina Dubatovka

Repository for Publications and Research Data (ETH Zurich) · 2020 · 10 citations · 0 references

DOIFull text

Open access

Concepts

Abstract

Deep learning relies on good initialization schemes and hyperparameter choices
\nprior to training a neural network. Random weight initializations induce random
\nnetwork ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles provide
\ntheoretical insights into the space of candidate models of which one is selected
\nduring training. The results obtained so far rely on mean field approximations
\nthat assume infinite layer width and that study average squared signals. We derive
\nthe joint signal output distribution exactly, without mean field assumptions, for
\nfully-connected networks with Gaussian weights and biases, and analyze deviations
\nfrom the mean field results. For rectified linear units, we further discuss limitations
\nof the standard initialization scheme, such as its lack of dynamical isometry, and
\npropose a simple alternative that overcomes these by initial parameter sharing.