Beyond Linearization: On Quadratic and Higher-Order Approximation of\n Wide Neural Networks

Abstract

Recent theoretical work has established connections between over-parametrized\nneural networks and linearized models governed by he Neural Tangent Kernels\n(NTKs). NTK theory leads to concrete convergence and generalization results,\nyet the empirical performance of neural networks are observed to exceed their\nlinearized models, suggesting insufficiency of this theory.\n Towards closing this gap, we investigate the training of over-parametrized\nneural networks that are beyond the NTK regime yet still governed by the Taylor\nexpansion of the network. We bring forward the idea of \\emph{randomizing} the\nneural networks, which allows them to escape their NTK and couple with\nquadratic models. We show that the optimization landscape of randomized\ntwo-layer networks are nice and amenable to escaping-saddle algorithms. We\nprove concrete generalization and expressivity results on these randomized\nnetworks, which lead to sample complexity bounds (of learning certain simple\nfunctions) that match the NTK and can in addition be better by a dimension\nfactor when mild distributional assumptions are present. We demonstrate that\nour randomization technique can be generalized systematically beyond the\nquadratic case, by using it to find networks that are coupled with higher-order\nterms in their Taylor series.\n