Algorithmic Regularization in Over-parameterized Matrix Sensing and\n Neural Networks with Quadratic Activations

Abstract

We show that the gradient descent algorithm provides an implicit\nregularization effect in the learning of over-parameterized matrix\nfactorization models and one-hidden-layer neural networks with quadratic\nactivations. Concretely, we show that given $\\tilde{O}(dr^{2})$ random linear\nmeasurements of a rank $r$ positive semidefinite matrix $X^{\\star}$, we can\nrecover $X^{\\star}$ by parameterizing it by $UU^\\top$ with $U\\in \\mathbb\nR^{d\\times d}$ and minimizing the squared loss, even if $r \\ll d$. We prove\nthat starting from a small initialization, gradient descent recovers\n$X^{\\star}$ in $\\tilde{O}(\\sqrt{r})$ iterations approximately. The results\nsolve the conjecture of Gunasekar et al.'17 under the restricted isometry\nproperty. The technique can be applied to analyzing neural networks with\none-hidden-layer quadratic activations with some technical modifications.\n