Open Collections

Description

Based on joint work with Jean-Christophe Mourrat. We study the gradient-based learning dynamics of a wide two-layers neural network on a misspecified single index regression task, in a two-timescale regime where the second layer weights are learned faster than the first layer weights. Conjectures for the solution to the dynamical system describing the training dynamics were obtained recently by R. Berthier, A. Montanari and K. Zhou; using the relative training speed between the two layers as a perturbative parameter and matched asymptotic expansions arguments. We provide rigorous counterparts to these predictions. Our proofs are based on a quantitative approximation result for dynamical systems evolving near target sets defined by integral constraints involving the empirical measure of the weights. When the latter is a point mass, the auxiliary system can be viewed as obtained from a skewed projection on the tangent space to the manifold defined by the constraint functions.