Are you an EPFL student looking for a semester project?
Work with us on data science and visualisation projects, and deploy your project as an app on top of Graph Search.
We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a general class of symmetries which, when satisfied by the target function f* and the input distribution, are preserved by the dynamics. We then study more specific cases. When f* is odd, we show that the dynamics of the predictor reduces to that of a (non -linearly parameterized) linear predictor, and its exponential convergence can be guaranteed. When f* has a low-dimensional structure, we prove that the gradient flow PDE reduces to a lower-dimensional PDE. Furthermore, we present informal and numerical arguments that suggest that the input neurons align with the lower-dimensional structure of the problem.
, ,
Marco Picasso, Alexandre Caboussat, Maude Girardin