Landscape and training regimes in deep learning

Deep learning algorithms are responsible for a technological revolution in a variety oftasks including image recognition or Go playing. Yet, why they work is not understood.Ultimately, they manage to classify data lying in high dimension – a feat genericallyimpossible due to the geometry of high dimensional space and the associatedcurse ofdimensionality. Understanding what kind of structure, symmetry or invariance makesdata such as images learnable is a fundamental challenge. Other puzzles include that(i) learning corresponds to minimizing a loss in high dimension, which is in generalnot convex and could well get stuck bad minima. (ii) Deep learning predicting powerincreases with the number of fitting parameters, even in a regime where data areperfectly fitted. In this manuscript, we review recent results elucidating (i, ii) andthe perspective they offer on the (still unexplained) curse of dimensionality paradox.We base our theoretical discussion on the (h,α) plane wherehcontrols the numberof parameters andαthe scale of the output of the network at initialization, andprovide new systematic measures of performance in that plane for two common imageclassification datasets. We argue that different learning regimes can be organized intoa phase diagram. A line of critical points sharply delimits an under-parametrized phasefrom an over-parametrized one. In over-parametrized nets, learning can operate intwo regimes separated by a smooth cross-over. At large initialization, it correspondsto a kernel method, whereas for small initializations features can be learnt, togetherwith invariants in the data. We review the properties of these different phases, ofthe transition separating them and some open questions. Our treatment emphasizesanalogies with physical systems, scaling arguments and the development of numericalobservables to quantitatively test these results empirically. Practical implications arealso discussed, including the benefit of averaging nets with distinct initial weights, orthe choice of parameters (h,α) optimizing performance.