Comparing dynamics: deep neural networks versus glassy systems

Mario Geiger, Levent Dogus Sagun, Stefano Spigler, Matthieu Wyart
IOP PUBLISHING LTD, 2019
Journal paper

Abstract

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Official source

https://infoscience.epfl.ch/record/275517?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

Mario Geiger, Levent Dogus Sagun, Stefano Spigler, Rainer Wesche

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are the complexity of the loss-landscape and of the dynamics within it, and to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and data-sets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, thus showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

2018

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

Mario Geiger, Levent Dogus Sagun, Stefano Spigler, Matthieu Wyart

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased, remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully connected deep networks a phase transition delimits the over- and underparametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the Hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity Delta which characterizes how well (Delta < 0) or badly (Delta > 0) a datum is learned. At the critical point it is power-law distributed on several decades, P+(Delta) similar to Delta(theta) for Delta > 0 and P_(Delta) similar to (-Delta)(-gamma) for Delta < 0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

AMER PHYSICAL SOC2019

The framework of complex networks has been shown to describe adequately a wide class of complex systems made up of a large number of interacting units. In this framework, a node is associated to each unit and two nodes are connected by an edge if the two units interact with each other. Examples of such systems can be found in living organisms—the map of interactions between proteins or the network of neurons in the brain. Moreover, artificial systems such as the WWW, electrical grids or airplane connections have been studied using the tools of complex networks. Finally networks have found many applications in social sciences to characterize for instance human interactions of different kinds underlying the spreading of an epidemic. For most of these systems, the complexity arises because of the large number of units and their intricate connection patterns. A natural approach is therefore to simplify the systems by decreasing their size. Different schemes can indeed be designed for each particular system, leading to effective but case-dependent methods. From a more global and statistical perspective, a promising alternative is to reduce the complexity of the corresponding networks. In order to simplify complex networks, two strategies are presented in this Thesis. The first approach refers to the well-known clustering paradigm. It aims at identifying groups of nodes densely connected between each other and much less to the rest of the network. Those groups are referred to as clusters or communities. For most real systems, nodes within a community share some similarity or common feature. For instance, in a synonymy network where nodes are words and edges connect synonymous words, we have shown that finding communities allowed us to identify words corresponding to a single concept. We have also studied a network describing the dynamics of a peptide by associating a node to a microscopic configuration and an edge to a transition. The community structure of the network was shown to provide a new methodology to explore the main characteristics of the peptide dynamics and to unravel the large-scale features of the underlying free-energy landscape. Finally we have designed a new technique to probe the robustness of the community structure against external perturbations of the network topology. This method allows us, among else, to assess whether communities correspond to a real structure of the network, or are simple artifacts of the clustering algorithms. Community detection techniques have found a large number of practical applications as a method to simplify networks since the number of clusters is often much smaller than the number of nodes. However, a crucial issue has often been disregarded: is the network of clusters truly representative of the initial one? In this Thesis, we show that this is indeed not verified for most networks. For example we have considered the evolution of random walks on the network of clusters and found that it behaves quite differently than in the initial network. This observation led us to develop a new strategy to simplify complex networks, ensuring that the reduced network is representative of the initial one. It is based on the idea of grouping nodes, akin to community detection. However, the aim is no longer to identify the "correct" clusters, but to find a smaller network which preserves the relevant features of the initial one, and especially the spectral properties. We therefore refer to our method as Spectral Coarse Graining, by analogy with the coarse graining framework used in Statistical Physics. Applying this method to various kinds of networks, we have shown that the coarse-grained network provides an excellent approximation of the initial one, while the size could be easily reduced by a factor of ten. Therefore, the Spectral Coarse Graining provides a well-defined way of studying large networks and their dynamics considering a much smaller coarse-grained version. Overall, we first discuss the use and the limits of the usual clustering approach to reduce the complexity of networks, and apply it to several real-world systems. In a second part, we develop a new coarse graining strategy to approximate large networks by smaller ones and provide several examples to illustrate the power and the novelty of the method.

EPFL2007

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

Mario Geiger, Levent Dogus Sagun, Stefano Spigler, Matthieu Wyart

AMER PHYSICAL SOC2019

EPFL2007