Publication

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Related publications (78)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Geodesic Convolutional Shape Optimization

Pascal Fua, François Fleuret, Pierre Bruno Baqué, Edoardo Remelli

Aerodynamic shape optimization has many industrial applications. Existing methods, however, are so computationally demanding that typical engineering practices are to either simply try a limited number of hand-designed shapes or restrict oneself to shapes ...

2018

On the linear convergence of the stochastic gradient method with constant step-size

Volkan Cevher, Cong Bang Vu

The strong growth condition (SGC) is known to be a sufficient condition for linear convergence of the stochastic gradient method using a constant step-size γ (SGM-CS). In this paper, we provide a necessary condition, for the linear convergence of SGM-CS, t ...

2018

Learning Proximal Operators with Gaussian Processes

Colin Neil Jones, Georgios Stathopoulos, Xuan Truong Nghiem

Several distributed-optimization setups involve a group of agents coordinated by a central entity (coordinator), altogether operating in a collaborative framework. In such environments, it is often common that the agents solve proximal minimization problem ...

IEEE2018

Theory of representation learning in cortical neural networks

Carlos Stein Naves de Brito

Our brain continuously self-organizes to construct and maintain an internal representation of the world based on the information arriving through sensory stimuli. Remarkably, cortical areas related to different sensory modalities appear to share the same f ...

EPFL2016

Adaptive data augmentation for image classification

Pascal Frossard, Alhussein Fawzi

Data augmentation is the process of generating samples by transforming training data, with the target of improving the accuracy and robustness of classifiers. In this paper, we propose a new automatic and adaptive algorithm for choosing the transformations ...

IEEE2016

Robust Online Time Series Prediction with Recurrent Neural Networks

Karl Aberer, Tian Guo

Time series forecasting for streaming data plays an important role in many real applications, ranging from IoT systems, cyber-networks, to industrial systems and healthcare. However the real data is often complicated with anomalies and change points, which ...

Ieee2016

Stochastic gradient descent with finite samples sizes

Ali H. Sayed, Stefan Vlaski, Bicheng Ying, Kun Yuan

The minimization of empirical risks over finite sample sizes is an important problem in large-scale machine learning. A variety of algorithms has been proposed in the literature to alleviate the computational burden per iteration at the expense of converge ...

IEEE2016

Stochastic Spectral Descent for Discrete Graphical Models

Volkan Cevher, Ya-Ping Hsieh, Edo Collins

Interest in deep probabilistic graphical models has increased in recent years, due to their state-of-the-art perfor- mance on many machine learning applications. Such models are typically trained with the stochastic gradient method, which can take a signif ...

Ieee-Inst Electrical Electronics Engineers Inc2016

Preconditioned Spectral Descent for Deep Learning

Volkan Cevher, Ya-Ping Hsieh, Edo Collins

Deep learning presents notorious computational challenges. These challenges in- clude, but are not limited to, the non-convexity of learning objectives and estimat- ing the quantities needed for optimization algorithms, such as gradients. While we do not a ...

2015

Stochastic Spectral Descent for Restricted Boltzmann Machines.

Volkan Cevher

Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a ...

2015