Publications related to Stochastic Spectral Descent for Restricted Boltzmann Machines.

Convergence of adaptive algorithms for constrained weakly convex optimization

Volkan Cevher, Ahmet Alacaoglu, Yurii Malitskyi

We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the

\tilde O(t^{-1/2})

rate of convergence for the squared norm of the gradient of Moreau envelope, ...

2021

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Lenka Zdeborová, Francesca Mignacco

In this paper we investigate how gradient-based algorithms such as gradient descent (GD), (multi-pass) stochastic GD, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best general ...

IOP PUBLISHING LTD2021

Extrapolation for Large-batch Training in Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batc ...

2020

Second-Order Guarantees In Federated Learning

Ali H. Sayed, Stefan Vlaski, Elsa Rizk

Federated learning is a useful framework for centralized learning from distributed data under practical considerations of heterogeneity, asynchrony, and privacy. Federated architectures are frequently deployed in deep learning settings, which generally giv ...

IEEE2020

Multi-armed Bandits in Action

Farnood Salehi

Making decisions is part and parcel of being human. Among a set of actions, we want to choose the one that has the highest reward. But the uncertainty of the outcome prevents us from always making the right decision. Making decisions under uncertainty can ...

EPFL2020

Byzantine tolerant gradient descent for distributed machine learning with adversaries

Rachid Guerraoui, El Mahdi El Mhamdi, Julien Michel Stainer, Peva François Blanchard

The present application concerns a computer-implemented method for training a machine learning model in a distributed fashion, using Stochastic Gradient Descent, SGD, wherein the method is performed by a first computer in a distributed computing environmen ...

2019

Local SGD Converges Fast and Communicates Little

Sebastian Urban Stich

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large ne ...

2019

Diffusion Learning In Non-Convex Environments

Ali H. Sayed, Stefan Vlaski

Driven by the need to solve increasingly complex optimization problems in signal processing and machine learning, recent years have seen rising interest in the behavior of gradient-descent based algorithms in non-convex environments. Most of the works on d ...

IEEE2019

Learning Positive Functions with Pseudo Mirror Descent

Negar Kiyavash

The nonparametric learning of positive-valued functions appears widely in machine learning, especially in the context of estimating intensity functions of point processes. Yet, existing approaches either require computing expensive projections or semidefin ...

Curran Associates, Inc.2019

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

Mohammad Emtiyaz Khan

Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distribution. In such situations, existing methods usually resort to a diagonal approxi ...

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)2018

Stochastic Spectral Descent for Restricted Boltzmann Machines.

Graph Chatbot

Chat with Graph Search

Convergence of adaptive algorithms for constrained weakly convex optimization

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Extrapolation for Large-batch Training in Deep Learning

Second-Order Guarantees In Federated Learning

Multi-armed Bandits in Action

Byzantine tolerant gradient descent for distributed machine learning with adversaries

Local SGD Converges Fast and Communicates Little

Diffusion Learning In Non-Convex Environments

Learning Positive Functions with Pseudo Mirror Descent

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient

Learning Positive Functions with Pseudo Mirror Descent

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Convergence of adaptive algorithms for constrained weakly convex optimization

Extrapolation for Large-batch Training in Deep Learning

Second-Order Guarantees In Federated Learning

Multi-armed Bandits in Action

Byzantine tolerant gradient descent for distributed machine learning with adversaries

Local SGD Converges Fast and Communicates Little

Diffusion Learning In Non-Convex Environments

SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient