Publications related to Stochastic gradient descent

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Volkan Cevher, Alp Yurtsever, Maria-Luiza Vladarean

We propose a stochastic conditional gradient method (CGM) for minimizing convex finitesum objectives formed as a sum of smooth and non-smooth terms. Existing CGM variants for this template either suffer from slow convergence rates, or require carefully inc ...

2022

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Patrick Thiran, Negar Kiyavash, Saber Salehkaleybar

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with

1\le\alpha\le2

which holds in a wide range of applications in machine learning and signal processing. This conditio ...

NeurIPS2022

System and method for privacy-preserving distributed training of neural network models on distributed datasets

Jean-Pierre Hubaux, Juan Ramón Troncoso-Pastoriza, Jean-Philippe Léonard Bossuat, Apostolos Pyrgelis, David Jules Froelicher, Sinem Sav

A computer-implemented method and a distributed computer system (100) for privacy- preserving distributed training of a global neural network model on distributed datasets (DS1 to DSn). The system has a plurality of data providers (DP1 to DPn) being commun ...

2022

Metrizing Fairness

Daniel Kuhn, Yves Rychener, Bahar Taskesen

We study supervised learning problems for predicting properties of individuals who belong to one of two demographic groups, and we seek predictors that are fair according to statistical parity. This means that the distributions of the predictions within th ...

2022

ADAGRAD Avoids Saddle Points

Kimon Antonakopoulos, Xiao Wang

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Momentum-Based Policy Gradient with Second-Order Information

Patrick Thiran, Negar Kiyavash, Mohammadsadegh Khorasani, Saber Salehkaleybar

Variance-reduced gradient estimators for policy gradient methods have been one of the main focus of research in the reinforcement learning in recent years as they allow acceleration of the estimation process. We propose a variance-reduced policy-gradient m ...

2022

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Nirupam Gupta

This paper considers the multi-agent linear least-squares problem in a server-agent network architecture. The system comprises multiple agents, each with a set of local data points. The agents are connected to a server, and there is no inter-agent communic ...

PERGAMON-ELSEVIER SCIENCE LTD2022

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Loucas Pillaud-Vivien

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...

2022

Stochastic gradient descent

Graph Chatbot

Chat with Graph Search

Masked Training of Neural Networks with Partial Gradients

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Towards Understanding Sharpness-Aware Minimization

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

System and method for privacy-preserving distributed training of neural network models on distributed datasets

Metrizing Fairness

ADAGRAD Avoids Saddle Points

Momentum-Based Policy Gradient with Second-Order Information

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Masked Training of Neural Networks with Partial Gradients

Metrizing Fairness

Momentum-Based Policy Gradient with Second-Order Information

Towards Understanding Sharpness-Aware Minimization

Iterative pre-conditioning for expediting the distributed gradient-descent method: The case of linear least-squares problem

ADAGRAD Avoids Saddle Points

System and method for privacy-preserving distributed training of neural network models on distributed datasets