Publications related to Stochastic Spectral Descent for Restricted Boltzmann Machines.

Understanding generalization and robustness in modern deep learning

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Brookline2024

The inductive bias of deep learning: Connecting weights and functions

Guillermo Ortiz Jimenez

Years of a fierce competition have naturally selected the fittest deep learning algorithms. Yet, although these models work well in practice, we still lack a proper characterization of why they do so. This poses serious questions about the robustness, trus ...

EPFL2023

Deep Learning Generalization with Limited and Noisy Labels

Mahsa Forouzesh

Deep neural networks have become ubiquitous in today's technological landscape, finding their way in a vast array of applications. Deep supervised learning, which relies on large labeled datasets, has been particularly successful in areas such as image cla ...

EPFL2023

Communication-efficient distributed training of machine learning models

Thijs Vogels

In this thesis, we explore techniques for addressing the communication bottleneck in data-parallel distributed training of deep learning models. We investigate algorithms that either reduce the size of the messages that are exchanged between workers, or th ...

EPFL2023

MULTI-TASK CURRICULUM LEARNING FOR PARTIALLY LABELED DATA

Jiancheng Yang, Stanislav Lukyanenko

Incomplete labels are common in multi-task learning for biomedical applications due to several practical difficulties, e.g., expensive annotation efforts by experts, limit of data collection, different sources of data. A naive approach to enable joint lear ...

New York2023

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Discriminative clustering with representation learning with any ratio of labeled to unlabeled data

We present a discriminative clustering approach in which the feature representation can be learned from data and moreover leverage labeled data. Representation learning can give a similarity-based clustering method the ability to automatically adapt to an ...

2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Patrick Thiran, Negar Kiyavash, Saber Salehkaleybar

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with

1\le\alpha\le2

which holds in a wide range of applications in machine learning and signal processing. This conditio ...

NeurIPS2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Stochastic Spectral Descent for Restricted Boltzmann Machines.

Graph Chatbot

Chat with Graph Search

Understanding generalization and robustness in modern deep learning

On the Generalization of Stochastic Gradient Descent with Momentum

The inductive bias of deep learning: Connecting weights and functions

Deep Learning Generalization with Limited and Noisy Labels

Communication-efficient distributed training of machine learning models

MULTI-TASK CURRICULUM LEARNING FOR PARTIALLY LABELED DATA

Masked Training of Neural Networks with Partial Gradients

Discriminative clustering with representation learning with any ratio of labeled to unlabeled data

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

The inductive bias of deep learning: Connecting weights and functions

Deep Learning Generalization with Limited and Noisy Labels

Communication-efficient distributed training of machine learning models

MULTI-TASK CURRICULUM LEARNING FOR PARTIALLY LABELED DATA

Discriminative clustering with representation learning with any ratio of labeled to unlabeled data

Understanding generalization and robustness in modern deep learning

On the Generalization of Stochastic Gradient Descent with Momentum

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Masked Training of Neural Networks with Partial Gradients

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks