Publications related to Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Understanding generalization and robustness in modern deep learning

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Phenomenological theory of variational quantum ground-state preparation

Ivano Tavernelli, Giuseppe Carleo

The variational approach is a cornerstone of computational physics, considering both conventional and quantum computing computational platforms. The variational quantum eigensolver algorithm aims to prepare the ground state of a Hamiltonian exploiting para ...

College Pk2023

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Ludovic Théo Stephan, Bruno Loureiro

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Andreas Loukas, Nikolaos Karalias

Integrating functions on discrete domains into neural networks is key to developing their capability to reason about discrete objects. But, discrete domains are (I) not naturally amenable to gradient-based optimization, and (II) incompatible with deep lear ...

2022

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

On the Double Descent of Random Features Models Trained with SGD

Volkan Cevher, Fanghui Liu

We study generalization properties of random features (RF) regression in high dimensions optimized by stochastic gradient descent (SGD) in under-/overparameterized regime. In this work, we derive precise non-asymptotic error bounds of RF regression under b ...

2022

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

Patrick Thiran, Negar Kiyavash, Saber Salehkaleybar

We study the performance of Stochastic Cubic Regularized Newton (SCRN) on a class of functions satisfying gradient dominance property with

1\le\alpha\le2

which holds in a wide range of applications in machine learning and signal processing. This conditio ...

NeurIPS2022

ADAGRAD Avoids Saddle Points

Kimon Antonakopoulos, Xiao Wang

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Momentum-Based Policy Gradient with Second-Order Information

Patrick Thiran, Negar Kiyavash, Mohammadsadegh Khorasani, Saber Salehkaleybar

Variance-reduced gradient estimators for policy gradient methods have been one of the main focus of research in the reinforcement learning in recent years as they allow acceleration of the estimation process. We propose a variance-reduced policy-gradient m ...

2022

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Graph Chatbot

Chat with Graph Search

Understanding generalization and robustness in modern deep learning

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Phenomenological theory of variational quantum ground-state preparation

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions

Masked Training of Neural Networks with Partial Gradients

On the Double Descent of Random Features Models Trained with SGD

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

ADAGRAD Avoids Saddle Points

Momentum-Based Policy Gradient with Second-Order Information

Understanding generalization and robustness in modern deep learning

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Momentum-Based Policy Gradient with Second-Order Information

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Masked Training of Neural Networks with Partial Gradients

ADAGRAD Avoids Saddle Points

Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Function

On the Double Descent of Random Features Models Trained with SGD

Phenomenological theory of variational quantum ground-state preparation

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions