Publications related to Sparse approximation

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

EPFL2024

Multi-Frequency Tracking via Group-Sparse Optimal Transport

Isabel Haasler

In this letter, we introduce an optimal transport framework for inferring power distributions over both spatial location and temporal frequency. Recently, it has been shown that optimal transport is a powerful tool for estimating spatial spectra that chang ...

Ieee-Inst Electrical Electronics Engineers Inc2024

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

The inductive bias of deep learning: Connecting weights and functions

Guillermo Ortiz Jimenez

Years of a fierce competition have naturally selected the fittest deep learning algorithms. Yet, although these models work well in practice, we still lack a proper characterization of why they do so. This poses serious questions about the robustness, trus ...

EPFL2023

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Nicolas Henri Bernard Flammarion, Scott William Pesme, Mathieu Even

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...

2023

Penalising the biases in norm regularisation enforces sparsity

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier

Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between parameters' norm and obtained estimators theoretically remains misunderstood. For one hidden ReLU layer networks ...

2023

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Scott William Pesme

In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minim ...

2023

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

Thomas Jean Debarre

The goal of this thesis is to study continuous-domain inverse problems for the reconstruction of sparse signals and to develop efficient algorithms to solve such problems computationally. The task is to recover a signal of interest as a continuous function ...

EPFL2022

A Structured Dictionary Perspective on Implicit Neural Representations

Pascal Frossard, Guillermo Ortiz Jimenez, Gizem Yüce, Beril Besbinar

Implicit neural representations (INRs) have recently emerged as a promising alternative to classical discretized representations of signals. Nevertheless, despite their practical success, we still do not understand how INRs represent signals. We propose a ...

IEEE COMPUTER SOC2022

From Undercomplete to Sparse Overcomplete Autoencoders to Improve LF-MMI Speech Recognition

Hervé Bourlard, Selen Hande Kabil

Starting from a strong Lattice-Free Maximum Mutual Information (LF-MMI) baseline system, we explore different autoencoder configurations to enhance Mel-Frequency Cepstral Coefficients (MFCC) features. Autoencoders are expected to generate new MFCC features ...

ISCA-INT SPEECH COMMUNICATION ASSOC2022

Sparse approximation

Graph Chatbot

Chat with Graph Search

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Multi-Frequency Tracking via Group-Sparse Optimal Transport

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

The inductive bias of deep learning: Connecting weights and functions

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Penalising the biases in norm regularisation enforces sparsity

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

A Structured Dictionary Perspective on Implicit Neural Representations

From Undercomplete to Sparse Overcomplete Autoencoders to Improve LF-MMI Speech Recognition

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Multi-Frequency Tracking via Group-Sparse Optimal Transport

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

A Structured Dictionary Perspective on Implicit Neural Representations

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

The inductive bias of deep learning: Connecting weights and functions

Variational Methods For Continuous-Domain Inverse Problems: the Quest for the Sparsest Solution

From Undercomplete to Sparse Overcomplete Autoencoders to Improve LF-MMI Speech Recognition

Penalising the biases in norm regularisation enforces sparsity