Publications by Nicolas Henri Bernard Flammarion

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Penalising the biases in norm regularisation enforces sparsity

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier

Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between parameters' norm and obtained estimators theoretically remains misunderstood. For one hidden ReLU layer networks ...

2023

Model agnostic methods meta-learn despite misspecifications

Nicolas Henri Bernard Flammarion, Oguz Kaan Yüksel, Etienne Patrice Boursier

Due to its empirical success on few shot classification and reinforcement learning, meta-learning recently received a lot of interest. Meta-learning leverages data from previous tasks to quickly learn a new task, despite limited data. In particular, model ...

2023

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Nicolas Henri Bernard Flammarion, Scott William Pesme, Mathieu Even

In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...

2023

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Scott William Pesme

In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minim ...

2023

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Loucas Pillaud-Vivien

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...

2022

Accelerated SGD for Non-Strongly-Convex Least Squares

Nicolas Henri Bernard Flammarion, Aditya Vardhan Varre

We consider stochastic approximation for the least squares regression problem in the non-strongly convex setting. We present the first practical algorithm that achieves the optimal prediction error rates in terms of dependence on the noise of the problem, ...

2022

Trace norm regularization for multi-task learning with scarce data

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Mikhail Konobeev

Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-scarce tasks, we consider a linear low-dimensional shared representation mode ...

2022

Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko, Francesco Croce

We propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficienc ...

ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE2022

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Nicolas Henri Bernard Flammarion

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Penalising the biases in norm regularisation enforces sparsity

Model agnostic methods meta-learn despite misspecifications

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Accelerated SGD for Non-Strongly-Convex Least Squares

Trace norm regularization for multi-task learning with scarce data

Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks

Towards Understanding Sharpness-Aware Minimization

Saddle-to-Saddle Dynamics in Diagonal Linear Networks

Model agnostic methods meta-learn despite misspecifications

Penalising the biases in norm regularisation enforces sparsity

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks

(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Towards Understanding Sharpness-Aware Minimization

Accelerated SGD for Non-Strongly-Convex Least Squares

Trace norm regularization for multi-task learning with scarce data