Person
Nicolas Henri Bernard Flammarion
Related publications (35)
Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks
Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme
In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size and momentum parameter that allows u ...
2024Penalising the biases in norm regularisation enforces sparsity
Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier
Controlling the parameters' norm often yields good generalisation when training neural networks. Beyond simple intuitions, the relation between parameters' norm and obtained estimators theoretically remains misunderstood. For one hidden ReLU layer networks ...
2023Model agnostic methods meta-learn despite misspecifications
Nicolas Henri Bernard Flammarion, Oguz Kaan Yüksel, Etienne Patrice Boursier
Due to its empirical success on few shot classification and reinforcement learning, meta-learning recently received a lot of interest. Meta-learning leverages data from previous tasks to quickly learn a new task, despite limited data. In particular, model ...
2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability
Nicolas Henri Bernard Flammarion, Scott William Pesme, Mathieu Even
In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscop ...
2023Saddle-to-Saddle Dynamics in Diagonal Linear Networks
Nicolas Henri Bernard Flammarion, Scott William Pesme
In this paper we fully describe the trajectory of gradient flow over diagonal linear networks in the limit of vanishing initialisation. We show that the limiting flow successively jumps from a saddle of the training loss to another until reaching the minim ...
2023Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Loucas Pillaud-Vivien
The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...
2022Accelerated SGD for Non-Strongly-Convex Least Squares
Nicolas Henri Bernard Flammarion, Aditya Vardhan Varre
We consider stochastic approximation for the least squares regression problem in the non-strongly convex setting. We present the first practical algorithm that achieves the optimal prediction error rates in terms of dependence on the noise of the problem, ...
2022Trace norm regularization for multi-task learning with scarce data
Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Mikhail Konobeev
Multi-task learning leverages structural similarities between multiple tasks to learn despite very few samples. Motivated by the recent success of neural networks applied to data-scarce tasks, we consider a linear low-dimensional shared representation mode ...
2022Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks
Nicolas Henri Bernard Flammarion, Maksym Andriushchenko, Francesco Croce
We propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficienc ...
ASSOC ADVANCEMENT ARTIFICIAL INTELLIGENCE2022