Publications related to Disentangling feature and lazy training in deep neural networks

Towards Stable and Efficient Adversarial Training against $l_1$ Bounded Adversarial Attacks

Sabine Süsstrunk, Mathieu Salzmann, Yulun Jiang, Chen Liu, Zhuoyi Huang

We address the problem of stably and efficiently training a deep neural network robust to adversarial perturbations bounded by an

l_1

norm. We demonstrate that achieving robustness against

l_1

-bounded perturbations is more challenging than in the

l_2

...

2023

Euclid: Identification of asteroid streaks in simulated images using deep learning

Frédéric Courbin, Georges Meylan, Yi Wang, Richard Massey

The material composition of asteroids is an essential piece of knowledge in the quest to understand the formation and evolution of the Solar System. Visual to near-infrared spectra or multiband photometry is required to constrain the material composition o ...

Les Ulis Cedex A2023

Stereo Confidence Estimation via Locally Adaptive Fusion and Knowledge Distillation

Pascal Frossard, Seungryong Kim

Stereo confidence estimation aims to estimate the reliability of the estimated disparity by stereo matching. Different from the previous methods that exploit the limited input modality, we present a novel method that estimates confidence map of an initial ...

IEEE COMPUTER SOC2023

Linear Complexity Self-Attention With 3rd Order Polynomials

Grigorios Chrysos, Filippos Kokkinos

Self-attention mechanisms and non-local blocks have become crucial building blocks for state-of-the-art neural architectures thanks to their unparalleled ability in capturing long-range dependencies in the input. However their cost is quadratic with the nu ...

Los Alamitos2023

Explicit representations for Banach subspaces of Lizorkin distributions

Michaël Unser, Sebastian Jonas Neumayer

The Lizorkin space is well suited to the study of operators like fractional Laplacians and the Radon transform. In this paper, we show that the space is unfortunately not complemented in the Schwartz space. In return, we show that it is dense in C0(Double- ...

WORLD SCIENTIFIC PUBL CO PTE LTD2023

On the symmetries in the dynamics of wide two-layer neural networks

Lénaïc Chizat

We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a general class of s ...

AMER INST MATHEMATICAL SCIENCES-AIMS2023

A Comparison of Residual-based Methods on Fault Detection

Olga Fink, Gaëtan Michel Frusque

An important initial step in fault detection for complex industrial systems is gaining an understanding of their health condition. Subsequently, continuous monitoring of this health condition becomes crucial to observe its evolution, track changes over tim ...

PHM Society2023

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Arthur Ulysse Jacot-Guillarmod

In the recent years, Deep Neural Networks (DNNs) have managed to succeed at tasks that previously appeared impossible, such as human-level object recognition, text synthesis, translation, playing games and many more. In spite of these major achievements, o ...

EPFL2022

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Angelos Katharopoulos

Deep neural networks have completely revolutionized the field of machinelearning by achieving state-of-the-art results on various tasks ranging fromcomputer vision to protein folding. However, their application is hindered bytheir large computational and m ...

EPFL2022

Improving the Training of Compact Neural Networks for Visual Recognition

Shuxuan Guo

During the Artificial Intelligence (AI) revolution of the past decades, deep neural networks have been widely used and have achieved tremendous success in visual recognition. Unfortunately, deploying deep models is challenging because of their huge model s ...

EPFL2022

Disentangling feature and lazy training in deep neural networks

Graph Chatbot

Chat with Graph Search

Towards Stable and Efficient Adversarial Training against $l_1$ Bounded Adversarial Attacks

Euclid: Identification of asteroid streaks in simulated images using deep learning

Stereo Confidence Estimation via Locally Adaptive Fusion and Knowledge Distillation

Linear Complexity Self-Attention With 3rd Order Polynomials

Explicit representations for Banach subspaces of Lizorkin distributions

On the symmetries in the dynamics of wide two-layer neural networks

A Comparison of Residual-based Methods on Fault Detection

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Improving the Training of Compact Neural Networks for Visual Recognition

Stereo Confidence Estimation via Locally Adaptive Fusion and Knowledge Distillation

Explicit representations for Banach subspaces of Lizorkin distributions

Stop Wasting my FLOPS: Improving the Efficiency of Deep Learning Models

Theory of Deep Learning: Neural Tangent Kernel and Beyond

Improving the Training of Compact Neural Networks for Visual Recognition

Towards Stable and Efficient Adversarial Training against $l_1$ Bounded Adversarial Attacks

Linear Complexity Self-Attention With 3rd Order Polynomials

Euclid: Identification of asteroid streaks in simulated images using deep learning

On the symmetries in the dynamics of wide two-layer neural networks

A Comparison of Residual-based Methods on Fault Detection