Publications related to Attention (machine learning)

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Topics in statistical physics of high-dimensional machine learning

Hugo Chao Cui

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

EPFL2024

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Scott William Pesme

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Seyed Mohammad Mahdi Johari

Recent advancements in deep learning have revolutionized 3D computer vision, enabling the extraction of intricate 3D information from 2D images and video sequences. This thesis explores the application of deep learning in three crucial challenges of 3D com ...

EPFL2024

Linear Complexity Self-Attention With 3rd Order Polynomials

Grigorios Chrysos, Filippos Kokkinos

Self-attention mechanisms and non-local blocks have become crucial building blocks for state-of-the-art neural architectures thanks to their unparalleled ability in capturing long-range dependencies in the input. However their cost is quadratic with the nu ...

Los Alamitos2023

Vision Transformer Adapters for Generalizable Multitask Learning

Sabine Süsstrunk, Mathieu Salzmann, Deblina Bhattacharjee

We introduce the first multitasking vision transformer adapters that learn generalizable task affinities which can be applied to novel tasks and domains. Integrated into an off-the-shelf vision transformer backbone, our adapters can simultaneously solve mu ...

2023

Exploration-based model learning with self-attention for risk-sensitive robot control

Sudong Lee

Model-based reinforcement learning for robot control offers the advantages of overcoming concerns on data collection and iterative processes for policy improvement in model-free methods. However, both methods use exploration strategy relying on heuristics ...

2023

Modeling Structured Data in Attention-based Models

Alireza Mohammadshahi

Natural language processing has experienced significant improvements with the development of Transformer-based models, which employ self-attention mechanism and pre-training strategies. However, these models still present several obstacles. A notable issue ...

EPFL2023

Transformer Models for Vision

Jean-Baptiste Francis Marie Juliette Cordonnier

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins.All these successful methods depend on a gradient-based learning algorithm to train a model on massive a ...

EPFL2023

Classification of Tropical Deforestation Drivers with Machine Learning and Satellite Image Time Series

Devis Tuia, Jan Pisl, Marc Conrad Russwurm, Lloyd Haydn Hughes

Tropical deforestation is a major environmental problem with severe consequences such as carbon emissions or biodiversity loss. While much research focuses on monitoring and mapping deforestation, less attention is paid to understanding the various reasons ...

The Institute of Electrical and Electronics Engineers, Inc.2023

Attention (machine learning)

Graph Chatbot

Chat with Graph Search

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Topics in statistical physics of high-dimensional machine learning

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Linear Complexity Self-Attention With 3rd Order Polynomials

Vision Transformer Adapters for Generalizable Multitask Learning

Exploration-based model learning with self-attention for risk-sensitive robot control

Modeling Structured Data in Attention-based Models

Transformer Models for Vision

Classification of Tropical Deforestation Drivers with Machine Learning and Satellite Image Time Series

Topics in statistical physics of high-dimensional machine learning

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Linear Complexity Self-Attention With 3rd Order Polynomials

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Modeling Structured Data in Attention-based Models

Transformer Models for Vision

Vision Transformer Adapters for Generalizable Multitask Learning

Exploration-based model learning with self-attention for risk-sensitive robot control

Classification of Tropical Deforestation Drivers with Machine Learning and Satellite Image Time Series