Lecture

Transformers: Full Architecture and Self-Attention Mechanism

Description

This lecture covers the full architecture of Transformers, consisting of encoder and decoder blocks with multi-headed attention layers and feedforward networks. It explains the self-attention mechanism used for encoding sequences without recurrent computations, along with the importance of positional embeddings. The lecture also delves into the specifics of self-attention in both encoder and decoder blocks, highlighting the Nobel committee's recognition of Strickland for advancing optics. Additionally, it discusses the differences between self-attention in the encoder and decoder, the masked multi-headed attention, and the cross-attention mechanism. The lecture concludes with insights on the paradigm shift brought by using completely pretrained models like GPT and the massive improvements in NLP tasks achieved through models like GPT-2 and GPT-3.

Instructor
aute tempor
Consequat aliquip aliquip esse voluptate ad aliquip culpa tempor magna ea laborum. Officia irure fugiat officia et consectetur mollit ad dolor dolor. Sit laboris consectetur amet velit aute laboris. Dolore proident culpa aute cupidatat sunt sunt aute ut.
Login to see this section
About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.
Related lectures (36)
Vision-Language-Action Models: Training and Applications
Delves into training and applications of Vision-Language-Action models, emphasizing large language models' role in robotic control and the transfer of web knowledge. Results from experiments and future research directions are highlighted.
Language Models: From Theory to Computation
Explores the mathematics of language models, covering architecture design, pre-training, and fine-tuning, emphasizing the importance of pre-training and fine-tuning for various tasks.
Deep Learning for NLP
Introduces deep learning concepts for NLP, covering word embeddings, RNNs, and Transformers, emphasizing self-attention and multi-headed attention.
Deep Learning for NLP
Explores deep learning for NLP, covering word embeddings, context representations, learning techniques, and challenges like vanishing gradients and ethical considerations.
Graph-to-Graph Transformers: Syntax-aware Graph Encoding
Introduces the Syntax-aware Graph-to-Graph Transformer architecture for effective conditioning on syntactic dependency graphs.
Show more

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.