Lecture

Transformers: Pre-Training

Description

This lecture covers the challenges of the Transformer model, such as quadratic compute in self-attention and the limitations of absolute position representations. It also explores recent advancements in reducing the quadratic self-attention cost and improving position representations using relative linear position attention. Additionally, it discusses the concept of graph-to-graph Transformers and summarizes the key features of Transformers, including bidirectional self-attention and the use of multiple attention heads. The lecture delves into pretraining models like ELMO, BERT, and GPT, and examines word structure and subword models, emphasizing the importance of subword tokenization in NLP.

About this result
This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.