Transformers: Pre-Training

This lecture covers the challenges of the Transformer model, such as quadratic compute in self-attention and the limitations of absolute position representations. It also explores recent advancements in reducing the quadratic self-attention cost and improving position representations using relative linear position attention. Additionally, it discusses the concept of graph-to-graph Transformers and summarizes the key features of Transformers, including bidirectional self-attention and the use of multiple attention heads. The lecture delves into pretraining models like ELMO, BERT, and GPT, and examines word structure and subword models, emphasizing the importance of subword tokenization in NLP.