Are you an EPFL student looking for a semester project?
Work with us on data science and visualisation projects, and deploy your project as an app on top of Graph Search.
This lecture covers the challenges of the Transformer model, such as quadratic compute in self-attention and the limitations of absolute position representations. It also explores recent advancements in reducing the quadratic self-attention cost and improving position representations using relative linear position attention. Additionally, it discusses the concept of graph-to-graph Transformers and summarizes the key features of Transformers, including bidirectional self-attention and the use of multiple attention heads. The lecture delves into pretraining models like ELMO, BERT, and GPT, and examines word structure and subword models, emphasizing the importance of subword tokenization in NLP.
This video is available exclusively on Mediaspace for a restricted audience. Please log in to MediaSpace to access it if you have the necessary permissions.
Watch on Mediaspace