Word Embeddings: Glove and Semantic Relationships

This lecture covers the concept of word embeddings, focusing on the Glove model and semantic relationships. It explains the learning of parameters using positive and negative examples, regularization techniques, and the computation of derivatives. The instructor discusses the use of subword embeddings, Byte Pair Encoding, and Hierarchical Softmax to improve efficiency. The lecture also delves into Glove's loss function, global co-occurrence counts, and modeling ratios. It explores word analogies, syntactic relationships, and semantic dimensions. Additionally, the lecture highlights the properties of word embeddings, such as clustering similar terms and encoding relationships. References to relevant articles in the field are provided.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (28)

Related concepts (48)

CS-423: Distributed information systems

This course introduces the foundations of information retrieval, data mining and knowledge bases, which constitute the foundations of today's Web-based distributed information systems.

Word embedding

In natural language processing (NLP), a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers.

Word2vec

Word2vec is a technique for natural language processing (NLP) published in 2013. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence. As the name implies, word2vec represents each distinct word with a particular list of numbers called a vector.

Large language model

A large language model (LLM) is a language model characterized by its large size. Their size is enabled by AI accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Transformer (machine learning model)

A transformer is a deep learning architecture that relies on the parallel multi-head attention mechanism. The modern transformer was proposed in the 2017 paper titled 'Attention Is All You Need' by Ashish Vaswani et al., Google Brain team. It is notable for requiring less training time than previous recurrent neural architectures, such as long short-term memory (LSTM), and its later variation has been prevalently adopted for training large language models on large (language) datasets, such as the Wikipedia corpus and Common Crawl, by virtue of the parallelized processing of input sequence.

Link-based Ranking: Fundamentals and Algorithms CS-423: Distributed information systems

Covers the fundamentals and algorithms of link-based ranking, including anchor text indexing, PageRank, HITS, and practical implementations.

Document Classification: Overview CS-423: Distributed information systems

Explores document classification methods, including k-Nearest-Neighbors, Naïve Bayes Classifier, transformer models, and multi-head attention.

Word Embedding Models: Optimization and Applications CS-423: Distributed information systems

Explores optimizing word embedding models, including loss function minimization and gradient descent, and introduces techniques like Fasttext and Byte Pair Encoding.

Word Embeddings: Models and Learning CS-423: Distributed information systems

Explores word embeddings, context importance, and learning algorithms for creating new representations.

Deep Learning for NLP CS-431: Introduction to natural language processing

Introduces deep learning concepts for NLP, covering word embeddings, RNNs, and Transformers, emphasizing self-attention and multi-headed attention.