Word embedding

In natural language processing (NLP), a word embedding is a representation of a word. The embedding is used in text analysis. Typically, the representation is a real-valued vector that encodes the meaning of the word in such a way that words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using language modeling and feature learning techniques, where words or phrases from the vocabulary are mapped to vectors of real numbers. Methods to generate this mapping include neural networks, dimensionality reduction on the word co-occurrence matrix, probabilistic models, explainable knowledge base method, and explicit representation in terms of the context in which words appear. Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as syntactic parsing and sentiment analysis. Development and history of the approach In distributional semantics

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (88)

Related publications

Related people (17)

Related people

Related units (10)

Related units

Related concepts (11)

Related concepts

Related courses (16)

Related courses

Related lectures (58)

Related lectures

Natural language processing

Natural language processing (NLP) is an interdisciplinary subfield of linguistics and computer science. It is primarily concerned with processing natural language datasets, such as text corpora or sp

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the

Word2vec

Word2vec is a technique for natural language processing (NLP) published in 2013. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained

CS-423: Distributed information systems

This course introduces the key concepts and algorithms from the areas of information retrieval, data mining and knowledge bases, which constitute the foundations of today's Web-based distributed information systems.

EE-608: Deep Learning For Natural Language Processing

The Deep Learning for NLP course provides an overview of neural network based methods applied to text. The focus is on models particularly suited to the properties of human language, such as categorical, unbounded, and structured representations, and very large input and output vocabularies.

CS-431: Introduction to natural language processing

The objective of this course is to present the main models, formalisms and algorithms necessary for the development of applications in the field of natural language information processing. The concepts introduced during the lectures will be applied during practical sessions.

Further results on latent discourse models and word embeddings

Youssef Allouah

We discuss some properties of generative models for word embeddings. Namely, (Arora et al., 2016) proposed a latent discourse model implying the concentration of the partition function of the word vectors. This concentration phenomenon led to an asymptotic linear relation between the pointwise mutual information (PMI) of pairs of words and the scalar product of their vectors. Here, we first revisit this concentration phenomenon and prove it under slightly weaker assumptions, for a set of random vectors symmetrically distributed around the origin. Second, we empirically evaluate the relation between PMI and scalar products of word vectors satisfying the concentration property. Our empirical results indicate that, in practice, this relation does not hold with arbitrarily small error. This observation is further supported by two theoretical results: (i) the error cannot be exactly zero because the corresponding shifted PMI matrix cannot be positive semidefinite; (ii) under mild assumptions, there exist pairs of words for which the error cannot be close to zero. We deduce that either natural language does not follow the assumptions of the considered generative model, or the current word vector generation methods do not allow the construction of the hypothesized word embeddings.

MICROTOME PUBL2021

Natural Language Processing (NLP) driven categorisation and detection of discourse in historical US patents

Jérôme Baudry, Nicolas Christophe Chachereau, Prakhar Gupta, Bhargav Srinivasa Desikan

Patents have traditionally been used in the history of technology as an indication of the thinking process of the inventors, of the challenges or “reverse salients” they faced, or of the social groups influencing the construction of technology. More recently, historians of science and technology also read them to interpret the way people described technology and how the specific inscriptions of inventions mattered for the justification and operation of the patent system. The digitization of historical patents opens up unique opportunities to assess the feasibility of unsupervised machine learning and natural language methods for such explorations. In this project, we analyze over a million US historical patents from 1830-1930 using a variety of text-based methods, with two major aims: 1) categorizing patents into coherent technical categories, 2) identifying discourses of safety, reflexivity, and environmental concern in technological innovation. We use both frequency-based and context-based methods, and find that a bag of words-based methods such as TDF-IDF and topic modeling do not perform well on semantic categorization due to the linguistic peculiarities of patent specifications. This suggests that a successful approach to categorizing patents would require contextual semantic representations such as Transformers-based methods (e.g. BERT), or static embedding based methods (e.g Word2Vec, Doc2Vec) which have relatively low computational costs but less expressive in some scenarios. We run early experiments using these methods and find that word embedding models are effective in learning semantics from the descriptions of the patents. In this poster, we will describe our early results, as well as exploratory data analysis on this massive historical patents dataset.

2022

Scalable Robust Graph Embedding with Spark

Karl Aberer, Chi Thang Duong, Trung-Dung Hoang, Quoc Viet Hung Nguyen

Graph embedding aims at learning a vector-based representation of vertices that incorporates the structure of the graph. This representation then enables inference of graph properties. Existing graph embedding techniques, however, do not scale well to large graphs. While several techniques to scale graph embedding using compute clusters have been proposed, they require continuous communication between the compute nodes and cannot handle node failure. We therefore propose a framework for scalable and robust graph embedding based on the MapReduce model, which can distribute any existing embedding technique. Our method splits a graph into subgraphs to learn their embeddings in isolation and subsequently reconciles the embedding spaces derived for the subgraphs. We realize this idea through a novel distributed graph decomposition algorithm. In addition, we show how to implement our framework in Spark to enable efficient learning of effective embeddings. Experimental results illustrate that our approach scales well, while largely maintaining the embedding quality.

ASSOC COMPUTING MACHINERY2021