Natural Language Generation: Evaluation

This lecture covers the evaluation of natural language generation models, focusing on content overlap metrics, model-based metrics, and human evaluations. The instructor discusses the challenges of evaluating the quality of generated text, the limitations of content overlap metrics, and the importance of human judgments in assessing factuality and correctness. Various metrics such as BLEU, ROUGE, and BERTScore are explained, along with their applications in different NLP tasks. The lecture emphasizes the need for better evaluation methods and highlights the role of humans in assessing text generation systems.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (33)

Related concepts (59)

CS-552: Modern natural language processing

Natural language processing is ubiquitous in modern intelligent technologies, serving as a foundation for language translators, virtual assistants, search engines, and many more. In this course, stude

Natural language generation

Natural language generation (NLG) is a software process that produces natural language output. A widely-cited survey of NLG methods describes NLG as "the subfield of artificial intelligence and computational linguistics that is concerned with the construction of computer systems than can produce understandable texts in English or other human languages from some underlying non-linguistic representation of information". While it is widely agreed that the output of any NLG process is text, there is some disagreement about whether the inputs of an NLG system need to be non-linguistic.

Metric system

The metric system is a system of measurement that succeeded the decimalised system based on the metre, which had been introduced in France in the 1790s. The historical development of these systems culminated in the definition of the International System of Units (SI) in the mid-20th century, under the oversight of an international standards body. Adopting the metric system is known as metrication. The historical evolution of metric systems has resulted in the recognition of several principles.

List of metric units

Metric units are units based on the metre, gram or second and decimal (power of ten) multiples or sub-multiples of these. The most widely used examples are the units of the International System of Units (SI). By extension they include units of electromagnetism from the CGS and SI units systems, and other units for which use of SI prefixes has become the norm.

Semantic similarity

Semantic similarity is a metric defined over a set of documents or terms, where the idea of distance between items is based on the likeness of their meaning or semantic content as opposed to lexicographical similarity. These are mathematical tools used to estimate the strength of the semantic relationship between units of language, concepts or instances, through a numerical description obtained according to the comparison of information supporting their meaning or describing their nature.

Automatic summarization

Automatic summarization is the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content. Artificial intelligence algorithms are commonly developed and employed to achieve this, specialized for different types of data. Text summarization is usually implemented by natural language processing methods, designed to locate the most informative sentences in a given document.

Vision-Language-Action Models: Training and Applications

Delves into training and applications of Vision-Language-Action models, emphasizing large language models' role in robotic control and the transfer of web knowledge. Results from experiments and future research directions are highlighted.

Evaluation in NLP CS-431: Introduction to natural language processing

Delves into NLP evaluation, covering gold standards, precision, recall, and statistical significance.

Deep Learning for NLP CS-431: Introduction to natural language processing

Introduces deep learning concepts for NLP, covering word embeddings, RNNs, and Transformers, emphasizing self-attention and multi-headed attention.

Binary Sentiment Classifier Training EE-608: Deep Learning For Natural Language Processing

Covers the training of a binary sentiment classifier using an RNN.

Deep Learning for NLP

Delves into Deep Learning for Natural Language Processing, exploring Neural Word Embeddings, Recurrent Neural Networks, and Attentive Neural Modeling with Transformers.