Publication

Linear Complexity Self-Attention With 3rd Order Polynomials

Related concepts (36)

Graph Chatbot

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

Attention

Attention is the concentration of awareness on some phenomenon to the exclusion of other stimuli. It is a process of selectively concentrating on a discrete aspect of information, whether considered subjective or objective. William James (1890) wrote that "Attention is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration, of consciousness are of its essence.

Cyclotomic polynomial

In mathematics, the nth cyclotomic polynomial, for any positive integer n, is the unique irreducible polynomial with integer coefficients that is a divisor of and is not a divisor of for any k < n. Its roots are all nth primitive roots of unity where k runs over the positive integers not greater than n and coprime to n (and i is the imaginary unit). In other words, the nth cyclotomic polynomial is equal to It may also be defined as the monic polynomial with integer coefficients that is the minimal polynomial over the field of the rational numbers of any primitive nth-root of unity ( is an example of such a root).

Irreducible polynomial

In mathematics, an irreducible polynomial is, roughly speaking, a polynomial that cannot be factored into the product of two non-constant polynomials. The property of irreducibility depends on the nature of the coefficients that are accepted for the possible factors, that is, the field to which the coefficients of the polynomial and its possible factors are supposed to belong. For example, the polynomial x2 − 2 is a polynomial with integer coefficients, but, as every integer is also a real number, it is also a polynomial with real coefficients.

Self-harm

Self-harm is intentional behavior that is considered harmful to oneself. This is most commonly regarded as direct injury of one's own skin tissues usually without a suicidal intention. Other terms such as cutting, self-injury, and self-mutilation have been used for any self-harming behavior regardless of suicidal intent. The most common form of self-harm is using a sharp object to cut the skin. Other forms include scratching, hitting, or burning body parts.

Attention deficit hyperactivity disorder

Attention deficit hyperactivity disorder (ADHD) is a neurodevelopmental disorder characterised by excessive amounts of inattention, hyperactivity, and impulsivity that are pervasive, impairing in multiple contexts, and otherwise age-inappropriate. ADHD symptoms arise from executive dysfunction, and emotional dysregulation is often considered a core symptom. In children, problems paying attention may result in poor school performance.

Time delay neural network

Time delay neural network (TDNN) is a multilayer artificial neural network architecture whose purpose is to 1) classify patterns with shift-invariance, and 2) model context at each layer of the network. Shift-invariant classification means that the classifier does not require explicit segmentation prior to classification. For the classification of a temporal pattern (such as speech), the TDNN thus avoids having to determine the beginning and end points of sounds before classifying them.

Reciprocal polynomial

In algebra, given a polynomial with coefficients from an arbitrary field, its reciprocal polynomial or reflected polynomial, denoted by p∗ or pR, is the polynomial That is, the coefficients of p∗ are the coefficients of p in reverse order. Reciprocal polynomials arise naturally in linear algebra as the characteristic polynomial of the inverse of a matrix. In the special case where the field is the complex numbers, when the conjugate reciprocal polynomial, denoted p†, is defined by, where denotes the complex conjugate of , and is also called the reciprocal polynomial when no confusion can arise.

Chebyshev polynomials

The Chebyshev polynomials are two sequences of polynomials related to the cosine and sine functions, notated as and . They can be defined in several equivalent ways, one of which starts with trigonometric functions: The Chebyshev polynomials of the first kind are defined by Similarly, the Chebyshev polynomials of the second kind are defined by That these expressions define polynomials in may not be obvious at first sight, but follows by rewriting and using de Moivre's formula or by using the angle sum formulas for and repeatedly.

Defence mechanism

In psychoanalytic theory, a defence mechanism (American English: defense mechanism) is an unconscious psychological operation that functions to protect a person from anxiety-producing thoughts and feelings related to internal conflicts and outer stressors. Defence mechanisms (Abwehrmechanismen) are unconscious psychological processes employed to defend against feelings of anxiety and unacceptable impulses at the level of consciousness.

Generative adversarial network

A generative adversarial network (GAN) is a class of machine learning framework and a prominent framework for approaching generative AI. The concept was initially developed by Ian Goodfellow and his colleagues in June 2014. In a GAN, two neural networks contest with each other in the form of a zero-sum game, where one agent's gain is another agent's loss. Given a training set, this technique learns to generate new data with the same statistics as the training set.