Automatic differentiation | EPFL Graph Search

Are you an EPFL student looking for a semester project?

Work with us on data science and visualisation projects, and deploy your project as an app on top of GraphSearch.

In mathematics and computer algebra, automatic differentiation (auto-differentiation, autodiff, or AD), also called algorithmic differentiation, computational differentiation, is a set of techniques to evaluate the partial derivative of a function specified by a computer program. Automatic differentiation exploits the fact that every computer calculation, no matter how complicated, executes a sequence of elementary arithmetic operations (addition, subtraction, multiplication, division, etc.) and elementary functions (exp, log, sin, cos, etc.). By applying the chain rule repeatedly to these operations, partial derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor of more arithmetic operations than the original program. Automatic differentiation is distinct from symbolic differentiation and numerical differentiation. Symbolic differentiation faces the difficulty of converting a computer program into a single mathematical expression and can lead to inefficient code. Numerical differentiation (the method of finite differences) can introduce round-off errors in the discretization process and cancellation. Both of these classical methods have problems with calculating higher derivatives, where complexity and errors increase. Finally, both of these classical methods are slow at computing partial derivatives of a function with respect to many inputs, as is needed for gradient-based optimization algorithms. Automatic differentiation solves all of these problems. Fundamental to automatic differentiation is the decomposition of differentials provided by the chain rule of partial derivatives of composite functions. For the simple composition the chain rule gives Usually, two distinct modes of automatic differentiation are presented. forward accumulation (also called bottom-up, forward mode, or tangent mode) reverse accumulation (also called top-down, reverse mode, or adjoint mode) Forward accumulation specifies that one traverses the chain rule from inside to outside (that is, first compute and then and at last ), while reverse accumulation has the traversal from outside to inside (first compute and then and at last ).

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (7)

Related people (4)

Related concepts (10)

Related courses (5)

Related lectures (54)

MATH-520: Mathematics of machine learning

Mathematical analysis of modern supervised machine learning techniques, from linear methods to artificial neural networks.

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

EE-608: Deep Learning For Natural Language Processing

The Deep Learning for NLP course provides an overview of neural network based methods applied to text. The focus is on models particularly suited to the properties of human language, such as categori

PyTorch

PyTorch is a machine learning framework based on the Torch library, used for applications such as computer vision and natural language processing, originally developed by Meta AI and now part of the Linux Foundation umbrella. It is free and open-source software released under the modified BSD license. Although the Python interface is more polished and the primary focus of development, PyTorch also has a C++ interface. A number of pieces of deep learning software are built on top of PyTorch, including Tesla Autopilot, Uber's Pyro, Hugging Face's Transformers, PyTorch Lightning, and Catalyst.

Automatic differentiation

Deep learning

Deep learning is part of a broader family of machine learning methods, which is based on artificial neural networks with representation learning. The adjective "deep" in deep learning refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

Differentiable Physically Based Rendering: Algorithms, Systems and Applications

Merlin Eléazar Nimier-David

Physically based rendering methods can create photorealistic images by simulating the propagation and interaction of light in a virtual scene. Given a scene description including the shape of objects,

EPFL2022

Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering

Wenzel Alban Jakob, Delio Aleardo Vicini, Sébastien Nicolas Speierer, Nicolas Roussel

Dr.Jit is a new just-in-time compiler for physically based rendering and its derivative. Dr.Jit expedites research on these topics in two ways: first, it traces high-level simulation code (e.g., writt

ASSOC COMPUTING MACHINERY2022

Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time

Wenzel Alban Jakob, Delio Aleardo Vicini, Sébastien Nicolas Speierer

Differentiable physically-based rendering has become an indispensable tool for solving inverse problems involving light. Most applications in this area jointly optimize a large set of scene parameters

ASSOC COMPUTING MACHINERY2021

Machine Learning on Earth System with Remote Sensing

Explores machine learning applications in Earth system analysis using remote sensing data, focusing on automatic image interpretation and explainable AI.

Deep Learning Building Blocks EE-556: Mathematics of data: from theory to computation

Covers tensors, loss functions, autograd, and convolutional layers in deep learning.

BackProp Algorithm: Pseudocode and Processing Steps CS-456: Artificial neural networks/reinforcement learning

Covers the BackProp algorithm, including initialization, signal propagation, error computation, weight updating, and complexity comparison with numerical differentiation.