Adversarial Training: Optimization and Applications

This lecture covers the optimization problem of adversarial training, focusing on non-smooth activation functions and the structure of neural networks. It also explores the practical implementation of stochastic adversarial training and its application in improving interpretability and fairness in machine learning. The lecture delves into the concepts of directional derivatives, Wasserstein distance, and neural network distances inspired by the 1-Wasserstein distance. Additionally, it discusses the formulation of Wasserstein GANs and practical implementation strategies for training Wasserstein GANs.

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related lectures (33)

Related concepts (55)

EE-556: Mathematics of data: from theory to computation

This course provides an overview of key advances in continuous optimization and statistical analysis for machine learning. We review recent learning formulations and models as well as their guarantees

Activation function

Activation function of a node in an artificial neural network is a function that calculates the output of the node (based on its inputs and the weights on individual inputs). Nontrivial problems can be solved only using a nonlinear activation function. Modern activation functions include the smooth version of the ReLU, the GELU, which was used in the 2018 BERT model, the logistic (sigmoid) function used in the 2012 speech recognition model developed by Hinton et al, the ReLU used in the 2012 AlexNet computer vision model and in the 2015 ResNet model.

Softmax function

The softmax function, also known as softargmax or normalized exponential function, converts a vector of K real numbers into a probability distribution of K possible outcomes. It is a generalization of the logistic function to multiple dimensions, and used in multinomial logistic regression. The softmax function is often used as the last activation function of a neural network to normalize the output of a network to a probability distribution over predicted output classes, based on Luce's choice axiom.

Rectifier (neural networks)

In the context of artificial neural networks, the rectifier or ReLU (rectified linear unit) activation function is an activation function defined as the positive part of its argument: where x is the input to a neuron. This is also known as a ramp function and is analogous to half-wave rectification in electrical engineering. This activation function was introduced by Kunihiko Fukushima in 1969 in the context of visual feature extraction in hierarchical neural networks.

Sigmoid function

A sigmoid function is a mathematical function having a characteristic "S"-shaped curve or sigmoid curve. A common example of a sigmoid function is the logistic function shown in the first figure and defined by the formula: Other standard sigmoid functions are given in the Examples section. In some fields, most notably in the context of artificial neural networks, the term "sigmoid function" is used as an alias for the logistic function.

Online machine learning

In computer science, online machine learning is a method of machine learning in which data becomes available in a sequential order and is used to update the best predictor for future data at each step, as opposed to batch learning techniques which generate the best predictor by learning on the entire training data set at once. Online learning is a common technique used in areas of machine learning where it is computationally infeasible to train over the entire dataset, requiring the need of out-of-core algorithms.

Kernel Methods: Neural Networks PHYS-467: Machine learning for physicists

Covers the fundamentals of neural networks, focusing on RBF kernels and SVM.

Neural Networks: Deep Neural Networks MATH-212: Analyse numérique et optimisation

Explores the basics of neural networks, with a focus on deep neural networks and their architecture and training.

Neural Networks: Training and Optimization ME-390: Foundations of artificial intelligence

Explores the training and optimization of neural networks, addressing challenges like non-convex loss functions and local minima.

Document Analysis: Topic Modeling DH-406: Machine learning for DH

Explores document analysis, topic modeling, and generative models for data generation in machine learning.

Neural Networks: Training and Optimization

Explores neural network training, optimization, and environmental considerations, with insights into PCA and K-means clustering.