Multiplicative weight update method

Summary

The multiplicative weights update method is an algorithmic technique most commonly used for decision making and prediction, and also widely deployed in game theory and algorithm design. The simplest use case is the problem of prediction from expert advice, in which a decision maker needs to iteratively decide on an expert whose advice to follow. The method assigns initial weights to the experts (usually identical initial weights), and updates these weights multiplicatively and iteratively according to the feedback of how well an expert performed: reducing it in case of poor performance, and increasing it otherwise. It was discovered repeatedly in very diverse fields such as machine learning (AdaBoost, Winnow, Hedge), optimization (solving linear programs), theoretical computer science (devising fast algorithm for LPs and SDPs), and game theory. Name "Multiplicative weights" implies the iterative rule used in algorithms derived from the multiplicative weight update method. It

Official source

https://en.wikipedia.org/wiki/Multiplicative_weight_update_method

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (7)

Related publications

Related people

Related units

Related concepts

Related courses (2)

Related courses

Related lectures (3)

Related lectures

Weight Initialization for High Order and Multilayer Perceptrons

Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than

200,000

simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than

20,000

simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals.

1994

, ,

Learning in the brain is poorly understood and learning rules that respect biological constraints, yet yield deep hierarchical representations, are still unknown. Here, we propose a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. Learning minimizes a simple layer-specific loss function and does not need to back-propagate error signals within or between layers. Instead, weight updates follow a local, Hebbian, learning rule that only depends on pre-and post-synaptic neuronal activity, predictive dendritic input and widely broadcasted modulation factors which are identical for large groups of neurons. The learning rule applies contrastive predictive learning to a causal, biological setting using saccades (ie rapid shifts in gaze direction). We find that networks trained with this self-supervised and local rule build deep hierarchical representations of images, speech and video.

2021

High Order and Multilayer Perceptron Initialization

Proper initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. This publication aims at determining the optimal variance (or range) for the initial weights and biases, which is the principal parameter of random initialization methods for both types of neural networks. An overview of random weight initialization methods for multilayer perceptrons is presented. These methods are extensively tested using eight real- world benchmark data sets and a broad range of initial weight variances by means of more than

30,000

simulations, in the aim to find the best weight initialization method for multilayer perceptrons. For high order networks, a large number of experiments (more than

200,000

simulations) was performed, using three weight distributions, three activation functions, several network orders, and the same eight data sets. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable initialization method for high order perceptrons. The conclusions on the initialization methods for both types of networks are justified by sufficiently small confidence intervals of the mean convergence times.

IEEE1997

Weight Initialization for High Order and Multilayer Perceptrons

200,000

20,000

1994

High Order and Multilayer Perceptron Initialization

30,000

simulations, in the aim to find the best weight initialization method for multilayer perceptrons. For high order networks, a large number of experiments (more than

200,000

IEEE1997