High Order and Multilayer Perceptron Initialization

Proper initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. This publication aims at determining the optimal variance (or range) for the initial weights and biases, which is the principal parameter of random initialization methods for both types of neural networks. An overview of random weight initialization methods for multilayer perceptrons is presented. These methods are extensively tested using eight real- world benchmark data sets and a broad range of initial weight variances by means of more than $30,000$ simulations, in the aim to find the best weight initialization method for multilayer perceptrons. For high order networks, a large number of experiments (more than $200,000$ simulations) was performed, using three weight distributions, three activation functions, several network orders, and the same eight data sets. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable initialization method for high order perceptrons. The conclusions on the initialization methods for both types of networks are justified by sufficiently small confidence intervals of the mean convergence times.

Optimal Setting of Weights, Learning Rate, and Gain

The optimal setting of the initial weights, learning rate, and gain of the activation function, which are key parameters of a neural network, influencing training time and generalization performance, are investigated by means of a large number of experiments using ten benchmarks using high order perceptrons. The results are used to illustrate the influence of these key parameters on the training time and generalization performance and permit general conclusions to be drawn on the behavior of high order perceptrons, some of which can be extended to the behavior of multilayer perceptrons. Furthermore, optimal values for the learning rate and the gain of the activation function are found and compared to those recommended by existing heuristics.

IDIAP1997

Weight Initialization for High Order and Multilayer Perceptrons

Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than

200,000

simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than

20,000

simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals.

1994

Two neural network construction methods

Two low complexity methods for neural network construction, that are applicable to various neural network models, are introduced and evaluated for high order perceptrons. The methods are based on a Boolean approximation of real-valued data. This approximation is used to construct an initial neural network topology which is subsequently trained on the original (real-valued) data. The methods are evaluated for their effectiveness in reducing the network size and increasing the network's generalization capabilities in comparison to fully connected high order perceptrons.

1997

Weight Initialization for High Order and Multilayer Perceptrons

200,000

20,000

1994