Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals.
Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu