Prior probability | EPFL Graph Search

A prior probability distribution of an uncertain quantity, often simply called the prior, is its assumed probability distribution before some evidence is taken into account. For example, the prior could be the probability distribution representing the relative proportions of voters who will vote for a particular politician in a future election. The unknown quantity may be a parameter of the model or a latent variable rather than an observable variable. In Bayesian statistics, Bayes' rule prescribes how to update the prior with new information to obtain the posterior probability distribution, which is the conditional distribution of the uncertain quantity given new data. Historically, the choice of priors was often constrained to a conjugate family of a given likelihood function, for that it would result in a tractable posterior of the same family. The widespread availability of Markov chain Monte Carlo methods, however, has made this less of a concern. There are many ways to constr

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Konstantinos Pitas

Deep neural networks have been empirically successful in a variety of tasks, however their theoretical understanding is still poor. In particular, modern deep neural networks have many more parameters than training data. Thus, in principle they should overfit the training samples and exhibit poor generalization to the complete data distribution. Counter intuitively however, they manage to achieve both high training accuracy and high testing accuracy. One can prove generalization using a validation set, however this can be difficult when training samples are limited and at the same time we do not obtain any information about why deep neural networks generalize well. Another approach is to estimate the complexity of the deep neural network. The hypothesis is that if a network with high training accuracy has high complexity it will have memorized the data, while if it has low complexity it will have learned generalizable patterns. In the first part of this thesis we explore Spectral Complexity, a measure of complexity that depends on combinations of norms of the weight matrices of the deep neural network. For a dataset that is difficult to classify, with no underlying model and/or no recurring pattern, for example one where the labels have been chosen randomly, spectral complexity has a large value, reflecting that the network needs to memorize the labels, and will not generalize well. Putting back the real labels, the spectral complexity becomes lower reflecting that some structure is present and the network has learned patterns that might generalize to unseen data. Spectral complexity results in vacuous estimates of the generalization error (the difference between the training and testing accuracy), and we show that it can lead to counterintuitive results when comparing the generalization error of different architectures. In the second part of the thesis we explore non-vacuous estimates of the generalization error. In Chapter 2 we analyze the case of PAC-Bayes where a posterior distribution over the weights of a deep neural network is learned using stochastic variational inference, and the generalization error is the KL divergence between this posterior and a prior distribution. We find that a common approximation where the posterior is constrained to be Gaussian with diagonal covariance, known as the mean-field approximation, limits significantly any gains in bound tightness. We find that, if we choose the prior mean to be the random network initialization, the generalization error estimate tightens significantly. In Chapter 3 we explore an existing approach to learning the prior mean, in PAC-Bayes, from the training set. Specifically, we explore differential privacy, which ensures that the training samples contribute only a limited amount of information to the prior, making it distribution and not training set dependent. In this way the prior should generalize well to unseen data (as it hasn't memorized individual samples) and at the same time any posterior distribution that is close to it in terms of the KL divergence will also exhibit good generalization.

EPFL2020

Morphology-driven automatic segmentation of MR images of the neonatal brain

Laura Ioana Gui, Michel Kocher, François Lazeyras, Radoslaw Lisowski

The segmentation of MR images of the neonatal brain is an essential step in the study and evaluation of infant brain development. State-of-the-art methods for adult brain MRI segmentation are not applicable to the neonatal brain, due to large differences in structure and tissue properties between newborn and adult brains. Existing newborn brain MRI segmentation methods either rely on manual interaction or require the use of atlases or templates, which unavoidably introduces a bias of the results towards the population that was used to derive the atlases. We propose a different approach for the segmentation of neonatal brain MRI, based on the infusion of high-level brain morphology knowledge, regarding relative tissue location, connectivity and structure. Our method does not require manual interaction, or the use of an atlas, and the generality of its priors makes it applicable to different neonatal populations, while avoiding atlas-related bias. The proposed algorithm segments the brain both globally (intracranial cavity, cerebellum, brainstem and the two hemispheres) and at tissue level (cortical and subcortical gray matter, myelinated and unmyelinated white matter, and cerebrospinal fluid). We validate our algorithm through visual inspection by medical experts, as well as by quantitative comparisons that demonstrate good agreement with expert manual segmentations. The algorithm's robustness is verified by testing on variable quality images acquired on different machines, and on subjects with variable anatomy (enlarged ventricles, preterm- vs. term-born). (C) 2012 Elsevier B.V. All rights reserved.

Elsevier Science Bv2012

On Vacuous and Non-Vacuous Generalization Bounds for Deep Neural Networks.

Konstantinos Pitas

EPFL2020