The impact of denoising on independent component analysis of functional magnetic resonance imaging data

Independent component analysis (ICA) is a suitable method for decomposing functional magnetic resonance imaging (fMRI) activity into spatially independent patterns. Practice has revealed that low-pass filtering prior to ICA may improve ICA results by reducing noise and possibly by increasing source smoothness, which may enhance source independence; however, it eliminates useful information in high frequency features and it amplifies low signal fluctuations leading to independence loss. On the other hand, high-pass filtering may increase the independence by preserving spatial information, but its denoising properties are weak. Thus, such filtering strategies did not lead to simultaneous enhancements in independence and noise reduction; therefore, band-pass filtering or more sophisticated filtering methods are expected to be more appropriate. We used advanced wavelet filtering procedures, such as wavelet-based methods relying upon hard and soft coefficient thresholding and non-stationary Gaussian modelling based on geometrical prior information, to denoise artificial and real fMRI data. We compared the performance of these methods with the performance of traditional Gaussian smoothing techniques. First, we demonstrated both analytically and empirically the consistent performance increase of spatial filtering prior to ICA using spatial correlation and statistical sensitivity as quality measures. Second, all filtering methods were computationally efficient. Finally, denoising using low-pass filters was needed to improve ICA, suggesting that noise reduction may have a more significant effect on the component independence than the preservation of information contained within high frequencies. (C) 2012 Elsevier B.V. All rights reserved.

Sparsity-Promoting Spatiotemporal Regularization for Data Mining in Functional Magnetic Resonance Imaging

Fikret Isik Karahanoglu

Magnetic resonance imaging (MRI) has opened unprecedented avenues to observe the human brain non-invasively. In particular, for about two decades, functional MRI (fMRI) has enabled to monitor brain function using the blood-oxygen-level-dependent (BOLD) contrast as a proxy for neuronal activity. The impact of fMRI on neurosciences, medicine, and psychology is ever increasing and has been mainly focussing on (1) understanding brain organization in terms of segregation (i.e., localized processing) and integration (i.e., distributed processing), specifically, related to sensory processing and cognition; (2) exploring temporal characteristics of brain processes. FMRI provides large spatiotemporal datasets, typically one whole-brain volume with spatial resolution of 1–3 mm in each dimension, every 1–4 seconds during several minutes. The structure of neurophysiological contributions in these data is complex and therefore requires advanced data processing. Conventional fMRI analysis is exploiting timing properties of a stimulation or task paradigm designed by the experimenter; i.e., evidence is sought for the presence of a hypothetical BOLD response. More recently, the community has shown increasing interest in spontaneous brain activity acquired during resting-state fMRI (RS-fMRI). In the absence of any task, data-driven or exploratory methods have found great use. In particular, blind source separation such as independent component analysis (ICA) has been widely applied to RS-fMRI data. One limitation of current data-driven methods is the lack of incorporating knowledge about the hemodynamic system, which governs any activity-related signal component in the fMRI measurements. In this dissertation, we build upon the latest advances in convex optimization and propose a novel framework that can reveal activity-inducing signals at the fMRI timescale. In particular, our regularization strategy, termed “total activation” (TA), allows deconvolving the fMRI signal to remove hemodynamic blur and to improve spatial contrast of activation patterns by incorporating knowledge about meaningful brain regions. The contribution of our method lies in adapting and tailoring state-of-the-art signal processing techniques with specific domain knowledge from fMRI and neurosciences. First, we extend “total variation” (TV), which is a well-recognized method in image processing for edge-preserving regularization. TV favors signals that are piecewise constant, and, therefore, whose derivatives are sparse. We generalize this concept for signals of which the derivative of an additional linear differential operator is sparse, and build a variational formulation for the denoising problem. The recovered signal can be also studied after applying the regularizing operators; e.g., applying the differential operator will lead to the piecewise constant driving signal, while applying an additional derivative reveals a sparse “innovation” signal. Fast and efficient schemes from convex optimization are deployed to solve the variational problem at hand. Second, we apply TA for fMRI data analysis to explore the underlying “deblurred” activity-inducing signals. The temporal regularization is based on generalization of TV where the differential operator is chosen to invert the (linearized) hemodynamic system. Consequently, this will favor block-type activity-inducing signals without restrictions on timing nor duration. The spatial regularization uses a mixed-norm regularization to favor coherent activity-inducing signals in brain regions chosen from an anatomical brain atlas. After demonstrating the feasibility of the proposed method using simulated data, we show results on experimental fMRI data consisting of long resting-state periods and a few unanticipated visual stimuli. The method is able to readily recover plausible activation patterns for the visual stimuli without any prior knowledge about their timing. More interestingly, we also recovered complex spatiotemporal patterns of spontaneous activity that were organized in “resting-state networks”, as such providing a new approach to study non-stationary dynamics of RS-fMRI, a research direction that will be of major interest in the coming years. Finally, we include convincing results for localizing epileptogenic brain regions in patients using simultaneous EEG-fMRI recordings.

EPFL2013

Efficient second-order methods for model compression

Sidak Pal Singh

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been a tremendous amount of work on utilizing this information for the current compute and memory-intensive deep neural networks, usually via coarse-grained approximations (such as diagonal, blockwise, or Kronecker-factorization). However, not much is known about the quality of these approximations. Our work addresses this question, and in particular, we propose a method called ‘WoodFisher’ that leverages the structure of the empirical Fisher information matrix, along with the Woodbury matrix identity, to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to the task of compressing neural networks, where we build on the classical Optimal Brain Damage/Surgeon framework (LeCun et al., 1990; Hassibi and Stork, 1993). We demonstrate that WoodFisher significantly outperforms magnitude pruning (isotropic Hessian), as well as methods that maintain other diagonal estimates. Further, even when gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for standard image classification datasets such as CIFAR-10, ImageNet. We also propose a variant called ‘WoodTaylor’, which takes into account the first-order gradient term, and can lead to additional improvements. An important advantage of our methods is that they allow us to automatically set the layer-wise pruning thresholds, avoiding the need for any manual tuning or sensitivity analysis.

2020

Feedback and Common Information: Bounds and Capacity for Gaussian Networks

Erixhen Sula

Network information theory studies the communication of information in a network and considers its fundamental limits. Motivating from the extensive presence of the networks in the daily life, the thesis studies the fundamental limits of particular networks including channel coding such as Gaussian multiple access channel with feedback and source coding such as lossy Gaussian Gray-Wyner network. On one part, we establish the sum-Capacity of the Gaussian multiple-access channel with feedback. The converse bounds that are derived from the dependence-balance argument of Hekstra and Willems meet the achievable scheme introduced by Kramer. Even though the problem is not convex, the factorization of lower convex envelope method that is introduced by Geng and Nair, combined with a Gaussian property are invoked to compute the sum-Capacity. Additionally, we characterize the rate region of lossy Gaussian Gray-Wyner network for symmetric distortion. The problem is not convex, thus the method of factorization of lower convex envelope is used to show the Gaussian optimality of the auxiliaries. Both of the networks, are a long-standing open problem. On the other part, we consider the common information that is introduced by Wyner and the natural relaxation of Wyner's common information. Wyner's common information is a measure that quantifies and assesses the commonality between two random variables. The operational significance of the newly introduced quantity is in Gray-Wyner network. Thus, computing the relaxed Wyner's common information is directly connected with computing the rate region in Gray-Wyner network. We derive a lower bound to Wyner's common information for any given source. The bound meets the exact Wyner's common information for sources that are expressed as sum of a common random variable and Gaussian noises. Moreover, we derive an upper bound on an extended variant of information bottleneck. Finally, we use Wyner's common information and its relaxation as a tool to extract common information between datasets. Thus, we introduce a novel procedure to construct features from data, referred to as Common Information Components Analysis (CICA). We establish that in the case of Gaussian statistics, CICA precisely reduces to Canonical Correlation Analysis (CCA), where the relaxing parameter determines the number of CCA components that are extracted. In this sense, we establish a novel rigorous connection between information measures and CCA, and CICA is a strict generalization of the latter. Moreover, we show that CICA has several desirable features, including a natural extension to beyond just two data sets.