Data-driven Measurement Designs for Magnetic Resonance Imaging

Despite being a powerful medical imaging technique which does not emit any ionizing radiation, magnetic resonance imaging (MRI) always had the major problem of long scanning times that can take up to an hour depending on the application. It also requires uncomfortable breath-holds due to the slow acquisition, sedation of children and repeated scans in the cases of degraded image quality due to body motion.

Recent years have seen new image reconstruction techniques that need less amount of acquired data (i.e., accelerated scans) for reconstructing MRI images: parallel imaging and compressed sensing (CS). Although much work has been done on the reconstruction side, there has been relatively less work on experimental design, i.e., which parts of the Fourier domain to acquire during the scan, an essential factor that considerably affects the performance of image reconstructions. The state-of-the-art experimental designs use random subsampling either based on parametric models or heuristical adaptive models. The requirement of extensive parameter tuning and the random nature of the performance render these methods impractical and unreliable for clinical use.

Can we systematically use the data acquired during past MRI scans for the design of accelerated scans with a reliable image quality? This problem is the focus of this thesis which proposes a data-driven scan design approach and training procedures which efficiently and effectively learn the structure inherent in the data, and accordingly, design the scans that directly acquire only the most relevant information during acquisition given an acceleration rate constraint. As a result, this boosts the performance of the existing state-of-the-art compressive sensing techniques on real-world datasets. Moreover, this approach provides strong theoretical guarantees by using tools from statistical learning theory.

The intensive computational training procedures of our approach are made feasible by large-scale implementations on a parallel computing cluster. In return, this approach avoids any dependence on parametric or heuristic models and provides a reliably consistent image reconstruction performance for accelerated scans. Our approach is flexible and capable of giving deterministic scan designs specific to the anatomy, to the acceleration rate in use, to the reconstruction algorithm and the scan settings such as static/dynamic and parallel imaging.

Apart from measurement designs for MRI, this thesis also considers the reconstruction problem. In particular, we focus on the inverse problems that involve a mixture of regularizers in the objective function, exploiting multiple structures at the same time. For these problems, we propose a reliable and systematic optimization framework and illustrate its effectiveness. Finally, in the last part of the thesis, we present a data-driven model and an optimization method for the design of nearly isometric, linear and dimensionality reducing embeddings.

From Classical to Unsupervised-Deep-Learning Methods for Solving Inverse Problems in Imaging.

Harshit Gupta

In this thesis, we propose new algorithms to solve inverse problems in the context of biomedical images. Due to ill-posedness, solving these problems require some prior knowledge of the statistics of the underlying images. The traditional algorithms, in the field, assume prior knowledge related to smoothness or sparsity of these images. Recently, they have been outperformed by the second generation algorithms which harness the power of neural networks to learn required statistics from training data. Even more recently, last generation deep-learning-based methods have emerged which require neither training nor training data. This thesis devises algorithms which progress through these generations. It extends these generations to novel formulations and applications while bringing more robustness. In parallel, it also progresses in terms of complexity, from proposing algorithms for problems with 1D data and an exact known forward model to the ones with 4D data and an unknown parametric forward model. We introduce five main contributions. The last three of them propose deep-learning-based latest-generation algorithms that require no prior training. 1) We develop algorithms to solve the continuous-domain formulation of inverse problems with both classical Tikhonov and total-variation regularizations. We formalize the problems, characterize the solution set, and devise numerical approaches to find the solutions. 2) We propose an algorithm that improves upon end-to-end neural-network-based second generation algorithms. In our method, a neural network is first trained as a projector on a training set, and is then plugged in as a projector inside the projected gradient descent (PGD). Since the problem is nonconvex, we relax the PGD to ensure convergence to a local minimum under some constraints. This method outperforms all the previous generation algorithms for Computed Tomography (CT). 3) We develop a novel time-dependent deep-image-prior algorithm for modalities that involve a temporal sequence of images. We parameterize them as the output of an untrained neural network fed with a sequence of latent variables. To impose temporal directionality, the latent variables are assumed to lie on a 1D manifold. The network is then tuned to minimize the data fidelity. We obtain state-of-the-art results in dynamic magnetic resonance imaging (MRI) and even recover intra-frame images. 4) We propose a novel reconstruction paradigm for cryo-electron-microscopy (CryoEM) called CryoGAN. Motivated by generative adversarial networks (GANs), we reconstruct a biomolecule's 3D structure such that its CryoEM measurements resemble the acquired data in a distributional sense. The algorithm is pose-or-likelihood-estimation-free, needs no ab initio, and is proven to have a theoretical guarantee of recovery of the true structure. 5) We extend CryoGAN to reconstruct continuously varying conformations of a structure from heterogeneous data. We parameterize the conformations as the output of a neural network fed with latent variables on a low-dimensional manifold. The method is shown to recover continuous protein conformations and their energy landscape.

EPFL2020

Reconstruction Methods for Cryo-Electron Microscopy: From Model-based to Data-driven

Laurène Donati

The topic of this thesis is the development of new reconstruction methods for cryo-electron microscopy (cryo-EM). Cryo-EM has revolutionized the field of structural biology over the last decade and now permits the regular discovery of biostructures. Yet, the technical challenges associated to cryo-EM are still numerous, and the measurements remain notoriously difficult to process. This calls for fast and robust algorithms that can reliably handle the challenging reconstruction task at hand. In this thesis, we investigated two reconstruction paradigms: model-based and data-driven. Model-based methods formulate the reconstruction task as an inverse problem and rely on a faithful model of the acquisition physics. By contrast, the central philosophy of data-driven approaches is to let the reconstruction algorithm be guided by the measured data through some learning procedure. Both paradigms share a tight link in all our works: their reliance on a rigorous mathematical formulation of the cryo-EM imaging model. The first cryo-EM method we considered is scanning transmission electron tomography (STET), a modality whose primary concern is to reduce the electron dosage required for accurate imaging. To handle this, we developed a tailored acquisition-reconstruction STET framework that relies on the principles of compressed sensing. This scheme permits high-quality reconstruction from a reduced number of measurements, hence greatly preserving the sample. We then designed several reconstruction algorithms for single-particle analysis (SPA), a popular cryo-EM method that enables the determination of structures at near-atomic resolution. A key challenge for the deployment of robust, iterative reconstruction methods in SPA is that they usually come with a prohibitive computational cost if not carefully engineered. To circumvent this problem, we developed a regularized reconstruction scheme whose cost-dominant operation is recast as a discrete convolution, which makes the use of our robust scheme feasible in SPA. Building on this development, we devised a joint optimization framework that efficiently alternates between the reconstruction and the estimation of the unknown orientations. We then explored a learning-based method to estimate the unknown orientations in SPA directly from the acquired dataset of projections. Capitalizing on our ability to model the cryo-EM procedure, we generated large synthetic SPA datasets to train a function---parametrized as a neural network---to predict the relative orientation between two projections based on their similarity. The framework relies on the postulate that it is possible to recover, from these estimated orientation distances, the orientations themselves through an appropriate minimization scheme, as supported by preliminary tests. Finally, we developed a completely new paradigm for SPA reconstruction that leverages the remarkable capability of deep neural networks to capture data distribution. The proposed algorithm uses a generative adversarial network to learn the 3D structure that has simulated projections that most closely match the real data in a distributional sense. By doing so, it can resolve a 3D structure in a single algorithmic run using only the dataset of projections and CTF estimations as inputs. Hence, it bypasses many processing steps that are necessary in the usual cryo-EM reconstruction pipeline, which opens new perspectives for reconstruction in SPA.

EPFL2020

Guest Editorial: Wavelets in Medical Imaging

Michaël Unser

I. Introduction Wavelets are the result of collective efforts that recognized common threads between ideas and concepts that had been independently developed and investigated by distinct research communities. They provide a unifying framework for decomposing images, volumes, and time-series data into their elementary constituents across scale. Although a relatively recent construct, wavelets have become a tool of choice for engineers, physicists, and mathematicians, leading to efficient solutions in time and space frequency analysis problems, as well as a multitude of other applications. One of the consequences is that wavelet methods of analysis and representation are presently having a significant impact on the science of medical imaging and the diagnosis of disease and screening protocols. Because of a powerful underlying mathematical theory, they offer exciting opportunities for the design of new multiresolution image processing algorithms, and novel acquisition methods such as wavelet-encoded magnetic resonance imaging (MRI). This special issue of the IEEE Transactions on Medical Imaging focuses on these recent developments and highlights progress that has been accomplished in the areas related to medical imaging. II. Sizing the Wave: Some Facts and Figures Rarely has a mathematical concept generated so much response and enthusiasm within and between the engineering and mathematical research communities at large. To give a rough idea of the phenomenon, we provide a brief chronology. While wavelets have been traced all the way back to Alfred Haar in 1910 [1], for many, the starting point of their modern history coincides with two publications in the late 1980s by S. Mallat [2] and I. Daubechies [3]. These groundbreaking papers established a solid mathematical footing which would both shape and define the field. In a nutshell, S. Mallat identified the important concept of multiresolution analysis which is the corner stone of modern wavelet theory, while I. Daubechies constructed the first orthogonal wavelet bases that were compactly supported. These two contributions count among the most cited papers in the scientific literature (over 1500 SCI citations each). From that point on, the number of contributions relating to wavelet-applications and theory has increased steadily on the order of 9000 journal papers published to date. This trend is likely to continue as suggested by the strong response to the call for papers for this special issue (over 30 submissions). Wavelets have become so popular that distinct communities continue to have conferences and scientific journals entirely devoted to them. There are already historical anecdotes and folklore associated with them; an entertaining account of which can be found in the book of B. Burke Hubbard [4]. Readers who want to dive deeper into the subject have the daunting task of choosing among over 200 books written on wavelets. Our only advice in this regard is: in case of doubt, stick with the classics. Given the size of the phenomenon, it is no surprise that wavelets have had an impact on a number of disciplines, medical imaging being no exception. A first record of activity in this particular area is the workshop on wavelets in medicine and biology that took place at the annual IEEE-EMBS meeting, Baltimore, MD, 1992. The first journal paper describing a wavelet application in medical imaging—noise reduction in MRI by soft-thresholding in the wavelet domain—also appeared in 1992 [5]. Note that this work, which is often overlooked, provided the earliest description of a wavelet denoising method that has become extremely popular through the impulsion of Donoho et al. A large palette of wavelet applications in medical imaging is provided in [6]. Two complementary review articles are also available; the first gives a complete account of the activity taking place from the beginning to 1996 [7], while the second covers the more recent papers until 2000 [8]. So far, the primary applications of wavelets in medical imaging have been the following: Compression of medical images. CT reconstruction; local tomography. Wavelet denoising (MRI, ultrasound). Wavelet-based feature extraction; texture and statistical descriptors Medical image enhancement (e.g. fluoroscopy, mammography). Analysis of functional images of the brain [positron emission tomography (PET), functional MRI (fMRI)]. Wavelet-encoded MRI: Most of these topics are still active areas of research, as illustrated by the papers that are published in this special issue. III. Scanning Through the Issue Functional imaging is an area were methods of wavelet processing hold great promise. This particular line of research was initiated by U. Ruttimann, a creative researcher and good friend, who sadly passed away shortly before the publication of his paper in this very journal [9]. Another first rate statistician who was also active in this area at an early stage is J. Raz. By a sad coincidence, he also suffered a sudden death about a week before he was to present his latest results on wavelet analysis of fMRI [10]. Despite the tragic loss of these two pioneers, research in this area is alive and well as exemplified by the first three papers of this issue. Turkheimer et al. [11] consider the problem of the analysis of dynamic PET data; in particular, they advocate the use of a linear (James-Stein) wavelet estimator as an alternative to the more classical wavelet shrinkage or threshold detectors. Hossein-Zadeh et al. propose a wavelet-technique for the detection of activation in fMRI data [12]. Their contribution is twofold: first, the use of a redundant wavelet transform for better translation invariance, and second, a nonparametric detection method based on a randomization procedure. F. Meyer also considers fMRI time series but applies wavelets differently, within the context of a generalized linear model, to detrend the data [13]; that is, to get rid of signal drifts and disturbances that are not related to the stimulus. Two other areas where wavelets have achieved great success is signal denoising (typically, by simple thresholding in the wavelet domain) and tomographic reconstruction, mainly, because the Radon operator is well localized in a wavelet basis. Pizurica et al. [14] propose a novel wavelet denoising method that uses a local statistical model for improved signal estimation and noise suppression. Willett and Nowak [15] introduce a new multiscale image model, using piecewise planar basis functions, and apply their method to the reconstruction of photon-limited data (with Poisson noise). They develop penalized maximum likelihood methods for image denoising, deconvolution, and tomographic reconstruction. Kalifa et al. present a direct method for the efficient reconstruction of PET and single photom emission computed tomography data [16]. Their approach includes a nonlinear noise reduction step that is implemented by thresholding in the transformed domain; the key here is to select a transform (wavelet packet) that is optimized for the problem and data at hand (sparse representation of the signal and near diagonalization of the Radon operator). Bonnet et al. [17] also develop a direct approach for the reconstruction of cone-beam data which is known to be challenging. In essence, their approach is a wavelet adaptation of the Feldkamp algorithm. The special issue also features two contributions relating to ultrasound imaging. Michailovich and Adam consider the problem of the estimation of the spectrum of a ultrasound pulse [18]. Specifically, they develop a modified (outlier-resistant) wavelet estimator that they apply to the log-spectrum of the radio-freqency sequence. Lee et al. present a pattern recognition system that uses an M-band wavelet filterbank to extract fractal and texture features from ultrasonic images of the liver [19]. They report promising classification results, differientiating normal liver, cirrhosis, and hepatoma using a hierarchical classifier. The recent development of commercial digital mammography imaging systems not only provides a significant improvement in image quality for traditional screening, but translates into a wealth of information for the analysis and detection of mammographic features by computer. The papers by Lemaur et al. [20]. and Heinlein et al. [21] focus on the goals of early detection and visual enhancement of microcalcifications, respectively. In the former, the regularity of a wavelet basis is used to identify microcalcification in clusters. The identification of microcalcifaction in clusters as opposed to individual occurrences is of clinical significance as clusters may suggest the likelihood of malignancy. In the later paper, a discretization of the continuous wavelet transform is developed which allows a filterbank to be adapted for the enhancement of mammographic features. This implementation allows for the reconstruction of modified wavelet coefficients at arbitrary scales and orientations without the introduction of artifacts or loss of completeness. The integration of such an interactive enhancement tool into digital mammographic screening systems will be of great importance as the wealth of dynamic range (contrast) provided by digital detectors become generally available to radiologist through the introduction of lower cost softcopy display systems. The paper of Davatzikos et al. [22] offers another illustration of the versatility of wavelets. Their proposal is to represent the contours of a shape in a wavelet bases and to use this sparse representation to derive active shape models. Their results are promising and significant in terms of providing an automated solution to problems in volume quantification. The amount of data generated by modern imaging devices is often very large, and ever increasing. Thus, an important problem is to find efficient ways of compressing and encoding this information to facilitate its transmission, storage, and retrieval. Since wavelet transforms provide sparse representations of signals, one of their privileged areas of application has been coding and compression. In fact, wavelets have had so much success in this area that they have already become an integral part of the new JPEG2000 compression standard. Unlike other applications, however, biomedical image compression raises delicate issues; lossy compression must be done carefully so as to preserve all medically relevant information while eventually suppressing irrelevant features such as noise or background. Thus, research in this area must address these specific needs and pay great attention to the issue of validation. The paper by Menegaz and Thiran [23] uses wavelet transform techniques for encoding 3D volumetric data sets to optimally compress the volumetric images while allowing efficient two-dimensional slices reconstruction. Schelkens et al. [24] propose new wavelet methods for coding volumetric medical datasets, and compares them to state-of-the-art compression techniques. The new methods perform favorably for both lossy and lossless compression. The paper of Z. Xiong et al. [25] also proposes a new method of wavelet compression that is competitive with other state-of-the-art compression techniques but suggests improved image quality. IV. Conclusion After having completed the tasks of putting this issue together and based on our collective knowledge in the field, it is our belief that wavelets are here to stay. The idea of decomposing a signal according to scale is as fundamental a notion for imaging as decomposing it into harmonic components (Fourier analysis); it is more intuitive and closer to what the visual system does. Another important point that has stimulated wavelet research is the quest for sparse representations of signals, as these offer obvious advantages for data compression, noise reduction and regularization purposes. As more progress is made in the years ahead, it is quite possible that the importance of wavelets as a research topic in its own right may diminish and that they will progressively reach the status of standard toolbox components for data processing (such as the fast Fourier transform). Indeed, we are seeing some of this maturity in the development of emerging commercial image analysis software and packages. Yet, given the flexibility of this powerful analysis tool, there is not much risk in betting that wavelets will continue to be widely used in a diversity of medical imaging applications. A beneficial side effect of all the interdisciplinary activity taking place around wavelets has been to create a common language between mathematicians and engineers and to instill rigor in the formulation of imaging problems. Wavelets have been a cross-cutting magnet encouraging engineers to use more sophisticated mathematics and mathematicians to consider applying their knowledge to real world problems. As Guest Editors, we hope that the papers in this issue will stimulate further progress in this direction. We believe that the best is yet to come. We are particularly grateful to the authors for their timely contributions and to the reviewers who responded quickly with insightful and thorough comments. We would also like to thank K. Escher for her editorial assistance and patience. Finally, we want to close by dedicating this special issue to U. Ruttimann and J. Raz. References A. Haar, "Zur Theorie der Orthogonalen Funktionensysteme," Annals of Mathematics, vol. 69, pp. 331-371, 1910. S.G. Mallat, "A Theory of Multiresolution Signal Decomposition: The Wavelet Representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 7, pp. 674-693, July 1989. I. Daubechies, "Orthogonal Bases of Compactly Supported Wavelets," Communications on Pure and Applied Mathematics, vol. 41, pp. 909-996, November 1988. B.B. Hubbard, The World According to Wavelets: The Story of a Mathematical Technique in the Making. A.K. Peters Ltd., Wellesley MA, USA, 286 p., 1998. J.B. Weaver, X. Yansun, D.M. Healy, Jr., L.D. Cromwell, "Filtering Noise from Images with Wavelet Transforms," Magnetic Resonance in Medicine, vol. 21, no. 2, pp. 288-295, 1991. A. Aldroubi, M.A. Unser, Wavelets in Medicine and Biology. CRC Press, Boca Raton FL, USA, 616 p., 1996. M. Unser, A. Aldroubi, "A Review of Wavelets in Biomedical Applications," Proceedings of the IEEE, vol. 84, no. 4, pp. 626-638, April 1996. A. Laine, "Wavelets in Temporal and Spatial Processing of Biomedical Images," Annual Review in Biomedical Engineering, vol. 2, pp. 511-550, 2000. U.E. Ruttimann, M. Unser, R.R. Rawlings, D. Rio, N.F. Ramsey, V.S. Mattay, D.W. Hommer, J.A. Frank, D.R. Weinberger, "Statistical Analysis of Functional MRI Data in the Wavelet Domain," IEEE Transactions on Medical Imaging, vol. 17, no. 2, pp. 142-154, April 1998. J. Raz, "FDR Wavelet Shrinkage Estimators for Inhomogeneous Poisson Processes," Proceedings of the SPIE Conference on Mathematical Imaging: Wavelet Applications in Signal and Image Processing VIII, San Diego CA, USA, July 31-August 4, 2000, vol. 4119, pp. . F.E. Turkheimer, J.A.D. Aston, R.B. Banati, C. Riddell, V.J. Cunningham , "A Linear Wavelet Filter for Parametric Imaging with Dynamic PET," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 289-301, March 2003. G.-A. Hossien-Zadeh, H. Soltanian-Zadeh, B.A. Ardekani, "Multiresolution fMRI Activation Detection Using Translation Invariant Wavelet Transform and Statistical Analysis Based on Resampling," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 302-314, March 2003. F.G. Meyer, "Wavelet Based Estimation of a Semi Parametric Generalized Linear Model of fMRI Time-Series," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 315-322, March 2003. A. Pizurica, W. Philips, "A Versatile Wavelet Domain Noise Filtration Technique for Medical Imaging," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 323-331, March 2003. R. Willett, R. Nowak, "Platelets: A Multiscale Approach for Recovering Edges and Surfaces in Photon Limited Medical Imaging," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 332-350, March 2003. J. Kalifa, A. Laine, P.D. Esser, "Regularization in Tomographic Reconstruction Using Thresholding Estimators," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 351-359, March 2003. S. Bonnet, F. Peyrin, F. Turjman, R. Prost, "Nonseparable Wavelet-Based Cone-Beam Reconstruction in 3-D Rotational Angiography," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 360-367, March 2003. O. Michailovich, D. Adam, "Robust Estimation of Ultrasound Pulses Using Outlier-Resistant De-Noising," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 368-381, March 2003. W.-L. Lee, Y.-C. Chen, K.-S. Hsei, "Ultrasonic Liver Tissues Classification by Fractal Feature Vector Based on M-Band Wavelet Transform," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 382-392, March 2003. G. Lemaur, K. Drouiche, J. DeConinck, "Highly Regular Wavelets for the Detection of Clustered Microcalcifications in Mammograms," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 393-401, March 2003. P. Heinlein, J. Drexl, W. Schneider, "Integrated Wavelets for Enhancement of Microcalcifications in Digital Mammography," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 402-413, March 2003. C. Davatzikos, X. Tao, D. Shen , "Hierarchical Active Shape Models, Using the Wavelet Transform," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 414-423, March 2003. G. Menegaz, J.-P. Thiran, "3D Encoding/2D Decoding of Medical Data," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 424-440, March 2003. P. Schelkens, A. Munteanu, J. Barbarien, M. Galca, X.G.I. Nieto, J. Cornelis, "Wavelet Coding of Volumetric Medical Datasets," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 441-458, March 2003. Z. Xiong, X. Wu, S. Cheng, J. Hua, "Lossy-To-Lossless Compression of Medical Volumetric Data Using Three-Dimensional Integer Wavelet Transforms," IEEE Transactions on Medical Imaging, vol. 22, no. 3, pp. 459-470, March 2003.

2003

From Classical to Unsupervised-Deep-Learning Methods for Solving Inverse Problems in Imaging.

Harshit Gupta

EPFL2020

Guest Editorial: Wavelets in Medical Imaging

Michaël Unser

2003

Reconstruction Methods for Cryo-Electron Microscopy: From Model-based to Data-driven

Laurène Donati

EPFL2020