Benefiting from Multitask Learning to Improve Single Image Super-Resolution

Despite significant progress toward super resolving more realistic images by deeper convolutional neural networks (CNNs), reconstructing fine and natural textures still remains a challenging problem. Recent works on single image super resolution (SISR) are mostly based on optimizing pixel and content wise similarity between recovered and high-resolution (HR) images and do not benefit from recognizability of semantic classes. In this paper, we introduce a novel approach using categorical information to tackle the SISR problem; we present a decoder architecture able to extract and use semantic information to super-resolve a given image by using multitask learning, simultaneously for image super-resolution and semantic segmentation. To explore categorical information during training, the proposed decoder only employs one shared deep network for two task-specific output layers. At run-time only layers resulting HR image are used and no segmentation label is required. Extensive perceptual experiments and a user study on images randomly selected from COCO-Stuff dataset demonstrate the effectiveness of our proposed method and it outperforms the state-of-the-art methods.

SROBB: Targeted Perceptual Loss for Single Image Super-Resolution

Hazim Kemal Ekenel, Mohammad Saeed Rad, Jean-Philippe Thiran

By benefiting from perceptual losses, recent studies have improved significantly the performance of the super-resolution task, where a high-resolution image is resolved from its low-resolution counterpart. Although such objective functions generate near-photorealistic results, their capability is limited, since they estimate the reconstruction error for an entire image in the same way, without considering any semantic information. In this paper, we propose a novel method to benefit from perceptual loss in a more objective way. We optimize a deep network-based decoder with a targeted objective function that penalizes images at different semantic levels using the corresponding terms. In particular, the proposed method leverages our proposed OBB (Object, Background and Boundary) labels, generated from segmentation labels, to estimate a suitable perceptual loss for boundaries, while considering texture similarity for backgrounds. We show that our proposed approach results in more realistic textures and sharper edges, and outperforms other state-of-the-art algorithms in terms of both qualitative results on standard benchmarks and results of extensive user studies.

IEEE COMPUTER SOC2019

Context-Aware Image Super-Resolution Using Deep Neural Networks

Mohammad Saeed Rad

Image super-resolution is a classic ill-posed computer vision and image processing problem, addressing the question of how to reconstruct a high-resolution image from its low-resolution counterpart. Current state-of-the-art methods have improved the performance of the single image super-resolution task significantly by benefiting from machine learning and AI-powered algorithms, and more specifically, with the advent of Deep Learning-based approaches. Although these advances allow a machine to learn and have better exploitation of an image and its content, recent methods are still unable to constrain the plausible solution space based on the available contextual information within an image. This limitation mostly results in poor reconstructions, even for well-known types of objects and textures easily recognizable for humans. In this thesis, we aim at proving that the categorical prior, which characterizes the semantic class of a region in an image (e.g., sky, building, plant), is crucial in super-resolution task for reaching a higher reconstruction quality. In particular, we propose several approaches to improve the perceived image quality and generalization capability of deep learning-based methods by exploiting the context and semantic meaning of images. To prove the effectiveness of this categorical information, we first propose a convolutional neural network-based framework that is able to extract and use semantic information to super-resolve a given image by using multitask learning, simultaneously for learning image super-resolution and semantic segmentation. The proposed decoder is forced to explore categorical information during training, as this setting employs only one shared deep network for both semantic segmentation and super-resolution tasks. We further investigate the possibility of using semantic information by a novel objective function to introduce additional spatial control over the training process. We propose penalizing images at different semantic levels using appropriate loss terms by benefiting from our new OBB (Object, Background, and Boundary) labels generated from segmentation labels. Then, we introduce a new test time adaptation-based technique to leverage high-resolution images with perceptually similar context to a given test image to improve the reconstruction quality. We further validate this approach's effectiveness by using a novel numerical experiment analyzing the correlation between filters learned by our network and what we define as `ideal' filters. Finally, we present a generic solution to enable adapting all our previous contributions in this thesis, as well as other recent super-resolution works trained on synthetic datasets, to real-world super-resolution problem. Real-world super-resolution refers to super-resolving images with real degradations caused by physical imaging systems, instead of low-resolution images from simulated datasets assuming a simple and uniform degradation model (i.e., bicubic downsampling). We study and develop an image-to-image translator to map the distribution of real low-resolution images to the well-understood distribution of bicubically downsampled images. This translator is used as a plug-in to integrate real inputs into any super-resolution framework trained on simulated datasets. We carry out extensive qualitative and quantitative experiments for each mentioned contribution, including user studies, to compare our proposed approaches to state-of-the-art method.

EPFL2021

Central and Cortical Gray Mater Segmentation of Magnetic Resonance Images of the Fetal Brain

Anouk André, Meritxell Bach Cuadra, Gabriele Vincenzo Bonanno, Marie Schaer, Jean-Philippe Thiran

Motivation. The study of human brain development in its early stage is today possible thanks to in vivo fetal magnetic resonance imaging (MRI) techniques. A quantitative analysis of fetal cortical surface represents a new approach which can be used as a marker of the cerebral maturation (as gyration) and also for studying central nervous system pathologies [1]. However, this quantitative approach is a major challenge for several reasons. First, movement of the fetus inside the amniotic cavity requires very fast MRI sequences to minimize motion artifacts, resulting in a poor spatial resolution and/or lower SNR. Second, due to the ongoing myelination and cortical maturation, the appearance of the developing brain differs very much from the homogenous tissue types found in adults. Third, due to low resolution, fetal MR images considerably suffer of partial volume (PV) effect, sometimes in large areas. Today extensive efforts are made to deal with the reconstruction of high resolution 3D fetal volumes [2,3,4] to cope with intra-volume motion and low SNR. However, few studies exist related to the automated segmentation of MR fetal imaging. [5] and [6] work on the segmentation of specific areas of the fetal brain such as posterior fossa, brainstem or germinal matrix. First attempt for automated brain tissue segmentation has been presented in [7] and in our previous work [8]. Both methods apply the Expectation-Maximization Markov Random Field (EM-MRF) framework but contrary to [7] we do not need from any anatomical atlas prior. Data set & Methods. Prenatal MR imaging was performed with a 1-T system (GE Medical Systems, Milwaukee) using single shot fast spin echo (ssFSE) sequences (TR 7000 ms, TE 180 ms, FOV 40 x 40 cm, slice thickness 5.4mm, in plane spatial resolution 1.09mm). Each fetus has 6 axial volumes (around 15 slices per volume), each of them acquired in about 1 min. Each volume is shifted by 1 mm with respect to the previous one. Gestational age (GA) ranges from 29 to 32 weeks. Mother is under sedation. Each volume is manually segmented to extract fetal brain from surrounding maternal tissues. Then, in-homogeneity intensity correction is performed using [9] and linear intensity normalization is performed to have intensity values that range from 0 to 255. Note that due to intra-tissue variability of developing brain some intensity variability still remains. For each fetus, a high spatial resolution image of isotropic voxel size of 1.09 mm is created applying [2] and using B-splines for the scattered data interpolation [10] (see Fig. 1). Then, basal ganglia (BS) segmentation is performed on this super reconstructed volume. Active contour framework with a Level Set (LS) implementation is used. Our LS follows a slightly different formulation from well-known Chan-Vese [11] formulation. In our case, the LS evolves forcing the mean of the inside of the curve to be the mean intensity of basal ganglia. Moreover, we add local spatial prior through a probabilistic map created by fitting an ellipsoid onto the basal ganglia region. Some user interaction is needed to set the mean intensity of BG (green dots in Fig. 2) and the initial fitting points for the probabilistic prior map (blue points in Fig. 2). Once basal ganglia are removed from the image, brain tissue segmentation is performed as described in [8]. Results. The case study presented here has 29 weeks of GA. The high resolution reconstructed volume is presented in Fig. 1. The steps of BG segmentation are shown in Fig. 2. Overlap in comparison with manual segmentation is quantified by the Dice similarity index (DSI) equal to 0.829 (values above 0.7 are considered a very good agreement). Such BG segmentation has been applied on 3 other subjects ranging for 29 to 32 GA and the DSI has been of 0.856, 0.794 and 0.785. Our segmentation of the inner (red and blue contours) and outer cortical surface (green contour) is presented in Fig. 3. Finally, to refine the results we include our WM segmentation in the Freesurfer software [12] and some manual corrections to obtain Fig.4. Discussion. Precise cortical surface extraction of fetal brain is needed for quantitative studies of early human brain development. Our work combines the well known statistical classification framework with the active contour segmentation for central gray mater extraction. A main advantage of the presented procedure for fetal brain surface extraction is that we do not include any spatial prior coming from anatomical atlases. The results presented here are preliminary but promising. Our efforts are now in testing such approach on a wider range of gestational ages that we will include in the final version of this work and studying as well its generalization to different scanners and different type of MRI sequences. References. [1] Guibaud, Prenatal Diagnosis 29(4) (2009). [2] Rousseau, Acad. Rad. 13(9), 2006, [3] Jiang, IEEE TMI 2007. [4] Warfield IADB, MICCAI 2009. [5] Claude, IEEE Trans. Bio. Eng. 51(4) (2004). [6] Habas, MICCAI (Pt. 1) 2008. [7] Bertelsen, ISMRM 2009 [8] Bach Cuadra, IADB, MICCAI 2009. [9] Styner, IEEE TMI 19(39 (2000). [10] Lee, IEEE Trans. Visual. And Comp. Graph. 3(3), 1997, [11] Chan, IEEE Trans. Img. Proc, 10(2), 2001 [12] Freesurfer, http://surfer.nmr.mgh.harvard.edu.