Unsupervised multiscale image segmentation

In the early stages of visual information processing one of the most fundamental and complex task is segmentation. This process is used to divide images or image sequences into meaningful regions or objects. Such a representation is definitely needed to go from the first rough description of the scene (i.e. light intensity) up to the semantic level, where one is able to discriminate shapes and other high-level visual primitives. The biological visual system performs this extremely difficult task with such ease that is still a major challenge for computer vision to reach this level of speed and accuracy. It is by now widely believed that a method inspired by biological vision has the best chances of accomplishing this goal. Therefore, our main goal in this thesis is to step on biologically inspired schemes to propose new and efficient techniques to solve the image segmentation problem. The starting point and key concept in this research, is the multiresolution paradigm. Indeed, the fact that the Human Visual System (HVS) processes the information in a multiresolution fashion is now well established. Several techniques have been introduced to model this behaviour. Scale-space is one of them and we will thus focus on such a formulation of the segmentation problem. Such a representation is composed by a stack of successive versions of the original data set at coarser scales. It is assumed that, the bigger the scale, the less information referred to local characteristics of the input data will appear. We also impose that general information applying to large scales will last through scale. Taking that into account, it is reasonable to think that local and high resolution scale information can be related to general and low resolution information. This will enable us to extract image structures. We thus propose to link the multiresolution approach to a computational model based on the analysis of the entire range of significant information across scales. The inherent multiscale image structure of the non-linear scale-space guides a coarse to fine segmentation using different strategies. The first one might be termed extrinsic because the non-linear scale-space is analyzed by an external process of linking information through scales. This leads to a completely adaptive scheme that can then be coupled to a labeling procedure in order to define a genuine segmentation. The second one could be labeled intrinsic because it adapts to the image through its inner properties and incorporate this in the labeling step of the main algorithm. These two approaches are particular examples of multiscale image segmentation. We also apply the same scale-space concept to define a segmentation variational formulation that leads to a region merging segmentation. In order to illustrate the proposed segmentation schemes, we apply our algorithms to gray-scale, noisy, color and texture images. Finally, it should be mentioned that the main problem addressed in this thesis, general image segmentation, plays an important role in the image understanding process, especially those that involve natural scenes rich in color and texture. The potential applications of the algorithms are thus numerous: medical image analysis, interactive multimedia developments, object based coding and compression, surveillance applications, image content based browsing and retrieval, etc.

Graph-based image representation learning

Renata Khasanova

Though deep learning (DL) algorithms are very powerful for image processing tasks, they generally require a lot of data to reach their full potential. Furthermore, there is no straightforward way to impose various properties, given by the prior knowledge about the target task, on the features extracted by a DL model. Therefore, in this thesis we propose several techniques that rely on the power of graph representations to embed prior knowledge inside the learning process. This allows to reduce the solution space and leads to faster optimization convergence and higher accuracy in the representation learning. In our first work, inspired by the ability of a human to correctly classify rotated, shifted or flipped objects, we propose an algorithm that permits to inherently encode invariance to isometric transformations of objects in an image. Our DL architecture is based on graph representations and consists of three novel layers, which we refer to as graph convolutional, dynamic pooling and statistical layers. Our experiments on the image classification tasks show that our network correctly recognizes isometrically transformed objects even though such types of transformation are not seen by the network at training time. Standard DL techniques are typically not able to succeed in solving such a problem without extensive data augmentation. Then, we propose to exploit the properties of graph-based approaches to efficiently process images with various types of projective geometry. In particular, we are interested in increasingly popular omnidirectional cameras, which have a 360 degree field of view. Despite their effectiveness, such cameras create images with specific geometric properties, which require special techniques for efficient processing. We propose an efficient way of adjusting the weights of the graph edges to adapt the filter responses to the geometric image properties introduced by omnidirectional cameras. Our experiments prove that using the proposed graph with properly adjusted edge weights permits to reach better performance as compared to using regular grid graph with equal weights. Finally, the approach described above relies on the isotropic filters, which work well within our transformation invariant architecture for image classification. However, for other problems (e.g. image compression) or even when used without dynamic pooling and statistical layers that are defined within the proposed architecture, these filters are unable to efficiently encode the information about the object. Thus, we introduce a different technique based on anisotropic filters that adapt their shape and size according to the omnidirectional image geometry. The main advantage of this approach compared to the previous one is the ability to encode the orientation of an image pattern, which is important for various tasks such as image compression. Our experiments show that our approach adapts to different image projective geometries and achieves state-of-the-art performance on image classification and compression tasks. Overall we propose several methods, which combine the power of DL and graph signal processing towards incorporating prior information about the target task inside the optimization procedure. We hope that the research efforts presented in this thesis will help the development of efficient DL algorithms that can use various types of prior knowledge to make them efficient even when the available training data is scarce.

EPFL2019

Image segmentation with variational active contours

Xavier Bresson

An important branch of computer vision is image segmentation. Image segmentation aims at extracting meaningful objects lying in images either by dividing images into contiguous semantic regions, or by extracting one or more specific objects in images such as medical structures. The image segmentation task is in general very difficult to achieve since natural images are diverse, complex and the way we perceive them vary according to individuals. For more than a decade, a promising mathematical framework, based on variational models and partial differential equations, have been investigated to solve the image segmentation problem. This new approach benefits from well-established mathematical theories that allow people to analyze, understand and extend segmentation methods. Moreover, this framework is defined in a continuous setting which makes the proposed models independent with respect to the grid of digital images. This thesis proposes four new image segmentation models based on variational models and the active contours method. The active contours or snakes model is more and more used in image segmentation because it relies on solid mathematical properties and its numerical implementation uses the efficient level set method to track evolving contours. The first model defined in this dissertation proposes to determine global minimizers of the active contour/snake model. Despite of great theoretic properties, the active contours model suffers from the existence of local minima which makes the initial guess critical to get satisfactory results. We propose to couple the geodesic/geometric active contours model with the total variation functional and the Mumford-Shah functional to determine global minimizers of the snake model. It is interesting to notice that the merging of two well-known and "opposite" models of geodesic/geometric active contours, based on the detection of edges, and active contours without edges provides a global minimum to the image segmentation algorithm. The second model introduces a method that combines at the same time deterministic and statistical concepts. We define a non-parametric and non-supervised image classification model based on information theory and the shape gradient method. We show that this new segmentation model generalizes, in a conceptual way, many existing models based on active contours, statistical and information theoretic concepts such as mutual information. The third model defined in this thesis is a variational model that extracts in images objects of interest which geometric shape is given by the principal components analysis. The main interest of the proposed model is to combine the three families of active contours, based on the detection of edges, the segmentation of homogeneous regions and the integration of geometric shape prior, in order to use simultaneously the advantages of each family. Finally, the last model presents a generalization of the active contours model in scale spaces in order to extract structures at different scales of observation. The mathematical framework which allows us to define an evolution equation for active contours in scale spaces comes from string theory. This theory introduces a mathematical setting to process a manifold such as an active contour embedded in higher dimensional Riemannian spaces such as scale spaces. We thus define the energy functional and the evolution equation of the multiscale active contours model which can evolve in the most well-known scale spaces such as the linear or the curvature scale space.

EPFL2005

Crowding and the importance of grouping and segmentation processes in human vision

Alban Bornet

Human vision has evolved to make sense of a world in which elements almost never appear in isolation. Surprisingly, the recognition of an element in a visual scene is strongly limited by the presence of other nearby elements, a phenomenon known as visual crowding. Crowding impacts vision at all levels and is thus a versatile tool to understand the fundamental mechanisms of vision. For decades, visual crowding was perfectly well explained by traditional feedforward models of vision. In these models, vision starts with the detection of low-level features. This information is combined locally along the hierarchy of the visual cortex to build more and more complex feature detectors, until neurons respond selectively and robustly to complex objects. Crowding happens when nearby elements interfere in this local feature combination process and impair target recognition. However, recent studies have shown that crowding is not determined by local interactions but by the global configuration across the entire visual field. Depending on how elements group together, crowding can even almost disappear, a phenomenon called uncrowding. Hence, crowding is rather a complex, global and high-level phenomenon, that simple feedforward models cannot explain. In this thesis, I first analyse which models of crowding can explain uncrowding. I compare the performance of diverse models, selected according to different architectural and functional features, such as feedforward vs. recurrent architecture, local or global information processing, including a grouping stage or not. I show that the only model that reproduces human behaviour includes a dedicated recurrent grouping processing stage. Second, I show that global effects in crowding cannot be explained by low-level accounts. It was argued that the Texture Tiling model, based on a complex and high-dimensional pooling stage, may account for global effects in crowding, without requiring any recurrent grouping stage. To test this model, I use a large pool of recent crowding data. I show that the Texture Tiling model is equivalent to a simple pooling model and is thus as limited as these models. Next, I focus on deep neural networks, which are well in the spirit of the feedforward framework of vision and have become state-of-the-art models both in computer vision and neuroscience. I test whether AlexNet and ResNet-50, which have been proposed as realistic models of the visual system, exhibit uncrowding. I show that these networks do not reproduce uncrowding for principled reasons. Finally, I use a genetic algorithm to generate stimuli based on the performance of different models, i.e., in a bottom-up manner. The goal is to avoid using stimuli that favour models of grouping from the start. I compare the distribution of stimuli that are produced by the models to the ones that are produced by humans. I show that only the models that include grouping and segmentations processes behave like humans. Taken together, the results in my thesis highlight the importance of recurrent grouping and segmentation processes in human vision when large portions of the visual field are involved. These results can be used as direct guidelines for future models of vision, in order to constraint how recurrent processing should be incorporated to improve the performance of deep neural networks and other feedforward models of vision, and help them generalize to more complex visual inputs.

EPFL2021