Historical newspaper semantic segmentation using visual and textual features

Mass digitization and the opening of digital libraries gave access to a huge amount of historical newspapers. In order to bring structure into these documents, current techniques generally proceed in two distinct steps. First, they segment the digitized images into generic articles and then classify the text of the articles into finer-grained categories. Unfortunately, by losing the link between layout and text, these two steps are not able to account for the fact that newspaper content items have distinctive visual features. This project proposes two main novelties. Firstly, it introduces the idea of merging the segmentation and classification steps, resulting in a fine- grained semantic segmentation of newspapers images. Secondly, it proposes to use textual features under the form of embeddings maps at segmentation step. The semantic segmentation with four categories (feuilleton, weather forecast, obituary, and stock exchange table) is done using a fully convolutional neural network and reaches a mIoU of 79.3%. The introduction of embeddings maps improves the overall performances by 3% and the generalization across time and newspapers by 8% and 12%, respectively. This shows a strong potential to consider the semantic aspect in the segmentation of newspapers and to use textual features to improve generalization.

Large-Scale Image Segmentation with Convolutional Networks

Pedro Henrique Oliveira Pinheiro

Object recognition is one of the most important problems in computer vision. However, visual recognition poses many challenges when tried to be reproduced by artificial systems. A main challenge is the problem of variability: objects can appear across huge variations in pose, appearance, illumination and occlusion, and a visual system need to be robust to all these changes. In the present thesis, we are interested in pixel-level recognition problems, ie, problems in which the objective is to partition a given image into multiple regions (overlapping or not) that are considered meaningful according to some criterion. Our interests are in algorithms that require the least amount of feature engineering and are easy to scale. Deep learning methods fit very well with this objective: these models alleviate the need of engineered features by discriminatively training a system from raw data (pixels). More precisely, we propose different convolutional neural network (CNN) based algorithms to deal with three important segmentation problems: semantic segmentation, object proposal generation and object detection with segments. The objective of semantic segmentation is to generate a categorical label to each pixel present in a scene. We first study the problem of fully supervised semantic segmentation. We propose a recurrent CNN that is able to consider a large input context (while limiting its capacity), which is essential to model long range pixel label dependencies. This approach achieves state-of-the-art performance without relying on any post-processing smoothing step. However, having densely labeled images to train a model can be expensive and require a lot of human labor. We also propose a CNN-based model that is able to infer object semantic segmentation by leveraging only the object category information from images. This is achieved by casting the problem into a multiple instance learning framework. This approach beats previous state of the art in weakly supervised semantic segmentation by a large margin. Object proposal algorithms generate a set of regions (segments) that are likely to contain objects, independent of their semantic category. Contrary to most approaches (which rely on low-level vision cues), we propose a CNN-based discriminative approach that is able to learn segmentation proposals from raw pixels. This approach is proven to be quite effective in this setting, achieving substantially higher recall using fewer proposals than other methods. The state of the art is pushed further with the introduction of a new top-down network augmentation. The resulting bottom-up/top-down network combines low-level rich spatial information with high-level object semantic information to improve segmentation, while remaining fast at test time. Finally, we show that the proposals generated by our approach, when coupled with a standard state-of-the-art object detection pipeline, achieve considerably better performance than previous proposals methods.

EPFL2017

Large-Scale Image Segmentation with Convolutional Networks

Pedro Henrique Oliveira Pinheiro

Sciences et Techniques de l’Ingénieur (STI)2017

Automatic Segmentation of Nuclei in Histopathology Images using Encoding-Decoding Convolutional Neural Networks

Beril Besbinar, Pascal Frossard, Deniz Sayin Mercadier

Accurate and fast segmentation of nuclei in histopathological images plays a crucial role in cancer research for detection and grading, as well as personal treatment. Despite the important efforts, current algorithms are still suboptimal in terms of speed, adaptivity and generalizability. Popular Deep Convolutional Neural Networks (DCNNs) have recently been utilized for nuclei segmentation, outperforming traditional approaches that exploit color and texture features in combination with shallow classifiers or segmentation algorithms. However, DCNNs need large annotated datasets that require extensive amount of time and expert knowledge. In addition, segmentation results obtained by either traditional or DCNN approaches often require a post-processing step to separate cluttered nuclei. In this paper, we propose a computationally efficient nuclei segmentation framework based on DCNNs exhibiting an encoding-decoding structure. We use a partially-annotated dataset and develop an effective training solution. We also use a weighted background model for network to give more importance to borders of nuclei to overcome the problem of clutters. The abolition of any pre-processing or post-processing step without any compromise on the performance leads to a fast and parameter-free system, which presents important advantages with respect to state-of-the-art.

IEEE2019

Large-Scale Image Segmentation with Convolutional Networks

Pedro Henrique Oliveira Pinheiro

Sciences et Techniques de l’Ingénieur (STI)2017

Large-Scale Image Segmentation with Convolutional Networks

Pedro Henrique Oliveira Pinheiro

EPFL2017