Neural networks for semantic segmentation of historical city maps: Cross-cultural performance and the impact of figurative diversity

In this work, we present a new semantic segmentation model for historical city maps that surpasses the state of the art in terms of flexibility and performance. Research in automatic map processing is largely focused on homogeneous corpora or even individual maps, leading to inflexible algorithms. Recently, convolutional neural networks have opened new perspectives for the development of more generic tools. Based on two new maps corpora, the first one centered on Paris and the second one gathering cities from all over the world, we propose a method for operationalizing the figuration based on traditional computer vision algorithms that allows large-scale quantitative analysis. In a second step, we propose a semantic segmentation model based on neural networks and implement several improvements. Finally, we analyze the impact of map figuration on segmentation performance and evaluate future ways to improve the representational flexibility of neural networks. To conclude, we show that these networks are able to semantically segment map data of a very large figurative diversity with efficiency.

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

The way our brain learns to disentangle complex signals into unambiguous concepts is fascinating but remains largely unknown. There is evidence, however, that hierarchical neural representations play a key role in the cortex. This thesis investigates biologically plausible models of unsupervised learning of hierarchical representations as found in the brain and modern computer vision models. We use computational modeling to address three main questions at the intersection of artificial intelligence (AI) and computational neuroscience.The first question is: What are useful neural representations and when are deep hierarchical representations needed? We approach this point with a systematic study of biologically plausible unsupervised feature learning in a shallow 2-layer networks on digit (MNIST) and object (CIFAR10) classification. Surprisingly, random features support high performance, especially for large hidden layers. When combined with localized receptive fields, random feature networks approach the performance of supervised backpropagation on MNIST, but not on CIFAR10. We suggest that future models of biologically plausible learning should outperform such random feature benchmarks on MNIST, or that such models should be evaluated in different ways.The second question is: How can hierarchical representations be learned with mechanisms supported by neuroscientific evidence? We cover this question by proposing a unifying Hebbian model, inspired by common models of V1 simple and complex cells based on unsupervised sparse coding and temporal invariance learning. In shallow 2-layer networks, our model reproduces learning of simple and complex cell receptive fields, as found in V1. In deeper networks, we stack multiple layers of Hebbian learning but find that it does not yield hierarchical representations of increasing usefulness. From this, we hypothesise that standard Hebbian rules are too constrained to build increasingly useful representations, as observed in higher areas of the visual cortex or deep artificial neural networks.The third question is: Can AI inspire learning models that build deep representations and are still biologically plausible? We address this question by proposing a learning rule that takes inspiration from neuroscience and recent advances in self-supervised deep learning. The proposed rule is Hebbian, i.e. only depends on pre- and post-synaptic neuronal activity, but includes additional local factors, namely predictive dendritic input and widely broadcasted modulation factors. Algorithmically, this rule applies self-supervised contrastive predictive learning to a causal, biological setting using saccades. We find that networks trained with this generalised Hebbian rule build deep hierarchical representations of images, speech and video.We see our modeling as a potential starting point for both, new hypotheses, that can be tested experimentally, and novel AI models that could benefit from added biological realism.

EPFL2021

Crowding and the importance of grouping and segmentation processes in human vision

Alban Bornet

Human vision has evolved to make sense of a world in which elements almost never appear in isolation. Surprisingly, the recognition of an element in a visual scene is strongly limited by the presence of other nearby elements, a phenomenon known as visual crowding. Crowding impacts vision at all levels and is thus a versatile tool to understand the fundamental mechanisms of vision. For decades, visual crowding was perfectly well explained by traditional feedforward models of vision. In these models, vision starts with the detection of low-level features. This information is combined locally along the hierarchy of the visual cortex to build more and more complex feature detectors, until neurons respond selectively and robustly to complex objects. Crowding happens when nearby elements interfere in this local feature combination process and impair target recognition. However, recent studies have shown that crowding is not determined by local interactions but by the global configuration across the entire visual field. Depending on how elements group together, crowding can even almost disappear, a phenomenon called uncrowding. Hence, crowding is rather a complex, global and high-level phenomenon, that simple feedforward models cannot explain. In this thesis, I first analyse which models of crowding can explain uncrowding. I compare the performance of diverse models, selected according to different architectural and functional features, such as feedforward vs. recurrent architecture, local or global information processing, including a grouping stage or not. I show that the only model that reproduces human behaviour includes a dedicated recurrent grouping processing stage. Second, I show that global effects in crowding cannot be explained by low-level accounts. It was argued that the Texture Tiling model, based on a complex and high-dimensional pooling stage, may account for global effects in crowding, without requiring any recurrent grouping stage. To test this model, I use a large pool of recent crowding data. I show that the Texture Tiling model is equivalent to a simple pooling model and is thus as limited as these models. Next, I focus on deep neural networks, which are well in the spirit of the feedforward framework of vision and have become state-of-the-art models both in computer vision and neuroscience. I test whether AlexNet and ResNet-50, which have been proposed as realistic models of the visual system, exhibit uncrowding. I show that these networks do not reproduce uncrowding for principled reasons. Finally, I use a genetic algorithm to generate stimuli based on the performance of different models, i.e., in a bottom-up manner. The goal is to avoid using stimuli that favour models of grouping from the start. I compare the distribution of stimuli that are produced by the models to the ones that are produced by humans. I show that only the models that include grouping and segmentations processes behave like humans. Taken together, the results in my thesis highlight the importance of recurrent grouping and segmentation processes in human vision when large portions of the visual field are involved. These results can be used as direct guidelines for future models of vision, in order to constraint how recurrent processing should be incorporated to improve the performance of deep neural networks and other feedforward models of vision, and help them generalize to more complex visual inputs.

EPFL2021

Understanding Deep Neural Networks using Adversarial Attacks

Krishna Kanth Nakka

Deep Neural Networks (DNNs) have achieved great success in a wide range of applications, such as image recognition, object detection, and semantic segmentation. Even thoughthe discriminative power of DNNs is nowadays unquestionable, serious concerns have arised ever since DNNs have shown to be vulnerable to adversarial examples craftedby adding imperceptible perturbations to clean images. The implications of these malicious attacks are even more significant for DNNs deployed in real-world systems, e.g.,autonomous driving and biometric authentication. Consequently, an intriguing question that we aim to understand is the underlying behavior of DNNs to adversarial attacks.This thesis contributes to a better understanding of the mechanism of adversarial attacks on DNNs. Our main contributions are broadly in two directions: (1) we proposeinterpretable architectures first to understand the reasons for the success of adversarial attacks and then to improve the robustness of DNNs; (2) we design intuitive adversarialattacks to both mislead and use as a tool to expand our present understanding of DNNs' internal workings and their limitations. In the first direction, we introduce deep architectures that allow humans to interpret the reasoning process of DNNs prediction. Specifically, we incorporate Bag-of-visual-wordsrepresentations from the pre-deep learning era into DNNs using an attention scheme. We find key reasons for adversarial attack success and use these insights to propose anadversarial defense by maximally separating the latent features of discriminative regions while minimizing the contribution of non-discriminative regions in the final prediction.The second direction deals with the design of adversarial attacks to understand DNNs' limitations in a real-world environment. To begin with, we show that existing state-of-the-art semantic segmentation networks that achieve superior performance by exploiting the context are highly susceptible to indirect local attacks. Furthermore, we demonstrate the existence of universal directional perturbations that are quasi-independent of the input template but still successfully fool unknown siamese-based visual object trackers. We then identify that the mid-level filter banks across different backbones bear strong similarities and thus can be potential common ground for attack. We, therefore, learn a generator that disrupts mid-level features with high transferability across different target architectures, datasets, and tasks. In short, our attacks highlight critical vulnerabilities of DNNs, which make their deployment challenging in the real-world environment, even in the extreme case when the attacker is unaware of the target architecture or the targetdata used to train it.Furthermore, we go beyond fooling networks and demonstrate the usefulness of adversarial attacks for studying the internal disentangled representations in self-supervised 3D pose estimation networks. We observe that adversarial manipulation of appearance information in the input image alters the pose output, indicating that the pose code contains appearance information and disentanglement is far from complete. Besides the above contributions, an underlying theme that arises multiple times in this thesis is counteracting the adversarial attacks by detecting them.

EPFL2022

Biologically plausible unsupervised learning in shallow and deep neural networks

Bernd Albert Illing

EPFL2021

Understanding Deep Neural Networks using Adversarial Attacks

Krishna Kanth Nakka

EPFL2022

Crowding and the importance of grouping and segmentation processes in human vision

Alban Bornet

EPFL2021