Learning an event sequence embedding for event-based deep stereo

Today, a frame-based camera is the sensor of choice for machine vision applications. However, these cameras, originally developed for acquisition of static images rather than for sensing of dynamic uncontrolled visual environments, suffer from high power consumption, data rate, latency and low dynamic range. An event-based image sensor addresses these drawbacks by mimicking a biological retina. Instead of measuring the intensity of every pixel in a fixed time-interval, it reports events of significant pixel intensity changes. Every such event is represented by its position, sign of change, and timestamp, accurate to the microsecond. Asynchronous event sequences require special handling, since traditional algorithms work only with synchronous, spatially gridded data. To address this problem we introduce a new module for event sequence embedding, for use in difference applications. The module builds a representation of an event sequence by firstly aggregating information locally across time, using a novel fully-connected layer for an irregularly sampled continuous domain, and then across discrete spatial domain. Based on this module, we design a deep learning-based stereo method for event-based cameras. The proposed method is the first learning-based stereo method for an event-based camera and the only method that produces dense results. We show that large performance increases on the Multi Vehicle Stereo Event Camera Dataset (MVSEC), which became the standard set for benchmarking of event-based stereo methods.

Optimization methods for collaborative learning

Sai Praneeth Reddy Karimireddy

A traditional machine learning pipeline involves collecting massive amounts of data centrally on a server and training models to fit the data. However, increasing concerns about the privacy and security of user's data, combined with the sheer growth in the data sizes has incentivized looking beyond such traditional centralized approaches. Collaborative learning (which encompasses distributed, federated, and decentralized learning) proposes instead for a network of data holders to collaborate together to train models without transmitting any data. This new paradigm minimizes data exposure, but inherently faces some fundamental challenges. In this thesis, we bring to bear the framework of stochastic optimization to formalize and develop new algorithms for these challenges. This serves not only to develop novel solutions, but also to test the utility of the optimization lens in modern deep learning.We study three fundamental problems. Firstly, collaborative training replaces a one-time transmission of raw data with repeated rounds of communicating partially trained models. However, this quickly runs against bandwidth constraints when dealing with large models. We propose to solve this bandwidth constraint using compressed communication. Next, collaborative training leverages the computation power of the data holders directly. However, this is not as reliable as using a data center with only a subset of them available at any given time. Thus, we require new algorithms which can efficiently utilize unreliable local computation of the data holders. Finally, collaborative training allows any data holder to participate in the training process, without being able to inspect their data or local computation. This may potentially open the system to malicious or faulty agents who seek to derail the training. We develop algorithms with Byzantine robustness which are guaranteed to be resilient to such attackers.

EPFL2021

Learning to Detect Objects with Minimal Supervision

Karim Ali

Many classes of objects can now be successfully detected with statistical machine learning techniques. Faces, cars and pedestrians, have all been detected with low error rates by learning their appearance in a highly generic manner from extensive training sets. These recent advances have enabled the use of reliable object detection components in real systems, such as automatic face focusing functions on digital cameras. One key drawback of these methods, and the issue addressed here, is the prohibitive requirement that training sets contain thousands of manually annotated examples. We present three methods which make headway toward reducing labeling requirements and in turn, toward a tractable solution to the general detection problem. First, we propose a new learning strategy for object detection. The proposed scheme forgoes the need to train a collection of detectors dedicated to homogeneous families of poses, and instead learns a single classifier that has the inherent ability to deform based on the signal of interest. We train a detector with a standard AdaBoost procedure by using combinations of pose-indexed features and pose estimators. This allows the learning process to select and combine various estimates of the pose with features able to compensate for variations in pose without the need to label data for training or explore the pose space in testing. We validate our framework on three types of data: hand video sequences, aerial images of cars, as well as face images. We compare our method to a standard Boosting framework, with access to the same ground truth, and show a reduction in the false alarm rate of up to an order of magnitude. Where possible, we compare our method to the state-of-the art, which requires pose annotations of the training data, and demonstrate comparable performance. Second, we propose a new learning method which exploits temporal consistency to successfully learn a complex appearance model from a sparsely labeled training video. Our approach consists in iteratively improving an appearance-based model built with a Boosting procedure, and the reconstruction of trajectories corresponding to the motion of multiple targets. We demonstrate the efficiency of our procedure by learning a pedestrian detector from videos and a cell detector from microscopy image sequences. In both cases, our method is demonstrated to reduce the labeling requirement by one to two orders of magnitude. We show that in some instances, our method trained with sparse labels on a video sequence is able to outperform a standard learning procedure trained with the fully labeled sequence. Third, we propose a new active learning procedure which exploits the spatial structure of image data and queries entire scenes or frames of a video rather than individual examples. We extend the Query by Committee approach allowing it to characterize the most informative scenes that are to be selected for labeling. We show that an aggressive procedure which exhibits zero tolerance to target localization error performs as well as more sophisticated strategies taking into account the trade-off between missed detections and localization error. Finally, we combine this method with our two proposed approaches above and demonstrate that the resulting algorithm can properly perform car detection from a small set of annotated image as well as pedestrian detection from a handful of labeled video frames.

EPFL2012

Towards Scalable Segmentation of 3D Image Stacks

Carlos Joaquin Becker

Imaging modalities such as Electron Microscopy (EM) and Light Microscopy (LM) can now deliver high-quality, high-resolution image stacks of neural structures. Though these imaging modalities can be used to analyze a variety of components that are critical to understanding brain function, the amount of human annotation effort required to analyze them remains a major bottleneck. This has triggered great interest in automating the annotation process, with most state-of-the-art algorithms nowadays relying on machine learning. However, such methods still require significant amounts of labeled examples for training, which can be highly time consuming and arduous, stressing the need for new approaches that require less amount of human effort. In light of this, we present here two efficient machine learning algorithms that incorporate expert knowledge to maximize prediction performance and simultaneously speed up analysis by reducing the required amount of labeled data. First, we present a new approach for the automated segmentation of synapses in image stacks acquired in EM that relies on image features specifically designed to take spatial context into account. These features are used to train a classifier that can effectively learn cues such as the presence of a nearby post-synaptic region. Our algorithm successfully distinguishes synapses from the numerous other organelles that appear within an EM volume, including those whose local textural properties are relatively similar. We evaluate our approach on three different datasets and demonstrate our ability to reliably collect shape, density, and orientation statistics over hundreds of synapses. Second, we focus on reducing the required amount of annotation effort. Due to changing experimental conditions in the image acquisition process, successive stacks often exhibit differences that are severe enough to make it difficult to use a classifier trained for a specific volume on another one. This means that the tedious annotation process has to be repeated for each new stack, resulting in a major bottleneck. We present a domain adaptation algorithm that addresses this issue by effectively leveraging labeled examples across different acquisitions and significantly reducing the annotation requirements. Our approach can handle complex, non-linear image feature transformations and scales to large microscopy datasets and high-dimensional feature spaces. We evaluate our approach on four EM and LM applications where annotation is very costly. We achieve a significant improvement over the state-of-the-art methods and demonstrate our ability to greatly reduce human annotation effort. Third, we apply our synapse segmentation approach to analyze and compare the structure and shape of synaptic densities in adult and aged mice, such as their area and number of perforations. This detailed analysis requires labeling each voxel within every synapse, making manual annotation unfeasible for large volumes. We show that we can bridge this gap with our approach and demonstrate its effectiveness on six large FIB/SEM brain stacks. Our approach generates segmentations that agree with expert annotations, while requiring very little annotation effort. To our knowledge, we are the first ones to analyze synapse shape in such detail on large stacks, as previous work has strongly relied on manual annotations, restricting analysis to small volumes.

EPFL2016