How Asynchronous Events Encode Video

As event-based sensing gains in popularity, theoretical understanding is needed to harness this technology’s potential. Instead of recording video by capturing frames, event-based cameras have sensors that emit events when their inputs change, thus encoding information in the timing of events. This creates new challenges in establishing reconstruction guarantees and algorithms, but also provides advantages over frame-based video. We use time encoding machines to model event-based sensors: TEMs also encode their inputs by emitting events characterized by their timing and reconstruction from time encodings is well understood. We consider the case of time encoding bandlimited video and demonstrate a dependence between spatial sensor density and overall spatial and temporal resolution. Such a dependence does not occur in frame-based video, where temporal resolution depends solely on the frame rate of the video and spatial resolution depends solely on the pixel grid. However, this dependence arises naturally in event-based video and allows oversampling in space to provide better time resolution. As such, event-based vision encourages using more sensors that emit fewer events over time.

Asynchrony Increases Efficiency: Time Encoding of Videos and Low-Rank Signals

Karen Adam, Adam James Scholefield, Martin Vetterli

In event-based sensing, many sensors independently and asynchronously emit events when there is a change in their input. Event-based sensing can present significant improvements in power efficiency when compared to traditional sampling, because (1) the output is a stream of events where the important information lies in the timing of the events, and (2) the sensor can easily be controlled to output information only when interesting activity occurs at the input. Moreover, event-based sampling can often provide better resolution than standard uniform sampling. Not only does this occur because individual event-based sensors have higher temporal resolution (Rebecq et al., 2021) it also occurs because the asynchrony of events within a sensor and therefore across sensors allows for less redundant and more informative encoding. We would like to explain how such curious results come about. To do so, we use ideal time encoding machines as a proxy for event-based sensors. We explore time encoding of signals with low rank structure, and apply the resulting theory to video. We then see how the asynchronous firing across time encoding machines can couple spatial sampling density with temporal resolution, leading to better reconstruction, whereas, in frame-based video, temporal resolution depends solely on the frame-rate and spatial resolution solely on the pixel grid used.

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Direct time-of-flight SPAD image sensors for light detection and ranging

Preethi Padmanabhan

Depth sensing is an increasingly important feature in many applications of consumer, automotive, augmented/virtual reality (AR-VR), space and bio-medical imaging. Long range, high depth resolution, high spatial resolution, and high frame rates are often conflicting requirements and difficult to be simultaneously achieved due to extreme operating conditions. Direct time-of-flight (DTOF) has evolved to becoming a powerful technique to perform light detection and ranging (LiDAR). Thanks to advances in low-jitter optical detectors, such as single-photon avalanche diodes (SPADs), and accurate chronometers like time-to-digital converters (TDCs), picosecond timing resolution is possible, thus enabling millimetric depth resolutions. High ambient light is an inevitable challenge in LiDAR applications, whose levels may exceed up to 100 klux on a bright sunny day, making it particularly challenging to detect a target submerged within an overwhelming noise floor. High ambient light operation can be accommodated by means of optical filtering, a higher laser power or temporal filtering techniques. Optical filtering is often restricted to a narrow, 10-50 nm bandwidth, insufficient at high ambient light levels. Higher laser power is not always possible, due to eye safety regulations and power constraints. Temporal filtering such as time gating and coincidence detection can thus be powerful tools to cope with high ambient light. This thesis focuses on the design of DTOF sensors for LiDAR. To that end, two SPAD-based DTOF sensors are designed. The first sensor is designed in a 3D-stacked 45/65 nm CMOS technology, thus, enabling a modular architecture where the module itself comprises of 8x16 pixels. With a 60 ps-resolution TDC at its core, the sensor provides centimetric accuracy up to 300 m range in free space. The second sensor, named Jatayu, advances the previous design by hosting 256x128 pixels, thereby, significantly improving on its spatial resolution. While retaining its modularity, Jatayu also enables multi-level coincidence detection and progressive time-gating to suppress background light. To the best of the authorâs knowledge, progressive gating has been implemented in a LiDAR for the first time in this thesis. Designed in a 3D-stacked 45/22 nm CMOS technology, the sensor achieves under 7 cm accuracy over 100 m ranging and 10 klux background light. With its capability of acquiring 128Â£128, 3D depth maps of high dynamic range scenes, Jatayu is highly suitable for a variety of imaging applications in many different scenarios.

EPFL2021

A 500 x 500 Dual-Gate SPAD Imager With 100% Temporal Aperture and 1 ns Minimum Gate Length for FLIM and Phasor Imaging Applications

Andrei Ardelean, Claudio Bruschini, Edoardo Charbon, Paul Mos, Arin Can Ülkü, Michael Alan Wayne

In this article, we report on SwissSPAD3 (SS3), a 500 x 500 pixel single-photon avalanche diode (SPAD) array, fabricated in 0.18-mu m CMOS technology. In this sensor, we introduce a novel dual-gate architecture with two contiguous temporal windows, or gates, guaranteed by the circuit architecture to be nonoverlapping and covering the totality of the sensor's exposure period. The gates can be adjusted with a temporal resolution of 17.9 ps, and the minimum measured gate width is 0.99 ns; to our knowledge, the shortest reported to date among large-format SPAD imagers. In the dual-channel mode, the burst frame rate is 49.8 and 97.7 kframes/s in the single-channel mode. A 2690-MB/s PCI express (PCIe) interface has been added to the data acquisition framework, enabling continuous operation at approximately 44 and 88 kframes/s. Due to optimizations of the gate-signal tree, we achieved a significant reduction to gate skew and gate width variation, which is negligible with respect to the SPAD temporal jitter. These improvements, along with sub-10-cps dark count rate (DCR) per pixel and 50% maximum photon detection probability (PDP), result in a sensor particularly well suited for fast acquisition fluorescence lifetime imaging microscopy (FLIM) experiments, for which we demonstrate reduced dispersion versus a single-gated sensor.

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Direct time-of-flight SPAD image sensors for light detection and ranging

Preethi Padmanabhan

EPFL2021