Graph-Based Light Field Super-Resolution

Light field cameras can capture the 3D information in a scene with a single exposure. This special feature makes light field cameras very appealing for a variety of applications: from post capture refocus, to depth estimation and image-based rendering. However, light field cameras exhibit a very limited spatial resolution, which should therefore be increased by computational methods. Off-the-shelf single-frame and multi-frame super-resolution algorithms are not ideal for light field data, as they ignore its particular structure. A few super-resolution algorithms explicitly devised for light field data exist, but they exhibit significant limitations, such as the need to carry out an explicit disparity estimation step for one or several light field views. In this work we present a new light field super-resolution algorithm meant to address these limitations. We adopt a multi- frame alike super-resolution approach, where the information in the different light field views is used to augment the spatial resolution of the whole light field. In particular, we show that coupling the multi-frame paradigma with a graph regularizer that enforces the light field structure permits to avoid the costly and challenging disparity estimation step. Our experiments show that the proposed method compares favorably to the state-of-the- art for light field super-resolution algorithms, both in terms of PSNR and visual quality.

Geometry-Consistent Light Field Super-Resolution via Graph-Based Regularization

Pascal Frossard, Mattia Rossi

Light field cameras capture the 3D information in a scene with a single exposure. This special feature makes light field cameras very appealing for a variety of applications: from post-capture refocus, to depth estimation and image-based rendering. However, light field cameras suffer by design from strong limitations in their spatial resolution, which should therefore be augmented by computational methods. On the one hand, off-the-shelf single-frame and multi-frame super-resolution algorithms are not ideal for light field data, as they do not consider its particular structure. On the other hand, the few super-resolution algorithms explicitly tailored for light field data exhibit significant limitations, such as the need to estimate an explicit disparity map at each view. In this work we propose a new light field super-resolution algorithm meant to address these limitations. We adopt a multi-frame alike super-resolution approach, where the complementary information in the different light field views is used to augment the spatial resolution of the whole light field. We show that coupling the multi-frame approach with a graph regularizer, that enforces the light field structure via nonlocal self similarities, permits to avoid the costly and challenging disparity estimation step for all the views. Extensive experiments show that the new algorithm compares favorably to the other state-of-the-art methods for light field super-resolution, both in terms of PSNR and visual quality.

2018

A Nonsmooth Graph-Based Approach to Light Field Super-Resolution

Mireille El Gheche, Pascal Frossard, Mattia Rossi

In this article we propose a new super-resolution algorithm tailored for light field cameras, which suffer by design from a limited spatial resolution. To do so, we cast the light field super-resolution problem into an optimization problem, where the particular structure of the light field data is captured by a nonsmooth graph-based regularizer, and all the light field views are super-resolved jointly. In our experiments, we show that the proposed method compares favorably to the state-of-the-art light field super-resolution algorithms in terms of PSNR and visual quality. In particular, the nonsmooth graph-based regularizer leads to sharper images while preserving fine details.

2018

Graph-based regularization of inverse problems in imaging

Mattia Rossi

As of today, the extension of the human visual capabilities to machines remains both a cornerstone and an open challenge in the attempt to develop intelligent systems. On the one hand, the development of more and more sophisticated imaging devices, capable of sensing richer information than a plain perspective projection of the real world, is critical in order to allow machines to understand the complex environment around them. On the other hand, despite the advances in imaging, the complexity of the real world cannot be fully captured by a single imaging device, either due to intrinsic hardware limitations or to the environment complexity itself. As a consequence, the attempt to extend the human visual capabilities to machines requires inevitably to estimate some unknown quantities, which could not be measured, from the available captured data. Equivalently, imaging requires the solution of arbitrarily complex inverse problems. In most scenarios, inverse problems are ill-posed and admit an infinite number of solutions, while only one or few of them are the desired ones. It becomes therefore crucial to reduce, equivalently \textit{to regularize}, the solution space by exploiting all the available prior information about the problem structure and, especially, about the target quantity to estimate. In this thesis we investigate the use of graph-based regularizers to encode our prior knowledge about the target quantity and to inject it directly into the inverse problem. In particular, we cast the inverse problem into an optimization task, where the target quantity is modelled as a graph whose topology captures our prior knowledge. In order to show the effectiveness and the flexibility of graph-based regularizers, we study their use in different inverse imaging problems, each one characterized by different geometrical constraints. We start by investigating how to augment the resolution of a light field. In fact, although light field cameras permit to capture the 3D information in a scene within a single exposure, thus providing much richer information than a perspective camera, their compact design limits their spatial resolution dramatically. We present a smooth graph-based regularizer which models the geometry of a light field explicitly and we use it to augment the light field spatial resolution while relying only on the complementary information encoded in the low resolution light field views. In particular, we show that the use of a graph-based regularizer permits to enforce the light field geometric structure without the need for a precise and costly disparity estimation step. Then we analyze the further benefits provided by adopting nonsmooth graph-based regularizers, as these better preserve edges and fine details than their smooth counterpart. In particular, we focus on a specific nonsmooth graph-based regularizer and show its effectiveness within two applications. The first application revolves again around light field super-resolution, which permits a comparison with the smooth regularizer adopted previously. The second applications is disparity estimation in omnidirectional stereo systems, where the two captured images and the target disparity map live on a spherical surface, hence a graph-based regularizer can be used to model the non trivial correlation underneath each signal. Finally, we investigate the refinement of a depth map and the estimation of the corresponding normal map. In fact, ...

EPFL2020

Graph-based regularization of inverse problems in imaging

Mattia Rossi

EPFL2020