Image Based Relighting of Cultural Artifacts

By incorporating computational methods into the image acquisition pipeline, computational photography has opened up new avenues in the representation and visualization of real world objects in the digital world. For example, we can sample a scene under a few specialized illuminations and a sparse set of viewpoints. We can later, computationally recover the complete light transport properties of the scene. Once we obtain the light transport characterization of cultural artifacts, we can enable users of virtual museums to interact with the artifacts in the same way as we experience these objects in the physical world. In particular, in this thesis, we develop algorithms and tools that facilitate the acquisition of relightable photographs of cultural artifacts, by acquiring their light transport matrix (LTM). A recurrent theme in this thesis is to exploit the low dimensionality of the LTM to develop efficient acquisition strategies for image based rendering. First, we propose a new acquisition and modeling framework for inverse rendering of stained glass windows. Stained glass windows are a dynamic art form that change their appearance constantly, due to the ever-changing outdoor illumination. They are therefore, an exceptional candidate for virtual relighting. However, as they are anchored and very large in size, it is often impossible to sample their entire light transport with controlled illumination. We build a material specific dictionary by studying the scattering properties of glass samples and exploiting the structure of their LTMs in a laboratory setup. We then pose the estimation of the LTM of stained glass from a small set of photographic observations, as a linear inverse problem that is constrained by sparsity in the custom dictionary. We show by experiments that our proposed solution preserves volume impurities under both controlled and uncontrolled, natural illuminations and that the retrieved LTM is heterogeneous, as in the case of real world objects. Next, equipped just with a dictionary to describe light transport in stained glass, we focus on the problem of designing a meaningful LTM, for the synthetic rendering of stained glass. Since this is an extremely ill-posed problem, we begin by exploring the physical properties of glass that can be used as constraints in light transport design. We then propose an iterative matrix completion algorithm that generates the LTM of a heterogeneous glass slab, given the dictionary and the physical constraints. We use this synthesis algorithm, in combination with an input texture to simulate stained glass windows in scenarios where inverse rendering is impossible or as an artist's preview tool. We also introduce a framework for the digital restoration of broken slabs of glass by first acquiring the LTM with inverse rendering and then using the proposed matrix completion framework to repair the fractures. Finally, we present an easy-to-use, handheld acquisition framework to sample the LTM of more general, reflective scenes. We first non-uniformly sample the scene reflectance by moving the LED attached to a smartphone along an arbitrary trajectory, while simultaneously tracking the position of the LED. The acquired reflectance is resampled to obtain a sparse set of samples on a uniform lattice. Using a compressive sensing framework, we recover an approximation to the uniformly sampled LTM, that is then used in scene relighting.