Rendering (computer graphics) | EPFL Graph Search

Rendering or image synthesis is the process of generating a photorealistic or non-photorealistic image from a 2D or 3D model by means of a computer program. The resulting image is referred to as the render. Multiple models can be defined in a scene file containing objects in a strictly defined language or data structure. The scene file contains geometry, viewpoint, texture, lighting, and shading information describing the virtual scene. The data contained in the scene file is then passed to a rendering program to be processed and output to a or raster graphics image file. The term "rendering" is analogous to the concept of an artist's impression of a scene. The term "rendering" is also used to describe the process of calculating effects in a video editing program to produce the final video output. Rendering is one of the major sub-topics of 3D computer graphics, and in practice it is always connected to the others. It is the last major step in the graphics pipeline, giving mod

Light Path Gradients for Forward and Inverse Rendering

Tizian Lucien Zeltner

Physically based rendering is a process for photorealistic digital image synthesis and one of the core problems in computer graphics. It involves simulating the light transport, i.e. the emission, propagation, and scattering of light through a virtual scene that is defined by a detailed description of object geometry and appearance. Research over the last decades has led to sophisticated rendering techniques and recently, the inversion of this process, i.e. recovering scene parameters from image observations, has also received significant attention. In this thesis, we investigate methods in both physically based forward and inverse rendering that exploit light path gradients.The first part is concerned with scattering from specular surfaces, which produces complex optical effects that are frequently encountered in realistic scenes: intricate caustics due to focused reflection, multiple refractions, and high-frequency glints from specular microstructure. Yet, despite their importance and considerable research to this end, sampling of light paths that cause these effects remains a formidable challenge of forward rendering.We propose a surprisingly simple and general path sampling strategy that targets the examples above. Valid light path configurations need to fulfill the physical laws of reflection and refraction, and we find these using a numerical root-finding process that is driven by geometric light path gradients. In contrast to prior work, our method supports high-frequency normal- or displacement-mapped geometry, samples specular-diffuse-specular (SDS) paths, and is compatible with standard Monte Carlo methods including unidirectional path tracing. We demonstrate our method on a range of challenging scenes and evaluate it against state-of-the-art methods for rendering caustics and glints.In the second part, we consider differentiable rendering algorithms. These propagate derivatives through the full light transport simulation to solve inverse rendering problems via gradient-based optimization. Recent progress has led to methods that can simultaneously compute derivatives with respect to millions of scene parameters. At the same time, elementary properties of these methods remain poorly understood.Current algorithms for differentiable rendering are constructed by mechanically differentiating a given primal algorithm. As differentiation fundamentally changes the underlying problem, this is often suboptimal and instead, primal and differential algorithms should be decoupled so that the latter can suitably adapt. This is surprisingly complex. Even the most basic Monte Carlo path tracer already involves several design choices concerning the techniques for sampling materials and emitters, and their combination, e.g. via multiple importance sampling (MIS). Differentiation causes a veritable explosion of this decision tree: should we differentiate only the estimator, or also the sampling technique? Should MIS be applied before or after differentiation? Are specialized derivative sampling strategies of any use? How should visibility-related discontinuities be handled when millions of parameters are differentiated simultaneously? We provide a taxonomy and analysis of different estimators for differential light transport to provide intuition about these and related questions.

EPFL2021

Differentiable Physically Based Rendering: Algorithms, Systems and Applications

Merlin Eléazar Nimier-David

Physically based rendering methods can create photorealistic images by simulating the propagation and interaction of light in a virtual scene. Given a scene description including the shape of objects, participating media, material properties, etc., the simulation computes an image representing the radiance reaching the sensor.This thesis, however, pursues the corresponding inverse problem: given observations such as pictures of a scene, we want to recover a plausible description of its components. Unfortunately, the rendering process is typically far too complex to invert analytically. We therefore turn to iterative gradient-based approximations of the inverse, that require efficiently estimating gradients of an objective function with respect to the scene parameters of interest.The first part of this thesis is dedicated to algorithms for gradient estimation. We consider the usage of automatic differentiation and examine the associated tradeoffs. Next, we introduce radiative backpropagation, an adjoint method that casts the gradient estimation problem into a modified light transport problem, unlocking vastly more efficient implementations. For the case of participating media, we propose differential ratio tracking, a sampling technique that addresses the bias and high variance found in existing gradient estimators.In the second part, we focus on the design of systems to support effective differentiable rendering research, including the efficient implementation of the algorithms above. We describe the architecture and features of Mitsuba 2, an open-source retargetable physically based renderer. Mitsuba 2 supports different representations of colors (RGB, spectral, polarized), computing platforms (scalar, CPU vectorized, GPU), and numerical precision, within a single codebase. Importantly, automatic differentiation can be applied throughout the system. Then, we extend this design by applying an automatic conversion from wavefront-style rendering to a megakernel-based approach, leveraging just-in-time compilation. We obtain a fast, flexible and memory-efficient framework for primal and differentiable physically based rendering.Finally, the third part showcases three applications of differentiable physically based rendering: caustic design, inverse volume rendering, and material & lighting estimation in real indoor scenes. In all cases, special care is taken to avoid sub-optimal local minima due to the ambiguous and non-convex nature of the reconstruction problems.

EPFL2022

Coherent rendering of virtual smile previews with fast neural style transfer

Valentin Vasiliu

Coherent rendering in augmented reality deals with synthesizing virtual content that seamlessly blends in with the real content. Unfortunately, capturing or modeling every real aspect in the virtual rendering process is often unfeasible or too expensive. We present a post-processing method that improves the look of rendered overlays in a dental virtual try-on application. We combine the original frame and the default rendered frame in an autoencoder neural network in order to obtain a more natural output, inspired by artistic style transfer research. Specifically, we apply the original frame as style on the rendered frame as content, repeating the process with each new pair of frames. Our method requires only a single forward pass, our shallow architecture ensures fast execution, and our internal feedback loop inherently enforces temporal consistency.

IEEE2019