Synthesis and Analysis of 3D shapes with Geometric Deep Learning in Computer-Aided Engineering

In this thesis, we advocate that Computer-Aided Engineering could benefit from a Geometric Deep Learning revolution, similarly to the way that Deep Learning revolutionized Computer Vision. To do so, we consider a variety of Computer-Aided Engineering problems, including physics simulation, design optimization, shape parameterization and shape reconstruction. For each of these problems, we develop novel algorithms that use Geometric Deep Learning to improve the capabilities of existing systems. First, we demonstrate how Geometric Deep Learning architectures can be used to learn to emulate physics simulations. Specifically, we design a neural architecture which, given as input a 3D surface mesh, directly regresses physical quantities of interest defined over the mesh surface. The key to making our approach practical is re-meshing the original shape using a polycube map, which makes it possible to perform computations on Graphic Process Units efficiently. This results in a speed up of 2 orders of magnitude with respect to physics simulators with little loss in accuracy: our main motivation is to provide lightweight performance feedback to improve interactivity in early design stages. Furthermore, being a neural network, our physics emulator is naturally differentiable with respect to input geometry parameters, allowing us to solve shape design problems through gradient-descent. The resulting algorithm outperforms state of-the-art methods by 5 to 20% for 2D optimization tasks and, in contrast to existing methods, our approach can be further used to optimize raw 3D geometry. This could empower designers and engineers to improve the performance of a given design automatically, i.e. without requiring any specific knowledge about the physics of the problem they are trying to solve. To perform shape optimization robustly, we develop novel parametric representations for 3D surface meshes that can be used as strong priors during the optimization process. To this end, we introduce a differentiable way to produce explicit surface mesh representations from Neural Signed Distance Functions. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying neural implicit field. This results in surface mesh parameterizations that can handle topology changes, something that is not feasible with currently available techniques. Finally, we propose a pipeline for reconstructing and editing 3D shapes from line drawings that leverages our end-to-end differentiable surface mesh representation. When integrated into a user interface that provides camera parameters for the sketches, we can exploit our latent parametrization to refine a 3D mesh so that its projections match the external contours outlined in the sketch. We show that this is crucial to make our approach robust with respect to domain gap. Furthermore, it can be used for shape refinement given only single pen strokes. This system could allow engineers and designers to translate legacy 2D sketches to real-world 3D models that can readily be used for downstream tasks such as physics simulations or fabrication, or to interact and modify 3D geometry in the most natural way possible, i.e. with a pen stroke.

Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches

Pascal Fua, Benoît Alain René Guillard, Edoardo Remelli, Pierre Mathieu Yvernay

Reconstructing 3D shape from 2D sketches has long been an open problem because the sketches only provide very sparse and ambiguous information. In this paper, we use an encoder/decoder architecture for the sketch to mesh translation. When integrated into a user interface that provides camera parameters for the sketches, this enables us to leverage its latent parametrization to represent and refine a 3D mesh so that its projections match the external contours outlined in the sketch. We will show that this approach is easy to deploy, robust to style changes, and effective. Furthermore, it can be used for shape refinement given only single pen strokes. We compare our approach to state-of-the-art methods on sketches---both hand-drawn and synthesized---and demonstrate that we outperform them.

IEEE2021

From Classical to Unsupervised-Deep-Learning Methods for Solving Inverse Problems in Imaging.

Harshit Gupta

In this thesis, we propose new algorithms to solve inverse problems in the context of biomedical images. Due to ill-posedness, solving these problems require some prior knowledge of the statistics of the underlying images. The traditional algorithms, in the field, assume prior knowledge related to smoothness or sparsity of these images. Recently, they have been outperformed by the second generation algorithms which harness the power of neural networks to learn required statistics from training data. Even more recently, last generation deep-learning-based methods have emerged which require neither training nor training data. This thesis devises algorithms which progress through these generations. It extends these generations to novel formulations and applications while bringing more robustness. In parallel, it also progresses in terms of complexity, from proposing algorithms for problems with 1D data and an exact known forward model to the ones with 4D data and an unknown parametric forward model. We introduce five main contributions. The last three of them propose deep-learning-based latest-generation algorithms that require no prior training. 1) We develop algorithms to solve the continuous-domain formulation of inverse problems with both classical Tikhonov and total-variation regularizations. We formalize the problems, characterize the solution set, and devise numerical approaches to find the solutions. 2) We propose an algorithm that improves upon end-to-end neural-network-based second generation algorithms. In our method, a neural network is first trained as a projector on a training set, and is then plugged in as a projector inside the projected gradient descent (PGD). Since the problem is nonconvex, we relax the PGD to ensure convergence to a local minimum under some constraints. This method outperforms all the previous generation algorithms for Computed Tomography (CT). 3) We develop a novel time-dependent deep-image-prior algorithm for modalities that involve a temporal sequence of images. We parameterize them as the output of an untrained neural network fed with a sequence of latent variables. To impose temporal directionality, the latent variables are assumed to lie on a 1D manifold. The network is then tuned to minimize the data fidelity. We obtain state-of-the-art results in dynamic magnetic resonance imaging (MRI) and even recover intra-frame images. 4) We propose a novel reconstruction paradigm for cryo-electron-microscopy (CryoEM) called CryoGAN. Motivated by generative adversarial networks (GANs), we reconstruct a biomolecule's 3D structure such that its CryoEM measurements resemble the acquired data in a distributional sense. The algorithm is pose-or-likelihood-estimation-free, needs no ab initio, and is proven to have a theoretical guarantee of recovery of the true structure. 5) We extend CryoGAN to reconstruct continuously varying conformations of a structure from heterogeneous data. We parameterize the conformations as the output of a neural network fed with latent variables on a low-dimensional manifold. The method is shown to recover continuous protein conformations and their energy landscape.

EPFL2020

Synthesis, modeling, and experimental validation of distributed robotic search

James Pugh

The work of this thesis centers around the research subject of distributed robotic search. Within the field of distributed robotic systems, a task of particular interest is attempting to locate one or more targets in a possibly unknown environment. While numerous studies have proposed and analyzed methods to accomplish this, there is currently a lack of a strong foundation of tools and techniques that can be used to facilitate the development and evaluation of different approaches to distributed robotic search. In this work, we aim to provide such a foundation through tools, methods, models, and analysis of experimentation. An often overlooked aspect of the distributed robotic research process is the development and analysis of tools and modules to be used with robotic systems. These may include plug-ins for realistic robotic simulators, software/hardware systems to track multiple mobile robots in real-time, extension boards for robotic platforms, and the robots themselves. Along with other tools developed and used for this work, we focus particularly on the development, characterization, and validation of a fast, accurate on-board system for relative positioning and communication between robots, a capability which is critical for effective distributed robotic search. Designing individual robot controllers to generate a specific group behavior is a difficult and often counter-intuitive process. A possible alternative to hand-crafting distributed search controllers is automatic synthesis using machine-learning techniques. We explore the effectiveness of using a noise-resistant version of the Particle Swarm Optimization algorithm to optimize the weights of an embedded Artificial Neural Network, allowing the robot to learn obstacle avoidance behavior (a common benchmark for robotic learning techniques); we find that this technique appears to offer superior performance as compared to the canonical approach of using Genetic Algorithms for this type of learning. A method for faster learning using distributed evaluation in a robot team is tested and is found to offer comparable performance using only a fraction of the original learning time. This technique can be used for fast, effective learning and adaptation in a distributed robotic system performing search. The process of designing and analyzing algorithms for distributed robotic systems can be greatly facilitated if models are available to describe the dynamics of the algorithm at an abstract level. Inspired by previous examples in the distributed robotic field, we work to design a model of robotic search that captures the system at different levels of abstraction, ranging from accurately recreating the details of individual robots to describing the entire system as an indivisible whole. To capture the entire search process, we model both the exploration phase, where robots cover an environment in an effort to detect traces of targets, and the localization phase, where robots use target emission sensing to navigate towards the target. The utility of our models is demonstrated by using them to develop an effective technique for the declaration phase of search, where robots decide that a target has been accurately localized and announce its position. In distributed robotics research, it is important that techniques developed with abstracted simulations and models are ultimately validated using real robotic platforms in order to verify their correctness. In that spirit, we run systematic sound search experiments using teams of up to ten real robots. These experiments utilize the tools developed throughout the research process, demonstrate the utility of our learning technique for fast search adaptation, and serve to validate our models of distributed robotic localization. In addition, they allow us to analyze and better understand the subtle dynamics of the search process, providing information which should be useful for any future work on distributed robotic search.