Structure-aware Multi-view 3D Reconstruction of Dislocations in TEM with Message Passing Neural Networks

Efficient analysis of the three-dimensional (3D) shape and distribution of curvilinear crystal defects, namely dislocations, is an open research topic in material science and computer vision. In order to determine the structural and opto-electrical characteristics of the material, accurate 3D reconstructions of dislocations are required. In its current state, an established way to obtain 3D information of dislocations is electron tomography which relies on acquiring dozens of images from tilted sample for wide range of angles. Although tomography yields sufficient results in many cases, it is manually demanding to acquire, register and process dozens of images. In previous works [1, 2], it is shown that a classical stereo reconstruction approach may be applied as an alternative in order to increase reconstruction throughput while providing good performance. A later work [3] employing modern data-driven computer vision models showed that this stereo approach can be largely automatized and be used in various imaging conditions. In this work, we further improve the performance and memory consumption of previous convolutional neural network (CNN) based stereo reconstruction approach and extend it to be applicable in cases where more than two images are available. Our proposed multi-view approach directly incorporates the structural prior of dislocations to the model"s architecture and estimates 3D line segments approximating 3D shape of dislocations. In its core, our method has two stages which are structural connectivity estimation and depth estimation. In first stage, we employ a CNN to estimate affinity of sampled image locations on dislocation cores. This affinity information is used in the later depth estimation stage to eliminate the depth discontinuities on dislocations (Figure1). Unlike previous deep data-driven method, our new approach uses structural knowledge to separate depth estimations of different dislocation segments and yields more accurate reconstructions. In Figure 2, an example affinity estimation is shown for 4 different image locations on the same dislocation. It may be seen that dislocation points that are connected by dislocation segment in 3D results in higher affinity. This information is crucial to connect 3D point estimations and obtain linear structures in 3D.

Resolving Ambiguities in Monocular 3D Reconstruction of Deformable Surfaces

Aydin Varol

In this thesis, we focus on the problem of recovering 3D shapes of deformable surfaces from a single camera. This problem is known to be ill-posed as for a given 2D input image there exist many 3D shapes that give visually identical projections. We present three methods which make headway towards resolving these ambiguities. We believe that our work represents a significant step towards making surface reconstruction methods of practical use. First, we propose a surface reconstruction method that overcomes the limitations of the state-of-the-art template-based and non-rigid structure from motion methods. We neither track points over many frames, nor require a sophisticated deformation model, or depend on a reference image. In our method, we establish correspondences between pairs of frames in which the shape is different and unknown. We then estimate homographies between corresponding local planar patches in both images. These yield approximate 3D reconstructions of points within each patch up to a scale factor. Since we consider overlapping patches, we can enforce them to be consistent over the whole surface. Finally, a local deformation model is used to fit a triangulated mesh to the 3D point cloud, which makes the reconstruction robust to both noise and outliers in the image data. Second, we propose a novel approach to recovering the 3D shape of a deformable surface from a monocular input by taking advantage of shading information in more generic contexts than conventional Shape-from-Shading (SfS) methods. This includes surfaces that may be fully or partially textured and lit by arbitrarily many light sources. To this end, given a lighting model, we learn the relationship between a shading pattern and the corresponding local surface shape. At run time, we first use this knowledge to recover the shape of surface patches and then enforce spatial consistency between the patches to produce a global 3D shape. Instead of treating texture as noise as in many SfS approaches, we exploit it as an additional source of information. We validate our approach quantitatively and qualitatively using both synthetic and real data. Third, we introduce a constrained latent variable model that inherently accounts for geometric constraints such as inextensibility defined on the mesh model. To this end, we learn a non-linear mapping from the latent space to the output space, which corresponds to vertex positions of a mesh model, such that the generated outputs comply with equality and inequality constraints expressed in terms of the problem variables. Since its output is encouraged to satisfy such constraints inherently, using our model removes the need for computationally expensive methods that enforce these constraints at run time. In addition, our approach is completely generic and could be used in many other different contexts as well, such as image classification to impose separation of the classes, and articulated tracking to constrain the space of possible poses.

EPFL2012

Stereo-vision three-dimensional reconstruction of curvilinear structures imaged with a TEM

Marco Cantoni, Sandro Ivo Sebastiano De Zanet, Pascal Fua, Cécile Hébert, Antoine Letouzey, Guillaume Lucas, Emadeddin Oveisi

Deriving accurate three-dimensional (3-D) structural information of materials at the nanometre level is often crucial for understanding their properties. Tomography in transmission electron microscopy (TEM) is a powerful technique that provides such information. It is however demanding and sometimes inapplicable, as it requires the acquisition of multiple images within a large tilt arc and hence prolonged exposure to electrons. In some cases, prior knowledge about the structure can tremendously simplify the 3-D reconstruction if incorporated adequately. Here, a novel algorithm is presented that is able to produce a full 3-D reconstruction of curvilinear structures from stereo pair of TEM images acquired within a small tilt range that spans from only a few to tens of degrees. Reliability of the algorithm is demonstrated through reconstruction of a model 3-D object from its simulated projections, and is compared with that of conventional tomography. This method is experimentally demonstrated for the 3-D visualization of dislocation arrangements in a deformed metallic micro-pillar. (C) 2017 Published by Elsevier B.V.

Elsevier2017

Microfluidics by Additive Manufacturing for Wearable Biosensors: A Review

Sandro Carrara, Christian Enz, Mahshid Alsadat Padash

Wearable devices are nowadays at the edge-front in both academic research as well as in industry, and several wearable devices have been already introduced in the market. One of the most recent advancements in wearable technologies for biosensing is in the area of the remote monitoring of human health by detection on-the-skin. However, almost all the wearable devices present in the market nowadays are still providing information not related to human ‘metabolites and/or disease’ biomarkers, excluding the well-known case of the continuous monitoring of glucose in diabetic patients. Moreover, even in this last case, the glycaemic level is acquired under-the-skin and not on-the-skin. On the other hand, it has been proven that human sweat is very rich in molecules and other biomarkers (e.g., ions), which makes sweat a quite interesting human liquid with regards to gathering medical information at the molecular level in a totally non-invasive manner. Of course, a proper collection of sweat as it is emerging on top of the skin is required to correctly convey such liquid to the molecular biosensors on board of the wearable system. Microfluidic systems have efficiently come to the aid of wearable sensors, in this case. These devices were originally built using methods such as photolithographic and chemical etching techniques with rigid materials. Nowadays, fabrication methods of microfluidic systems are moving towards three-dimensional (3D) printing methods. These methods overcome some of the limitations of the previous method, including expensiveness and non-flexibility. The 3D printing methods have a high speed and according to the application, can control the textures and mechanical properties of an object by using multiple materials in a cheaper way. Therefore, the aim of this paper is to review all the most recent advancements in the methods for 3D printing to fabricate wearable fluidics and provide a critical frame for the future developments of a wearable device for the remote monitoring of the human metabolism directly on-the-skin.