Structure-aware Multi-view 3D Reconstruction of Dislocations in TEM with Message Passing Neural Networks

Efficient analysis of the three-dimensional (3D) shape and distribution of curvilinear crystal defects, namely dislocations, is an open research topic in material science and computer vision. In order to determine the structural and opto-electrical characteristics of the material, accurate 3D reconstructions of dislocations are required. In its current state, an established way to obtain 3D information of dislocations is electron tomography which relies on acquiring dozens of images from tilted sample for wide range of angles. Although tomography yields sufficient results in many cases, it is manually demanding to acquire, register and process dozens of images. In previous works [1, 2], it is shown that a classical stereo reconstruction approach may be applied as an alternative in order to increase reconstruction throughput while providing good performance. A later work [3] employing modern data-driven computer vision models showed that this stereo approach can be largely automatized and be used in various imaging conditions. In this work, we further improve the performance and memory consumption of previous convolutional neural network (CNN) based stereo reconstruction approach and extend it to be applicable in cases where more than two images are available. Our proposed multi-view approach directly incorporates the structural prior of dislocations to the model"s architecture and estimates 3D line segments approximating 3D shape of dislocations. In its core, our method has two stages which are structural connectivity estimation and depth estimation. In first stage, we employ a CNN to estimate affinity of sampled image locations on dislocation cores. This affinity information is used in the later depth estimation stage to eliminate the depth discontinuities on dislocations (Figure1). Unlike previous deep data-driven method, our new approach uses structural knowledge to separate depth estimations of different dislocation segments and yields more accurate reconstructions. In Figure 2, an example affinity estimation is shown for 4 different image locations on the same dislocation. It may be seen that dislocation points that are connected by dislocation segment in 3D results in higher affinity. This information is crucial to connect 3D point estimations and obtain linear structures in 3D.