Imaging through multimode fibers using deep learning: The effects of intensity versus holographic recording of the speckle pattern

Information transmission through multimode fibers (MMFs) has been a topic of great interest for many years. Deep learning algorithms have been applied successfully to MMFs in particular to fiber endoscopy. In this work, we show how Deep Neural Networks (DNNs) can be a versatile technique for classification and recovery of input images that have been significantly distorted while propagating along the MMF forming a speckle pattern. A comparison between holographic and intensity-only recording of the speckle output, which is used as an input to the DNNs, shows that high performance can be achieved without having the full field information (amplitude and phase). Impressive reconstruction fidelity and classification accuracy of the fiber inputs from the intensity-only images of the speckle patterns is reported.

Wavefront shaping and deep learning in fiber endoscopy

Eirini Kakkava

Fiber endoscopy plays an important role in the clinical diagnosis and treatment processes involved in modern medicine. Thin fiber probes can relay information from confined places in the human body that are inaccessible for conventional bulky microscopes. Therefore, they can provide a minimally invasive platform for optical diagnosis assisting the fast recovery of patients. Aiming for more compact fiber endoscope devices, we investigate different types of fibers that can integrate not only superior imaging modalities but also microsurgery capabilities while maintaining an ultrathin size (less than 400um). In our experiments, the imaging potential of multimode and multicore fibers is assessed. On the one hand, multimode fibers provide high information capacity for small core sizes but scramble the input field after its propagation through the fiber length, while on the other hand, multicore fibers are the most common choice for endoscopic probes since they are able to directly relay images from the inspection location to the observation side. However, the final images produced by multicore fibers are of poor quality because of the discretization effect induced by the individual core sampling and the final resolution is related to the core spacing. We show that each fiber type can be selected based on the desired application and the imaging result can be significantly improved. Two approaches for imaging through the different fibers are investigated. In the first part of the present thesis, wavefront shaping using the transmission matrix approach to generate a focus spot at the distal fiber side is presented. The limitations concerning the maximum peak intensity guided through the different endoscopes is investigated for high power femtosecond pulses where nonlinear optical phenomena can hinder the overall performance of the system. Femtosecond laser ablation is demonstrated through multimode fibers for a range of materials. Furthermore, laser ablation is combined with two-photon fluorescence imaging in the same multimode fiber endoscope showing for the first time selective tissue modifications at a cellular level. In the second part of this thesis, deep learning of light propagation through the two fiber types is studied. Datasets of known input images and their respective fiber output images are generated to train deep neural network algorithms to map the fiber output to the fiber input for either classification or image reconstruction purposes. The deep neural networks show impressive performance to recover the information from intensity-only images of the speckle patterns emerging from multimode fibers, removing the need to record the full field information. Moreover, deep learning for multimode fiber imaging proved to be resilient to perturbations related to mechanical, thermal and even wavelength drifts introduced during the measurements. In the case of multicore fibers, deep neural networks are trained to remove pixelation artifacts from the final image and resolve features with an improved resolution. Finally, deep learning is employed to transform a bright field imaging-based commercial endoscope to a phase contrast imaging probe by training deep neural networks to map the intensity-only recorded image to its corresponding phase map.

EPFL2020

Deep Neural Networks for seeing through multimode fibers

Navid Borhani, Eirini Kakkava, Christophe Moser, Demetri Psaltis

Image delivery through multimode fibers (MMFs) suffers from modal scrambling which results in a speckle pattern at the fiber output. In this work, we use Deep Neural Networks (DNNs) for recovery and/or classification of the input image from the intensity-only images of the speckle patterns at the distal end of the fiber. We train the DNNs using 16,000 images of handwritten digits of the MNIST database and we test the accuracy of classification and reconstruction on another 2,000 new digits. Very positive results and robustness were observed for up to 1 km long MMF showing 90% reconstruction fidelity. The classification accuracy of the system for different inputs (phase-only, amplitude-only, hologram intensity etc.) to the DNN classifier was also tested.

SPIE-INT SOC OPTICAL ENGINEERING2019

Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media

Navid Borhani, Eirini Kakkava, Damien Claude-Marie Loterie, Christophe Moser, Demetri Psaltis, Babak Rahmani, Ugur Tegin

The output of physical systems, such as the scrambled pattern formed by shining the spot of a laser pointer through fog, is often easily accessible by direct measurements. However, selection of the input of such a system to obtain a desired output is difficult, because it is an ill-posed problem; that is, there are multiple inputs yielding the same output. Information transmission through scattering media is an example of this problem. Machine learning approaches for imaging have been implemented very successfully in photonics to recover the original input phase and amplitude objects of the scattering system from the distorted intensity diffraction pattern outputs. However, controlling the output of such a system, without having examples of inputs that can produce outputs in the class of the output objects the user wants to produce, is a challenging problem. Here, we propose an online learning approach for the projection of arbitrary shapes through a multimode fibre when a sample of intensity-only measurements is taken at the output. This projection system is nonlinear, because the intensity, not the complex amplitude, is detected. We show an image projection fidelity as high as similar to 90%, which is on par with the gold-standard methods that characterize the system fully by phase and amplitude measurements. The generality and simplicity of the proposed approach could potentially provide a new way of target-oriented control in real-world applications when only partial measurements are available.

SPRINGERNATURE2020

Wavefront shaping and deep learning in fiber endoscopy

Eirini Kakkava

EPFL2020