Spatially-Variant CNN-Based Point Spread Function Estimation for Blind Deconvolution and Depth Estimation in Optical Microscopy

Optical microscopy is an essential tool in biology and medicine. Imaging thin, yet non-flat objects in a single shot (without relying on more sophisticated sectioning setups) remains challenging as the shallow depth of field that comes with high-resolution microscopes leads to unsharp image regions and makes depth localization and quantitative image interpretation difficult. Here, we present a method that improves the resolution of light microscopy images of such objects by locally estimating image distortion while jointly estimating object distance to the focal plane. Specifically, we estimate the parameters of a spatially-variant Point Spread Function (PSF) model using a Convolutional Neural Network (CNN), which does not require instrument- or object-specific calibration. Our method recovers PSF parameters from the image itself with up to a squared Pearson correlation coefficient of 0.99 in ideal conditions, while remaining robust to object rotation, illumination variations, or photon noise. When the recovered PSFs are used with a spatially-variant and regularized Richardson-Lucy (RL) deconvolution algorithm, we observed up to 2.1 dB better Signal-to-Noise Ratio (SNR) compared to other Blind Deconvolution (BD) techniques. Following microscope-specific calibration, we further demonstrate that the recovered PSF model parameters permit estimating surface depth with a precision of $\mathrm {2 \mu \text {m} }$ and over an extended range when using engineered PSFs. Our method opens up multiple possibilities for enhancing images of non-flat objects with minimal need for a priori knowledge about the optical setup.

Weakly Supervised Deep Learning Methods for Biomicroscopy

Adrian Shajkofci

Optical microscopy, an invaluable tool in biology and medicine to observe and quantify cellular function, organ development, or disease mechanisms, requires constant trade-offs between spatial, temporal, and spectral resolution, invasiveness, acquisition time, and post-processing effort. Deep learning technologies have enabled multiple applications that are transforming our day-to-day routines, including the way we approach microscopy. Yet despite the ever-increasing computational power, it is often the lack of labeled training data that is the limiting factor for wide adoption in this domain. Annotating data is often a lengthy and expensive task, since it involves tedious work, generally by skilled experts. In this thesis, I explored "weakly supervised" learning methods targeted at a variety of applications to enhance microscopy images and extract physical information from a single image. The specificity of these "weakly supervised" methods is the fact that they use very little prior information about the image in order to keep the effort to annotate training data as low as possible. Specifically, I reduced the dimensionality of the learning problem by targeting the experiment towards estimating the parameters of a spatially-variant point-spread function (PSF) model using a convolutional neural network (CNN), which does not require instrument- or object-specific calibration. Using such a model permitted to simulate realistically accurate training data that could be generalized, once the model was trained, to real microscopy images. I extensively benchmarked different network architectures, training datasets and simulation modalities towards the optimal PSF prediction performance and robustness to image degradation. Starting from the estimated PSF model parameters, I developed a variety of applications, such as a semi-blind spatially-variant deconvolution method for image deblurring and enhancement, a robust and fast microscopy auto-focus, a method for the estimation of the object surface from a single 2D image, and a method for the estimation of the object velocity in a fluid, all of them with minimal need for a priori knowledge about the optical setup.

EPFL2021

Weakly Supervised Deep Learning Methods for Biomicroscopy

Adrian Shajkofci

EPFL2021