Empowering Low-Cost CMOS Cameras by Image Processing to Reach Comparable Results with Costly CCDs

Despite the huge research effort to improve the performance of the complementary metal oxide semiconductor (CMOS) image sensors, charge-coupled devices (CCDs) still dominate the cell biology-related conventional fluorescence microscopic imaging market where low or ultra-low noise imaging is required. A detailed comparison of the sensor specifications and performance is usually not provided by the manufacturers which leads the end users not to go out of the habitude and choose a CCD camera instead of a CMOS one. However, depending on the application, CMOS cameras, when empowered by image processing algorithms, can become cost-efficient solutions for conventional fluorescence microscopy. In this paper, we introduce an application-based comparative study between the default CCD camera of an inverted microscope (Nikon Ti-S Eclipse) and a custom-designed CMOS camera and apply efficient image processing algorithms to improve the performance of CMOS cameras. Quantum micro-bead samples (emitting fluorescence light at different intensity levels), breast cancer diagnostic tissue cell samples, and Caco-2 cell samples are imaged by both CMOS and CCD cameras. The results are provided to show the reliability of CMOS camera processed images and finally to be of assistance when scientists select their cameras for desired applications.

Quantitative Comparison of Commercial CCD and Custom-Designed CMOS Camera for Biological Applications

René Beuchat, Sandro Carrara, Giovanni De Micheli, Julien Michel Ghaye, Gözen Köklü, Yusuf Leblebici

In biological applications and systems where even the smallest details have a meaning, CCD cameras are mostly preferred and they hold most of the market share despite their high costs. In this paper, we propose a custom-designed CMOS camera to compete with the default CCD camera of an inverted microscope for fluorescence imaging. The custom-designed camera includes a commercially available mid-performance CMOS image sensor and a Field-Programmable Gate Array (FPGA) based hardware platform (FPGA4U). The high cost CCD camera of the microscope is replaced by the custom-designed CMOS camera and the two are quantitatively compared for a specific application where an Estrogen Reception (ER) expression in breast cancer diagnostic samples that emits light at 665nm has been imaged by both cameras. The gray-scale images collected by both cameras show a very similar intensity distribution. In addition, normalized white pixels after thresholding resulted in 4.96% for CCD and 3.38% for CMOS. The results and images after thresholding show that depending on the application even a mid-performance CMOS camera can provide enough image quality when the target is localization of fluorescent stained biological details. Therefore the cost of the cameras can be drastically reduced while benefiting from the inherent advantages of CMOS devices plus adding more features and flexibility to the camera systems with FPGAs.

2012

Beyond Cameras: Real-Time High-Resolution 3D and Panoramic MultiView Vision Systems with Their Applications

Bilal Demir

The cameras are invented by imitating the human visual system to capture the scene. The camera technologies have been substantially advanced in recent years. 108 MP resolution with 100x hybrid zoom has become standard features for smartphone flagships. In spite of the recent developments, the cameras are still restricted in terms of FoV, depth perception, pixel size, lens system, and more. That is why multicamera systems have progressively become prevalent. The multicamera systems can be a remedy to overcome the limitations that single cameras hold. The FoV can be raised to

360^\circ

through image stitching. The depth can be computed by stereo matching methods. The lens flaws and imperfections can be handled thanks to computer vision and image processing algorithms. Significant computational power is required to process a vast amount of pixels coming from multiple cameras. GPU and FPGA are promising platforms to implement computer vision and video processing applications due to their sophisticated parallelization features. FPGA platforms present more prevailing features, especially for real-time and portable vision applications. FPGAs provide less latency as they hold the connection to image sensors from a low level. FPGA enables the implementation of a system architecture dedicated to the target application. The compact systems can be designed by designing custom PCBs using only necessary ports. Also, FPGAs consume lower power and cheaper option compared to GPUs. On the other hand, GPUs propose more versatility and easy design and upgrade time. In the light of these observations, software and hardware integrated real-time high-resolution multi-view 3D and panoramic systems are presented in the scope of this thesis. Firstly, the depth estimation system is presented in the first part of the thesis. The proposed depth estimation system runs in real-time performance for up to 2K depth map resolution. The system adopts the trinocular scheme to address the occlusion problem. The pixel correspondence challenge in textureless-regions, from which the conventional stereo matching-based depth estimation systems suffer, is tackled by projecting artificial patterns through the integrated pico-projector. The application-specific system architecture ensures the high-performance depth map streaming. Secondly, the drone detection and tracking system is presented. The proposed drone detection system is capable of simultaneously monitoring 360° environment and detecting the drone from a long-range in real-time performance. The distributed architecture design enables ultra-high-resolution image processing. data coming from the hardware part. The GPU design is opted due to its high level of parallelization capability. The proposed system is appropriate to be employed for surveillance applications such as drone detection, passive radar system, vast terrain, and border control applications. Thirdly, the 3D stereoscopic panorama construction system is presented. The proposed system generated 2 separate panoramas for the left and right eyes to achieve 3D perception. The system offers the novel camera arrangement and the 3D panorama generation algorithm. The cameras are positioned to minimize the intra-panorama parallax while raising the inter-panorama parallax to augment 3D perception. Finally, the real-time vision systems are discussed with their pros and cons, and future predictions are presented in the conclusion part of the thesis.

EPFL2020

Augmented reality for non-rigid surfaces

Julien Pilet

Augmented Reality (AR) is the process of integrating virtual elements in reality, often by mixing computer graphics into a live video stream of a real scene. It requires registration of the target object with respect to the cameras. To this end, some approaches rely on dedicated hardware, such as magnetic trackers or infra-red cameras, but they are too expensive and cumbersome to reach a large public. Others are based on specifically designed markers which usually look like bar-codes. However, they alter the look of objects to be augmented, thereby hindering their use in application for which visual design matters. Recent advances in Computer Vision have made it possible to track and detect objects by relying on natural features. However, no such method is commonly used in the AR community, because the maturity of available packages is not sufficient yet. As far as deformable surfaces are concerned, the choice is even more limited, mainly because initialization is so difficult. Our main contribution is therefore a new AR framework that can properly augment deforming surfaces in real-time. Its target platform is a standard PC and a single webcam. It does not require any complex calibration procedure, making it perfectly suitable for novice end-users. To satisfy to the most demanding application designers, our framework does not require any scene engineering, renders virtual objects illuminated by real light, and let real elements occlude virtual ones. To meet this challenge, we developed several innovative techniques. Our approach to real-time registration of a deforming surface is based on wide-baseline feature matching. However, traditional outlier elimination techniques such as RANSAC are unable to handle the non-rigid surface's large number of degrees of freedom. We therefore proposed a new robust estimation scheme that allows both 2–D and 3–D non-rigid surface registration. Another issue of critical importance in AR to achieve realism is illumination handling, for which existing techniques often require setup procedures or devices such as reflective spheres. By contrast, our framework includes methods to estimate illumination for rendering purposes without sacrificing ease of use. Finally, several existing approaches to handling occlusions in AR rely on multiple cameras or can only deal with occluding objects modeled beforehand. Our requires only one camera and models occluding objects at runtime. We incorporated these components in a consistent and flexible framework. We used it to augment many different objects such as a deforming T-shirt or a sheet of paper, under challenging conditions, in real-time, and with correct handling of illumination and occlusions. We also used our non-rigid surface registration technique to measure the shape of deformed sails. We validated the ease of deployment of our framework by distributing a software package and letting an artist use it to create two AR applications.

EPFL2008