Advanced Mixed Reality Technologies for Surveillance and Risk Prevention Applications

We present a system that exploits advanced Mixed and Virtual Reality technologies to create a surveillance and security system that could be also extended to define emergency prevention plans in crowdy environments. Surveillance cameras are carried by a mini Blimp which is tele-operated using an innovative Virtual Reality interface with haptic feedback. An interactive control room (CAVE) receives multiple video streams from airborne and fixed cameras. Eye tracking technology allows for turning the user’s gaze into the main interaction mechanism; the user in charge can examine, zoom and select specific views by looking at them. Video streams selected at the control room can be redirected to agents equipped with a PDA. On-field agents can examine the video sent by the control center and locate the actual position of the airborne cameras in aGPS-driven map. The aerial video would be augmented with real-time 3D crowd to create more realist risk and emergency prevention plans. The prototype we present shows the added value of integrating AR/VR technologies into a complex application and opens up several research directions in the areas of tele-operation, Multimodal Interfaces, simulation, risk and emergency prevention plans, etc.

Advanced virtual reality technologies for surveillance and security applications

Renaud Ott, Daniel Thalmann, Frédéric Vexo

We present a system that exploits advanced Virtual Reality technologies to create a surveillance and security system. Surveillance cameras are carried by a mini Blimp which is tele-operated using an innovative Virtual Reality interface with haptic feedback. An interactive control room (CAVE) receives multiple video streams from airborne and fixed cameras. Eye tracking technology allows for turning the user's gaze into the main interaction mechanism; the user in charge can examine, zoom and select specific views by looking at them. Video streams selected at the control room can be redirected to agents equipped with a PDA. On-field agents can examine the video sent by the control center and locate the actual position of the airborne cameras in a GPS-driven map. The PDA interface reacts to the user's gestures. A tilt sensor recognizes the position in which the PDA is held and adapts the interface accordingly. The prototype we present shows the added value of integrating VR technologies into a complex application and opens up several research directions in the areas of tele-operation, Multimodal Interfaces, etc. Copyright © 2006 by the Association for Computing Machinery, Inc.

2006

Application of 3D range camera in virtual human-computer interfaces

Huan Du

In this work, we propose new ways to employ 3D-ranging systems in advanced human-computer interfaces and show that they can be used for precise hand-tracking systems aiming at virtual keyboard or mouse applications. We first implement a Structured Light (SL) based 3D-ranging system as the testbed for acquiring depth information. The ranging system uses an off-the-shelf projector combined with an industrial color camera, where the projector projects well-designed line patterns onto the scene and the camera captures the reflected light from the scene and the target. The depth information is recovered from the deformed line pattern and the pre-calibrated projector-camera system. With this system we can capture dense depth maps with moderate resolution and high frame rate as the simulated input to the hand-tracking system. We then explore an approach to 3D hand matching that combines a detailed 3D hand model (skeletal + polygonal model) with a Physical-based Model Fitting technique (PMF). Virtual forces are assigned between the depth measurements and the polygonal vertices of the model. The matching problem is formulated as the minimization of the distances between the measurement and the model surface with the guidance of virtual forces. This can employ 3D information simply and directly onto the parameter space, and yield good results under challenging conditions. To demonstrate the effectiveness of the approach we show that we can track hand motions such as typing or grasping frame by frame. The major limitation of this method is that it requires a good initialization of the hand model and the computation time to achieve convergence with good fitting results is quite long. We therefore investigated more complex hand models and feature-based matching techniques. A three-level hand model (skeletal + primitive + polygonal model) is proposed for the Multi-Level Feature Matching approach (MLFM). The matching is carried out from the bottom to top levels of different hand presentations, and distinctive features for each level extracted from both the measurements and the model are matched with Scaled Conjugate Gradient method (SCG). The matching result from lower level is used as the initial state of the model parameters for higher level matching. This hierarchical structure speeds up the optimization process and enhances the precision of the matching result. We demonstrate our approach with the same hand motion as in the previous method and compare the performance of these two methods. We show that the MLFM method is faster and more robust to hand occlusions than the PMF method. Finally, we present a more powerful approach based on the Back-Constrained Gaussian Process Latent Variable Model (BC-GPLVM) that import prior information of categorized hand motion such as typing. BC-GPLVM provides a low-dimensional embedding of hand motion data, with a density function that gives higher probability to gestures and motions close to normal typing motion. This model is learned from a small set of clusters of vast captured motion data. The prior model is then combined with the previous multi-level models and features to generate a uniform cost function for optimization. We show the tracking result to demonstrate that this method can be used for human-computer interfaces requiring precise 3D hand tracking.

EPFL2008

Adapting Virtual Embodiment through Reinforcement Learning

Olaf Blanke, Ronan Boulic, Bruno Herbelin, Yawen Hou

In Virtual Reality, having a virtual body opens a wide range of possibilities as the participant's avatar can appear to be quite different from oneself for the sake of the targeted application (e.g. for perspective-taking). In addition, the system can partially manipulate the displayed avatar movement through some distortion to make the overall experience more enjoyable and effective (e.g. training, exercising, rehabilitation). Despite its potential, an excessive distortion may become noticeable and break the feeling of being embodied into the avatar. Past researches have shown that individuals have a relatively high tolerance to movement distortions and a great variability of individual sensitivities to distortions. In this paper, we propose a method taking advantage of Reinforcement Learning (RL) to efficiently identify the magnitude of the maximum distortion that does not get noticed by an individual (further noted the detection threshold). We show through a controlled experiment with subjects that the RL method finds a more robust detection threshold compared to the adaptive staircase method, i.e. it is more able to prevent subjects from detecting the distortion when its amplitude is equal or below the threshold. Finally, the associated majority voting system makes the RL method able to handle more noise within the forced choices input than adaptive staircase. This last feature is essential for future use with physiological signals as these latter are even more susceptible to noise. It would then allow to calibrate embodiment individually to increase the effectiveness of the proposed interactions.

2021

Application of 3D range camera in virtual human-computer interfaces

Huan Du

EPFL2008