M/ORIS - Medical / Operating Room Interaction System

The routine use of Computer-Assisted Surgery (CAS) in a wide range of surgical procedures and specialties is being hindered by the lack of suitable user interface for surgical environments. While many computer techniques and software exist to assist surgeons in their tasks, the inability of these tools to fit into existing surgical setups and procedures (both physically and practically) significantly reduces their efficiency and limits their deployment. There are two major drawbacks to existing user interfaces for CAS: (1) they consist mostly of poor adaptations of the existing desktop computer keyboard and mouse interaction paradigm, and (2) they are designed to function just like desktop applications, where the software is the focus of attention of the user, therefore adding to the per-operative workload of the surgeon. I propose a new approach to user interfaces for CAS by introducing M/ORIS, a Medical/ Operating Room Interaction System. M/ORIS is an intelligent collection of sensors and logic modules that are designed to specifically address the need for suitable user interaction in the operating room (OR). M/ORIS provides two layers of interaction that address the fundamental limitations of existing surgical user interfaces: (1) a set of new interaction modalities that are compatible with surgical constraints (i.e. sterile and non-obstructing) and (2) a new interaction paradigm, based on these new modalities, that automates some interaction tasks in order to reduce the interruptions to the surgical process currently required to interact with the computer in the OR. This work describes a detailed analysis of Human-Computer Interaction methods for CAS, and proposes a new approach to Surgeon-Computer Interaction. The design of the new interaction paradigm, as well as the implementation of its building modules, are presented, including a new method to perform real-time, visual detection and tracking of obstructed body parts in indoors environments and a mechanism to perform surgical activity monitoring. The performance of each component of M/ORIS and the behavior of the whole system are then analyzed, and considerations for future improvements are given.

Brain-Computer Interface in Multimedia Systems

Ashkan Yazdani

A brain-computer interface (BCI) is a system that allows a user to communicate with the environment only through cerebral activity, without using muscular output channels. To establish a direct link between the brain and a computer, the electroencephalogram (EEG) signal of a user is measured and then analyzed with the help of signal processing and machine learning algorithms. Once a certain mental activity has been detected by the computer, a response can be displayed on a screen or a command can be sent to a peripheral device, for example a wheelchair or a television. The main application area for BCI is assistive technology for handicapped people. For example, one can imagine artificial limbs controlled by a BCI, a BCI-based spelling device, or an environment control system based on a BCI. Nevertheless, during the last years it has been convincingly shown that communication via a BCI is feasible for able-bodied as well as handicapped users and several new applications such as entertainment and gaming, and neuro-feedback have been proposed for BCI systems. This thesis describes the author’s contributions to the research on brain computer interface systems, focusing on exploring and studying new applications of BCIs especially in multimedia communication domain. After providing a general overview of the EEG signal processing techniques and current BCI systems, three topics of interest have been identified. These topics include BCI for salient image retrieval and image triage, BCI for affect recognition during multimedia consumption, and BCI for studying the EEG correlates of pleasant and unpleasant odors. In the first topic, we present a BCI system, capable of identifying interesting images in an image database. Using this system, images of a database are presented to several users at a fast rate, and the information on whether a given image is salient, is implicitly extracted by means of processing the EEG signals, acquired when users are watching the image sequence. Furthermore, in order to investigate the impact of expertise on a BCI-based salient image retrieval system, the changes in the EEG signals of expert and novice users (with respect to image content) are studied. We show that it is possible to define experimental protocol and judiciously apply signal processing tools such that a practical BCI system can be set up for detection of interest in users during the watching of image sequences. Furthermore, we show that a relatively high retrieval accuracy can be obtained for most of the users of this system. The second presented topic deals with an affective BCI system, which recognizes the emotional states of users While they watch different music video clips. More precisely, the users are asked to watch several emotive music video clips, while their EEG and other peripheral physiological signals are acquired. Signal processing and machine learning algorithms are then developed to infer the emotional state of the user, induced while watching the music video clips. We propose the application of this BCI system in implicit emotional annotation of multimedia contents and sketch a iii iv road map towards developing a hybrid BCI system for emotional search and retrieval of multimedia contents. In the last topic, we present the results of our study on exploring the EEG alterations during the perception of pleasant and unpleasant odorant stimuli. We identify the regions of the brain cortex, which contribute to differentiation of pleasant and unpleasant odors and show that a relatively high accuracy for classification of EEG signals during perception of hedonically different odorant stimuli can be achieved. Furthermore, we identify a novel clinical application for the developed system, namely to study the EEG changes in comatose patients during stimulation with hedonically different odors in order to estimate the depth of coma.

EPFL2012

Comprehensive Interactive Soft Interfaces for Wearable Tactile Feedback

Harshal Arun Sonar

As the field of robotics continues to grow outside the manufacturing environment into our daily lives, the interactions between humans and robots are increasingly becoming close and dynamic. This type of environment requires robots to be less rigid, multi-functional, safer, and compliant with human bodies. Such robotics interfaces are ideally suited for medical rehabilitation, elderly assistance to at-home entertainment devices. However, the traditional rigid robotic devices fail to address some of the design criteria posed for safety, compliance, and physical limitations on the mechanical design. In the case of wearable technologies, the robotic device additionally requires a lightweight, compliant, and safe interface to provide smooth communication and interaction with humans. The traditional robotic systems are fast, accurate, and can handle large torques. However, they are also, application-specific and not suitable as it is for the wearable scenario due to the contradictory design requirements. In the past decade, soft robotics has emerged as a novel approach to solve the complex problems faced by rigid robots using inherent softness and compliant material properties. The design for the future interactive wearable interfaces thus may lie in the intersection of developments in the fields of wearable technology and soft robotics.Designing a wearable interactive interface with soft materials for wearability, portability, cost-effectiveness, and modularity would be one of the ideal solutions to tackle the wearability challenge. It can be a cost-effective solution for at-home assistive rehabilitation or entertainment. It also allows developing novel methods of soft sensing, actuation, control strategies, and human in loop protocols to maximize the utility of inherent material properties and environment around the robotic device.In my Ph.D. research, I develop and create hardware and software towards an immersive interactive soft virtual-tactile environment focusing on the wearability, portability, easy customization, and modularity aspects. I developed a low profile soft pneumatic actuator-skin (SPA-skin) with distributed sensing and actuation capabilities for testing various levels of complex vibrotactile feedback: the multilayer composite of high-sensitivity PZT pressure sensors and vibratory SPA can produce up to 3 N output force at 0-100 Hz. This demonstrated that SPA-skin produces distinctive and dynamic haptic force feedback under modulated varying time, force, and spatial resolution. Furthermore, a complete framework for designing a tactile feedback skin for specific wearable applications is developed, starting from the material selection and actuator design, and concluding with the validation of the subjectâs perceived feedback. As soon as, the SPA-skin is required to be worn by humans, the complexity of the problem multiplies due to lack of proper grounding to ensure the blocked forces. The platform is further optimized for application space in fMRI compliant environment and a user study protocol for somatosensory thresholds is designed. The modulable and controllable nature of SPA-skinâs tactile feedback provides much-needed validation using BOLD signals from brain imaging. SPA-skin and findings presented herein, provide a foundational platform to further investigate the sensitivity of human skin and decipher mechanoreceptor reactions for a diverse range of human-robot interactions and wearable technology.

EPFL2021

Adapting Virtual Embodiment through Reinforcement Learning

Olaf Blanke, Ronan Boulic, Bruno Herbelin, Yawen Hou

In Virtual Reality, having a virtual body opens a wide range of possibilities as the participant's avatar can appear to be quite different from oneself for the sake of the targeted application (e.g. for perspective-taking). In addition, the system can partially manipulate the displayed avatar movement through some distortion to make the overall experience more enjoyable and effective (e.g. training, exercising, rehabilitation). Despite its potential, an excessive distortion may become noticeable and break the feeling of being embodied into the avatar. Past researches have shown that individuals have a relatively high tolerance to movement distortions and a great variability of individual sensitivities to distortions. In this paper, we propose a method taking advantage of Reinforcement Learning (RL) to efficiently identify the magnitude of the maximum distortion that does not get noticed by an individual (further noted the detection threshold). We show through a controlled experiment with subjects that the RL method finds a more robust detection threshold compared to the adaptive staircase method, i.e. it is more able to prevent subjects from detecting the distortion when its amplitude is equal or below the threshold. Finally, the associated majority voting system makes the RL method able to handle more noise within the forced choices input than adaptive staircase. This last feature is essential for future use with physiological signals as these latter are even more susceptible to noise. It would then allow to calibrate embodiment individually to increase the effectiveness of the proposed interactions.