Supernumerary Robotic Arm for Three-Handed Surgical Application: Behavioral Study and Design of Human-Machine Interface

In surgical to industrial manipulation, the operator needs assistance for tasks requiring more than two hands. Teamwork may be the source of errors and inefficiency, especially if the assistant is a novice or unfamiliar with the main operator. The need for assistance may become problematic in case of lack of human resources e.g. in emergency surgical cases in the late hours of the night. Our objective is to improve the surgeon's autonomy and dexterity by a robotic arm under his own control. Although a number of robotic instrument holders have been developed, the best way to control such devices is still an open question. No behavioral study has been conducted on the best control strategy and human performance in three-handed tasks. We have selected the foot for commanding the third arm on the basis of a literature review. A series of experiments in virtual environments has been conducted to study the feasibility of this choice. The first experiment compares performance in the same task using two or three hands. Results show that three-handed manipulation is preferred to two-handed manipulation in demanding tasks. The second experiment investigated the type of tasks to be aimed in three-handed manipulation and the learning curve of users. Moving the hands and a foot simultaneously in opposite directions was perceived as difficult compared to a more active task with liberty in choosing the limbs coordination. Limbs were moved in parallel rather than serially. The performance improved within a few minutes of practice. Also, the sense of ownership improved constantly during the experiment. Two other experiments were aimed at handling the endoscope in laparoscopic surgery. Surgeons and medical students participated in these studies. Residents had a more positive approach towards foot usage and performed better compared to more experienced surgeons. This proves that the best training period for surgeons to use a foot controlled robotic arm is during their residency. A realistic virtual abdominal cavity has been developed for the last experiment. This had a positive influence on the participants' performance and emphasizes the importance of using a familiar context for training such a "three-handed surgery". Finally, two different foot interfaces were developed to investigate the most intuitive third arm commanding strategy. A robotic arm is hence controlled by the foot's translation or rotation in one interface (isotonic interface), and by force or torque in the other one (isometric interface). An experimental behavioral study was conducted to compare the two devices. Isometric rate control was preferred to isotonic position control due to the lower physical burden and higher movement accuracy of the robot. It was shown that the proposed device for isometric rate control could be used for intuitive control of four DoFs of a slave robotic arm. This thesis is the first step in a systematic investigation of a three-handed manipulation, two biological hands and a foot controlled robotic assistant. Findings suggest a high potential in using the foot to become more autonomous in surgery as well as other fields. Users can learn the control paradigm in a short period of time with little mental and physical burden. We expect the developed foot interfaces to be the basis of future development of more intuitive control interfaces. We believe that foot controlled robotic arms will be commonly used in various surgical as well as industrial applications.

Control of a Supernumerary Robotic Hand by Foot: An Experimental Study in Virtual Reality

Elahe Abdi, Hannes Bleuler, Mohamed Bouri, Etienne Burdet

In the operational theater, the surgical team could highly benefit from a robotic supplementary hand under the surgeon's full control. The surgeon may so become more autonomous; this may reduce communication errors with the assistants and take over difficult tasks such as holding tools without tremor. In this paper, we therefore examine the possibility to control a third robotic hand with one foot's movements. Three experiments in virtual reality were designed to assess the feasibility of this control strategy, the learning curve of the subjects in different tasks and the coordination of foot movements with the two natural hands. Results show that the limbs are moved simultaneously, in parallel rather than serially. Participants' performance improved within a few minutes of practice without any specific difficulty to complete the tasks. Subjective assessment by the subjects indicated that controlling a third hand by foot has been easy and required only negligible physical and mental efforts. The sense of ownership was reported to improve through the experiments. The mental burden was not directly related to the level of motion required by a task, but depended on the type of activity and practice. The most difficult task was moving two hands and foot in opposite directions. These results suggest that a combination of practice and appropriate tasks can enhance the learning process for controlling a robotic hand by foot.

Public Library of Science2015

Active: a unified platform for building intelligent applications

Didier Guzzoni

Computers have become affordable, small, omnipresent and are often connected to the Internet. However, despite the availability of such rich environments, user interfaces have not been adapted to fully leverage their potential. In our view, user interfaces will evolve to become more than simple tools that act using a click-and-act paradigm – people want software that can act as assistants to whom tasks can be delegated. This new type of software will provide us with more user-centric systems, able to interact naturally with human users and with the information environment. Although progress has been made in this direction over the last decade, current research in the field has shown that building intelligent assistants is a complex task that requires expertise in many fields, ranging from artificial intelligence to core software and hardware engineering. The difficulty of deeply integrating all the technologies and AI methodologies required to produce robust intelligent assistants has greatly limited the impact and widespread adoption of this form of software. In this thesis, we propose to design, implement and evaluate a new methodology and associated tool suite aimed to ease and accelerate the development of intelligent assistant software. Active, our solution, introduces the original concept of Active Ontologies, and combines it with a service oriented architecture to serve as the foundation for user-centric applications. The Active software suite features a programming editor, a runtime server and associated management tools. Using this unified platform, we developed techniques for rapidly creating intelligent assistant applications that weave together language processing, process modeling and dynamic service orchestration. To validate our approach, three prototypes have been implemented and evaluated. First, we created an assistant that helps mobile users retrieve data and access online services. The system allows users to fetch information through natural language dialog about restaurants, hotels, points of interest, flights status and weather forecasts. As a second prototype, we have implemented an assistant to help surgeons in the context of the operating room. The system uses a combination of voice and gesture recognition to help surgeons navigate through pre-operative information, visualize and control a live video stream coming from an endoscope mounted on a robotic arm. Lastly, we have created a system that helps organize meetings through emails and instant messages with the organizer and attendees. These three prototypes are deployed in very different application domains, and yet are built with the same tools and methods; this demonstrates the flexibility and versatility of our approach. We conducted a user study to validate our claim that, in specific domains, intelligent assistant software can perform better than conventional software approaches. First, we compared our mobile assistant against Google Mobile™, the leading commercially available search application for mobile users. We asked a population of users to accomplish ten travel-related tasks with both systems. Results show that, for the travel domain, our assistant-based system performs significantly better both in terms of effectiveness and time to completion than the more conventional, keyword-based search engine. In the medical domain, we asked a small population of surgeons to accomplish a set of tasks with our second prototype. After providing us with a positive feedback, surgeons used our prototype to discuss and experiment with innovative forms of user interaction in the context of the operating room. We also performed evaluations to support our claim that our unified approach accelerates the software development of intelligent assistants. First, we gave basic training on the Active platform to a population of software developers, asking them to go through a simple tutorial. Once trained, they were able to successfully program the core component of the mobile assistant prototype in under two hours. As a second validation, the meeting organizing assistant was designed, implemented and tested within three working days – comparable systems published in the literature were the product of significantly more time and resource investment. This demonstrates that a system encompassing numerous AI and software development aspects, including language processing, multiple modalities, contextual dialog, processing logic and backend services integration, can be rapidly implemented and tested using the Active framework. Our work shows that for many task-oriented domains, intelligent assistant applications can be significantly more effective than conventional systems. Through our Active Ontology approach and associated methodologies and tools, we have shown that complex intelligent assistant applications, perhaps for the first time, can be developed with low cost and high reliability. In addition to receiving a positive echo from the research world, our work is also causing ripples in the commercial world.

EPFL2008

Personalized Body-Machine Interfaces for Advanced Human-Robot Interaction

Matteo Macchini

Robotic systems are becoming more and more pervasive in modern industrial, scientific and personal activities through recent years and will play a fundamental role in the future society. Despite their increasing level of automation, teleoperation is still needed in many robotic applications. Human-Robot Interfaces (HRIs) are a central element in the diffusion of robots and are a crucial component to make them available to the use of most people. However, standard interfaces such as remote controllers still need time and extensive training to be proficiently mastered by users. Body-Machine Interfaces (BoMIs) represent a promising alternative to such devices as they leverage the innate ability of humans to finely control their movements. This thesis aims to provide new insights about humans' natural motor behaviors when interfacing with robots and to propose novel approaches for the implementation of HRIs based on body motion. The core contribution of this work is the design, realization and qualification of machine learning-based methods to translate the motion of a person into control inputs for a robot. The methods described here allowed naive users to effectively control real and simulated robots based on the processing of their body movements with a faster learning rate compared to classic interfaces. We extended our core algorithm to leverage knowledge about the natural organization of human motion resulting a more general method that was successfully applied for the control of a set of morphologically different machines. Through the implementation of an online adaptive functionality, we enabled users to modify their motor behaviour during teleoperation resulting in a lower perceived physical and cognitive workload and augmented performance. Human factors such as the sense of presence in virtual or distant environments, or the motor characteristics of distinct body areas are a key factor for advanced telerobotic applications. We studied the impact of the use of Virtual Reality for the control of non-anthropomorphic machines, and the effects of designing BoMIs based on different limbs opening to new knowledge about the effects of such factors in human-robot interaction. The design of a wearable haptic interface for drone operation showed the potential of augmenting their senses to perceive a distal environment through the sense of touch. Thanks to this novel design, nonexpert users were enabled to navigate drones through narrow passages with limited visibility, a task almost impossible to perform with standard interfaces.We believe that this thesis contributes to pave the way towards a deeper understanding of humans in human-robot interaction, as well as providing a new set of tools for the design of simple, intuitive interfaces with the goal of democratizing robotic teleoperation for the masses.