Control of a Supernumerary Robotic Hand by Foot: An Experimental Study in Virtual Reality

In the operational theater, the surgical team could highly benefit from a robotic supplementary hand under the surgeon's full control. The surgeon may so become more autonomous; this may reduce communication errors with the assistants and take over difficult tasks such as holding tools without tremor. In this paper, we therefore examine the possibility to control a third robotic hand with one foot's movements. Three experiments in virtual reality were designed to assess the feasibility of this control strategy, the learning curve of the subjects in different tasks and the coordination of foot movements with the two natural hands. Results show that the limbs are moved simultaneously, in parallel rather than serially. Participants' performance improved within a few minutes of practice without any specific difficulty to complete the tasks. Subjective assessment by the subjects indicated that controlling a third hand by foot has been easy and required only negligible physical and mental efforts. The sense of ownership was reported to improve through the experiments. The mental burden was not directly related to the level of motion required by a task, but depended on the type of activity and practice. The most difficult task was moving two hands and foot in opposite directions. These results suggest that a combination of practice and appropriate tasks can enhance the learning process for controlling a robotic hand by foot.

In a demanding task, three-handed manipulation is preferred to two-handed manipulation

Elahe Abdi, Hannes Bleuler, Mohamed Bouri, Etienne Burdet

Equipped with a third hand under their direct control, surgeons may be able to perform certain surgical interventions alone; this would reduce the need for a human assistant and related coordination difficulties. However, does human performance improve with three hands compared to two hands? To evaluate this possibility, we carried out a behavioural study on the performance of naive adults catching objects with three virtual hands controlled by their two hands and right foot. The subjects could successfully control the virtual hands in a few trials. With this control strategy, the workspace of the hands was inversely correlated with the task velocity. The comparison of performance between the three and two hands control revealed no significant difference of success in catching falling objects and in average effort during the tasks. Subjects preferred the three handed control strategy, found it easier, with less physical and mental burden. Although the coordination of the foot with the natural hands increased trial after trial, about two minutes of practice was not sufficient to develop a sense of ownership towards the third arm.

Nature Publishing Group2016

Foot-Interfaces Control and Human-Robot Interaction in Four-Handed Manipulation

Jacob Hernandez Sanchez

Humans have a great capacity to perform complex manipulations. However, there are many tasks for which two hands are not enough, such as in surgery. For such tasks, a dyad of people is required. However, when not used to work together, a dyad collaboration can be intermittent, slow and prone to misunderstandings if relied solely on verbal cues. The current PhD thesis report concerns the design and control of foot interfaces to enable solo four-handed manipulations. In the first part, we present the platform design in five DoF, with position mapping and haptic force feedback. We tackle the known issue of fatigue when using position mapping, by partially compensating for the dynamics of the leg. Finally, we perform a series of feasibility validations on bipedal teleoperation in 3 and 4 and 5 DoF. In the second part, a behavioural study investigates a symmetric bipedal holding and transport of an object while working on it with the hands. This study sheds light to the human capacity to use the redundancy given by augmentation with the feet, to assist the task of the hands. We investigate the effect of shared-control for lower limb teleoperation. Two modalities are used: 1. synergistic control of two robotic arms with one foot, 2. force assistance. We evaluate these strategies in terms of objective and subjective metrics, such as physical and cognitive workload and fluency.Aiming at enabling individualized foot dexterous control, in the third part of the thesis, we addressed the problem of bio-mechanical coupling of foot rotations. This is done by detecting intention to grasp, and selectively easing this action while haptically constraining the foot gestures for tool alignment. A user study was conducted to evaluate our proposed approach in a surgical laparoscopic gripper.In the fourth and final part, we focus on investigating more demanding tasks for the feet, as well as the coordinated control of four hands, during hybrid-robotic laparoscopic surgery. One foot controls a camera, while the other one performs a grasping task. Noteworthy is that all four-arms can be controlled simultaneously. For easing the solo surgical task, we design haptic assistance, for autonomous grasping operated by the foot. Finally, we perform a systematic behavioural study to understand the effect of our haptic-shared-control, along with an assistance modality for camera tool-tracking. We cover uni-pedal, bipedal and four-handed task scenarios. Our main conclusion is that foot gestures can be conditioned with haptic feedback, to effectively control multiple degrees of freedom of a supernumerary robotic arm. Thereby enabling four-arm simultaneous manipulations. This is done specifically by leveraging 1. a foot kinesthetic haptic platform that allows for the use of two feet, 2. leg compensation, and virtual assistive impedance, which are useful to increase comfort and alleviate fatigue. 3. Hybridization of control mappings for foot gestures, with limited range of motion and oblique axes of rotation, 4. Haptic shared-control strategies for prehensile force assistance, coordination assistance, and DoF-individuation assistance. We found that robotic assistance is highly appreciated, and has an important role in simplifying four-handed tasks. Indeed, these are instrumental considerations to enable a more fluent, less demanding, and more performant interaction, in tasks where the feet are supplementing the task of hands.

EPFL2022

Supernumerary Robotic Arm for Three-Handed Surgical Application: Behavioral Study and Design of Human-Machine Interface

Elahe Abdi

In surgical to industrial manipulation, the operator needs assistance for tasks requiring more than two hands. Teamwork may be the source of errors and inefficiency, especially if the assistant is a novice or unfamiliar with the main operator. The need for assistance may become problematic in case of lack of human resources e.g. in emergency surgical cases in the late hours of the night. Our objective is to improve the surgeon's autonomy and dexterity by a robotic arm under his own control. Although a number of robotic instrument holders have been developed, the best way to control such devices is still an open question. No behavioral study has been conducted on the best control strategy and human performance in three-handed tasks. We have selected the foot for commanding the third arm on the basis of a literature review. A series of experiments in virtual environments has been conducted to study the feasibility of this choice. The first experiment compares performance in the same task using two or three hands. Results show that three-handed manipulation is preferred to two-handed manipulation in demanding tasks. The second experiment investigated the type of tasks to be aimed in three-handed manipulation and the learning curve of users. Moving the hands and a foot simultaneously in opposite directions was perceived as difficult compared to a more active task with liberty in choosing the limbs coordination. Limbs were moved in parallel rather than serially. The performance improved within a few minutes of practice. Also, the sense of ownership improved constantly during the experiment. Two other experiments were aimed at handling the endoscope in laparoscopic surgery. Surgeons and medical students participated in these studies. Residents had a more positive approach towards foot usage and performed better compared to more experienced surgeons. This proves that the best training period for surgeons to use a foot controlled robotic arm is during their residency. A realistic virtual abdominal cavity has been developed for the last experiment. This had a positive influence on the participants' performance and emphasizes the importance of using a familiar context for training such a "three-handed surgery". Finally, two different foot interfaces were developed to investigate the most intuitive third arm commanding strategy. A robotic arm is hence controlled by the foot's translation or rotation in one interface (isotonic interface), and by force or torque in the other one (isometric interface). An experimental behavioral study was conducted to compare the two devices. Isometric rate control was preferred to isotonic position control due to the lower physical burden and higher movement accuracy of the robot. It was shown that the proposed device for isometric rate control could be used for intuitive control of four DoFs of a slave robotic arm. This thesis is the first step in a systematic investigation of a three-handed manipulation, two biological hands and a foot controlled robotic assistant. Findings suggest a high potential in using the foot to become more autonomous in surgery as well as other fields. Users can learn the control paradigm in a short period of time with little mental and physical burden. We expect the developed foot interfaces to be the basis of future development of more intuitive control interfaces. We believe that foot controlled robotic arms will be commonly used in various surgical as well as industrial applications.