Role of Gaze Cues in Interpersonal Motor Coordination: Towards Higher Affiliation in Human-Robot Interaction

Background The ability to follow one another's gaze plays an important role in our social cognition; especially when we synchronously perform tasks together. We investigate how gaze cues can improve performance in a simple coordination task (i.e., the mirror game), whereby two players mirror each other's hand motions. In this game, each player is either a leader or follower. To study the effect of gaze in a systematic manner, the leader's role is played by a robotic avatar. We contrast two conditions, in which the avatar provides or not explicit gaze cues that indicate the next location of its hand. Specifically, we investigated (a) whether participants are able to exploit these gaze cues to improve their coordination, (b) how gaze cues affect action prediction and temporal coordination, and (c) whether introducing active gaze behavior for avatars makes them more realistic and human-like (from the user point of view). Methodology/Principal Findings 43 subjects participated in 8 trials of the mirror game. Each subject performed the game in the two conditions (with and without gaze cues). In this within-subject study, the order of the conditions was randomized across participants, and subjective assessment of the avatar's realism was assessed by administering a post-hoc questionnaire. When gaze cues were provided, a quantitative assessment of synchrony between participants and the avatar revealed a significant improvement in subject reaction-time (RT). This confirms our hypothesis that gaze cues improve the follower's ability to predict the avatar's action. An analysis of the pattern of frequency across the two players' hand movements reveals that the gaze cues improve the overall temporal coordination across the two players. Finally, analysis of the subjective evaluations from the questionnaires reveals that, in the presence of gaze cues, participants found it not only more human-like/realistic, but also easier to interact with the avatar. Conclusion/Significance This work confirms that people can exploit gaze cues to predict another person's movements and to better coordinate their motions with their partners, even when the partner is a computer-animated avatar. Moreover, this study contributes further evidence that implementing biological features, here task-relevant gaze cues, enable the humanoid robotic avatar to appear more human-like, and thus increase the user's sense of affiliation.

Enhancing our understanding of human fine manipulation skills and advancing robot dexterity in grasping

Kunpeng Yao

From surgery to watchmaking, fine-manipulation skills highly rely on the dexterity afforded by both hands. Coordination is key to human dexterity. Specifically, humans need not only to govern the abundant intrinsic degrees of freedom (DOFs) to allocate controls of task-demanded variables, but also to adapt postures in response to extrinsic task conditions. In spite of the recent advances in robotics research, human dexterity remains unattainable for robots, especially in terms of flexibility and adaptability. Therefore, it is necessary to gain insights into human fine-manipulation skills, to advance the dexterity of robots in similar tasks. This thesis deepens our understanding of human dexterity by investigating the human coordination in fine-manipulation skills taken from watchmaking craftsmanship.The first part of this thesis investigates both intrinsic and extrinsic coordination of upper-limbs in a non-symmetrical and non-rhythmic bimanual fine-manipulation task. We conduct a comparative study in the assembly of a spring on the watch face. Analysis of motion kinematics reveals that professional subjects mitigate the task challenge by actively modifying task conditions. Moreover, we offer a novel perspective to understand the skill acquisition in humans. We hypothesize that the evolution of coordination during skill improvement is driven by the changes in the structure of the optimal criterion of the central nervous system. We employ a bi-level optimization framework to infer the structure of this optimal criterion.The second part investigates how roles and control variables are distributed across hands and fingers. We compare task performance of human subjects under two experimental conditions when dismounting a screw from a watch face. When the watch face needed positioning, the role distribution of both hands was strongly influenced by hand dominance; when the watch face was stationary, a variety of hand pose combinations emerged. We propose a taxonomy of bimanual hand pose combinations and develop a graphical matrix-based representation approach to afford analysis of experimental observations. Our analysis suggests that the control of independent task demands is distributed across either hands or functional groups of fingers.In the third part, we take inspiration from human coordination principle to advance the dexterity of a robotic hand by exploiting its redundant DOFs. We propose a human-like algorithm to enable a robotic hand to grasp objects using arbitrary surface regions, no longer restricted to regular grasp types, such as pinch or power grasp. We present an iterative process to empower the robotic hand to grasp multiple objects in sequence. Moreover, we formulate a strategy to facilitate the exploitation of redundant DOFs for multitask planning. Our approaches have been validated both in simulation and on a real robotic hand.In summary, this thesis not only offers a deeper understanding of human dexterity by revealing both the intrinsic and extrinsic coordination of upper-limbs and the role distribution across hands in bimanual fine-manipulation tasks, but also proposes algorithms that enable multi-fingered robotic hands to achieve human-like dexterous grasping of a single or even multiple objects using arbitrary surface regions. This thesis offers the prospect of developing algorithms for robotic dexterous grasping and manipulation, and also inspires the design of novel robotic hands and manipulators.

EPFL2022

Imitation Learning of Motion Coordination in Robots

Elena Gribovskaya

The ease with which humans coordinate all their limbs is fascinating. Such a simplicity is the result of a complex process of motor coordination, i.e. the ability to resolve the biomechanical redundancy in an efficient and repeatable manner. Coordination enables a wide variety of everyday human activities from filling in a glass with water to pair figure skating. Therefore, it is highly desirable to endow robots with similar skills. Despite the apparent diversity of coordinated motions, all of them share a crucial similarity: these motions are dictated by underlying constraints. The constraints shape the formation of the coordination patterns between the different degrees of freedom. Coordination constraints may take a spatio-temporal form; for instance, during bimanual object reaching or while catching a ball on the fly. They also may relate to the dynamics of the task; for instance, when one applies a specific force profile to carry a load. In this thesis, we develop a framework for teaching coordination skills to robots. Coordination may take different forms, here, we focus on teaching a robot intra-limb and bimanual coordination, as well as coordination with a human during physical collaborative tasks. We use tools from well-established domains of Bayesian semiparametric learning (Gaussian Mixture Models and Regression, Hidden Markov Models), nonlinear dynamics, and adaptive control. We take a biologically inspired approach to robot control. Specifically, we adopt an imitation learning perspective to skill transfer, that offers a seamless and intuitive way of capturing the constraints contained in natural human movements. As the robot is taught from motion data provided by a human teacher, we exploit evidence from human motor control of the temporal evolution of human motions that may be described by dynamical systems. Throughout this thesis, we demonstrate that the dynamical system view on movement formation facilitates coordination control in robots. We explain how our framework for teaching coordination to a robot is built up, starting from intra-limb coordination and control, moving to bimanual coordination, and finally to physical interaction with a human. The dissertation opens with the discussion of learning discrete task-level coordination patterns, such as spatio-temporal constraints emerging between the two arms in bimanual manipulation tasks. The encoding of bimanual constraints occurs at the task level and proceeds through a discretization of the task as sequences of bimanual constraints. Once the constraints are learned, the robot utilizes them to couple the two dynamical systems that generate kinematic trajectories for the hands. Explicit coupling of the dynamical systems ensures accurate reproduction of the learned constraints, and proves to be crucial for successful accomplishment of the task. In the second part of this thesis, we consider learning one-arm control policies. We present an approach to extracting non-linear autonomous dynamical systems from kinematic data of arbitrary point-to-point motions. The proposed method aims to tackle the fundamental questions of learning robot coordination: (i) how to infer a motion representation that captures a multivariate coordination pattern between degrees of freedom and that generalizes this pattern to unseen contexts; (ii) whether the policy learned directly from demonstrations can provide robustness against spatial and temporal perturbations. Finally, we demonstrate that the developed dynamical system approach to coordination may go beyond kinematic motion learning. We consider physical interactions between a robot and a human in situations where they jointly perform manipulation tasks; in particular, the problem of collaborative carrying and positioning of a load. We extend the approach proposed in the second part of this thesis to incorporate haptic information into the learning process. As a result, the robot adapts its kinematic motion plan according to human intentions expressed through the haptic signals. Even after the robot has learned the task model, the human still remains a complex contact environment. To ensure robustness of the robot behavior in the face of the variability inherent to human movements, we wrap the learned task model in an adaptive impedance controller with automatic gain tuning. The techniques, developed in this thesis, have been applied to enable learning of unimanual and bimanual manipulation tasks on the robotics platforms HOAP-3, KATANA, and i-Cub, as well as to endow a pair of simulated robots with the ability to perform a manipulation task in the physical collaboration.

EPFL2012

Imitation Learning of Motion Coordination in Robots

Elena Gribovskaya

EPFL2012

Enhancing our understanding of human fine manipulation skills and advancing robot dexterity in grasping

Kunpeng Yao

EPFL2022