Generalizing Robot Imitation Learning with Invariant Hidden Semi-Markov Models

Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations. In this paper, we propose an algorithm that learns a joint probability density function of the demonstrations with invariant formulations of hidden semi-Markov models to extract invariant segments (also termed as sub-goals or options), and smoothly follow the generated sequence of states with a linear quadratic tracking controller. The algorithm takes as input the demonstrations with respect to different coordinate systems describing virtual landmarks or objects of interest with a task-parameterized formulation, and adapt the segments according to the environmental changes in a systematic manner. We present variants of this algorithm in latent space with low-rank covariance decompositions, semi-tied covariances, and non-parametric online estimation of model parameters under small variance asymptotics; yielding considerably low sample and model complexity for acquiring new manipulation skills. The algorithm allows a Baxter robot to learn a pick-and-place task while avoiding a movable obstacle based on only 4 kinesthetic demonstrations.

Generative Models for Learning Robot Manipulation Skills from Humans

Ajay Kumar Tanwani

A long standing goal in artificial intelligence is to make robots seamlessly interact with humans in performing everyday manipulation skills. Learning from demonstrations or imitation learning provides a promising route to bridge this gap. In contrast to direct trajectory learning from demonstrations, many problems arise in interactive robotic applications that require higher contextual level understanding of the environment. This requires learning invariant mappings in the demonstrations that can generalize across different environmental situations such as size, position, orientation of objects, viewpoint of the observer, etc. In this thesis, we address this challenge by encapsulating invariant patterns in the demonstrations using probabilistic learning models for acquiring dexterous manipulation skills. We learn the joint probability density function of the demonstrations with a hidden semi-Markov model, and smoothly follow the generated sequence of states with a linear quadratic tracking controller. The model exploits the invariant segments (also termed as sub-goals, options or actions) in the demonstrations and adapts the movement in accordance with the external environmental situations such as size, position and orientation of the objects in the environment using a task-parameterized formulation. We incorporate high-dimensional sensory data for skill acquisition by parsimoniously representing the demonstrations using statistical subspace clustering methods and exploit the coordination patterns in latent space. To adapt the models on the fly and/or teach new manipulation skills online with the streaming data, we formulate a non-parametric scalable online sequence clustering algorithm with Bayesian non-parametric mixture models to avoid the model selection problem while ensuring tractability under small variance asymptotics. We exploit the developed generative models to perform manipulation skills with remotely operated vehicles over satellite communication in the presence of communication delays and limited bandwidth. A set of task-parameterized generative models are learned from the demonstrations of different manipulation skills provided by the teleoperator. The model captures the intention of teleoperator on one hand and provides assistance in performing remote manipulation tasks on the other hand under varying environmental situations. The assistance is formulated under time-independent shared control, where the model continuously corrects the remote arm movement based on the current state of the teleoperator; and/or time-dependent autonomous control, where the model synthesizes the movement of the remote arm for autonomous skill execution. Using the proposed methodology with the two-armed Baxter robot as a mock-up for semi-autonomous teleoperation, we are able to learn manipulation skills such as opening a valve, pick-and-place an object by obstacle avoidance, hot-stabbing (a specialized underwater task akin to peg-in-a-hole task), screw-driver target snapping, and tracking a carabiner in as few as 4 - 8 demonstrations. Our study shows that the proposed manipulation assistance formulations improve the performance of the teleoperator by reducing the task errors and the execution time, while catering for the environmental differences in performing remote manipulation tasks with limited bandwidth and communication delays.

EPFL2018

Continuous extraction of task constraints in a robot programming by demonstration framework

Sylvain Calinon

Robot Programming by Demonstration (RbD), also referred to as Learning by Imitation, explores user-friendly means of teaching a robot new skills. Recent advances in RbD have identified a number of key-issues for ensuring a generic approach to the transfer of skills across various agents and contexts. This thesis focuses on the two generic questions of what-to-imitate and how-to-imitate, which are respectively concerned with the problem of extracting the essential features of a task and determining a way to reproduce these essential features in different situations. The perspective adopted in this work is that a skill can be described efficiently at a trajectory level and that the robot may infer what are the important characteristics of the skill by observing multiple demonstrations of it, assuming that the important characteristics are invariant across the demonstrations. The proposed approach is based on the use of well-established statistical methods in a RbD application, by using Hidden Markov Model (HMM) as a first approach and then moving on to the joint use of Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR). Even if these methods were applied extensively in various fields of research including human motion analysis and robotics, their use essentially focused on gesture recognition rather than on gesture reproduction. Moreover, the models were usually trained offline using large datasets. Thus, the use and combination of these machine learning techniques in a Learning by Imitation framework is challenging and has not been extensively studied yet. In this thesis, we show that these methods are well suited for incremental and active teaching scenarios where the user only provides a small number of demonstrations, either by wearing a set of motions sensors attached to his/her body, or by helping the robot refine its skill by kinesthetic teaching, that is, by embodying the robot and putting it through the motion. These techniques are then applied for enabling a Fujitsu HOAP-2 and HOAP-3 humanoid robot to extract automatically different constraints by observing several manipulation skills and to reproduce these skills in various situations through Gaussian Mixture Regression (GMR). The contributions of this thesis are threefold: (1) it contributes to RbD by proposing a generic probabilistic framework to deal with recognition, generalization, reproduction and evaluation issues, and more specifically to deal with the automatic extraction of task constraints and with the determination of a controller satisfying several constraints simultaneously to reproduce the skill in a different context; (2) it contributes to Human-Robot Interaction (HRI) by proposing active teaching methods that puts the human teacher "in the loop" of the robot's learning by using an incremental scaffolding process and by using different modalities to produce the demonstrations; (3) it finally contributes to robotics and HRI through various real-world applications of the framework showing learning and reproduction of communicative gestures and manipulation skills. The generality of the proposed framework is also demonstrated through its joint use with other learning approaches in collaborative experiments.

EPFL2007

Learning Search Strategies from Human Demonstrations

Guillaume Pierre Luc De Chambrier

Decision making and planning with partial state information is a problem faced by all forms of intelligent entities. The formulation of a problem under partial state information leads to an exorbitant set of choices with associated probabilistic outcomes making its resolution difficult when using traditional planning methods. Human beings have acquired the ability of acting under uncertainty through education and self-learning. Transferring our know-how to artificial agents and robots will make it faster for them to learn and even improve upon us in tasks in which incomplete knowledge is available, which is the objective of this thesis. We model how humans reason with respect to their beliefs and transfer this knowledge in the form of a parameterised policy, following a Programming by Demonstration framework, to a robot apprentice for two spatial navigation tasks: the first task consists of localising a wooden block on a table and for the second task a power socket must be found and connected. In both tasks the human teacher and robot apprentice only rely on haptic and tactile information. We model the human and robot's beliefs by a probability density function which we update through recursive Bayesian state space estimation. To model the reasoning processes of human subjects performing the search tasks we learn a generative joint distribution over beliefs and actions (end-effector velocities) which were recorded during the executions of the task. For the first search task the direct mapping from belief to actions is learned whilst for the second task we incorporate a cost function used to adapt the policy parameters in a Reinforcement Learning framework and show a considerable improvement over solely learning the behaviour with respect to the distance taken to accomplish the task. Both search tasks above can be considered as active localisation as the uncertainty originates only from the position of the agent in the world. We consider searches in which both the position of the robot and features of the environment are uncertain. Given the unstructured nature of the belief a histogram parametrisation of the joint distribution of the robots position and features is necessary. However, naively doing so becomes quickly intractable as the space and time complexity is exponential. We demonstrate that by only parametrising the marginals and by memorising the parameters of the measurement likelihood functions we can recover the exact same solution as the naive parametrisations at a cost which is linear in space and time complexity.

EPFL2016