Generative Models for Learning Robot Manipulation Skills from Humans

A long standing goal in artificial intelligence is to make robots seamlessly interact with humans in performing everyday manipulation skills. Learning from demonstrations or imitation learning provides a promising route to bridge this gap. In contrast to direct trajectory learning from demonstrations, many problems arise in interactive robotic applications that require higher contextual level understanding of the environment. This requires learning invariant mappings in the demonstrations that can generalize across different environmental situations such as size, position, orientation of objects, viewpoint of the observer, etc.

In this thesis, we address this challenge by encapsulating invariant patterns in the demonstrations using probabilistic learning models for acquiring dexterous manipulation skills. We learn the joint probability density function of the demonstrations with a hidden semi-Markov model, and smoothly follow the generated sequence of states with a linear quadratic tracking controller. The model exploits the invariant segments (also termed as sub-goals, options or actions) in the demonstrations and adapts the movement in accordance with the external environmental situations such as size, position and orientation of the objects in the environment using a task-parameterized formulation. We incorporate high-dimensional sensory data for skill acquisition by parsimoniously representing the demonstrations using statistical subspace clustering methods and exploit the coordination patterns in latent space. To adapt the models on the fly and/or teach new manipulation skills online with the streaming data, we formulate a non-parametric scalable online sequence clustering algorithm with Bayesian non-parametric mixture models to avoid the model selection problem while ensuring tractability under small variance asymptotics.

We exploit the developed generative models to perform manipulation skills with remotely operated vehicles over satellite communication in the presence of communication delays and limited bandwidth. A set of task-parameterized generative models are learned from the demonstrations of different manipulation skills provided by the teleoperator. The model captures the intention of teleoperator on one hand and provides assistance in performing remote manipulation tasks on the other hand under varying environmental situations. The assistance is formulated under time-independent shared control, where the model continuously corrects the remote arm movement based on the current state of the teleoperator; and/or time-dependent autonomous control, where the model synthesizes the movement of the remote arm for autonomous skill execution. Using the proposed methodology with the two-armed Baxter robot as a mock-up for semi-autonomous teleoperation, we are able to learn manipulation skills such as opening a valve, pick-and-place an object by obstacle avoidance, hot-stabbing (a specialized underwater task akin to peg-in-a-hole task), screw-driver target snapping, and tracking a carabiner in as few as 4 - 8 demonstrations. Our study shows that the proposed manipulation assistance formulations improve the performance of the teleoperator by reducing the task errors and the execution time, while catering for the environmental differences in performing remote manipulation tasks with limited bandwidth and communication delays.

Generalizing Robot Imitation Learning with Invariant Hidden Semi-Markov Models

Sylvain Calinon, Swati Krishnan, Ajay Kumar Tanwani

Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations. In this paper, we propose an algorithm that learns a joint probability density function of the demonstrations with invariant formulations of hidden semi-Markov models to extract invariant segments (also termed as sub-goals or options), and smoothly follow the generated sequence of states with a linear quadratic tracking controller. The algorithm takes as input the demonstrations with respect to different coordinate systems describing virtual landmarks or objects of interest with a task-parameterized formulation, and adapt the segments according to the environmental changes in a systematic manner. We present variants of this algorithm in latent space with low-rank covariance decompositions, semi-tied covariances, and non-parametric online estimation of model parameters under small variance asymptotics; yielding considerably low sample and model complexity for acquiring new manipulation skills. The algorithm allows a Baxter robot to learn a pick-and-place task while avoiding a movable obstacle based on only 4 kinesthetic demonstrations.

2018

Learning Reach-to-Grasp Motions From Human Demonstrations

Ashwini Shukla

Reaching over to grasp an item is arguably the most commonly used motor skill by humans. Even under sudden perturbations, humans seem to react rapidly and adapt their motion to guarantee success. Despite the apparent ease and frequency with which we use this ability, a complete understanding of the underlying mechanisms cannot be claimed. It is partly due to such incomplete knowledge that adaptive robot motion for reaching and grasping under perturbations is not perfectly achieved. In this thesis, we take the discriminative approach for modelling trajectories of reach-to-grasp motion from expert demonstrations. Throughout this thesis, we will employ time-independent (autonomous) flow based representations to learn reactive motion controllers which can then be ported onto robots. This thesis is divided into three main parts. The first part is dedicated to biologically inspired modelling of reach-to-grasp motions with respect to the hand-arm coupling. We build upon previous work in motion modelling using autonomous dynamical systems (DS) and present a coupled dynamical system (CDS) model of these two subsystems. The coupled model ensures satisfaction of the constraints between the hand and the arm subsystems which are critical to the success of a reach-to-grasp task. Moreover, it reduces the complexity of the overall motion planning problem as compared to considering a combined problem for the hand and the arm motion. In the second part we extend the CDS approach to incorporate multiple grasping points. Such a model is beneficial due to the fact that many daily life objects afford multiple grasping locations on their surface. We combine a DS based approach with energy-function learning to learn a multiple attractor dynamical system where the attractors are mapped to the desired grasping points. We present the Augmented-SVM (ASVM) model that combines the classical SVM formulation with gradient constraints arising from the energy function to learn the desired dynamical function for motion generation. In the last part of this thesis, we address the problem of inverse-kinematics and obstacle avoidance by combining our flow-based motion generator with global configuration-space planners. We claim that the two techniques complement each other. On one hand, the fast reactive nature of our flow based motion generator can used to guide the search of a randomly exploring random tree (RRT) based global planner. On the other hand, global planners can efficiently handle arbitrary obstacles and avoid local minima present in the dynamical function learned from demonstrations. We show that combining the information from demonstrations with global planning in the form of a energy-map considerably decreases the computational complexity of state-of-the-art sampling based planners. We believe that this thesis has the following contributions to Robotics and Machine Learning. First, we have developed algorithms for fast and adaptive motion generation for reach-grasp motions. Second, we formulated an extension to the classical SVM formulation that takes into account the gradient information from data. We showed that instead of being limited as a classifier or a regressor, the SVM framework can be used as a more general function approximation technique. Lastly, we have combined our local methods with global approaches for planning to achieve arbitrary obstacle avoidance and considerable reduction in the computation complexity of the global planners.

EPFL2014

Learning Search Strategies from Human Demonstrations

Guillaume Pierre Luc De Chambrier

Decision making and planning with partial state information is a problem faced by all forms of intelligent entities. The formulation of a problem under partial state information leads to an exorbitant set of choices with associated probabilistic outcomes making its resolution difficult when using traditional planning methods. Human beings have acquired the ability of acting under uncertainty through education and self-learning. Transferring our know-how to artificial agents and robots will make it faster for them to learn and even improve upon us in tasks in which incomplete knowledge is available, which is the objective of this thesis. We model how humans reason with respect to their beliefs and transfer this knowledge in the form of a parameterised policy, following a Programming by Demonstration framework, to a robot apprentice for two spatial navigation tasks: the first task consists of localising a wooden block on a table and for the second task a power socket must be found and connected. In both tasks the human teacher and robot apprentice only rely on haptic and tactile information. We model the human and robot's beliefs by a probability density function which we update through recursive Bayesian state space estimation. To model the reasoning processes of human subjects performing the search tasks we learn a generative joint distribution over beliefs and actions (end-effector velocities) which were recorded during the executions of the task. For the first search task the direct mapping from belief to actions is learned whilst for the second task we incorporate a cost function used to adapt the policy parameters in a Reinforcement Learning framework and show a considerable improvement over solely learning the behaviour with respect to the distance taken to accomplish the task. Both search tasks above can be considered as active localisation as the uncertainty originates only from the position of the agent in the world. We consider searches in which both the position of the robot and features of the environment are uncertain. Given the unstructured nature of the belief a histogram parametrisation of the joint distribution of the robots position and features is necessary. However, naively doing so becomes quickly intractable as the space and time complexity is exponential. We demonstrate that by only parametrising the marginals and by memorising the parameters of the measurement likelihood functions we can recover the exact same solution as the naive parametrisations at a cost which is linear in space and time complexity.