Resource Management for Power-Constrained HEVC Transcoding Using Reinforcement Learning

The advent of online video streaming services along with the users' demand for high-quality contents require High Efficiency Video Coding (HEVC), which provides higher quality and compression at the cost of increased complexity. On one hand, HEVC exposes a set of dynamically tunable parameters to provide trade-offs among Quality-of-Service (QoS), performance, and power consumption of multi-core servers. On the other hand, resource management of modern multi-core servers is in charge of adapting system-level parameters, as operating frequency and multithreading, to deal with concurrent applications and their requirements. Therefore, efficient multi-user HEVC streaming necessitates joint adaptation of application- and system-level parameters. Nonetheless, dealing with such a large and dynamic design-space is difficult to address through conventional strategies. In this work, we develop a multiagent Reinforcement Learning framework to jointly adjust application- and system-level parameters at runtime to satisfy the QoS of multi-user HEVC streaming in power-constrained servers. The benefits of our approach are revealed in terms of adaptability and quality (up to to 4x improvements in terms of QoS when compared to a static scheme), and learning time (6x faster than an equivalent mono-agent implementation). Finally, we show how power-capping techniques formulated outperform the hardware-based power capping with respect to quality.

Learning Search Strategies from Human Demonstrations

Guillaume Pierre Luc De Chambrier

Decision making and planning with partial state information is a problem faced by all forms of intelligent entities. The formulation of a problem under partial state information leads to an exorbitant set of choices with associated probabilistic outcomes making its resolution difficult when using traditional planning methods. Human beings have acquired the ability of acting under uncertainty through education and self-learning. Transferring our know-how to artificial agents and robots will make it faster for them to learn and even improve upon us in tasks in which incomplete knowledge is available, which is the objective of this thesis. We model how humans reason with respect to their beliefs and transfer this knowledge in the form of a parameterised policy, following a Programming by Demonstration framework, to a robot apprentice for two spatial navigation tasks: the first task consists of localising a wooden block on a table and for the second task a power socket must be found and connected. In both tasks the human teacher and robot apprentice only rely on haptic and tactile information. We model the human and robot's beliefs by a probability density function which we update through recursive Bayesian state space estimation. To model the reasoning processes of human subjects performing the search tasks we learn a generative joint distribution over beliefs and actions (end-effector velocities) which were recorded during the executions of the task. For the first search task the direct mapping from belief to actions is learned whilst for the second task we incorporate a cost function used to adapt the policy parameters in a Reinforcement Learning framework and show a considerable improvement over solely learning the behaviour with respect to the distance taken to accomplish the task. Both search tasks above can be considered as active localisation as the uncertainty originates only from the position of the agent in the world. We consider searches in which both the position of the robot and features of the environment are uncertain. Given the unstructured nature of the belief a histogram parametrisation of the joint distribution of the robots position and features is necessary. However, naively doing so becomes quickly intractable as the space and time complexity is exponential. We demonstrate that by only parametrising the marginals and by memorising the parameters of the measurement likelihood functions we can recover the exact same solution as the naive parametrisations at a cost which is linear in space and time complexity.

EPFL2016

Learning Reach-to-Grasp Motions From Human Demonstrations

Ashwini Shukla

Reaching over to grasp an item is arguably the most commonly used motor skill by humans. Even under sudden perturbations, humans seem to react rapidly and adapt their motion to guarantee success. Despite the apparent ease and frequency with which we use this ability, a complete understanding of the underlying mechanisms cannot be claimed. It is partly due to such incomplete knowledge that adaptive robot motion for reaching and grasping under perturbations is not perfectly achieved. In this thesis, we take the discriminative approach for modelling trajectories of reach-to-grasp motion from expert demonstrations. Throughout this thesis, we will employ time-independent (autonomous) flow based representations to learn reactive motion controllers which can then be ported onto robots. This thesis is divided into three main parts. The first part is dedicated to biologically inspired modelling of reach-to-grasp motions with respect to the hand-arm coupling. We build upon previous work in motion modelling using autonomous dynamical systems (DS) and present a coupled dynamical system (CDS) model of these two subsystems. The coupled model ensures satisfaction of the constraints between the hand and the arm subsystems which are critical to the success of a reach-to-grasp task. Moreover, it reduces the complexity of the overall motion planning problem as compared to considering a combined problem for the hand and the arm motion. In the second part we extend the CDS approach to incorporate multiple grasping points. Such a model is beneficial due to the fact that many daily life objects afford multiple grasping locations on their surface. We combine a DS based approach with energy-function learning to learn a multiple attractor dynamical system where the attractors are mapped to the desired grasping points. We present the Augmented-SVM (ASVM) model that combines the classical SVM formulation with gradient constraints arising from the energy function to learn the desired dynamical function for motion generation. In the last part of this thesis, we address the problem of inverse-kinematics and obstacle avoidance by combining our flow-based motion generator with global configuration-space planners. We claim that the two techniques complement each other. On one hand, the fast reactive nature of our flow based motion generator can used to guide the search of a randomly exploring random tree (RRT) based global planner. On the other hand, global planners can efficiently handle arbitrary obstacles and avoid local minima present in the dynamical function learned from demonstrations. We show that combining the information from demonstrations with global planning in the form of a energy-map considerably decreases the computational complexity of state-of-the-art sampling based planners. We believe that this thesis has the following contributions to Robotics and Machine Learning. First, we have developed algorithms for fast and adaptive motion generation for reach-grasp motions. Second, we formulated an extension to the classical SVM formulation that takes into account the gradient information from data. We showed that instead of being limited as a classifier or a regressor, the SVM framework can be used as a more general function approximation technique. Lastly, we have combined our local methods with global approaches for planning to achieve arbitrary obstacle avoidance and considerable reduction in the computation complexity of the global planners.

EPFL2014

Machine Learning-Based Quality-Aware Power and Thermal Management of Multistream HEVC Encoding on Multicore Servers

David Atienza Alonso, Arman Iranfar, Marina Zapater Sancho

The emergence of video streaming applications, together with the users’ demand for high-resolution contents, has led to the development of new video coding standards, such as High Efficiency Video Coding (HEVC). HEVC provides high efficiency at the cost of increased complexity. This higher computational burden results in increased power consumption in current multicore servers. To tackle this challenge, algorithmic optimizations need to be accompanied by content-aware application-level strategies, able to reduce power while meeting compression and quality requirements. In this paper, we propose a machine learning-based power and thermal management approach that dynamically learns and selects the best encoding configuration and operating frequency for each of the videos running on multicore servers, by using information from frame compression, quality, encoding time, power, and temperature. In addition, we present a resolution-aware video assignment and migration strategy that reduces the peak and average temperature of the chip while maintaining the desirable encoding time. We implemented our approach in an enterprise multicore server and evaluated it under several common scenarios for video providers. On average, compared to a state-of-the-art technique, for the most realistic scenario, our approach improves BD-PSNR and BD-rate by 0.54 dB, and 8%, respectively, and reduces the encoding time, power consumption, and average temperature by 15.3%, 13%, and 10%, respectively. Moreover, our proposed approach increases BD-PSNR and BD-rate compared to the HEVC Test Model (HM), by 1.19 dB and 24%, respectively, without any encoding time degradation, when power and temperature constraints are relaxed.