Wearable and Self-Aware Machine Learning System for Online Cognitive Workload Monitoring and Drone Control

The use of robots in search and rescue is gaining particular interest, but singular skills are required to ensure efficient deployments in real missions. To face this problem, there is a need to develop more intuitive control interfaces. Moreover, to ensure high performance during high cognitive demanding tasks, such as in search and rescue missions, there is a need to combine both humanâs and robotâs skills. Respectively, humans and robots can adapt to new situations and optimize the execution of repetitive tasks. In this regard, novel share-control techniques have been developed to adapt the human-robot interaction, but to dynamically adapt this interaction, the information about the human state is missing.

To address these problems, I first developed a novel wearable system that enhances the control of drones providing a more intuitive flying experience. As shown in chapter 2, this wearable system tracks the upper body movements and translates them into commands for a drone. The system has been tested with a simulator and demonstrated for the teleoperation of a real drone. Moreover, to ensure enduring operations, I proposed a method that drastically reduces communication, and consequently, improves energy efficiency by 11.9%.

Second, in chapter 3, I presented a machine-learning approach for monitoring the cognitive workload level of a drone operator involved in search and rescue missions. My model combines the information of different features extracted from physiological signals, such as respiratory activity, electrocardiogram, photoplethysmogram, and skin temperature, acquired in a non-invasive way. To reduce both subject and day inter-variability of the signals, I explored different feature normalization techniques. Moreover, I adjusted the learning method for support vector machines to allow subject-specific optimizations. On a test set acquired from 34 volunteers, the proposed model distinguished between low and high cognitive workloads with an average accuracy of 87.3% and 91.2%, while controlling a drone simulator using both a traditional controller and the proposed FlyJacket design, respectively.

Third, in chapter 4, I presented the integration of the method developed for cognitive workload monitoring, on a new single wearable embedded system, that also integrates the proposed drone controller design. On the hardware side, it includes a multi-channel physiological signals acquisition and a low-power processing platform that is suited for cognitive workload monitoring. On the software side, the proposed system includes novel energy-aware bio-signal processing and embedded machine learning methods. Moreover, to exploit the trade-offs between the required accuracy of the available energy of the system, I presented a new application of the concept of a scalable machine-learning method with different power-saving levels. Results showed that the proposed self-aware approach yields an increase of 78% of the battery lifetime without really affecting the classification accuracy.

The proposed system, comprising a drone controller integrating a unit for cognitive workload monitoring, lays the foundations for the development of new-generation human-robot interfaces. With the information about the human state, we can close the loop of traditional share-control techniques, which will be able to dynamically adapt the level of interaction with semi-autonomous machines based on the need for the operator.

Wearable Technologies for Embodied Human-Robot Interaction

Carine Rognon

Robotic teleoperation is fundamental to augment the resilience, precision, and force of robots with the cognition of the operator. However, current interfaces, such as joysticks and remote controllers, are often complicated to handle since they require cognitive effort and learned skills. Wearable interfaces can enable more natural and intuitive interactions with robots, which would make robotic teleoperation accessible to a larger population of users for demanding tasks, such as manipulation or search-and-rescue. The aim of this thesis is to explore solutions to simplify our interactions with robots promoting their access to a broader range of the population. To achieve this, we are presenting a soft upper body exoskeleton, called the FlyJacket, for the bidirectional control of drones. Drones can greatly benefit us as they extend our perception and range of action. The exoskeleton controls a drone by recording torso movement and, through embedded haptic feedback devices, renders either kinesthetic guidance to improve the flight performance or tactile feedback to render the sensation of flying. We developed and tested an interface to control both a simulated and a real drone. The FlyJacket is a soft exoskeleton with arm support conceived to address the challenges of adapting to different morphologies and supporting the user during flight to prevent fatigue. We demonstrated that this novel interface allowed more consistent performance than when performing the same task with a remote controller and users felt more immersed into the flight. Interacting with a robot can be greatly enhanced by having multiple channels of sensorial feedback to increase the awareness of the operator. Information on the state of the drone can be intuitively rendered with haptic feedback. To create this bidirectional interaction with the drone, the two types of haptic feedback - kinesthetic and tactile - have been explored. Kinesthetic feedback was implemented with a cable-driven system to give guidance to the userâs torso position. Performing user studies, we could determine that the embedded guidance improved the flight performance and that a quadratically shaped force feedback curve was the most adequate profile to guide the user. We also established the minimal force difference, defined the perceived magnitude of this system and studied the learning process of users. Tactile feedback was investigated to render the sensation of flying by enhancing flight awareness, realism and immersion. To this end, we developed and embedded a new type of soft actuator that was compliant and lightweight such that it remained wearable and portable. Four devices, placed on the torso, provided feedback by compressing closed air pouches against the skin rendering the sensation of air pressure. A mechanical model and simulation of the pouch device were developed to determine appropriate parameters. We evaluated whether it conveyed useful information to the user and whether it enhanced the experience of flying. We demonstrated that users were able to understand the direction of the cues without prompting, could distinguish the cues quickly, and do so with high accuracy. The device was also used in a simulated flight task and users indicated that it increased the flight realism. We believe that the contributions of this thesis provides insights to the design of intuitive interfaces for human-robot interaction and increases their accessibility to a wider range of the population.

EPFL2019

Learning How to Write with a Social Robot

Shruti Chandra

Learning handwriting is one of the main occupations for children in primary education. It is a complex skill that has been associated with children’s learning and development. Poor handwriting skills affect children negatively concerning their academic performance and motivation. In schools, children with handwriting problems need extra support and one-to-one interventions from teachers. Personalised one-to-one support requires additional resources that most schools generally cannot afford. Therefore, there is a need to explore different methods of learning and contemporary technologies to develop new educational tools to foster children’s learning process of handwriting. The research presented in this thesis addresses the question of - “How to create a social robot that can help children to acquire handwriting skills?". Thus, we looked into three different research domains: education; pedagogic sciences; and social human-robot interaction (HRI). On the one hand, we investigated the types of errors commonly present in children’s handwriting. On the other hand, we needed to explore different educational theories to find alternate learning methods that would maximise children’s learning gains. In addition, we investigated various possibilities for integrating a social robot into educational scenarios and measured its impact on the children’s learning process. This thesis presents five studies that follow an incremental evolution of the research. The first three studies were based on the interaction between two children, in the presence of either a human or a robotic facilitator, with the children performing a collaborative writing activity. We explored the ‘peer-learning’ and ‘peer-tutoring’ methods and compared the differences in terms of interaction behaviour and learning gains. Further, we analysed children’s interpersonal distance and feelings of responsibility towards their peers, both in the presence of a robot and human facilitator. Finally, we examined children’s assessment behaviour towards their peers and self-disclosure towards the robotic facilitator. The outcomes of these first studies provided enough data to generate a taxonomy of children handwriting errors to be used in the subsequent studies. They hinted that the ‘peer-tutoring’ method had a positive influence on children’s learning and corrective feedback to their peers in the presence of the social robot, providing positive support to the applicability of social robots as ‘facilitators’ in handwriting learning. The next step was to study the effects of the robot in the peer-tutoring scenario when assuming the role of a peer, directly interacting and performing a collaborative writing activity with the children. Thus, we developed an educational system with an autonomous social behaviour of a robot exhibiting various levels of learning competencies. The tutor-child provides corrective feedback on the writing of the learner-robot and in turn, enhances his writing skills. The robot’s writing was generated by an algorithm incorporating human-inspired movements and could reproduce a set of writing errors from the error taxonomy. We tested the system by conducting two longitudinal studies, each of them composed of four sessions. In the fourth study, we assigned the robot two contrasting competencies: ‘learning’ and ‘non-learning’. We measured the differences in children’s learning gains and changes in their perceptions towards the learner-robot. The fifth study followed a similar interaction scenario and research questions, but this time the robot performed three learning competencies: ‘continuous-learning’;’non-learning’ and ‘personalised-learning’. The findings of these studies show that the children improved more with the robot exhibiting learning, continuous-learning and personalised-learning competencies compared to the robot showing non-learning competency. The way children’s learning and perception of the robot evolved over time confirmed the need for multiple sessions for the last two studies. This thesis presents some evidence that social robots can be successfully used as tools for enhancing children’s handwriting skills in peer-learning scenarios.

EPFL2019

A Quality-Scalable and Energy-Efficient Approach for Spectral Analysis of Heart Rate Variability

Mohamed Mostafa Sabry Aly, David Atienza Alonso, Andreas Peter Burg, Georgios Karakonstantis, Aviinaash Sankaranarayanan

Today there is a growing interest in the integration of health monitoring applications in portable devices necessitating the development of methods that improve the energy efficiency of such systems. In this paper, we present a systematic approach that enables energy-quality trade-offs in spectral analysis systems for bio-signals, which are useful in monitoring various health conditions as those associated with the heart-rate. To enable such trade-offs, the processed signals are expressed initially in a basis in which significant components that carry most of the relevant information can be easily distinguished from the parts that influence the output to a lesser extent. Such a classification allows the pruning of operations associated with the less significant signal components leading to power savings with minor quality loss since only less useful parts are pruned under the given requirements. To exploit the attributes of the modified spectral analysis system, thresholding rules are determined and adopted at design- and run-time, allowing the static or dynamic pruning of less-useful operations based on the accuracy and energy requirements. The proposed algorithm is implemented on a typical sensor node simulator and results show up-to 82% energy savings when static pruning is combined with voltage and frequency scaling, compared to the conventional algorithm in which such trade-offs were not available. In addition, experiments with numerous cardiac samples of various patients show that such energy savings come with a 4.9% average accuracy loss, which does not affect the system detection capability of sinus-arrhythmia which was used as a test case.

IEEE2014

Wearable Technologies for Embodied Human-Robot Interaction

Carine Rognon

EPFL2019

Learning How to Write with a Social Robot

Shruti Chandra

EPFL2019