A Comparative Psychophysical and EEG Study of Different Feedback Modalities for HRI

This paper presents a comparison between six different ways to convey navigational information provided by a robot to a human. Visual, auditory, and tactile feedback modalities were selected and designed to suggest a direction of travel to a human user, who can then decide if he agrees or not with the robot's proposition. This work builds upon a previous research on a novel semi-autonomous navigation system in which the human supervises an autonomous system, providing corrective monitoring signals whenever necessary.We recorded both qualitative (user impressions based on selected criteria and ranking of their feelings) and quantitative (response time and accuracy) information regarding different types of feedback. In addition, a preliminary analysis of the influence of the different types of feedback on brain activity is also shown. The result of this study may provide guidelines for the design of such a human-robot interaction system, depending on both the task and the human user.

Ricardo Andres Chavarriaga Lozano, Xavier Perrin, Roland Siegwart

This paper presents a comparison between six different ways to convey navigational information provided by a robot to a human. Visual, auditory, and tactile feedback mo-da-li-ties were selected and designed to suggest a direction of travel to a human user, who can then decide if he agrees or not with the robot's proposition. This work builds upon a previous research on a novel semi-autonomous navigation system in which the human supervises an autonomous system, providing corrective monitoring signals whenever necessary.We recorded both qualitative (user impressions based on selected criteria and ranking of their feelings) and quantitative (response time and accuracy) information regarding different types of feedback. In addition, a preliminary analysis of the influence of the different types of feedback on brain activity is also shown. The result of this study may provide guidelines for the design of such a human-robot interaction system, depending on both the task and the human user.

IDIAP2007

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Plamen Prodanov

Mobile service robots are going to play an increasing role in the society of humans. Voice-enabled interaction with service robots becomes very important, if such robots are to be deployed in real-world environments and accepted by the vast majority of potential human users. The research presented in this thesis addresses the problem of speech recognition integration in an interactive voice-enabled interface of a service robot, in particular a tour-guide robot. The task of a tour-guide robot is to engage visitors to mass exhibitions (users) in dialogue providing the services it is designed for (e.g. exhibit presentations) within a limited time. In managing tour-guide dialogues, extracting the user goal (intention) for requesting a particular service at each dialogue state is the key issue. In mass exhibition conditions speech recognition errors are inevitable because of noisy speech and uncooperative users of robots with no prior experience in robotics. They can jeopardize the user goal identification. Wrongly identified user goals can lead to communication failures. Therefore, to reduce the risk of such failures, methods for detecting and compensating for communication failures in human-robot dialogue are needed. During the short-term interaction with visitors, the interpretation of the user goal at each dialogue state can be improved by combining speech recognition in the speech modality with information from other available robot modalities. The methods presented in this thesis exploit probabilistic models for fusing information from speech and auxiliary modalities of the robot for user goal identification and communication failure detection. To compensate for the detected communication failures we investigate multimodal methods for recovery from communication failures. To model the process of modality fusion, taking into account the uncertainties in the information extracted from each input modality during human-robot interaction, we use the probabilistic framework of Bayesian networks. Bayesian networks are graphical models that represent a joint probability function over a set of random variables. They are used to model the dependencies among variables associated with the user goals, modality related events (e.g. the event of user presence that is inferred from the laser scanner modality of the robot), and observed modality features providing evidence in favor of these modality events. Bayesian networks are used to calculate posterior probabilities over the possible user goals at each dialogue state. These probabilities serve as a base in deciding if the user goal is valid, i.e. if it can be mapped into a tour-guide service (e.g. exhibit presentation) or is undefined – signaling a possible communication failure. The Bayesian network can be also used to elicit probabilities over the modality events revealing information about the possible cause for a communication failure. Introducing new user goal aspects (e.g. new modality events and related features) that provide auxiliary information for detecting communication failures makes the design process cumbersome, calling for a systematic approach in the Bayesian network modelling. Generally, introducing new variables for user goal identification in the Bayesian networks can lead to complex and computationally expensive models. In order to make the design process more systematic and modular, we adapt principles from the theory of grounding in human communication. When people communicate, they resolve understanding problems in a collaborative joint effort of providing evidence of common shared knowledge (grounding). We use Bayesian network topologies, tailored to limited computational resources, to model a state-based grounding model fusing information from three different input modalities (laser, video and speech) to infer possible grounding states. These grounding states are associated with modality events showing if the user is present in range for communication, if the user is attending to the interaction, whether the speech modality is reliable, and if the user goal is valid. The state-based grounding model is used to compute probabilities that intermediary grounding states have been reached. This serves as a base for detecting if the the user has reached the final grounding state, or wether a repair dialogue sequence is needed. In the case of a repair dialogue sequence, the tour-guide robot can exploit the multiple available modalities along with speech. For example, if the user has failed to reach the grounding state related to her/his presence in range for communication, the robot can use its move modality to search and attract the attention of the visitors. In the case when speech recognition is detected to be unreliable, the robot can offer the alternative use of the buttons modality in the repair sequence. Given the probability of each grounding state, and the dialogue sequence that can be executed in the next dialogue state, a tour-guide robot has different preferences on the possible dialogue continuation. If the possible dialogue sequences at each dialogue state are defined as actions, the introduced principle of maximum expected utility (MEU) provides an explicit way of action selection, based on the action utility, given the evidence about the user goal at each dialogue state. Decision networks, constructed as graphical models based on Bayesian networks are proposed to perform MEU-based decisions, incorporating the utility of the actions to be chosen at each dialogue state by the tour-guide robot. These action utilities are defined taking into account the tour-guide task requirements. The proposed graphical models for user goal identification and dialogue error handling in human-robot dialogue are evaluated in experiments with multimodal data. These data were collected during the operation of the tour-guide robot RoboX at the Autonomous System Lab of EPFL and at the Swiss National Exhibition in 2002 (Expo.02). The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation. On the component level, the technical evaluation is done by calculating accuracies, as objective measures of the performance of the grounding model, and the resulting performance of the user goal identification in dialogue. The benefit of the proposed error handling framework is demonstrated comparing the accuracy of a baseline interactive system, employing only speech recognition for user goal identification, and a system equipped with multimodal grounding models for error handling.

EPFL2006

Learning How to Write with a Social Robot

Shruti Chandra

Learning handwriting is one of the main occupations for children in primary education. It is a complex skill that has been associated with children’s learning and development. Poor handwriting skills affect children negatively concerning their academic performance and motivation. In schools, children with handwriting problems need extra support and one-to-one interventions from teachers. Personalised one-to-one support requires additional resources that most schools generally cannot afford. Therefore, there is a need to explore different methods of learning and contemporary technologies to develop new educational tools to foster children’s learning process of handwriting. The research presented in this thesis addresses the question of - “How to create a social robot that can help children to acquire handwriting skills?". Thus, we looked into three different research domains: education; pedagogic sciences; and social human-robot interaction (HRI). On the one hand, we investigated the types of errors commonly present in children’s handwriting. On the other hand, we needed to explore different educational theories to find alternate learning methods that would maximise children’s learning gains. In addition, we investigated various possibilities for integrating a social robot into educational scenarios and measured its impact on the children’s learning process. This thesis presents five studies that follow an incremental evolution of the research. The first three studies were based on the interaction between two children, in the presence of either a human or a robotic facilitator, with the children performing a collaborative writing activity. We explored the ‘peer-learning’ and ‘peer-tutoring’ methods and compared the differences in terms of interaction behaviour and learning gains. Further, we analysed children’s interpersonal distance and feelings of responsibility towards their peers, both in the presence of a robot and human facilitator. Finally, we examined children’s assessment behaviour towards their peers and self-disclosure towards the robotic facilitator. The outcomes of these first studies provided enough data to generate a taxonomy of children handwriting errors to be used in the subsequent studies. They hinted that the ‘peer-tutoring’ method had a positive influence on children’s learning and corrective feedback to their peers in the presence of the social robot, providing positive support to the applicability of social robots as ‘facilitators’ in handwriting learning. The next step was to study the effects of the robot in the peer-tutoring scenario when assuming the role of a peer, directly interacting and performing a collaborative writing activity with the children. Thus, we developed an educational system with an autonomous social behaviour of a robot exhibiting various levels of learning competencies. The tutor-child provides corrective feedback on the writing of the learner-robot and in turn, enhances his writing skills. The robot’s writing was generated by an algorithm incorporating human-inspired movements and could reproduce a set of writing errors from the error taxonomy. We tested the system by conducting two longitudinal studies, each of them composed of four sessions. In the fourth study, we assigned the robot two contrasting competencies: ‘learning’ and ‘non-learning’. We measured the differences in children’s learning gains and changes in their perceptions towards the learner-robot. The fifth study followed a similar interaction scenario and research questions, but this time the robot performed three learning competencies: ‘continuous-learning’;’non-learning’ and ‘personalised-learning’. The findings of these studies show that the children improved more with the robot exhibiting learning, continuous-learning and personalised-learning competencies compared to the robot showing non-learning competency. The way children’s learning and perception of the robot evolved over time confirmed the need for multiple sessions for the last two studies. This thesis presents some evidence that social robots can be successfully used as tools for enhancing children’s handwriting skills in peer-learning scenarios.