Low-level grounding in a multimodal mobile service robot conversational system using graphical models

Anil Alexander, Andrzej Drygajlo, Plamen Prodanov, Jonas Richiardi
Springer-Verlag, 2008
Journal paper

Abstract

The main task of a service robot with a voice-enabled communication interface is to engage a user in dialogue providing an access to the services it is designed for. In managing such interaction, inferring the user goal (intention) from the request for a service at each dialogue turn is the key issue. In service robot deployment conditions speech recognition limitations with noisy speech input and inexperienced users may jeopardize user goal identification. In this paper, we introduce a grounding state-based model motivated by reducing the risk of communication failure due to incorrect user goal identification. The model exploits the multiple modalities available in the service robot system to provide evidence for reaching grounding states. In order to handle the speech input as sufficiently grounded (correctly understood) by the robot, four proposed states have to be reached. Bayesian networks combining speech and non-speech modalities during user goal identification are used to estimate probability that each grounding state has been reached. These probabilities serve as a base for detecting whether the user is attending to the conversation, as well as for deciding on an alternative input modality (e.g., buttons) when the speech modality is unreliable. The Bayesian networks used in the grounding model are specially designed for modularity and computationally efficient inference. The potential of the proposed model is demonstrated comparing a conversational system for the mobile service robot RoboX employing only speech recognition for user goal identification, and a system equipped with multimodal grounding.The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation with multimodal data collected during the conversations of the robot RoboX with users.

Official source

https://infoscience.epfl.ch/record/141830?ln=en

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related concepts

Related publications

Anil Alexander, Andrzej Drygajlo, Plamen Prodanov, Jonas Richiardi
Springer-Verlag, 2008
Journal paper

Abstract

Official source

https://infoscience.epfl.ch/record/141830?ln=en

About this result

Related concepts (20)

Related concepts

Related publications (21)

Related publications

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Plamen Prodanov

Mobile service robots are going to play an increasing role in the society of humans. Voice-enabled interaction with service robots becomes very important, if such robots are to be deployed in real-world environments and accepted by the vast majority of potential human users. The research presented in this thesis addresses the problem of speech recognition integration in an interactive voice-enabled interface of a service robot, in particular a tour-guide robot. The task of a tour-guide robot is to engage visitors to mass exhibitions (users) in dialogue providing the services it is designed for (e.g. exhibit presentations) within a limited time. In managing tour-guide dialogues, extracting the user goal (intention) for requesting a particular service at each dialogue state is the key issue. In mass exhibition conditions speech recognition errors are inevitable because of noisy speech and uncooperative users of robots with no prior experience in robotics. They can jeopardize the user goal identification. Wrongly identified user goals can lead to communication failures. Therefore, to reduce the risk of such failures, methods for detecting and compensating for communication failures in human-robot dialogue are needed. During the short-term interaction with visitors, the interpretation of the user goal at each dialogue state can be improved by combining speech recognition in the speech modality with information from other available robot modalities. The methods presented in this thesis exploit probabilistic models for fusing information from speech and auxiliary modalities of the robot for user goal identification and communication failure detection. To compensate for the detected communication failures we investigate multimodal methods for recovery from communication failures. To model the process of modality fusion, taking into account the uncertainties in the information extracted from each input modality during human-robot interaction, we use the probabilistic framework of Bayesian networks. Bayesian networks are graphical models that represent a joint probability function over a set of random variables. They are used to model the dependencies among variables associated with the user goals, modality related events (e.g. the event of user presence that is inferred from the laser scanner modality of the robot), and observed modality features providing evidence in favor of these modality events. Bayesian networks are used to calculate posterior probabilities over the possible user goals at each dialogue state. These probabilities serve as a base in deciding if the user goal is valid, i.e. if it can be mapped into a tour-guide service (e.g. exhibit presentation) or is undefined – signaling a possible communication failure. The Bayesian network can be also used to elicit probabilities over the modality events revealing information about the possible cause for a communication failure. Introducing new user goal aspects (e.g. new modality events and related features) that provide auxiliary information for detecting communication failures makes the design process cumbersome, calling for a systematic approach in the Bayesian network modelling. Generally, introducing new variables for user goal identification in the Bayesian networks can lead to complex and computationally expensive models. In order to make the design process more systematic and modular, we adapt principles from the theory of grounding in human communication. When people communicate, they resolve understanding problems in a collaborative joint effort of providing evidence of common shared knowledge (grounding). We use Bayesian network topologies, tailored to limited computational resources, to model a state-based grounding model fusing information from three different input modalities (laser, video and speech) to infer possible grounding states. These grounding states are associated with modality events showing if the user is present in range for communication, if the user is attending to the interaction, whether the speech modality is reliable, and if the user goal is valid. The state-based grounding model is used to compute probabilities that intermediary grounding states have been reached. This serves as a base for detecting if the the user has reached the final grounding state, or wether a repair dialogue sequence is needed. In the case of a repair dialogue sequence, the tour-guide robot can exploit the multiple available modalities along with speech. For example, if the user has failed to reach the grounding state related to her/his presence in range for communication, the robot can use its move modality to search and attract the attention of the visitors. In the case when speech recognition is detected to be unreliable, the robot can offer the alternative use of the buttons modality in the repair sequence. Given the probability of each grounding state, and the dialogue sequence that can be executed in the next dialogue state, a tour-guide robot has different preferences on the possible dialogue continuation. If the possible dialogue sequences at each dialogue state are defined as actions, the introduced principle of maximum expected utility (MEU) provides an explicit way of action selection, based on the action utility, given the evidence about the user goal at each dialogue state. Decision networks, constructed as graphical models based on Bayesian networks are proposed to perform MEU-based decisions, incorporating the utility of the actions to be chosen at each dialogue state by the tour-guide robot. These action utilities are defined taking into account the tour-guide task requirements. The proposed graphical models for user goal identification and dialogue error handling in human-robot dialogue are evaluated in experiments with multimodal data. These data were collected during the operation of the tour-guide robot RoboX at the Autonomous System Lab of EPFL and at the Swiss National Exhibition in 2002 (Expo.02). The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation. On the component level, the technical evaluation is done by calculating accuracies, as objective measures of the performance of the grounding model, and the resulting performance of the user goal identification in dialogue. The benefit of the proposed error handling framework is demonstrated comparing the accuracy of a baseline interactive system, employing only speech recognition for user goal identification, and a system equipped with multimodal grounding models for error handling.

EPFL2006

On developing a voice-enabled interface for interactive tour-guide robots

Andrzej Drygajlo, Plamen Prodanov

This paper considers design methodologies in order to develop voice-enabled interfaces for tour-guide robots deployed at the Robotics Exposition of the Swiss National Exhibition (Expo.02). Human–robot voice communication presents new challenges for design of fully autonomous mobile robots, in that interactivity must be robot-initiated in conversation and within a dynamic adverse environment.We approached these general problems for a voice-enabledinterface, tailored to limited computational resources of one on-board processor,when integrating smart speech signal acquisition, automatic speech recognition and synthesis, as well as a dialogue system into the multi-modal, multi-sensor interface for the expo tour-guide robot. We also focus on particular issues that needed to be addressed in voice-based interaction when planning specific tasks and research experiments for Expo.02, where tour-guide robots had to interact with hundreds of thousands of visitors over 6 months, 7 days a week, 10 h per day.

2003

Robots are employed to assist humans in lengthy, challenging, and repetitive tasks. However, the fields of rehabilitation, haptics, and assistive robotics have shown a significant need to support and interact with people in their everyday life. To facilitate these further needs, robots must overcome constraints in compactness, degrees of freedom (DoF), and mechanical performance. However, developing a robot with these characteristics is challenging for conventional mechanics. First, creating a meso-scale device requires the assembly of small components, which is laborious and increases manufacturing cost, time, and complexity. In addition, devices with several DoF and functionalities also require more parts to create numerous links and joints, actuators, and thus space, affecting the compactness of systems. Finally, aiming for mechanical performance in terms of force, speed, and motion range requires transmissions or bulky actuators, which also influence the size of the system. This thesis attempts to overcome the limitations of conventional mechanics by using origami robots, an alternative design approach for creating meso-scale structures with numerous folding joints. It consists in stacking layers of functional material to achieve low-profile and scalable structures that fold into 3D. Using this method, we developed three hybrid robots combining the decades of experience of conventional mechanics with the advantages of origami robotics. First, to enhance the performances of robots at the meso-scale, the creation of compact low-profile compliant transmissions benefiting from origami's minimal assembly of functional layers was explored. The transmissions obtained can reproduce conventional kinematics or generate unconventional motions using materials properties. Second, the development of multi-DoF compact robots was studied using the aforementioned low-profile compliant transmissions to create a multi-functional structure. The resulting robot is a fingertip haptic device called Haptigami that can render vibrotactile and three-DoF force feedback without compromising bulkiness. Then, this study aimed at creating a meso-scale, multi-DoF, and mechanically efficient robot. We combined the previous device contributions and developed Flexure Variable Stiffness Actuators (F-VSA), a novel compliant actuator that uses the origami flexure hinges' inherent stiffness to avoid hindering mechanical simplicity. Finally, we developed a software architecture that enables high-level control of the human-robot hardware and is compatible with multiple simulation engines. We used this architecture to control our robots and provide feedback to the user, based on interactions in a virtual environment. These novel robots rely on non-conventional structures, actuators, and kinematics that require exploring and testing suitable control strategies. Hence, kinematic and stiffness models were developed and used to tune the position, force, and stiffness of the end-effector. Several characterization platforms and experimental protocols suitable for testing our unique robotic systems were also developed to assess their precision and efficiency. Consequently, the results of this study pave the way for the development of robots that adapt to multiple environments, interactions, and application scenarios, compatible with the human bandwidth without hindering the user's motion and comfort in everyday life.

EPFL2022

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Plamen Prodanov

EPFL2006

EPFL2022