Devanagari Character Recognition towards natural Human-Computer Interaction

Human-computer interaction is a growing research area. There are several ways of interaction with the computer. Handwriting has continued to persist as a means of communication and recording information in the day to day life even with the introduction of new technologies. Due to the growth of technology in India, it becomes important to devise ways that allow people to communicate with computer in Indian languages. Hindi being the national language of India, we present a way to communicate with the computer in Hindi or more precisely, ‘Devanagari script’. Due to absence of a global font to represent Devanagari characters, it is important that the computer recognises the characters written by the user in order to interact with him. The algorithm implemented for character recognition first segments the image containing Devanagari text fed to the software into lines, lines to words and words to characters. The obtained characters are then brought down to a standard size. The Kohonen Neural Network based recogniser then comes into action and recognises the text character by character and provides the output in Unicode format. The network has been designed with no hidden layer to support quick recognition. Apart from text recognition from an image, we also provided the option to recognise individual handwritten characters drawn using a mouse. Such a system provides keyboard less computer interaction. The technique is implemented using Java. The overall recognition rate for a fixed font machine printed characters is 90.26% and for hand written characters, it is 83.33%.

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Plamen Prodanov

Mobile service robots are going to play an increasing role in the society of humans. Voice-enabled interaction with service robots becomes very important, if such robots are to be deployed in real-world environments and accepted by the vast majority of potential human users. The research presented in this thesis addresses the problem of speech recognition integration in an interactive voice-enabled interface of a service robot, in particular a tour-guide robot. The task of a tour-guide robot is to engage visitors to mass exhibitions (users) in dialogue providing the services it is designed for (e.g. exhibit presentations) within a limited time. In managing tour-guide dialogues, extracting the user goal (intention) for requesting a particular service at each dialogue state is the key issue. In mass exhibition conditions speech recognition errors are inevitable because of noisy speech and uncooperative users of robots with no prior experience in robotics. They can jeopardize the user goal identification. Wrongly identified user goals can lead to communication failures. Therefore, to reduce the risk of such failures, methods for detecting and compensating for communication failures in human-robot dialogue are needed. During the short-term interaction with visitors, the interpretation of the user goal at each dialogue state can be improved by combining speech recognition in the speech modality with information from other available robot modalities. The methods presented in this thesis exploit probabilistic models for fusing information from speech and auxiliary modalities of the robot for user goal identification and communication failure detection. To compensate for the detected communication failures we investigate multimodal methods for recovery from communication failures. To model the process of modality fusion, taking into account the uncertainties in the information extracted from each input modality during human-robot interaction, we use the probabilistic framework of Bayesian networks. Bayesian networks are graphical models that represent a joint probability function over a set of random variables. They are used to model the dependencies among variables associated with the user goals, modality related events (e.g. the event of user presence that is inferred from the laser scanner modality of the robot), and observed modality features providing evidence in favor of these modality events. Bayesian networks are used to calculate posterior probabilities over the possible user goals at each dialogue state. These probabilities serve as a base in deciding if the user goal is valid, i.e. if it can be mapped into a tour-guide service (e.g. exhibit presentation) or is undefined – signaling a possible communication failure. The Bayesian network can be also used to elicit probabilities over the modality events revealing information about the possible cause for a communication failure. Introducing new user goal aspects (e.g. new modality events and related features) that provide auxiliary information for detecting communication failures makes the design process cumbersome, calling for a systematic approach in the Bayesian network modelling. Generally, introducing new variables for user goal identification in the Bayesian networks can lead to complex and computationally expensive models. In order to make the design process more systematic and modular, we adapt principles from the theory of grounding in human communication. When people communicate, they resolve understanding problems in a collaborative joint effort of providing evidence of common shared knowledge (grounding). We use Bayesian network topologies, tailored to limited computational resources, to model a state-based grounding model fusing information from three different input modalities (laser, video and speech) to infer possible grounding states. These grounding states are associated with modality events showing if the user is present in range for communication, if the user is attending to the interaction, whether the speech modality is reliable, and if the user goal is valid. The state-based grounding model is used to compute probabilities that intermediary grounding states have been reached. This serves as a base for detecting if the the user has reached the final grounding state, or wether a repair dialogue sequence is needed. In the case of a repair dialogue sequence, the tour-guide robot can exploit the multiple available modalities along with speech. For example, if the user has failed to reach the grounding state related to her/his presence in range for communication, the robot can use its move modality to search and attract the attention of the visitors. In the case when speech recognition is detected to be unreliable, the robot can offer the alternative use of the buttons modality in the repair sequence. Given the probability of each grounding state, and the dialogue sequence that can be executed in the next dialogue state, a tour-guide robot has different preferences on the possible dialogue continuation. If the possible dialogue sequences at each dialogue state are defined as actions, the introduced principle of maximum expected utility (MEU) provides an explicit way of action selection, based on the action utility, given the evidence about the user goal at each dialogue state. Decision networks, constructed as graphical models based on Bayesian networks are proposed to perform MEU-based decisions, incorporating the utility of the actions to be chosen at each dialogue state by the tour-guide robot. These action utilities are defined taking into account the tour-guide task requirements. The proposed graphical models for user goal identification and dialogue error handling in human-robot dialogue are evaluated in experiments with multimodal data. These data were collected during the operation of the tour-guide robot RoboX at the Autonomous System Lab of EPFL and at the Swiss National Exhibition in 2002 (Expo.02). The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation. On the component level, the technical evaluation is done by calculating accuracies, as objective measures of the performance of the grounding model, and the resulting performance of the user goal identification in dialogue. The benefit of the proposed error handling framework is demonstrated comparing the accuracy of a baseline interactive system, employing only speech recognition for user goal identification, and a system equipped with multimodal grounding models for error handling.

EPFL2006

The Vernissage Corpus: A Multimodal Human-Robot-Interaction Dataset

Daniel Gatica-Perez, Dinesh Babu Jayagopi, Laurent Son Nguyen, Jean-Marc Odobez, Samira Sheikhi

We introduce a new multimodal interaction dataset with extensive annotations in a conversational Human-Robot-Interaction (HRI) scenario. It has been recorded and annotated to benchmark many relevant perceptual tasks, towards enabling a robot to converse with multiple humans, such as speaker localization, key word spotting, speech recognition in audio domain; tracking, pose estimation, nodding, visual focus of attention estimation in visual domain; and an audio-visual task such as addressee detection. Some of the above mentioned tasks could benefit from information sensed from several modalities and recorded states of the robot. As compared to recordings done with a static camera, this corpus involves the head-movement of a humanoid robot (due to gaze change, nodding), making it challenging for tracking. Also, the significant background noise present in a real HRI setting makes tasks in the auditory domain more challenging. From the interaction point of view, our scenario, where the robot explains paintings in a room and then quizzes the participants, allows to analyze the quality of the interaction and the behavior of the human interaction partners.

2012

Offline Cursive Handwriting: From Word To Text Recognition

Alessandro Vinciarelli

This thesis presents a PhD work on offline cursive handwriting recognition, the automatic transcription of cursive data when only its image is available. Two main approaches were used in the literature to solve the problem. The first one attempts to segment the words into letters and then applies Dynamic Programming techniques to perform the recognition. The second one converts the words into sequences of observation vectors and then recognizes them with Hidden Markov Models. Some efforts were made to apply human reading inspired models, but their success was limited to applications involving small lexica (20-30 words). The HMM based approach became dominant after some years and now few works still attempt to segment the words into characters. The Sayre's paradox (a word cannot be segmented before being recognized and cannot be recognized before being segmented~\cite{say73}) was shown to be an important limit of any segmentation based approach. The research focused essentially on two applications: postal address recognition and bankcheck reading. In both cases, the application environment provides information helpful for the recognition (the zip code and the courtesy amount respectively). This simplified the problem and had an important effect on the state of the art where several issues appear to be neglected: 1) The steps preceeding the recognition (preprocessing, normalization, segmentation and feature extraction) are based on heuristics often involving data dependent algorithms and parameters to be set empirically. This leads to systems that are not robust with respect to a change of data and require a heavy experimental effort to find an optimal configuration of the parameter set. 2) Although HMMs offer a solid theoretic framework to deal with the recognition problem, most of the systems presented in the literature are still far from taking full advantage from their application. 3) The research focused on single word recognition neglecting, with very few exceptions, the transcription of texts or word sequences (e.g. dates) where language modeling can be used. 4) The experimental setup is not always rigorous. It is frequent to find works where a clear distinction between training and test set is not made. The systems are often fitted to the test set by tuning the hyperparameters over it. This leads to an overestimation of the performance. In this thesis, the problem of cursive handwriting recognition is addressed taking into account the above issues. The system we developed is based on a sliding window approach. This allows one to avoid the segmentation, a step that typically involves ad hoc algorithms and heuristics that risk to be data dependent and not robust with respect to a change of data. The normalization (slant and slope removal) is based on a statistical approach that assumes very general hypotheses about the words and that it is completely adaptive. No hyperparameters are involved so that, when changing the data, no effort is needed to reconfigure this step for the new problem. Several modeling aspects have been investigated in order to take more advantage from the HMM framework. The use of transforms (linear and nonlinear Principal Component Analysis, Independent Component Analysis) making the feature vectors more suitable for the recognition are shown to improve significantly the recognition rate. HMM adaptation techniques (Maximum Likelihood Linear Regression and Maximum A Posteriori adaptation) have been used to improve the performance of a system trained using multiple writer data over the words written by a specific person. The recognition has been extended from single word to text line recognition. In this case, the only knowledge we have about the data to be recognized is that they are written in English and they must reproduce on average the statistics of any english text. Such hypothesis is used by applying Statistical Language Models (unigrams, bigrams and trigrams). Their application results in a significant improvement of the system. All the performed experiments were designed to be correct from a statistical point of view, to avoid any fitting of the system to the test set and to give realistic measures of the performance.