Designable Visual Markers for Mobile Human-Computer Interaction

Visual markers are graphic symbols designed to be easily recognised by machines. They are traditionally used to track goods, but there is increasing interest in their application to mobile human-computer interaction (HCI). By scanning a visual marker through a camera phone, users can retrieve localised information and access mobile services. In particular the dissertation examines the application of visual markers to physical tagging: practices of association of digital information with physical items. One missed opportunity in current visual marker systems is that the markers themselves cannot be visually designed; they are not expressive to humans, and thus fail to convey information before being scanned. To address this limitation, this dissertation introduces the idea of designable markers, visual markers that are both machine-readable and visually communicative to humans, and presents an investigation of the ways in which they can support mobile human-computer interaction. The application of designable visual markers to the creation of mobile interfaces is explored through a variety of methods: through formal usability experiments, through the creation and analysis of example designs, as well as through the qualitative analysis of two field trials. All three approaches were enabled by the engineering and development of d-touch, an actual recognition system that supports designable visual markers and by its integration in a variety of applications and experimental probes. D-touch is based on image topology, and its markers are defined in terms of constraints on the nesting of dark and light regions. The constraints imposed by d-touch are flexible enough to allow novice users to create markers which are visually expressive and at the same time machine readable. A user study demonstrates how such system enables people to design their own functional visual markers, determining their aesthetic qualities and what they visually communicate to others. A desktop application to support users in the creation of valid markers, the d-touch analyser, is presented and its usefulness is demonstrated through the same study. A formal usability experiment comparing five variations of marker-based interfaces on keypad and touch-screen phones shows that all of them allow users to reliably select targets within, on average, less than 4 seconds. Participants of the experiment reported a strong preference for interfaces that involve only marker scanning, compared to those that require a combination of marker scanning and key-presses or touch selections. Example designs of mobile interface generated by the author as well as others are presented to expose how the d-touch recognition system can be integrated in mobile applications. The examples illustrate a variety of ways in which markers can be used to augment printed materials such as cards, books and product packages, adding to them interactive capabilities. The examples show also different approaches to marker design, ranging from simple and recognisable iconic design, to symbols that integrate cues about the interactive functionality, to making them invisible by hiding them in existing graphics. Finally, the dissertation reports and analyses two field trials conducted to study what practices of physical tagging can emerge from, and be supported by, the use of markers. The trials were centred around the use of uWiki, a functional prototype based on d-touch, that allows users to associate digital content to markers printed on physical tags that can be affixed to objects or buildings. Observations show that a variety of practices emerge around the use of this technology, indicating that they provide a rich medium that has potential to attract the interest of real users. Though the results of this work are preliminary, they serve to demonstrate the range of potential for the future of such systems.

Designable Visual Markers

Enrico Costanza, Jeffrey Huang

Visual markers are graphic symbols designed to be easily recognised by machines. They are traditionally used to,track goods, but there is increasing interest in their application to mobile HCI. By scanning a visual marker through a camera phone, users can retrieve localised information and access mobile services. One missed opportunity in current visual marker systems is that the markers themselves cannot be visually designed; they are not expressive to humans, and thus fail to convey information before being scanned. This paper provides an overview of d-touch, an open source system that allows users to create their own markers, and control their aesthetic qualities. The system runs in real-time on mobile phones and desktop computers. To increase computational efficiency, d-touch imposes constraints on the design of the markers in terms of the relationship of dark and light regions in the symbols. We report a user study in which pairs of novice users generated between 3 and 27 valid and expressive markers within one hour of being introduced to the system, demonstrating its flexibility and ease of use.

Acm Order Department, P O Box 64145, Baltimore, Md 21264 Usa2009

The Complementarity of Tangible and Paper Interfaces in Tabletop Environments for Collaborative Learning

Guillaume Zufferey

The current trend in Human-Computer Interaction aims at bridging the gap between the digital and the real world, exploring novel ways to engage users with computational devices. Computers take new forms that are better integrated into our environment and can be embedded in buildings, furniture or clothes. Novel forms of interfaces take advantage of people's intuitive knowledge of everyday objects to offer more direct and natural interactions. Tangible User Interfaces (TUIs) allow users to interact with digital objects through tangible artifacts, building on their rich physical affordances. Paper User Interfaces (PUIs) add digital capabilities to paper documents, synchronizing for instance their content with their digital counterpart. Unique properties of paper are also used to create engaging and intuitive interfaces to computer applications. This dissertation is interested in the complementarity of tangible and paper interfaces in tabletop environments. We introduce the concept of Tangible and Paper Environments (TaPEs) where Interactive Paper Forms (IPFs), a particular type of PUIs based on the paper form metaphor, are used as a complementary interface to a TUI. We evaluate the potential of IPFs to overcome two main shortcomings of TUIs, in terms of scalability and pedagogy. The scalability issue comes from the limited expressiveness of task-specific physical artifacts, which offer rich physical affordances but limit the complexity of applications that can be controlled by a TUI. The pedagogy issue is raised by the lack of consistent evidence regarding the use of physical manipulatives in educational settings, which is one of the main application domain of TUIs. IPFs overcome the scalability issue by offering a set of generic interaction elements that allow TaPEs to cope with applications of any complexity. In a pedagogical setting, IPFs present learners with abstract representation which facilitate understanding by the embodied and concrete representations offered by tangible artifacts. A TaPE, the Tinker Environment, has been developed with two logistics teachers in the context of the Swiss vocational training system. It consists of a warehouse physical small-scale model (TUI) and TinkerSheets, our implementation of IPFs. It aims at helping apprentices understand theoretical concepts presented at schools. We followed a Design-based Research (DBR) approach: ten studies were conducted during the development of the Tinker Environment in authentic classroom settings. Controlled experiments were conducted to address specific questions. v The general research questions concern the respective affordances of paper and tangible components of TaPEs. The analysis is not limited to usability aspects but also considers their impact on group problem-solving activities and their potential in terms of integration of the system in its context of use. A descriptive model is proposed, built around three interaction circles: individual (usability), group (collaboration) and context (integration). Results identify design guidelines that limit the impact of the less direct interaction modality offered by IPFs, allowing TaPEs to overcome the scalability issue while supporting rich interactions. At the group level, observations of groups of apprentices solving problems around the Tinker Environment show that the consistent physical interaction modality offered by TaPEs naturally supports collaborative interactions. Apprentices tend to take implicit roles based on their location around the system. Regarding the context circle, we observed that carefully designed IPFs play the role of bridges between offline and online activities and contribute to a tight integration of the system in a its context (i.e. a classroom). The specific research questions address the potential of the Tinker Environment in this pedagogical context and its appropriation by teachers. The observations conducted with the Tinker Environment show that the warehouse small-scale model reduces the complexity of problems and allows apprentices to engage in meaningful problem-solving activities. Controlled experiments comparing a TUI to a mulitouch interface demonstrate that tangible artifacts lead to a higher learning gain and an increased performance in a problem-solving activity. Collaboration quality and perceived playfulness are also improved. The teacher plays a central role in the use of the environment, guiding apprentices through activities and encouraging reflections during debriefing sessions. The design of IPFs, emphasizing either their interface or document nature, has a strong influence on their ability to support teachers. We finally discuss the two-way adaptation process that took place between teachers and the system during the development of the Tinker Environment.

EPFL2010

Error handling in multimodal voice-enabled interfaces of tour-guide robots using graphical models

Plamen Prodanov

Mobile service robots are going to play an increasing role in the society of humans. Voice-enabled interaction with service robots becomes very important, if such robots are to be deployed in real-world environments and accepted by the vast majority of potential human users. The research presented in this thesis addresses the problem of speech recognition integration in an interactive voice-enabled interface of a service robot, in particular a tour-guide robot. The task of a tour-guide robot is to engage visitors to mass exhibitions (users) in dialogue providing the services it is designed for (e.g. exhibit presentations) within a limited time. In managing tour-guide dialogues, extracting the user goal (intention) for requesting a particular service at each dialogue state is the key issue. In mass exhibition conditions speech recognition errors are inevitable because of noisy speech and uncooperative users of robots with no prior experience in robotics. They can jeopardize the user goal identification. Wrongly identified user goals can lead to communication failures. Therefore, to reduce the risk of such failures, methods for detecting and compensating for communication failures in human-robot dialogue are needed. During the short-term interaction with visitors, the interpretation of the user goal at each dialogue state can be improved by combining speech recognition in the speech modality with information from other available robot modalities. The methods presented in this thesis exploit probabilistic models for fusing information from speech and auxiliary modalities of the robot for user goal identification and communication failure detection. To compensate for the detected communication failures we investigate multimodal methods for recovery from communication failures. To model the process of modality fusion, taking into account the uncertainties in the information extracted from each input modality during human-robot interaction, we use the probabilistic framework of Bayesian networks. Bayesian networks are graphical models that represent a joint probability function over a set of random variables. They are used to model the dependencies among variables associated with the user goals, modality related events (e.g. the event of user presence that is inferred from the laser scanner modality of the robot), and observed modality features providing evidence in favor of these modality events. Bayesian networks are used to calculate posterior probabilities over the possible user goals at each dialogue state. These probabilities serve as a base in deciding if the user goal is valid, i.e. if it can be mapped into a tour-guide service (e.g. exhibit presentation) or is undefined – signaling a possible communication failure. The Bayesian network can be also used to elicit probabilities over the modality events revealing information about the possible cause for a communication failure. Introducing new user goal aspects (e.g. new modality events and related features) that provide auxiliary information for detecting communication failures makes the design process cumbersome, calling for a systematic approach in the Bayesian network modelling. Generally, introducing new variables for user goal identification in the Bayesian networks can lead to complex and computationally expensive models. In order to make the design process more systematic and modular, we adapt principles from the theory of grounding in human communication. When people communicate, they resolve understanding problems in a collaborative joint effort of providing evidence of common shared knowledge (grounding). We use Bayesian network topologies, tailored to limited computational resources, to model a state-based grounding model fusing information from three different input modalities (laser, video and speech) to infer possible grounding states. These grounding states are associated with modality events showing if the user is present in range for communication, if the user is attending to the interaction, whether the speech modality is reliable, and if the user goal is valid. The state-based grounding model is used to compute probabilities that intermediary grounding states have been reached. This serves as a base for detecting if the the user has reached the final grounding state, or wether a repair dialogue sequence is needed. In the case of a repair dialogue sequence, the tour-guide robot can exploit the multiple available modalities along with speech. For example, if the user has failed to reach the grounding state related to her/his presence in range for communication, the robot can use its move modality to search and attract the attention of the visitors. In the case when speech recognition is detected to be unreliable, the robot can offer the alternative use of the buttons modality in the repair sequence. Given the probability of each grounding state, and the dialogue sequence that can be executed in the next dialogue state, a tour-guide robot has different preferences on the possible dialogue continuation. If the possible dialogue sequences at each dialogue state are defined as actions, the introduced principle of maximum expected utility (MEU) provides an explicit way of action selection, based on the action utility, given the evidence about the user goal at each dialogue state. Decision networks, constructed as graphical models based on Bayesian networks are proposed to perform MEU-based decisions, incorporating the utility of the actions to be chosen at each dialogue state by the tour-guide robot. These action utilities are defined taking into account the tour-guide task requirements. The proposed graphical models for user goal identification and dialogue error handling in human-robot dialogue are evaluated in experiments with multimodal data. These data were collected during the operation of the tour-guide robot RoboX at the Autonomous System Lab of EPFL and at the Swiss National Exhibition in 2002 (Expo.02). The evaluation experiments use component and system level metrics for technical (objective) and user-based (subjective) evaluation. On the component level, the technical evaluation is done by calculating accuracies, as objective measures of the performance of the grounding model, and the resulting performance of the user goal identification in dialogue. The benefit of the proposed error handling framework is demonstrated comparing the accuracy of a baseline interactive system, employing only speech recognition for user goal identification, and a system equipped with multimodal grounding models for error handling.