Multimodal interaction

Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data. Introduction Multimodal human-computer interaction refers to the "interaction with the virtual and physical environment through natural modes of communication", This implies that multimodal interaction enables a more free and natural communication, interfacing users with automated systems in both input and output. Specifically, multimodal systems can offer a flexible, efficient and usable environment allowing users to interact through input modalities, such as speech, handwriting, hand gesture and gaze, and to receive information by the system through output modalities, such as speech synthesis, smart graphics and other modalities, opportunely combined. Then a multimodal system has to recognize the inputs from the different modalities combining them according to temporal and contextual constraints in

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications

Related people

Related units

Related concepts

Related courses

Related lectures

Summary

Official source

About this result

Related publications (100)

Related publications

Related people (31)

Related people

Related units (14)

Related units

Related concepts (4)

Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken

Human–computer interaction (HCI) is research in the design and the use of computer technology, which focuses on the interfaces between people (users) and computers. HCI researchers observe the ways h

Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms. It is a subdiscipline of computer vision. Gesture

Related concepts

Related courses (5)

Related courses

Related lectures (3)

Related lectures

Enhanced Performance of Multimodal Biometric Systems by Confidence Estimation

The main scope of this project is to identify the best method of confidence estimator whose performance could be reliable in comparison to multimodal fusion alone. To do that, three alternative approaches to prediction confidence estimation are presented and compared. Among the three methods, the first one is the Gaussian hypothesis, the second one is the non parametric and the third one is the proposed distance method, which will be discuss later on in section 4.3 The three techniques are tested and compared on two different types of fusion (face and speech) methods namely multi layer perceptrons (MLP) and support vector machine (SVM) and the basic modalities were speech and face.

IDIAP2003

Design of multimodal dialogue-based systems

Miroslav Melichar

Multimodal dialogue systems integrate advanced (often spoken) language technologies within human-computer interaction methods. Such complex systems cannot be designed without extensive human expertise and systematic design guidelines taking into account the limitations of the underlying technologies. Therefore, this thesis aims at reducing the time and effort needed to build such systems by creating application-independent techniques, tools and algorithms that automate the design process and make it accessible for non-expert application developers. The thesis proposes an interactive system prototyping methodology, which (together with its software implementation) allows for rapid building of multimodal dialogue-based information seeking systems. When designed with our methodology, even partially implemented system prototypes can immediately be tested with users through Wizard of Oz simulations (which are integrated into the methodology) that reveal user behavior and modality use models. Involving users in early development phases increases the chances for the targeted system to be well accepted by end-users. With respect to dialogue system design, we propose a two-layered dialogue model as a variant of the standard frame-based approach. The two layers of the proposed dialogue model correspond to local and global dialogue strategies. One of the important findings of our research is that the two-layered dialogue model is easily extendable to multimodal systems. The methodology is illustrated in full detail through the design and implementation of the Archivus system – a multimodal (mouse, pen, touchscreen, keyboard and voice) interface that allows users to access and search a database of recorded and annotated meetings (the Smart Meeting Room application). The final part of the thesis is dedicated to an overall qualitative evaluation of the Archivus system (user's performance, satisfaction, analysis of encountered problems) and to a quantitative evaluation of all the implemented dialogue strategies. Our methodology is intended (1) for designers of multimodal systems who want to quickly develop a multimodal system in their application domain, (2) for researchers who want to better understand human-machine multimodal interaction through experimenting with working prototypes, (3) for researchers who want to test new modalities within the context of a complete application, and (4) for researchers interested in new approaches to specific issues related to multimodal systems (e.g. the multimodal fusion problem).

EPFL2008

Using sign language, speech, and haptics as communication modalities, a virtual treasure-hunting game serves as an entertainment and educational tool for visually and hearing impaired users. In recent years, there has been an increasing interest in human-computer interaction (HCI) for multimodal interfaces. Since Sutherland's SketchPad in 1961 and Xerox's Alto in 1973, computer users have long been acquainted with technol ogies other than the traditional keyboard for interacting with a system. Recently, with the desire for increased productivity, seamless in teraction, immersion, and e-inclusion of people with disabilities, along with progress in fields such as multimedia, multimodal signal analy sis, and HCI, multimodal interaction has emerged as an active field of research.

2011

Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms. It is a subdiscipline of computer vision. Gesture

EE-612: Fundamentals in statistical pattern recognition

This course provides in-depth understanding of the most fundamental algorithms in statistical pattern recognition or machine learning (including Deep Learning) as well as concrete tools (as Python source code) to PhD students for their work.

EE-623: Perception and learning from multimodal sensors

The course will cover different aspects of multimodal processing (complementarity vs redundancy; alignment and synchrony; fusion), with an emphasis on the analysis of people, behaviors and interactions from multimodal sensor, using statistical models and deep learning as main modeling tools.

CS-444: Virtual reality

The goal of VR is to embed the users in a potentially complex virtual environment while ensuring that they are able to react as if this environment were real. The course provides a human perception-action background and describes the key programming techniques for achieving efficient VR applications