A component-based software framework for hybrid real-time and best-effort embedded applications

Embedded systems play an increasing role in our society. From consumer electronics to driving assistance in cars, from medical devices to space exploration, they are ubiquitous. To better assist human beings, embedded systems become more complex, and more intelligent or even autonomous behaviors are expected. To handle their interaction with the real-world, those systems rely on artificial intelligence methods but are also subject to real-time constraints imposed by their environment. Taking advantage of increasingly powerful embedded processors, most of the new intelligent functionalities are implemented in software. As a result, the complexity of embedded software increases dramatically and can no longer be managed by domain specialists that are neither computer scientists nor software engineers. Unfortunately, the computer science community has left aside embedded systems for many years, and most modern software engineering paradigms cannot be applied directly to develop embedded applications. In this work, we present a hybrid framework based on software components that embed value-adding contributions from domain specialists. The framework relies on a dual-kernel approach that executes both a real-time and a best-effort operating system on a single processor. It is centered on dual-portability of software components across hardware platforms and across execution modes. The hybrid framework empowers domain specialists and promotes reuse of valuable software assets. It also supports the generation of hybrid embedded applications that combine real-time and best-effort software components. Such applications can take advantage of both time determinism and high computational throughput to face uncertain and dynamic real-world environments in an intelligent manner. The application of the hybrid framework is illustrated through real-world case studies with autonomous mobile robots from ASL (Autonomous Systems Laboratory) and NASA (National Aeronautics and Space Administration). It is also adapted to support the development of integrated satellite payload controllers for ESA (European Space Agency).

Active: a unified platform for building intelligent applications

Didier Guzzoni

Computers have become affordable, small, omnipresent and are often connected to the Internet. However, despite the availability of such rich environments, user interfaces have not been adapted to fully leverage their potential. In our view, user interfaces will evolve to become more than simple tools that act using a click-and-act paradigm – people want software that can act as assistants to whom tasks can be delegated. This new type of software will provide us with more user-centric systems, able to interact naturally with human users and with the information environment. Although progress has been made in this direction over the last decade, current research in the field has shown that building intelligent assistants is a complex task that requires expertise in many fields, ranging from artificial intelligence to core software and hardware engineering. The difficulty of deeply integrating all the technologies and AI methodologies required to produce robust intelligent assistants has greatly limited the impact and widespread adoption of this form of software. In this thesis, we propose to design, implement and evaluate a new methodology and associated tool suite aimed to ease and accelerate the development of intelligent assistant software. Active, our solution, introduces the original concept of Active Ontologies, and combines it with a service oriented architecture to serve as the foundation for user-centric applications. The Active software suite features a programming editor, a runtime server and associated management tools. Using this unified platform, we developed techniques for rapidly creating intelligent assistant applications that weave together language processing, process modeling and dynamic service orchestration. To validate our approach, three prototypes have been implemented and evaluated. First, we created an assistant that helps mobile users retrieve data and access online services. The system allows users to fetch information through natural language dialog about restaurants, hotels, points of interest, flights status and weather forecasts. As a second prototype, we have implemented an assistant to help surgeons in the context of the operating room. The system uses a combination of voice and gesture recognition to help surgeons navigate through pre-operative information, visualize and control a live video stream coming from an endoscope mounted on a robotic arm. Lastly, we have created a system that helps organize meetings through emails and instant messages with the organizer and attendees. These three prototypes are deployed in very different application domains, and yet are built with the same tools and methods; this demonstrates the flexibility and versatility of our approach. We conducted a user study to validate our claim that, in specific domains, intelligent assistant software can perform better than conventional software approaches. First, we compared our mobile assistant against Google Mobile™, the leading commercially available search application for mobile users. We asked a population of users to accomplish ten travel-related tasks with both systems. Results show that, for the travel domain, our assistant-based system performs significantly better both in terms of effectiveness and time to completion than the more conventional, keyword-based search engine. In the medical domain, we asked a small population of surgeons to accomplish a set of tasks with our second prototype. After providing us with a positive feedback, surgeons used our prototype to discuss and experiment with innovative forms of user interaction in the context of the operating room. We also performed evaluations to support our claim that our unified approach accelerates the software development of intelligent assistants. First, we gave basic training on the Active platform to a population of software developers, asking them to go through a simple tutorial. Once trained, they were able to successfully program the core component of the mobile assistant prototype in under two hours. As a second validation, the meeting organizing assistant was designed, implemented and tested within three working days – comparable systems published in the literature were the product of significantly more time and resource investment. This demonstrates that a system encompassing numerous AI and software development aspects, including language processing, multiple modalities, contextual dialog, processing logic and backend services integration, can be rapidly implemented and tested using the Active framework. Our work shows that for many task-oriented domains, intelligent assistant applications can be significantly more effective than conventional systems. Through our Active Ontology approach and associated methodologies and tools, we have shown that complex intelligent assistant applications, perhaps for the first time, can be developed with low cost and high reliability. In addition to receiving a positive echo from the research world, our work is also causing ripples in the commercial world.

EPFL2008

Software integration in mobile robotics, a science to scale up machine intelligence

Stéphane Magnenat

The present work tackles integration in mobile robotics. Integration is often considered to be a mere technique, unworthy of scientific investigation. On the contrary, we show that integrating capabilities in a mobile robot entails new questions that the parts alone do not feature. These questions reflect the structure of the application and the physics of the world. We also show that a successful integration process transforms the parts themselves and allows to scale up mobile-robot intelligence in real-world applications. In Chapter 2 we present the hardware. In Chapter 3, we show that building a low-level control architecture considering the mechanic and electronic reality of the robot improves the performances and allows to integrate a large number of sensors and actuators. In Chapter 4, we show that globally optimising mechatronic parameters considering the robot as a whole allows to implement slam using an inexpensive sensor with a low processor load. In Chapter 5, we show that based on the output from the slam algorithm, we can combine infrared proximity sensors and vision to detect objects and to build a semantic map of the environment. We show how to find free paths for the robot and how to create a dual geometric-symbolic representation of the world. In Chapter 6, we show that the nature of scenarios influences the implementation of a task-planning algorithm and changes its execution properties. All these chapters contribute results that together prove that integration is a science. In Chapter 7, we show that combining these results improves the state of the art in a difficult application : autonomous construction in unknown environments with scarce resources. This application is interesting because it is challenging at multiple levels : For low-level control, manipulating objects in the real world to build structures is difficult. At the level of perceptions, the fusion of multiple heterogeneous inexpensive sensors is not trivial, because these sensors are noisy and the noise is non-Gaussian. At the level of cognition, reasoning about elements from an unknown world in real time on a miniature robot is demanding. Building this application upon our other results proves that integration allows to scale up machine intelligence, because this application shows intelligence that is beyond the state of the art, still only combining basic components that are individually slightly behind the state of the art.

EPFL2010

Tales from a Robotic World. How Intelligent Machines Will Shape Our Future

Dario Floreano, Nicola Nosengo

Stories from the future of intelligent machines—from rescue drones to robot spouses—and accounts of cutting-edge research that could make it all possible. Tech prognosticators promised us robots—autonomous humanoids that could carry out any number of tasks. Instead, we have robot vacuum cleaners. But, as Dario Floreano and Nicola Nosengo report, advances in robotics could bring those rosy predictions closer to reality. A new generation of robots, directly inspired by the intelligence and bodies of living organisms, will be able not only to process data but to interact physically with humans and the environment. In this book, Floreano, a roboticist, and Nosengo, a science writer, bring us tales from the future of intelligent machines—from rescue drones to robot spouses—along with accounts of the cutting-edge research that could make it all possible. These stories from the not-so-distant future show us robots that can be used for mitigating effects of climate change, providing healthcare, working with humans on the factory floor, and more. Floreano and Nosengo tell us how an application of swarm robotics could protect Venice from flooding, how drones could reduce traffic on the congested streets of mega-cities like Hong Kong, and how a “long-term relationship model” robot could supply sex, love, and companionship. After each fictional scenario, they explain the technologies that underlie it, describing advances in such areas as soft robotics, swarm robotics, aerial and mobile robotics, humanoid robots, wearable robots, and even biohybrid robots based on living cells. Robotics technology is no silver bullet for all the world's problems—but it can help us tackle some of the most pressing challenges we face.