Active: a unified platform for building intelligent applications

Computers have become affordable, small, omnipresent and are often connected to the Internet. However, despite the availability of such rich environments, user interfaces have not been adapted to fully leverage their potential. In our view, user interfaces will evolve to become more than simple tools that act using a click-and-act paradigm – people want software that can act as assistants to whom tasks can be delegated. This new type of software will provide us with more user-centric systems, able to interact naturally with human users and with the information environment. Although progress has been made in this direction over the last decade, current research in the field has shown that building intelligent assistants is a complex task that requires expertise in many fields, ranging from artificial intelligence to core software and hardware engineering. The difficulty of deeply integrating all the technologies and AI methodologies required to produce robust intelligent assistants has greatly limited the impact and widespread adoption of this form of software. In this thesis, we propose to design, implement and evaluate a new methodology and associated tool suite aimed to ease and accelerate the development of intelligent assistant software. Active, our solution, introduces the original concept of Active Ontologies, and combines it with a service oriented architecture to serve as the foundation for user-centric applications. The Active software suite features a programming editor, a runtime server and associated management tools. Using this unified platform, we developed techniques for rapidly creating intelligent assistant applications that weave together language processing, process modeling and dynamic service orchestration. To validate our approach, three prototypes have been implemented and evaluated. First, we created an assistant that helps mobile users retrieve data and access online services. The system allows users to fetch information through natural language dialog about restaurants, hotels, points of interest, flights status and weather forecasts. As a second prototype, we have implemented an assistant to help surgeons in the context of the operating room. The system uses a combination of voice and gesture recognition to help surgeons navigate through pre-operative information, visualize and control a live video stream coming from an endoscope mounted on a robotic arm. Lastly, we have created a system that helps organize meetings through emails and instant messages with the organizer and attendees. These three prototypes are deployed in very different application domains, and yet are built with the same tools and methods; this demonstrates the flexibility and versatility of our approach. We conducted a user study to validate our claim that, in specific domains, intelligent assistant software can perform better than conventional software approaches. First, we compared our mobile assistant against Google Mobile™, the leading commercially available search application for mobile users. We asked a population of users to accomplish ten travel-related tasks with both systems. Results show that, for the travel domain, our assistant-based system performs significantly better both in terms of effectiveness and time to completion than the more conventional, keyword-based search engine. In the medical domain, we asked a small population of surgeons to accomplish a set of tasks with our second prototype. After providing us with a positive feedback, surgeons used our prototype to discuss and experiment with innovative forms of user interaction in the context of the operating room. We also performed evaluations to support our claim that our unified approach accelerates the software development of intelligent assistants. First, we gave basic training on the Active platform to a population of software developers, asking them to go through a simple tutorial. Once trained, they were able to successfully program the core component of the mobile assistant prototype in under two hours. As a second validation, the meeting organizing assistant was designed, implemented and tested within three working days – comparable systems published in the literature were the product of significantly more time and resource investment. This demonstrates that a system encompassing numerous AI and software development aspects, including language processing, multiple modalities, contextual dialog, processing logic and backend services integration, can be rapidly implemented and tested using the Active framework. Our work shows that for many task-oriented domains, intelligent assistant applications can be significantly more effective than conventional systems. Through our Active Ontology approach and associated methodologies and tools, we have shown that complex intelligent assistant applications, perhaps for the first time, can be developed with low cost and high reliability. In addition to receiving a positive echo from the research world, our work is also causing ripples in the commercial world.