Partially observable Markov decision process

A partially observable Markov decision process (POMDP) is a generalization of a Markov decision process (MDP). A POMDP models an agent decision process in which it is assumed that the system dynamics are determined by an MDP, but the agent cannot directly observe the underlying state. Instead, it must maintain a sensor model (the probability distribution of different observations given the underlying state) and the underlying MDP. Unlike the policy function in MDP which maps the underlying states to the actions, POMDP's policy is a mapping from the history of observations (or belief states) to the actions. The POMDP framework is general enough to model a variety of real-world sequential decision processes. Applications include robot navigation problems, machine maintenance, and planning under uncertainty in general. The general framework of Markov decision processes with imperfect information was described by Karl Johan Åström in 1965 in the case of a discrete state space, and it was

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Related publications (25)

Related publications

Related people (3)

Related people

Related units (1)

Related units

Related concepts (1)

Related concepts

Related courses (6)

Related courses

Related lectures (9)

Related lectures

Measurement Scheduling for Soil Moisture Sensing: From Physical Models to Optimal Control

Ke Li, Aditya Mahajan, David Shuman

In this paper, we consider the problem of monitoring soil moisture evolution using a wireless network of in situ sensors. Continuously sampling moisture levels with these sensors incurs high-maintenance and energy consumption costs, which are particularly undesirable for wireless networks. Our main hypothesis is that a sparser set of measurements can meet the monitoring objectives in an energy-efficient manner. The underlying idea is that we can trade off some inaccuracy in estimating soil moisture evolution for a significant reduction in energy consumption. We investigate how to dynamically schedule the sensor measurements so as to balance this tradeoff. Unlike many prior studies on sensor scheduling that make generic assumptions on the statistics of the observed phenomenon, we obtain statistics of soil moisture evolution from a physical model. We formulate the optimal measurement scheduling and estimation problem as a partially observable Markov decision problem (POMDP). We then utilize special features of the problem to approximate the POMDP by a computationally simpler finite-state Markov decision problem (MDP). The result is a scalable, implementable technology that we have tested and validated numerically and in the field.

2010

Design and Performance Evaluation of an Algorithm Based on Source Term Estimation for Odor Source Localization

Ali Marjovi, Alcherio Martinoli, Faezeh Rahbar

Finding sources of airborne chemicals with mobile sensing systems finds applications across safety, security, environmental monitoring, and medical domains. In this paper, we present an algorithm based on Source Term Estimation for odor source localization that is coupled with a navigation method based on partially observable Markov decision processes. We propose a novel strategy to balance exploration and exploitation in navigation. Moreover, we study two variants of the algorithm, one exploiting a global and the other one a local framework. The method was evaluated through high-fidelity simulations and in a wind tunnel emulating a quasi-laminar air flow in a controlled environment, in particular by systematically investigating the impact of multiple algorithmic and environmental parameters (wind speed and source release rate) on the overall performance. The outcome of the experiments showed that the algorithm is robust to different environmental conditions in the global framework, but, in the local framework, it is only successful in relatively high wind speeds. In the local framework, on the other hand, the algorithm is less demanding in terms of energy consumption as it does not require any absolute positioning information from the environment and the robot travels less distance compared to the global framework.

2019

An Algorithm for Odor Source Localization based on Source Term Estimation

Ali Marjovi, Alcherio Martinoli, Faezeh Rahbar

Finding sources of airborne chemicals with mobile sensing systems finds applications across the security, safety, domestic, medical, and environmental domains. In this paper, we present an algorithm based on source term estimation for odor source localization that is coupled with a navigation method based on partially observable Markov decision processes. We propose an innovative strategy to balance exploration and exploitation in navigation. The method has been evaluated systematically through high-fidelity simulations and in a wind tunnel emulating realistic and repeatable conditions. The impact of multiple algorithmic and environmental parameters has been studied in the experiments.

2019

CS-430: Intelligent agents

Software agents are widely used to control physical, economic and financial processes. The course presents practical methods for implementing software agents and multi-agent systems, supported by programming exercises, and the theoretical underpinnings including computational game theory.

MGT-484: Applied probability & stochastic processes

This course focuses on dynamic models of random phenomena, and in particular, the most popular classes of such models: Markov chains and Markov decision processes. We will also study applications in queuing theory, finance, project management, etc.

CS-421: Machine learning for behavioral data

Computer environments such as educational games, interactive simulations, and web services provide large amounts of data, which can be analyzed and serve as a basis for adaptation. This course will cover the core methods of user modeling and personalization, with a focus on educational data.

Artificial intelligence

Artificial intelligence (AI) is the intelligence of machines or software, as opposed to the intelligence of human beings or animals. AI applications include advanced web search engines (e.g., Google