Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning

In reinforcement learning, agents learn by performing actions and observing their outcomes. Sometimes, it is desirable for a human operator to \textit{interrupt} an agent in order to prevent dangerous situations from happening. Yet, as part of their learning process, agents may link these interruptions, that impact their reward, to specific states and deliberately avoid them. The situation is particularly challenging in a multi-agent context because agents might not only learn from their own past interruptions, but also from those of other agents. Orseau and Armstrong~\cite{orseau2016safely} defined \emph{safe interruptibility} for one learner, but their work does not naturally extend to multi-agent systems. This paper introduces \textit{dynamic safe interruptibility}, an alternative definition more suited to decentralized learning problems, and studies this notion in two learning frameworks: \textit{joint action learners} and \textit{independent learners}. We give realistic sufficient conditions on the learning algorithm to enable dynamic safe interruptibility in the case of joint action learners, yet show that these conditions are not sufficient for independent learners. We show however that if agents can detect interruptions, it is possible to prune the observations to ensure dynamic safe interruptibility even for independent learners.

Multi-agent modelling of electricity markets

Abdelaziz Lahlou

This thesis describes the development of three conceptual models built to serve as decision support tools in liberalised electricity markets. The introduction of competition, higher uncertainty and decentralised planning requires new planning and analysis tools on the medium to long term to support decision making at the level of the industry as well as at the level of the market authorities. We have mainly developed models of market simulation and investment decision by investigating how multi-agent systems can contribute to the understanding of the decentralised nature of competitive decision making in a liberalised electricity market. The thesis focused particularly on bilateral transactions arguing that such transaction processes are specially relevant from the point of view of risk management when considering medium to long term decisions (for both portfolio management and investment decision making). We have used decision analysis under uncertainty and dynamic modeling approaches under a multi-agent framework by considering that such approaches are more adequate in a competitive environment due to the market actors' exposure to risks and need for adaptation in a changing environment. Monthly spot market prices are taken into account through scenarios that reproduce volatility by lognormal processes. Monte Carlo simulation is used to quantify the risks among which the spot prices are taken as the most significant source of uncertainty. The main objective of this thesis is to contribute to the problem of modeling investment decisions in new generation capacities using electricity market simulation from a planning point of view. This global objective is driven according to the: Modeling of the bilateral transaction processes between generation companies and demand companies at the level of wholesale market by taking into account the possible tradeoffs between spot and bilateral markets. The transaction processes models represent the market simulation modules by: Distinguishing the different forms the transaction processes can take (negotiation vs. auctions). Modeling of the market actors' portfolio management from a planning perspective by taking into account the market uncertainties (sales portfolio for generation companies and supply portfolios for demand companies). Modeling of the investment decisions in new electricity generation power plants. "Negotiation based portfolio management in a liberalised electricity market: a multiagent approach" The first model of the thesis formalizes the transactions of demand companies that would have to face a single generation company. The transaction process is here modeled as a bilateral negotiation algorithm between buyers and a seller. The negotiation is driven on two issues: prices and quantities of monthly contracts. It is simulated as a succession of offers and counteroffers during a limited period of time. Each market actor is searching for efficient transactions that satisfy its objectives, preferences and constraints. The reasoning process of the agents is based on the multi-attribute tradeoffs under uncertainty seeking each month the optimal configuration of portfolios composed of bilateral and spot contracts. The simulated output of the negotiation does not necessarily optimise the social welfare as both parties have opposing interests and interact under incomplete information. The efficiency of the algorithm is characterized by a ratio relatively to the Pareto optimal contracts. It was found that a supplier with nuanced risk aversion have high efficiency beyond the Pareto optimal points (between 100 and 120%). It indicates that the single generation company has a higher bargaining power than the individual demand companies independently of the preferences and negotiation behaviour of its counterparts. The negotiation efficiency of the different demand companies ranges from 80 to 100% and is related to their attitude toward risks. Risk averse demand companies accept higher risk premiums in order to avoid spot market risk exposure unlike other agents that may have more nuanced risk preferences. The bilateral contracts concluded covers the electricity demand partially or totally depending on the risk attitude. Demand companies with nuanced risk aversion negotiate contracts in order to manage some flexibility and take benefit from potential spot market opportunities. "Bilateral transactions in a competitive electricity market: multiple auctions using an agent based approach" The second transaction process models bilateral transactions between demand companies and several generation companies through multiple auctions. In this context, the generation companies must compete between each other to satisfy the call-for-bids (CfBs) launched simultaneously by each demand company, i.e. each buyer in the system opens an auction by expressing its needs and preferences by product (base and peak). After receiving the CfBs, the suppliers send offers responding to the multiple auctions opened by the demand companies. The offers timing is sequenced by a priority list that ranks the demand companies according to their value. The construction of the offers is modeled with a function of the evaluated utility of the customer based on its CfB. The demand companies, at the closure of their individual auction, compare and select the offers with maximum value by including the evaluation of the possible tradeoffs between bilateral and spot markets. The dynamic of competition requires the suppliers to adapt their monthly pricing strategies to satisfy their profit and market share objectives. The adaptive behaviour of the generation companies is here modeled using a directed learning approach. Learning is performed on the basis of two variables (risk premiums and market shares) which previous month instances permit causal inferences on eight rules of pricing parameters adaptation. The results of a case study show that the agents in the system build flexible portfolios (partial or full supply contracts, supply portfolio with several producers, spot contracts ...). The agents learning process tend to satisfy their objectives of profit and market shares. The bilateral market transaction volume represents between 70 and 90% of the total transaction volume (10 to 30% of spot contracts) as a result of the tradeoffs between the two markets. "Electricity generation capacity expansion under competition: A multi-agent dynamic programming model" The last model proposed in this thesis intends to model the decentralized investment decision making in a competitive electricity market. The capacity expansion model proposed aims at evaluating the medium to long term investment decisions according to criteria of risks and profits maximisation. We use the agent-based multiple-auction module to simulate he electricity market. The investment decisions of each supplier agent in the system are assessed according to a dynamic programming algorithm that explores different potential expansion plans in predefined periods. We consider that capital rationing, preservation of capital under risk and market share targeted ranges are three constraints that bound the expansion plans. Two major sources of uncertainties are considered: market price risk and strategic risk. Strategic risks are evaluated through a customer and competitors focus by putting into perspective the investment decision relatively to the competitors' potential expansion plans. Hence, the evaluation of an expansion plan is built as the convolution of the realizable incomes through the potential configurations of competitors strategic investment decisions. In our model, a supplier agent is seeking to take market positions relatively to its competitor through the market share that he can expect to secure its investment decisions. It appears that risk reduction highlighted here through bilateral transactions is a more realistic approach when assessing investment decisions in a competitive and uncertain electricity market.

EPFL2007

Model-based reinforcement learning and navigation in animals and machines

Dane Sterling Corneil

For decades, neuroscientists and psychologists have observed that animal performance on spatial navigation tasks suggests an internal learned map of the environment. More recently, map-based (or model-based) reinforcement learning has become a highly active research area in machine learning. With a learned model of their environment, both animals and artificial agents can generalize between tasks and learn rapidly. In this thesis, I present approaches for developing efficient model--based behaviour in machines and explaining model--based behaviour in animals. From a neuroscience perspective, I focus on the hippocampus, believed to be a major substrate of model-based behaviour in the brain. I consider how hippocampal connectivity enable path--finding between different locations in an environment. The model describes how environments with boundaries and barriers can be represented in recurrent neural networks (i.e. attractor networks), and how the transient activity in these networks, after being stimulated with a goal location, could be used for determining a path to the goal. I also propose how the connectivity of these map--like networks can be learned from the spatial firing patterns observed in the input pathway to the hippocampus (i.e. grid cells and border cells). From a machine learning perspective, I describe a reinforcement learning model that integrates model-based methods and "episodic control", an approach to reinforcement learning based on episodic memory. According to episodic control, the agent learns how to act in the environment by storing snapshot-like memories of its observations, then comparing its current observations to similar snapshot memories where it took an action that resulted in high reward. In our approach, the agent augments these real-world memories with episodes simulated offline using a learned model of the environment. These ``simulated memories'' allow the agent to adapt faster when the reward locations change. Next, I describe Variational State Tabulation (VaST), a model--based method for learning quickly with continuous and high-dimensional observations (like those found in 3D navigation tasks). The VaST agent learns to map its observations to a limited number of discrete abstract states, and build a transition model over those abstract states. The long--term values of different actions in each state are updated continuously and efficiently in the background as the agent explores the environment. I show how the VaST agent can learn faster than other state-of-the-art algorithms, even changing its policy after a single new experience, and how it can respond quickly to changing rewards in complex 3D environments. The models I present allow the agent to rapidly adapt to changing goals and rewards, a key component of intelligence. They use a combination of features attributed to model-based and episodic controllers, suggesting that the division between the two fields is not strict. I therefore also consider the consequences of these findings on theories of model-based learning, episodic control and hippocampal function.

EPFL2018

Adaptive Laser Welding Control: A Reinforcement Learning Approach

Giulio Masinelli, Silvio Zanoli

Despite extensive research efforts in the field of laser welding, the imperfect repeatability of the weld quality still represents an open topic. Indeed, the inherent complexity of the underlying physical phenomena prevents the implementation of an effective controller using conventional regulators. To close this gap, we propose the application of Reinforcement Learning for closed-loop adaptive control of welding processes. The presented system is able to autonomously learn a control law that achieves a predefined weld quality independently from the starting conditions and without prior knowledge of the process dynamics. Specifically, our control unit influences the welding process by modulating the laser power and uses optical and acoustic emission signals as sensory input. The algorithm consists of three elements: a smart agent interacting with the process, a feedback network for quality monitoring, and an encoder that retains only the quality critic events from the sensory input. Based on the data representation provided by the encoder, the smart agent decides the output laser power accordingly. The corresponding input signals are then analyzed by the feedback network to determine the resulting process quality. Depending on the distance to the targeted quality, a reward is given to the agent. The latter is designed to learn from its experience by taking the actions that maximize not just its immediate reward, but the sum of all the rewards that it will receive from that moment on. Two learning schemes were tested for the agent, namely S-Learning and Policy Gradient. The required training time to reach the targeted quality was 20 min for the former technique and 33 min for the latter.