Self-Exciting Point Processes: Identification and Control

This thesis addresses theoretical and practical aspects of identification and subsequent control of self-exciting point processes. The main contributions correspond to four separate scientific papers.In the first paper, we address the challenge of robust identification of controlled Hawkes processes in applications with sparsely available data. Specifically, we propose an alternative approach based on an expectation-maximization algorithm, which instrumentalizes the internal branching structure of the process, thus improving the estimator's convergence behavior. Additionally, we show that our method provides a tight lower bound for maximum-likelihood estimates. The relevance of the proposed technique is demonstrated on the practical application of credit collections and trading in the presence of macroeconomic news. The second and third paper focus on the optimal control of self-exciting point processes using the reinforcement-learning paradigm. Contrary to traditional reinforcement learning applications, environments driven by Hawkes-like dynamics feature an asynchronous action-reward relationship which complicates attributing actions to their consequent rewards, and thus hinders learning. To this end, we formulate a novel reward shaping theorem that provides a continuous reward analogue that enables learning in such environments. Furthermore, with the growing need for interpretable machine-learning models we formulate a monotonicity regularizer that embeds domain expertise into the learning. Our formulation overcomes the challenge of learning interpretable policies by constraining the policy space with a priori expected structural properties, producing state-feedback control laws that can be readily understood and implemented by human decision-makers. Again the results are developed in the context of credit collections but are straightforwardly applicable to other problems with self-exciting dynamics.Finally, the last paper consists of an empirical investigation of cryptocurrency market microstructure through the optics of Hawkes processes. We construct a 'reflexivity' index that measures the activity generated endogenously within cryptocurrency markets by fitting a univariate self-exciting Hawkes process with two classes of parametric kernels to high-frequency trading data. Our parsimonious model allows for an elegant separation and quantification of endogenous and exogenous dynamics, and thus allows for a direct market microstructure comparison with traditional asset classes in terms of identified branching ratios. Furthermore, we formulate a 'Hawkes disorder problem,' as a generalization of the established Poisson disorder problem, and provide a simulation-based approach to determining an optimal observation horizon---a critical consideration in the high-frequency finance context. Our analysis suggests that Bitcoin mid-price dynamics feature long-memory properties, well explained by the power-law kernel, at a level of criticality similar to fiat-currency markets.

Learning to Reduce Annotation Load

Ksenia Konyushkova

Modern machine learning methods and their applications in computer vision are known to crave for large amounts of training data to reach their full potential. Because training data is mostly obtained through humans who manually label samples, it induces a significant cost. Therefore, the problem of reducing the annotation load is of great importance for the success of machine learning methods. We study the problem of reducing the annotation load from two viewpoints, by answering the questions âWhat to annotate?â and âHow to annotate?â. The question âWhat?â addresses the selection of a small portion of the data that would be sufficient to train an accurate model. The question âHow? focuses on minimising the effort of labelling each datapoint. The question âWhat to annotate?â becomes particularly compelling if we can select data to be annotated in an iterative and adaptive way, a setting known as active learning (AL). The key challenge in AL is to identify the datapoints that are the most informative for the model at a given stage. We propose several techniques to address this challenge. Firstly, we consider the problem of segmenting natural images and image volumes. We take advantage of image priors, such as smoothness of objects of interest, and use them in a novel form of geometric uncertainty. Using this, we design an AL technique to efficiently annotate data that is tailored to segmentation applications. Next, we notice that no single manually-designed strategy outperforms others in every application and that often the burden of designing new strategies outweighs the benefits of AL. To overcome this problem we suggest learning an AL strategy from data by formulating the AL problem as a regression task that predicts the reduction in the generalisation error achieved by labelling each datapoint. This enables us to learn AL strategies from simulated data and to transfer them to new datasets. Finally, we turn towards non-myopic data-driven AL strategies. To this end, we formulate the AL problem as a Markov decision process and find the best selection policy using reinforcement learning. We design the decision process such that the policy can be learnt for any ML model and transferred to diverse application domains. Effectively addressing the question âHow to annotate?â is of no less importance as large cost savings can be achieved by labelling each datapoint more efficiently. This can be done with intelligent interfaces that interact with a human annotator. We make two contributions towards answering the question âHow?â. Firstly, we propose an efficient technique to annotate 3D image volumes for image segmentation. Annotating data in 3D is cumbersome and an obvious way to facilitate it is to select a subset of the data lying on a 2D plane. To find the optimal plane (i.e. the one containing the most informative datapoints) we design a branch-and-bound algorithm that quickly eliminates hypotheses about the optimal projection. Secondly, we propose an intelligent data annotation method to train object detectors. Instead of always asking the human annotator to draw bounding boxes in images, we detect automatically in which cases we can rely on the current detector and verify its proposal.

EPFL2019

Surprise-based model estimation in reinforcement learning: algorithms and brain signatures

Vasiliki Liakoni

Learning how to act and adapting to unexpected changes are remarkable capabilities of humans and other animals. In the absence of a direct recipe to follow in life, behaviour is often guided by rewarding and by surprising events. A positive or a negative outcome influences the tendency to repeat some actions, and a sudden unexpected event signals the possible need to act differently or to update one's view about the world. Advances in computational, behavioral, and cognitive neuroscience have indicated that animals employ multiple strategies to learn from interaction. However, our understanding of learning strategies and how they are combined is largely restricted. The main goal of this thesis is to study the use of surprise by ever-adapting biological agents, its contributions to reward-based learning, and its manifestation in the human brain. We first study surprise from a theoretical perspective. In a probabilistic model of changing environments, we show that exact and approximate Bayesian inference give rise to a trade-off between forgetting old observations and integrating them with new ones, modulated by a naturally emerging surprise measure. We develop novel surprised-based algorithms that can adapt in the face of abrupt changes and accurately estimate the model of the world, and that could potentially be implemented in the brain. Next, we focus on the contributions of surprise-based model estimation to reinforcement learning. We couple one of our adaptive algorithms as well as simpler non-adaptive methods with reinforcement learning agents and evaluate their performance on environments exhibiting different characteristics. Abrupt changes that directly affect the agent's policy call for surprise-based adaptation, in order to achieve higher performance. Often, however, the agent does not need to invest in maintaining an accurate model of the environment to obtain high reward levels. More specifically, in stochastic environments or in environments with distal changes, simpler methods, equipped with exploration capacity, perform equally well compared to more elaborate methods. Finally, we turn to human learning behaviour and brain signals of surprise- and reward-based learning. We design a novel sequential decision making task of multiple steps where strategic use of surprising events allows us to dissociate fMRI brain correlates of reward learning and model estimation. We show that Bayesian inference on this task leads to the same surprise measure we found earlier, where the trade-off is now between ignoring new observations and integrating them with the old belief, and we develop reinforcement learning algorithms that perform outlier detection via this surprise-modulated trade-off. At the level of behaviour we find evidence for a model-free policy learning architecture, with potential influences from a model estimation system. At the level of brain responses we identify signatures of both reward- and model estimation signals, supporting the existence of multiple parallel learning systems in the brain. This thesis presents a comparative analysis of surprise-based model estimation methods in theory and simulations, provides insights in the type of approximations that biological agents may adopt, and identifies signatures of model estimation in the human brain. Our results may aid future work aiming at building efficient adaptive agents and at understanding the learning algorithms and the surprise measures implemented in the brain.

EPFL2021

Optimal recovery of unsecured debt via interpretable reinforcement learning

Huanxi Liu, Michael Mark, Thomas Alois Weber

This paper addresses the issue of interpretability and auditability of reinforcement-learning agents employed in the recovery of unsecured consumer debt. To this end, we develop a deterministic policy-gradient method that allows for a natural integration of domain expertise into the learning procedure so as to encourage learning of consistent, and thus interpretable, policies. Domain knowledge can often be expressed in terms of policy monotonicity and/or convexity with respect to relevant state inputs. We augment the standard actor–critic policy approximator using a monotonically regularized loss function which integrates domain expertise into the learning. Our formulation overcomes the challenge of learning interpretable policies by constraining the search to policies satisfying structural-consistency properties. The resulting state-feedback control laws can be readily understood and implemented by human decision makers. This new domain-knowledge enhanced learning approach is applied to the problem of optimal debt recovery which features a controlled Hawkes process and an asynchronous action–feedback relationship.