Apprentissage par renforcement profond

L'apprentissage par renforcement profond (en anglais : deep reinforcement learning ou deep RL) est un sous-domaine de l'apprentissage automatique (en anglais : machine learning) qui combine l'apprentissage par renforcement et l'apprentissage profond (en anglais : deep learning). L'apprentissage par renforcement considère le problème d'un agent informatique (par exemple, un robot, un agent conversationnel, un personnage dans un jeu vidéo, etc.) qui apprend à prendre des décisions par essais et erreurs. L'apprentissage par renforcement profond intègre l'apprentissage profond dans la résolution, permettant aux agents de prendre des décisions à partir de données d'entrée non structurées sans intervention manuelle sur l'espace des états. Les algorithmes d'apprentissage par renforcement profond sont capables de prendre en compte de très grandes quantités de données (par exemple, chaque pixel affiché à l'écran dans un jeu vidéo) et de décider des actions à effectuer pour optimiser un objectif

A prescriptive Dirichlet power allocation policy with deep reinforcement learning

Yuan Tian

Prescribing optimal operation based on the condition of the system, and thereby potentially prolonging its remaining useful lifetime, has tremendous potential in terms of actively managing the availability, maintenance, and costs of complex systems. Reinforcement learning (RL) algorithms are particularly suitable for this type of problem given their learning capabilities. A special case of a prescriptive operation is the power allocation task, which can be considered as a sequential allocation problem whereby the action space is bounded by a simplex constraint. A general continuous action-space solution of such sequential allocation problems has still remained an open research question for RL algorithms. In continuous action space, the standard Gaussian policy applied in reinforcement learning does not support simplex constraints, while the Gaussian-softmax policy introduces a bias during training. In this work, we propose the Dirichlet policy for continuous allocation tasks and analyze the bias and variance of its policy gradients. We demonstrate that the Dirichlet policy is bias-free and provides significantly faster convergence, better performance, and better robustness to hyperparameter changes as compared to the Gaussian-softmax policy. Moreover, we demonstrate the applicability of the proposed algorithm on a prescriptive operation case in which we propose the Dirichlet power allocation policy and evaluate its performance on a case study of a set of multiple lithium-ion (Li-I) battery systems. The experimental results demonstrate the potential to prescribe optimal operation, improving the efficiency and sustainability of multi-power source systems.

2022

Technology Strategy in Dynamic Environments: A Computational Analysis of the Automation of Routines, the Organization of AI, and the Evaluation of Technology Risk

In this dissertation, I develop theory and evidence to argue that new technologies are central to how firms organize to create and capture value. I use computational methods such as reinforcement learning and probabilistic topic modeling to investigate three topics: the automation of routines, the organization of artificial intelligence (AI), and the evaluation of technology risk. Overall, I argue that new technologies are not a panacea for the firm but require deliberate strategic planning to manage the potential downsides of myopic automation, AI interdependencies, and the disclosure of technology risks.In the first essay, I argue that while automation can increase productivity by reducing the costs of coordinating individuals, the automation of routines can also incur an indirect opportunity cost due to slow adaptation to environmental change. I develop a reinforcement learning simulation to model the impact of automation on the returns from the division of labor in dynamic environments and to show how automation incurs opportunity costs through lost learning and slow adaptation. Moreover, automation can be suboptimal when it brings about myopic behavior, i.e. high returns from the division of labor in the short term, but negative returns in the long term. Given the simulation results, I argue that firms need dynamic routines to simultaneously balance learning and automation. I open-source the simulation platform as OrgSim-RL on GitHub.In the second essay, I argue that a data-driven culture - what I define as a Data Clan - can help to coordinate complex interdependencies between AI components within a firm. I analyze in-depth semi-structured interview data with a hierarchical stochastic block model (hSBM) and hand-coding to find that managers focus primarily on building a strong culture and establishing high-quality data assets when allocating resources to AI initiatives. Given the results, I inductively develop implications for theory and argue that the emergence of a Data Clan can be a governance mechanism to reduce coordination frictions and build a competitive advantage in the age of AI.In the third essay, I argue that investors require a higher initial return to take on more technology risk disclosure during an IPO. I quantify the magnitude of disclosed risk and the risk disclosure topics based on a latent Dirichlet allocation (LDA) topic model of IPO prospectus text and find a return-for-risk association between text-based technology risk disclosure and underpricing. The study also finds evidence that owning granted patents is associated with a lower return-for-risk association, suggesting that intellectual property allows the disclosure of risk without losing the competitive advantage. I open-source the code for quantifying risk disclosure as RiskyData-LDA on GitHub.In summary, this dissertation develops theory and finds evidence across three essays to argue that leveraging new technologies requires deliberate strategic planning to manage potential downsides of new technologies, such as the opportunity costs of automation, coordination costs, and costs associated with raising capital. The results suggest three mitigating solutions: dynamic routines to balance learning and automation, a Data Clan to improve coordination, and disclosure through patents to reduce underpricing.

EPFL2022

Auto-tuning deep forest for shear stiffness prediction of headed stud connectors

Yuqing Liu

The shear stiffness of headed stud connector is a critical parameter for the calculation of deflection and inter-facial shear force for steel-concrete composite structure. Thus, this study presented a promising data-driven model auto-tuning Deep Forest (ATDF) to precisely predict the stud shear stiffness, where the novel Deep For-est algorithm is integrated with the Sequential Model-Based Optimization method to achieve automatic hyper -parameter optimization. Six variables having causal relationships with shear stiffness were extracted via mechanism and model analysis, including the effect of weld collar that cannot be considered in existing models and subsequently constituting a database of 425 push-out tests. Then the ATDF model was trained by combining the advantages of deep learning, ensemble learning, and auto-tuning techniques. It was approved to significantly outperform representative benchmark models with R values of 0.91 and 0.87 for training and testing sets. The ATDF was subjected to attribute importance analysis, which quantified the stud diameter and concrete elastic modulus as the most significant variables for shear stiffness, with the stud elastic modulus having the minimal effect. The model uncertainty of ATDF was further evaluated, revealing that it had the lowest bias and variability than those in existing empirical or semi-empirical models. Finally, the reliability analysis was conducted and the partial factors of ATDF under specified target reliability were derived.

2022

Technology Strategy in Dynamic Environments: A Computational Analysis of the Automation of Routines, the Organization of AI, and the Evaluation of Technology Risk

EPFL2022