Detecting Latent Training Needs Using Large Datasets

In today's world, there is no shortage of disruptors acting on various professional domains. The Fourth Industrial Revolution, with its AI-driven and automation-focused technologies, has fundamentally changed many domains -- particularly the Information and Communication Technologies (ICT) domain -- and continues to do so. The COVID-19 pandemic and its ensuing lockdowns have resulted in dramatic change in the workplace. These disruptors are fast-acting and rapidly change the skills landscape of many professional domains, introducing new skills and rendering old ones obsolete. In such a situation, it is imperative for educational institutions to keep their curricula updated in order to (re)train the workforce with the right set of skills for the new economy. However, most of the methodologies commonly used for analyzing training needs and effecting curricular change were developed during a time when the pace of change was slower, and thus have trouble quickly and effectively responding to today's rapid changes. As a result, the development of new approaches for the analysis of training needs is of utmost importance.The present dissertation is the result of an effort to develop such new approaches, using different types of data and approaching the problem from different perspectives. These approaches use big data methods on data sources that are large, pre-existing, and continually updated in order to identify the skill needs of the present and/or predict the skill needs of the future for various professions. Focusing on the ICT domain but also making forays into vocational domains, the present work examines the usefulness and feasibility of methodologies such as the estimation of topical difficulty, the prediction of the appearance of new skills, and the prediction of emerging skills (less popular skills expected to experience a surge in hiring demand) for the identification of training needs. The findings of this dissertation are twofold. Firstly, they demonstrate the pivotal role that decentralized educational platforms such as Stack Overflow (an online Q&A platform for ICT) and Udemy (a MOOC platform where anyone can create a course) play in lifelong learning, essentially making the case for a policy of sponsorship and promotion of such platforms in other professional domains. Secondly and more importantly, the results showcase the feasibility of predicting the emerging skills of the near future. The successful emerging skills methodology can therefore be used to provide valuable insights to training providers at a moment's notice, giving them time to prepare for the skills landscape of the future in advance. Thus, AI can be used to help solve the challenges it creates for educational institutions and the workforce. Future work, both in the form of policymaking and in the form of research, can help capitalize and expand on the results of this dissertation, furthering the cause of rapid curricular change.