Provenance, workflows, and crystallographic tools in materials science: AiiDA, spglib, and seekpath

The near-exponential expansion in computing resources over the last few decades has enabled a rapid increase in the capabilities of computational science, including applications to materials research. In order to harness the available resources and accelerate the field of materials design, it is critically important to develop robust and reusable automation software for preparing and performing multistep computational workflows, starting with crystal structures and ending with material properties. In the domain of first-principles calculations of crystalline materials, we highlight emerging tools for automated symmetry analysis of the atomic and electronic structure. With automation capabilities in hand, the ever-increasing amount of data also becomes a serious bottleneck in terms of organization, analysis, and reproducibility. We describe some of the progress and strategic challenges in the development of a general infrastructure for coupling computational automation with data management, emphasizing data reproducibility and provenance capture.

Thermal transport in low dimensions

Andrea Cepellotti

Lattice vibrations are the microscopic mechanism responsible for a large, if not dominant, contribution to heat transport in crystalline insulators. These vibrations are described in terms of phonons, collective excitations (or quasiparticles) in the form of waves of atomic displacements inside a crystal. Phonons are traditionally considered to be the quasiparticles responsible for carrying heat through the material. Heat transport is considered as a flux of a phonon gas, diffusing from hot areas (high phonon densities) to cold areas (low phonon densities) in an attempt to reestablish equilibrium, with phonon collisions being the source of heat flux dissipation. However, as dimensionality is reduced, the motion of phonons stimulated by temperature perturbations becomes correlated and this gas-like picture of thermal transport in terms of phonons becomes invalid. In this Thesis, we lay out an interpretation of thermal transport in 2D materials based on the Boltzmann transport equation in the form of collective excitations of phonons. These collective phonon excitations give raise to complex phenomena, such as high thermal conductivities, that are otherwise unexplained. As another example, collective excitations, at variance with conventional diffusive transport, can induce wave-like heat propagation, or second sound. This had been found only in a few exotic materials at cryogenic temperatures, but is present instead routinely in 2D materials at room temperature. The correlated-phonon description of heat transfer can be rationalised by introducing a new collective excitation, called 'relaxon', which is defined as the eigenstate of the collision operator. Whereas only oversimplifying assumptions endow phonons with well defined relaxation times (the average interval of time between collisions), relaxons have always well defined relaxation times and permit an exact description of thermal transport. The complex dynamic of heat transport in 2D is thus greatly simplified and a kinetic gas theory of thermal transport still applies, provided that the gas is not constituted by phonons, but by relaxons. Our work on thermal transport is part of a larger effort, aiming at the creation of a database of numerically computed properties of materials. The high-throughput production of simulated properties is a challenging task, since it necessitates the understanding of a physical model, but it also needs to face a myriad of technicalities and problems that hinder the execution of a large number of calculations. In order to allow the creation of computational materials databases, we developed AiiDA, an open-source automated interactive infrastructure and database for computational science. This platform tackles the problems of creation, management, analysis and sharing of data and simulations, summarized in the pillars of Automation, Data, Environment and Sharing. Automation is achieved by management of remote computational resources and the encoding of workflows, that allows the execution of complex sequences of calculations. The tight coupling between automation and data storage, handled by the platform, enables full reproducibility of the results and a suitable database design allows for efficient data analysis tools. Sharing of scientific knowledge is addressed by providing tools for distribution of data and of the underlying workflows that generated them, creating an ecosystem for computational materials science.

EPFL2016

Just-in-time Analytics Over Heterogeneous Data and Hardware

Manolis Karpathiotakis

Industry and academia are continuously becoming more data-driven and data-intensive, relying on the analysis of a wide variety of datasets to gain insights. At the same time, data variety increases continuously across multiple axes. First, data comes in multiple formats, such as the binary tabular data of a DBMS, raw textual files, and domain-specific formats. Second, different datasets follow different data models, such as the relational and the hierarchical one. Data location also varies: Some datasets reside in a central "data lake", whereas others lie in remote data sources. In addition, users execute widely different analysis tasks over all these data types. Finally, the process of gathering and integrating diverse datasets introduces several inconsistencies and redundancies in the data, such as duplicate entries for the same real-world concept. In summary, heterogeneity significantly affects the way data analysis is performed. In this thesis, we aim for data virtualization: Abstracting data out of its original form and manipulating it regardless of the way it is stored or structured, without a performance penalty. To achieve data virtualization, we design and implement systems that i) mask heterogeneity through the use of heterogeneity-aware, high-level building blocks and ii) offer fast responses through on-demand adaptation techniques. Regarding the high-level building blocks, we use a query language and algebra to handle multiple collection types, such as relations and hierarchies, express transformations between these collection types, as well as express complex data cleaning tasks over them. In addition, we design a location-aware compiler and optimizer that masks away the complexity of accessing multiple remote data sources. Regarding on-demand adaptation, we present a design to produce a new system per query. The design uses customization mechanisms that trigger runtime code generation to mimic the system most appropriate to answer a query fast: Query operators are thus created based on the query workload and the underlying data models; the data access layer is created based on the underlying data formats. In addition, we exploit emerging hardware by customizing the system implementation based on the available heterogeneous processors â CPUs and GPGPUs. We thus pair each workload with its ideal processor type. The end result is a just-in-time database system that is specific to the query, data, workload, and hardware instance. This thesis redesigns the data management stack to natively cater for data heterogeneity and exploit hardware heterogeneity. Instead of centralizing all relevant datasets, converting them to a single representation, and loading them in a monolithic, static, suboptimal system, our design embraces heterogeneity. Overall, our design decouples the type of performed analysis from the original data layout; users can perform their analysis across data stores, data models, and data formats, but at the same time experience the performance offered by a custom system that has been built on demand to serve their specific use case.

EPFL2017

AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance

Casper Welzel Andersen, Andrea Cepellotti, Fernando Gargiulo, Sebastiaan Philippe Huber, Conrad Johnston, Leonid Kahle, Boris Kozinsky, Snehal Pramod Kumbhar, Nicola Marzari, Andrius Merkys, Nicolas Frank Mounet, Elsa Passaro, Giovanni Pizzi, Christoph Leopold Talirz, Martin Uhrin, Aliaksandr Yakutovich, Spyros Zoupanos

The ever-growing availability of computing power and the sustained development of advanced computational methods have contributed much to recent scientific progress. These developments present new challenges driven by the sheer amount of calculations and data to manage. Next-generation exascale supercomputers will harden these challenges, such that automated and scalable solutions become crucial. In recent years, we have been developing AiiDA (aiida.net), a robust open-source high-throughput infrastructure addressing the challenges arising from the needs of automated workflow management and data provenance recording. Here, we introduce developments and capabilities required to reach sustained performance, with AiiDA supporting throughputs of tens of thousands processes/hour, while automatically preserving and storing the full data provenance in a relational database making it queryable and traversable, thus enabling high-performance data analytics. AiiDA's workflow language provides advanced automation, error handling features and a flexible plugin model to allow interfacing with external simulation software. The associated plugin registry enables seamless sharing of extensions, empowering a vibrant user community dedicated to making simulations more robust, user-friendly and reproducible.