Application of large-scale computing infrastructure for diverse environmental research applications using GC3Pie

The Swiss Experiment platform (SwissEx: http://www.swiss-experiment.ch) provides a distributed storage and processing infrastructure for environmental research experiments. The aim of the second phase project (the Open Support Platform for Environmental Research, OSPER, 2012-2015) is to develop the existing infrastructure to provide scientists with an improved workflow. This improved workflow will include pre-defined, documented and connected processing routines. A large-scale computing and data facility is required to provide reliable and scalable access to data for analysis, and it is desirable that such an infrastructure should be free of traditional data handling methods. Such an infrastructure has been developed using the cloud-based part of the Swiss national infrastructure SMSCG (http://www.smscg.ch) and Academic Cloud. The infrastructure under construction supports two main usage models: 1) Ad-hoc data analysis scripts: These scripts are simple processing scripts, written by the environmental researchers themselves, which can be applied to large data sets via the high power infrastructure. Examples of this type of script are spatial statistical analysis scripts (R-based scripts), mostly computed on raw meteorological and/or soil moisture data. These provide processed output in the form of a grid, a plot, or a kml. 2) Complex models: A more intense data analysis pipeline centered (initially) around the physical process model, Alpine3D, and the MeteoIO plugin; depending on the data set, this may require a tightly coupled infrastructure. SMSCG already supports Alpine3D executions as both regular grid jobs and as virtual software appliances. A dedicated appliance with the Alpine3D specific libraries has been created and made available through the SMSCG infrastructure. The analysis pipelines are activated and supervised by simple control scripts that, depending on the data fetched from the meteorological stations, launch new instances of the Alpine3D appliance, execute location-based subroutines at each grid point and store the results back into the central repository for post-processing. An optional extension of this infrastructure will be to provide a 'ring buffer'-type database infrastructure, such that model results (e.g. test runs made to check parameter dependency or for development) can be visualised and downloaded after completion without submitting them to a permanent storage infrastructure. Data organization Data collected from sensors are archived and classified in distributed sites connected with an open-source software middleware, GSN. Publicly available data are available through common web services and via a cloud storage server (based on Swift). Collocation of the data and processing in the cloud would eventually eliminate data transfer requirements. Execution control logic Execution of the data analysis pipelines (for both the R-based analysis and the Alpine3D simulations) has been implemented using the GC3Pie framework developed by UZH. (https://code.google.com/p/gc3pie/). This allows large-scale, fault-tolerant execution of the pipelines to be described in terms of software appliances. GC3Pie also allows supervision of the execution of large campaigns of appliances as a single simulation. This poster will present the fundamental architectural components of the data analysis pipelines together with initial experimental results.

Mathias Thierry Pierre Bavay, Michael Lehning, Sofiane Sarni

The Swiss Experiment platform (SwissEx: http://www.swiss-experiment.ch) provides a distributed storage and processing infrastructure for environmental research experiments. The aim of the second phase project (the Open Support Platform for Environmental Research, OSPER, 2012-2015) is to develop the existing infrastructure to provide scientists with an improved workflow. This improved workflow will include pre-defined, documented and connected processing routines. A large-scale computing and data facility is required to provide reliable and scalable access to data for analysis, and it is desirable that such an infrastructure should be free of traditional data handling methods. Such an infrastructure has been developed using the cloud-based part of the Swiss national infrastructure SMSCG (http://www.smscg.ch) and Academic Cloud. The infrastructure under construction supports two main usage models: 1) Ad-hoc data analysis scripts: These scripts are simple processing scripts, written by the environmental researchers themselves, which can be applied to large data sets via the high power infrastructure. Examples of this type of script are spatial statistical analysis scripts (R-based scripts), mostly computed on raw meteorological and/or soil moisture data. These provide processed output in the form of a grid, a plot, or a kml. 2) Complex models: A more intense data analysis pipeline centered (initially) around the physical process model, Alpine3D, and the MeteoIO plugin; depending on the data set, this may require a tightly coupled infrastructure. SMSCG already supports Alpine3D executions as both regular grid jobs and as virtual software appliances. A dedicated appliance with the Alpine3D specific libraries has been created and made available through the SMSCG infrastructure. The analysis pipelines are activated and supervised by simple control scripts that, depending on the data fetched from the meteorological stations, launch new instances of the Alpine3D appliance, execute locationbased subroutines at each grid point and store the results back into the central repository for post-processing. An optional extension of this infrastructure will be to provide a ’ring buffer’-type database infrastructure, such that model results (e.g. test runs made to check parameter dependency or for development) can be visualised and downloaded after completion without submitting them to a permanent storage infrastructure. Data organization Data collected from sensors are archived and classified in distributed sites connected with an open-source software middleware, GSN. Publicly available data are available through common web services and via a cloud storage server (based on Swift). Collocation of the data and processing in the cloud would eventually eliminate data transfer requirements. Execution control logic Execution of the data analysis pipelines (for both the R-based analysis and the Alpine3D simulations) has been implemented using the GC3Pie framework developed by UZH. (https://code.google.com/p/gc3pie/). This allows large-scale, fault-tolerant execution of the pipelines to be described in terms of software appliances. GC3Pie also allows supervision of the execution of large campaigns of appliances as a single simulation. This poster will present the fundamental architectural components of the data analysis pipelines together with initial experimental results.

2013

A self-organized, fault-tolerant and scalable replication scheme for cloud storage

Karl Aberer, Nicolas Bonvin

Failures of any type are common in current datacenters, partly due to the higher scales of the data stored. As data scales up, its availability becomes more complex, while different availability levels per application or per data item may be required. In this paper, we propose a self- managed key-value store that dynamically allocates the resources of a data cloud to several applications in a cost- efficient and fair way. Our approach offers and dynamically maintains multiple differentiated availability guarantees to each different application despite failures. We employ a virtual economy, where each data partition (i.e. a key range in a consistent-hashing space) acts as an individual optimizer and chooses whether to migrate, replicate or remove itself based on net benefit maximization regarding the utility offered by the partition and its storage and maintenance cost. As proved by a game-theoretical model, no migrations or replications occur in the system at equilibrium, which is soon reached when the query load and the used storage are stable. Moreover, by means of extensive simulation experiments, we have proved that our approach dynamically finds the optimal resource allocation that balances the query processing overhead and satisfies the availability objectives in a cost-efficient way for different query rates and storage requirements. Finally, we have implemented a fully working prototype of our approach that clearly demonstrates its applicability in real settings.

ACM2010

Exploring the coupled ocean and atmosphere system with a data science approach applied to observations from the Antarctic Circumnavigation Expedition

Andrea Baccarini, Irina Gorodetskaya, Christel Sybille Hassler, Sebastian Johannes Heinz-Josef Landwehr, Katherine Colby Leonard, Robin Lewis Modini, Fernando Perez Cruz, Julia Schmale, Jenny Thomas

The Southern Ocean is a critical component of Earth's climate system, but its remoteness makes it challenging to develop a holistic understanding of its processes from the small scale to the large scale. As a result, our knowledge of this vast region remains largely incomplete. The Antarctic Circumnavigation Expedition (ACE, austral summer 2016/2017) surveyed a large number of variables describing the state of the ocean and the atmosphere, the freshwater cycle, atmospheric chemistry, and ocean biogeochemistry and microbiology. This circumpolar cruise included visits to 12 remote islands, the marginal ice zone, and the Antarctic coast. Here, we use 111 of the observed variables to study the latitudinal gradients, seasonality, shorter-term variations, geographic setting of environmental processes, and interactions between them over the duration of 90 d. To reduce the dimensionality and complexity of the dataset and make the relations between variables interpretable we applied an unsupervised machine learning method, the sparse principal component analysis (sPCA), which describes environmental processes through 14 latent variables. To derive a robust statistical perspective on these processes and to estimate the uncertainty in the sPCA decomposition, we have developed a bootstrap approach. Our results provide a proof of concept that sPCA with uncertainty analysis is able to identify temporal patterns from diurnal to seasonal cycles, as well as geographical gradients and “hotspots” of interaction between environmental compartments. While confirming many well known processes, our analysis provides novel insights into the Southern Ocean water cycle (freshwater fluxes), trace gases (interplay between seasonality, sources, and sinks), and microbial communities (nutrient limitation and island mass effects at the largest scale ever reported). More specifically, we identify the important role of the oceanic circulations, frontal zones, and islands in shaping the nutrient availability that controls biological community composition and productivity; the fact that sea ice controls sea water salinity, dampens the wave field, and is associated with increased phytoplankton growth and net community productivity possibly due to iron fertilisation and reduced light limitation; and the clear regional patterns of aerosol characteristics that have emerged, stressing the role of the sea state, atmospheric chemical processing, and source processes near hotspots for the availability of cloud condensation nuclei and hence cloud formation. A set of key variables and their combinations, such as the difference between the air and sea surface temperature, atmospheric pressure, sea surface height, geostrophic currents, upper-ocean layer light intensity, surface wind speed and relative humidity played an important role in our analysis, highlighting the necessity for Earth system models to represent them adequately. In conclusion, our study highlights the use of sPCA to identify key ocean–atmosphere interactions across physical, chemical, and biological processes and their associated spatio-temporal scales. It thereby fills an important gap between simple correlation analyses and complex Earth system models. The sPCA processing code is available as open-access from the following link: https://renkulab.io/gitlab/ACE-ASAID/spca-decomposition (last access: 29 March 2021). As we show here, it can be used for an exploration of environmental data that is less prone to cognitive biases (and confirmation biases in particular) compared to traditional regression analysis that might be affected by the underlying research question.

2021

Exploring the coupled ocean and atmosphere system with a data science approach applied to observations from the Antarctic Circumnavigation Expedition

Andrea Baccarini, Irina Gorodetskaya, Christel Sybille Hassler, Sebastian Johannes Heinz-Josef Landwehr, Katherine Colby Leonard, Robin Lewis Modini, Fernando Perez Cruz, Julia Schmale, Jenny Thomas

2021

Application of large-scale computing infrastructure for diverse environmental research applications using GC3Pie

Mathias Thierry Pierre Bavay, Michael Lehning, Sofiane Sarni

The Swiss Experiment platform (SwissEx: http://www.swiss-experiment.ch) provides a distributed storage and processing infrastructure for environmental research experiments. The aim of the second phase project (the Open Support Platform for Environmental Research, OSPER, 2012-2015) is to develop the existing infrastructure to provide scientists with an improved workflow. This improved workflow will include pre-defined, documented and connected processing routines. A large-scale computing and data facility is required to provide reliable and scalable access to data for analysis, and it is desirable that such an infrastructure should be free of traditional data handling methods. Such an infrastructure has been developed using the cloud-based part of the Swiss national infrastructure SMSCG (http://www.smscg.ch) and Academic Cloud. The infrastructure under construction supports two main usage models: 1) Ad-hoc data analysis scripts: These scripts are simple processing scripts, written by the environmental researchers themselves, which can be applied to large data sets via the high power infrastructure. Examples of this type of script are spatial statistical analysis scripts (R-based scripts), mostly computed on raw meteorological and/or soil moisture data. These provide processed output in the form of a grid, a plot, or a kml. 2) Complex models: A more intense data analysis pipeline centered (initially) around the physical process model, Alpine3D, and the MeteoIO plugin; depending on the data set, this may require a tightly coupled infrastructure. SMSCG already supports Alpine3D executions as both regular grid jobs and as virtual software appliances. A dedicated appliance with the Alpine3D specific libraries has been created and made available through the SMSCG infrastructure. The analysis pipelines are activated and supervised by simple control scripts that, depending on the data fetched from the meteorological stations, launch new instances of the Alpine3D appliance, execute locationbased subroutines at each grid point and store the results back into the central repository for post-processing. An optional extension of this infrastructure will be to provide a ’ring buffer’-type database infrastructure, such that model results (e.g. test runs made to check parameter dependency or for development) can be visualised and downloaded after completion without submitting them to a permanent storage infrastructure. Data organization Data collected from sensors are archived and classified in distributed sites connected with an open-source software middleware, GSN. Publicly available data are available through common web services and via a cloud storage server (based on Swift). Collocation of the data and processing in the cloud would eventually eliminate data transfer requirements. Execution control logic Execution of the data analysis pipelines (for both the R-based analysis and the Alpine3D simulations) has been implemented using the GC3Pie framework developed by UZH. (https://code.google.com/p/gc3pie/). This allows large-scale, fault-tolerant execution of the pipelines to be described in terms of software appliances. GC3Pie also allows supervision of the execution of large campaigns of appliances as a single simulation. This poster will present the fundamental architectural components of the data analysis pipelines together with initial experimental results.

2013