Stream processing | EPFL Graph Search

In computer science, stream processing (also known as event stream processing, data stream processing, or distributed stream processing) is a programming paradigm which views streams, or sequences of events in time, as the central input and output objects of computation. Stream processing encompasses dataflow programming, reactive programming, and distributed data processing. Stream processing systems aim to expose parallel processing for data streams and rely on streaming algorithms for efficient implementation. The software stack for these systems includes components such as programming models and query languages, for expressing computation; stream management systems, for distribution and scheduling; and hardware components for acceleration including floating-point units, graphics processing units, and field-programmable gate arrays. The stream processing paradigm simplifies parallel software and hardware by restricting the parallel computation that can be performed. Given a sequen

DSPE: Domain Specific Language for Parallel Real-Time Stream Processing

Tiziano Leidi

Today, the evolution of software solutions for parallel processing is strong, as a consequence of the mainstream introduction of chip-level multiprocessors. These types of parallel processors, which include general-purpose multi-cores and graphical processing units (GPUs), allow the execution of very intensive computations. However, depending on the nature of the executed applications, their effective exploitation may represent a complex task. In terms of parallel processing, the family of stream-processing applications is appealing because it features some forms of parallelism that may be exploited systematically. However, the processing performed by these types of applications is often real-time and, therefore, special-purpose support for parallel processing is required. The main contribution of this thesis is the DIGITAL STREAM-PROCESSING ENVIRONMENT (DSPE): an open-source development environment for rapid prototyping and customizing parallel real-time stream-processing applications. DSPE features a domain-specific language (DSL) based on model-driven generative programming that allows specifying applications at a high abstraction level. From the information provided with the DSL, generators are utilized to automatically produce the source code of applications. The source code generated by DSPE features dedicated infrastructures for parallel processing in stream-processing applications. These infrastructures are based on event-driven scheduling and on dynamic load balancing of lightweight tasks, which are techniques that limit the impact of load imbalances when parallel stream processing. Furthermore, DSPE supports the integration of kernels for GPUs for the exploitation of heterogeneous combinations of multi-core processors and GPUs. In DSPE, the generated source code can be configured to fit application- and hardware-specific needs. Furthermore, profile-guided, model-to-model transformations ease the migration from a serial version of an application to a parallel processing version. Experiments performed with DSPE on micro-benchmarks and sample applications demonstrate that the adopted approaches facilitate obtaining linear or near-to-linear scalings of the speed-up, when progressively incrementing the number of utilized processor cores.

EPFL2012

Cyclone: Unified Stream and Batch Processing

Matus Harvan, Thomas Locher, Ana Claudia Sima

Due to the rising demand for large-scale data processing, there is a growing interest in both batch processing, where large volumes of data are processed offline, and stream processing, where large quantities of streaming data are processed online. The dichotomy between these vastly different computing paradigms has led to the development of substantially different methodologies and systems. As there is an increasing number of applications requiring stream and batch processing, there is a need to bridge this gap and offer support for both paradigms. We introduce a new direction for the unification of stream and batch processing, which, contrary to other proposed approaches, uses a stream processing platform as its foundation and supports batch processing on top. Our proof-of-concept implementation of such a middleware layer, called Cyclone, offers the widely popular MapReduce programming model and translates MapReduce jobs for execution on the underlying streaming platform. Cyclone not only achieves a tight integration of batch and stream processing, our evaluation further shows significant performance gains, in particular for sequential and iterative jobs, which naturally arise in many applications.

Ieee2016

Computing effective properties of random heterogeneous materials on heterogeneous parallel processors

Tiziano Leidi, Alberto Ortona, Jean-Philippe Thiran

In recent decades, finite element (FE) techniques have been extensively used for predicting effective properties of random heterogeneous materials. In the case of very complex microstructures, the choice of numerical methods for the solution of this problem can offer some advantages over classical analytical approaches, and it allows the use of digital images obtained from real material samples (e.g., using computed tomography). On the other hand, having a large number of elements is often necessary for properly describing complex microstructures, ultimately leading to extremely time-consuming computations and high memory requirements. With the final objective of reducing these limitations, we improved an existing freely available FE code for the computation of effective conductivity (electrical and thermal) of microstructure digital models. To allow execution on hardware combining multi-core CPUs and a CPU, we first translated the original algorithm from Fortran to C, and we subdivided it into software components. Then, we enhanced the C version of the algorithm for parallel processing with heterogeneous processors. With the goal of maximizing the obtained performances and limiting resource consumption, we utilized a software architecture based on stream processing, event-driven scheduling, and dynamic load balancing. The parallel processing version of the algorithm has been validated using a simple microstructure consisting of a single sphere located at the centre of a cubic box, yielding consistent results. Finally, the code was used for the calculation of the effective thermal conductivity of a digital model of a real sample (a ceramic foam obtained using X-ray computed tomography). On a computer equipped with dual hexa-core Intel Xeon X5670 processors and an NVIDIA Tesla C2050, the parallel application version features near to linear speed-up progression when using only the CPU cores. It executes more than 20 times faster when additionally using the CPU. (C) 2012 Elsevier B.V. All rights reserved.