Lac de données

Résumé

Un lac de données (en anglais data lake) est une méthode de stockage de données massives utilisée par le big data (mégadonnées en français). Ces données sont gardées dans leurs formats originaux ou sont très peu transformées. Le lac de données donne la priorité au stockage rapide et volumineux de données hétérogènes en adoptant une architecture en cluster. Il n'est pas optimisé pour les requêtes SQL comme les SGBD relationnels classiques, et s'écarte des Propriétés ACID traditionnelles. On parle depuis 2010 de SGBD NoSQL. Big Data On trouve donc dans un lac de données des données de natures et de sources différentes, telles que :

des données structurées issues notamment de bases de données relationnelles (lignes et colonnes) ;
des données variées issues de bases NoSQL ;
des données semi-structurées (fichiers CSV, journaux, XML, JSON...) ;
des données non structurées (emails, documents, PDF) ;
des fichiers de type blob (images, audio, vidéo notamment).

Ces données sont c

Source officielle

https://fr.wikipedia.org/wiki/Lac_de_donn%C3%A9es

À propos de ce résultat

Cette page est générée automatiquement et peut contenir des informations qui ne sont pas correctes, complètes, à jour ou pertinentes par rapport à votre recherche. Il en va de même pour toutes les autres pages de ce site. Veillez à vérifier les informations auprès des sources officielles de l'EPFL.

Publications associées (7)

Publications associées

Chargement

Personnes associées

Chargement

Unités associées

Chargement

Concepts associés (3)

Concepts associés

Chargement

Cours associés (7)

Cours associés

Chargement

Séances de cours associées (14)

Séances de cours associées

Chargement

Pixels: An Efficient Column Store for Cloud Data Lakes

Anastasia Ailamaki, Haoqiong Bian

To benefit from the cloud’s higher elasticity and price-efficiency, most modern data-lake engines support S3-like cloud object storage (COS) services as their optional or preferred underlying storage. Meanwhile, the widespread column stores, such as Parquet, are applied in these data lakes to improve analytical performance. However, as these column stores were designed for on-premise HDFS, they often suffer from the high latency of COS and deliver sub-optimal query performance. We observe that by optimizing the storage layout and data access pattern, we can effectively hide and mitigate the high latency. In this paper, we present Pixels, a column store optimized for the cloud that solves the problem by (1) the workload-driven storage layout optimization within and across the row group boundaries; (2) the I/O scheduling concerning the optimized storage layout and the performance characteristics of COS. They collectively improve the analytical performance in a transparent way that does not affect data ingestion and query execution in data lakes. Evaluations show that Pixels outperforms the state-of- the-art column store on COS by more than one order of magnitude on real-world workload and by 1.93x on TPC-H. Moreover, the performance of Pixels is also portable to HDFS.

IEEE2022

Chargement

High-dimensional Data Cubes

Sachin Basil John, Christoph Koch

This paper introduces an approach to supporting high-dimensional data cubes at interactive query speeds and moderate storage cost. The approach is based on binary(-domain) data cubes that are judiciously partially materialized; the missing information can be quickly reconstructed using statistical or linear programming techniques. This enables new applications such as exploratory data analysis for feature engineering and other fields of data science. Moreover, it allows us to use data cubes in data warehouses and data lakes without design regret. It removes the need to compromise when building a data cube — all columns that we might ever wish to use can be included as dimensions. As a collateral benefit, our approach also speeds up certain dice, roll-up, and drill-down operations on data cubes with hierarchical dimensions compared to traditional data cubes.

2022

Chargement

Industry and academia are continuously becoming more data-driven and data-intensive, relying on the analysis of a wide variety of datasets to gain insights. At the same time, data variety increases continuously across multiple axes. First, data comes in multiple formats, such as the binary tabular data of a DBMS, raw textual files, and domain-specific formats. Second, different datasets follow different data models, such as the relational and the hierarchical one. Data location also varies: Some datasets reside in a central "data lake", whereas others lie in remote data sources. In addition, users execute widely different analysis tasks over all these data types. Finally, the process of gathering and integrating diverse datasets introduces several inconsistencies and redundancies in the data, such as duplicate entries for the same real-world concept. In summary, heterogeneity significantly affects the way data analysis is performed. In this thesis, we aim for data virtualization: Abstracting data out of its original form and manipulating it regardless of the way it is stored or structured, without a performance penalty. To achieve data virtualization, we design and implement systems that i) mask heterogeneity through the use of heterogeneity-aware, high-level building blocks and ii) offer fast responses through on-demand adaptation techniques. Regarding the high-level building blocks, we use a query language and algebra to handle multiple collection types, such as relations and hierarchies, express transformations between these collection types, as well as express complex data cleaning tasks over them. In addition, we design a location-aware compiler and optimizer that masks away the complexity of accessing multiple remote data sources. Regarding on-demand adaptation, we present a design to produce a new system per query. The design uses customization mechanisms that trigger runtime code generation to mimic the system most appropriate to answer a query fast: Query operators are thus created based on the query workload and the underlying data models; the data access layer is created based on the underlying data formats. In addition, we exploit emerging hardware by customizing the system implementation based on the available heterogeneous processors â CPUs and GPGPUs. We thus pair each workload with its ideal processor type. The end result is a just-in-time database system that is specific to the query, data, workload, and hardware instance. This thesis redesigns the data management stack to natively cater for data heterogeneity and exploit hardware heterogeneity. Instead of centralizing all relevant datasets, converting them to a single representation, and loading them in a monolithic, static, suboptimal system, our design embraces heterogeneity. Overall, our design decouples the type of performed analysis from the original data layout; users can perform their analysis across data stores, data models, and data formats, but at the same time experience the performance offered by a custom system that has been built on demand to serve their specific use case.

EPFL2017

Chargement

EPFL2017