Are you an EPFL student looking for a semester project?
Work with us on data science and visualisation projects, and deploy your project as an app on top of Graph Search.
This lecture introduces the Spark runtime architecture, covering the history of Spark, its key features, flexibility, and basic data abstractions like Resilient Distributed Datasets (RDDs). It explains the Spark architecture overview, the roles of Driver and Worker, RDD operations, transformations, actions, caching, and partitioning. The lecture also delves into Spark's deployment flexibility, supported languages, and specialized libraries. Practical aspects such as initializing Spark, creating and transforming RDDs, and caching for performance optimization are discussed.