Are you an EPFL student looking for a semester project?
Work with us on data science and visualisation projects, and deploy your project as an app on top of Graph Search.
This lecture introduces Apache Spark, a unified analytics engine for large-scale data processing, highlighting its key features like interactive data exploration, in-memory data processing, and fault-tolerance. It covers the history of Spark, its usage in various applications, and its flexibility in deployment. The lecture explains Resilient Distributed Datasets (RDDs), the primary interface of Spark applications, and their importance in fault-tolerant and efficient iterative algorithms. It also delves into Spark's architecture, including the roles of the Driver and Worker nodes. Additionally, it explores RDD operations, transformations, actions, caching, and partitioning, providing insights into Spark's distributed computing framework.