Building a Simplified Haystack Store for Image Management

This lecture presents a project inspired by Facebook's 2010 Haystack system, focusing on efficient image storage and retrieval. The instructor outlines the challenges faced by Facebook in managing billions of photos and the limitations of traditional file systems. The lecture emphasizes the need for a specialized storage solution that optimizes performance and locality. Students will learn to build a simplified version of the Haystack store, which includes a super block, an index, and a data block for JPEG images. The project will involve creating a command-line interface and a simple HTTP server, allowing for image management and retrieval. Key skills covered include working with the POSIX filesystem, using libraries for image compression, and employing debugging tools. The instructor stresses the importance of collaboration and version control using Git, as students will work in pairs. The lecture concludes with an overview of project milestones and grading criteria, ensuring students understand the expectations and resources available for their success.