This document summarizes the architecture of Prospero Media Storage, which manages 100TB of small files. It discusses the challenges of managing a large number of small files and connections. Prospero uses a distributed architecture with identical server nodes to allow linear scaling. Each node runs on commodity hardware and uses techniques like non-blocking IO, asynchronous file IO, and zero-copy operations to optimize performance. The system is designed to fail safely and guarantee delivery through techniques like fallback options and journaling.

1. Prospero Media Storage Managing 100TB of small files… IGT – Event July 2011

2. Numbers 70TBused space 700 million files 200GBand 250,000 files uploaded every day 1200Mbpsbandwidth throughput in peak 180TBof data is being served out monthly 3700 Hits per second in peak 40 storage node servers – 300TB raw space $0.13 per GB

3. Motivation Web 2.0 content serving paradigm shift Too many files 12M users x 1 file = very long tail Too many connections 1M users + keepalive = 1M connections Living with modern content in web 2.0 1 file x (thumbnail + iPhone + Mac) = 3 file copies

4. Traditional Architecture HTTP IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

5. Traditional Architecture HTTP – TOO MANY CONNECTIONS IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

6. Traditional Architecture HTTP IO IO IO IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

7. Traditional Architecture HTTP IO IO IO IO IO IO IO Too much IO

8. Traditional Architecture HTTP Cache IO IO IO IO IO IO IO Centralized Storage (NAS, SAN, DAS etc.)

9. “There are only two hard things in Computer Science: cache invalidation and naming things”. -- Tim Bray quoting Phil Karlton

10. Architecture goals Symmetric identical server nodes Simplified management and scaling Linear scaling out No functional / role servers No single point of failure No performance bottlenecks Multiple datacenters support DRP support Geo load distribution

11. Meet Prospero Distributed Web content storage system Full blown HTTP support Runs on low cost commodity hardware Adjustable file level replication controls redundancy policy for every content type Provides dynamic image manipulation

12. How do we do it?

13. Designed to fail Fallback for every operation Geographical, machine, storage medium Write never fails All files will reach their destination Journaling Tracking all uploaded files Pending jobs Guaranteed file distribution

14. How do we achieve this Control the input define the only unified API Functional process isolation every function deserves its own process by default watchdogs monitors alerts

15. get 37D815B5.jpg Go to 37 range servers Fallback if not found 2.static 6.static 0.static 4.static HTTP HTTP HTTP 20-3f 60-7f 00-1f 40-5f 7.static 3.static 1.static 5.static HTTP HTTP HTTP

16. Fallback Example

17. Node Architecture

18. Real Life

19. It’s all about performance Non blocking IO, readiness notification (epoll) Asynchronous file IO (AIO) Zero copy (sendfile) Memory maps Inter-process binary protocols UNIX socket Minimize dynamic memory allocation lighttpd memory footprint: 50MB

20. Lessons learnt Be symmetric Control the input Design to failure Performance matters again Simple is hard but a must