This document summarizes the architecture of Prospero Media Storage, which manages 100TB of small files. It discusses the challenges of managing a large number of small files and connections. Prospero uses a distributed architecture with identical server nodes to allow linear scaling. Each node runs on commodity hardware and uses techniques like non-blocking IO, asynchronous file IO, and zero-copy operations to optimize performance. The system is designed to fail safely and guarantee delivery through techniques like fallback options and journaling.

1. Prospero Media StorageManaging 100TB of small files…IGT – EventJuly 2011

2. Numbers70TBused space700 million files200GBand 250,000 files uploaded every day1200Mbpsbandwidth throughput in peak180TBof data is being served out monthly3700 Hits per second in peak 40 storage node servers – 300TB raw space$0.13 per GB

3. MotivationWeb 2.0 content serving paradigm shiftToo many files12M users x 1 file = very long tailToo many connections1M users + keepalive = 1M connectionsLiving with modern content in web 2.01 file x (thumbnail + iPhone + Mac) = 3 file copies

4. Traditional ArchitectureHTTPIOIOIOIOCentralized Storage (NAS, SAN, DAS etc.)

5. Traditional ArchitectureHTTP – TOO MANY CONNECTIONSIOIOIOIOCentralized Storage (NAS, SAN, DAS etc.)

6. Traditional ArchitectureHTTPIOIOIOIOIOIOIOCentralized Storage (NAS, SAN, DAS etc.)

7. Traditional ArchitectureHTTPIOIOIOIOIOIOIOToo much IO

8. Traditional ArchitectureHTTPCacheIOIOIOIOIOIOIOCentralized Storage (NAS, SAN, DAS etc.)

9. “There are only two hard things in Computer Science: cache invalidation and naming things”. -- Tim Bray quoting Phil Karlton

10. Architecture goalsSymmetric identical server nodesSimplified management and scalingLinear scaling outNo functional / role serversNo single point of failureNo performance bottlenecksMultiple datacenters supportDRP supportGeo load distribution

11. Meet ProsperoDistributed Web content storage systemFull blown HTTP supportRuns on low cost commodity hardwareAdjustable file level replication controls redundancy policy for every content typeProvides dynamic image manipulation

12. How do we do it?

13. Designed to failFallback for every operationGeographical, machine, storage mediumWrite never failsAll files will reach their destinationJournalingTracking all uploaded filesPending jobs Guaranteed file distribution

14. How do we achieve thisControl the inputdefine the only unified API Functional process isolationevery function deserves its own process by defaultwatchdogsmonitorsalerts

15. get 37D815B5.jpgGo to 37 range serversFallback if not found2.static6.static0.static4.staticHTTPHTTPHTTP20-3f60-7f00-1f40-5f7.static3.static1.static5.staticHTTPHTTPHTTP

16. Fallback Example

17. Node Architecture

18. Real Life

19. It’s all about performanceNon blocking IO, readiness notification (epoll)Asynchronous file IO (AIO)Zero copy (sendfile)Memory mapsInter-process binary protocolsUNIX socketMinimize dynamic memory allocationlighttpd memory footprint: 50MB

20. Lessons learntBe symmetricControl the inputDesign to failurePerformance matters againSimple is hard but a must