OpenStack Swift overview oscon2011

5,885 views

Published on

Published in: Technology, Business

OpenStack Swift overview oscon2011

  1. 1. Wednesday, July 27, 11
  2. 2. Swift Overview John Dickinson Swift Project Technical LeadWednesday, July 27, 11
  3. 3. What We’ll Cover ‣ Intro to Swift ‣ Architectural Overview ‣ Best Practices for Deploying ‣ Future workWednesday, July 27, 11
  4. 4. Swift is Scalable Storage ‣ Scales as the cluster grows ‣ Designed to run on commodity hardware ‣ No single point of failureWednesday, July 27, 11
  5. 5. Swift is not ‣ RAID ‣ Distributed Filesystem ‣ CDN ‣ SAN/NAS/DASWednesday, July 27, 11
  6. 6. Swift Use Cases ‣ Multi-tenant ‣ Backups ‣ Typically large ‣ Write-once, read-never ‣ Web Content ‣ Typically small ‣ Write-many, read-very-manyWednesday, July 27, 11
  7. 7. Swift Use Cases (cont.) ‣ Document management ‣ Medical imaging ‣ Disaster recovery ‣ Caching ‣ Storage appliances ‣ Scientific dataWednesday, July 27, 11
  8. 8. Example Usage ‣ Standard HTTP verbs ‣ Standard HTTP response codesWednesday, July 27, 11
  9. 9. Designing Swift ‣ At scale, everything breaks ‣ Design for failure ‣ The “Happy Path” is trivial ‣ At scale, even rare events happen a lot ‣ “Simple is better than complex.”Wednesday, July 27, 11
  10. 10. Swift History ‣ Rackspace Cloud Files (circa 2008) ‣ Distributed Storage ‣ Centralized MetadataWednesday, July 27, 11
  11. 11. Swift History (cont.) ‣ Rackspace Cloud Files (circa 2010) ‣ Redesigned from the ground up ‣ Distributed storage and metadata ‣ Worked closely with opsWednesday, July 27, 11
  12. 12. Modular Design of Swift ‣ Proxy Server ‣ The Ring ‣ Storage Servers ‣ Consistency ServersWednesday, July 27, 11
  13. 13. Proxy Server ‣ Public face of swift ‣ Determines appropriate storage nodes ‣ https://swift.example.com/v1/account/container/object ‣ Coordinates responsesWednesday, July 27, 11
  14. 14. The Ring ‣ Maps requests to storage nodes ‣ Availability zones ‣ Expandable without affecting most entitiesWednesday, July 27, 11
  15. 15. The Ring, in detail ‣ Started as a consistent-hashing ring ‣ Availability zones ‣ Became a static map of partitions to volumes ‣ What is a partition? ‣ What is a volume?Wednesday, July 27, 11
  16. 16. Storage Servers ‣ Account server ‣ Container server ‣ Object serverWednesday, July 27, 11
  17. 17. Storage Servers (Account and Container) ‣ SQLite databases ‣ Groups containers and objects into accounts ‣ Simple schema ‣ Table for listings ‣ Table for metadata ‣ Scale concerns?Wednesday, July 27, 11
  18. 18. Storage Servers (Object) ‣ Use the file system to store files ‣ Any file system that supports xattrs ‣ Files named by timestamp ‣ Tombstones ‣ Directory structure ‣ /mount/data_dir/partition/hash_suffix/hash/object.ts ‣ Directory metadata can easily exhaust RAMWednesday, July 27, 11
  19. 19. Consistency Servers ‣ Replicators ‣ Updaters ‣ AuditorsWednesday, July 27, 11
  20. 20. How to Deploy ‣ Swift All-In-One (SAIO) ‣ Small cluster ‣ Large clusterWednesday, July 27, 11
  21. 21. How to Deploy (SAIO) ‣ All services in one VM ‣ Use loopback devices for storage ‣ Great for dev work and validation testingWednesday, July 27, 11
  22. 22. How to Deploy (Small Cluster) ‣ Two load-balanced proxies ‣ Five storage nodes ‣ This area needs workWednesday, July 27, 11
  23. 23. How to Deploy (Large Cluster) ‣ Five zones ‣ Two proxies per zone ‣ Many storage nodes per zoneWednesday, July 27, 11
  24. 24. How to Deploy (Large Cluster, networking) ‣ 10g external network ‣ 1g (or 10g) internal network ‣ Use-case concerns ‣ Internal proxy network divided by replica countWednesday, July 27, 11
  25. 25. How to Deploy (Large Cluster, IO) ‣ Dedicated Container and Account servers ‣ Few, relative to object servers ‣ Many IOPS (i.e. SSDs) ‣ Cheap storage for Object serversWednesday, July 27, 11
  26. 26. Future Dev Work ‣ Better Ring ‣ Layered zones ‣ Unique-of-zones-as-possible ‣ Tiered storage ‣ More modular deployment ‣ Cabinets, not zones ‣ More featuresWednesday, July 27, 11
  27. 27. What Swift is “Missing” ‣ Auth ‣ ID management ‣ Stats ‣ Quotas ‣ Billing ‣ CDNWednesday, July 27, 11
  28. 28. Questions? ‣ #openstack on freenode ‣ http://launchpad.net/swift ‣ http://d.not.mn/swift_overview_oscon2011.pdfWednesday, July 27, 11
  29. 29. John Dickinson ‣ @notmyname ‣ http://gplus.to/notmyname ‣ me@not.mn ‣ http://not.mnWednesday, July 27, 11

×