Your SlideShare is downloading. ×

A Storage Story #ChefConf2013


Published on

Published in: Technology
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Kyle Storage Story
  • 2. About MeDad, husband, technologist.Sr. Systems Engineer@DreamHostfree software linux internals storagenetworking security monitoringdistributed systems automationDreamHost
  • 3. OutlineDreamHost Storage HistoryAnatomy of CephAutomating StorageDreamHost
  • 4. Destro- DreamHosts first web server- Pentium 100- SCSI storage- Shared T1 lineDreamHost
  • 5. DH2DreamHost
  • 6. NetAppNetApp Fabric Attached Storage- 15k Fiber channel drives- Filer heads serve NFS- Fast failover- Large failure domains- Expensive- Low densityDreamHost
  • 7. CoraidCoraid SAN- Shelves carry SATA devices, provide AoE volumes- Head units mount AoE volumes, XFS, NFS shares- Linux!- Fast failover- Large failure domains (single L2 segment)DreamHost
  • 8. ThumperSun Sunfire X4500- 45 Drives in a 4U chassis- Legendary hardware- Fast failover- High density- Large failure domains- SATA- HeavyDreamHost
  • 9. DreamHost Solution: Hybrid
  • 10. BlueArc "Titanic"- Switched fiber channel- Head units serve NFS- Tiered storage, FC/SATA- Fast failover- Larger failure domain (than NetApp)- Software bugs :(- Tiering: find -atimeDreamHost BlueArc
  • 11. DreamHost UNLIMITED
  • 12. DreamHost Thoughting..
  • 13. Mixed Strategy- Separate email and web storage- Email IO is heavy random, lots of small files- Web storage needs to be dense- FC NAS for email- SATA RAID for web storage- SATA ZRAID for backupsDreamHost
  • 14. Local RAIDLocal RAID- RAID6, RAID10, RAID6- SATA, SAS disks- ext3, XFS- Shrink failure domain- Great density- Slower failover- RAID Controllers..DreamHost
  • 15. SighDreamHost
  • 16. CephCeph- Open source- Build with COTS hardware- Distributed and replicated- No single point of failure- Consist- Self healing and self managingDreamHost
  • 17. Building BlocksMonitors:- Maintain cluster map- Provide consensus for distributed decision making- Must have an odd number- These do not serve stored objects to clientsOSDs:- One per disk (recommended)- Serve stored objects to clients- Intelligently peer to perform replication tasks- Supports object classesDreamHost
  • 18. Building BlocksOSD StatesUp available and readyDown not availableIn current member of clusterOut not member of clusterDreamHost
  • 19. CephstoreDreamHostXFSBTRFSEXT4
  • 20. ClusterDreamHost
  • 21. Ceph ConsumersDreamHost
  • 22. Creating a MapDreamHost
  • 23. CRUSHDreamHostCRUSH- Pseudo-random placement algorithm- Ensures statistically even distribution- Repeatable, deterministic- Rule based configuration- Replica count- Infrastructure Topology- Weighting
  • 24. CRUSHDreamHost
  • 25. OSD DOWN!DreamHost
  • 26. Remap and BackfillDreamHost
  • 27. Ceph AnatomyDreamHost
  • 28. RESTful Storage ServiceDreamHost
  • 29. DreamObjectsDreamObjects:- Ceph Storage Cluster- Ubuntu Linux (12.04)- Managed by Opscode Chef- S3 and Swift RESTful interfaces- Highly durable (8 nines)- 2+ PB raw capacityDreamHost
  • 30. DreamComputeDreamCompute:- Ceph Storage Cluster (RDB)- Ubuntu Linux (12.04)- Managed by Opscode Chef- OpenStack- Virtualized L2 and L3 networking- Highly durable (8 nines)- 3+ PB raw capacityDreamHost
  • 31. AutomateDreamHost- Bootstrap cluster- Packages and configuration- Creates, Destroys and Encrypts OSDS- Roles map to pdsh genders- User and SSH key management- Push monitoring configurations
  • 32. Hard StuffDreamHostKey managementLeader election
  • 33. What we useDreamHost- Attributes- Environments- Search- No databags
  • 34. EnvironmentsDreamHost- Ceph package versions- VIPs for API endpoints- Package repository URI- Ceph configuration data driven by attributes
  • 35. Gated EnvironmentsDreamHostDevelopmentStagingProduction
  • 36. Operational Feedback- Continuous functional testing- Metrics, metrics, metrics- DashboardsDreamHost
  • 37. Chef Infrastructure- Chef cluster per datacenter- Private Chef from Opscode- Erchef is awesome- Migrating legacy automationDreamHost
  • 38. Chef for Ceph- Prototyped Ceph cluster automation with Chef- Adapted Ceph to ease configuration- Pushed some automation down into Ceph- Move towards being CM agnostic- Simplify Chef recipesDreamHost
  • 39. Chef the network- DreamCompute utilizes ODM switches- Cumulus Networks provides Linux based OS- Custom Chef omnibus builds for PPC- Ohai networking!DreamHost
  • 40. Resiliency EngineeringDreamHost- Amazon (GameDay)- Etsy (GameDay)- Google (DiRT)
  • 41. Infra. a CodeDreamHost- Bare metal servers configured by code- Network devices configured by code- Block storage configured by code- Virtual networking configured by code
  • 42. ThanksDreamHostDreamHostSage Weil and InktankOpenStack DevelopersOpscode
  • 43. DreamObjectsDreamHostQuestions?