Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Storage Story #ChefConf2013


Published on

Published in: Technology

A Storage Story #ChefConf2013

  1. 1. Kyle Storage Story
  2. 2. About MeDad, husband, technologist.Sr. Systems Engineer@DreamHostfree software linux internals storagenetworking security monitoringdistributed systems automationDreamHost
  3. 3. OutlineDreamHost Storage HistoryAnatomy of CephAutomating StorageDreamHost
  4. 4. Destro- DreamHosts first web server- Pentium 100- SCSI storage- Shared T1 lineDreamHost
  5. 5. DH2DreamHost
  6. 6. NetAppNetApp Fabric Attached Storage- 15k Fiber channel drives- Filer heads serve NFS- Fast failover- Large failure domains- Expensive- Low densityDreamHost
  7. 7. CoraidCoraid SAN- Shelves carry SATA devices, provide AoE volumes- Head units mount AoE volumes, XFS, NFS shares- Linux!- Fast failover- Large failure domains (single L2 segment)DreamHost
  8. 8. ThumperSun Sunfire X4500- 45 Drives in a 4U chassis- Legendary hardware- Fast failover- High density- Large failure domains- SATA- HeavyDreamHost
  9. 9. DreamHost Solution: Hybrid
  10. 10. BlueArc "Titanic"- Switched fiber channel- Head units serve NFS- Tiered storage, FC/SATA- Fast failover- Larger failure domain (than NetApp)- Software bugs :(- Tiering: find -atimeDreamHost BlueArc
  11. 11. DreamHost UNLIMITED
  12. 12. DreamHost Thoughting..
  13. 13. Mixed Strategy- Separate email and web storage- Email IO is heavy random, lots of small files- Web storage needs to be dense- FC NAS for email- SATA RAID for web storage- SATA ZRAID for backupsDreamHost
  14. 14. Local RAIDLocal RAID- RAID6, RAID10, RAID6- SATA, SAS disks- ext3, XFS- Shrink failure domain- Great density- Slower failover- RAID Controllers..DreamHost
  15. 15. SighDreamHost
  16. 16. CephCeph- Open source- Build with COTS hardware- Distributed and replicated- No single point of failure- Consist- Self healing and self managingDreamHost
  17. 17. Building BlocksMonitors:- Maintain cluster map- Provide consensus for distributed decision making- Must have an odd number- These do not serve stored objects to clientsOSDs:- One per disk (recommended)- Serve stored objects to clients- Intelligently peer to perform replication tasks- Supports object classesDreamHost
  18. 18. Building BlocksOSD StatesUp available and readyDown not availableIn current member of clusterOut not member of clusterDreamHost
  19. 19. CephstoreDreamHostXFSBTRFSEXT4
  20. 20. ClusterDreamHost
  21. 21. Ceph ConsumersDreamHost
  22. 22. Creating a MapDreamHost
  23. 23. CRUSHDreamHostCRUSH- Pseudo-random placement algorithm- Ensures statistically even distribution- Repeatable, deterministic- Rule based configuration- Replica count- Infrastructure Topology- Weighting
  24. 24. CRUSHDreamHost
  25. 25. OSD DOWN!DreamHost
  26. 26. Remap and BackfillDreamHost
  27. 27. Ceph AnatomyDreamHost
  28. 28. RESTful Storage ServiceDreamHost
  29. 29. DreamObjectsDreamObjects:- Ceph Storage Cluster- Ubuntu Linux (12.04)- Managed by Opscode Chef- S3 and Swift RESTful interfaces- Highly durable (8 nines)- 2+ PB raw capacityDreamHost
  30. 30. DreamComputeDreamCompute:- Ceph Storage Cluster (RDB)- Ubuntu Linux (12.04)- Managed by Opscode Chef- OpenStack- Virtualized L2 and L3 networking- Highly durable (8 nines)- 3+ PB raw capacityDreamHost
  31. 31. AutomateDreamHost- Bootstrap cluster- Packages and configuration- Creates, Destroys and Encrypts OSDS- Roles map to pdsh genders- User and SSH key management- Push monitoring configurations
  32. 32. Hard StuffDreamHostKey managementLeader election
  33. 33. What we useDreamHost- Attributes- Environments- Search- No databags
  34. 34. EnvironmentsDreamHost- Ceph package versions- VIPs for API endpoints- Package repository URI- Ceph configuration data driven by attributes
  35. 35. Gated EnvironmentsDreamHostDevelopmentStagingProduction
  36. 36. Operational Feedback- Continuous functional testing- Metrics, metrics, metrics- DashboardsDreamHost
  37. 37. Chef Infrastructure- Chef cluster per datacenter- Private Chef from Opscode- Erchef is awesome- Migrating legacy automationDreamHost
  38. 38. Chef for Ceph- Prototyped Ceph cluster automation with Chef- Adapted Ceph to ease configuration- Pushed some automation down into Ceph- Move towards being CM agnostic- Simplify Chef recipesDreamHost
  39. 39. Chef the network- DreamCompute utilizes ODM switches- Cumulus Networks provides Linux based OS- Custom Chef omnibus builds for PPC- Ohai networking!DreamHost
  40. 40. Resiliency EngineeringDreamHost- Amazon (GameDay)- Etsy (GameDay)- Google (DiRT)
  41. 41. Infra. a CodeDreamHost- Bare metal servers configured by code- Network devices configured by code- Block storage configured by code- Virtual networking configured by code
  42. 42. ThanksDreamHostDreamHostSage Weil and InktankOpenStack DevelopersOpscode
  43. 43. DreamObjectsDreamHostQuestions?