DreamObjectsCloud Object Storage Powered by Ceph
About Me•  Senior systems engineer at dreamhost•  Been with dreamhost since 2007•  Systems lead for DreamObjects•  Excited...
Anatomy of DreamObjects
Nuts and Bolts - Storage Nodes•  Intel nehalem or AMD lisbon•  32 GB of ram•  8-12 enterprise 3.5" sas disks•  LSI megarai...
Nuts and Bolts - Support Nodes•  Intel westmere-ep•  8 GB ram•  3x Gb nics•  4x SAS drives•  IPMI 2.0 BMCceph monitors, ra...
Tubes•  IPv6 native•  Dual-stack application proxy / balancer•  Application, rados, and cluster networks•  One gig links, ...
Traffic•  IPv6 and IPv4 api endpoints•  IPv6 to radosgw nodes•  Stud for ssl termination•  HAproxy for load balancing•  10...
Ceph Numbers•  Two plus petabytes of raw storage•  Triplicate replication•  900+ OSDs•  7 nines durability•  Single datace...
CRUSH Configuration•  Row, rack, node, and disk fault tolerant•  Weighted, even distribution of data•  Object striping•  O...
OSD Configuration•  Single disk RAID0•  Two partitions per RAID0, data + journal•  Writeback enabled•  Disk cache disabled...
Application Tuning•  No keepalive on apache•  Increase apache thread pool•  Increase radosgw thread pool•  Raise radosgw u...
Argonaut•  Lots of failure testing•  Tested on large clusters (~900 osds)•  Weight osds in/out of the cluster
Bobtail•  Tunables for less aggressive remapping•  CNAME support•  OSD performance ++
Control and Management•  chef•  pdsh•  runit/upstart•  jenkins•  sensu•  collectd•  graphite•  logstash
Chef•  Bootstrap cluster, new nodes•  Packages and configuration•  Creates, destroys and encrypts osds•  Search node metad...
pdshParallel distributed shell• Ceph monitors are a natural control point• pdsh from monitor to other nodes• ssh-agent + s...
runit/upstartKeep ceph daemons running• ceph-osd• ceph-mon• radosgw
Jenkins•  Continuous api functional tests•  Monitor chef convergence•  Generates alerts on failures
Sensu•  Relatively new monitoring system•  Open source•  Written in ruby•  State stored in Redis•  Pub-sub with local agen...
collectdSystem metrics• cpu• memory• disks• entropy• interfaces
graphite•  bucky - collectd, statsd, metricsd api•  carbon cache•  whisper for metric storage (sequential writes)•  graphi...
logstash•  lumberjack agent to ship logs to logstash•  logstash graphite output•  logstash elasticsearch output•  elastics...
Future•  East coast region•  Pool replication to another datacenter•  Object expiration, root site support•  Content deliv...
Thanks             Questions?    kyle.bader [at] dreamhost.com            @mmgaggle
Inktank’s Professional ServicesConsulting Services: •    Technical Overview •    Infrastructure Assessment •    Proof of C...
Check out our upcoming and ondemand webinars from InktankUpcoming:  Advanced Features of Ceph Distributed Storage  (delive...
Contact InktankInfo@inktank.com and 1-855-INKTANKDon’t forget to follow us on:Twitter: https://twitter.com/inktankFacebook...
THANK YOU!
Upcoming SlideShare
Loading in...5
×

Webinar - DreamObjects/Ceph Case Study

1,764

Published on

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,764
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
54
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Webinar - DreamObjects/Ceph Case Study

  1. 1. DreamObjectsCloud Object Storage Powered by Ceph
  2. 2. About Me•  Senior systems engineer at dreamhost•  Been with dreamhost since 2007•  Systems lead for DreamObjects•  Excited about storage, distributed systems, networking, linux internals, automation and monitoring•  I built my first ceph cluster in april, 2010twitter: @mmgaggle
  3. 3. Anatomy of DreamObjects
  4. 4. Nuts and Bolts - Storage Nodes•  Intel nehalem or AMD lisbon•  32 GB of ram•  8-12 enterprise 3.5" sas disks•  LSI megaraid 1078/2108•  5x Gb nics•  IPMI 2.0 BMC
  5. 5. Nuts and Bolts - Support Nodes•  Intel westmere-ep•  8 GB ram•  3x Gb nics•  4x SAS drives•  IPMI 2.0 BMCceph monitors, radosgw,proxies, object sync
  6. 6. Tubes•  IPv6 native•  Dual-stack application proxy / balancer•  Application, rados, and cluster networks•  One gig links, bonded•  LLDP for discovery
  7. 7. Traffic•  IPv6 and IPv4 api endpoints•  IPv6 to radosgw nodes•  Stud for ssl termination•  HAproxy for load balancing•  10Gb backend and frontend
  8. 8. Ceph Numbers•  Two plus petabytes of raw storage•  Triplicate replication•  900+ OSDs•  7 nines durability•  Single datacenter
  9. 9. CRUSH Configuration•  Row, rack, node, and disk fault tolerant•  Weighted, even distribution of data•  Object striping•  Online expansion and contraction
  10. 10. OSD Configuration•  Single disk RAID0•  Two partitions per RAID0, data + journal•  Writeback enabled•  Disk cache disabled•  Using XFS for filesystem
  11. 11. Application Tuning•  No keepalive on apache•  Increase apache thread pool•  Increase radosgw thread pool•  Raise radosgw ulimits
  12. 12. Argonaut•  Lots of failure testing•  Tested on large clusters (~900 osds)•  Weight osds in/out of the cluster
  13. 13. Bobtail•  Tunables for less aggressive remapping•  CNAME support•  OSD performance ++
  14. 14. Control and Management•  chef•  pdsh•  runit/upstart•  jenkins•  sensu•  collectd•  graphite•  logstash
  15. 15. Chef•  Bootstrap cluster, new nodes•  Packages and configuration•  Creates, destroys and encrypts osds•  Search node metadata, restful api•  Roles mapped to pdsh genders•  User and ssh key management•  Loads more..
  16. 16. pdshParallel distributed shell• Ceph monitors are a natural control point• pdsh from monitor to other nodes• ssh-agent + ssh agent forwarding• Works when chef api doesnt
  17. 17. runit/upstartKeep ceph daemons running• ceph-osd• ceph-mon• radosgw
  18. 18. Jenkins•  Continuous api functional tests•  Monitor chef convergence•  Generates alerts on failures
  19. 19. Sensu•  Relatively new monitoring system•  Open source•  Written in ruby•  State stored in Redis•  Pub-sub with local agent•  Handlers monitor state and generate alerts
  20. 20. collectdSystem metrics• cpu• memory• disks• entropy• interfaces
  21. 21. graphite•  bucky - collectd, statsd, metricsd api•  carbon cache•  whisper for metric storage (sequential writes)•  graphite webui
  22. 22. logstash•  lumberjack agent to ship logs to logstash•  logstash graphite output•  logstash elasticsearch output•  elasticsearch restful api•  kibana webui sits on top of elasticsearch
  23. 23. Future•  East coast region•  Pool replication to another datacenter•  Object expiration, root site support•  Content delivery network..not necessarily in that order ;)
  24. 24. Thanks Questions? kyle.bader [at] dreamhost.com @mmgaggle
  25. 25. Inktank’s Professional ServicesConsulting Services: •  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance TuningSupport Subscriptions: •  Pre-Production Support •  Production SupportA full description of our services can be found at the following:Consulting Services: http://www.inktank.com/consulting-services/Support Subscriptions: http://www.inktank.com/support-services/
  26. 26. Check out our upcoming and ondemand webinars from InktankUpcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph) February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/On Demand: Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/
  27. 27. Contact InktankInfo@inktank.com and 1-855-INKTANKDon’t forget to follow us on:Twitter: https://twitter.com/inktankFacebook: http://www.facebook.com/inktankYouTube: http://www.youtube.com/inktankstorage
  28. 28. THANK YOU!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×