Webinar - DreamObjects/Ceph Case Study
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Webinar - DreamObjects/Ceph Case Study



Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.



Total Views
Views on SlideShare
Embed Views



3 Embeds 351

http://www.inktank.com 230
http://dev.gluesys.com 118
https://twitter.com 3



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Webinar - DreamObjects/Ceph Case Study Presentation Transcript

  • 1. DreamObjectsCloud Object Storage Powered by Ceph
  • 2. About Me•  Senior systems engineer at dreamhost•  Been with dreamhost since 2007•  Systems lead for DreamObjects•  Excited about storage, distributed systems, networking, linux internals, automation and monitoring•  I built my first ceph cluster in april, 2010twitter: @mmgaggle
  • 3. Anatomy of DreamObjects
  • 4. Nuts and Bolts - Storage Nodes•  Intel nehalem or AMD lisbon•  32 GB of ram•  8-12 enterprise 3.5" sas disks•  LSI megaraid 1078/2108•  5x Gb nics•  IPMI 2.0 BMC
  • 5. Nuts and Bolts - Support Nodes•  Intel westmere-ep•  8 GB ram•  3x Gb nics•  4x SAS drives•  IPMI 2.0 BMCceph monitors, radosgw,proxies, object sync
  • 6. Tubes•  IPv6 native•  Dual-stack application proxy / balancer•  Application, rados, and cluster networks•  One gig links, bonded•  LLDP for discovery
  • 7. Traffic•  IPv6 and IPv4 api endpoints•  IPv6 to radosgw nodes•  Stud for ssl termination•  HAproxy for load balancing•  10Gb backend and frontend
  • 8. Ceph Numbers•  Two plus petabytes of raw storage•  Triplicate replication•  900+ OSDs•  7 nines durability•  Single datacenter
  • 9. CRUSH Configuration•  Row, rack, node, and disk fault tolerant•  Weighted, even distribution of data•  Object striping•  Online expansion and contraction
  • 10. OSD Configuration•  Single disk RAID0•  Two partitions per RAID0, data + journal•  Writeback enabled•  Disk cache disabled•  Using XFS for filesystem
  • 11. Application Tuning•  No keepalive on apache•  Increase apache thread pool•  Increase radosgw thread pool•  Raise radosgw ulimits
  • 12. Argonaut•  Lots of failure testing•  Tested on large clusters (~900 osds)•  Weight osds in/out of the cluster
  • 13. Bobtail•  Tunables for less aggressive remapping•  CNAME support•  OSD performance ++
  • 14. Control and Management•  chef•  pdsh•  runit/upstart•  jenkins•  sensu•  collectd•  graphite•  logstash
  • 15. Chef•  Bootstrap cluster, new nodes•  Packages and configuration•  Creates, destroys and encrypts osds•  Search node metadata, restful api•  Roles mapped to pdsh genders•  User and ssh key management•  Loads more..
  • 16. pdshParallel distributed shell• Ceph monitors are a natural control point• pdsh from monitor to other nodes• ssh-agent + ssh agent forwarding• Works when chef api doesnt
  • 17. runit/upstartKeep ceph daemons running• ceph-osd• ceph-mon• radosgw
  • 18. Jenkins•  Continuous api functional tests•  Monitor chef convergence•  Generates alerts on failures
  • 19. Sensu•  Relatively new monitoring system•  Open source•  Written in ruby•  State stored in Redis•  Pub-sub with local agent•  Handlers monitor state and generate alerts
  • 20. collectdSystem metrics• cpu• memory• disks• entropy• interfaces
  • 21. graphite•  bucky - collectd, statsd, metricsd api•  carbon cache•  whisper for metric storage (sequential writes)•  graphite webui
  • 22. logstash•  lumberjack agent to ship logs to logstash•  logstash graphite output•  logstash elasticsearch output•  elasticsearch restful api•  kibana webui sits on top of elasticsearch
  • 23. Future•  East coast region•  Pool replication to another datacenter•  Object expiration, root site support•  Content delivery network..not necessarily in that order ;)
  • 24. Thanks Questions? kyle.bader [at] dreamhost.com @mmgaggle
  • 25. Inktank’s Professional ServicesConsulting Services: •  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance TuningSupport Subscriptions: •  Pre-Production Support •  Production SupportA full description of our services can be found at the following:Consulting Services: http://www.inktank.com/consulting-services/Support Subscriptions: http://www.inktank.com/support-services/
  • 26. Check out our upcoming and ondemand webinars from InktankUpcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph) February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/On Demand: Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/
  • 27. Contact InktankInfo@inktank.com and 1-855-INKTANKDon’t forget to follow us on:Twitter: https://twitter.com/inktankFacebook: http://www.facebook.com/inktankYouTube: http://www.youtube.com/inktankstorage
  • 28. THANK YOU!