Your SlideShare is downloading. ×
Webinar - DreamObjects/Ceph Case Study
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Webinar - DreamObjects/Ceph Case Study


Published on

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. DreamObjectsCloud Object Storage Powered by Ceph
  • 2. About Me•  Senior systems engineer at dreamhost•  Been with dreamhost since 2007•  Systems lead for DreamObjects•  Excited about storage, distributed systems, networking, linux internals, automation and monitoring•  I built my first ceph cluster in april, 2010twitter: @mmgaggle
  • 3. Anatomy of DreamObjects
  • 4. Nuts and Bolts - Storage Nodes•  Intel nehalem or AMD lisbon•  32 GB of ram•  8-12 enterprise 3.5" sas disks•  LSI megaraid 1078/2108•  5x Gb nics•  IPMI 2.0 BMC
  • 5. Nuts and Bolts - Support Nodes•  Intel westmere-ep•  8 GB ram•  3x Gb nics•  4x SAS drives•  IPMI 2.0 BMCceph monitors, radosgw,proxies, object sync
  • 6. Tubes•  IPv6 native•  Dual-stack application proxy / balancer•  Application, rados, and cluster networks•  One gig links, bonded•  LLDP for discovery
  • 7. Traffic•  IPv6 and IPv4 api endpoints•  IPv6 to radosgw nodes•  Stud for ssl termination•  HAproxy for load balancing•  10Gb backend and frontend
  • 8. Ceph Numbers•  Two plus petabytes of raw storage•  Triplicate replication•  900+ OSDs•  7 nines durability•  Single datacenter
  • 9. CRUSH Configuration•  Row, rack, node, and disk fault tolerant•  Weighted, even distribution of data•  Object striping•  Online expansion and contraction
  • 10. OSD Configuration•  Single disk RAID0•  Two partitions per RAID0, data + journal•  Writeback enabled•  Disk cache disabled•  Using XFS for filesystem
  • 11. Application Tuning•  No keepalive on apache•  Increase apache thread pool•  Increase radosgw thread pool•  Raise radosgw ulimits
  • 12. Argonaut•  Lots of failure testing•  Tested on large clusters (~900 osds)•  Weight osds in/out of the cluster
  • 13. Bobtail•  Tunables for less aggressive remapping•  CNAME support•  OSD performance ++
  • 14. Control and Management•  chef•  pdsh•  runit/upstart•  jenkins•  sensu•  collectd•  graphite•  logstash
  • 15. Chef•  Bootstrap cluster, new nodes•  Packages and configuration•  Creates, destroys and encrypts osds•  Search node metadata, restful api•  Roles mapped to pdsh genders•  User and ssh key management•  Loads more..
  • 16. pdshParallel distributed shell• Ceph monitors are a natural control point• pdsh from monitor to other nodes• ssh-agent + ssh agent forwarding• Works when chef api doesnt
  • 17. runit/upstartKeep ceph daemons running• ceph-osd• ceph-mon• radosgw
  • 18. Jenkins•  Continuous api functional tests•  Monitor chef convergence•  Generates alerts on failures
  • 19. Sensu•  Relatively new monitoring system•  Open source•  Written in ruby•  State stored in Redis•  Pub-sub with local agent•  Handlers monitor state and generate alerts
  • 20. collectdSystem metrics• cpu• memory• disks• entropy• interfaces
  • 21. graphite•  bucky - collectd, statsd, metricsd api•  carbon cache•  whisper for metric storage (sequential writes)•  graphite webui
  • 22. logstash•  lumberjack agent to ship logs to logstash•  logstash graphite output•  logstash elasticsearch output•  elasticsearch restful api•  kibana webui sits on top of elasticsearch
  • 23. Future•  East coast region•  Pool replication to another datacenter•  Object expiration, root site support•  Content delivery network..not necessarily in that order ;)
  • 24. Thanks Questions? kyle.bader [at] @mmgaggle
  • 25. Inktank’s Professional ServicesConsulting Services: •  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance TuningSupport Subscriptions: •  Pre-Production Support •  Production SupportA full description of our services can be found at the following:Consulting Services: Subscriptions:
  • 26. Check out our upcoming and ondemand webinars from InktankUpcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph) February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET Demand: Getting Started with Ceph Intro to Ceph with OpenStack
  • 27. Contact and 1-855-INKTANKDon’t forget to follow us on:Twitter:
  • 28. THANK YOU!