Webinar - DreamObjects/Ceph Case Study
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Webinar - DreamObjects/Ceph Case Study

  • 2,449 views
Uploaded on

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

Slides from our DreamObjects/Ceph webinar. This webinar was guest hosted by Kyle Bader of DreamHost.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,449
On Slideshare
2,093
From Embeds
356
Number of Embeds
3

Actions

Shares
Downloads
50
Comments
0
Likes
3

Embeds 356

http://www.inktank.com 235
http://dev.gluesys.com 118
https://twitter.com 3

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. DreamObjectsCloud Object Storage Powered by Ceph
  • 2. About Me•  Senior systems engineer at dreamhost•  Been with dreamhost since 2007•  Systems lead for DreamObjects•  Excited about storage, distributed systems, networking, linux internals, automation and monitoring•  I built my first ceph cluster in april, 2010twitter: @mmgaggle
  • 3. Anatomy of DreamObjects
  • 4. Nuts and Bolts - Storage Nodes•  Intel nehalem or AMD lisbon•  32 GB of ram•  8-12 enterprise 3.5" sas disks•  LSI megaraid 1078/2108•  5x Gb nics•  IPMI 2.0 BMC
  • 5. Nuts and Bolts - Support Nodes•  Intel westmere-ep•  8 GB ram•  3x Gb nics•  4x SAS drives•  IPMI 2.0 BMCceph monitors, radosgw,proxies, object sync
  • 6. Tubes•  IPv6 native•  Dual-stack application proxy / balancer•  Application, rados, and cluster networks•  One gig links, bonded•  LLDP for discovery
  • 7. Traffic•  IPv6 and IPv4 api endpoints•  IPv6 to radosgw nodes•  Stud for ssl termination•  HAproxy for load balancing•  10Gb backend and frontend
  • 8. Ceph Numbers•  Two plus petabytes of raw storage•  Triplicate replication•  900+ OSDs•  7 nines durability•  Single datacenter
  • 9. CRUSH Configuration•  Row, rack, node, and disk fault tolerant•  Weighted, even distribution of data•  Object striping•  Online expansion and contraction
  • 10. OSD Configuration•  Single disk RAID0•  Two partitions per RAID0, data + journal•  Writeback enabled•  Disk cache disabled•  Using XFS for filesystem
  • 11. Application Tuning•  No keepalive on apache•  Increase apache thread pool•  Increase radosgw thread pool•  Raise radosgw ulimits
  • 12. Argonaut•  Lots of failure testing•  Tested on large clusters (~900 osds)•  Weight osds in/out of the cluster
  • 13. Bobtail•  Tunables for less aggressive remapping•  CNAME support•  OSD performance ++
  • 14. Control and Management•  chef•  pdsh•  runit/upstart•  jenkins•  sensu•  collectd•  graphite•  logstash
  • 15. Chef•  Bootstrap cluster, new nodes•  Packages and configuration•  Creates, destroys and encrypts osds•  Search node metadata, restful api•  Roles mapped to pdsh genders•  User and ssh key management•  Loads more..
  • 16. pdshParallel distributed shell• Ceph monitors are a natural control point• pdsh from monitor to other nodes• ssh-agent + ssh agent forwarding• Works when chef api doesnt
  • 17. runit/upstartKeep ceph daemons running• ceph-osd• ceph-mon• radosgw
  • 18. Jenkins•  Continuous api functional tests•  Monitor chef convergence•  Generates alerts on failures
  • 19. Sensu•  Relatively new monitoring system•  Open source•  Written in ruby•  State stored in Redis•  Pub-sub with local agent•  Handlers monitor state and generate alerts
  • 20. collectdSystem metrics• cpu• memory• disks• entropy• interfaces
  • 21. graphite•  bucky - collectd, statsd, metricsd api•  carbon cache•  whisper for metric storage (sequential writes)•  graphite webui
  • 22. logstash•  lumberjack agent to ship logs to logstash•  logstash graphite output•  logstash elasticsearch output•  elasticsearch restful api•  kibana webui sits on top of elasticsearch
  • 23. Future•  East coast region•  Pool replication to another datacenter•  Object expiration, root site support•  Content delivery network..not necessarily in that order ;)
  • 24. Thanks Questions? kyle.bader [at] dreamhost.com @mmgaggle
  • 25. Inktank’s Professional ServicesConsulting Services: •  Technical Overview •  Infrastructure Assessment •  Proof of Concept •  Implementation Support •  Performance TuningSupport Subscriptions: •  Pre-Production Support •  Production SupportA full description of our services can be found at the following:Consulting Services: http://www.inktank.com/consulting-services/Support Subscriptions: http://www.inktank.com/support-services/
  • 26. Check out our upcoming and ondemand webinars from InktankUpcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph) February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/On Demand: Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/
  • 27. Contact InktankInfo@inktank.com and 1-855-INKTANKDon’t forget to follow us on:Twitter: https://twitter.com/inktankFacebook: http://www.facebook.com/inktankYouTube: http://www.youtube.com/inktankstorage
  • 28. THANK YOU!