This document summarizes DreamObjects, an object storage platform powered by Ceph. It discusses the hardware used in storage and support nodes, including Intel and AMD processors, RAM, disks, and networking components. The document also provides details on Ceph configuration including replication, CRUSH mapping, OSD configuration, and application tuning. Monitoring tools discussed include Chef, pdsh, Sensu, collectd, graphite, logstash, Jenkins and future plans.

1. DreamObjects
Cloud Object Storage
Powered by Ceph

2. About Me
• Senior systems engineer at dreamhost
• Been with dreamhost since 2007
• Systems lead for DreamObjects
• Excited about storage, distributed systems,
networking, linux internals, automation and
monitoring
• I built my first ceph cluster in april, 2010
twitter: @mmgaggle

3. Anatomy of DreamObjects

4. Nuts and Bolts - Storage Nodes
• Intel nehalem or AMD lisbon
• 32 GB of ram
• 8-12 enterprise 3.5" sas disks
• LSI megaraid 1078/2108
• 5x Gb nics
• IPMI 2.0 BMC

5. Nuts and Bolts - Support Nodes
• Intel westmere-ep
• 8 GB ram
• 3x Gb nics
• 4x SAS drives
• IPMI 2.0 BMC
ceph monitors, radosgw,
proxies, object sync

6. Tubes
• IPv6 native
• Dual-stack application proxy / balancer
• Application, rados, and cluster networks
• One gig links, bonded
• LLDP for discovery

7. Traffic
• IPv6 and IPv4 api endpoints
• IPv6 to radosgw nodes
• Stud for ssl termination
• HAproxy for load balancing
• 10Gb backend and frontend

8. Ceph Numbers
• Two plus petabytes of raw storage
• Triplicate replication
• 900+ OSDs
• 7 nines durability
• Single datacenter

9. CRUSH Configuration
• Row, rack, node, and disk fault tolerant
• Weighted, even distribution of data
• Object striping
• Online expansion and contraction

10. OSD Configuration
• Single disk RAID0
• Two partitions per RAID0, data + journal
• Writeback enabled
• Disk cache disabled
• Using XFS for filesystem

11. Application Tuning
• No keepalive on apache
• Increase apache thread pool
• Increase radosgw thread pool
• Raise radosgw ulimits

12. Argonaut
• Lots of failure testing
• Tested on large clusters (~900 osds)
• Weight osds in/out of the cluster

13. Bobtail
• Tunables for less aggressive remapping
• CNAME support
• OSD performance ++

14. Control and Management
• chef
• pdsh
• runit/upstart
• jenkins
• sensu
• collectd
• graphite
• logstash

15. Chef
• Bootstrap cluster, new nodes
• Packages and configuration
• Creates, destroys and encrypts osds
• Search node metadata, restful api
• Roles mapped to pdsh genders
• User and ssh key management
• Loads more..

16. pdsh
Parallel distributed shell
• Ceph monitors are a natural control point
• pdsh from monitor to other nodes
• ssh-agent + ssh agent forwarding
• Works when chef api doesn't

17. runit/upstart
Keep ceph daemons running
• ceph-osd
• ceph-mon
• radosgw

18. Jenkins
• Continuous api functional tests
• Monitor chef convergence
• Generates alerts on failures

19. Sensu
• Relatively new monitoring system
• Open source
• Written in ruby
• State stored in Redis
• Pub-sub with local agent
• Handlers monitor state and generate alerts

20. collectd
System metrics
• cpu
• memory
• disks
• entropy
• interfaces

21. graphite
• bucky - collectd, statsd, metricsd api
• carbon cache
• whisper for metric storage (sequential writes)
• graphite webui

22. logstash
• lumberjack agent to ship logs to logstash
• logstash graphite output
• logstash elasticsearch output
• elasticsearch restful api
• kibana webui sits on top of elasticsearch

23. Future
• East coast region
• Pool replication to another datacenter
• Object expiration, root site support
• Content delivery network
..not necessarily in that order ;)

24. Thanks
Questions?
kyle.bader [at] dreamhost.com
@mmgaggle

25. Inktank’s Professional Services
Consulting Services:
• Technical Overview
• Infrastructure Assessment
• Proof of Concept
• Implementation Support
• Performance Tuning
Support Subscriptions:
• Pre-Production Support
• Production Support
A full description of our services can be found at the following:
Consulting Services: http://www.inktank.com/consulting-services/
Support Subscriptions: http://www.inktank.com/support-services/

26. Check out our upcoming and on
demand webinars from Inktank
Upcoming:
Advanced Features of Ceph Distributed Storage
(delivered by Sage Weil, creator of Ceph)
February 12, 2013
10:00AM PT, 12:00PM CT, 1:00PM ET
http://www.inktank.com/news-events/webinars/
On Demand:
Getting Started with Ceph
http://www.inktank.com/news-events/webinars/
Intro to Ceph with OpenStack
http://www.inktank.com/news-events/webinars/

27. Contact Inktank
Info@inktank.com and 1-855-INKTANK
Don’t forget to follow us on:
Twitter: https://twitter.com/inktank
Facebook: http://www.facebook.com/inktank
YouTube: http://www.youtube.com/inktankstorage

28. THANK YOU!