Integrating CloudStack & Ceph


Published on

Wido den Hollander (@widoh) did a great presentation on his work to integrate Cloudstack to Ceph

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Integrating CloudStack & Ceph

  1. 1. Ceph as storage for CloudStack Wido den Hollander <>
  2. 2. Who am I? ● Wido den Hollander – – Co-owner of a dutch hosting company – ● Part of the Ceph community since 2010 Committer and PMC member for Apache CloudStack Developed: – – rados-java – libvirt RBD storage pool support – ● phprados CloudStack integration Work as a Ceph and CloudStack consultant
  3. 3. Ceph Ceph is a unified, open source distributed object store
  4. 4. Auto recovery ● ● Recovery when a OSD fails Data migration when the cluster expands or contracts
  5. 5. Traditional vs Distributed Traditional storage systems don't scale that well – All have their limitations: Number of disks, shelfs, CPUs, network connections, etc – Scaling usually meant buying a second system ● ● Ceph clusters can grow and shrink without service interruptions – ● Migrating data requires service windows We don't want to watch rsync copying over data and wasting our time Ceph runs on commodity hardware – Just add more nodes to add capacity – Ceph fits in smaller budgets
  6. 6. Hardware failure is the rule ● As systems grow hardware failure becomes more frequent – – ● A system with 1.000 nodes will see daily hardware issues We don't want to get out of bed when a machine fails at 03:00 on Sunday morning Commodity hardware is cheaper, but less reliable. Ceph mitigates that
  7. 7. RBD: the RADOS Block Device ● Ceph is a object store – – ● Store billions of objects in pools RADOS is the heart of Ceph RBD block devices are striped over RADOS objects – Default stripe size is 4MB – All objects are distributed over all available Object Store Daemons (OSDs) – 40GB image consists out of 10.000 potential objects – Thin provisioned
  8. 8. RADOS Block Device
  9. 9. RBD for Primary Storage ● In 4.0 RBD support for Primary Storage for KVM was added – No support for VMware or Xen – Xen support is being worked on (not by me) ● Live migration is supported ● Snapshot and backup support (4.2) ● Cloning when deploying from templates ● Run System VMs from RBD (4.2) ● Uses the rados-java bindings
  10. 10. RBD for Primary Storage
  11. 11. System Virtual Machines ● Perform cluster tasks, e.g.: – – Serving metadata to Instances – Loadbalancing – Copying data between clusters – ● DHCP Run in between user Instances They can now run from RBD due to a change in the way they get their metadata – Old way was dirty and had to be replaced ● It created a small disk with metadata files
  12. 12. rados-java bindings ● Developed to have the KVM Agent perform snapshotting and cloning – libvirt doesn't know how to do this, but it would be best if it did ● Uses JNA, so easy deployment ● Binds both librados and librbd ● Available on
  13. 13. Future plans ● Add RBD write caching – Write-cache setting per Disk Offering ● – ● none (default), write-back and write-through Probably in 4.3 Native RADOS support for Secondary Storage – – ● Secondary Storage already supports S3 Ceph has a S3-compatible gateway Moving logic from the KVM Agent into libvirt – Like snapshotting and cloning RBD images
  14. 14. Help is needed! ● Code is tested, but testing is always welcome ● Adding more RBD logic into libvirt – – Cloning RBD images – ● Snapshotting RBD images This makes the CloudStack code cleaner and helps other users who also use libvirt with RBD Improving the rados-java bindings – Not feature complete yet
  15. 15. Thanks ● Find me on: – E-Mail: – IRC: widodh @ Freenode / wido @ OFTC – Skype: widodh / contact42on – Twitter: widodh