Zoltan Arnold Nagy, IBM

1. Zoltan Arnold Nagy
23. 10. 2014.
Ceph at IBM and IBM Research
Zoltan Arnold Nagy
IBM Research – Zurich
nag@zurich.ibm.com
© 2014 IBM Corporation

2. Zoltan Arnold Nagy - Ceph at IBM
Legal disclaimer
All information contained in this document is subject to change
without notice. The products described in this document are
generally available via the normal IBM sales channels. The
information contained in this document does not affect or change
IBM product specifications or warranties. Nothing in this
document shall operate as an express or implied license or
indemnity under the intellectual property rights of IBM or third
parties. All information contained in this document was obtained
in specific environments, and is presented as an illustration. The
results obtained in other operating environments may vary.
© 2014 2/16 IBM Corporation

3. Zoltan Arnold Nagy - Ceph at IBM
Agenda
Ceph at IBM Research - Zurich
Ceph on Softlayer
Plans for the future
© 2014 3/16 IBM Corporation

4. Zoltan Arnold Nagy - Ceph at IBM
Agenda
Ceph at IBM Research - Zurich
Ceph on Softlayer
Plans for the future
© 2014 4/16 IBM Corporation

5. Zoltan Arnold Nagy - Ceph at IBM
The need for a local cloud at the
Zurich Research Lab
Highly available and fast compute nodes
— Simulations need reliable infrastructure
Projects with high storage needs
— Not suitable for migrating to other site
— Legal issues about data handling
Network and cloud security research
— Needs a very responsive admin team
Few hundred users
— But not usual end-users
— Different OS needs
© 2014 5/16 IBM Corporation

6. Zoltan Arnold Nagy - Ceph at IBM
First iteration
3 compute nodes with 396 GB of RAM and 36 CPUs
— System x3550 M4
4 storage nodes with 13.5 TB of total raw space on SSDs
— 2 storage nodes are hybrid compute & storage (not optimal)
— System x3550 M4, Supermicro SC216E16-R1200LPB
Everything connected with 2x10GbE
OpenStack Havana & Ceph Dumpling (0.67) on Ubuntu 12.04 LTS
— KVM hypervisors
© 2014 6/16 IBM Corporation

7. Zoltan Arnold Nagy - Ceph at IBM
Boundaries hit quickly
Storage need growth is ~1TB / month
— Ceph is CPU hungry when doing small block IO
—We've grown to 38 SSD disks (17TB RAW)
at the time of sunset (Oct 2014)
Had a too high price point for general volume storage
— Some hadoop users would need ~30TB
Network is doing well, not saturated
© 2014 7/16 IBM Corporation

8. Zoltan Arnold Nagy - Ceph at IBM
Second iteration
6x System x3650 M4 BD storage boxes
— Ceph 0.80.7 on Ubuntu 14.04
— 144 TB raw capacity on HDDs
 12x2TB OSD per node
 2x120GB SSD for journal
— OS on dedicated SSDs
— 6 disk journals to one SSD
3 monitors running in Vms
Will repurpose the SSD nodes to provide
a second cinder pool for special use-cases
© 2014 8/16 IBM Corporation

9. Zoltan Arnold Nagy - Ceph at IBM
Agenda
Ceph at IBM Research – Zurich
Ceph on Softlayer
Plans for the future
© 2014 9/16 IBM Corporation

10. Zoltan Arnold Nagy - Ceph at IBM
Agenda
Ceph at IBM Research – Zurich
Ceph on Softlayer
Plans for the future
© 2014 10/16 IBM Corporation

11. Zoltan Arnold Nagy - Ceph at IBM
IBM's cloud participation is growing
SoftLayer acquired in 2013
— Mostly bare-metal hosting, but VMs are also available
OpenStack Foundation platinum member
— Ranking among the top contributors release by release
Strong commitment to cloud technologies
© 2014 11/16 IBM Corporation

12. Zoltan Arnold Nagy - Ceph at IBM
Ceph on Softlayer
Hosted managed private cloud
—Using Softlayer as a physical foundation
—Using ceph for volume storage! (ephemeral is still on local RAID10)
Consist of
—3 controller nodes running in VMs on 3 physical “controllers”
—1GbE / 10GbE connectivity across hypervisors/storage nodes
depending on physical provisioning location
—Fixed storage size (between 8TB and 96TB usable in the initial offering)
—2-way replication instead of 3 to keep costs down
 No erasure coding yet
© 2014 12/16 IBM Corporation

13. Zoltan Arnold Nagy - Ceph at IBM
Agenda
Ceph at IBM Research – Zurich
Ceph on Softlayer
Plans for the future
© 2014 13/16 IBM Corporation

14. Zoltan Arnold Nagy - Ceph at IBM
Plans for the future
Currently nova uses a central cephx token to access ceph
—Would be nice to have per-tenant cephx keys
—Need to propagate image ACLs to ceph auth rules
Working on proper rbd snapshotting
—Apparently will need an approved blueprint first...
Simplified multi-DC support for stretched ceph deployments
—Via playing with CRUSH physical data separation is doable, but not so
convenient
—Need async replication support so we can tolerate latency
© 2014 14/16 IBM Corporation

15. Zoltan Arnold Nagy - Ceph at IBM
Questions?
© 2014 15/16 IBM Corporation

16. Zoltan Arnold Nagy - Ceph at IBM
Thank You!
© 2014 16/16 IBM Corporation