Burkhard Noltensmeier
teuto.net Netzdienste GmbH
Erkan Yanar
Consultant
teuto.net Netzdienste GmbH
● 18 Mitarbeiter
● Linux Systemhaus und
Webdevelopment
● Ubuntu Advantage Partner
● Openstack C...
Why Openstack ?
Infrastructure as a Sevice
● Cloud Init (automated Instance provisioning)
● Network Virtualization
● Multi...
● closed beta since September 2013
● updated to Havana in October
● Ubuntu Cloud Archive
● 20 Compute Nodes
● 5 Ceph Nodes...
Provisioning and Orchestration
Openstack Storage Types
● Block Storage
● Object Storage
● Image Repository
● Internal Cluster Storage
– Temorary Image St...
Storage Requirements
● Scalability
● Redundancy
● Performance
● Efficient Pooling
Key Facts for our Decision
● One Ceph Cluster fits all Openstack needs
● no „single Point of Failure“
● POSIX compatibilit...
Rados Block Storage
● Live migration
● Efficient Snapshots
● Different types of storage avaiable (tiering)
● Cloning for f...
How to start
● determine Clustersize
uneven amount of Nodes to enable negotiation
● Small start with at least 5 Nodes
● ei...
Rough calculation
● 3 Nodes, 8 Disks per Node, 2 replica
● Netto = Brutto / 2 replica – 1 Node (33%) =
33%
Cluster Brutto
...
Rough calculation
● 5 Nodes, 8 Disks per Node, 3 replica
● Netto = Brutto / 3 replica – 1 Node (20%) =
27%
Cluster Brutto
...
Ceph specifics
● Data is distributed throughout the Cluster
● Unfortunately this destroys Data locality
tradeoff between b...
Operational Challenges
● Performance
● Availability
● Qos (Quality of Service)
Ceph Monitoring in ostack
● Ensure Quality with Monitoring
● Easy spotting of congestion Problems
● Event Monitoring (e.g....
What we did
● Disk monitoring with Icinga
● Collect data via Ceph Admin Socket Json
interface
● put it into Graphite
● enr...
Cumulated osd Performance
Single osd performance
Sum by Openstack tenant
Verify Ceph Performance
● Fio Benchmark with fixed file size
fio ­­fsync=<n> ­­runtime=60 ­­size=1g –bs=<n> ...
● Differen...
Cinder Quality of Service
$ cinder qos­create high­iops consumer="front­end" 
  read_iops_sec=100       write_iops_sec=100...
Speed per Cinder Qos
Does it scale
Effect of syncing Files
Different Blocksize with sync
Ceph is somewhat complex, but
● reliable
● No unpleasent suprises (so far!)
● Monitoring is important for resource
managem...
Using Ceph in OStack.de - Ceph Day Frankfurt
Using Ceph in OStack.de - Ceph Day Frankfurt
Using Ceph in OStack.de - Ceph Day Frankfurt
Upcoming SlideShare
Loading in …5
×

Using Ceph in OStack.de - Ceph Day Frankfurt

797 views

Published on

Burkhard Noltensmeier, Teuto

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
797
On SlideShare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
56
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Using Ceph in OStack.de - Ceph Day Frankfurt

  1. 1. Burkhard Noltensmeier teuto.net Netzdienste GmbH Erkan Yanar Consultant
  2. 2. teuto.net Netzdienste GmbH ● 18 Mitarbeiter ● Linux Systemhaus und Webdevelopment ● Ubuntu Advantage Partner ● Openstack Ceph Service ● Büros und Datacenter in Bielefeld
  3. 3. Why Openstack ? Infrastructure as a Sevice ● Cloud Init (automated Instance provisioning) ● Network Virtualization ● Multiple Storage Options ● Multiple APIs for Automation
  4. 4. ● closed beta since September 2013 ● updated to Havana in October ● Ubuntu Cloud Archive ● 20 Compute Nodes ● 5 Ceph Nodes ● Additional Monitoring with Graphite
  5. 5. Provisioning and Orchestration
  6. 6. Openstack Storage Types ● Block Storage ● Object Storage ● Image Repository ● Internal Cluster Storage – Temorary Image Store – Databases (Mysql Galera,MongoDB)
  7. 7. Storage Requirements ● Scalability ● Redundancy ● Performance ● Efficient Pooling
  8. 8. Key Facts for our Decision ● One Ceph Cluster fits all Openstack needs ● no „single Point of Failure“ ● POSIX compatibility via Rados Block Device ● seamless scalability ● commercial support by Inktank ● GPL
  9. 9. Rados Block Storage ● Live migration ● Efficient Snapshots ● Different types of storage avaiable (tiering) ● Cloning for fast restore or scaling
  10. 10. How to start ● determine Clustersize uneven amount of Nodes to enable negotiation ● Small start with at least 5 Nodes ● either 8 or 12 Disks per Chassis ● One Jounal per Disk ● 2 Journal SSD per Chassis
  11. 11. Rough calculation ● 3 Nodes, 8 Disks per Node, 2 replica ● Netto = Brutto / 2 replica – 1 Node (33%) = 33% Cluster Brutto ● 24 2TB Sata Disks, 100 IOPS each Cluster Netto ● 15,8 Terrabyte, 790 IOPS
  12. 12. Rough calculation ● 5 Nodes, 8 Disks per Node, 3 replica ● Netto = Brutto / 3 replica – 1 Node (20%) = 27% Cluster Brutto ● 40 2TB Sata Disks, 100 IOPS each Cluster Netto ● 21,3 Terrabyte, 1066 IOPS
  13. 13. Ceph specifics ● Data is distributed throughout the Cluster ● Unfortunately this destroys Data locality tradeoff between blocksize an iops. ● The bigger Blocks, the better is sequential performance ● Double Write, SSD Journals strongly advised ● Longterm fragmentation by small writes
  14. 14. Operational Challenges ● Performance ● Availability ● Qos (Quality of Service)
  15. 15. Ceph Monitoring in ostack ● Ensure Quality with Monitoring ● Easy spotting of congestion Problems ● Event Monitoring (e.g. disk failure) ● Capacity management
  16. 16. What we did ● Disk monitoring with Icinga ● Collect data via Ceph Admin Socket Json interface ● put it into Graphite ● enrich it with Meta Data – with Openstack tennant – Ceph Node – OSD
  17. 17. Cumulated osd Performance
  18. 18. Single osd performance
  19. 19. Sum by Openstack tenant
  20. 20. Verify Ceph Performance ● Fio Benchmark with fixed file size fio ­­fsync=<n> ­­runtime=60 ­­size=1g –bs=<n> ... ● Different sync option nosync, 1, 100 ● Different Cinder Qos Service Options ● Blocksize 64k 512k 1024k 4096k ● 1 up to 4 VM Clients ● Resulting in 500 Benchmark runs..
  21. 21. Cinder Quality of Service $ cinder qos­create high­iops consumer="front­end"    read_iops_sec=100       write_iops_sec=100    read_bytes_sec=41943040 write_bytes_sec=41943040 $ cinder qos­create low­iops consumer="front­end"    read_iops_sec=50        write_iops_sec=50    read_bytes_sec=20971520 write_bytes_sec=20971520 $ cinder qos­create ultra­low­iopsconsumer="front­end"   read_iops_sec=10        write_iops_sec=10    read_bytes_sec=10485760  write_bytes_sec=10485760
  22. 22. Speed per Cinder Qos
  23. 23. Does it scale
  24. 24. Effect of syncing Files
  25. 25. Different Blocksize with sync
  26. 26. Ceph is somewhat complex, but ● reliable ● No unpleasent suprises (so far!) ● Monitoring is important for resource management and availabilty !

×