Ceph Day Tokyo - Bit-Isle's 3 years footprint with Ceph
•
0 likes•205 views
Bit-isle has been using Ceph storage with OpenStack for 3 years, starting with a proof of concept in 2013. They have three Ceph environments - a development environment using OpenStack Havana and Ceph Dumpling, a staging environment using OpenStack Juno and Ceph Giant, and a production customer environment using OpenStack Kilo and Ceph Hammer. They chose Ceph because it provides high performance scalable storage without the need for expensive dedicated storage appliances or many storage engineers. Their initial POC was successful and showed Ceph could provide fault tolerance and cooperate well with OpenStack.
Recommended
Recommended
More Related Content
What's hot
What's hot (17)
Viewers also liked
Viewers also liked (20)
Similar to Ceph Day Tokyo - Bit-Isle's 3 years footprint with Ceph
Similar to Ceph Day Tokyo - Bit-Isle's 3 years footprint with Ceph (20)
Recently uploaded
Recently uploaded (20)
Ceph Day Tokyo - Bit-Isle's 3 years footprint with Ceph
- 1. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights ReservedCopyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Bit-‐‑‒isle's three years footprint with Ceph Ikuo Kumagai – Bit-‐‑‒isle Equinix Inc.
- 2. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved ビットアイル・エクイニクス株式会社 会社概要 ■資本⾦金金 ■代表者 ■設 ⽴立立 ■所在地 ■URL ■株主 ■主要サービス ■グループ会社： ：35億69百万円 ：代表取締役 古⽥田敬 ：2000年年6⽉月14⽇日 ：〒140-‐‑‒0002 東京都品川区東品川2-‐‑‒2-‐‑‒28 Tビル Tel:03-‐‑‒5805-‐‑‒8151（代表） Fax:03-‐‑‒3474-‐‑‒5540 ：http://www.bit-‐‑‒isle.jp ：QAON合同会社（エクイニクスグループ） ：データセンターサービス、 クラウドサービス、運⽤用サービス、 システムインテグレーション 2
- 3. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Open my Stack •Ikuo Kumagai（@kumagai19o） • Blog : Bit-‐‑‒isle R&D institute blog (Japanese only) Career • 2012 ~∼ OpenStack R&D & Providing hosted private OpenStack • 2011 ~∼ Developing a cloud system based VMware and BIG-‐‑‒IP • 2007 ~∼ Designing a Financial System Infrastructure • 2006 ~∼ Developing a Job Scheduler Package • 2004 ~∼ Production Control System Developer • 2001 ~∼ Visual Basic/Java Programmer 3
- 4. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved 2011年年 • POC with midokura 2012年年/2013年年 • POC (Essex) • Swift β (for in-‐‑‒house) • RD-‐‑‒Cloud service (Folsom/in-‐‑‒house) • POC of Ceph 2014年年 •RD-‐‑‒Cloud-‐‑‒1 service (Havana/in-‐‑‒house) •OpenStack Training (for in-‐‑‒house) 2015年年 •OpenStack Training (for external) •RD-‐‑‒Cloud-‐‑‒2 service (Juno/in-‐‑‒house) •Join in OpenStack Days Tokyo 2015 •Have a Session in OpenStack Summit Tokyo •Start hosting customer's OpenStack cloud(Kilo) 2016年年 Bit-‐‑‒isle's OpenStack History 4
- 5. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Our Ceph Environment and History • Our Ceph history is almost same as OpenStack history since 2013. Each environment has POC term 5 2013 2014 2015 2016 RD-‐‑‒Cloud-‐‑‒1 for Develop (for POC/Testing) (OpenStack Havana & Ceph Dumpling ) RD-‐‑‒Cloud-‐‑‒2 – (for Staging & Develop) (OpenStack Juno & Ceph Giant) Customer-‐‑‒Cloud for Production (OpenStack Kilo & Ceph Hammer)
- 6. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Table of Contents •Why do we chose and use Ceph? •The 3 environment of in operation •Things that I am thinking now 6
- 7. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Why do we choose and use Ceph? 7
- 8. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Why OpenStack users need Ceph storage • Ceph is popular enough in OpenStack Community. 8
- 9. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved This is why we need Ceph •We needs a storage with no IOPS saturation ‣LVM Cinder backend is saturated by one action. •We do not have budget and Storage appliance but we have some servers. •We also do not have engineers enough to operate troublesome storage. •However, we want to provide OpenStack environment for our employees. 9
- 10. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved First POC •Point ‣Deploy, Basic Storage Features ‣Cooperation with OpenStack ‣Fault tolerance 種別 台数 スペック 特記事項 Ceph OSD 3 CPU 12 Core Memory 96GB 512GB HDD×4 10Gnic ×2 HDD used to -‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒-‐‑‒ For journal ×1 For OSD ×3 MON MDS 1 CPU 12 Core Memory 96GB 10Gnic ×2 Put Ceph-‐‑‒mds and cinder-‐‑‒ volume on same node Openstack Volume & image Compute 2 CPU 12 Core Memory 96GB 10Gnic ×2 Ceph Public NW （10Gbps) MON.1 OSD.0 OSD.3 OSD.6 Nova Compute 1 Nova Compute 2 Volume MDS Image Ceph Cluster NW 10G接続 Openstack 内部接続 （100Mbps) MON.2 OSD.1 OSD.4 OSD.7 MON.2 OSD.2 OSD.5 OSD.8
- 11. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Result of First POC •Deploy / Fine (by ceph-‐‑‒deploy) •Fault tolerance / Very Good (Required 10GbE NW) •Cooperation with OpenStack / Very Good •Operationability / Very Good (No rebalance operation) •Performance ‣Parallelism / Very Good ‣Top Performance / Not expected ‣This result is enough to use Ceph for our test environment. 11
- 12. About 3 environment of in operation 12
- 13. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved MON.X OSD.0 HDD 0.5TB JOURNAL (SATA HDD) OSD.1 HDD 0.5TB OSD.2 HDD 0.5TB RD-‐‑‒Cloud-‐‑‒1 – (for Develop) -‐‑‒ outline ‣OpenStack Havana & Ceph Dumpling ‣12 OpenStack Compute nodes ‣3 nodes Ceph Cluster all 4.5TB / 2replica – Effective 2.2TB ‣Full HDD / 10G NW for cluster only ‣Hand made(ceph-‐‑‒deploy) on Ubuntu13.10 13 Ceph Public NW (1Gbps) Ceph Cluster NW (10Gbps) OpensStack Compute 1OpensStack Compute 2OpensStack Compute 12 OpenStack ControllerOpenStack ControllerOpenStack Controller MON.X OSD.0 HDD 0.5TB JOURNAL (SATA HDD) OSD.1 HDD 0.5TB OSD.2 HDD 0.5TB MON.X OSD.0 HDD 0.5TB JOURNAL (SATA HDD) OSD.1 HDD 0.5TB OSD.2 HDD 0.5TB
- 14. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved RD-‐‑‒Cloud-‐‑‒1 – (for Develop) – Point 14 •Design ‣Do not need 10GbE NIC/Port , when we add Compute node. ‣Storage performance is not so important , but itʼ’s important not stopping. ‣It is also important that no data lost. •Result ‣No data lost ‣No IO stop ‣Almost operation free ‣Performance is not good.
- 15. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved RD-‐‑‒Cloud-‐‑‒2 – (for Staging & Develop) -‐‑‒ outline ‣OpenStack Juno & Ceph Giant ‣10 OpenStack Compute nodes ‣5 nodes Ceph Cluster all 15TB / 3 replica – Effective 5 TB ‣SATA SSD for for journal /OSD =HDD / 10G NW for cluster only ‣Hand made(ceph-‐‑‒deploy) on CentOS 7 15 Ceph Public NW (1Gbps) Ceph Cluster NW (10Gbps) OpensStack Compute 1OpensStack Compute 2OpensStack Compute 10 OpenStack ControllerOpenStack ControllerOpenStack Controller MON.X OSD.0 (HDD 1TB) OSD.1 (HDD 1TB) OSD.2 (HDD 1TB) JOURNAL (SATA SSD) MON.X OSD.0 (HDD 1TB) OSD.1 (HDD 1TB) OSD.2 (HDD 1TB) JOURNAL (SATA SSD) MON.X OSD.0 (HDD 1TB) OSD.1 (HDD 1TB) OSD.2 (HDD 1TB) JOURNAL (SATA SSD) MON.X OSD.0 (HDD 1TB) OSD.1 (HDD 1TB) OSD.2 (HDD 1TB) JOURNAL (SATA SSD) MON.X OSD.0 (HDD 1TB) OSD.1 (HDD 1TB) OSD.2 (HDD 1TB) JOURNAL (SATA SSD)
- 16. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved RD-‐‑‒Cloud-‐‑‒2 – (for Staging & Develop) -‐‑‒ point •Design ‣Performance will be better than RD-‐‑‒Cloud-‐‑‒1. •Result ‣IOPS is bellow 16 サイズ seq-‐‑‒read seq-‐‑‒write rand-‐‑‒read rand-‐‑‒wite 4k 13721 4243 13538 1063 8k 12635 3701 12294 1009 16k 6830 2827 6831 877 32k 3516 2135 3431 655
- 17. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Customerʼ’s Production -‐‑‒ outline ‣OpenStack Kilo & Ceph Hammer ‣30 OpenStack Compute nodes ‣30 Ceph Cluster nodes all 180TB / 3 replica – Effective 60 TB ‣Both of that deployed on same physical server ‣Jounal PCIe SSD / 40G NW for All ‣Deployed from Juju/MAAS 17
- 18. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved •Network device ‣1 × 40G Network for All Service ‣1 × 1G Network for IPMI •OpenStack Nodes ‣1 Control and NW ‣5 Compute and Storage •Deployment Node ‣Juju /Maas Server Customerʼ’s Production -‐‑‒ Basic Structure Compute&Storage 30 node CTRL/NW 3 node Deployment Router CTRL/NW Compute/OSD Compute/OSD Compute/OSD MAAS/Juju OpenStack Segment IPMI Segment Compute/OSD Compute/OSD
- 19. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved •Resource Server 40GB/Ethernet Hyper Visor Customerʼ’s Production – detail KVM VM VM VM VM VM VM KVM VM VM VM VM VM VM KVM VM VM VM VM VM VM KVM VM VM VM VM VM VM Ceph Cluster OSD (HDD) Journal (PCIe SSD) Server fundamentals Server HP ProLiant DL360 Gen9 CPU E5-‐‑‒2690v3 2.60GHz 1P/12C * 2 HDD SAS 1TB HDD *2 RAID1 for OS SAS 1TB HDD *6 RAID0 for OSD Memory 96GB mem per node PCIeSSD Fusion-‐‑‒io iomemory 1.6TB * 1 for Journal 40Gbps NIC Mellanox ConnectX3-‐‑‒Pro OpenStack Components xxx xxx xxx xxx Ceph MON OSD (HDD) Journal (PCIe SSD) OSD (HDD) Journal (PCIe SSD) OSD (HDD) Journal (PCIe SSD) OpenStack Components xxx xxx xxx xxx Ceph MON OpenStack Components xxx xxx xxx xxx Ceph MON
- 20. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Deployed by Juju/MAAS ‣Deployed by Juju/MAAS • Same as other OpenStack Components • Parameters are set by juju charm (with Canonical support) -‐‑‒ reason -‐‑‒ ‣For avoiding to depend on individual skill. ‣For reducing operation cost, when HW failures.
- 21. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Customerʼ’s Production -‐‑‒ point •Design ‣Designed like Hyper Converged • For saving number of servers. •Performance ‣fio result is bellow(100 VM parallel summary) 21 ブロックサイズ SeqRead SeqWrite RandRead RandWrite 4k 333,286 43,216 211,394 31,121 8k 333,255 50,218 223,061 30,274 16k 295,515 46,719 220,171 17,791 32k 212,678 52,005 179,464 14,457 ブロックサイズ SeqRead SeqWrite RandRead RandWrite 4k 3,333 432 2,114 311 8k 3,333 502 2,231 303 16k 2,955 467 2,202 178 32k 2,127 520 1,795 145 1台当たり 100台合計
- 22. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Customerʼ’s Production -‐‑‒ problem •A problem occurred from concentration of 9000/19000 PGʼ’s deep-‐‑‒scrub. ‣VMʼ’s IO stopped temporarily •Temporarily 600 ksps = 300MB/sec • We made a script that do deep-‐‑‒scrub scheduled. 22
- 23. Things that I am thinking now 23
- 24. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved Current Worries •Capax ‣Ceph Cluster for Production is cheap enough, but… • It is premise that you already have a lot of servers and ssd and network device for over 10GbE. And also spaces(=racks and power). •Opex ‣Ceph Cluster does not need operation cost. but… • It is premise that hardware failure does not occur or Software troubles does not appearance. ‣If the customerʼ’s initial size is enough to use other storage appliance, should we recommend using ceph to them? 24
- 25. Copyright © 2016 Bit-‐‑‒isle Equinix Inc. All Rights Reserved My wishes •Start small, scale limitless ‣Both of Size and IOPS ‣Both of Capex(especially Initial Cost) and Opex • Appliance storage it too experience for startup users. • But it's necessary after when they will become big. •Separate effect of ceph process and kvm ‣For hyper-‐‑‒converged use case. •Ceph should be more popular. ‣It is not easy to operate ceph storage for our members currently. •Ceph Storage Status monitoring services 25
- 26. ビットアイル・エクイニクス株式会社 TEL 03-‐‑‒5805-‐‑‒8154 FAX 03-‐‑‒3474-‐‑‒5538 URL http://www.bit-‐‑‒isle.jp/ 26