Our first 5 years with Ceph
Lars Marowsky-Brée
lmb@suse.com
Distinguished Engineer, Architect Software-Defined-Storage
2
Welcome!
33
Enterprise Data Storage Challenges
4
Common Limitations of Traditional
Enterprise Storage
Unable to scale and
manage data
growth
Expensive Won’t extend to
the software-defined
data center
$
PRESENT
FUTURE
£
€
$
5
Data Continues to Grow Very Fast
Medical Data
EmailsVideos
Mobile Data IoT Data
Transactional Data
6
Data Protection Problem is Getting
Compounded
• Inability to keep enough data online
• Not being able to recover lost data fast enough
• Migrations to larger disk or dedupe appliances every year
Increasing volumes of data compound the data protection problem
*SUSE Software Defined Storage Study 2016
Cost of storage
Performance / availability
Challenges with back up, disaster recovery and archiving
Increasing volumes of data
Security and data governance
45%
46%
46%
54%
56%
Customer Facing Numerous Backup Challenges:
Top 5 Data Management Challenges*
7
Open Source is the only sustainable option
88
About SUSE Enterprise Storage
9
SUSE’s history with Ceph
August 2012 SUSE Cloud 1 Argonaut
February 2015 SUSE Enterprise Storage 1 Firefly
October 2015 SUSE Enterprise Storage 2 Hammer
June 2016 SUSE Enterprise Storage 3 Jewel
October 2016 openATTIC team joins SUSE!
November 2016 SUSE Enterprise Storage 4 Jewel
October 2017 SUSE Enterprise Storage 5 Luminous
Later 2018 SUSE Enterprise Storage 6 Mimic
10
●
Curate open source solutions – projects and features
●
Track, test, and manage software dependencies
●
Incorporate and provide patches and backports
●
Advise customers on recommended practices and hardware
●
Represent customers and partners in the community
●
Bridge the worlds of community and IHVs/ISVs
●
Provide high quality support
Vendor value add
11
●
Strong contributor to Ceph community & Ceph Advisory Board
●
True Open Source - “Upstream first”
●
iSCSI with Multipathing/VMware support
●
First supported Ceph distribution for ARM64
●
First to support CephFS for production deployments
●
Salt-based orchestration for upgrade and FileStore/BlueStore migration
●
Lead on openATTIC, Prometheus & Grafana, now merged into core
Ceph management functionality
Major SUSE contributions
12
Monitoring with openATTIC
13
Management with openATTIC
14
openATTIC merging into the Ceph dashboard
1515
Strong industry partnerships
16
Use Case Focused Solutions
Backup to Disk Solution Compliant
Archives
SAP HANA
Storage Solution
Appliance
HPC Archives
Certified RAs
Cloud
SOC SES
IoT
Configuration CSPs +
SES
Mode1
Customers
Mode2
Customers
Example SUSE Enterprise Storage Partners
17
Growth of our Ceph deployments
1818
Industry validation
PUBLIC
Marc Koderer, SAP
March 2018
Case Study: Ceph at SAP
How to Build a Cattle Cloud
20PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Integraton
Environment
SAP Cloud Platform, private edition – High Level
Security Platorm
Services
SAP Cloud Platorm Private Cloud
Data & Storage
Foundaton
XS Advanced Development
IaaSSAP HANA
Database
Hardware / Bare Metal
Applicaton / IoT
ERP
21PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Traditional applications vs cloud native applications (pets vs cattle)
DB
Standby
DB
Active
App App
Centralized
Storage
Characteristics:
• Runs on bare metal and/or VM’s
• HW failure is exceptional
• Centralized instead of distributed
monolithic payloads
Characteristics:
• Runs on VMs or in Containers
• HW failure can happen all the time
• Highly distributed
Distributed Storage / SDS
DB2DB1
Service Service Service…
DBn…
Distributed/scale-out payloads
22PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
AZ1 AZ2 AZ3
Private cloud architecture
23PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
AZ1 AZ2 AZ3
OpenStack Control Plane
C
N
C
N
C
N
C
N
C
N
C
N
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
Private cloud architecture
24PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
AZ1 AZ2 AZ3
Ceph
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
OpenStack Control Plane
C
N
C
N
C
N
C
N
C
N
C
N
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
H
V
Private cloud architecture
25PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Ceph in detail
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
spine01 spine02 spine03 spine04
leaf02a leaf02b leaf03a leaf03bleaf01a leaf01b
26PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Ceph in detail
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
S
T
spine01 spine02 spine03 spine04
leaf02a leaf02b leaf03a leaf03bleaf01a leaf01b
• Hardware characteristics
• 108 storage nodes
• 24 disk per osd node; 2 NVMEs
• 4x 25 Gbit (2x 25 Gbit bond frontend; 2x 25 Gbit bond backend)
• Spine/leaf network architecture across all AZs
• RBD and Radosgw usage
• The Ceph cluster is designed to compensate the outage of one Availability Zone
• Max performance ~60 GB/sec (writes; 4MB)
27PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
Ceph in detail: RGW
Loadbalancer
VM VM VM VM VM
mon1 mon2 mon3
osd1 osd2 osd3 osdn…
mon4 mon5
rgw1 rgw2 rgw3 rgw30…
28PUBLIC© 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
RGW stress test
# of RGW
(6 vcpus)
Max
concurrent
connection
Method Throughput
1 512 put 250 MB/sec
1 512 get 300 MB/sec
3 1536 put 700 MB/sec
3 1536 get 900 MB/sec
9 4608 put 1950 MB/sec
9 4608 get 2100 MB/sec
2929
The next 5 years
30
SUSE Enterprise Storage 6 – based on Mimic
•Incorporates Ceph Mimic
•Based on SUSE Linux Enterprise 15
•Improve interoperability
•Internationalization and localization
•Improved scale-out user experience
•Eventing and alerting
•Metric reporting and telemetry
31
Hyper-converged infrastructure
32
SUSE Container as a Service Platform
33
openATTIC <3 Ceph’s Dashboard
Provide a better user experience
Make complex tasks easier
Assist users along the way of sizing,
deploying, and managing a Ceph cluster
34
Machine Learning and AI
35
Ceph? Not if. When.
Open source cloud operating systems and software-defined storage
platforms are based on the Linux operating system.
SUSE is a Linux OS pioneer and successful software vendor with
thousands of installations. Customers should expect to receive
nothing less than expert support for their software-based storage.
Learn more at suse.com/storage/
Ceph's journey at SUSE

Ceph's journey at SUSE

  • 1.
    Our first 5years with Ceph Lars Marowsky-Brée lmb@suse.com Distinguished Engineer, Architect Software-Defined-Storage
  • 2.
  • 3.
  • 4.
    4 Common Limitations ofTraditional Enterprise Storage Unable to scale and manage data growth Expensive Won’t extend to the software-defined data center $ PRESENT FUTURE £ € $
  • 5.
    5 Data Continues toGrow Very Fast Medical Data EmailsVideos Mobile Data IoT Data Transactional Data
  • 6.
    6 Data Protection Problemis Getting Compounded • Inability to keep enough data online • Not being able to recover lost data fast enough • Migrations to larger disk or dedupe appliances every year Increasing volumes of data compound the data protection problem *SUSE Software Defined Storage Study 2016 Cost of storage Performance / availability Challenges with back up, disaster recovery and archiving Increasing volumes of data Security and data governance 45% 46% 46% 54% 56% Customer Facing Numerous Backup Challenges: Top 5 Data Management Challenges*
  • 7.
    7 Open Source isthe only sustainable option
  • 8.
  • 9.
    9 SUSE’s history withCeph August 2012 SUSE Cloud 1 Argonaut February 2015 SUSE Enterprise Storage 1 Firefly October 2015 SUSE Enterprise Storage 2 Hammer June 2016 SUSE Enterprise Storage 3 Jewel October 2016 openATTIC team joins SUSE! November 2016 SUSE Enterprise Storage 4 Jewel October 2017 SUSE Enterprise Storage 5 Luminous Later 2018 SUSE Enterprise Storage 6 Mimic
  • 10.
    10 ● Curate open sourcesolutions – projects and features ● Track, test, and manage software dependencies ● Incorporate and provide patches and backports ● Advise customers on recommended practices and hardware ● Represent customers and partners in the community ● Bridge the worlds of community and IHVs/ISVs ● Provide high quality support Vendor value add
  • 11.
    11 ● Strong contributor toCeph community & Ceph Advisory Board ● True Open Source - “Upstream first” ● iSCSI with Multipathing/VMware support ● First supported Ceph distribution for ARM64 ● First to support CephFS for production deployments ● Salt-based orchestration for upgrade and FileStore/BlueStore migration ● Lead on openATTIC, Prometheus & Grafana, now merged into core Ceph management functionality Major SUSE contributions
  • 12.
  • 13.
  • 14.
    14 openATTIC merging intothe Ceph dashboard
  • 15.
  • 16.
    16 Use Case FocusedSolutions Backup to Disk Solution Compliant Archives SAP HANA Storage Solution Appliance HPC Archives Certified RAs Cloud SOC SES IoT Configuration CSPs + SES Mode1 Customers Mode2 Customers Example SUSE Enterprise Storage Partners
  • 17.
    17 Growth of ourCeph deployments
  • 18.
  • 19.
    PUBLIC Marc Koderer, SAP March2018 Case Study: Ceph at SAP How to Build a Cattle Cloud
  • 20.
    20PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ Integraton Environment SAP Cloud Platform, private edition – High Level Security Platorm Services SAP Cloud Platorm Private Cloud Data & Storage Foundaton XS Advanced Development IaaSSAP HANA Database Hardware / Bare Metal Applicaton / IoT ERP
  • 21.
    21PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ Traditional applications vs cloud native applications (pets vs cattle) DB Standby DB Active App App Centralized Storage Characteristics: • Runs on bare metal and/or VM’s • HW failure is exceptional • Centralized instead of distributed monolithic payloads Characteristics: • Runs on VMs or in Containers • HW failure can happen all the time • Highly distributed Distributed Storage / SDS DB2DB1 Service Service Service… DBn… Distributed/scale-out payloads
  • 22.
    22PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ AZ1 AZ2 AZ3 Private cloud architecture
  • 23.
    23PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ AZ1 AZ2 AZ3 OpenStack Control Plane C N C N C N C N C N C N H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V Private cloud architecture
  • 24.
    24PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ AZ1 AZ2 AZ3 Ceph S T S T S T S T S T S T S T S T S T OpenStack Control Plane C N C N C N C N C N C N H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V H V Private cloud architecture
  • 25.
    25PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ Ceph in detail S T S T S T S T S T S T S T S T S T spine01 spine02 spine03 spine04 leaf02a leaf02b leaf03a leaf03bleaf01a leaf01b
  • 26.
    26PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ Ceph in detail S T S T S T S T S T S T S T S T S T spine01 spine02 spine03 spine04 leaf02a leaf02b leaf03a leaf03bleaf01a leaf01b • Hardware characteristics • 108 storage nodes • 24 disk per osd node; 2 NVMEs • 4x 25 Gbit (2x 25 Gbit bond frontend; 2x 25 Gbit bond backend) • Spine/leaf network architecture across all AZs • RBD and Radosgw usage • The Ceph cluster is designed to compensate the outage of one Availability Zone • Max performance ~60 GB/sec (writes; 4MB)
  • 27.
    27PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ Ceph in detail: RGW Loadbalancer VM VM VM VM VM mon1 mon2 mon3 osd1 osd2 osd3 osdn… mon4 mon5 rgw1 rgw2 rgw3 rgw30…
  • 28.
    28PUBLIC© 2017 SAPSE or an SAP affiliate company. All rights reserved. ǀ RGW stress test # of RGW (6 vcpus) Max concurrent connection Method Throughput 1 512 put 250 MB/sec 1 512 get 300 MB/sec 3 1536 put 700 MB/sec 3 1536 get 900 MB/sec 9 4608 put 1950 MB/sec 9 4608 get 2100 MB/sec
  • 29.
  • 30.
    30 SUSE Enterprise Storage6 – based on Mimic •Incorporates Ceph Mimic •Based on SUSE Linux Enterprise 15 •Improve interoperability •Internationalization and localization •Improved scale-out user experience •Eventing and alerting •Metric reporting and telemetry
  • 31.
  • 32.
    32 SUSE Container asa Service Platform
  • 33.
    33 openATTIC <3 Ceph’sDashboard Provide a better user experience Make complex tasks easier Assist users along the way of sizing, deploying, and managing a Ceph cluster
  • 34.
  • 35.
    35 Ceph? Not if.When. Open source cloud operating systems and software-defined storage platforms are based on the Linux operating system. SUSE is a Linux OS pioneer and successful software vendor with thousands of installations. Customers should expect to receive nothing less than expert support for their software-based storage. Learn more at suse.com/storage/