Storage is one of the main 3 pillars of any data center, along with compute and networking. OpenStack provides flexibility and automation for storage provisioning, no matter if one uses iSCSI integrated with Cinder or Ceph for block and object storage. But what about performance? How can one enjoy storage flexibility without compromising on state of the art, low-latency, high-throughput storage, that is required by today’s applications? In this session, we will present three storage solutions for OpenStack and how they can be accelerated natively in OpenStack with Remote Direct memory Access (RDMA) technology. Join us to learn how RDMA boosts storage performance in the cloud.

1. © 2017 Mellanox Technologies 1
Aviram Bar-Haim, SW Manager, Cloud solutions
Openstack Israel 2017
Born to be fast!
Accelerating Openstack storage with RDMA

2. © 2017 Mellanox Technologies 2
A World of Exponential Data Growth
▪ More than a 1.94B active Facebook users > 1/4 of the earth
population. (Wikipedia)
▪ Every 1 second, 1 Hour of video is uploaded to YouTube, which
is a century of videos every 10 days. (YouTube)
▪ More than 90% of today’s data created in 2 years. (IBM, 2016)
▪ In 2017, 8.4B connected devices will be in use worldwide - 1B
more than humans. (Garthner, 2017)

3. © 2017 Mellanox Technologies 3
The Future Depends on the Fastest Interconnects
10Gb/s 200Gb/s
1Gb/
s
Data center standards are evolving in order to meet services demand

4. © 2017 Mellanox Technologies 4
Some Ways to Accelerate Cloud Storage Solutions
Faster Wires
Faster Architecture
Faster Protocols
Faster Storage

5. © 2017 Mellanox Technologies 5
TCP/UDP -> Remote Direct Memory Access (RDMA)
RDMA over IB / Ethernet –
200Gb/s

6. © 2017 Mellanox Technologies 6
RDMA benefits example
▪ Without RDMA
• 5.7 GB/s throughput
• 20-26% CPU utilization
• 4 cores 100% consumed by moving data
▪ With Hardware RDMA
• 11.1 GB/s throughput at half the latency
• 13-14% CPU utilization
• More CPU power for applications, better ROI
x x x x
100GbE With CPU Onload 100 GbE With Network Offload
2X Better Bandwidth
Half the Latency
33% Lower CPU
See the demo (over Microsoft): https://www.youtube.com/watch?v=u8ZYhUjSUoI
Network protocols in Top500 HPC clusters

7. © 2017 Mellanox Technologies 7
Memory
Storage RDMA Products by Market Categories
▪ Pure Bandwidth – up to 100Gb/s
▪ Flash over Block, File & Object
▪ RDMA
• InfiniBand & RoCE
• iSER
• SMB Direct, NFSoRDMA
• Ceph over RDMA
▪ Non-Volatile Memory (NVM)
• NVMe over Fabrics
• PMf (3D-XPoint)
Block
File
Object
SMB (CIFS)
NFS
iSCSI
Ceph
RDMA
Swift
PMf
iSERSMB Direct
NFSoRDMA
Ceph
over
RDMA
NVMe-oF

8. © 2017 Mellanox Technologies 8
iSER
iSCSI Extensions for RDMA

9. © 2017 Mellanox Technologies 9
What is iSER?
▪ iSER – iSCSI Extensions for RDMA
▪ Maps the iSCSI protocol over RDMA fabrics
▪ Leverages iSCSI management infrastructure
▪ The transport layer iSER and/or iSCSI/TCP are transparent.
▪ An initiator Read is actually a target Write operation: Ethernet (RoCE) / InfiniBand
SCSI Application Layer
iSCSI
iSER
iSCSIInitiator
iSER
HCA
HCA
iSERTarget
TargetStorage
Send_Control (SCSI
Read Cmd)
RDMA Write for Data
Send_Control + Buffer
advertisement
Control_Notify
Data_Put
(Data-In PDU)
for Read
Control_Notify
Send_Control (SCSI
Response)
Send_Control (status,
sense data)

10. © 2017 Mellanox Technologies 10
Performance

11. © 2017 Mellanox Technologies 11
▪ iSER is fully integrated with Openstack since Havana release. no additional software is required!
▪ Supported with LVM volume driver and others, using TGT or LIO targets.
iSER – The Fastest Storage Solution in OpenStack
Hypervisor (KVM)
OS
VM
OS
VM
OS
VM
Adapter
Open-iSCSI
(iscsiadm)
Compute Servers
Switching Fabric
iSER Target
(TGT/LIO)
Adapter Local Disks
RDMA Cache
Storage Servers
Cinder driver
iSER provides up to ~x5 More Bandwidth, ~x5 lower I/O latency, and lower CPU%
LIO:
TGT
:

12. © 2017 Mellanox Technologies 12
Conclusion: iSER Is the FASTEST Networked SCSI Transport
▪ What it is: iSCSI With RDMA Transport
• Runs over Ethernet or InfiniBand at 10, 25, 40, 50, 56, 100Gb/s +
• Works with all applications that support SCSI/iSCSI
▪ Benefits
• Most bandwidth, Highest IOPs, Lowest Latency, Lowest CPU utilization
• iSCSI storage features, management and tools (security, HA, discovery...)
• Faster than iSCSI, FC, FCoE; Easier to manage than SRP
• Fully integrated with Openstack!
▪ Ideal For
• SSD
• Latency-sensitive workloads; Small, random I/O
• Databases, Virtualization

13. © 2017 Mellanox Technologies 13
NVMeOF
Non-Volatile Memory Express Over Fabrics

14. © 2017 Mellanox Technologies 14
Faster Storage – Flash/SSD Becomes Super Popular
Today

15. © 2017 Mellanox Technologies 15
Storage Performance Dramatically Increases = Network load

16. © 2017 Mellanox Technologies 16
NVMe & NVMeOF
▪ What is NVME
• Non-volatile memory express, is a specification that allows a
solid state drive to make effective use of a high-speed
Peripheral Component Interconnect Express (PCIe) bus in a
computer.
▪ What is NVMEoF
• A common architecture that supports a range of storage
networking fabrics (specifically – RDMA) for NVMe block
storage protocol over a storage networking fabric.

17. © 2017 Mellanox Technologies 17
▪ NVMe SSD
BW ~22/10 Gbps
Latency ~20 µs
IOPs (4KB) ~450/250K per device
Latency effect:
Bandwidth effect:
The bottleneck is moving from the storage to the network
▪ SATA HDD
BW ~0.4 Gbps
Latency ~6000 µs
IOPs (4KB) ~180 per device
Latency effect:
Bandwidth effect:
>250K IOPs
20
180 IOPs
RDMA based network latency
TCP based network latency
100
GE
40 GE
10 GE
10 GE is not enough to
sustain 1 NVMe!
10 GE is enough to
sustain 24 HDDs!
100
GE
40 GE
10 GE

18. © 2017 Mellanox Technologies 18
NVMeOF Openstack activity
▪ Cinder blueprint accepted for Ocata
▪ Two main activities:
1. Supporting Linux NVMe target (nvmetcli) with LVM
- A draft implementation is ready
- Requires support in: Cinder (LVM driver target), OS-Brick, Nova
- ETA: Pike
2. Supporting SPDK Cinder driver and target:
- A draft implementation created for Barcelona
- Working on support for snapshots and CI
- Requires support in: Cinder (SPDK driver), OS-Brick, Nova
- ETA: Queens
Hypervisor (KVM)
OS
VM
OS
VM
OS
VM
Adapter
nvmecli
Compute Servers
nvmetcli
Adapter Local SSDs
RDMA Cache
Storage Servers
Cinder LVM driver
Hypervisor (KVM)
OS
VM
OS
VM
OS
VM
Adapter
nvmecli
Compute Servers
Adapter
RDMA Cache
Storage Servers
Cinder NVMe
driver
Local SSDs
1
2

19. © 2017 Mellanox Technologies 19
Ceph over RDMA

20. © 2017 Mellanox Technologies 20
CEPH
▪ Open source SDS, maintained by Red Hat (Started by InkTank in April 2014).
▪ Capable of block, object, and file storage, in a scale-out, distributed architecture.
▪ The most common Cinder backend (>65% of all deployments).
▪ Connects to Nova, glance and swift as well.
▪ Ceph Provides:
• Block device service (RBD)
• Object Store (RADOSGW)
• File System (CEPH FS)

21. © 2017 Mellanox Technologies 21
CEPH major Issues
▪ Ceph generates a lot of traffic and calculations
• data replications
• Self healing
• Self monitoring
▪ Ceph cluster components communicate using TCP
• Leads to a high CPU usage compared to other storage solutions.

22. © 2017 Mellanox Technologies 22
RDMA with Ceph
▪ Mellanox, Xsky, Samsung, SanDisk and Red Hat contributors
▪ Ceph RDMA Beta Included in Ceph Hammer from June 2015
• RDMA for both public and cluster networks; for block and object storage
• High performance but limited scalability
• Easy Openstack integration – 2 lines in /etc/ceph.conf
▪ Updates pushed upstream in Ceph community constantly
• Focus on increased scale and performance
How to guide: https://community.mellanox.com/docs/DOC-2693

23. © 2017 Mellanox Technologies 23
New Ethernet Speeds: 25 and 50 vs. 10 GbE
▪ Aggregate performance of 4 Ceph servers
• 25GbE has 92% more bandwidth than 10GbE
• 25GbE has 86% more IOPS than 10GbE
▪ Internet search results seem to recommend one 10GbE NIC for each ~15 HDDs in an OSD
• Mirantis, Red Hat, Supermicro, etc.

24. © 2017 Mellanox Technologies 24
Recovery backend traffic 1x times lost data
The Need For Speed During Data Recovery (Replication)
▪ Example - Time to recover
• Net networking time to move data
• 10GbE
- 2TB - 26min.
- 20TB - 4.4hrs
- 200TB - 44hrs
• 25GbE
- 2TB - 10.6min.
- 20TB - 1.8hrs
- 200TB - 17.6hrs
• 40GbE
- 2TB - 6.6min.
- 20TB - 1.1hrs
- 200TB - 11.1hrs
• 100GbE
- 2TB - 2.6min.
- 20TB - 26.4min
- 200TB - 4.4hrs
Client OSD OSD OSD OSD
Read
Read Reply

25. © 2017 Mellanox Technologies 25
Improving Ceph’s Network Architecture
40GbE
10GbE
25, 40, 50, 100GbE

26. © 2017 Mellanox Technologies 26
Conclusions

27. © 2017 Mellanox Technologies 27
Summary
▪ Faster storage solutions leads to a network load – 10Gbps is not enough!
• Especially if using >15 HDD or SSDs
▪ iSER is the fastest block storage solution that is fully integrated in Openstack
▪ SSD NVMe is the best available storage, requires RDMA’s latency to enjoy its benefits
• Openstack integration is in progress
▪ Few ways to accelerate Ceph
• Use flash SSDs over a fast Cluster network (>10 Gbps)
• Use RDMA in Cluster and Public networks

28. © 2017 Mellanox Technologies 28
Thank YouThank
Youaviramb@mellanox.com