"Session ID: HKG18-112 Session Name: HKG18-112 - Accelerate Ceph by SPDK on AArch64 Speaker: Jun He Track: Enterprise ★ Session Summary ★ Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability. Ceph can supply block storage service within Clould production. SPDK (Storage Performance Development Kit) is a technology to improve the performance of nonvolatile media (NVMe SSD) and networking. The paper will introduce how to accelerate Ceph by SPDK on AArch64 platform. SPDK can improve the IOPs of Ceph cluster, reduce the data transmission latency and reduce the CPU cost. It will make Ceph cluster profit from multi-core advantage on AArch64 platform. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/hkg18/hkg18-112/ Presentation: http://connect.linaro.org.s3.amazonaws.com/hkg18/presentations/hkg18-112.pdf Video: http://connect.linaro.org.s3.amazonaws.com/hkg18/videos/hkg18-112.mp4 --------------------------------------------------- ★ Event Details ★ Linaro Connect Hong Kong 2018 (HKG18) 19-23 March 2018 Regal Airport Hotel Hong Kong --------------------------------------------------- Keyword: Enterprise 'http://www.linaro.org' 'http://connect.linaro.org' --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961"

1. © 2018 Arm Limited
Jun He, jun.he@arm.com
Tone Zhang, tone.zhang@arm.com
• 2018/3/9
Accelerate Ceph
By SPDK on
AArch64

2. © 2018 Arm Limited
SPDK

3. 3 © 2018 Arm Limited
SPDK
What’s SPDK?
• Storage Performance Development Kit
• A set of tools and libraries to create high
performance, scalable, user mode
storage applications
• Designed for new storage HW devices
(NVMe). Can achieve millions of IOPS
per core. Better tail latency.
Architecture diagram

4. 4 © 2018 Arm Limited
SPDK on AArch64
• Several ARM related patches are merged
• Memory_barrier
• VA address space
• 17.10 release verified
• Kernel: 4.11, 48bit/42bit VA, 4KB pagesize
• UIO/VFIO

5. 5 © 2018 Arm Limited
SPDK Performance on AArch64
• SPDK perf
• UIO/4K pagesize
• FIO
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
1800000
2000000
IOPS Bandwidth Latency
RandRead
Kernel SPDK
0
200000
400000
600000
800000
1000000
1200000
IOPS Bandwidth Latency
RandWrite
Kernel SPDK
FIO configuration: direct=1, bs=4096, rwmixread=50, iodepth=32, ramp=30s, run_time=180s, jobs=1

6. 6 © 2018 Arm Limited
SPDK Performance on AArch64
• FIO
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
IOPS Bandwidth Latency
RandRW - read
Kernel SPDK
0
50000
100000
150000
200000
250000
300000
350000
IOPS Bandwidth Latency
RandRW - write
Kernel SPDK
FIO configuration: direct=1, bs=4096, rwmixread=50, iodepth=32, ramp=30s, run_time=180s, jobs=1

7. 7 © 2018 Arm Limited
What’s the next?
• Optimization with ASIMD and Crypto extensions
• Tuning with different page-size(16KB/64KB)
• Cache strategy improvement for better read/write performance

8. © 2018 Arm Limited
Ceph

9. 9 © 2018 Arm Limited
Ceph
What’s Ceph?
• Ceph is a unified, distributed storage
system designed for excellent
performance, reliability and scalability
• Ceph can supply following services
• Object storage
• Block storage
• File system
• The backend storage types
• FileStore
• BlueStore

10. 10 © 2018 Arm Limited
BlueStore
BlueStore is a new storage backend for Ceph.
• Full data built-in compression
• Full data checksum
• Boasts better performance
• Get rid of file system, and write all data to RAW
device via asynchronous libaio infrastructure

11. 11 © 2018 Arm Limited
Ceph on AArch64
• Has already been integrated with OpenStack
• Has been validated and released by Linaro SDI team
• Has committed many patches to fix the functional faults and improve the performance
• Has validated “Ceph + SPDK” on top of NVMe devices
• Tuned Ceph performance on AArch64

12. 12 © 2018 Arm Limited
Ceph + SPDK on AArch64
• Dependencies
• NVMe device
• SPDK/DPDK
• BlueStore
• Enabled SPDK in Ceph on AArch64
• Extended virtual address map bits from 47 to 48 bits in DPDK

13. 13 © 2018 Arm Limited
Ceph + SPDK on AArch64
BlueStore is a new storage backend for Ceph.
• BlueStore can utilize SPDK
• Replace kernel driver with SPDK user
space NVMe driver
• Abstract BlockDevice on top of SPDK
NVMe driver
NVMe device
Kernel NVMe driver
BlueFS
BlueRocksENV
RocksDB
metadata
NVMe device
SPDK NVMe driver
BlueFS
BlueRocksENV
RocksDB
metadata
FileStore BlueStore
CEPH RBD Service
BlockDevice
CEPH Object Service CEPHFS Service

14. 14 © 2018 Arm Limited
Ceph + SPDK Performance test on AArch64
Test case
• Ceph cluster
• Two OSD, one MON, no MDS and RGW
• One NVMe card per OSD
• CPU: 2.4GHz multi-core
• Client
• CPU: 2.0GHz multi-core
• Test tool
• Fio (v2.2.10)
• Test case:
• Sequential write with different block_size (4KB,
8KB and 16KB)
• 1 and 2 fio streams
Ceph cluster
OSD1 OSD2
MON
Client

15. 15 © 2018 Arm Limited
Write performance result
1 stream
1000
1500
2000
2500
3000
3500
IOPS - 4KB
1coe 2cores 4cores
Kernel NVMe
1core 2cores 4cores
SPDK
1000
1500
2000
2500
3000
3500
IOPS - 8KB
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
SPDK
1000
1500
2000
2500
3000
3500
IOPS - 16KB
100
150
200
250
300
350
latency - 4KB
msec
100
150
200
250
300
350
latency - 8KB
msec
100
150
200
250
300
350
latency - 16KB
msec
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
SPDK
1core 2cores 4cores
SPDK
1core 2cores 4cores
SPDK
1core 2cores 4cores
SPDK
1 fio stream, FIO configuration: bs=4K/8K/16K, rw=write, iodepth=384, run_time=40s, jobs=1, ioengine=rbd

16. 16 © 2018 Arm Limited
Write performance result
2 streams
1000
1500
2000
2500
3000
3500
4000
IOPS - 4K
1000
1500
2000
2500
3000
3500
4000
IOPS - 8K
1000
1500
2000
2500
3000
3500
4000
IOPS - 16K
1core 2cores 4cores
SPDK
1core 2cores 4cores
SPDK
1core 2cores 4cores
SPDK
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
Kernel NVMe
1core 2cores 4cores
Kernel NVMe
2 fio streams, FIO configuration: bs=4K/8K/16K, rw=write, iodepth=384, run_time=40s, jobs=1, ioengine=rbd

17. 17 © 2018 Arm Limited
Performance improvement
SPDK accelerated Ceph in below:
• More IOPS
• Lower latency
• Linear scaling associate with the number of CPU cores

18. 18 © 2018 Arm Limited
What’s the next?
• Continue improving Ceph performance on top of SPDK
• Enable NVMe-OF and RDMA
• Enable zero-copy in Ceph
• Simplify the locking in Ceph to improve the OSD daemon performance
• Switch PAGE_SIZE to 16KB and 64KB to improve the memory performance
• Modify NVMEDEVICE to improve its performance associate with different PAGE_SIZE

19. 1919
Thank You
Danke
Merci
谢谢
ありがとう
Gracias
Kiitos
감사합니다
धन्यवाद
‫תודה‬
© 2018 Arm Limited