Successfully reported this slideshow.

Ceph Tech Talk -- Ceph Benchmarking Tool

1

Share

1 of 23
1 of 23

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Ceph Tech Talk -- Ceph Benchmarking Tool

  1. 1. Ceph Benchmarking Tool (CBT) Kyle BaderCeph Tech Talk May 26, 2016
  2. 2. INTRO TO CBT
  3. 3. • Benchmarking framework written in python • Began as a engineering benchmark tool for upstream developlment • Adopted for downstream performance and sizing • Used by many people in Ceph community • Red Hat • Intel / Samsung / SanDisk • Quanta QCT / Supermicro / Dell WHAT IS IT?
  4. 4. CBT PERSONALITIES HEAD • CBT checkout • Key based authentication to all other hosts • Including itself.. • PDSH packages • Space to store results archives • YAML testplans
  5. 5. CBT PERSONALITIES CLIENT • Generates load against the SUT • Ceph admin keyring readable by cbt user • Needs loadgen tools installed • FIO • COSbench • Should be a VM for kvmrbdfio • Can be containerized (good for rbdfio)
  6. 6. CBT PERSONALITIES MON • Nodes to setup monitors on OSD • Nodes to setup OSDs
  7. 7. • RADOS Bench • FIO with RBD engine • FIO on KRBD on EXT4 • FIO on KVM (vdb) on EXT4 • COSBench for S3/Swift against RGW CBT BENCHMARKS
  8. 8. • Cluster creation ( optional, use_existing: true ) • Cache tier configuration • Replicated and Erasure coded pools • Collects monitoring information from every node • Collectl – cpu/disk/net/etc. CBT EXTRAS
  9. 9. • SSH Key on head • Pub key in all hosts authorized_keys (including head) • Ceph packages on all hosts • PDSH packages on all hosts (for pdcp) • Collectl installed on all hosts BASIC SETUP
  10. 10. • Test network beforehand, bad network easily impairs performance • All-to-All iperf • Check network routes, interfaces • Bonding • Switches should use 5-tuple-hashing for LACP • Nodes should use LACP xmit_hash_policy=layer3+4 TEST METHODOLOGY
  11. 11. • Use multiple iterations for micro benchmarks • Use client sweeps to establish point of contention / max throughput • Client sweeps should always start with X(1) ~ 1 client • Should have 4-6 different increments of clients • Eg. client1, client[1-2], client[1-3], client[1-4] TEST METHODOLOGY
  12. 12. Testplan Examples
  13. 13. CBT CLUSTER CONFIGURATION cluster: head: "ceph@head” clients: ["ceph@client"] osds: ["ceph@osd"] mons: ["ceph@mon"] osds_per_node: 1 fs: xfs mkfs_opts: -f -i size=2048 mount_opts: -o inode64,noatime,logbsize=256k conf_file: /etc/ceph.conf ceph.conf: /etc/ceph/cepf.conf iterations: 3 rebuild_every_test: False tmp_dir: "/tmp/cbt" pool_profiles: replicated: pg_size: 4096 pgp_size: 4096 replication: 'replicated'
  14. 14. CLIENT SWEEPS cluster: head: "ceph@head” clients: ["ceph@client1"] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, ”ceph@client3”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, “ceph@client3”,”ceph@client4”] osds: ["ceph@osd"] mons: ["ceph@mon"]
  15. 15. • Spawns RADOS bench processes on each client • Establish raw RADOS throughput • Works against replicated or EC pools RADOS BENCH benchmarks: radosbench: op_size: [ 4194304, 524288, 4096 ] write_only: False time: 300 concurrent_ops: [ 128 ] concurrent_procs: 1 use_existing: True pool_profile: replicated osd_ra: [256]
  16. 16. • Spawns FIO proccesses on each client • Uses RBD ioengine • Establish raw librbd performance • No VM / container setup required FIO WITH RBD IO ENGINE benchmarks: librbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] procs_per_volume: [ 1 ] volumes_per_client: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100 use_existing_volumes: true
  17. 17. • Maps KRBD volume to each client • Creates EXT4 filesystem on KRBD • Mounts filesystem • Spawns FIO process per client • Uses AIO IO Engine on filesystem • Client can be container or bare metal • Establishes KRBD performance potential FIO WITH KRBD ON EXT4 benchmarks: rbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
  18. 18. • Create KVM instances outside CBT • KVM instances listed as clients • Creates EXT4 filesystem on /dev/vdb • Mounts filesystem • Spanws FIO process per client • Uses AIO IO Engine • Establish RBD performance with QEMU IO susbsystems FIO WITH KVM (VDB) ON EXT4 benchmarks: kvmrbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
  19. 19. • Install COSBench on head/clients outside CBT • Install / Configure RGW outside CBT • Translates CBT YAML to COSBench XML • Runs COSBench COSBENCH benchmarks: cosbench: cosbench_dir: /root/0.4.1.0 cosbench_xml_dir: /home/ceph-admin/plugin/cbt/conf/cosbench/ controller: client01 auth: config: username=cosbench:operator;password=intel2012;url=… obj_size: [128KB] template: [default] mode: [write] ratio: [100] …. Example at cbt/docs/cosbench.README
  20. 20. Running CBT # Loop through each test plan for clients in $(seq 1 6);do cbt/cbt –archive=/tmp/${clients}-clients-results path/to/test.yaml done
  21. 21. ANALYZING DATA
  22. 22. • No robust tools for analysis • Nested archive directory based on YAML options • Archive/000000/Librbdfio/osd_ra-00000128… • Usually awk/grep/cut-fu to csv • Plot charts with gnplot, Excel, R ANALYZING DATA
  23. 23. THANK YOU!

×