Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ceph Tech Talk -- Ceph Benchmarking Tool

519 views

Published on

26 May presentation for Ceph Tech Talks.

http://ceph.com/ceph-tech-talks/

Published in: Technology
  • Be the first to comment

Ceph Tech Talk -- Ceph Benchmarking Tool

  1. 1. Ceph Benchmarking Tool (CBT) Kyle BaderCeph Tech Talk May 26, 2016
  2. 2. INTRO TO CBT
  3. 3. • Benchmarking framework written in python • Began as a engineering benchmark tool for upstream developlment • Adopted for downstream performance and sizing • Used by many people in Ceph community • Red Hat • Intel / Samsung / SanDisk • Quanta QCT / Supermicro / Dell WHAT IS IT?
  4. 4. CBT PERSONALITIES HEAD • CBT checkout • Key based authentication to all other hosts • Including itself.. • PDSH packages • Space to store results archives • YAML testplans
  5. 5. CBT PERSONALITIES CLIENT • Generates load against the SUT • Ceph admin keyring readable by cbt user • Needs loadgen tools installed • FIO • COSbench • Should be a VM for kvmrbdfio • Can be containerized (good for rbdfio)
  6. 6. CBT PERSONALITIES MON • Nodes to setup monitors on OSD • Nodes to setup OSDs
  7. 7. • RADOS Bench • FIO with RBD engine • FIO on KRBD on EXT4 • FIO on KVM (vdb) on EXT4 • COSBench for S3/Swift against RGW CBT BENCHMARKS
  8. 8. • Cluster creation ( optional, use_existing: true ) • Cache tier configuration • Replicated and Erasure coded pools • Collects monitoring information from every node • Collectl – cpu/disk/net/etc. CBT EXTRAS
  9. 9. • SSH Key on head • Pub key in all hosts authorized_keys (including head) • Ceph packages on all hosts • PDSH packages on all hosts (for pdcp) • Collectl installed on all hosts BASIC SETUP
  10. 10. • Test network beforehand, bad network easily impairs performance • All-to-All iperf • Check network routes, interfaces • Bonding • Switches should use 5-tuple-hashing for LACP • Nodes should use LACP xmit_hash_policy=layer3+4 TEST METHODOLOGY
  11. 11. • Use multiple iterations for micro benchmarks • Use client sweeps to establish point of contention / max throughput • Client sweeps should always start with X(1) ~ 1 client • Should have 4-6 different increments of clients • Eg. client1, client[1-2], client[1-3], client[1-4] TEST METHODOLOGY
  12. 12. Testplan Examples
  13. 13. CBT CLUSTER CONFIGURATION cluster: head: "ceph@head” clients: ["ceph@client"] osds: ["ceph@osd"] mons: ["ceph@mon"] osds_per_node: 1 fs: xfs mkfs_opts: -f -i size=2048 mount_opts: -o inode64,noatime,logbsize=256k conf_file: /etc/ceph.conf ceph.conf: /etc/ceph/cepf.conf iterations: 3 rebuild_every_test: False tmp_dir: "/tmp/cbt" pool_profiles: replicated: pg_size: 4096 pgp_size: 4096 replication: 'replicated'
  14. 14. CLIENT SWEEPS cluster: head: "ceph@head” clients: ["ceph@client1"] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, ”ceph@client3”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, “ceph@client3”,”ceph@client4”] osds: ["ceph@osd"] mons: ["ceph@mon"]
  15. 15. • Spawns RADOS bench processes on each client • Establish raw RADOS throughput • Works against replicated or EC pools RADOS BENCH benchmarks: radosbench: op_size: [ 4194304, 524288, 4096 ] write_only: False time: 300 concurrent_ops: [ 128 ] concurrent_procs: 1 use_existing: True pool_profile: replicated osd_ra: [256]
  16. 16. • Spawns FIO proccesses on each client • Uses RBD ioengine • Establish raw librbd performance • No VM / container setup required FIO WITH RBD IO ENGINE benchmarks: librbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] procs_per_volume: [ 1 ] volumes_per_client: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100 use_existing_volumes: true
  17. 17. • Maps KRBD volume to each client • Creates EXT4 filesystem on KRBD • Mounts filesystem • Spawns FIO process per client • Uses AIO IO Engine on filesystem • Client can be container or bare metal • Establishes KRBD performance potential FIO WITH KRBD ON EXT4 benchmarks: rbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
  18. 18. • Create KVM instances outside CBT • KVM instances listed as clients • Creates EXT4 filesystem on /dev/vdb • Mounts filesystem • Spanws FIO process per client • Uses AIO IO Engine • Establish RBD performance with QEMU IO susbsystems FIO WITH KVM (VDB) ON EXT4 benchmarks: kvmrbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
  19. 19. • Install COSBench on head/clients outside CBT • Install / Configure RGW outside CBT • Translates CBT YAML to COSBench XML • Runs COSBench COSBENCH benchmarks: cosbench: cosbench_dir: /root/0.4.1.0 cosbench_xml_dir: /home/ceph-admin/plugin/cbt/conf/cosbench/ controller: client01 auth: config: username=cosbench:operator;password=intel2012;url=… obj_size: [128KB] template: [default] mode: [write] ratio: [100] …. Example at cbt/docs/cosbench.README
  20. 20. Running CBT # Loop through each test plan for clients in $(seq 1 6);do cbt/cbt –archive=/tmp/${clients}-clients-results path/to/test.yaml done
  21. 21. ANALYZING DATA
  22. 22. • No robust tools for analysis • Nested archive directory based on YAML options • Archive/000000/Librbdfio/osd_ra-00000128… • Usually awk/grep/cut-fu to csv • Plot charts with gnplot, Excel, R ANALYZING DATA
  23. 23. THANK YOU!

×