ZFSperftools2012

29,574 views

Published on

zfsday talk (a video is on the last slide). The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.

Published in: Technology

ZFSperftools2012

  1. 1. ZFS Performance Analysis and Tools Brendan Gregg Lead Performance Engineer brendan@joyent.com @brendangregg ZFS Day October, 2012
  2. 2. whoami • • • G’Day, I’m Brendan These days I do systems performance analysis of the cloud Regarding ZFS: • Perf analysis of ZFS (mostly using DTrace) for 5+ years, both enterprise and cloud usage • • Wrote many DTrace-based ZFS perf analysis tools including those in the DTrace book Developed ZFS L2ARC while at Sun
  3. 3. Who is Joyent • • Cloud computing provider (public cloud + software) Use ZFS as much as possible: • • • Host storage Guest storage: OS virtualization (SmartOS), and KVM guests (Linux, Windows) We use ZFS because • Reliability: checksums, COW • Features: snapshots, clones, compression, ... • Performance: large ARCs • It can boil oceans
  4. 4. Joyent, cont. • • We build tools for ZFS automation and observability. Performance is a key company feature. • Need to solve FS/disk issues fast.
  5. 5. Agenda • My top 12 tools for ZFS performance analysis (unsorted): • • • • • • • • • • • • iostat vfsstat zfsslower.d iosnoop iostacks metaslab_free.d spasync.d arcstat arcaccess.d latency counters scatter plots heat maps (CA) For cloud computing from within a Zone, add: • • mysqld_pid_fslatency.d syscall with fi_fs == zfs
  6. 6. Functional diagram: full stack • Unix 101 Process User-Land Kernel logical I/O VFS ZFS ... Block Device Interface physical I/O Disks Syscall Interface
  7. 7. Functional diagram: full stack, cont. • Unix 102 User-Land Kernel sync. Process Syscall Interface VFS ZFS ... Block Device Interface Disks iostat(1) often async: write buffering, read ahead
  8. 8. Functional diagram: full stack, cont. • DTrace 301 mysql_pid_fslatency.d syscall with fi_fs == zfs Process User-Land Kernel VFS zioslower.d spasync.d metaslab_free.d arcstat.pl arcaccess.d Syscall Interface vfsstat ZFS ... iostacks.d Block Device Interface kernel drivers as needed see DTrace book chap 4 Disks iostat iosnoop
  9. 9. ZFS Internals • • That’s just my top 12 Use more as needed
  10. 10. ZFS Internals • • That’s just my top 12 Use more as needed DTRACE

×