ZFS Performance
Analysis and Tools
Brendan Gregg
Lead Performance Engineer
brendan@joyent.com
@brendangregg

ZFS Day
Octob...
whoami

•
•
•

G’Day, I’m Brendan
These days I do systems performance analysis of the cloud
Regarding ZFS:

•

Perf analys...
Who is Joyent

•
•

Cloud computing provider (public cloud + software)
Use ZFS as much as possible:

•
•
•

Host storage
G...
Joyent, cont.

•
•

We build tools for ZFS automation and observability.
Performance is a key company feature.

•

Need to...
Agenda

•

My top 12 tools for ZFS performance analysis (unsorted):

•
•
•
•
•
•
•
•
•
•
•
•

iostat
vfsstat
zfsslower.d
i...
Functional diagram: full stack

•

Unix 101

Process
User-Land
Kernel

logical I/O

VFS
ZFS

...

Block Device Interface
p...
Functional diagram: full stack, cont.

•

Unix 102

User-Land
Kernel

sync.

Process
Syscall
Interface
VFS

ZFS

...

Bloc...
Functional diagram: full stack, cont.

•

DTrace 301

mysql_pid_fslatency.d
syscall with
fi_fs == zfs

Process
User-Land
K...
ZFS Internals

•
•

That’s just my top 12
Use more as needed
ZFS Internals

•
•

That’s just my top 12
Use more as needed

DTRACE
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
ZFSperftools2012
Upcoming SlideShare
Loading in...5
×

ZFSperftools2012

25,062

Published on

zfsday talk (a video is on the last slide). The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.

Published in: Technology
1 Comment
28 Likes
Statistics
Notes
No Downloads
Views
Total Views
25,062
On Slideshare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
311
Comments
1
Likes
28
Embeds 0
No embeds

No notes for slide

ZFSperftools2012

  1. 1. ZFS Performance Analysis and Tools Brendan Gregg Lead Performance Engineer brendan@joyent.com @brendangregg ZFS Day October, 2012
  2. 2. whoami • • • G’Day, I’m Brendan These days I do systems performance analysis of the cloud Regarding ZFS: • Perf analysis of ZFS (mostly using DTrace) for 5+ years, both enterprise and cloud usage • • Wrote many DTrace-based ZFS perf analysis tools including those in the DTrace book Developed ZFS L2ARC while at Sun
  3. 3. Who is Joyent • • Cloud computing provider (public cloud + software) Use ZFS as much as possible: • • • Host storage Guest storage: OS virtualization (SmartOS), and KVM guests (Linux, Windows) We use ZFS because • Reliability: checksums, COW • Features: snapshots, clones, compression, ... • Performance: large ARCs • It can boil oceans
  4. 4. Joyent, cont. • • We build tools for ZFS automation and observability. Performance is a key company feature. • Need to solve FS/disk issues fast.
  5. 5. Agenda • My top 12 tools for ZFS performance analysis (unsorted): • • • • • • • • • • • • iostat vfsstat zfsslower.d iosnoop iostacks metaslab_free.d spasync.d arcstat arcaccess.d latency counters scatter plots heat maps (CA) For cloud computing from within a Zone, add: • • mysqld_pid_fslatency.d syscall with fi_fs == zfs
  6. 6. Functional diagram: full stack • Unix 101 Process User-Land Kernel logical I/O VFS ZFS ... Block Device Interface physical I/O Disks Syscall Interface
  7. 7. Functional diagram: full stack, cont. • Unix 102 User-Land Kernel sync. Process Syscall Interface VFS ZFS ... Block Device Interface Disks iostat(1) often async: write buffering, read ahead
  8. 8. Functional diagram: full stack, cont. • DTrace 301 mysql_pid_fslatency.d syscall with fi_fs == zfs Process User-Land Kernel VFS zioslower.d spasync.d metaslab_free.d arcstat.pl arcaccess.d Syscall Interface vfsstat ZFS ... iostacks.d Block Device Interface kernel drivers as needed see DTrace book chap 4 Disks iostat iosnoop
  9. 9. ZFS Internals • • That’s just my top 12 Use more as needed
  10. 10. ZFS Internals • • That’s just my top 12 Use more as needed DTRACE
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×