C4Media, profile picture
Uploaded byC4Media
783 views

Broken Performance Tools

The document discusses the challenges and intricacies associated with performance tools in software development, particularly focusing on observability, benchmarking, and various anti-patterns that can lead to misleading interpretations of system metrics. Brendan Gregg, a senior performance architect at Netflix, shares insights on common pitfalls in analyzing CPU usage, memory management, and disk I/O, emphasizing the importance of understanding the underlying mechanics of these metrics. He highlights the significance of accurate documentation and intuitive metrics design to enhance usability for practitioners in the field.

Technology◦
Related topics:
performance-tuning•

Embed presentation

1 / 131
2 / 131
3 / 131
4 / 131
5 / 131
6 / 131
7 / 131
8 / 131
9 / 131
10 / 131
11 / 131
12 / 131
13 / 131
14 / 131
15 / 131
16 / 131
17 / 131
18 / 131
19 / 131
20 / 131
21 / 131
22 / 131
23 / 131
24 / 131
25 / 131
26 / 131
27 / 131
28 / 131
29 / 131
30 / 131
31 / 131
32 / 131
33 / 131
34 / 131
35 / 131
36 / 131
37 / 131
38 / 131
39 / 131
40 / 131
41 / 131
42 / 131
43 / 131
44 / 131
45 / 131
46 / 131
47 / 131
48 / 131
49 / 131
50 / 131
51 / 131
52 / 131
53 / 131
54 / 131
55 / 131
56 / 131
57 / 131
58 / 131
59 / 131
60 / 131
61 / 131
62 / 131
63 / 131
64 / 131
65 / 131
66 / 131
67 / 131
68 / 131
69 / 131
70 / 131
71 / 131
72 / 131
73 / 131
74 / 131
75 / 131
76 / 131
77 / 131
78 / 131
79 / 131
80 / 131
81 / 131
82 / 131
83 / 131
84 / 131
85 / 131
86 / 131
87 / 131
88 / 131
89 / 131
90 / 131
91 / 131
92 / 131
93 / 131
94 / 131
95 / 131
96 / 131
97 / 131
98 / 131
99 / 131
100 / 131
101 / 131
102 / 131
103 / 131
104 / 131
105 / 131
106 / 131
107 / 131
108 / 131
109 / 131
110 / 131
111 / 131
112 / 131
113 / 131
114 / 131
115 / 131
116 / 131
117 / 131
118 / 131
119 / 131
120 / 131
121 / 131
122 / 131
123 / 131
124 / 131
125 / 131
126 / 131
127 / 131
128 / 131
129 / 131
130 / 131
131 / 131
Broken Performance Tools Brendan Gregg Senior Performance Architect, Netflix Nov  2015  
InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /broken-performance-tools
Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
CAUTION: PERFORMANCE TOOLS
•  Over 60 million subscribers •  AWS EC2 Linux cloud •  FreeBSD CDN •  Awesome place to work
This Talk •  Observability, benchmarking, anti-patterns, and lessons •  Broken and misleading things that are surprising Note: problems with current implementations are discussed, which may be fixed/improved in the future
Observability: System Metrics
LOAD AVERAGES
RFC  546  
Load Averages (1, 5, 15 min) •  "load" –  Usually CPU demand (scheduler run queue length/latency) –  On Linux, task demand: CPU + uninterruptible disk I/O (?) •  "average" –  Exponentially damped moving sum •  "1, 5, and 15 minutes" –  Constants used in the equation •  Don't study these for longer than 10 seconds $ uptime 22:08:07 up 9:05, 1 user, load average: 11.42, 11.87, 12.12
t=0 Load begins (1 thread) 1 5 15 @ 1 min: 1 min avg =~ 0.62
Load Average "1 minute load average" really means… "The exponentially damped moving sum of CPU + uninterruptible disk I/O that uses a value of 60 seconds in its equation"
TOP %CPU
top %CPU •  Who is consuming CPU? •  And by how much? $ top - 20:15:55 up 19:12, 1 user, load average: 7.96, 8.59, 7.05 Tasks: 470 total, 1 running, 468 sleeping, 0 stopped, 1 zombie %Cpu(s): 28.1 us, 0.4 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.1 st KiB Mem: 61663100 total, 61342588 used, 320512 free, 9544 buffers KiB Swap: 0 total, 0 used, 0 free. 3324696 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11959 apiprod 20 0 81.731g 0.053t 14476 S 935.8 92.1 13568:22 java 12595 snmp 20 0 21240 3256 1392 S 3.6 0.0 2:37.23 snmp-pass 10447 snmp 20 0 51512 6028 1432 S 2.0 0.0 2:12.12 snmpd 18463 apiprod 20 0 23932 1972 1176 R 0.7 0.0 0:00.07 top […]
top: Missing %CPU •  Short-lived processes can be missing entirely –  Process creates and exits in-between sampling /proc. e.g., software builds. –  Try atop(1), or sampling using perf(1) •  Stop clearing the screen! –  No option to turn this off. Your eyes can miss updates. –  I often use pidstat(1) on Linux instead. Scroll back for history.
top: Misinterpreting %CPU •  Different top(1)s use different calculations -  On different OSes, check the man page, and run a test! •  %CPU can mean: –  A) Sum of per-CPU percents (0-Ncpu x 100%) consumed during the last interval –  B) Percentage of total CPU capacity (0-100%) consumed during the last interval –  C) (A) but historically damped (like load averages) –  D) (B) " " "
top: %Cpu vs %CPU •  This 4 CPU system is consuming: –  130% total CPU, via %Cpu(s) –  190% total CPU, via %CPU •  Which one is right? Is either? $ top - 15:52:58 up 10 days, 21:58, 2 users, load average: 0.27, 0.53, 0.41 Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.2 us, 24.5 sy, 0.0 ni, 67.2 id, 0.2 wa, 0.0 hi, 6.6 si, 0.4 st KiB Mem: 2872448 total, 2778160 used, 94288 free, 31424 buffers KiB Swap: 4151292 total, 76 used, 4151216 free. 2411728 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12678 root 20 0 96812 1100 912 S 100.4 0.0 0:23.52 iperf 12675 root 20 0 170544 1096 904 S 88.8 0.0 0:20.83 iperf 215 root 20 0 0 0 0 S 0.3 0.0 0:27.73 jbd2/sda1-8 […]
CPU Summary Statistics •  %Cpu row is from /proc/stat •  linux/Documentation/cpu-load.txt: •  /proc/stat is used by everything for CPU stats In most cases the `/proc/stat' information reflects
 the reality quite closely, however due to the nature
 of how/when the kernel collects this data
 sometimes it can not be trusted at all.
%CPU
What is %CPU anyway? •  "Good" %CPU: –  Retiring instructions (provided they aren't a spin loop) –  High IPC (Instructions-Per-Cycle) •  "Bad" %CPU: –  Stall cycles waiting on resources, usually memory I/O –  Low IPC –  Buying faster processors may make little difference •  %CPU alone is ambiguous –  Would love top(1) to split %CPU into cycles retiring vs stalled –  Although, it gets worse…
A CPU Mystery… •  As load increased, CPU ms per request lowered (blue) –  up to 1.84x faster •  Was it due to: -  Cache warmth? no -  Different code? no -  Turbo boost? no •  (Same test, but problem fixed, is shown in red)
CPU Speed Variation •  Clock speed can vary thanks to: –  Intel Turbo Boost: by hardware, based on power, temp, etc –  Intel Speed Step: by software, controlled by the kernel •  %CPU is still ambiguous, given IPC. Need to know the clock speed as well •  CPU counters nowadays have "reference cycles"
Out-of-order Execution •  CPUs execute uops out-of- order and in parallel across multiple functional units •  %CPU doesn't account for how many units are active •  Accounting each cycles as "stalled" or “retiring" is a simplification •  Nowadays it's a lot of work to truly understand what CPUs are doing h/ps://upload.wikimedia.org/wikipedia/commons/6/64/Intel_Nehalem_arch.svg  
I/O WAIT
I/O Wait •  Suggests system is disk I/O bound, but often misleading •  Comparing I/O wait between system A and B: -  higher might be bad: slower disks, more blocking -  lower might be bad: slower processor and architecture consumes more CPU, obscuring I/O wait •  Solaris implementation was also broken and later hardwired to zero •  Can be very useful when understood: another idle state $ mpstat -P ALL 1 08:06:43 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 08:06:44 PM all 53.45 0.00 3.77 0.00 0.00 0.39 0.13 0.00 42.26 […]
I/O Wait Venn Diagram "CPU"   "I/O Wait"  "CPU"   "Idle"   CPU   Waiting for disk I/O   Per CPU:  
FREE MEMORY
Free Memory •  "free" is near-zero: I'm running out of memory! -  No, it's in the file system cache, and is still free for apps to use •  Linux free(1) explains it, but other tools, e.g. vmstat(1), don't •  Some file systems (e.g., ZFS) may not be shown in the system's cached metrics at all www.linuxatemyram.com   $ free -m total used free shared buffers cached Mem: 3750 1111 2639 0 147 527 -/+ buffers/cache: 436 3313 Swap: 0 0 0  
VMSTAT
vmstat(1) •  Linux: first line has some summary since boot values — confusing! •  Other implementations: -  "r" may be sampled once per second. Almost useless. -  Columns like "de" for deficit, making much less sense for non- page scanned situations $ vmstat –Sm 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 8 0 0 1620 149 552 0 0 1 179 77 12 25 34 0 0 7 0 0 1598 149 552 0 0 0 0 205 186 46 13 0 0 8 0 0 1617 149 552 0 0 0 8 210 435 39 21 0 0 8 0 0 1589 149 552 0 0 0 0 218 219 42 17 0 0 […]  
NETSTAT -S
netstat -s $ netstat -s Ip: 7962754 total packets received 8 with invalid addresses 0 forwarded 0 incoming packets discarded 7962746 incoming packets delivered 8019427 requests sent out Icmp: 382 ICMP messages received 0 input ICMP message failed. ICMP input histogram: destination unreachable: 125 timeout in transit: 257 3410 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 3410 IcmpMsg: InType3: 125 InType11: 257 OutType3: 3410 Tcp: 17337 active connections openings 395515 passive connection openings 8953 failed connection attempts 240214 connection resets received 3 connections established 7198375 segments received 7504939 segments send out 62696 segments retransmited 10 bad segments received. 1072 resets sent InCsumErrors: 5 Udp: 759925 packets received 3412 packets to unknown port received. 0 packet receive errors 784370 packets sent UdpLite: TcpExt: 858 invalid SYN cookies received 8951 resets received for embryonic SYN_RECV sockets 14 packets pruned from receive queue because of socket buffer overrun 6177 TCP sockets finished time wait in fast timer 293 packets rejects in established connections because of timestamp 733028 delayed acks sent 89 delayed acks further delayed because of locked socket Quick ack mode was activated 13214 times 336520 packets directly queued to recvmsg prequeue. 43964 packets directly received from backlog 11406012 packets directly received from prequeue 1039165 packets header predicted 7066 packets header predicted and directly queued to user 1428960 acknowledgments not containing data received 1004791 predicted acknowledgments 1 times recovered from packet loss due to fast retransmit 5044 times recovered from packet loss due to SACK data 2 bad SACKs received Detected reordering 4 times using SACK Detected reordering 11 times using time stamp 13 congestion windows fully recovered 11 congestion windows partially recovered using Hoe heuristic TCPDSACKUndo: 39 2384 congestion windows recovered after partial ack 228 timeouts after SACK recovery 100 timeouts in loss state 5018 fast retransmits 39 forward retransmits 783 retransmits in slow start 32455 other TCP timeouts TCPLossProbes: 30233 TCPLossProbeRecovery: 19070 992 sack retransmits failed 18 times receiver scheduled too late for direct processing 705 packets collapsed in receive queue due to low socket buffer 13658 DSACKs sent for old packets 8 DSACKs sent for out of order packets 13595 DSACKs received 33 DSACKs for out of order packets received 32 connections reset due to unexpected data 108 connections reset due to early user close 1608 connections aborted due to timeout TCPSACKDiscard: 4 TCPDSACKIgnoredOld: 1 TCPDSACKIgnoredNoUndo: 8649 TCPSpuriousRTOs: 445 TCPSackShiftFallback: 8588 TCPRcvCoalesce: 95854 TCPOFOQueue: 24741 TCPOFOMerge: 8 TCPChallengeACK: 1441 TCPSYNChallenge: 5 TCPSpuriousRtxHostQueues: 1 TCPAutoCorking: 4823 IpExt: InOctets: 1561561375 OutOctets: 1509416943 InNoECTPkts: 8201572 InECT1Pkts: 2 InECT0Pkts: 3844 InCEPkts: 306
netstat -s […] Tcp: 17337 active connections openings 395515 passive connection openings 8953 failed connection attempts 240214 connection resets received 3 connections established 7198870 segments received 7505329 segments send out 62697 segments retransmited 10 bad segments received. 1072 resets sent InCsumErrors: 5 […]
netstat -s •  Many metrics on Linux (can be over 200) •  Still doesn't include everything: getting better, but don't assume everything is there •  Includes typos & inconsistencies •  Might be more readable to: cat /proc/net/snmp /proc/net/netstat •  Totals since boot can be misleading •  On Linux, -s needs -c support •  Often no documentation outside kernel source code •  Requires expertise to comprehend
DISK METRICS
Disk Metrics •  All disk metrics are misleading •  Disk %utilization / %busy –  Logical devices (volume managers) can process requests in parallel, and may accept more I/O at 100% •  Disk IOPS –  High IOPS is "bad"? That depends… •  Disk latency –  Does it matter? File systems and volume managers try hard to hide latency and make latency asynchronous –  Better measuring latency via application->FS calls
Rules for Metrics Makers •  They must work –  As well as possible. Clearly document caveats. •  They must be useful –  Document a real use case (eg, my example.txt files). If you get stuck, it's not useful – ditch it. •  Aim to be intuitive –  Document it. If it's too weird to explain, redo it. •  As few as possible –  Respect end-user's time •  Good system examples: –  iostat -x: workload columns, then resulting perf columns –  Linux sar: consistency, units on columns, logical groups
Observability: Profilers
PROFILERS
Linux perf •  Can sample stack traces and summarize output: # perf report -n -stdio […] # Overhead Samples Command Shared Object Symbol # ........ ............ ....... ................. ............................. # 20.42% 605 bash [kernel.kallsyms] [k] xen_hypercall_xen_version | --- xen_hypercall_xen_version check_events | |--44.13%-- syscall_trace_enter | tracesys | | | |--35.58%-- __GI___libc_fcntl | | | | | |--65.26%-- do_redirection_internal | | | do_redirections | | | execute_builtin_or_function | | | execute_simple_command [… ~13,000 lines truncated …]
Too Much Output
… as a Flame Graph
PROFILER VISIBILITY
System Profilers with Java Java Kernel TCP/IP GC Idle thread Time Locks epoll JVM
System Profilers with Java •  e.g., Linux perf •  Visibility –  JVM (C++) –  GC (C++) –  libraries (C) –  kernel (C) •  Typical problems (x86): –  Stacks missing for Java and other runtimes –  Symbols missing for Java methods •  Profile everything except Java and similar runtimes
Java Profilers Java GC Kernel, libraries, JVM
Java Profilers •  Visibility –  Java method execution –  Object usage –  GC logs –  Custom Java context •  Typical problems: –  Sampling often happens at safety/yield points (skew) –  Method tracing has massive observer effect –  Misidentifies RUNNING as on-CPU (e.g., epoll) –  Doesn't include or profile GC or JVM CPU time –  Tree views not quick (proportional) to comprehend •  Inaccurate (skewed) and incomplete profiles
COMPILER OPTIMIZATIONS
Broken System Stack Traces •  Profiling Java on x86 using perf •  The stacks are 1 or 2 levels deep, and have junk values # perf record –F 99 –a –g – sleep 30 # perf script […] java 4579 cpu-clock: ffffffff8172adff tracesys ([kernel.kallsyms]) 7f4183bad7ce pthread_cond_timedwait@@GLIBC_2… java 4579 cpu-clock: 7f417908c10b [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f4179101c97 [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f41792fc65f [unknown] (/tmp/perf-4458.map) a2d53351ff7da603 [unknown] ([unknown]) java 4579 cpu-clock: 7f4179349aec [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f4179101d0f [unknown] (/tmp/perf-4458.map) […]
Why Stacks are Broken •  On x86 (x86_64), hotspot uses the frame pointer register (RBP) as general purpose •  This "compiler optimization" breaks stack walking •  Once upon a time, x86 had fewer registers, and this made much more sense •  gcc provides -fno-omit-frame-pointer to avoid doing this –  JDK8u60+ now has this as -XX:+PreserveFramePoiner
Missing Symbols 12.06% 62 sed sed [.] re_search_internal | --- re_search_internal | |--96.78%-- re_search_stub | rpl_re_search | match_regex | do_subst | execute_program | process_files | main | __libc_start_main 71.79% 334 sed sed [.] 0x000000000001afc1 | |--11.65%-- 0x40a447 | 0x40659a | 0x408dd8 | 0x408ed1 | 0x402689 | 0x7fa1cd08aec5 broken not broken •  Missing symbols may show up as hex; e.g., Linux perf:
Fixing Symbols •  For applications, install debug symbol package •  For JIT'd code, Linux perf already looks for an externally provided symbol file: /tmp/perf-PID.map •  Find for a way to create this for your runtime # perf script Failed to open /tmp/perf-8131.map, continuing without symbols […] java 8131 cpu-clock: 7fff76f2dce1 [unknown] ([vdso]) 7fd3173f7a93 os::javaTimeMillis() (/usr/lib/jvm… 7fd301861e46 [unknown] (/tmp/perf-8131.map) […]
INSTRUCTION PROFILING
Instruction Profiling # perf annotate -i perf.data.noplooper --stdio Percent | Source code & Disassembly of noplooper -------------------------------------------------------- : Disassembly of section .text: : : 00000000004004ed <main>: 0.00 : 4004ed: push %rbp 0.00 : 4004ee: mov %rsp,%rbp 20.86 : 4004f1: nop 0.00 : 4004f2: nop 0.00 : 4004f3: nop 0.00 : 4004f4: nop 19.84 : 4004f5: nop 0.00 : 4004f6: nop 0.00 : 4004f7: nop 0.00 : 4004f8: nop 18.73 : 4004f9: nop 0.00 : 4004fa: nop 0.00 : 4004fb: nop 0.00 : 4004fc: nop 19.08 : 4004fd: nop 0.00 : 4004fe: nop 0.00 : 4004ff: nop 0.00 : 400500: nop 21.49 : 400501: jmp 4004f1 <main+0x4> •  Often broken nowadays due to skid, out-of-order execution, and sampling the resumption instruction •  Better with PEBS support
Observability: Overhead
TCPDUMP
tcpdump •  Packet tracing doesn't scale. Overheads: –  CPU cost of per-packet tracing (improved by [e]BPF) •  Consider CPU budget per-packet at 10/40/100 GbE –  Transfer to user-level (improved by ring buffers) –  File system storage (more CPU, and disk I/O) –  Possible additional network transfer •  Can also drop packets when overloaded •  You should only trace send/receive as a last resort –  I solve problems by tracing lower frequency TCP events $ tcpdump -i eth0 -w /tmp/out.tcpdump tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes ^C7985 packets captured 8996 packets received by filter 1010 packets dropped by kernel
STRACE
strace •  Before: •  After: •  442x slower. This is worst case. •  strace(1) pauses the process twice for each syscall. This is like putting metering lights on your app. –  "BUGS: A traced process runs slowly." – strace(1) man page –  Use buffered tracing / in-kernel counters instead, e.g. DTrace $ dd if=/dev/zero of=/dev/null bs=1 count=500k […] 512000 bytes (512 kB) copied, 0.103851 s, 4.9 MB/s $ strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=500k […] 512000 bytes (512 kB) copied, 45.9599 s, 11.1 kB/s
DTRACE
DTrace •  Overhead often negligible, but not always •  Before: •  After: •  384x slower. Fairly worst case: frequent pid probes. # time wc systemlog 262600 2995200 23925200 systemlog real 0m1.098s user 0m1.085s sys 0m0.012s # time dtrace -n 'pid$target:::entry { @[probefunc] = count(); }' -c 'wc systemlog' dtrace: description 'pid$target:::entry ' matched 3756 probes 262600 2995200 23925200 systemlog […] real 7m2.896s user 7m2.650s sys 0m0.572s
Tracing Dangers •  Overhead potential exists for all tracers –  Overhead = event instrumentation cost X frequency of event •  Costs –  Lower: counters, in-kernel aggregations –  Higher: event dumps, stack traces, string copies, copyin/outs •  Frequencies –  Lower: process creation & destruction, disk I/O (usually), … –  Higher: instructions, functions in I/O hot path, malloc/free, Java methods, … •  Advice –  < 10,000 events/sec, probably ok –  > 100,000 events/sec, overhead may start to be measurable
DTraceToolkit •  My own tools that can cause massive overhead: –  dapptrace/dappprof: can trace all native functions –  Java/j_flow.d, ...: can trace all Java methods with +ExtendedDTraceProbes •  Useful for debugging, but should warn about overheads # j_flow.d C PID TIME(us) -- CLASS.METHOD 0 311403 4789112583163 -> java/lang/Object.<clinit> 0 311403 4789112583207 -> java/lang/Object.registerNatives 0 311403 4789112583323 <- java/lang/Object.registerNatives 0 311403 4789112583333 <- java/lang/Object.<clinit> 0 311403 4789112583343 -> java/lang/String.<clinit> 0 311403 4789112583732 -> java/lang/String$CaseInsensitiveComparator.<init> 0 311403 4789112583743 -> java/lang/String$CaseInsensitiveComparator.<init> 0 311403 4789112583752 -> java/lang/Object.<init> [...]
VALGRIND
Valgrind •  A suite of tools including an extensive leak detector •  To its credit it does warn the end user "Your  program  will  run  much  slower   (eg.  20  to  30  Imes)  than  normal"     –  h/p://valgrind.org/docs/manual/quick-­‐start.html  
JAVA PROFILERS
Java Profilers •  Some Java profilers have two modes: –  Sampling stacks: eg, at 100 Hertz –  Tracing methods: instrumenting and timing every method •  Method timing has been described as "highly accurate", despite slowing the target by up to 1000x! •  Issues & advice already covered at QCon: –  Nitsan Wakart "Profilers are Lying Hobbitses" earlier today –  Java track tomorrow
Observability: Monitoring
MONITORING
Monitoring •  By now you should recognize these pathologies: –  Let's just graph the system metrics! •  That's not the problem that needs solving –  Let's just trace everything and post process! •  Now you have one million problems per second •  Monitoring adds additional problems: –  Let's have a cloud-wide dashboard update per-second! •  From every instance? Packet overheads? –  Now we have billions of metrics!
Observability: Statistics
STATISTICS
Statistics "Then  there  is  the  man  who  drowned  crossing  a   stream  with  an  average  depth  of  six  inches."         –  W.I.E.  Gates  
Statistics •  Averages can be misleading –  Hide latency outliers –  Per-minute averages can hide multi-second issues •  Percentiles can be misleading –  Probability of hitting 99.9th latency may be more than 1/1000 after many dependency requests •  Show the distribution: –  Summarize: histogram, density plot, frequency trail –  Over-time: scatter plot, heat map •  See Gil Tene's "How Not to Measure Latency" QCon talk from earlier today
Average Latency •  When the index of central tendency isn't…
Observability: Visualizations
VISUALIZATIONS
Tachometers …especially with arbitrary color highlighting
Pie Charts …for real-time metrics usr   sys   wait   idle  
Doughnuts usr   sys   wait   idle   …like pie charts but worse
Traffic Lights …when used for subjective metrics These can be used for objective metrics RED == BAD (usually) GREEN == GOOD (hopefully)
Benchmarking
BENCHMARKING
~100% of benchmarks are wrong
"Most popular benchmarks are flawed" Source: Traeger, A., E. Zadok, N. Joukov, and C. Wright. “A Nine Year Study of File System and Storage Benchmarking,” ACM Transactions on Storage, 2008. Not only can a popular benchmark be broken, but so can all alternates.
REFUTING BENCHMARKS
The energy needed to refute benchmarks is multiple orders of magnitude bigger than to run them It can take 1-2 weeks of senior performance engineering time to debug a single benchmark.
Benchmarking •  Benchmarking is a useful form of experimental analysis –  Try observational first; benchmarks can perturb •  Accurate and realistic benchmarking is vital for technical investments that improve our industry •  However, benchmarking is error prone
COMMON MISTAKES
Common Mistakes 1.  Testing the wrong target –  eg, FS cache instead of disk; misconfiguration 2.  Choosing the wrong target –  eg, disk instead of FS cache … doesn’t resemble real world 3.  Invalid results –  benchmark software bugs 4.  Ignoring errors –  error path may be fast! 5.  Ignoring variance or perturbations –  real workload isn't steady/consistent, which matters 6.  Misleading results –  you benchmark A, but actually measure B, and conclude you measured C
PRODUCT EVALUATIONS
Product Evaluations •  Benchmarking is used for product evaluations & sales •  The Benchmark Paradox: –  If your product’s chances of winning a benchmark are 50/50, you’ll usually lose –  To justify a product switch, a customer may run several benchmarks, and expect you to win them all –  May mean winning a coin toss at least 3 times in a row –  http://www.brendangregg.com/blog/2014-05-03/the-benchmark-paradox.html •  Solving this seeming paradox (and benchmarking): –  Confirm benchmark is relevant to intended workload –  Ask: why isn't it 10x?
Active Benchmarking •  Root cause performance analysis while the benchmark is still running –  Use observability tools –  Identify the limiter (or suspected limiter) and include it with the benchmark results –  Answer: why not 10x? •  This takes time, but uncovers most mistakes
MICRO BENCHMARKS
Micro Benchmarks •  Test a specific function in isolation. e.g.: –  File system maximum cached read operations/sec –  Network maximum throughput •  Examples of bad microbenchmarks: –  gitpid() in a tight loop –  speed of /dev/zero and /dev/null •  Common problems: –  Testing a workload that is not very relevant –  Missing other workloads that are relevant
MACRO BENCHMARKS
Macro Benchmarks •  Simulate application user load. e.g.: –  Simulated web client transaction •  Common problems: –  Misplaced trust: believed to be realistic, but misses variance, errors, perturbations, e.t.c. –  Complex to debug, verify, and root cause
KITCHEN SINK BENCHMARKS
Kitchen Sink Benchmarks •  Run everything! –  Mostly random benchmarks found on the Internet, where most are are broken or irrelevant –  Developers focus on collecting more benchmarks than verifying or fixing the existing ones •  Myth that more benchmarks == greater accuracy –  No, use active benchmarking (analysis)
AUTOMATION
Automated Benchmarks •  Completely automated procedure. e.g.: –  Cloud benchmarks: spin up an instance, benchmark, destroy. Automate. •  Little or no provision for debugging •  Automation is only part of the solution
Benchmarking: More Examples
BONNIE++
bonnie++ •  "simple tests of hard drive and file system performance" •  First metric printed by (thankfully) older versions: per character sequential output •  What was actually tested: –  1 byte writes to libc (via putc()) –  4 Kbyte writes from libc -> FS (depends on OS; see setbuffer()) –  128 Kbyte async writes to disk (depends on storage stack) –  Any file system throttles that may be present (eg, ZFS) –  C++ code, to some extent (bonnie++ 10% slower than Bonnie) •  Actual limiter: –  Single threaded write_block_putc() and putc() calls
APACHE BENCH
Apache Bench •  HTTP web server benchmark •  Single thread limited (use wrk for multi-threaded) •  Keep-alive option (-k): –  without: Can become an unrealistic TCP session benchmark –  with: Can become an unrealistic server throughput test •  Performance issues of ab's own code
UNIXBENCH
UnixBench •  The original kitchen-sink micro benchmark from 1984, published in BYTE magazine •  Innovative & useful for the time, but that time has passed •  More problems than I can shake a stick at •  Starting with…
COMPILERS
UnixBench Makefile •  Default (by ./Run) for Linux. Would you edit it? Then what? ## Very generic #OPTON = -O ## For Linux 486/Pentium, GCC 2.7.x and 2.8.x #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math # -m486 -malign-loops=2 -malign-jumps=2 -malign-functions=2 ## For Linux, GCC previous to 2.7.0 #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math -m486 #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math # -m386 -malign-loops=1 -malign-jumps=1 -malign-functions=1 ## For Solaris 2, or general-purpose GCC 2.7.x OPTON = -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ## For Digital Unix v4.x, with DEC cc v5.x #OPTON = -O4 #CFLAGS = -DTIME -std1 -verbose -w0
UnixBench Makefile •  "Fixing" the Makefile improved the first result, Dhrystone 2, by 64% •  Is everyone "fixing" it the same way, or not? Are they using the same compiler version? Same OS? (No.)
UnixBench Documentation "The results will depend not only on your hardware, but on your operating system, libraries, and even compiler." "So you may want to make sure that all your test systems are running the same version of the OS; or at least publish the OS and compuiler versions with your results."
SYSTEM MICROBENCHMARKS
UnixBench Tests •  Results summarized as "The BYTE Index". From USAGE: •  What can go wrong? Everything. system: dhry2reg Dhrystone 2 using register variables whetstone-double Double-Precision Whetstone syscall System Call Overhead pipe Pipe Throughput context1 Pipe-based Context Switching spawn Process Creation execl Execl Throughput fstime-w File Write 1024 bufsize 2000 maxblocks fstime-r File Read 1024 bufsize 2000 maxblocks fstime File Copy 1024 bufsize 2000 maxblocks fsbuffer-w File Write 256 bufsize 500 maxblocks fsbuffer-r File Read 256 bufsize 500 maxblocks fsbuffer File Copy 256 bufsize 500 maxblocks fsdisk-w File Write 4096 bufsize 8000 maxblocks fsdisk-r File Read 4096 bufsize 8000 maxblocks fsdisk File Copy 4096 bufsize 8000 maxblocks shell1 Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1") shell8 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8") shell16 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16")
Anti-Patterns
ANTI-PATTERNS
Street Light Anti-Method 1.  Pick observability tools that are: –  Familiar –  Found on the Internet –  Found at random 2.  Run tools 3.  Look for obvious issues
Blame Someone Else Anti-Method 1.  Find a system or environment component you are not responsible for 2.  Hypothesize that the issue is with that component 3.  Redirect the issue to the responsible team 4.  When proven wrong, go to 1
Performance Tools Team •  Having a separate performance tools team, who creates tools but doesn't use them (no production exposure) •  At Netflix: –  The performance engineering team builds tools and uses tools for both service consulting and live production triage •  Mogul, Vector, … –  Other teams (CORE, traffic, …) also build performance tools and use them during issues •  Good performance tools are built out of necessity
Messy House Fallacy •  Fallacy: my code is a mess, I bet yours is immaculate, therefore the bug must be mine •  Reality: everyone's code is terrible and buggy •  When analyzing performance, don't overlook the system: kernel, libraries, etc.
Lessons
PERFORMANCE TOOLS
Observability •  Trust nothing, verify everything –  Cross-check with other observability tools –  Write small "known" workloads, and confirm metrics match –  Find other sanity tests: e.g. check known system limits –  Determine how metrics are calculated, averaged, updated •  Find metrics to solve problems –  Instead of understanding hundreds of system metrics –  What problems do you want to observe? What metrics would be sufficient? Find, verify, and use those. e.g., USE Method. –  The metric you want may not yet exist •  File bugs, get these fixed/improved
Observe Everything •  Use functional diagrams to pose Q's and find missing metrics:
Java Mixed-Mode Flame Graph Profile Everything Java JVM Kernel GC
Visualize Everything
Benchmark Nothing •  Trust nothing, verify everything •  Do Active Benchmarking: 1.  Configure the benchmark to run in steady state, 24x7 2.  Do root-cause analysis of benchmark performance 3.  Answer: why is it not 10x?
Links & References •  https://www.rfc-editor.org/rfc/rfc546.pdf •  https://upload.wikimedia.org/wikipedia/commons/6/64/Intel_Nehalem_arch.svg •  http://www.linuxatemyram.com/ •  http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provider-overhead/ •  Traeger, A., E. Zadok, N. Joukov, and C. Wright. “A Nine Year Study of File System and Storage Benchmarking,” ACM Trans- actions on Storage, 2008. •  http://www.brendangregg.com/blog/2014-06-09/java-cpu-sampling-using-hprof.html •  http://www.brendangregg.com/blog/2014-05-03/the-benchmark-paradox.html •  http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html •  https://blogs.oracle.com/roch/entry/decoding_bonnie •  http://www.brendangregg.com/blog/2014-05-02/compilers-love-messing-with- benchmarks.html •  https://code.google.com/p/byte-unixbench/ •  https://qconsf.com/sf2015/presentation/how-not-measure-latency •  https://qconsf.com/sf2015/presentation/profilers-lying •  Caution signs drawn be me, inspired by real-world signs
THANKS
Thanks •  Questions? •  http://techblog.netflix.com •  http://slideshare.net/brendangregg •  http://www.brendangregg.com •  bgregg@netflix.com •  @brendangregg Nov  2015  
Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/broken- performance-tools

More Related Content

PDF
Systems Performance: Enterprise and the Cloud
byBrendan Gregg
 
PDF
Stateless Hypervisors at Scale
byAntony Messerl
 
PDF
Lisa12 methodologies
byBrendan Gregg
 
PDF
Summit2014 riel chegu_w_0340_automatic_numa_balancing_0
bysprdd
 
PDF
200.1,2-Capacity Planning
bybehrad eslamifar
 
PPTX
Fast Userspace OVS with AF_XDP, OVS CONF 2018
byCheng-Chun William Tu
 
PDF
The New Systems Performance
byBrendan Gregg
 
PDF
LISA2010 visualizations
byBrendan Gregg
 
Systems Performance: Enterprise and the Cloud
byBrendan Gregg
 
Stateless Hypervisors at Scale
byAntony Messerl
 
Lisa12 methodologies
byBrendan Gregg
 
Summit2014 riel chegu_w_0340_automatic_numa_balancing_0
bysprdd
 
200.1,2-Capacity Planning
bybehrad eslamifar
 
Fast Userspace OVS with AF_XDP, OVS CONF 2018
byCheng-Chun William Tu
 
The New Systems Performance
byBrendan Gregg
 
LISA2010 visualizations
byBrendan Gregg
 

What's hot

PDF
Performance Analysis: new tools and concepts from the cloud
byBrendan Gregg
 
PDF
Linux Systems Performance 2016
byBrendan Gregg
 
PDF
Java on Linux for devs and ops
byaragozin
 
PPTX
Am I reading GC logs Correctly?
byTier1 App
 
PDF
Open Source Systems Performance
byBrendan Gregg
 
PDF
What every Java developer should know about network?
byaragozin
 
PDF
20150918 klug el performance tuning-v1.4
byJinkoo Han
 
PPTX
DIY Java Profiler
byaragozin
 
PDF
How Netflix Tunes EC2 Instances for Performance
byBrendan Gregg
 
PDF
Java black box profiling JUG.EKB 2016
byaragozin
 
PPTX
Java profiling Do It Yourself
byaragozin
 
ZIP
Operational Efficiency Hacks Web20 Expo2009
byJohn Allspaw
 
PDF
I know why your Java is slow
byaragozin
 
PPTX
Java profiling Do It Yourself (jug.msk.ru 2016)
byaragozin
 
PPTX
Pick diamonds from garbage
byTier1 App
 
PPTX
A fun cup of joe with open liberty
byAndy Mauer
 
TXT
Awrrpt 1 3004_3005
byKam Chan
 
PPTX
Don't dump thread dumps
byTier1app
 
PDF
Understand and optimize Linux I/O
byAndrea Righi
 
POTX
Performance Tuning EC2 Instances
byBrendan Gregg
 
Performance Analysis: new tools and concepts from the cloud
byBrendan Gregg
 
Linux Systems Performance 2016
byBrendan Gregg
 
Java on Linux for devs and ops
byaragozin
 
Am I reading GC logs Correctly?
byTier1 App
 
Open Source Systems Performance
byBrendan Gregg
 
What every Java developer should know about network?
byaragozin
 
20150918 klug el performance tuning-v1.4
byJinkoo Han
 
DIY Java Profiler
byaragozin
 
How Netflix Tunes EC2 Instances for Performance
byBrendan Gregg
 
Java black box profiling JUG.EKB 2016
byaragozin
 
Java profiling Do It Yourself
byaragozin
 
Operational Efficiency Hacks Web20 Expo2009
byJohn Allspaw
 
I know why your Java is slow
byaragozin
 
Java profiling Do It Yourself (jug.msk.ru 2016)
byaragozin
 
Pick diamonds from garbage
byTier1 App
 
A fun cup of joe with open liberty
byAndy Mauer
 
Awrrpt 1 3004_3005
byKam Chan
 
Don't dump thread dumps
byTier1app
 
Understand and optimize Linux I/O
byAndrea Righi
 
Performance Tuning EC2 Instances
byBrendan Gregg
 

Similar to Broken Performance Tools

PPTX
QCon 2015 Broken Performance Tools
byBrendan Gregg
 
PPTX
Broken Linux Performance Tools 2016
byBrendan Gregg
 
PDF
YOW2020 Linux Systems Performance
byBrendan Gregg
 
PDF
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
byNETWAYS
 
PDF
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
byNETWAYS
 
PDF
Linux Performance Profiling and Monitoring
byGeorg Schönberger
 
PDF
OSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
byNETWAYS
 
PDF
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
byNETWAYS
 
PDF
Analyzing OS X Systems Performance with the USE Method
byBrendan Gregg
 
PDF
MeetBSD2014 Performance Analysis
byBrendan Gregg
 
PDF
Velocity 2015 linux perf tools
byBrendan Gregg
 
PDF
test
byWentingLiu4
 
PDF
LISA2019 Linux Systems Performance
byBrendan Gregg
 
PPT
Servers and Processes: Behavior and Analysis
bydreamwidth
 
PDF
Linux Performance Tools
byBrendan Gregg
 
PDF
Linux Performance Tools 2014
byBrendan Gregg
 
PDF
Help, my computer is sluggish
byKevin OBrien
 
PDF
Deep Dive on Amazon EC2 Instances (March 2017)
byJulien SIMON
 
PDF
linux monitoring and performance tunning
byiman darabi
 
PPTX
Using Libtracecmd to Analyze Your Latency and Performance Troubles
byScyllaDB
 
QCon 2015 Broken Performance Tools
byBrendan Gregg
 
Broken Linux Performance Tools 2016
byBrendan Gregg
 
YOW2020 Linux Systems Performance
byBrendan Gregg
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
byNETWAYS
 
OSDC 2015: Georg Schönberger | Linux Performance Profiling and Monitoring
byNETWAYS
 
Linux Performance Profiling and Monitoring
byGeorg Schönberger
 
OSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
byNETWAYS
 
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
byNETWAYS
 
Analyzing OS X Systems Performance with the USE Method
byBrendan Gregg
 
MeetBSD2014 Performance Analysis
byBrendan Gregg
 
Velocity 2015 linux perf tools
byBrendan Gregg
 
test
byWentingLiu4
 
LISA2019 Linux Systems Performance
byBrendan Gregg
 
Servers and Processes: Behavior and Analysis
bydreamwidth
 
Linux Performance Tools
byBrendan Gregg
 
Linux Performance Tools 2014
byBrendan Gregg
 
Help, my computer is sluggish
byKevin OBrien
 
Deep Dive on Amazon EC2 Instances (March 2017)
byJulien SIMON
 
linux monitoring and performance tunning
byiman darabi
 
Using Libtracecmd to Analyze Your Latency and Performance Troubles
byScyllaDB
 

More from C4Media

PDF
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
byC4Media
 
PDF
Next Generation Client APIs in Envoy Mobile
byC4Media
 
PDF
Software Teams and Teamwork Trends Report Q1 2020
byC4Media
 
PDF
Understand the Trade-offs Using Compilers for Java Applications
byC4Media
 
PDF
Kafka Needs No Keeper
byC4Media
 
PDF
High Performing Teams Act Like Owners
byC4Media
 
PDF
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
byC4Media
 
PDF
Service Meshes- The Ultimate Guide
byC4Media
 
PDF
Shifting Left with Cloud Native CI/CD
byC4Media
 
PDF
CI/CD for Machine Learning
byC4Media
 
PDF
Fault Tolerance at Speed
byC4Media
 
PDF
Architectures That Scale Deep - Regaining Control in Deep Systems
byC4Media
 
PDF
ML in the Browser: Interactive Experiences with Tensorflow.js
byC4Media
 
PDF
Build Your Own WebAssembly Compiler
byC4Media
 
PDF
User & Device Identity for Microservices @ Netflix Scale
byC4Media
 
PDF
Scaling Patterns for Netflix's Edge
byC4Media
 
PDF
Make Your Electron App Feel at Home Everywhere
byC4Media
 
PDF
The Talk You've Been Await-ing For
byC4Media
 
PDF
Future of Data Engineering
byC4Media
 
PDF
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
byC4Media
 
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
byC4Media
 
Next Generation Client APIs in Envoy Mobile
byC4Media
 
Software Teams and Teamwork Trends Report Q1 2020
byC4Media
 
Understand the Trade-offs Using Compilers for Java Applications
byC4Media
 
Kafka Needs No Keeper
byC4Media
 
High Performing Teams Act Like Owners
byC4Media
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
byC4Media
 
Service Meshes- The Ultimate Guide
byC4Media
 
Shifting Left with Cloud Native CI/CD
byC4Media
 
CI/CD for Machine Learning
byC4Media
 
Fault Tolerance at Speed
byC4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
byC4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
byC4Media
 
Build Your Own WebAssembly Compiler
byC4Media
 
User & Device Identity for Microservices @ Netflix Scale
byC4Media
 
Scaling Patterns for Netflix's Edge
byC4Media
 
Make Your Electron App Feel at Home Everywhere
byC4Media
 
The Talk You've Been Await-ing For
byC4Media
 
Future of Data Engineering
byC4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
byC4Media
 

Recently uploaded

PDF
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PPTX
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
PDF
December Patch Tuesday
byIvanti
 
PDF
AI Technology important knowledge ......
byutkarshj2007
 
PDF
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
PDF
Recursive Self Improvement vs Continuous Learning
byBob Marcus
 
PPTX
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
PDF
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
PDF
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
PPTX
The Landscape of Computer Software Development Companies
byYash Singh
 
PDF
Generative AI in UiPath: Mastering the Generative Extractor for Intelligent D...
byDianaGray10
 
PDF
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
PDF
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
PDF
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
PDF
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
PDF
Agentic Automation for Testers – A Hands-On Deep Dive
byUiPathCommunity
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
RISE with SAP for Automotive - S4 HANA Value.pptx
byAmira Jemi
 
December Patch Tuesday
byIvanti
 
AI Technology important knowledge ......
byutkarshj2007
 
eResource Scheduler Enterprise Resource Management and Scheduling Software.pdf
byeResource Scheduler
 
Recursive Self Improvement vs Continuous Learning
byBob Marcus
 
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
Guided Substation Engineering with CoMPAS: Simplifying IEC 61850
byDanBrown980551
 
The Landscape of Computer Software Development Companies
byYash Singh
 
Generative AI in UiPath: Mastering the Generative Extractor for Intelligent D...
byDianaGray10
 
AI and Computer Architecture: 200 Years Together
byDmitry Zinoviev
 
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
Knowing and Doing: Knowledge graphs, AI, and work
bymarainglezakis1
 
Agentic Automation for Testers – A Hands-On Deep Dive
byUiPathCommunity
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 

Broken Performance Tools

  • 1.
  • 2.
    InfoQ.com: News &Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /broken-performance-tools
  • 3.
    Purpose of QCon -to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  • 4.
  • 5.
    •  Over 60million subscribers •  AWS EC2 Linux cloud •  FreeBSD CDN •  Awesome place to work
  • 6.
    This Talk •  Observability,benchmarking, anti-patterns, and lessons •  Broken and misleading things that are surprising Note: problems with current implementations are discussed, which may be fixed/improved in the future
  • 7.
  • 8.
  • 9.
  • 10.
    Load Averages (1,5, 15 min) •  "load" –  Usually CPU demand (scheduler run queue length/latency) –  On Linux, task demand: CPU + uninterruptible disk I/O (?) •  "average" –  Exponentially damped moving sum •  "1, 5, and 15 minutes" –  Constants used in the equation •  Don't study these for longer than 10 seconds $ uptime 22:08:07 up 9:05, 1 user, load average: 11.42, 11.87, 12.12
  • 11.
    t=0 Load begins (1 thread) 1 5 15 @1 min: 1 min avg =~ 0.62
  • 12.
    Load Average "1 minuteload average" really means… "The exponentially damped moving sum of CPU + uninterruptible disk I/O that uses a value of 60 seconds in its equation"
  • 13.
  • 14.
    top %CPU •  Whois consuming CPU? •  And by how much? $ top - 20:15:55 up 19:12, 1 user, load average: 7.96, 8.59, 7.05 Tasks: 470 total, 1 running, 468 sleeping, 0 stopped, 1 zombie %Cpu(s): 28.1 us, 0.4 sy, 0.0 ni, 71.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.1 st KiB Mem: 61663100 total, 61342588 used, 320512 free, 9544 buffers KiB Swap: 0 total, 0 used, 0 free. 3324696 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11959 apiprod 20 0 81.731g 0.053t 14476 S 935.8 92.1 13568:22 java 12595 snmp 20 0 21240 3256 1392 S 3.6 0.0 2:37.23 snmp-pass 10447 snmp 20 0 51512 6028 1432 S 2.0 0.0 2:12.12 snmpd 18463 apiprod 20 0 23932 1972 1176 R 0.7 0.0 0:00.07 top […]
  • 15.
    top: Missing %CPU • Short-lived processes can be missing entirely –  Process creates and exits in-between sampling /proc. e.g., software builds. –  Try atop(1), or sampling using perf(1) •  Stop clearing the screen! –  No option to turn this off. Your eyes can miss updates. –  I often use pidstat(1) on Linux instead. Scroll back for history.
  • 16.
    top: Misinterpreting %CPU • Different top(1)s use different calculations -  On different OSes, check the man page, and run a test! •  %CPU can mean: –  A) Sum of per-CPU percents (0-Ncpu x 100%) consumed during the last interval –  B) Percentage of total CPU capacity (0-100%) consumed during the last interval –  C) (A) but historically damped (like load averages) –  D) (B) " " "
  • 17.
    top: %Cpu vs%CPU •  This 4 CPU system is consuming: –  130% total CPU, via %Cpu(s) –  190% total CPU, via %CPU •  Which one is right? Is either? $ top - 15:52:58 up 10 days, 21:58, 2 users, load average: 0.27, 0.53, 0.41 Tasks: 180 total, 1 running, 179 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.2 us, 24.5 sy, 0.0 ni, 67.2 id, 0.2 wa, 0.0 hi, 6.6 si, 0.4 st KiB Mem: 2872448 total, 2778160 used, 94288 free, 31424 buffers KiB Swap: 4151292 total, 76 used, 4151216 free. 2411728 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12678 root 20 0 96812 1100 912 S 100.4 0.0 0:23.52 iperf 12675 root 20 0 170544 1096 904 S 88.8 0.0 0:20.83 iperf 215 root 20 0 0 0 0 S 0.3 0.0 0:27.73 jbd2/sda1-8 […]
  • 18.
    CPU Summary Statistics • %Cpu row is from /proc/stat •  linux/Documentation/cpu-load.txt: •  /proc/stat is used by everything for CPU stats In most cases the `/proc/stat' information reflects
 the reality quite closely, however due to the nature
 of how/when the kernel collects this data
 sometimes it can not be trusted at all.
  • 19.
  • 20.
    What is %CPUanyway? •  "Good" %CPU: –  Retiring instructions (provided they aren't a spin loop) –  High IPC (Instructions-Per-Cycle) •  "Bad" %CPU: –  Stall cycles waiting on resources, usually memory I/O –  Low IPC –  Buying faster processors may make little difference •  %CPU alone is ambiguous –  Would love top(1) to split %CPU into cycles retiring vs stalled –  Although, it gets worse…
  • 21.
    A CPU Mystery… • As load increased, CPU ms per request lowered (blue) –  up to 1.84x faster •  Was it due to: -  Cache warmth? no -  Different code? no -  Turbo boost? no •  (Same test, but problem fixed, is shown in red)
  • 22.
    CPU Speed Variation • Clock speed can vary thanks to: –  Intel Turbo Boost: by hardware, based on power, temp, etc –  Intel Speed Step: by software, controlled by the kernel •  %CPU is still ambiguous, given IPC. Need to know the clock speed as well •  CPU counters nowadays have "reference cycles"
  • 23.
    Out-of-order Execution •  CPUsexecute uops out-of- order and in parallel across multiple functional units •  %CPU doesn't account for how many units are active •  Accounting each cycles as "stalled" or “retiring" is a simplification •  Nowadays it's a lot of work to truly understand what CPUs are doing h/ps://upload.wikimedia.org/wikipedia/commons/6/64/Intel_Nehalem_arch.svg  
  • 24.
  • 25.
    I/O Wait •  Suggestssystem is disk I/O bound, but often misleading •  Comparing I/O wait between system A and B: -  higher might be bad: slower disks, more blocking -  lower might be bad: slower processor and architecture consumes more CPU, obscuring I/O wait •  Solaris implementation was also broken and later hardwired to zero •  Can be very useful when understood: another idle state $ mpstat -P ALL 1 08:06:43 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 08:06:44 PM all 53.45 0.00 3.77 0.00 0.00 0.39 0.13 0.00 42.26 […]
  • 26.
    I/O Wait VennDiagram "CPU"   "I/O Wait"  "CPU"   "Idle"   CPU   Waiting for disk I/O   Per CPU:  
  • 27.
  • 28.
    Free Memory •  "free"is near-zero: I'm running out of memory! -  No, it's in the file system cache, and is still free for apps to use •  Linux free(1) explains it, but other tools, e.g. vmstat(1), don't •  Some file systems (e.g., ZFS) may not be shown in the system's cached metrics at all www.linuxatemyram.com   $ free -m total used free shared buffers cached Mem: 3750 1111 2639 0 147 527 -/+ buffers/cache: 436 3313 Swap: 0 0 0  
  • 29.
  • 30.
    vmstat(1) •  Linux: firstline has some summary since boot values — confusing! •  Other implementations: -  "r" may be sampled once per second. Almost useless. -  Columns like "de" for deficit, making much less sense for non- page scanned situations $ vmstat –Sm 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 8 0 0 1620 149 552 0 0 1 179 77 12 25 34 0 0 7 0 0 1598 149 552 0 0 0 0 205 186 46 13 0 0 8 0 0 1617 149 552 0 0 0 8 210 435 39 21 0 0 8 0 0 1589 149 552 0 0 0 0 218 219 42 17 0 0 […]  
  • 31.
  • 32.
    netstat -s $ netstat-s Ip: 7962754 total packets received 8 with invalid addresses 0 forwarded 0 incoming packets discarded 7962746 incoming packets delivered 8019427 requests sent out Icmp: 382 ICMP messages received 0 input ICMP message failed. ICMP input histogram: destination unreachable: 125 timeout in transit: 257 3410 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 3410 IcmpMsg: InType3: 125 InType11: 257 OutType3: 3410 Tcp: 17337 active connections openings 395515 passive connection openings 8953 failed connection attempts 240214 connection resets received 3 connections established 7198375 segments received 7504939 segments send out 62696 segments retransmited 10 bad segments received. 1072 resets sent InCsumErrors: 5 Udp: 759925 packets received 3412 packets to unknown port received. 0 packet receive errors 784370 packets sent UdpLite: TcpExt: 858 invalid SYN cookies received 8951 resets received for embryonic SYN_RECV sockets 14 packets pruned from receive queue because of socket buffer overrun 6177 TCP sockets finished time wait in fast timer 293 packets rejects in established connections because of timestamp 733028 delayed acks sent 89 delayed acks further delayed because of locked socket Quick ack mode was activated 13214 times 336520 packets directly queued to recvmsg prequeue. 43964 packets directly received from backlog 11406012 packets directly received from prequeue 1039165 packets header predicted 7066 packets header predicted and directly queued to user 1428960 acknowledgments not containing data received 1004791 predicted acknowledgments 1 times recovered from packet loss due to fast retransmit 5044 times recovered from packet loss due to SACK data 2 bad SACKs received Detected reordering 4 times using SACK Detected reordering 11 times using time stamp 13 congestion windows fully recovered 11 congestion windows partially recovered using Hoe heuristic TCPDSACKUndo: 39 2384 congestion windows recovered after partial ack 228 timeouts after SACK recovery 100 timeouts in loss state 5018 fast retransmits 39 forward retransmits 783 retransmits in slow start 32455 other TCP timeouts TCPLossProbes: 30233 TCPLossProbeRecovery: 19070 992 sack retransmits failed 18 times receiver scheduled too late for direct processing 705 packets collapsed in receive queue due to low socket buffer 13658 DSACKs sent for old packets 8 DSACKs sent for out of order packets 13595 DSACKs received 33 DSACKs for out of order packets received 32 connections reset due to unexpected data 108 connections reset due to early user close 1608 connections aborted due to timeout TCPSACKDiscard: 4 TCPDSACKIgnoredOld: 1 TCPDSACKIgnoredNoUndo: 8649 TCPSpuriousRTOs: 445 TCPSackShiftFallback: 8588 TCPRcvCoalesce: 95854 TCPOFOQueue: 24741 TCPOFOMerge: 8 TCPChallengeACK: 1441 TCPSYNChallenge: 5 TCPSpuriousRtxHostQueues: 1 TCPAutoCorking: 4823 IpExt: InOctets: 1561561375 OutOctets: 1509416943 InNoECTPkts: 8201572 InECT1Pkts: 2 InECT0Pkts: 3844 InCEPkts: 306
  • 33.
    netstat -s […] Tcp: 17337 activeconnections openings 395515 passive connection openings 8953 failed connection attempts 240214 connection resets received 3 connections established 7198870 segments received 7505329 segments send out 62697 segments retransmited 10 bad segments received. 1072 resets sent InCsumErrors: 5 […]
  • 34.
    netstat -s •  Manymetrics on Linux (can be over 200) •  Still doesn't include everything: getting better, but don't assume everything is there •  Includes typos & inconsistencies •  Might be more readable to: cat /proc/net/snmp /proc/net/netstat •  Totals since boot can be misleading •  On Linux, -s needs -c support •  Often no documentation outside kernel source code •  Requires expertise to comprehend
  • 35.
  • 36.
    Disk Metrics •  Alldisk metrics are misleading •  Disk %utilization / %busy –  Logical devices (volume managers) can process requests in parallel, and may accept more I/O at 100% •  Disk IOPS –  High IOPS is "bad"? That depends… •  Disk latency –  Does it matter? File systems and volume managers try hard to hide latency and make latency asynchronous –  Better measuring latency via application->FS calls
  • 37.
    Rules for MetricsMakers •  They must work –  As well as possible. Clearly document caveats. •  They must be useful –  Document a real use case (eg, my example.txt files). If you get stuck, it's not useful – ditch it. •  Aim to be intuitive –  Document it. If it's too weird to explain, redo it. •  As few as possible –  Respect end-user's time •  Good system examples: –  iostat -x: workload columns, then resulting perf columns –  Linux sar: consistency, units on columns, logical groups
  • 38.
  • 39.
  • 40.
    Linux perf •  Cansample stack traces and summarize output: # perf report -n -stdio […] # Overhead Samples Command Shared Object Symbol # ........ ............ ....... ................. ............................. # 20.42% 605 bash [kernel.kallsyms] [k] xen_hypercall_xen_version | --- xen_hypercall_xen_version check_events | |--44.13%-- syscall_trace_enter | tracesys | | | |--35.58%-- __GI___libc_fcntl | | | | | |--65.26%-- do_redirection_internal | | | do_redirections | | | execute_builtin_or_function | | | execute_simple_command [… ~13,000 lines truncated …]
  • 41.
  • 42.
    … as aFlame Graph
  • 43.
  • 44.
    System Profilers withJava Java Kernel TCP/IP GC Idle thread Time Locks epoll JVM
  • 45.
    System Profilers withJava •  e.g., Linux perf •  Visibility –  JVM (C++) –  GC (C++) –  libraries (C) –  kernel (C) •  Typical problems (x86): –  Stacks missing for Java and other runtimes –  Symbols missing for Java methods •  Profile everything except Java and similar runtimes
  • 46.
  • 47.
    Java Profilers •  Visibility – Java method execution –  Object usage –  GC logs –  Custom Java context •  Typical problems: –  Sampling often happens at safety/yield points (skew) –  Method tracing has massive observer effect –  Misidentifies RUNNING as on-CPU (e.g., epoll) –  Doesn't include or profile GC or JVM CPU time –  Tree views not quick (proportional) to comprehend •  Inaccurate (skewed) and incomplete profiles
  • 48.
  • 49.
    Broken System StackTraces •  Profiling Java on x86 using perf •  The stacks are 1 or 2 levels deep, and have junk values # perf record –F 99 –a –g – sleep 30 # perf script […] java 4579 cpu-clock: ffffffff8172adff tracesys ([kernel.kallsyms]) 7f4183bad7ce pthread_cond_timedwait@@GLIBC_2… java 4579 cpu-clock: 7f417908c10b [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f4179101c97 [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f41792fc65f [unknown] (/tmp/perf-4458.map) a2d53351ff7da603 [unknown] ([unknown]) java 4579 cpu-clock: 7f4179349aec [unknown] (/tmp/perf-4458.map) java 4579 cpu-clock: 7f4179101d0f [unknown] (/tmp/perf-4458.map) […]
  • 50.
    Why Stacks areBroken •  On x86 (x86_64), hotspot uses the frame pointer register (RBP) as general purpose •  This "compiler optimization" breaks stack walking •  Once upon a time, x86 had fewer registers, and this made much more sense •  gcc provides -fno-omit-frame-pointer to avoid doing this –  JDK8u60+ now has this as -XX:+PreserveFramePoiner
  • 51.
    Missing Symbols 12.06% 62sed sed [.] re_search_internal | --- re_search_internal | |--96.78%-- re_search_stub | rpl_re_search | match_regex | do_subst | execute_program | process_files | main | __libc_start_main 71.79% 334 sed sed [.] 0x000000000001afc1 | |--11.65%-- 0x40a447 | 0x40659a | 0x408dd8 | 0x408ed1 | 0x402689 | 0x7fa1cd08aec5 broken not broken •  Missing symbols may show up as hex; e.g., Linux perf:
  • 52.
    Fixing Symbols •  Forapplications, install debug symbol package •  For JIT'd code, Linux perf already looks for an externally provided symbol file: /tmp/perf-PID.map •  Find for a way to create this for your runtime # perf script Failed to open /tmp/perf-8131.map, continuing without symbols […] java 8131 cpu-clock: 7fff76f2dce1 [unknown] ([vdso]) 7fd3173f7a93 os::javaTimeMillis() (/usr/lib/jvm… 7fd301861e46 [unknown] (/tmp/perf-8131.map) […]
  • 53.
  • 54.
    Instruction Profiling # perfannotate -i perf.data.noplooper --stdio Percent | Source code & Disassembly of noplooper -------------------------------------------------------- : Disassembly of section .text: : : 00000000004004ed <main>: 0.00 : 4004ed: push %rbp 0.00 : 4004ee: mov %rsp,%rbp 20.86 : 4004f1: nop 0.00 : 4004f2: nop 0.00 : 4004f3: nop 0.00 : 4004f4: nop 19.84 : 4004f5: nop 0.00 : 4004f6: nop 0.00 : 4004f7: nop 0.00 : 4004f8: nop 18.73 : 4004f9: nop 0.00 : 4004fa: nop 0.00 : 4004fb: nop 0.00 : 4004fc: nop 19.08 : 4004fd: nop 0.00 : 4004fe: nop 0.00 : 4004ff: nop 0.00 : 400500: nop 21.49 : 400501: jmp 4004f1 <main+0x4> •  Often broken nowadays due to skid, out-of-order execution, and sampling the resumption instruction •  Better with PEBS support
  • 55.
  • 56.
  • 57.
    tcpdump •  Packet tracingdoesn't scale. Overheads: –  CPU cost of per-packet tracing (improved by [e]BPF) •  Consider CPU budget per-packet at 10/40/100 GbE –  Transfer to user-level (improved by ring buffers) –  File system storage (more CPU, and disk I/O) –  Possible additional network transfer •  Can also drop packets when overloaded •  You should only trace send/receive as a last resort –  I solve problems by tracing lower frequency TCP events $ tcpdump -i eth0 -w /tmp/out.tcpdump tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes ^C7985 packets captured 8996 packets received by filter 1010 packets dropped by kernel
  • 58.
  • 59.
    strace •  Before: •  After: • 442x slower. This is worst case. •  strace(1) pauses the process twice for each syscall. This is like putting metering lights on your app. –  "BUGS: A traced process runs slowly." – strace(1) man page –  Use buffered tracing / in-kernel counters instead, e.g. DTrace $ dd if=/dev/zero of=/dev/null bs=1 count=500k […] 512000 bytes (512 kB) copied, 0.103851 s, 4.9 MB/s $ strace -eaccept dd if=/dev/zero of=/dev/null bs=1 count=500k […] 512000 bytes (512 kB) copied, 45.9599 s, 11.1 kB/s
  • 60.
  • 61.
    DTrace •  Overhead oftennegligible, but not always •  Before: •  After: •  384x slower. Fairly worst case: frequent pid probes. # time wc systemlog 262600 2995200 23925200 systemlog real 0m1.098s user 0m1.085s sys 0m0.012s # time dtrace -n 'pid$target:::entry { @[probefunc] = count(); }' -c 'wc systemlog' dtrace: description 'pid$target:::entry ' matched 3756 probes 262600 2995200 23925200 systemlog […] real 7m2.896s user 7m2.650s sys 0m0.572s
  • 62.
    Tracing Dangers •  Overheadpotential exists for all tracers –  Overhead = event instrumentation cost X frequency of event •  Costs –  Lower: counters, in-kernel aggregations –  Higher: event dumps, stack traces, string copies, copyin/outs •  Frequencies –  Lower: process creation & destruction, disk I/O (usually), … –  Higher: instructions, functions in I/O hot path, malloc/free, Java methods, … •  Advice –  < 10,000 events/sec, probably ok –  > 100,000 events/sec, overhead may start to be measurable
  • 63.
    DTraceToolkit •  My owntools that can cause massive overhead: –  dapptrace/dappprof: can trace all native functions –  Java/j_flow.d, ...: can trace all Java methods with +ExtendedDTraceProbes •  Useful for debugging, but should warn about overheads # j_flow.d C PID TIME(us) -- CLASS.METHOD 0 311403 4789112583163 -> java/lang/Object.<clinit> 0 311403 4789112583207 -> java/lang/Object.registerNatives 0 311403 4789112583323 <- java/lang/Object.registerNatives 0 311403 4789112583333 <- java/lang/Object.<clinit> 0 311403 4789112583343 -> java/lang/String.<clinit> 0 311403 4789112583732 -> java/lang/String$CaseInsensitiveComparator.<init> 0 311403 4789112583743 -> java/lang/String$CaseInsensitiveComparator.<init> 0 311403 4789112583752 -> java/lang/Object.<init> [...]
  • 64.
  • 65.
    Valgrind •  A suiteof tools including an extensive leak detector •  To its credit it does warn the end user "Your  program  will  run  much  slower   (eg.  20  to  30  Imes)  than  normal"     –  h/p://valgrind.org/docs/manual/quick-­‐start.html  
  • 66.
  • 67.
    Java Profilers •  SomeJava profilers have two modes: –  Sampling stacks: eg, at 100 Hertz –  Tracing methods: instrumenting and timing every method •  Method timing has been described as "highly accurate", despite slowing the target by up to 1000x! •  Issues & advice already covered at QCon: –  Nitsan Wakart "Profilers are Lying Hobbitses" earlier today –  Java track tomorrow
  • 68.
  • 69.
  • 70.
    Monitoring •  By nowyou should recognize these pathologies: –  Let's just graph the system metrics! •  That's not the problem that needs solving –  Let's just trace everything and post process! •  Now you have one million problems per second •  Monitoring adds additional problems: –  Let's have a cloud-wide dashboard update per-second! •  From every instance? Packet overheads? –  Now we have billions of metrics!
  • 71.
  • 72.
  • 73.
    Statistics "Then  there  is  the  man  who  drowned  crossing  a   stream  with  an  average  depth  of  six  inches."         –  W.I.E.  Gates  
  • 74.
    Statistics •  Averages canbe misleading –  Hide latency outliers –  Per-minute averages can hide multi-second issues •  Percentiles can be misleading –  Probability of hitting 99.9th latency may be more than 1/1000 after many dependency requests •  Show the distribution: –  Summarize: histogram, density plot, frequency trail –  Over-time: scatter plot, heat map •  See Gil Tene's "How Not to Measure Latency" QCon talk from earlier today
  • 75.
    Average Latency •  Whenthe index of central tendency isn't…
  • 76.
  • 77.
  • 78.
  • 79.
    Pie Charts …for real-timemetrics usr   sys   wait   idle  
  • 80.
    Doughnuts usr   sys   wait   idle   …like pie charts but worse
  • 81.
    Traffic Lights …when usedfor subjective metrics These can be used for objective metrics RED == BAD (usually) GREEN == GOOD (hopefully)
  • 82.
  • 83.
  • 84.
  • 85.
    "Most popular benchmarksare flawed" Source: Traeger, A., E. Zadok, N. Joukov, and C. Wright. “A Nine Year Study of File System and Storage Benchmarking,” ACM Transactions on Storage, 2008. Not only can a popular benchmark be broken, but so can all alternates.
  • 86.
  • 87.
    The energy needed torefute benchmarks is multiple orders of magnitude bigger than to run them It can take 1-2 weeks of senior performance engineering time to debug a single benchmark.
  • 88.
    Benchmarking •  Benchmarking isa useful form of experimental analysis –  Try observational first; benchmarks can perturb •  Accurate and realistic benchmarking is vital for technical investments that improve our industry •  However, benchmarking is error prone
  • 89.
  • 90.
    Common Mistakes 1.  Testingthe wrong target –  eg, FS cache instead of disk; misconfiguration 2.  Choosing the wrong target –  eg, disk instead of FS cache … doesn’t resemble real world 3.  Invalid results –  benchmark software bugs 4.  Ignoring errors –  error path may be fast! 5.  Ignoring variance or perturbations –  real workload isn't steady/consistent, which matters 6.  Misleading results –  you benchmark A, but actually measure B, and conclude you measured C
  • 91.
  • 92.
    Product Evaluations •  Benchmarkingis used for product evaluations & sales •  The Benchmark Paradox: –  If your product’s chances of winning a benchmark are 50/50, you’ll usually lose –  To justify a product switch, a customer may run several benchmarks, and expect you to win them all –  May mean winning a coin toss at least 3 times in a row –  http://www.brendangregg.com/blog/2014-05-03/the-benchmark-paradox.html •  Solving this seeming paradox (and benchmarking): –  Confirm benchmark is relevant to intended workload –  Ask: why isn't it 10x?
  • 93.
    Active Benchmarking •  Rootcause performance analysis while the benchmark is still running –  Use observability tools –  Identify the limiter (or suspected limiter) and include it with the benchmark results –  Answer: why not 10x? •  This takes time, but uncovers most mistakes
  • 94.
  • 95.
    Micro Benchmarks •  Testa specific function in isolation. e.g.: –  File system maximum cached read operations/sec –  Network maximum throughput •  Examples of bad microbenchmarks: –  gitpid() in a tight loop –  speed of /dev/zero and /dev/null •  Common problems: –  Testing a workload that is not very relevant –  Missing other workloads that are relevant
  • 96.
  • 97.
    Macro Benchmarks •  Simulateapplication user load. e.g.: –  Simulated web client transaction •  Common problems: –  Misplaced trust: believed to be realistic, but misses variance, errors, perturbations, e.t.c. –  Complex to debug, verify, and root cause
  • 98.
  • 99.
    Kitchen Sink Benchmarks • Run everything! –  Mostly random benchmarks found on the Internet, where most are are broken or irrelevant –  Developers focus on collecting more benchmarks than verifying or fixing the existing ones •  Myth that more benchmarks == greater accuracy –  No, use active benchmarking (analysis)
  • 100.
  • 101.
    Automated Benchmarks •  Completelyautomated procedure. e.g.: –  Cloud benchmarks: spin up an instance, benchmark, destroy. Automate. •  Little or no provision for debugging •  Automation is only part of the solution
  • 102.
  • 103.
  • 104.
    bonnie++ •  "simple testsof hard drive and file system performance" •  First metric printed by (thankfully) older versions: per character sequential output •  What was actually tested: –  1 byte writes to libc (via putc()) –  4 Kbyte writes from libc -> FS (depends on OS; see setbuffer()) –  128 Kbyte async writes to disk (depends on storage stack) –  Any file system throttles that may be present (eg, ZFS) –  C++ code, to some extent (bonnie++ 10% slower than Bonnie) •  Actual limiter: –  Single threaded write_block_putc() and putc() calls
  • 105.
  • 106.
    Apache Bench •  HTTPweb server benchmark •  Single thread limited (use wrk for multi-threaded) •  Keep-alive option (-k): –  without: Can become an unrealistic TCP session benchmark –  with: Can become an unrealistic server throughput test •  Performance issues of ab's own code
  • 107.
  • 108.
    UnixBench •  The originalkitchen-sink micro benchmark from 1984, published in BYTE magazine •  Innovative & useful for the time, but that time has passed •  More problems than I can shake a stick at •  Starting with…
  • 109.
  • 110.
    UnixBench Makefile •  Default(by ./Run) for Linux. Would you edit it? Then what? ## Very generic #OPTON = -O ## For Linux 486/Pentium, GCC 2.7.x and 2.8.x #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math # -m486 -malign-loops=2 -malign-jumps=2 -malign-functions=2 ## For Linux, GCC previous to 2.7.0 #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math -m486 #OPTON = -O2 -fomit-frame-pointer -fforce-addr -fforce-mem -ffast-math # -m386 -malign-loops=1 -malign-jumps=1 -malign-functions=1 ## For Solaris 2, or general-purpose GCC 2.7.x OPTON = -O2 -fomit-frame-pointer -fforce-addr -ffast-math -Wall ## For Digital Unix v4.x, with DEC cc v5.x #OPTON = -O4 #CFLAGS = -DTIME -std1 -verbose -w0
  • 111.
    UnixBench Makefile •  "Fixing"the Makefile improved the first result, Dhrystone 2, by 64% •  Is everyone "fixing" it the same way, or not? Are they using the same compiler version? Same OS? (No.)
  • 112.
    UnixBench Documentation "The resultswill depend not only on your hardware, but on your operating system, libraries, and even compiler." "So you may want to make sure that all your test systems are running the same version of the OS; or at least publish the OS and compuiler versions with your results."
  • 113.
  • 114.
    UnixBench Tests •  Resultssummarized as "The BYTE Index". From USAGE: •  What can go wrong? Everything. system: dhry2reg Dhrystone 2 using register variables whetstone-double Double-Precision Whetstone syscall System Call Overhead pipe Pipe Throughput context1 Pipe-based Context Switching spawn Process Creation execl Execl Throughput fstime-w File Write 1024 bufsize 2000 maxblocks fstime-r File Read 1024 bufsize 2000 maxblocks fstime File Copy 1024 bufsize 2000 maxblocks fsbuffer-w File Write 256 bufsize 500 maxblocks fsbuffer-r File Read 256 bufsize 500 maxblocks fsbuffer File Copy 256 bufsize 500 maxblocks fsdisk-w File Write 4096 bufsize 8000 maxblocks fsdisk-r File Read 4096 bufsize 8000 maxblocks fsdisk File Copy 4096 bufsize 8000 maxblocks shell1 Shell Scripts (1 concurrent) (runs "looper 60 multi.sh 1") shell8 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 8") shell16 Shell Scripts (8 concurrent) (runs "looper 60 multi.sh 16")
  • 115.
  • 116.
  • 117.
    Street Light Anti-Method 1. Pick observability tools that are: –  Familiar –  Found on the Internet –  Found at random 2.  Run tools 3.  Look for obvious issues
  • 118.
    Blame Someone ElseAnti-Method 1.  Find a system or environment component you are not responsible for 2.  Hypothesize that the issue is with that component 3.  Redirect the issue to the responsible team 4.  When proven wrong, go to 1
  • 119.
    Performance Tools Team • Having a separate performance tools team, who creates tools but doesn't use them (no production exposure) •  At Netflix: –  The performance engineering team builds tools and uses tools for both service consulting and live production triage •  Mogul, Vector, … –  Other teams (CORE, traffic, …) also build performance tools and use them during issues •  Good performance tools are built out of necessity
  • 120.
    Messy House Fallacy • Fallacy: my code is a mess, I bet yours is immaculate, therefore the bug must be mine •  Reality: everyone's code is terrible and buggy •  When analyzing performance, don't overlook the system: kernel, libraries, etc.
  • 121.
  • 122.
  • 123.
    Observability •  Trust nothing,verify everything –  Cross-check with other observability tools –  Write small "known" workloads, and confirm metrics match –  Find other sanity tests: e.g. check known system limits –  Determine how metrics are calculated, averaged, updated •  Find metrics to solve problems –  Instead of understanding hundreds of system metrics –  What problems do you want to observe? What metrics would be sufficient? Find, verify, and use those. e.g., USE Method. –  The metric you want may not yet exist •  File bugs, get these fixed/improved
  • 124.
    Observe Everything •  Usefunctional diagrams to pose Q's and find missing metrics:
  • 125.
    Java Mixed-Mode FlameGraph Profile Everything Java JVM Kernel GC
  • 126.
  • 127.
    Benchmark Nothing •  Trustnothing, verify everything •  Do Active Benchmarking: 1.  Configure the benchmark to run in steady state, 24x7 2.  Do root-cause analysis of benchmark performance 3.  Answer: why is it not 10x?
  • 128.
    Links & References • https://www.rfc-editor.org/rfc/rfc546.pdf •  https://upload.wikimedia.org/wikipedia/commons/6/64/Intel_Nehalem_arch.svg •  http://www.linuxatemyram.com/ •  http://dtrace.org/blogs/brendan/2011/02/18/dtrace-pid-provider-overhead/ •  Traeger, A., E. Zadok, N. Joukov, and C. Wright. “A Nine Year Study of File System and Storage Benchmarking,” ACM Trans- actions on Storage, 2008. •  http://www.brendangregg.com/blog/2014-06-09/java-cpu-sampling-using-hprof.html •  http://www.brendangregg.com/blog/2014-05-03/the-benchmark-paradox.html •  http://www.brendangregg.com/ActiveBenchmarking/bonnie++.html •  https://blogs.oracle.com/roch/entry/decoding_bonnie •  http://www.brendangregg.com/blog/2014-05-02/compilers-love-messing-with- benchmarks.html •  https://code.google.com/p/byte-unixbench/ •  https://qconsf.com/sf2015/presentation/how-not-measure-latency •  https://qconsf.com/sf2015/presentation/profilers-lying •  Caution signs drawn be me, inspired by real-world signs
  • 129.
  • 130.
    Thanks •  Questions? •  http://techblog.netflix.com • http://slideshare.net/brendangregg •  http://www.brendangregg.com •  bgregg@netflix.com •  @brendangregg Nov  2015  
  • 131.
    Watch the videowith slide synchronization on InfoQ.com! http://www.infoq.com/presentations/broken- performance-tools