This document outlines 10 common mistakes made when performing performance tuning: 1) Setting up a tiger team to diagnose issues by committee, 2) Giving a system more hardware without understanding the underlying bottleneck, 3) Having poor system visibility that provides too much unclear monitoring data, 4) Diving into code without properly identifying the primary bottleneck, 5) Not being able to profile a production system due to statistical sampling issues, 6) Not listening to signals from the system about where problems exist, 7) Assuming performance issues are due to known system behaviors without properly diagnosing, 8) Tuning based on settings from Google without understanding the specific system context, 9) Tuning based on folklore from blogs without empirical evidence, and 10) Not

1. Performance Tuning
You’re doing it wrong!

2. Number 10:
No performance
requirements
• No requirements == no problem

3. Number 9:
Tuning by folklore
• Performance happens in a context
• admin manuals are devoid of
context
• blogs don’t speak to your context
®
• Measure don’t guess
• use measurements to guide all
decisions

4. Number 8:
Tuning by Google
-XX:InitialHeapSize=1610612736 -XX:MaxHeapSize=2147483648
-XX:MaxNewSize=697933824 -XX:NewSize=697933824 -XX:OldSize=1395867648
-XX:OldPLABSize=16
-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution
-XX:+PrintGCApplicationStoppedTime
-XX:SurvivorRatio=1 -XX:TargetSurvivorRatio=90
-XX:-UseAdaptiveSizePolicy -XX:+PrintAdaptiveSizePolicy
-XX:+UseCompressedClassPointers -XX:+UseCompressedOops
-XX:-UseLargePagesIndividualAllocation
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled

5. Number 7:
That is how my system
works
• Often the diagnosis takes 15
minutes
• convincing the client that the
implement is a problem can
take hours!!!

6. Number 6:
Not listening to the
system
• Presented groups of developers
with a performance problem
• not one group has solved it in 8
years!!!
• once they learn to read the
signals the problem can be
diagnosed in minutes!

7. Number 5:
Not being able to profile
in prod • Would you let your doctor diagnose you
by running the tests on a newt?
• Profiling is statistical sampling technique
• Performance bottleneck is a statistically
significant event
• Profilers dilute the answer with
expensive to capture useless data
points

8. Number 4:
Diving into the code
• “Know” what the problem is based on local
knowledge
• even if you tell them what the problem is
they won’t change course!!!
• In one exercise >97% of developers fail to
identify and fix the primary bottleneck
• even when they’ve been emphatically told
where they will fail

9. Number 3:
Poor system visibility
• Monitoring pukes a ton of data into your
lap
• often leave people scratching their
heads trying to understand where to
start!!!
• Often app is only logging
• logs are optimized for development, not
ops

10. Number 2:
Give it more hardware
• If you don’t understand your bottleneck…..
• Won’t work if you can’t use the hardware
you already have
• Seemingly perfectly parallelizable batch job
needed to run in 1 hour
• single server took 4 hours
• 4 servers took more than 8 hours to
complete!!!

11. Number 1:
Setting up a tiger team
• Seriously!!! resort to diagnosis by
committee?
• Clear indication that your team is
using technology that either don’t
understand or can’t manage
• Nice way to trigger “tribal” behavior.

12. jClarity Illuminate
Performance Diagnostic
Engine
• Picks up where monitoring stops
• Minimize ambiguity
• uses heuristics to identify performance
bottlenecks in live system
• points out casual execution path!
• Profiles with minimal impact on (already
poor) performance