SlideShare a Scribd company logo
1 of 36
Download to read offline
the art of
performance tuning
Jonathan Ross
the art of
performance tuning
Jonathan Ross
science
engineering
trading company

big in options and futures 

based in Amsterdam
imc Trading
me
Playing with JVMs since 1997

Paid to do so by IMC Chicago

Theoretical physics background

CJUG board member

JCP associate member
@JoroRoss
it’s too slow
have you tried profiling it?
it’s too slow
this talk
Part 1: theory
• challenges

• methodology

• implementation

Part 2: practice
• how to use a profiler

• hands-on jmh benchmarks
this talk
is not about gc tuning
Part 1: theory
• challenges

• methodology

• implementation

Part 2: practice
• how to use a profiler

• hands-on jmh benchmarks
Part 1: theory
challenges
objects

modern cpus

lack of determinism

virtualization

tooling
ignorance

confirmation bias

premature optimisation

over-engineering

legacy
challenges
ignorance

confirmation bias

premature optimisation

over-engineering

legacy
objects

modern cpus

lack of determinism

virtualization

tooling
Methodology
find a (proxy) metric for 

implement 

automate
“success”

a way of measuring it

a way of monitoring it
implementation
make it easy to 

find root causes using





fix and add regression
tests using

learn to use and
understand





reproduce issues

production metrics

macro benchmarks

micro benchmarks

profilers

implementation
make it easy to 

find root causes using





fix and add regression
tests using

learn to use and
understand

if the going gets tough



reproduce issues

production metrics

macro benchmarks

micro benchmarks

profilers

-XX+UnlockDiagnosticVMOptions

-XX:+LogCompilation

-XX:+PrintAssembly
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir• try harder
• rearchitect
• seek new job
erm…
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir
1. Only work on improving performance
of actual bottle-necks
2. Don’t keep a fix unless it improves
performance (or makes the code
better)
the main take-away:
Part 2: practice
A STORY OF ATAN
a (redacted) real world performance

tuning adventure involving my love for 

math and some short-sightedness
it’s too
slow
let me try to profile it
To profile a production system, start your server with the JVM argument:

-agentpath:<profiler directory>/bin/linux-x86-64/libyjpagent.so
stack telemetry (Yourkit)
weeding out irrelevant data
(tip: turn off filters by default)
export threads were not supposed to be this busy…
ordering callees by “own time” shows us the bottleneck
where is the bottleneck called from?
select method back-traces
(wouldn’t you like to know!)
2 bottlenecks for the price of one
my favorite feature of Yourkit: focusing on sub-set of data
eureka!
next favorite feature: merged callees
To the micro benchmarks!
(demo)
normalizedAtan = 2
⇡ tan 1 ⇡
2 x
tan 1
(x) ⇡
⇡(0.596227x + x2
)
2(1 + 1.192524x + x2)
Back To the micro benchmarks!
(demo)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2164.391 ± 49.520 us/op
original
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2043.580 ± 80.292 us/op
approximate, outside loop (6%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 1991.929 ± 50.966 us/op
approximate (8% speedup)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2045.587 ± 55.951 us/op
precise, outside loop (6%)
Java mission control
(demo)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2393.484 ± 111.872 us/op
original
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 1163.414 ± 17.417 us/op
map of maps (52%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 660.802 ± 5.795 us/op
mutable double (72%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 894.116 ± 9.971 us/op
map of maps (2) (63%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 547.549 ± 4.122 us/op
atan outside loop again (77%)
challenges revisited
ignorance

confirmation bias

premature optimisation

overengineering

legacy
objects

modern cpus

lack of determinism

virtualization

tooling
?
@JoroRoss

www.imc.com

www.cjug.org
http://openjdk.java.net/projects/code-tools/jmh/

http://hirt.se - Marcus Hirt’s blog on JMC/JFR

https://visualvm.github.io
The demo code is available at

https://github.com/JoroRoss/art-of-performance

More Related Content

Similar to The Art Of Performance Tuning

Optimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone MLOptimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone MLSpark Summit
 
Tales of Linux micro-benchmarks
Tales of Linux micro-benchmarksTales of Linux micro-benchmarks
Tales of Linux micro-benchmarksMatt Fleming
 
Grow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM StackGrow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM StackKeitaSugiyama1
 
Crossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCsCrossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCsAndreas Koschak
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand
 
From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014Gloria Lovera
 
(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and backDavid Rodenas
 
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Jan Margeta
 
Sista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performanceSista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performanceESUG
 
Incremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software EngineeringIncremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software EngineeringÁkos Horváth
 
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStormServerless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStormDmitri Zimine
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Hiroto Honda
 
Advanced Production Debugging
Advanced Production DebuggingAdvanced Production Debugging
Advanced Production DebuggingTakipi
 
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experimentsSFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experimentsSouth Tyrol Free Software Conference
 

Similar to The Art Of Performance Tuning (20)

10 Ways To Improve Your Code
10 Ways To Improve Your Code10 Ways To Improve Your Code
10 Ways To Improve Your Code
 
Optimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone MLOptimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone ML
 
Tales of Linux micro-benchmarks
Tales of Linux micro-benchmarksTales of Linux micro-benchmarks
Tales of Linux micro-benchmarks
 
When Should I Use Simulation?
When Should I Use Simulation?When Should I Use Simulation?
When Should I Use Simulation?
 
Grow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM StackGrow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM Stack
 
Crossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCsCrossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCs
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
 
From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014
 
Py conie 2014
Py conie 2014Py conie 2014
Py conie 2014
 
(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back
 
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
 
Sista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performanceSista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performance
 
Incremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software EngineeringIncremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software Engineering
 
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStormServerless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)
 
Java Micro-Benchmarking
Java Micro-BenchmarkingJava Micro-Benchmarking
Java Micro-Benchmarking
 
Advanced Production Debugging
Advanced Production DebuggingAdvanced Production Debugging
Advanced Production Debugging
 
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experimentsSFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
 
computer architecture.
computer architecture.computer architecture.
computer architecture.
 

Recently uploaded

What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxTechnogeeks
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...OnePlan Solutions
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dbaRemote DBA Services
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUsamruddhijedgule2004
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxRemote DBA Services
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfkalichargn70th171
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Reviewjw364beach
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapIshara Amarasekera
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...Bert Jan Schrijver
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxAS Design & AST.
 

Recently uploaded (20)

What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docx
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dba
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptx
 
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdfPros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
Pros and Cons of Selenium In Automation Testing_ A Comprehensive Assessment.pdf
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery Roadmap
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
Mastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptxMastering Project Planning with Microsoft Project 2016.pptx
Mastering Project Planning with Microsoft Project 2016.pptx
 

The Art Of Performance Tuning