SlideShare a Scribd company logo
the art of
performance tuning
Jonathan Ross
the art of
performance tuning
Jonathan Ross
science
engineering
trading company

big in options and futures 

based in Amsterdam
imc Trading
me
Playing with JVMs since 1997

Paid to do so by IMC Chicago

Theoretical physics background

CJUG board member

JCP associate member
@JoroRoss
it’s too slow
have you tried profiling it?
it’s too slow
this talk
Part 1: theory
• challenges

• methodology

• implementation

Part 2: practice
• how to use a profiler

• hands-on jmh benchmarks
this talk
is not about gc tuning
Part 1: theory
• challenges

• methodology

• implementation

Part 2: practice
• how to use a profiler

• hands-on jmh benchmarks
Part 1: theory
challenges
objects

modern cpus

lack of determinism

virtualization

tooling
ignorance

confirmation bias

premature optimisation

over-engineering

legacy
challenges
ignorance

confirmation bias

premature optimisation

over-engineering

legacy
objects

modern cpus

lack of determinism

virtualization

tooling
Methodology
find a (proxy) metric for 

implement 

automate
“success”

a way of measuring it

a way of monitoring it
implementation
make it easy to 

find root causes using





fix and add regression
tests using

learn to use and
understand





reproduce issues

production metrics

macro benchmarks

micro benchmarks

profilers

implementation
make it easy to 

find root causes using





fix and add regression
tests using

learn to use and
understand

if the going gets tough



reproduce issues

production metrics

macro benchmarks

micro benchmarks

profilers

-XX+UnlockDiagnosticVMOptions

-XX:+LogCompilation

-XX:+PrintAssembly
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir• try harder
• rearchitect
• seek new job
erm…
handy-dandy flowchart
production
metrics
reproduce
in benchmark
profile
implement
fix
metrics
better?
ditch fix
nope
yup
good
enough?
nope
merge and verify
in production
yup
?
find
bottleneck?
yes sir
1. Only work on improving performance
of actual bottle-necks
2. Don’t keep a fix unless it improves
performance (or makes the code
better)
the main take-away:
Part 2: practice
A STORY OF ATAN
a (redacted) real world performance

tuning adventure involving my love for 

math and some short-sightedness
it’s too
slow
let me try to profile it
To profile a production system, start your server with the JVM argument:

-agentpath:<profiler directory>/bin/linux-x86-64/libyjpagent.so
stack telemetry (Yourkit)
weeding out irrelevant data
(tip: turn off filters by default)
export threads were not supposed to be this busy…
ordering callees by “own time” shows us the bottleneck
where is the bottleneck called from?
select method back-traces
(wouldn’t you like to know!)
2 bottlenecks for the price of one
my favorite feature of Yourkit: focusing on sub-set of data
eureka!
next favorite feature: merged callees
To the micro benchmarks!
(demo)
normalizedAtan = 2
⇡ tan 1 ⇡
2 x
tan 1
(x) ⇡
⇡(0.596227x + x2
)
2(1 + 1.192524x + x2)
Back To the micro benchmarks!
(demo)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2164.391 ± 49.520 us/op
original
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2043.580 ± 80.292 us/op
approximate, outside loop (6%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 1991.929 ± 50.966 us/op
approximate (8% speedup)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2045.587 ± 55.951 us/op
precise, outside loop (6%)
Java mission control
(demo)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 2393.484 ± 111.872 us/op
original
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 1163.414 ± 17.417 us/op
map of maps (52%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 660.802 ± 5.795 us/op
mutable double (72%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 894.116 ± 9.971 us/op
map of maps (2) (63%)
Benchmark Mode Cnt Score Error Units
FoobaratorBenchmark.benchmark avgt 100 547.549 ± 4.122 us/op
atan outside loop again (77%)
challenges revisited
ignorance

confirmation bias

premature optimisation

overengineering

legacy
objects

modern cpus

lack of determinism

virtualization

tooling
?
@JoroRoss

www.imc.com

www.cjug.org
http://openjdk.java.net/projects/code-tools/jmh/

http://hirt.se - Marcus Hirt’s blog on JMC/JFR

https://visualvm.github.io
The demo code is available at

https://github.com/JoroRoss/art-of-performance

More Related Content

Similar to The Art Of Performance Tuning

Optimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone MLOptimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone ML
Spark Summit
 
Tales of Linux micro-benchmarks
Tales of Linux micro-benchmarksTales of Linux micro-benchmarks
Tales of Linux micro-benchmarks
Matt Fleming
 
When Should I Use Simulation?
When Should I Use Simulation?When Should I Use Simulation?
When Should I Use Simulation?
SIMUL8 Corporation
 
Grow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM StackGrow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM Stack
KeitaSugiyama1
 
Crossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCsCrossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCs
Andreas Koschak
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
Lionel Briand
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
SAP HANA Cloud Platform
 
From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014
Gloria Lovera
 
(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back
David Rodenas
 
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Jan Margeta
 
Sista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performanceSista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performance
ESUG
 
Incremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software EngineeringIncremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software Engineering
Ákos Horváth
 
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStormServerless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
Dmitri Zimine
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)
Hiroto Honda
 
Advanced Production Debugging
Advanced Production DebuggingAdvanced Production Debugging
Advanced Production Debugging
Takipi
 
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experimentsSFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
South Tyrol Free Software Conference
 
computer architecture.
computer architecture.computer architecture.
computer architecture.
Shivalik college of engineering
 

Similar to The Art Of Performance Tuning (20)

10 Ways To Improve Your Code
10 Ways To Improve Your Code10 Ways To Improve Your Code
10 Ways To Improve Your Code
 
Optimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone MLOptimizing Terascale Machine Learning Pipelines with Keystone ML
Optimizing Terascale Machine Learning Pipelines with Keystone ML
 
Tales of Linux micro-benchmarks
Tales of Linux micro-benchmarksTales of Linux micro-benchmarks
Tales of Linux micro-benchmarks
 
When Should I Use Simulation?
When Should I Use Simulation?When Should I Use Simulation?
When Should I Use Simulation?
 
Grow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM StackGrow and Shrink - Dynamically Extending the Ruby VM Stack
Grow and Shrink - Dynamically Extending the Ruby VM Stack
 
Crossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCsCrossing the Boundaries: Development Strategies for (P)SoCs
Crossing the Boundaries: Development Strategies for (P)SoCs
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
 
From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014From Black Box to Black Magic, Pycon Ireland 2014
From Black Box to Black Magic, Pycon Ireland 2014
 
Py conie 2014
Py conie 2014Py conie 2014
Py conie 2014
 
(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back(automatic) Testing: from business to university and back
(automatic) Testing: from business to university and back
 
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
Distributed computing with Ray. Find your hyper-parameters, speed up your Pan...
 
Sista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performanceSista: Improving Cog’s JIT performance
Sista: Improving Cog’s JIT performance
 
Incremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software EngineeringIncremental Model Queries for Model-Dirven Software Engineering
Incremental Model Queries for Model-Dirven Software Engineering
 
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStormServerless on OpenStack with Docker Swarm, Mistral, and StackStorm
Serverless on OpenStack with Docker Swarm, Mistral, and StackStorm
 
Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)Tackling Open Images Challenge (2019)
Tackling Open Images Challenge (2019)
 
Java Micro-Benchmarking
Java Micro-BenchmarkingJava Micro-Benchmarking
Java Micro-Benchmarking
 
Advanced Production Debugging
Advanced Production DebuggingAdvanced Production Debugging
Advanced Production Debugging
 
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experimentsSFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
SFScon 2020 - Hlib Babii - DVC version control your datasets and ML experiments
 
computer architecture.
computer architecture.computer architecture.
computer architecture.
 

Recently uploaded

2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
Georgi Kodinov
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Globus
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
Paco van Beckhoven
 
BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024
Ortus Solutions, Corp
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 
Pro Unity Game Development with C-sharp Book
Pro Unity Game Development with C-sharp BookPro Unity Game Development with C-sharp Book
Pro Unity Game Development with C-sharp Book
abdulrafaychaudhry
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
wottaspaceseo
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
Cyanic lab
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdfVitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
Google
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
e20449
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
Globus
 
Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"
Donna Lenk
 

Recently uploaded (20)

2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
 
Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024Globus Connect Server Deep Dive - GlobusWorld 2024
Globus Connect Server Deep Dive - GlobusWorld 2024
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
 
BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 
Pro Unity Game Development with C-sharp Book
Pro Unity Game Development with C-sharp BookPro Unity Game Development with C-sharp Book
Pro Unity Game Development with C-sharp Book
 
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket ManagementUtilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
Utilocate provides Smarter, Better, Faster, Safer Locate Ticket Management
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
Vitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdfVitthal Shirke Java Microservices Resume.pdf
Vitthal Shirke Java Microservices Resume.pdf
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
 
How to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good PracticesHow to Position Your Globus Data Portal for Success Ten Good Practices
How to Position Your Globus Data Portal for Success Ten Good Practices
 
Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"Navigating the Metaverse: A Journey into Virtual Evolution"
Navigating the Metaverse: A Journey into Virtual Evolution"
 

The Art Of Performance Tuning