Submit Search
Upload
Benchmarking - you're doing it wrong - Aysylu Greenberg
•
0 likes
•
218 views
J
JAXLondon_Conference
Follow
JAX London 2015 talk
Read less
Read more
Software
Report
Share
Report
Share
1 of 86
Download now
Download to read offline
Recommended
TDC2016SP - Otimização Prematura: a Raíz de Todo o Mal
TDC2016SP - Otimização Prematura: a Raíz de Todo o Mal
tdc-globalcode
Premature optimisation: The Root of All Evil
Premature optimisation: The Root of All Evil
Fabio Akita
SEO Live! Higher Ed Web SE
SEO Live! Higher Ed Web SE
Fathom: A Digital Marketing Agency
Lesson 6 influences on tate perception
Lesson 6 influences on tate perception
coburgpsych
Cassandra and Spark - Tim Berglund
Cassandra and Spark - Tim Berglund
JAXLondon_Conference
GIF Storytelling-sp15
GIF Storytelling-sp15
phdserena
Ehab_ahmed
Ehab_ahmed
Ehab Ahmed
συμβολη της παιδοδοντιατρικης στο παιδι με σχιστια
συμβολη της παιδοδοντιατρικης στο παιδι με σχιστια
Katerina Koleventi
Recommended
TDC2016SP - Otimização Prematura: a Raíz de Todo o Mal
TDC2016SP - Otimização Prematura: a Raíz de Todo o Mal
tdc-globalcode
Premature optimisation: The Root of All Evil
Premature optimisation: The Root of All Evil
Fabio Akita
SEO Live! Higher Ed Web SE
SEO Live! Higher Ed Web SE
Fathom: A Digital Marketing Agency
Lesson 6 influences on tate perception
Lesson 6 influences on tate perception
coburgpsych
Cassandra and Spark - Tim Berglund
Cassandra and Spark - Tim Berglund
JAXLondon_Conference
GIF Storytelling-sp15
GIF Storytelling-sp15
phdserena
Ehab_ahmed
Ehab_ahmed
Ehab Ahmed
συμβολη της παιδοδοντιατρικης στο παιδι με σχιστια
συμβολη της παιδοδοντιατρικης στο παιδι με σχιστια
Katerina Koleventi
The legal factors
The legal factors
FutureTDM
Measurement
Measurement
phdserena
Perceptual constancy and set
Perceptual constancy and set
coburgpsych
Siete dolores ang pitong hapis ng mahal
Siete dolores ang pitong hapis ng mahal
majumalon
IoT Attack Surfaces -- DEFCON 2015
IoT Attack Surfaces -- DEFCON 2015
Daniel Miessler
IoT Server - Device Type Plugin
IoT Server - Device Type Plugin
Charitha Goonetilleke
Perfect Competition
Perfect Competition
tutor2u
Oligopoly - The Kinked Demand Curve
Oligopoly - The Kinked Demand Curve
tutor2u
Benchmarking (DevNexus 2015)
Benchmarking (DevNexus 2015)
Aysylu Greenberg
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Aysylu Greenberg
Benchmarking (RICON 2014)
Benchmarking (RICON 2014)
Aysylu Greenberg
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
CanadaHelps / MyCharityConnects
ADMSP Introduction to Social Media Measurement
ADMSP Introduction to Social Media Measurement
Altos del Mar Sculpture Park
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with Snowplow
Giuseppe Gaviani
Conexão Kinghost - Otimização Prematura
Conexão Kinghost - Otimização Prematura
Fabio Akita
EscConf - Deep Dive Frontend Optimization
EscConf - Deep Dive Frontend Optimization
Jonathan Klein
Trending with Purpose
Trending with Purpose
Jason Dixon
Nondeterministic Software for the Rest of Us
Nondeterministic Software for the Rest of Us
Tomer Gabel
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
Stacy Deere
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
Clix Marketing
Front end performance on Shopify.com
Front end performance on Shopify.com
Bryson Gilbert
Zipline - A Declarative Feature Engineering Framework
Zipline - A Declarative Feature Engineering Framework
Databricks
More Related Content
Viewers also liked
The legal factors
The legal factors
FutureTDM
Measurement
Measurement
phdserena
Perceptual constancy and set
Perceptual constancy and set
coburgpsych
Siete dolores ang pitong hapis ng mahal
Siete dolores ang pitong hapis ng mahal
majumalon
IoT Attack Surfaces -- DEFCON 2015
IoT Attack Surfaces -- DEFCON 2015
Daniel Miessler
IoT Server - Device Type Plugin
IoT Server - Device Type Plugin
Charitha Goonetilleke
Perfect Competition
Perfect Competition
tutor2u
Oligopoly - The Kinked Demand Curve
Oligopoly - The Kinked Demand Curve
tutor2u
Viewers also liked
(8)
The legal factors
The legal factors
Measurement
Measurement
Perceptual constancy and set
Perceptual constancy and set
Siete dolores ang pitong hapis ng mahal
Siete dolores ang pitong hapis ng mahal
IoT Attack Surfaces -- DEFCON 2015
IoT Attack Surfaces -- DEFCON 2015
IoT Server - Device Type Plugin
IoT Server - Device Type Plugin
Perfect Competition
Perfect Competition
Oligopoly - The Kinked Demand Curve
Oligopoly - The Kinked Demand Curve
Similar to Benchmarking - you're doing it wrong - Aysylu Greenberg
Benchmarking (DevNexus 2015)
Benchmarking (DevNexus 2015)
Aysylu Greenberg
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Aysylu Greenberg
Benchmarking (RICON 2014)
Benchmarking (RICON 2014)
Aysylu Greenberg
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
CanadaHelps / MyCharityConnects
ADMSP Introduction to Social Media Measurement
ADMSP Introduction to Social Media Measurement
Altos del Mar Sculpture Park
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with Snowplow
Giuseppe Gaviani
Conexão Kinghost - Otimização Prematura
Conexão Kinghost - Otimização Prematura
Fabio Akita
EscConf - Deep Dive Frontend Optimization
EscConf - Deep Dive Frontend Optimization
Jonathan Klein
Trending with Purpose
Trending with Purpose
Jason Dixon
Nondeterministic Software for the Rest of Us
Nondeterministic Software for the Rest of Us
Tomer Gabel
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
Stacy Deere
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
Clix Marketing
Front end performance on Shopify.com
Front end performance on Shopify.com
Bryson Gilbert
Zipline - A Declarative Feature Engineering Framework
Zipline - A Declarative Feature Engineering Framework
Databricks
Introduction to Google Analytics
Introduction to Google Analytics
Mary Kay Jerige Lofurno
Digital Data Tips Tuesday
Digital Data Tips Tuesday
Martijn Scheijbeler
Serverless: What happens next will blow your mind!
Serverless: What happens next will blow your mind!
Chris Williams
DevCommerce Conference 2016: Performance, anti-patterns e stacks pra desenvol...
DevCommerce Conference 2016: Performance, anti-patterns e stacks pra desenvol...
iMasters
Advanced Guide to Seo (Third Sector - Leeds Digital Festival 2016)
Advanced Guide to Seo (Third Sector - Leeds Digital Festival 2016)
Dan Taylor
SEO Analysis for Life Insurance Website - SCBLife Case Study
SEO Analysis for Life Insurance Website - SCBLife Case Study
TNC Digital
Similar to Benchmarking - you're doing it wrong - Aysylu Greenberg
(20)
Benchmarking (DevNexus 2015)
Benchmarking (DevNexus 2015)
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Benchmarking: You're Doing It Wrong (StrangeLoop 2014)
Benchmarking (RICON 2014)
Benchmarking (RICON 2014)
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
Mark Farmer - Google Analytics: Business Intelligence for Non-profits
ADMSP Introduction to Social Media Measurement
ADMSP Introduction to Social Media Measurement
How Gousto is moving to just-in-time personalization with Snowplow
How Gousto is moving to just-in-time personalization with Snowplow
Conexão Kinghost - Otimização Prematura
Conexão Kinghost - Otimização Prematura
EscConf - Deep Dive Frontend Optimization
EscConf - Deep Dive Frontend Optimization
Trending with Purpose
Trending with Purpose
Nondeterministic Software for the Rest of Us
Nondeterministic Software for the Rest of Us
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
Build It And They Will Come: User Adoption SharePoint 2013 (SPS Charlotte)
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
3 AdWords Reports You Didn't Know About (Or Didn't Know What To Do With)
Front end performance on Shopify.com
Front end performance on Shopify.com
Zipline - A Declarative Feature Engineering Framework
Zipline - A Declarative Feature Engineering Framework
Introduction to Google Analytics
Introduction to Google Analytics
Digital Data Tips Tuesday
Digital Data Tips Tuesday
Serverless: What happens next will blow your mind!
Serverless: What happens next will blow your mind!
DevCommerce Conference 2016: Performance, anti-patterns e stacks pra desenvol...
DevCommerce Conference 2016: Performance, anti-patterns e stacks pra desenvol...
Advanced Guide to Seo (Third Sector - Leeds Digital Festival 2016)
Advanced Guide to Seo (Third Sector - Leeds Digital Festival 2016)
SEO Analysis for Life Insurance Website - SCBLife Case Study
SEO Analysis for Life Insurance Website - SCBLife Case Study
More from JAXLondon_Conference
All change! How the new Economics of Cloud will make you think differently ab...
All change! How the new Economics of Cloud will make you think differently ab...
JAXLondon_Conference
The Unit Test is dead. Long live the Unit Test! - Colin Vipurs
The Unit Test is dead. Long live the Unit Test! - Colin Vipurs
JAXLondon_Conference
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
JAXLondon_Conference
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
JAXLondon_Conference
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
JAXLondon_Conference
Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
JAXLondon_Conference
VC from the inside - a techie's perspective - Adrian Colyer
VC from the inside - a techie's perspective - Adrian Colyer
JAXLondon_Conference
Use your type system; write less code - Samir Talwar
Use your type system; write less code - Samir Talwar
JAXLondon_Conference
Thinking fast and slow with software development - Daniel Bryant
Thinking fast and slow with software development - Daniel Bryant
JAXLondon_Conference
The java memory model and the mutability matrix of pain - Jamie Allen
The java memory model and the mutability matrix of pain - Jamie Allen
JAXLondon_Conference
The art of shifting perspectives - Rachel Davies
The art of shifting perspectives - Rachel Davies
JAXLondon_Conference
Spring Boot in the Web Tier - Dave Syer
Spring Boot in the Web Tier - Dave Syer
JAXLondon_Conference
Microservices from dream to reality in an hour - Dr. Holly Cummins
Microservices from dream to reality in an hour - Dr. Holly Cummins
JAXLondon_Conference
Love your architecture - Alexander von Zitzewitz
Love your architecture - Alexander von Zitzewitz
JAXLondon_Conference
Lambdas puzzler - Peter Lawrey
Lambdas puzzler - Peter Lawrey
JAXLondon_Conference
Java vs. Java Script for enterprise web applications - Chris Bailey
Java vs. Java Script for enterprise web applications - Chris Bailey
JAXLondon_Conference
Java generics past, present and future - Raoul-Gabriel Urma, Richard Warburton
Java generics past, present and future - Raoul-Gabriel Urma, Richard Warburton
JAXLondon_Conference
Java 8 best practices - Stephen Colebourne
Java 8 best practices - Stephen Colebourne
JAXLondon_Conference
Intuitions for scaling data centric architectures - Benjamin Stopford
Intuitions for scaling data centric architectures - Benjamin Stopford
JAXLondon_Conference
Hybrid solutions – combining in memory solutions with SSD - Christos Erotocritou
Hybrid solutions – combining in memory solutions with SSD - Christos Erotocritou
JAXLondon_Conference
More from JAXLondon_Conference
(20)
All change! How the new Economics of Cloud will make you think differently ab...
All change! How the new Economics of Cloud will make you think differently ab...
The Unit Test is dead. Long live the Unit Test! - Colin Vipurs
The Unit Test is dead. Long live the Unit Test! - Colin Vipurs
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
Stop guessing, start testing – mobile testing done right - Timo Euteneuer
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Java Generics Past, Present and Future - Richard Warburton, Raoul-Gabriel Urma
Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
VC from the inside - a techie's perspective - Adrian Colyer
VC from the inside - a techie's perspective - Adrian Colyer
Use your type system; write less code - Samir Talwar
Use your type system; write less code - Samir Talwar
Thinking fast and slow with software development - Daniel Bryant
Thinking fast and slow with software development - Daniel Bryant
The java memory model and the mutability matrix of pain - Jamie Allen
The java memory model and the mutability matrix of pain - Jamie Allen
The art of shifting perspectives - Rachel Davies
The art of shifting perspectives - Rachel Davies
Spring Boot in the Web Tier - Dave Syer
Spring Boot in the Web Tier - Dave Syer
Microservices from dream to reality in an hour - Dr. Holly Cummins
Microservices from dream to reality in an hour - Dr. Holly Cummins
Love your architecture - Alexander von Zitzewitz
Love your architecture - Alexander von Zitzewitz
Lambdas puzzler - Peter Lawrey
Lambdas puzzler - Peter Lawrey
Java vs. Java Script for enterprise web applications - Chris Bailey
Java vs. Java Script for enterprise web applications - Chris Bailey
Java generics past, present and future - Raoul-Gabriel Urma, Richard Warburton
Java generics past, present and future - Raoul-Gabriel Urma, Richard Warburton
Java 8 best practices - Stephen Colebourne
Java 8 best practices - Stephen Colebourne
Intuitions for scaling data centric architectures - Benjamin Stopford
Intuitions for scaling data centric architectures - Benjamin Stopford
Hybrid solutions – combining in memory solutions with SSD - Christos Erotocritou
Hybrid solutions – combining in memory solutions with SSD - Christos Erotocritou
Recently uploaded
chapter--4-software-project-planning.ppt
chapter--4-software-project-planning.ppt
kotipi9215
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
stazi3110
Asset Management Software - Infographic
Asset Management Software - Infographic
Hr365.us smith
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service Consultant
AxelRicardoTrocheRiq
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
umasea
The Evolution of Karaoke From Analog to App.pdf
The Evolution of Karaoke From Analog to App.pdf
Power Karaoke
software engineering Chapter 5 System modeling.pptx
software engineering Chapter 5 System modeling.pptx
nada99848
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
9953056974 Low Rate Call Girls In Saket, Delhi NCR
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
Hanief Utama
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
Sujith Sukumaran
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
Wave PLM
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
MyIntelliSource, Inc.
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
Ortus Solutions, Corp
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
gurkirankumar98700
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
soniya singh
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
Alina Yurenko
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
Christina Lin
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStack
VICTOR MAESTRE RAMIREZ
Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data
Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data
Alluxio, Inc.
Recently uploaded
(20)
chapter--4-software-project-planning.ppt
chapter--4-software-project-planning.ppt
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Asset Management Software - Infographic
Asset Management Software - Infographic
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service Consultant
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
The Evolution of Karaoke From Analog to App.pdf
The Evolution of Karaoke From Analog to App.pdf
software engineering Chapter 5 System modeling.pptx
software engineering Chapter 5 System modeling.pptx
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
Hot Sexy call girls in Patel Nagar🔝 9953056974 🔝 escort Service
React Server Component in Next.js by Hanief Utama
React Server Component in Next.js by Hanief Utama
Intelligent Home Wi-Fi Solutions | ThinkPalm
Intelligent Home Wi-Fi Solutions | ThinkPalm
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
GOING AOT WITH GRAALVM – DEVOXX GREECE.pdf
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStack
Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data
Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data
Benchmarking - you're doing it wrong - Aysylu Greenberg
1.
Benchmarking:- You’re Doing It
Wrong Aysylu-Greenberg- @aysylu22- October-2015-
2.
Aysylu-Greenberg- --------@aysylu22- - -
3.
4.
To-Write-Good-Benchmarks…- Need-to-be-Full-Stack-
5.
- - your-process-vs-goal- your-process-vs-best-pracFces- - Benchmark-=-How-Fast?-
6.
Today- • How-Not-to-Write-Benchmarks- • Benchmark-Setup-&-Results:- - -You’re-wrong-about-machines- - -You’re-wrong-about-stats- - -You’re-wrong-about-what-maOers- •
Becoming-Less-Wrong-
7.
HOW$NOT$TO$WRITE$BENCHMARKS$
8.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
9.
WHAT’S$WRONG$WITH$THIS$ BENCHMARK?$$
10.
YOU’RE$WRONG$ABOUT$THE$MACHINE$ $
11.
Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!-
12.
It’s-Caches-All-The-Way-Down- Web-Request- Server- S3-Cache-
13.
It’s-Caches-All-The-Way-Down-
14.
Prefetching:-Program-
15.
16.
Prefetching:-Disabled-
17.
Prefetching:-Enabled-
18.
19.
Caches-in-Benchmarks- Prof.-Saman-Amarasinghe,-MIT-2009--
20.
Caches-in-Benchmarks- Prof.-Saman-Amarasinghe,-MIT-2009--
21.
Caches-in-Benchmarks- Prof.-Saman-Amarasinghe,-MIT-2009--
22.
Caches-in-Benchmarks- Prof.-Saman-Amarasinghe,-MIT-2009--
23.
Caches-in-Benchmarks- Prof.-Saman-Amarasinghe,-MIT-2009--
24.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
25.
Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming-
26.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
27.
Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- •
Periodic-interference-
28.
Periodic-Interference-
29.
Periodic-Interference-
30.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
31.
Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- •
Periodic-interference- • Test-!=-Prod-
32.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
33.
Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- •
Periodic-interference- • Test-!=-Prod- • Power-mode-changes-
34.
Power-Modes- $-cat-/sys/devices/system/cpu/*/cpufreq/scaling_governor- “ondemand”-OR-“performance”- - Current-CPU-frequencies:- $-grep-"MHz"-/proc/cpuinfo-
35.
YOU’RE$WRONG$ABOUT$THE$STATS$ $
36.
Wrong-About-Stats- • Too-few-samples- -
37.
0- 20- 40- 60- 80- 100- 120- 0- 10- 20-
30- 40- 50- 60- Latency$ #$Runs$ Convergence$of$Median$on$Samples$ Stable-Samples- Stable-Median- Decaying-Samples- Decaying-Median-
38.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-machine- Web-Request- Server- S3-Cache-
39.
Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)-
40.
Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- •
Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-machine- Web-Request- Server- S3-Cache-
41.
Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- •
MulFmodal-distribuFon-
42.
MulFmodal-DistribuFon- 50%- 99%- #-occurrences- Latency- 5-ms- 10-ms-
43.
MulFmodal-DistribuFon-
44.
Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- •
MulFmodal-distribuFon- • Outliers-
45.
Coordinated-Omission- 0- request- response- request- response- 10- request- 20- 30- 40-
50- 60- 70- 80- response- Fme- request- response- request-
46.
Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- •
MulFmodal-distribuFon- • Outliers-
47.
YOU’RE$WRONG$ABOUT$WHAT$MATTERS$ $
48.
Wrong-About-What-MaOers- • Premature-opFmizaFon-
49.
“Programmers-waste-enormous-amounts-of- Fme-thinking-about-…-the-speed-of- noncriFcal-parts-of-their-programs-...-Forget- about-small-efficiencies-…97%-of-the-Fme:- premature$opImizaIon$is$ the$root$of$all$evil.-Yet-we- should-not-pass-up-our- opportuniFes-in-that-criFcal- 3%.”- - pp-Donald-Knuth-
50.
Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads-
51.
Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- •
Memory-pressure-
52.
Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- •
Memory-pressure- • Hidden-components-
53.
Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- •
Memory-pressure- • Hidden-components- • Reproducibility-of-measurements-
54.
BECOMING$LESS$WRONG$
55.
User-AcFons-MaOer- - X->-Y-for-workload-Z- with-trade-offs-A,-B,-and-C- p-hOp://www.toomuchcode.org/-
56.
Profiling- -
57.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
58.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
59.
perf- #-Various-basic-CPU-staFsFcs,-system-wide,-for-10-seconds- perf-stat-pe-cycles,instrucFons,cachepmisses-pa-sleep-10- #-Count-system-calls-for-the-enFre-system,-for-5-seconds- perf-stat-pe-'syscalls:sys_enter_*'-pa-sleep-5- #-Sample-CPU-stack-traces,-once-every-10,000-Level-1-data- cache-misses,-for-5-seconds- perf-record-pe-L1pdcacheploadpmisses-pc-10000-pag-pp-sleep-5- hOp://www.brendangregg.com/perf.html-
60.
perf- hOp://www.brendangregg.com/perf.html-
61.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
62.
gprof:-Where-Does-It-Spend-Its-Time?- • Compile-with-profiling- - • Execute-the-code- - •
Run-the-gprof- hOp://www.thegeekstuff.com/2012/08/gprofptutorial/-
63.
gprof:-Where-Does-It-Spend-Its-Time?- hOp://www.thegeekstuff.com/2012/08/gprofptutorial/-
64.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
65.
hOp://www.brendangregg.com/linuxperf.html-
66.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
67.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
68.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
69.
Profiling- perf- gprof-&- Oprofile- YourKit-&- jProfiler- jVisualVM- cProfile-
70.
Profiling- Code-instrumentaFon- Aggregate-over-logs- Traces- -
71.
Microbenchmarking:-Blessing-&-Curse- + Quick-&-cheap- + Answers-narrow-?s-well- - O|en-misleading-results- - Not-representaFve-of-the-program-
72.
Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- -
73.
Choose-Your-N-Wisely- Prof.-Saman-Amarasinghe,-MIT-2009--
74.
Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects-
75.
Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- •
Beware-of-clock-resoluFon-
76.
Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- •
Beware-of-clock-resoluFon- • Dead-Code-EliminaFon-
77.
Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- •
Beware-of-clock-resoluFon- • Dead-Code-EliminaFon- • Constant-work-per-iteraFon-
78.
NonpConstant-Work-Per-IteraFon-
79.
What-Should-a-Benchmark-Do?- Measure-behavior-of-system- - Represent-realisFc-workload- - Run-for-sufficiently-long-Fme- - Compare-in-the-same-context- - Output-predictable-and-reproducible-results-
80.
Followpup-Material- • How$NOT$to$Measure$Latency$by-Gil-Tene- – hOp://www.infoq.com/presentaFons/latencyppi}alls- •
Taming$the$Long$Latency$Tail-on-highscalability.com- – hOp://highscalability.com/blog/2012/3/12/googleptamingptheplongplatencyp tailpwhenpmorepmachinespequal.html- • Performance$Analysis$Methodology$by-Brendan-Gregg- – hOp://www.brendangregg.com/methodology.html- • Silverman’s$Mode$Detec@on$Method-by-MaO-Adereth- – hOp://adereth.github.io/blog/2014/10/12/silvermanspmodepdetecFonp methodpexplained/- • How$Not$To$Measure$System$Performance-by-James-Bornholt$ – hOps://homes.cs.washington.edu/~bornholt/post/performancep evaluaFon.html- • Trust$No$One,$Not$Even$Performance$Counters-by-Paul-Khuong$ – hDp://www.pvk.ca/Blog/2014/10/19/performancePop@misa@onP~Pwri@ngPanP essay/#trustPnoPone$
81.
Followpup-Material- hOp://wwwpplan.cs.colorado.edu/diwan/asplos09.pdf-
82.
Followpup-Material- • List-of-media-for-learning-more-about-measurement-bias-in- system-benchmarks:- hOps://gist.github.com/aysylu/58ab5d67314d684a7f4c- -
83.
Takeaway-#1:-Cache-
84.
Takeaway-#2:-Outliers-
85.
Takeaway-#3:-Workload-
86.
Benchmarking:- You’re Doing It
Wrong Aysylu-Greenberg- @aysylu22-
Download now