SlideShare a Scribd company logo

PAC 2020 Santorin - Edoardo Varani

N
Neotys

Seeing is knowing, Measuring CPU throttling in containerized environments

1 of 33
PERFORMANCE
IS NOT A MYTH
P E R F O R M A N C E A D V I S O R Y C O U N C I L
SANTORINI GREECE
FEBRUARY 26 - 27 2020
Seeing is knowing:
Measuring CPU throttling in
containerized environments
Edoardo Varani
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Organizations love containers
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Complexity is growing
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Bottleneck Analysis complexity
--cpus = 1
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
First outcomes
• Avg response times
increase with the load
• Bad spikes up to 4x
• CPU??
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Jpetstore bottleneck
0,65 avg CPUsLimit set to 1 CPU
• CPU Util is far from
critical limits (80-
90%)
• So?
• We start to look at
other resources to
find the bottleneck
Ad

Recommended

PAC 2020 Santorin - Joerek Van Gaalen
PAC 2020 Santorin - Joerek Van GaalenPAC 2020 Santorin - Joerek Van Gaalen
PAC 2020 Santorin - Joerek Van GaalenNeotys
 
PAC 2020 Santorin - Giovanni Paolo Gibilisco
PAC 2020 Santorin - Giovanni Paolo GibiliscoPAC 2020 Santorin - Giovanni Paolo Gibilisco
PAC 2020 Santorin - Giovanni Paolo GibiliscoNeotys
 
PAC 2020 Santorin - Gopalkrishnan Yadav
PAC 2020 Santorin - Gopalkrishnan YadavPAC 2020 Santorin - Gopalkrishnan Yadav
PAC 2020 Santorin - Gopalkrishnan YadavNeotys
 
PAC 2020 Santorin - Stijn Schepers
PAC 2020 Santorin - Stijn SchepersPAC 2020 Santorin - Stijn Schepers
PAC 2020 Santorin - Stijn SchepersNeotys
 
PAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan RamachandranPAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan RamachandranNeotys
 
Monitoring and Logging in Wonderland
Monitoring and Logging in WonderlandMonitoring and Logging in Wonderland
Monitoring and Logging in WonderlandPaul Seiffert
 
Gluecon 2013 Keynote Ravello Systems
Gluecon 2013 Keynote Ravello SystemsGluecon 2013 Keynote Ravello Systems
Gluecon 2013 Keynote Ravello Systemsnavinthadani
 
Amaya_Presentation
Amaya_PresentationAmaya_Presentation
Amaya_PresentationIsaias Amaya
 

More Related Content

What's hot

SRE - drupal day aveiro 2016
SRE - drupal day aveiro 2016SRE - drupal day aveiro 2016
SRE - drupal day aveiro 2016Ricardo Amaro
 
Refinery stream modeling walkthrough
Refinery stream modeling walkthroughRefinery stream modeling walkthrough
Refinery stream modeling walkthroughCapstone
 
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...Pål Evensen
 
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...Pankaj Gupta, PhD
 
IEEE Buenaventura cs Chapter March 9 2016 v4
IEEE Buenaventura cs Chapter March 9 2016  v4IEEE Buenaventura cs Chapter March 9 2016  v4
IEEE Buenaventura cs Chapter March 9 2016 v4Sailaja Tennati
 
DevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDNDevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDNSailaja Tennati
 

What's hot (6)

SRE - drupal day aveiro 2016
SRE - drupal day aveiro 2016SRE - drupal day aveiro 2016
SRE - drupal day aveiro 2016
 
Refinery stream modeling walkthrough
Refinery stream modeling walkthroughRefinery stream modeling walkthrough
Refinery stream modeling walkthrough
 
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...
Event Processing Applied to Streams of TV Channel Zaps and Sensor Middleware ...
 
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
 
IEEE Buenaventura cs Chapter March 9 2016 v4
IEEE Buenaventura cs Chapter March 9 2016  v4IEEE Buenaventura cs Chapter March 9 2016  v4
IEEE Buenaventura cs Chapter March 9 2016 v4
 
DevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDNDevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDN
 

Similar to PAC 2020 Santorin - Edoardo Varani

Introduction to Java Profiling
Introduction to Java ProfilingIntroduction to Java Profiling
Introduction to Java ProfilingJerry Yoakum
 
Profiling PyTorch for Efficiency & Sustainability
Profiling PyTorch for Efficiency & SustainabilityProfiling PyTorch for Efficiency & Sustainability
Profiling PyTorch for Efficiency & Sustainabilitygeetachauhan
 
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.The Linux Foundation
 
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...Edge AI and Vision Alliance
 
Broken Performance Tools
Broken Performance ToolsBroken Performance Tools
Broken Performance ToolsC4Media
 
Ocp updating the ocp compute voltage step response specification
Ocp  updating the ocp compute voltage step response specificationOcp  updating the ocp compute voltage step response specification
Ocp updating the ocp compute voltage step response specificationPenguin Computing
 
Computer Performance Microscopy with SHIM
Computer Performance Microscopy with SHIMComputer Performance Microscopy with SHIM
Computer Performance Microscopy with SHIMhiyangxi
 
Galvin-operating System(Ch6)
Galvin-operating System(Ch6)Galvin-operating System(Ch6)
Galvin-operating System(Ch6)dsuyal1
 
Whitepaper: Where did my CPU go?
Whitepaper: Where did my CPU go?Whitepaper: Where did my CPU go?
Whitepaper: Where did my CPU go?Kristofferson A
 
CPU scheduling algorithms in OS
CPU scheduling algorithms in OSCPU scheduling algorithms in OS
CPU scheduling algorithms in OSharini0810
 
Cpu Schedule Algorithm
Cpu Schedule AlgorithmCpu Schedule Algorithm
Cpu Schedule AlgorithmArchit Jain
 
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons Learned
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons LearnedCeph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons Learned
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons LearnedCeph Community
 
4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz KowalczewskiPROIDEA
 
QCon 2015 Broken Performance Tools
QCon 2015 Broken Performance ToolsQCon 2015 Broken Performance Tools
QCon 2015 Broken Performance ToolsBrendan Gregg
 

Similar to PAC 2020 Santorin - Edoardo Varani (20)

Introduction to Java Profiling
Introduction to Java ProfilingIntroduction to Java Profiling
Introduction to Java Profiling
 
Profiling PyTorch for Efficiency & Sustainability
Profiling PyTorch for Efficiency & SustainabilityProfiling PyTorch for Efficiency & Sustainability
Profiling PyTorch for Efficiency & Sustainability
 
Presentation
PresentationPresentation
Presentation
 
Overclocking
OverclockingOverclocking
Overclocking
 
Overclocking
OverclockingOverclocking
Overclocking
 
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
XPDDS18: Real Time in XEN on ARM - Andrii Anisov, EPAM Systems Inc.
 
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...
“Efficiently Map AI and Vision Applications onto Multi-core AI Processors Usi...
 
Broken Performance Tools
Broken Performance ToolsBroken Performance Tools
Broken Performance Tools
 
Ocp updating the ocp compute voltage step response specification
Ocp  updating the ocp compute voltage step response specificationOcp  updating the ocp compute voltage step response specification
Ocp updating the ocp compute voltage step response specification
 
Computer Performance Microscopy with SHIM
Computer Performance Microscopy with SHIMComputer Performance Microscopy with SHIM
Computer Performance Microscopy with SHIM
 
Galvin-operating System(Ch6)
Galvin-operating System(Ch6)Galvin-operating System(Ch6)
Galvin-operating System(Ch6)
 
Whitepaper: Where did my CPU go?
Whitepaper: Where did my CPU go?Whitepaper: Where did my CPU go?
Whitepaper: Where did my CPU go?
 
W10: Interrupts
W10: InterruptsW10: Interrupts
W10: Interrupts
 
CPU scheduling algorithms in OS
CPU scheduling algorithms in OSCPU scheduling algorithms in OS
CPU scheduling algorithms in OS
 
Cpu Schedule Algorithm
Cpu Schedule AlgorithmCpu Schedule Algorithm
Cpu Schedule Algorithm
 
Everybody Lies
Everybody LiesEverybody Lies
Everybody Lies
 
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons Learned
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons LearnedCeph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons Learned
Ceph Day Chicago - Ceph Deployment at Target: Best Practices and Lessons Learned
 
4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski
 
Measure to fail
Measure to failMeasure to fail
Measure to fail
 
QCon 2015 Broken Performance Tools
QCon 2015 Broken Performance ToolsQCon 2015 Broken Performance Tools
QCon 2015 Broken Performance Tools
 

More from Neotys

PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner Neotys
 
PAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis ChatzinasiosPAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis ChatzinasiosNeotys
 
PAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur JainPAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur JainNeotys
 
PAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen TownshendPAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen TownshendNeotys
 
PAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro MelendezPAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro MelendezNeotys
 
PAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen TownshendPAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen TownshendNeotys
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo   PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo Neotys
 
PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez Neotys
 
PAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark TomlinsonPAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark TomlinsonNeotys
 
PAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli AparnaPAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli AparnaNeotys
 
PAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan GeorgePAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan GeorgeNeotys
 
PAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van GaalenPAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van GaalenNeotys
 
PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan  PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan Neotys
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux Neotys
 
PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg  PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg Neotys
 
PAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine ToulmePAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine ToulmeNeotys
 
PAC 2019 virtual Scott Moore
PAC 2019  virtual   Scott Moore PAC 2019  virtual   Scott Moore
PAC 2019 virtual Scott Moore Neotys
 
PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni   PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni Neotys
 
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRANPAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRANNeotys
 
PAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn SchepersPAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn SchepersNeotys
 

More from Neotys (20)

PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner
 
PAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis ChatzinasiosPAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis Chatzinasios
 
PAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur JainPAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur Jain
 
PAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen TownshendPAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen Townshend
 
PAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro MelendezPAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro Melendez
 
PAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen TownshendPAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen Townshend
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo   PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
 
PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez
 
PAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark TomlinsonPAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark Tomlinson
 
PAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli AparnaPAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli Aparna
 
PAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan GeorgePAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan George
 
PAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van GaalenPAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van Gaalen
 
PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan  PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux
 
PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg  PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg
 
PAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine ToulmePAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine Toulme
 
PAC 2019 virtual Scott Moore
PAC 2019  virtual   Scott Moore PAC 2019  virtual   Scott Moore
PAC 2019 virtual Scott Moore
 
PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni   PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni
 
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRANPAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
 
PAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn SchepersPAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn Schepers
 

Recently uploaded

Center Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxCenter Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxsjzzztc
 
Introduction about Technology roadmap for Industry 4.0
Introduction about Technology roadmap for Industry 4.0Introduction about Technology roadmap for Industry 4.0
Introduction about Technology roadmap for Industry 4.0RaishKhanji
 
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4SahithiGurlinka
 
SR Globals Profile - Building Vision, Exceeding Expectations.
SR Globals Profile -  Building Vision, Exceeding Expectations.SR Globals Profile -  Building Vision, Exceeding Expectations.
SR Globals Profile - Building Vision, Exceeding Expectations.srglobalsenterprises
 
ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024Savipriya Raghavendra
 
Introduction to Binary Tree and Conersion of General tree to Binary Tree
Introduction to Binary Tree  and Conersion of General tree to Binary TreeIntroduction to Binary Tree  and Conersion of General tree to Binary Tree
Introduction to Binary Tree and Conersion of General tree to Binary TreeSwarupaDeshpande4
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDBMarian Marinov
 
Sample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeSample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeHarshith A S
 
Architectural Preservation - Heritage, focused on Saudi Arabia
Architectural Preservation - Heritage, focused on Saudi ArabiaArchitectural Preservation - Heritage, focused on Saudi Arabia
Architectural Preservation - Heritage, focused on Saudi ArabiaIgnacio J. Palma, Arch PhD.
 
S. Kim, NeurIPS 2023, MLILAB, KAISTAI
S. Kim,  NeurIPS 2023,  MLILAB,  KAISTAIS. Kim,  NeurIPS 2023,  MLILAB,  KAISTAI
S. Kim, NeurIPS 2023, MLILAB, KAISTAIMLILAB
 
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...GauravBhartie
 
Deluck Technical Works Company Profile.pdf
Deluck Technical Works Company Profile.pdfDeluck Technical Works Company Profile.pdf
Deluck Technical Works Company Profile.pdfartpoa9
 
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIKiranKandhro1
 
neural network non-linear hypothesis.pdf
neural network non-linear hypothesis.pdfneural network non-linear hypothesis.pdf
neural network non-linear hypothesis.pdfSamikshaGoel10
 
PM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxPM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxnissamant
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsPurva Nikam
 
chap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignchap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignteddymebratie
 
GDSC solution challenge Android ppt.pptx
GDSC solution challenge Android ppt.pptxGDSC solution challenge Android ppt.pptx
GDSC solution challenge Android ppt.pptxAnandMenon54
 
MedTech R&D - Tamer Emara - resume @2024
MedTech R&D - Tamer Emara - resume @2024MedTech R&D - Tamer Emara - resume @2024
MedTech R&D - Tamer Emara - resume @2024Tamer Emara
 
nervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesnervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesPhebeLois1
 

Recently uploaded (20)

Center Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docxCenter Enamel is the leading bolted steel tanks manufacturer in China.docx
Center Enamel is the leading bolted steel tanks manufacturer in China.docx
 
Introduction about Technology roadmap for Industry 4.0
Introduction about Technology roadmap for Industry 4.0Introduction about Technology roadmap for Industry 4.0
Introduction about Technology roadmap for Industry 4.0
 
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4GDSC  Google Cloud Study jam Web Bootcamp - Day-4  Session 4
GDSC Google Cloud Study jam Web Bootcamp - Day-4 Session 4
 
SR Globals Profile - Building Vision, Exceeding Expectations.
SR Globals Profile -  Building Vision, Exceeding Expectations.SR Globals Profile -  Building Vision, Exceeding Expectations.
SR Globals Profile - Building Vision, Exceeding Expectations.
 
ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024ExtraordinAIre Monthly Newsletter Jan 2024
ExtraordinAIre Monthly Newsletter Jan 2024
 
Introduction to Binary Tree and Conersion of General tree to Binary Tree
Introduction to Binary Tree  and Conersion of General tree to Binary TreeIntroduction to Binary Tree  and Conersion of General tree to Binary Tree
Introduction to Binary Tree and Conersion of General tree to Binary Tree
 
Introduction and replication to DragonflyDB
Introduction and replication to DragonflyDBIntroduction and replication to DragonflyDB
Introduction and replication to DragonflyDB
 
Sample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its OutcomeSample Case Study of industry 4.0 and its Outcome
Sample Case Study of industry 4.0 and its Outcome
 
Architectural Preservation - Heritage, focused on Saudi Arabia
Architectural Preservation - Heritage, focused on Saudi ArabiaArchitectural Preservation - Heritage, focused on Saudi Arabia
Architectural Preservation - Heritage, focused on Saudi Arabia
 
S. Kim, NeurIPS 2023, MLILAB, KAISTAI
S. Kim,  NeurIPS 2023,  MLILAB,  KAISTAIS. Kim,  NeurIPS 2023,  MLILAB,  KAISTAI
S. Kim, NeurIPS 2023, MLILAB, KAISTAI
 
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
Microstrip Bandpass Filter Design using EDA Tolol such as keysight ADS and An...
 
Deluck Technical Works Company Profile.pdf
Deluck Technical Works Company Profile.pdfDeluck Technical Works Company Profile.pdf
Deluck Technical Works Company Profile.pdf
 
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHIINTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
INTERACTIVE AQUATIC MUSEUM AT BAGH IBN QASIM CLIFTON KARACHI
 
neural network non-linear hypothesis.pdf
neural network non-linear hypothesis.pdfneural network non-linear hypothesis.pdf
neural network non-linear hypothesis.pdf
 
PM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptxPM24_Oral_Presentation_Template_Guidelines.pptx
PM24_Oral_Presentation_Template_Guidelines.pptx
 
Laser And its Application's - Engineering Physics
Laser And its Application's - Engineering PhysicsLaser And its Application's - Engineering Physics
Laser And its Application's - Engineering Physics
 
chap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processignchap. 3. lipid deterioration oil and fat processign
chap. 3. lipid deterioration oil and fat processign
 
GDSC solution challenge Android ppt.pptx
GDSC solution challenge Android ppt.pptxGDSC solution challenge Android ppt.pptx
GDSC solution challenge Android ppt.pptx
 
MedTech R&D - Tamer Emara - resume @2024
MedTech R&D - Tamer Emara - resume @2024MedTech R&D - Tamer Emara - resume @2024
MedTech R&D - Tamer Emara - resume @2024
 
nervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nervesnervous system ppt pptx anatomy system of nerves
nervous system ppt pptx anatomy system of nerves
 

PAC 2020 Santorin - Edoardo Varani

  • 1. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Seeing is knowing: Measuring CPU throttling in containerized environments Edoardo Varani
  • 2. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Organizations love containers
  • 3. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Complexity is growing
  • 4. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Bottleneck Analysis complexity --cpus = 1
  • 5. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L First outcomes • Avg response times increase with the load • Bad spikes up to 4x • CPU??
  • 6. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Jpetstore bottleneck 0,65 avg CPUsLimit set to 1 CPU • CPU Util is far from critical limits (80- 90%) • So? • We start to look at other resources to find the bottleneck
  • 7. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Bottleneck analysis complexity What if the bottleneck is exactly in jpetstore CPU?
  • 8. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CPU limits under the hood • Based on Linux Kernel cgroups • Completely Fair Scheduler (CFS) Bandwidth control: • Quota • Period • Shares (Soft limit) • A cgroup can use at most his CPU time Quota in each wall-clock time Period • Each cgroup can still use all the physical CPUs • If the Quota is burned before the new Period, the cgroup gets throttled
  • 9. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Throttling metrics from cgroup files • CFS native metrics: • CPU periods • CPU throttled periods • CPU throttling time Cgroup folder for the container
  • 10. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Best APMs show throttling (in seconds)
  • 11. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L A metric from Kubecon Δnr_throttled • By Dave Chiluk @ Indeed • Gives an approximate indication • It does not tell how bad the throttling spikes are Δ nr_periods Throttled% =
  • 12. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L A metric from Kubecon Commits: de53fd7aedb1 & 763a9ec06c40 Applied to 5.4 Kernel Backported to: 4.14.154+, 4.19.84+, 5.3.9+ Distro kernels: • Ubuntu 5.3.0-24+ • Ubuntu 4.15.0-67+ • RHEL7 - kernel-3.10.0-1062.8.1.el7 • RHEL8.2 - WIP
  • 13. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L So what? Ok, so we probably need to increase jpetstore quota. BUT How much more?
  • 14. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Hard choices CONTAINER PERFORMANCE CLUSTER RELIABILITY • The higher the limits, the higher the risk • We want to avoid multiple performance tests just to know the right container quota • We need to know how far we are from the desired CPUs
  • 15. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L We need to see the Throttled CPUs + 1.3 Max Throttled CPUs ThrottledCPUs
  • 16. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Just add the 1.3 Throttled CPUs to the quota --cpus = 1--cpus = 2,3+1,3 desired CPUs
  • 17. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Quota increased – no more throttling Quota=2.3Cpus Throttling < 0.1 CPUs
  • 18. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L New Sizing – CPU footprint is similar 1 CPU quota 2.3 CPU quota 0.7 CPUs 0.7 CPUs
  • 19. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Benefits Any real benefit from this sizing?
  • 20. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L New Sizing – Response time cut About 200ms on average About 105ms on averageBefore After
  • 21. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L New Sizing – GC Pauses cut About 230ms for a Full GCs About 100ms of Full GCs AfterBefore
  • 22. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L How is it calculated? + 1.3 Max Throttled CPUs ThrottledCPUs
  • 23. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Throttled CPUs - PromQL rate(container_cpu_cfs_throttled_seconds_total[interval]) • It’s just the “throttled seconds per second” rate • Close to standard "top-like container cpu utilization“ • You want this to be close to 0 but not 0
  • 24. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Stress-ng – Batch Case Scenario docker run lorel/docker-stress-ng stress-ng --cpu 2 • Easy workload generator • It completely use the number of CPUs provided • Three experiments: • --cpus = 0.5 • --cpus = 1 • --cpus = 1.9
  • 25. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Batch workloads – 0.5 CPU limit 1.5 Throttled CPUs 100% Throttled periods 0.5 Used CPUs
  • 26. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Batch workloads – 1 CPU limit 100% Throttled periods 1 Used CPUs 1 Throttled CPUs
  • 27. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Batch workloads – 1.9 CPU limit Still 100% Throttled periods0.1 Throttled CPUs 1.9 Used CPUs
  • 28. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Quota dependency - Renassaince • Open Source • 30+ Benchmarks • Akka actors • Spark batches • Genetic algorithms • Jdk-streams • Reactive streams • … • Committee from Universities and Oracle Labs • Cool logo
  • 29. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L JVM Ergonomics – Quota dependency • Increasing the quota caused a demand increase • Why?
  • 30. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L JVM Ergonomics – Quota dependency • More threads are created • Parallel work is offloaded to the new threads • JVM see more availableProcessors
  • 31. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L availableProcessors • Runtime.availableProcessors is used to set: • Compile threads • GC threads • Fork join pool size • Libraries / App Servers threads • Prior to JDK 8u131, there was no container awareness • In JDK11 PreferContainerQuotaForCPUCount is ON by default • If OFF, JVM will use the CPU Shares
  • 32. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Takeaways • Choose a meaningful throttling metric • Give JVMs some room to spike, BUT • Try to downscale your threads if you are spiking too much • Compare throttling to compare the spikiness between releases • Tailor the limits around your workload to preserve the cluster • Upgrade your Kernels and JDKs • Hang tight for cgroup v2
  • 33. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Questions?