SlideShare a Scribd company logo
1 of 39
Download to read offline
Preventing Guests from Spinning Around


     How to Deal with Lock-Holder Preemption
                              Thomas Friebel

                              June 23, 2008
Spinlock Basics

Spinlocks wait actively as opposed to sleeping locks

Used for short critical sections




 2              How to Deal with Lock-Holder Preemption
Spinlock Wait Times – Kernbench




3           How to Deal with Lock-Holder Preemption
Spinlock Wait Times – Kernbench




4           How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




5   How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




6            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




7            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




8            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




9            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




10           How to Deal with Lock-Holder Preemption
Is lock-holder preemption problematic?




11    How to Deal with Lock-Holder Preemption
Kernbench in a Guest




12          How to Deal with Lock-Holder Preemption
Kernbench vs. 'while(true)'




13          How to Deal with Lock-Holder Preemption
Time, not Times




14          How to Deal with Lock-Holder Preemption
And in Numbers?




                      guest time                  time spent spinning
                              [s]                      [s]       [%]
single kernbench          109.0                        0.2      0.2%
kernbench vs while(1)     117.3                        9.0      7.6%
difference                 7.6%




15             How to Deal with Lock-Holder Preemption
What can we do about it?




16   How to Deal with Lock-Holder Preemption
Dealing with lock-holder preemption

LHP avoidance
 No spinlock held in userspace
 Idea: Avoid preempting guest in kernel space
 Postpone guest switch to kernel exit
 Problem: extraordinary long critical sections, e.g.
  Apache using sendfile()

Helping locks
 Instead of busy waiting, switch to preempted lock-
  holder
 Problem: finding the preempted lock-holder



17              How to Deal with Lock-Holder Preemption
Helping locks: Ingredients

1) Guest kernel: new 'yield' hypercall when waiting
   unusually long
         Modify spinlock loop
     ●




2) Reasonable threshold for 'unusually long'
         Histograms help
     ●




3) Selecting which VCPU to switch to




18                  How to Deal with Lock-Holder Preemption
Threshold: Upper boundary




19         How to Deal with Lock-Holder Preemption
Threshold: Lower boundary




20         How to Deal with Lock-Holder Preemption
Scheduling Strategy

Good choices:
 VCPUs of the same VM to make progress locally
 (Potential) preempted lock-holders
 Cache-„near“ VCPUs

Neither/nor:
 VCPUs in user space

Bad choices:
 VCPUs which yielded recently




21              How to Deal with Lock-Holder Preemption
What about performance?




22   How to Deal with Lock-Holder Preemption
Histogram with 'yield' hypercall




23           How to Deal with Lock-Holder Preemption
Performance




             wall clock guest time time spent spinning
                    [s]         [s]      [s]      [%]
LHP                34.8      117.3       9.0     7.6%
yield              33.5      108.4       0.0     0.0%
difference      -3.9%       -7.6%               -7.6%




24            How to Deal with Lock-Holder Preemption
Efficiency




25           How to Deal with Lock-Holder Preemption
Efficiency




26           How to Deal with Lock-Holder Preemption
Efficiency




27           How to Deal with Lock-Holder Preemption
Efficiency




28           How to Deal with Lock-Holder Preemption
Efficiency




29           How to Deal with Lock-Holder Preemption
Efficiency




30           How to Deal with Lock-Holder Preemption
Efficiency




31           How to Deal with Lock-Holder Preemption
Efficiency




                      117 sec
                                  ×7.6 % = 3.7 %
                 117 sec 126 sec

                 Real result of 3.9% is reasonable
             ➔


                 Highly efficient
             ➔




32           How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks




33   How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks

Next ticket in dispenser: queue tail

„Now serving“ display at counter: queue head




Lock: atomic( ticket = tail++ ); while ( head != ticket );

Unlock: atomic( head++ );



34              How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks




35           How to Deal with Lock-Holder Preemption
Ticket locks and virtualization




           wall clock guest time                 time spent spinning
                  [s]         [s]                      [s]      [%]
LHP          2825.1     22434.2                   22270.4     99.3%




36           How to Deal with Lock-Holder Preemption
Ticket locks and virtualization




           wall clock guest time                 time spent spinning
                  [s]         [s]                      [s]      [%]
LHP          2825.1     22434.2                   22270.4     99.3%
yield            34.1     123.6                       6.6      5.4%




37           How to Deal with Lock-Holder Preemption
Conclusion


       Lock-holder preemption quite serious:
             7.6% guest time wasted

                  Helping locks:
      3.9% system performance improvement!
           (Amdahl's law explains why)

               New ticket spinlocks:
        30 secs kernbench takes 45 minutes

             Helping locks help here, too




38           How to Deal with Lock-Holder Preemption
Trademark Attribution


AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or
other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks
of their respective owners.

©2008 Advanced Micro Devices, Inc. All rights reserved.




 39                               How to Deal with Lock-Holder Preemption

More Related Content

Similar to XS Boston 2008 Guest Spinning

Preemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloudPreemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloud
Jiannan Ouyang, PhD
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
HostedbyConfluent
 

Similar to XS Boston 2008 Guest Spinning (19)

Preemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloudPreemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloud
 
Meltdown & spectre
Meltdown & spectreMeltdown & spectre
Meltdown & spectre
 
Meltdown & Spectre
Meltdown & Spectre Meltdown & Spectre
Meltdown & Spectre
 
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilitiesBlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
 
Distributed Lock Manager
Distributed Lock ManagerDistributed Lock Manager
Distributed Lock Manager
 
Sangam 18 - Database Development: Return of the SQL Jedi
Sangam 18 - Database Development: Return of the SQL JediSangam 18 - Database Development: Return of the SQL Jedi
Sangam 18 - Database Development: Return of the SQL Jedi
 
Need for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applicationsNeed for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applications
 
An Introduction to Locks in Go
An Introduction to Locks in GoAn Introduction to Locks in Go
An Introduction to Locks in Go
 
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and LatencyOptimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
 
opt-mem-trx
opt-mem-trxopt-mem-trx
opt-mem-trx
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
 
ioDrive de benchmarking 2011 1209_zem_distribution
ioDrive de benchmarking 2011 1209_zem_distributionioDrive de benchmarking 2011 1209_zem_distribution
ioDrive de benchmarking 2011 1209_zem_distribution
 
MSc_thesis_defence
MSc_thesis_defenceMSc_thesis_defence
MSc_thesis_defence
 
Smallsat 2021
Smallsat 2021Smallsat 2021
Smallsat 2021
 
Psychtoolbox (PTB) practical course by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
Psychtoolbox (PTB) practical course  by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...Psychtoolbox (PTB) practical course  by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
Psychtoolbox (PTB) practical course by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
 
On the way to low latency (2nd edition)
On the way to low latency (2nd edition)On the way to low latency (2nd edition)
On the way to low latency (2nd edition)
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
 

More from The Linux Foundation

More from The Linux Foundation (20)

ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
 
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
 
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
 
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
 
XPDDS19 Keynote: Unikraft Weather Report
XPDDS19 Keynote:  Unikraft Weather ReportXPDDS19 Keynote:  Unikraft Weather Report
XPDDS19 Keynote: Unikraft Weather Report
 
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
 
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, XilinxXPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
 
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
 
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, BitdefenderXPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
 
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
 
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making... OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, CitrixXPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
 
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltdXPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
 
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
 
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&DXPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
 
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
 
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
 
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
 
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSEXPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
 

Recently uploaded

Recently uploaded (20)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 

XS Boston 2008 Guest Spinning

  • 1. Preventing Guests from Spinning Around How to Deal with Lock-Holder Preemption Thomas Friebel June 23, 2008
  • 2. Spinlock Basics Spinlocks wait actively as opposed to sleeping locks Used for short critical sections 2 How to Deal with Lock-Holder Preemption
  • 3. Spinlock Wait Times – Kernbench 3 How to Deal with Lock-Holder Preemption
  • 4. Spinlock Wait Times – Kernbench 4 How to Deal with Lock-Holder Preemption
  • 5. Spinlocks and Virtualization 5 How to Deal with Lock-Holder Preemption
  • 6. Spinlocks and Virtualization 6 How to Deal with Lock-Holder Preemption
  • 7. Spinlocks and Virtualization 7 How to Deal with Lock-Holder Preemption
  • 8. Spinlocks and Virtualization 8 How to Deal with Lock-Holder Preemption
  • 9. Spinlocks and Virtualization 9 How to Deal with Lock-Holder Preemption
  • 10. Spinlocks and Virtualization 10 How to Deal with Lock-Holder Preemption
  • 11. Is lock-holder preemption problematic? 11 How to Deal with Lock-Holder Preemption
  • 12. Kernbench in a Guest 12 How to Deal with Lock-Holder Preemption
  • 13. Kernbench vs. 'while(true)' 13 How to Deal with Lock-Holder Preemption
  • 14. Time, not Times 14 How to Deal with Lock-Holder Preemption
  • 15. And in Numbers? guest time time spent spinning [s] [s] [%] single kernbench 109.0 0.2 0.2% kernbench vs while(1) 117.3 9.0 7.6% difference 7.6% 15 How to Deal with Lock-Holder Preemption
  • 16. What can we do about it? 16 How to Deal with Lock-Holder Preemption
  • 17. Dealing with lock-holder preemption LHP avoidance  No spinlock held in userspace  Idea: Avoid preempting guest in kernel space  Postpone guest switch to kernel exit  Problem: extraordinary long critical sections, e.g. Apache using sendfile() Helping locks  Instead of busy waiting, switch to preempted lock- holder  Problem: finding the preempted lock-holder 17 How to Deal with Lock-Holder Preemption
  • 18. Helping locks: Ingredients 1) Guest kernel: new 'yield' hypercall when waiting unusually long Modify spinlock loop ● 2) Reasonable threshold for 'unusually long' Histograms help ● 3) Selecting which VCPU to switch to 18 How to Deal with Lock-Holder Preemption
  • 19. Threshold: Upper boundary 19 How to Deal with Lock-Holder Preemption
  • 20. Threshold: Lower boundary 20 How to Deal with Lock-Holder Preemption
  • 21. Scheduling Strategy Good choices:  VCPUs of the same VM to make progress locally  (Potential) preempted lock-holders  Cache-„near“ VCPUs Neither/nor:  VCPUs in user space Bad choices:  VCPUs which yielded recently 21 How to Deal with Lock-Holder Preemption
  • 22. What about performance? 22 How to Deal with Lock-Holder Preemption
  • 23. Histogram with 'yield' hypercall 23 How to Deal with Lock-Holder Preemption
  • 24. Performance wall clock guest time time spent spinning [s] [s] [s] [%] LHP 34.8 117.3 9.0 7.6% yield 33.5 108.4 0.0 0.0% difference -3.9% -7.6% -7.6% 24 How to Deal with Lock-Holder Preemption
  • 25. Efficiency 25 How to Deal with Lock-Holder Preemption
  • 26. Efficiency 26 How to Deal with Lock-Holder Preemption
  • 27. Efficiency 27 How to Deal with Lock-Holder Preemption
  • 28. Efficiency 28 How to Deal with Lock-Holder Preemption
  • 29. Efficiency 29 How to Deal with Lock-Holder Preemption
  • 30. Efficiency 30 How to Deal with Lock-Holder Preemption
  • 31. Efficiency 31 How to Deal with Lock-Holder Preemption
  • 32. Efficiency 117 sec ×7.6 % = 3.7 % 117 sec 126 sec Real result of 3.9% is reasonable ➔ Highly efficient ➔ 32 How to Deal with Lock-Holder Preemption
  • 33. FIFO ticket spinlocks 33 How to Deal with Lock-Holder Preemption
  • 34. FIFO ticket spinlocks Next ticket in dispenser: queue tail „Now serving“ display at counter: queue head Lock: atomic( ticket = tail++ ); while ( head != ticket ); Unlock: atomic( head++ ); 34 How to Deal with Lock-Holder Preemption
  • 35. FIFO ticket spinlocks 35 How to Deal with Lock-Holder Preemption
  • 36. Ticket locks and virtualization wall clock guest time time spent spinning [s] [s] [s] [%] LHP 2825.1 22434.2 22270.4 99.3% 36 How to Deal with Lock-Holder Preemption
  • 37. Ticket locks and virtualization wall clock guest time time spent spinning [s] [s] [s] [%] LHP 2825.1 22434.2 22270.4 99.3% yield 34.1 123.6 6.6 5.4% 37 How to Deal with Lock-Holder Preemption
  • 38. Conclusion Lock-holder preemption quite serious: 7.6% guest time wasted Helping locks: 3.9% system performance improvement! (Amdahl's law explains why) New ticket spinlocks: 30 secs kernbench takes 45 minutes Helping locks help here, too 38 How to Deal with Lock-Holder Preemption
  • 39. Trademark Attribution AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks of their respective owners. ©2008 Advanced Micro Devices, Inc. All rights reserved. 39 How to Deal with Lock-Holder Preemption