© 2009 IBM Corporation
CPU Hotplug, RCU, and big.LITTLE
Paul E. McKenney, IBM Distinguished Engineer, Linux Technology Cen...
© 2009 IBM Corporation
2
CPU Hotplug: Work In Progress
CPU Hotplug, RCU, and big.LITTLE
© 2009 IBM Corporation
3
CPU Hotplug: Work In Progress
CPU Hotplug, RCU, and big.LITTLE
 Reduce task creation/destruction...
© 2009 IBM Corporation
4
RCU Callbacks, HPC, Real Time and big.LITTLE
CPU Hotplug, RCU, and big.LITTLE
© 2009 IBM Corporation
5
RCU Callbacks, HPC, Real Time and big.LITTLE
CPU Hotplug, RCU, and big.LITTLE
 For an overview, ...
© 2009 IBM Corporation
6
The Problem With RCU Callbacks
CPU Hotplug, RCU, and big.LITTLE
 The problem with RCU callbacks:...
© 2009 IBM Corporation
7
The Problem With RCU Callbacks
CPU Hotplug, RCU, and big.LITTLE
 The problem with RCU callbacks:...
© 2009 IBM Corporation
8
The Problem With RCU Callbacks
CPU Hotplug, RCU, and big.LITTLE
 The problem with RCU callbacks:...
© 2009 IBM Corporation
9
The Problem With RCU Callbacks and big.LITTLE
CPU Hotplug, RCU, and big.LITTLE
big CPU
call_rcu()...
© 2009 IBM Corporation
10
What big.LITTLE Might Want Instead
CPU Hotplug, RCU, and big.LITTLE
big CPU
call_rcu()
CB
Grace ...
© 2009 IBM Corporation
11
CB
What big.LITTLE Might Want Instead: But Not Free
CPU Hotplug, RCU, and big.LITTLE
big CPU
cal...
© 2009 IBM Corporation
12
Callback Invoked
RCU Callbacks, Houston/Korty for TREE_RCU
CPU Hotplug, RCU, and big.LITTLE
CPU ...
© 2009 IBM Corporation
13
Callback Invoked
RCU Callbacks, Houston/Korty for TREE_RCU
CPU Hotplug, RCU, and big.LITTLE
CPU ...
© 2009 IBM Corporation
14
Callback Invoked
RCU Callbacks, Houston/Korty for TREE_RCU
CPU Hotplug, RCU, and big.LITTLE
CPU ...
© 2009 IBM Corporation
15
Offloadable RCU Callbacks: Limitations and Futures
CPU Hotplug, RCU, and big.LITTLE
 Must reboo...
© 2009 IBM Corporation
16
Offloadable RCU Callbacks: Plans
CPU Hotplug, RCU, and big.LITTLE
Initial version in 3.8 or 3.9...
© 2009 IBM Corporation
17
To Probe More Deeply...
CPU Hotplug, RCU, and big.LITTLE
What Is New In RCU for Real Time (RTLW...
© 2009 IBM Corporation
18
Legal Statement
This work represents the view of the author and does not
necessarily represent ...
© 2009 IBM Corporation
19
Questions?
CPU Hotplug, RCU, and big.LITTLE
© 2009 IBM Corporation
20
Scheduling Topics
Target CPU Selection (Vincent Guittot)
Sharing Scheduler Information (Vincen...
© 2009 IBM Corporation
21
CPU Hotplug Primer: Online Process
CPU_UP_PREPARE
CPU_STARTING CPU_UP_CANCELLED
FailedSucceeded
...
© 2009 IBM Corporation
22
CPU Hotplug Primer: Offline Process
CPU_DOWN_PREPARE
CPU_DYING CPU_DOWN_FAILED
FailedSucceeded
C...
© 2009 IBM Corporation
23
CPU Hotplug Is Not Atomic
Scheduler
IPIs RCU
Valid notifier order for online:
●
IPIs
●
RCU
●
Sch...
© 2009 IBM Corporation
24
CPU Hotplug: Valid Online Order (1/3)
IPIs
Valid notifier order for online:
●
IPIs
●
RCU
●
Sched...
© 2009 IBM Corporation
25
CPU Hotplug: Valid Online Order (2/3)
IPIs RCU
Valid notifier order for online:
●
IPIs
●
RCU
●
S...
© 2009 IBM Corporation
26
CPU Hotplug: Valid Online Order (3/3)
Scheduler
IPIs RCU
Valid notifier order for online:
●
IPIs...
© 2009 IBM Corporation
27
Suppose We Offline In The Same Order...
Scheduler
IPIs RCU
Valid notifier order for online:
●
IP...
© 2009 IBM Corporation
28
CPU Hotplug: Invalid Offline Order
Scheduler
RCU???
Valid notifier order for online:
●
IPIs
●
RC...
© 2009 IBM Corporation
29
CPU Hotplug: Valid Offline Order (1/3)
Scheduler
IPIs RCU
Valid notifier order for online:
●
IPI...
© 2009 IBM Corporation
30
CPU Hotplug: Valid Online Order (2/3)
IPIs RCU
Valid notifier order for online:
●
IPIs
●
RCU
●
S...
© 2009 IBM Corporation
31
CPU Hotplug: Valid Offline Order (3/3)
IPIs
Valid notifier order for online:
●
IPIs
●
RCU
●
Sche...
© 2009 IBM Corporation
32
Other Topics
Synthetic Mobile Workloads
–Progress, but no nice solution yet
Emulating big.LITT...
© 2009 IBM Corporation
33
Summary
Excellent progress on many fronts
But much work still to be done!
CPU Hotplug, RCU, an...
Upcoming SlideShare
Loading in …5
×

LCE12: CPU Hotplug, RCU, and big.LITTLE

692 views

Published on

Resource: LCE12
Name: CPU Hotplug, RCU, and big.LITTLE
Date: 01-11-2012
Speaker: Paul McKenney

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
692
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
20
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

LCE12: CPU Hotplug, RCU, and big.LITTLE

  1. 1. © 2009 IBM Corporation CPU Hotplug, RCU, and big.LITTLE Paul E. McKenney, IBM Distinguished Engineer, Linux Technology Center (Linaro) November 1, 2012 2012 Linux Plumbers Conference, Scheduling Microconference
  2. 2. © 2009 IBM Corporation 2 CPU Hotplug: Work In Progress CPU Hotplug, RCU, and big.LITTLE
  3. 3. © 2009 IBM Corporation 3 CPU Hotplug: Work In Progress CPU Hotplug, RCU, and big.LITTLE  Reduce task creation/destruction overhead – Thomas Gleixner's generic-idle (mainline) and park/unpark (-tip) – Tejun Heo's fix for workqueues  Numerous fixes from Srivatsa Bhat (3.4-3.6)  Disturbance-free SRCU (Lai Jiangshan, 3.4)  Wean RCU from __stop_machine() (PEM, 3.5-3.7)  Wean scheduler from __stop_machine() (Peter, 3.7)  Guard for_each_online_cpu() (Silas Boyd-Wickizer, 3.7-3.8)  Switch hotplug from __stop_machine() (PEM, 3.8?)  Tell RCU exactly when CPU goes offline (PEM, 3.8?)  Reverse notifier order, reworking notifier concepts (TBD)  Apply park/unpark everywhere (TBD)  Good progress, some work left
  4. 4. © 2009 IBM Corporation 4 RCU Callbacks, HPC, Real Time and big.LITTLE CPU Hotplug, RCU, and big.LITTLE
  5. 5. © 2009 IBM Corporation 5 RCU Callbacks, HPC, Real Time and big.LITTLE CPU Hotplug, RCU, and big.LITTLE  For an overview, see http://lwn.net/Articles/262464/ For the purposes of this presentation, think of RCU as something that defers work, with one work item per callback –Each callback has a function pointer and an argument –Callbacks are queued on per-CPU lists, invoked after grace period • Invocation can result in OS jitter and real-time latency –Global list handles callbacks from offlined CPUs: adopted quickly
  6. 6. © 2009 IBM Corporation 6 The Problem With RCU Callbacks CPU Hotplug, RCU, and big.LITTLE  The problem with RCU callbacks: CPU 0 call_rcu() Callback Invoked Grace Period Likely disrupting whatever was intended to execute at about this time...
  7. 7. © 2009 IBM Corporation 7 The Problem With RCU Callbacks CPU Hotplug, RCU, and big.LITTLE  The problem with RCU callbacks: CPU 0 call_rcu() Callback Invoked Grace Period Likely disrupting whatever was intended to execute at about this time... Such as a big CPU's desire to remain idle.
  8. 8. © 2009 IBM Corporation 8 The Problem With RCU Callbacks CPU Hotplug, RCU, and big.LITTLE  The problem with RCU callbacks: CPU 0 call_rcu() Callback Invoked Grace Period Likely disrupting whatever was intended to execute at about this time... Such as a big CPU's desire to remain idle. Not so good for HPC and real time, either
  9. 9. © 2009 IBM Corporation 9 The Problem With RCU Callbacks and big.LITTLE CPU Hotplug, RCU, and big.LITTLE big CPU call_rcu() CB Grace Period LITTLE CPU Busy Busy Busy Busy
  10. 10. © 2009 IBM Corporation 10 What big.LITTLE Might Want Instead CPU Hotplug, RCU, and big.LITTLE big CPU call_rcu() CB Grace Period LITTLE CPU Busy Busy Busy Busy
  11. 11. © 2009 IBM Corporation 11 CB What big.LITTLE Might Want Instead: But Not Free CPU Hotplug, RCU, and big.LITTLE big CPU call_rcu() CB Grace Period LITTLE CPU Busy Busy Busy Busy
  12. 12. © 2009 IBM Corporation 12 Callback Invoked RCU Callbacks, Houston/Korty for TREE_RCU CPU Hotplug, RCU, and big.LITTLE CPU 2 Grace Period rcuo1 kthread CPU 1 Callback Invoked Grace Period rcuo0 kthread call_rcu() call_rcu()
  13. 13. © 2009 IBM Corporation 13 Callback Invoked RCU Callbacks, Houston/Korty for TREE_RCU CPU Hotplug, RCU, and big.LITTLE CPU 2 Grace Period rcuo1 kthread CPU 1 Callback Invoked Grace Period rcuo0 kthread call_rcu() call_rcu() Manually place rcuoN kthreads on LITTLE CPUs
  14. 14. © 2009 IBM Corporation 14 Callback Invoked RCU Callbacks, Houston/Korty for TREE_RCU CPU Hotplug, RCU, and big.LITTLE CPU 2 Grace Period rcuo1 kthread No big-CPU wakeups!!! CPU 1 Callback Invoked Grace Period rcuo0 kthread call_rcu() call_rcu() Manually place rcuoN kthreads on LITTLE CPUs
  15. 15. © 2009 IBM Corporation 15 Offloadable RCU Callbacks: Limitations and Futures CPU Hotplug, RCU, and big.LITTLE  Must reboot to reconfigure no-CBs CPUs – Should be just fine for big.LITTLE – Hopefully also OK for HPC and real-time workloads  Scalability: 1,000 no-CBs CPUs would not do well – Not a problem for big.LITTLE, but must be solved for HPC  Must be at least one non-no-CBs CPU (namely, CPU 0) – OK if CPU 0 is a LITTLE CPU, but expect to relax this constraint “real soon now”  No energy-efficiency code: lazy & non-lazy CBs? Non-lazy! – But non-lazy Cbs are common case, so deferring interpretation of laziness.  No-CBs CPUs' kthreads not subject to priority boosting – Probably not a near-term problem for big.LITTLE  Setting all no-CBs CPUs' kthreads to RT prio w/out pinning them: bad! – At least on large systems: Probably OK near-term for big.LITTLE  Note: I do not expect no-CBs path to completely replace current CB path
  16. 16. © 2009 IBM Corporation 16 Offloadable RCU Callbacks: Plans CPU Hotplug, RCU, and big.LITTLE Initial version in 3.8 or 3.9, depending on testing results –Thanks to Robin Randhawa and Paul Gortmaker for testing efforts thus far
  17. 17. © 2009 IBM Corporation 17 To Probe More Deeply... CPU Hotplug, RCU, and big.LITTLE What Is New In RCU for Real Time (RTLWS 2012) – http://www.rdrop.com/users/paulmck/realtime/paper/RTLWS2012occcRT.2012. 10.19e.pdf • Slides 21-on Getting RCU Further Out of the Way (Plumbers 2012) – http://www.rdrop.com/users/paulmck/realtime/paper/nocb.2012.08.31a.pdf Cleaning Up Linux’s CPU Hotplug For Real Time and Energy Management (ECRTS 2012) – http://www.rdrop.com/users/paulmck/realtime/paper/hotplug- ecrts.2012.06.11a.pdf And hot off the press: “Relocating RCU callbacks” –http://lwn.net/Articles/522262/
  18. 18. © 2009 IBM Corporation 18 Legal Statement This work represents the view of the author and does not necessarily represent the view of IBM. IBM and IBM (logo) are trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries. Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others. CPU Hotplug, RCU, and big.LITTLE
  19. 19. © 2009 IBM Corporation 19 Questions? CPU Hotplug, RCU, and big.LITTLE
  20. 20. © 2009 IBM Corporation 20 Scheduling Topics Target CPU Selection (Vincent Guittot) Sharing Scheduler Information (Vincent Guittot) Task Placement for Asymmetric Cores (Morten Rasmussen) Scheduling and the big.LITTLE Architecture (you are here!) Dynamic CPU Core Management (Peter De Schrijver and Antti Miettinen) Application of Deadline Scheduling for Power-Saving Strategies (Juri Lelli) Main focus of this talk is progress on CPU hotplug –See Peter's and Antti's talk for alternatives CPU Hotplug, RCU, and big.LITTLE
  21. 21. © 2009 IBM Corporation 21 CPU Hotplug Primer: Online Process CPU_UP_PREPARE CPU_STARTING CPU_UP_CANCELLED FailedSucceeded CPU_ONLINE cpu_hotplug.lock held cpu_add_remove_lock held Succeeded Failed CPU Hotplug, RCU, and big.LITTLE
  22. 22. © 2009 IBM Corporation 22 CPU Hotplug Primer: Offline Process CPU_DOWN_PREPARE CPU_DYING CPU_DOWN_FAILED FailedSucceeded CPU_DEAD CPU_POST_DEAD All other CPUs spinning with interrupts off while outgoing CPU is in CPU_DYING notifiers cpu_hotplug.lock held cpu_add_remove_lock held Succeeded Failed CPU Hotplug, RCU, and big.LITTLE
  23. 23. © 2009 IBM Corporation 23 CPU Hotplug Is Not Atomic Scheduler IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  24. 24. © 2009 IBM Corporation 24 CPU Hotplug: Valid Online Order (1/3) IPIs Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  25. 25. © 2009 IBM Corporation 25 CPU Hotplug: Valid Online Order (2/3) IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  26. 26. © 2009 IBM Corporation 26 CPU Hotplug: Valid Online Order (3/3) Scheduler IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  27. 27. © 2009 IBM Corporation 27 Suppose We Offline In The Same Order... Scheduler IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  28. 28. © 2009 IBM Corporation 28 CPU Hotplug: Invalid Offline Order Scheduler RCU??? Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  29. 29. © 2009 IBM Corporation 29 CPU Hotplug: Valid Offline Order (1/3) Scheduler IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  30. 30. © 2009 IBM Corporation 30 CPU Hotplug: Valid Online Order (2/3) IPIs RCU Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  31. 31. © 2009 IBM Corporation 31 CPU Hotplug: Valid Offline Order (3/3) IPIs Valid notifier order for online: ● IPIs ● RCU ● Scheduler Must reverse order for offline Reality will intrude... ● RCU depends on scheduler ● Circular dependency! Must further decompose RCU and scheduler interaction CPU Hotplug, RCU, and big.LITTLE
  32. 32. © 2009 IBM Corporation 32 Other Topics Synthetic Mobile Workloads –Progress, but no nice solution yet Emulating big.LITTLE on commodity systems –An embarrassment of riches: http://lwn.net/Articles/501501/ CPU Hotplug, RCU, and big.LITTLE
  33. 33. © 2009 IBM Corporation 33 Summary Excellent progress on many fronts But much work still to be done! CPU Hotplug, RCU, and big.LITTLE

×