More about Linaro Connect: connect.linaro.org
More about Linaro: www.linaro.org/about/
More about Linaro engineering: www....
2
EUROPE 2012 (LCE12)
www.linaro.org
Linaro Connect Hong-Kong 2013
Who disturbs my
slumber ?!
3
EUROPE 2012 (LCE12)
www.linaro.org
Introduction
The power management is a wide area
Power aware scheduler, P-States, C-S...
4
EUROPE 2012 (LCE12)
www.linaro.org
PM diagram blocks
5
EUROPE 2012 (LCE12)
www.linaro.org
Introduction
“Idle” opposite to “Running”, in between an event occurs.
We want a cpu ...
6
EUROPE 2012 (LCE12)
www.linaro.org
Idle vs Running
When there is no more tasks to run, the scheduler choose
the special ...
7
EUROPE 2012 (LCE12)
www.linaro.org
Idle vs Running – UP
8
EUROPE 2012 (LCE12)
www.linaro.org
Idle vs Running – SMP
9
EUROPE 2012 (LCE12)
www.linaro.org
Package Idle
The deep idle state could be reached only when all cpus are
idle
One cpu...
10
EUROPE 2012 (LCE12)
www.linaro.org
Package idle
11
EUROPE 2012 (LCE12)
www.linaro.org
/proc/interrupts – Snowball U8500
CPU0 CPU1
29: 3150 4124 GIC twd
36: 783 0 GIC Noma...
12
EUROPE 2012 (LCE12)
www.linaro.org
/proc/interrupts – Pandaboard OMAP4
CPU0 CPU1
29: 293 395 GIC twd
41: 0 0 GIC l3-dbg...
13
EUROPE 2012 (LCE12)
www.linaro.org
Challenging the interrupts
Where are they coming from ?
When do they occur ?
How can...
14
EUROPE 2012 (LCE12)
www.linaro.org
From where ?
Hardware interrupt
Network, keyboard, mouse, timer, MMC, USB, serial, ....
15
EUROPE 2012 (LCE12)
www.linaro.org
Hardware interrupt
HW interrupt can happen at arbitrary times
The system could be tw...
16
EUROPE 2012 (LCE12)
www.linaro.org
Hardware interrupt
Interrupt Deterministic Comment
Network No The network stack swit...
17
EUROPE 2012 (LCE12)
www.linaro.org
Timers
A complex infrastructure to handle most of the kernel and
time services for u...
18
EUROPE 2012 (LCE12)
www.linaro.org
Timers
Timer Watchdog (twd)
The timer is local to the CPU
Also known as “local timer...
19
EUROPE 2012 (LCE12)
www.linaro.org
Timers
Used by the kernel for:
Network stack, especially the TCP/IP protocol
Timed-o...
20
EUROPE 2012 (LCE12)
www.linaro.org
IPI : Inter Processor Interrupt
A softirq
Limited to 16 on ARM with the GIC
5 used n...
21
EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (1/3)
Occurs only if SMP and cpuidle with at least retention ...
22
EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (2/3)
[1] : the cpuidle driver tells the time
framework the l...
23
EUROPE 2012 (LCE12)
www.linaro.org
IPI1 : Timer broadcast (3/3)
[1] : the timer expires raising an
interrupt
[2] : the ...
24
EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (1/4)
Used by the scheduler to wake up a processor in ...
25
EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (2/4)
CPU0 is running an application
CPU1 is idle
A ta...
26
EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (3/4)
The application on CPU0 releases
the lock
The sc...
27
EUROPE 2012 (LCE12)
www.linaro.org
IPI2 : Rescheduling interrupt (4/4)
The CPU1 is awake
The scheduler put in the run q...
28
EUROPE 2012 (LCE12)
www.linaro.org
IPI3 : Function Call interrupt
Used to do a remote function invocation on all the CP...
29
EUROPE 2012 (LCE12)
www.linaro.org
IPI3 : Function Call interrupt
Used to do a remote function invocation on all the CP...
30
EUROPE 2012 (LCE12)
www.linaro.org
IPI4 : Single Function Call interrupt
Same as the IPI3 but for a targeted CPU
Happen...
31
EUROPE 2012 (LCE12)
www.linaro.org
How to reduce the number of interrupts ?
Understand the framework and identify the u...
32
EUROPE 2012 (LCE12)
www.linaro.org
Deferrable timers
Used for non critical timers and change their expiration to
occur ...
33
EUROPE 2012 (LCE12)
www.linaro.org
Round jiffies
Used to group timers to expire at the same time slot
How to use it ?
I...
34
EUROPE 2012 (LCE12)
www.linaro.org
Timer and workqueue migration
Schedule work onto running CPUs instead of waking up a...
35
EUROPE 2012 (LCE12)
www.linaro.org
Misc improvements
The kernel does not do everything, userspace applications
have to ...
36
EUROPE 2012 (LCE12)
www.linaro.org
Misc improvements
Kill all busyloop and replace them with an event based
mainloop
ti...
Upcoming SlideShare
Loading in …5
×

LCA13: Who Disturbs My Slumber

539
-1

Published on

Resource: LCA13
Name: Who Disturbs My Slumber
Date: 05-03-2013
Speaker: Daniel Lezcano

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
539
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

LCA13: Who Disturbs My Slumber

  1. 1. More about Linaro Connect: connect.linaro.org More about Linaro: www.linaro.org/about/ More about Linaro engineering: www.linaro.org/engineering/ Hong-Kong (LCE13)
  2. 2. 2 EUROPE 2012 (LCE12) www.linaro.org Linaro Connect Hong-Kong 2013 Who disturbs my slumber ?!
  3. 3. 3 EUROPE 2012 (LCE12) www.linaro.org Introduction The power management is a wide area Power aware scheduler, P-States, C-States, optimizations for idling the cpu, regulators ... We will focus on the sources of wake up for ARM architecture
  4. 4. 4 EUROPE 2012 (LCE12) www.linaro.org PM diagram blocks
  5. 5. 5 EUROPE 2012 (LCE12) www.linaro.org Introduction “Idle” opposite to “Running”, in between an event occurs. We want a cpu to be idle as much as possible without impacting the performance of the system The cpuidle framework takes care of entering the idle state depending on the predictable events on the system The depth of the sleep will depend on the next event on the system What is this event ? An interrupt.
  6. 6. 6 EUROPE 2012 (LCE12) www.linaro.org Idle vs Running When there is no more tasks to run, the scheduler choose the special idle task This one is an infinite loop entering and exiting the arch specific idle function When an interrupt occurs, the function exits and the idle task is yield or enters again into the idle function
  7. 7. 7 EUROPE 2012 (LCE12) www.linaro.org Idle vs Running – UP
  8. 8. 8 EUROPE 2012 (LCE12) www.linaro.org Idle vs Running – SMP
  9. 9. 9 EUROPE 2012 (LCE12) www.linaro.org Package Idle The deep idle state could be reached only when all cpus are idle One cpu exiting the idle state will lead the package to exit the idle state also Software idle states management is challenging on multicore SoCs
  10. 10. 10 EUROPE 2012 (LCE12) www.linaro.org Package idle
  11. 11. 11 EUROPE 2012 (LCE12) www.linaro.org /proc/interrupts – Snowball U8500 CPU0 CPU1 29: 3150 4124 GIC twd 36: 783 0 GIC Nomadik Timer Tick ... 57: 1592 0 GIC dma40 58: 73 0 GIC uart-pl011 72: 0 0 GIC ab8500 79: 187 0 GIC prcmu 87: 0 0 GIC nmk-i2c 92: 4687 0 GIC mmci-pl18x (cmd) 131: 14897 0 GIC mmci-pl18x (cmd) ... IPI0: 0 0 CPU wakeup interrupts IPI1: 0 167 Timer broadcast interrupts IPI2: 386 4908 Rescheduling interrupts IPI3: 0 0 Function call interrupts IPI4: 3 67 Single function call interrupts IPI5: 0 0 CPU stop interrupts Err: 0
  12. 12. 12 EUROPE 2012 (LCE12) www.linaro.org /proc/interrupts – Pandaboard OMAP4 CPU0 CPU1 29: 293 395 GIC twd 41: 0 0 GIC l3-dbg-irq 42: 0 0 GIC l3-app-irq 44: 0 0 GIC DMA 69: 61 0 GIC gp_timer 88: 0 0 GIC i2c.9 89: 0 0 GIC i2c.10 93: 0 0 GIC i2c.11 94: 0 0 GIC i2c.12 106: 93 0 GIC OMAP UART2 169: 0 0 PRCM hwmod_io IPI0: 0 0 Timer broadcast interrupts IPI1: 1424 1260 Rescheduling interrupts IPI2: 0 0 Function call interrupts IPI3: 81 90 Single function call interrupts IPI4: 0 0 CPU stop interrupts Err: 0
  13. 13. 13 EUROPE 2012 (LCE12) www.linaro.org Challenging the interrupts Where are they coming from ? When do they occur ? How can we reduce them ?
  14. 14. 14 EUROPE 2012 (LCE12) www.linaro.org From where ? Hardware interrupt Network, keyboard, mouse, timer, MMC, USB, serial, ... Inter Processor Interrupt : Wisely used today, they are tricky to optimize The architecture differs across the SoC vendor but they are slightly similar for the IPI
  15. 15. 15 EUROPE 2012 (LCE12) www.linaro.org Hardware interrupt HW interrupt can happen at arbitrary times The system could be tweaked depending on the hardware One deterministic interrupt is interesting : the timer
  16. 16. 16 EUROPE 2012 (LCE12) www.linaro.org Hardware interrupt Interrupt Deterministic Comment Network No The network stack switch to polling on high traffic Keyboard No Mouse No MMC No Timer Yes Two kinds of timer USB No Serial No
  17. 17. 17 EUROPE 2012 (LCE12) www.linaro.org Timers A complex infrastructure to handle most of the kernel and time services for userspace programs Two kinds of timer Per cpu timer Global timer
  18. 18. 18 EUROPE 2012 (LCE12) www.linaro.org Timers Timer Watchdog (twd) The timer is local to the CPU Also known as “local timer” It goes down when the processor logic is shutdown Timer device (architecture dependant) Less accurate (eg. 32KHz) Usually always out of the CPU’s power domain Used as backup when the CPU are sleeping
  19. 19. 19 EUROPE 2012 (LCE12) www.linaro.org Timers Used by the kernel for: Network stack, especially the TCP/IP protocol Timed-out IO Delayed work queues Scheduling tasks ... Used by the userspace for: Asynchronous IO and timeout Threading timed lock Mainloop (poll, epoll, select) Posix timers ... In other words : widely used in the system
  20. 20. 20 EUROPE 2012 (LCE12) www.linaro.org IPI : Inter Processor Interrupt A softirq Limited to 16 on ARM with the GIC 5 used nowadays IPI0 : defined but unused so far IPI1 : timer broadcast IPI2 : Rescheduling interrupt IPI3 : Function Call interrupt IPI4 : Single Function Call interrupt IPI5 : Cpu Stop interrupt
  21. 21. 21 EUROPE 2012 (LCE12) www.linaro.org IPI1 : Timer broadcast (1/3) Occurs only if SMP and cpuidle with at least retention mode state Broadcast interrupt could occur on any cpu: idle or not concerned by the timer expiration or not Could be optimized with dynamic timer irq affinity https://lkml.org/lkml/2013/2/19/555 A summary of how timer broadcast works https://lkml.org/lkml/2013/2/20/216
  22. 22. 22 EUROPE 2012 (LCE12) www.linaro.org IPI1 : Timer broadcast (2/3) [1] : the cpuidle driver tells the time framework the local timer is no longer a valid source [2] : the timer framework changes the source and program the timer device [3] / [3'] : the cpuidle driver power downs the cpus
  23. 23. 23 EUROPE 2012 (LCE12) www.linaro.org IPI1 : Timer broadcast (3/3) [1] : the timer expires raising an interrupt [2] : the cpu is woken up [3] : the cpu handles the timer callback and through the time framework … [4] … it sends an IPI to the cpu which are concerned by the expiration of this timer
  24. 24. 24 EUROPE 2012 (LCE12) www.linaro.org IPI2 : Rescheduling interrupt (1/4) Used by the scheduler to wake up a processor in order to run a task Could happen for different reasons: Load balancing / process migration An event occurs for a specific task IO (eg. network ingress packet) A lock has been released for a blocked task Signal delivery
  25. 25. 25 EUROPE 2012 (LCE12) www.linaro.org IPI2 : Rescheduling interrupt (2/4) CPU0 is running an application CPU1 is idle A task on CPU1 is blocked on a lock
  26. 26. 26 EUROPE 2012 (LCE12) www.linaro.org IPI2 : Rescheduling interrupt (3/4) The application on CPU0 releases the lock The scheduler takes the decision to wake up the CPU1... … and send an IPI to it
  27. 27. 27 EUROPE 2012 (LCE12) www.linaro.org IPI2 : Rescheduling interrupt (4/4) The CPU1 is awake The scheduler put in the run queue the task which was previously blocked
  28. 28. 28 EUROPE 2012 (LCE12) www.linaro.org IPI3 : Function Call interrupt Used to do a remote function invocation on all the CPUs when the code must be run in the processor context Setting the timer frequency Setting up the clock event notify framework It happens rarely on the ARM system, but often on x86 system under the term of “TLB shootdowns” which happens at fork time
  29. 29. 29 EUROPE 2012 (LCE12) www.linaro.org IPI3 : Function Call interrupt Used to do a remote function invocation on all the CPUs when the code must be run in the processor context Setting the timer frequency Setting up the clock event notify framework It happens rarely on the ARM system, but often on x86 system under the term of “TLB shootdowns” which happens at fork time
  30. 30. 30 EUROPE 2012 (LCE12) www.linaro.org IPI4 : Single Function Call interrupt Same as the IPI3 but for a targeted CPU Happens rarely on the system
  31. 31. 31 EUROPE 2012 (LCE12) www.linaro.org How to reduce the number of interrupts ? Understand the framework and identify the unnecessary wake up Deferrable timers http://lwn.net/Articles/228143/ Round jiffies http://lkml.org/lkml/2006/10/10/189 RCU no callback http://lwn.net/Articles/522262 Timer and workqueue migration on non idle cpu https://lkml.org/lkml/2012/9/27/188
  32. 32. 32 EUROPE 2012 (LCE12) www.linaro.org Deferrable timers Used for non critical timers and change their expiration to occur when the CPU wakes up How to use it ? Identify non critical timers Flag them deferrable They will be handled when the CPU wakes up for another reason
  33. 33. 33 EUROPE 2012 (LCE12) www.linaro.org Round jiffies Used to group timers to expire at the same time slot How to use it ? Identify timers for non precise timeout Group them to the same time slot by using round_jiffies
  34. 34. 34 EUROPE 2012 (LCE12) www.linaro.org Timer and workqueue migration Schedule work onto running CPUs instead of waking up an idle one Still work in progress
  35. 35. 35 EUROPE 2012 (LCE12) www.linaro.org Misc improvements The kernel does not do everything, userspace applications have to be improved for power management The application could group the timers to expire at the same time if high precision is not needed Round-jiffies like framework ? A specific flag for timer syscalls, in order to make them deferrable ? prctl : PR_SET_TIMERSLACK
  36. 36. 36 EUROPE 2012 (LCE12) www.linaro.org Misc improvements Kill all busyloop and replace them with an event based mainloop timerfd, signalfd, eventfd, etc … Increase MTU size for private network area when possible Use CPU affinity when multiple tasks have to run to sequentially Put pressure on developers to fix the applications and make them more power aware, the kernel can not overcome bad applications

×