1. The document discusses introducing generic energy-awareness into the upstream Linux kernel through a new Energy Aware Scheduling (EAS) framework. 2. EAS aims to provide a common solution for scheduling tasks across different CPU topologies in a way that minimizes energy usage based on an energy model of the hardware. 3. The EAS framework integrates idle state, DVFS, and big.LITTLE support and is designed to be clean, based on measurable energy data, and support future CPU topologies with reduced software maintenance costs.

1. EAS Update
September 2015
Amit Kucheria – LinaroTechnical lead – Power Management
Robin Randhawa –ARM PowersoftTech lead
Ian Rickards –ARM Powersoft Product Manager

2. 2
Motivation
 Hardwaretopologiesarebecomingmorevaried,
accommodatingdifferentpower/performancebudgets:
 SMP,multi-clusterSMP,ARMbig.LITTLEtechnology.
 Percore/perclusterDVFS
(DynamicVoltage&FrequencyScaling)
 Linuxpowermanagementframeworksare
uncoordinatedandhardtotunefordifferenttopologies.
 Weneedacommonupstreamsolution tominimize
softwarecosts.
_All_ policy, all metrics, all
averaging should happen at the
scheduler power saving level, in a
single place, and then the
scheduler should directly drive
the new low level idle state driver
mechanism.
IngoMolnar(31 Mar2013)

3. 3
Goals
Introducegenericenergy-awarenessinupstreamLinux:
1. IntegrateIdle,DVFS,schedulerbig.LITTLEsupport
2. Cleandesignratherthanshort-cuts.
3. Basedonmeasureableenergymodeldataratherthan
magictunables.
4. SupportfutureCPUtopologies
5. MaintainedinupstreamLinux,reducedsoftware
maintenancecosts.
EAS
Energy
model driven
scheduling
Scheduler
driven
DVFS
Analysis
& Tuning
flows
Tools
Performance
enhancements
Idle CPU
improvementsSimple
tunability
Linaro
development
ARM
development

4. 4
StaticPower
 Areaofsilicon(mm2)
 Thresholdvoltage(Vt)
 “LowVt”implementationfasterbut
moreleaky
 “HighVt”implementationslower
 Temperature
DynamicPower
 Togglingnodesxcapacitancexvoltage
Power Fundamentals
Compute
capacity
(Performance)
Little
core
Big
core
Power

5. 5
OS task scheduling– throughputpolicy
Schedulingpolicydecidestaskplacement
 Affectperformanceandenergy
consumption.
MainlineLinuxpolicyis‘workpreserving’
 Considersonlymaximizingthroughput.
 DVFSandidle-statescontrolledby
independentpolicygovernors.
DesignedforSMP,notenergy-aware
Capacity Utilization
0 1 2 3
Max capacity
Waking task
?
CPU
cluster 0 1
Current
capacity

6. 6
OS task scheduling– energy-awarepolicy
Energy-AwareScheduling(EAS)policy:
 PickCPUwithsufficientsparecapacityand
smallestenergyimpact.
Requirements:
1. Trackingoftaskutilization.
2. Platformenergymodel.
Supportsalltopologies,includingSMPand
big.LITTLE.
Capacity Utilization
0 1 2 3
Max capacity
big
Waking task
?
CPU
cluster little big
Current
capacity
Max capacity
little

7. 7
What is EAS – the energy model
Compute
capacity
(Performance)
Capacity Utilization
0 1 2 3
Max capacity
big
Waking task
?
CPU
cluster little big
Current
capacity
Max capacity
little Placing task on cpu1:
P-state change for
cpu0 and cpu1.
Placing task on cpu3:
No P-state changes.
CPU 0, 1
CPU 2, 3
Little
core
Big
core
Power

8. 8
EAS vs big.LITTLEHMP
NewEnergyAwareScheduling
 Genericenergymodelbasedapproach
fitsallplatformsandtopologies.
 Foundationforfurtherenhancements.
ExistingHeterogeneousMPpatchset
 big.LITTLEtopologyonly.
 Hardcodedbehaviors.
 InLinaroLSKkernels (notmainline).
Generic EAS
Foundation
b.L
HMP
(big.LITTLE specific)
+1
+2
Designed-for-
Upstream
Additional
performance
patches
+3
Partner
patches
+1
Additional
tuning patches
Upstreaming
EAS
Linux Linux
Source
patches
Source
patches

9. 9
DVFS in Linux (cpufreq)
 Samplingbasedgovernorsareslowto
respondandhardtotune.
 Samplingtoofast:OPPchangesforsmall
utilizationspikes.
 Samplingtooslow:Suddenburstof
utilizationmightnotgetthenecessaryOPP
changeintime.
cpu
utilization
Time
20% 20% 20%
cpufreq governor
sampling
React!cpu
utilization
0% 30% 100%
Time
Average too low, no response.

10. 10
 WithschedulertaskutilizationtrackingDVFScan
benotifiedimmediatelywhenCPUutilization
changes=improvedresponsiveness.
Scheduler-drivenDVFS
Capacity Utilization
0 1 2 3
Max capacity
big
Waking task
?
CPU
cluster little big
Current
capacity
Max capacity
little
cpu 1
utilization
Time
Now Next sample
Capacity

11. 11
SchedTune
Current:
 Asetofgovernor-specifictunables.
Goal:
 Singletunabletobiastheenergy/performance
trade-off.
Prototypes:
 Globalboosttunable:
/proc/sys/kernel/sched_cfs_boost
 Taskgroup(cgroup)basedtuning:
/sys/fs/cgroup/stune/<group>/schedtune.boost
cpu
utilization
Time
Capacity
Performance
margin

12. 12
Tunability improvements
Existing CFS with HMP
HMP tunables hmp_domains,
up_threshold,
down_threshold,
packing_enable,
packing_limit,
frequency_invariant_load
_scale
Interactive
governor
min_sample_time,
hispeed_freq,
go_hispeed_load,
above_hispeed_delay,
timer_rate, input_boost,
boost, boostpulse
EAS
EAS tunables NONE - energy
model only
SchedTune ‘boost’ margin
{ boost, boostpulse }

13. 13
Analysis tools
Tool name / function Location
rt-app/ WorkloadGen (Linaro)
Variable-intensity workload generator for Linux
https://wiki.linaro.org/WorkingGroups/PowerManagem
ent/Resources/Tools/WorkloadGen
workload-automation (ARM)
Automating benchmark runs and ftrace log capture
(Linux, Android, ChromeOS)
https://github.com/ARM-software/workload-
automation
TRAPpy (ARM)
Python-based visualization tool to help analyze ftrace data generated on a
device. Uses ipython & javascript
https://github.com/ARM-software/trappy
BART (ARM)
Behavior Analysis Regression Testing
Thread residency checker, used as the framework for regression testing for
EAS.
https://github.com/ARM-software/bart
Idlestat (Linaro)
Idlestat uses kernel ftrace to monitor and capture C-state and P-state
transitions of CPUs over a time interval.
https://wiki.linaro.org/WorkingGroups/PowerManagem
ent/Resources/Tools/Idlestat
kernelshark
X11/GTK tool for analysis of ftrace data, useful for detailed scheduler analysis
but does not offer the API capability of ‘trappy’ above.
http://people.redhat.com/srostedt/kernelshark/HTML/

14. 14
Upstream progress
2014 20152013
_All_ policy, all metrics, all averaging should
happen at the scheduler power saving level, in a
single place, and then the scheduler should directly
drive the new low level idle state driver
mechanism.
--- Ingo Molnar (31 Mar 2013)
ARM EAS
RFCv3
Feb-2015
ARM EAS
RFCv4
May-2015
Linaro
sched-
DVFS
Mar-2015
Linaro
shallowest
idle
Sep-2014
MERGED
ARM EAS
RFCv5
Jul-2015
ARM
Sched
Tune RFC
Aug-2015
ARM
Foundation
Patches
Aug-2015
LPC

15. 15
Current patchsets for review/testing
 Request reviewers to send‘tested-by’ or‘acked-by’
Patchset URL
Scheduler driven DVFS PATCH v3 https://lkml.org/lkml/2015/6/26/620
EAS RFCv5 https://lkml.org/lkml/2015/7/7/754
SchedTune proposal https://lkml.org/lkml/2015/8/19/419
Foundational Patches
(frequency and microarchitecture contribution to capacity/utilization,
split out from RFCv5) (already queued for merging!)
https://lkml.org/lkml/2015/8/14/296
Yuyang Du PELT rewrite v10 containing ARM enhancements to
utilization calculation (already queued for merging!)
https://lkml.org/lkml/2015/7/15/159

16. 16
EAS RFCv5 update – posted 7-Jul-2015
 Linaro sched-DVFS integration +ARM
improvements
 Maps all 6 HMP behaviors
 Landed on ChromeOS
 (Linux 3.18 kernel)
 SchedTune equivalent to‘interactive’
HMP behaviors Equivalent in EAS?
Wake migration Yes – from wakeup pathways
Fork migration Yes – new task initialized to max load
Forced migration Yes – from periodic load balance
Idle-pull migration Yes – from idle load balance
Offload migration Yes – one task per CPU
Small task packing Yes – built into design. From energy
model.

17. 17
Results – RFCv5 @ARMTC2
Energy Performance
0
20
40
60
80
100
120
rt-app 6% rt-app 13% rt-app 19% rt-app 25% sysbench
Mainline
+capacity awareness
+EAS
0
20
40
60
80
100
120
sysbench
Lower is better. Higher is better.

18. 18
LSK3.18
 (Linaro–targeting15.10)allowingdirectHMPvs.EAScomparisons
Testing
 Use-cases (ChromeOSthenAndroid)
 Moreplatforms–canyouhelptest?
Androidtesting&tuning targetingDecember2015
 StartingwithHMPsched_tests,migratingto‘bart’tests
Productization
 Analysistools/testsuites/tuningflow&documentation
 Energymodelflow (basedonpower/perfmeasurementsofdhrystoneorsysbench)
EAS Near-term Plans

19. 19
EAS NeedsYOU!
Now get on the list and ACK some patches!