SlideShare a Scribd company logo
1
Trusted Firmware Deep
Dive
Dan Handley
Charles Garcia-Tobin
2
Agenda
 Architecture overview
 Memory usage
 Code organisation
 Cold boot deep dive
 PSCI deep dive
3
Example System Architecture
Guest A
App 1
Guest A
App 2
Guest Linux Kernel A
Guest B
App 1
Guest B
App 2
Trusted
App 1
Trusted
App 2
Trusted OS
Hypervisor
EL0
EL1
EL2
EL3
Normal World Secure World
Boot
Firmware
Boot
Firmware
Runtime EL3 Firmware (Secure Monitor)
EL1 / EL0 Execution
Secure EL1 / EL0 Execution
EL2 Execution
Key
EL3 Execution
UEFI
Guest Linux Kernel B
SMC
Secure EL0
Secure EL1
 SMC Calling Convention Interface (ARM DEN 0028A) is gateway to:
 Runtime EL3 Firmware
 Trusted OS / TEE services
 Power State Coordination Interface (PSCI) (ARM DEN 0022B.b)
 Transported by SMC calls
 Also see ARMv8-A Architecture Manual (AR150-DA-70000)
4
Guest A
App 1
Guest A
App 2
Guest Linux Kernel A
Guest B
App 1
Guest B
App 2
Trusted
App 1
Trusted
App 2
Trusted OS
Hypervisor
EL0
EL1
EL2
EL3
Normal World Secure World
Boot
Firmware
Boot
Firmware
Runtime EL3 Firmware (Secure Monitor)
EL1 / EL0 Execution
Secure EL1 / EL0 Execution
EL2 Execution
Key
EL3 Execution
UEFI
Guest Linux Kernel B
SMC
Secure EL0
Secure EL1
Example System Architecture
 SMC Calling Convention Interface (ARM DEN 0028A) is gateway to:
 Runtime EL3 Firmware
 Trusted OS / TEE services
 Power State Coordination Interface (PSCI) (ARM DEN 0022B.b)
 Transported by SMC calls
 Also see ARMv8-A Architecture Manual (AR150-DA-70000)
ARM Trusted
Firmware
5
ARM Trusted Firmware Architecture
EL3 Firmware - BL31
(Secure Monitor)
SMC Interface
Service Router
Other EL3 Interfaces Interrupt Handler
World Switcher
PSCI
Pwr Ctrl
Driver
EL3 Arch Context
Save/Restore
Normal World Secure World
Interface Usage
External Interface
EL1 Execution
Secure EL1 Execution
EL2 Execution
KeyGlossary
BL - Boot Loader
EDK2 - EFI Development Kit 2
EL - Exception Level
NV - Non-Volatile
PSCI - Power State Control Interface
SMC - Secure Monitor Call
UEFI - Unified Enhanced Firmware Interface
EL3 Execution
Potential Interface
UEFI - BL33
UEFI Secure
Boot
EDK2 Core
I/O Drivers
Boot ROM - BL1
Trusted Board
Boot 1
Trusted Boot
Firmware - BL2
Trusted Board
Boot 2
Cold/Warm
Boot Detection
NV Storage
Driver
Boot Time Arch
+ Platform Init
Temp SMC
Handler
Boot Time Arch
+ Platform Init
Test Secure EL1 Payload - BL32
PSCI
Test
Service Router
Other
Test
S-EL1 Arch
Context
Save/Restore
Interrupt
Handler
Runtime Arch +
Platform Init
Test Suite – BL33_ALT
PSCI
Tests
EL1 Arch Context
Save/Restore
EL2 Arch Context
Save/Restore
Other
Tests
Interrupt
Handler
Runtime Arch
+ Platform InitException Trapper
6
EL3 Firmware - BL31
(Secure Monitor)
SMC Interface
Service Router
Other EL3 Interfaces Interrupt Handler
World Switcher
PSCI
Pwr Ctrl
Driver
EL3 Arch Context
Save/Restore
Normal World Trusted World
Interface Usage
External Interface
EL1 Execution
Secure EL1 Execution
EL2 Execution
KeyGlossary
BL - Boot Loader
EDK2 - EFI Development Kit 2
EL - Exception Level
NV - Non-Volatile
PSCI - Power State Control Interface
SMC - Secure Monitor Call
UEFI - Unified Enhanced Firmware Interface
EL3 Execution
Potential Interface
UEFI - BL33
UEFI Secure
Boot
EDK2 Core
I/O Drivers
Boot ROM - BL1
Trusted Board
Boot 1
Trusted Boot
Firmware - BL2
Trusted Board
Boot 2
Cold/Warm
Boot Detection
NV Storage
Driver
Boot Time Arch
+ Platform Init
Temp SMC
Handler
Boot Time Arch
+ Platform Init
Test Secure EL1 Payload - BL32
PSCI
Test
Service Router
Other
Test
S-EL1 Arch
Context
Save/Restore
Interrupt
Handler
Runtime Arch +
Platform Init
Test Suite – BL33_ALT
PSCI
Tests
EL1 Arch Context
Save/Restore
EL2 Arch Context
Save/Restore
Other
Tests
Interrupt
Handler
Runtime Arch
+ Platform InitException Trapper
ARM Trusted Firmware Architecture
Not Available Yet
Partially Available
7
EL3 Firmware - BL31
(Secure Monitor)
SMC Interface
Service Router
Other EL3 Interfaces Interrupt Handler
World Switcher
PSCI
Pwr Ctrl
Driver
EL3 Arch Context
Save/Restore
Normal World Trusted World
Interface Usage
External Interface
EL1 Execution
Secure EL1 Execution
EL2 Execution
KeyGlossary
BL - Boot Loader
EDK2 - EFI Development Kit 2
EL - Exception Level
NV - Non-Volatile
PSCI - Power State Control Interface
SMC - Secure Monitor Call
UEFI - Unified Enhanced Firmware Interface
EL3 Execution
Potential Interface
UEFI - BL33
UEFI Secure
Boot
EDK2 Core
I/O Drivers
Boot ROM - BL1
Trusted Board
Boot 1
Trusted Boot
Firmware - BL2
Trusted Board
Boot 2
Cold/Warm
Boot Detection
NV Storage
Driver
Boot Time Arch
+ Platform Init
Temp SMC
Handler
Boot Time Arch
+ Platform Init
Test Secure EL1 Payload - BL32
PSCI
Test
Service Router
Other
Test
S-EL1 Arch
Context
Save/Restore
Interrupt
Handler
Runtime Arch +
Platform Init
Test Suite – BL33_ALT
PSCI
Tests
EL1 Arch Context
Save/Restore
EL2 Arch Context
Save/Restore
Other
Tests
Interrupt
Handler
Runtime Arch
+ Platform InitException Trapper
ARM Trusted Firmware Boot Flow
BL33
BL33
(Alternative)
BL32 BL2
BL1
BL31
RESET
2nd level
Boot Loader
(BL2) loads
all 3rd level
images
1st level
Boot Loader
(BL1) loads
2nd level
image
To
Hypervisor /
Linux Kernel
8
Current Memory Usage on FVP
 Secure ROM
 BL1 code
 Semi-hosting
 BL1, BL2, BL31
and Linux kernel
code.
 NOR
 UEFI code
 Virtio Block
 Linux file system
 Intend to use this
for BL images
instead of semi-
hosting
Storage
9
Current Memory Usage on FVP
 Secure ROM
 BL1 code
 Semi-hosting
 BL1, BL2, BL31
and Linux kernel
code.
 NOR
 UEFI code
 Virtio Block
 Linux file system
 Intend to use this
for BL images
instead of semi-
hosting
 Secure ROM
 BL1 code
 Secure SRAM
 BL1 data (bottom)
 BL2 code & data (top)
 BL31 code & data (middle, above BL1)
 Secure DRAM
 Hand-off structures between BL images
 When secure_memory parameter is defined,
will need to program TZC to access DRAM
 NOR
 Early UEFI code
 DRAM
 UEFI code and data
 Linux code and data
Storage Execution
10
Code Organisation
 arch - Architecture specific code (just AArch64 for now)
 plat - Platform specific code (i.e. porting layer)
 bl<X> - BL specific code
 common - architecture/platform neutral code used by all BLs
 lib - library functionality common to all other code
 drivers - e.g. CCI, UART, FVP power controller
 include
 docs
 fdts - not necessarily the final home for these!
 Conforms to Linux coding standard
11
COLD BOOT DEEP DIVE
12
CPU States
 Running. CPU is executing normally
 Idle. States where OS is aware of CPU being idle
 Execution resumes if new OS thread needs scheduling or interrupt is received
 Idle Standby. Typically entered via WFI/WFE and exited via wake-up event
 All caches and memories required for execution remain powered on and
coherent
 All CPU state (context) is preserved
 Execution resumes from line after WFI/WFE
 Idle Retention. Similar to standby except cannot access debug registers
 Idle Power Down. CPU is powered off
 CPU state at each EL must be saved
 Execution starts at the reset vector, after which CPU state at each EL must
be restored
 Local caches may be off
 Off. States where the OS is not using the CPU for scheduling
 Execution starts from the reset vector and no CPU state is restored
 Enabling a CPU in this state requires a Hotplug
13
./bl1/aarch64/bl1_entrypoint.S
14
./bl1/aarch64/bl1_entrypoint.S
Always do basic CPU init
15
./bl1/aarch64/bl1_entrypoint.S
If PSCI provided an entrypoint, then jump to BL31
(either hotplug or resume from idle)
16
./bl1/aarch64/bl1_entrypoint.S
If there's no entrypoint and this is the primary CPU,
then continue with cold boot
17
./bl1/aarch64/bl1_entrypoint.S
Otherwise put the secondary CPU in a safe
state (e.g. On FVP, power off CPU)
18
./plat/fvp/aarch64/bl1_plat_helpers.S
19
./plat/fvp/aarch64/bl1_plat_helpers.S
Initialize the secure memory used by BL1
(On FVP, just zero out the entrypoint mailboxes)
20
./plat/fvp/aarch64/bl1_plat_helpers.S
Create a small stack in "always uncached" memory
to allow "C" code execution as soon as possible
21
./plat/fvp/aarch64/bl1_plat_helpers.S
"early" means "before MMU is enabled"
In this case, just calculate extents of memory
22
./plat/fvp/aarch64/bl1_plat_helpers.S
Now create simple page tables and enable MMU / caches
23
./plat/fvp/aarch64/bl1_plat_helpers.S
Create a stack in normal memory
24
./plat/fvp/aarch64/bl1_plat_helpers.S
Branch to main "C" function,
bl1_main()
25
./bl1/bl1_main.c
26
./bl1/bl1_main.c
Do remaining architectural / platform setup now we are
executing in normal memory with MMU / caches enabled
E.g. control registers, generic timer, CCI snoops, console, ...
27
./bl1/bl1_main.c
Calculate where to load
BL2, load it, then run it
28
BL2 and BL31
 Entrypoints are similar to BL1's platform_cold_boot_init()
 Create a small stack in coherent / always uncached memory)
 Early platform setup (e.g. unpack image hand-off information)
 Platform-specific architectural setup (e.g. enable MMU / caches)
 Create a stack in normal memory
 Branch to main "C" function for remaining platform / arch setup
 BL2 loads BL3<X> images similarly to how BL1 loads BL2
 BL2 passes information about BL33 (e.g. UEFI) to BL31,
before running BL31
 Means that BL31 can jump to BL33 without going back to BL2
 Has to go via BL1 SMC handler to jump from Secure EL1 to EL3
 See SynchronousExceptionA64 in
./bl1/aarch64/early_exceptions.S
29
BL31 Initialization
 Can override any BL1 initialization
 Reinitializes exception vectors, MMU, control registers, etc ...
 Installs runtime SMC handler
 Initializes platform for normal world software
 Initializes GIC, runtime services (e.g. PSCI)
 Returns from exception into normal world boot loader, BL33
(e.g UEFI)
30
PSCI DEEP DIVE
31
PSCI Status
 Work in progress. Key functions for boot and hotplug are
functional. Idle is next on the radar
PSCI Function Implementation Status
PSCI_VERSION OK
CPU_ON OK
CPU_SUSPEND NOK (code present not ready)
CPU_OFF OK
AFFINTY_INFO OK
MIGRATE Not present
MIGRATE_INFO_TYPE Not present
SYSTEM_OFF Not present
SYSTEM_RESET Not present
32
PSCI Topology
 PSCI needs to build a map of system topology
 How many clusters, cores per cluster etc
 Used for last man tracking
 Topology information is provided by the platform using the
following functions:
int plat_get_max_afflvl()
int plat_get_aff_count(unsigned int aff_lvl,
unsigned long mpidr)
unsigned int plat_get_aff_state(unsigned int aff_lvl,
unsigned long mpidr)
33
PSCI Topology
plat_get_max_afflvl()
Returns highest affinity level implemented by the
platform
e.g. For FVP models:
int plat_get_max_afflvl()
{
return MPIDR_AFFLVL1;
}
34
PSCI Topology
plat_get_aff_count(aff_lvl, mpidr)
Given an MPIDR and a level of affinity return how many
instances are implemented at that affinity level
For example consider 2x3 system: 2 clusters, 2 cores in cluster
0, and 3 in cluster 1
mpidr aff_lvl Return Value
0000 (Cluster 0, Core 0)
0001 (Cluster 0, Core 1)
0 2 (two affinity level 0 instances, or
cores, in cluster 0)
1 2 (There are two affinity level 1
instances or clusters in the system)
0100 (Cluster 1, Core 0)
0101 (Cluster 1, Core 1)
0 3 (three affinity level 0 instances, or
cores, in cluster 1)
1 2 (There are two affinity level 1
instances or clusters in the system)
35
PSCI Topology
plat_get_aff_state(aff_lvl, mpidr)
 Returns whether an affinity instance is present or absent
 You can use it to deal with hierarchies that are asymmetric
 For example a cluster and a single core sharing an interconnect
 Saves on having to take locks for affinity levels that don’t exist
36
bl31_entrypoint
bl31_main
bl31_platform_setup
plat_setup_topology
runtime_svc_init
psci_setup
FVP topology
 FVP model sets up topology information as part of cold boot
path (called from primary CPU)
 plat_setup_topology() sets up necessary data to allow
the following to work
 plat_get_max_aff_lvl(),
plat_get_aff_count(), plat_get_aff_state()
 BL31 then moves on to set up PSCI
Bl31/AArch64/bl31_entrypoint.S
Bl31/bl31_main.c
Plat/fvp/bl31_plat_setup.c
Plat/fvp/Fvp_topology.c
37
PSCI Deep Dive
 After cold boot path calls plat_setup_topology(), it calls
psci_setup()
 This functions create a topology map for the system based on the
platform specific functions
 Map is an array of aff_map_node pointers
typedef struct {
unsigned long mpidr; //mpidr of node
unsigned char state; //[present||absent]|[PSCI state]
char level; //aff level
unsigned int data; //cookie (holds index into
//non-secure data for CPU_ON/CPU_SUSPEND
bakery_lock lock; //Lock for node
} aff_map_node;
(bakery_lock will be replaced with a more abstract lock API)
See psci_setup.c/psci_common.c/psci_private.h
38
PSCI Topology
 psci_aff_map holds topology tree (array of aff_map_node (s))
 Held in device ordered memory
 Array is populated in a breadth first way
Aff 3 entities Aff 2 entities Aff 1 entities Aff 0 entities
 e.g. for a 2x2 system
Cluster 0 Cluster 1 CPU 0.0 CPU 0.1 CPU 1.0 CPU 1.1
 Additional affinity information arrays:
psci_aff_limits : indices to start and end of each affinty level in
topology tree
psci_ns_entry_info: array of ns_entry_info. Structure to hold
entry information into EL[2|1] for CPU_ON/CPU_SUSPEND
psci_secure_context: array of secure_context. Structure to hold
secure context information that needs saving when powering down
39
PSCI Topology
 psci_setup_up completes by:
 Initialising the state of the primary CPU (and containing higher affinity
levels, e.g cluster) to PSCI_STATE_ON
 All others default to PSCI_STATE_OFF
 Calls into platform to set up platform specific PSCI operations
platfvpFvp_pm.c
int platform_setup_pm(plat_pm_ops **plat_ops)
{
*plat_ops = &fvp_plat_pm_ops;
return 0;
}
static plat_pm_ops fvp_plat_pm_ops = {
0, // standby (not used in FVP)
fvp_affinst_on, // cpu_on will come here
fvp_affinst_off, // cpu_off will come here
fvp_affinst_suspend, // cpu_suspend
fvp_affinst_on_finish, // called on wake up path of cpu being turned on
fvp_affinst_suspend_finish, // called on wake up path of cpu waking from suspend
};
40
__psci_cpu_off
common/psci/psci_entry.S
__psci_cpu_off:
func_prologue
sub sp, sp, #0x10
stp x19, x20, [sp, #0]
mov x19, sp
bl read_mpidr
bl platform_set_coherent_stack << Switch stack
bl psci_cpu_off
mov x1, #PSCI_E_SUCCESS
cmp x0, x1
b.eq final_wfi
mov sp, x19
ldp x19, x20, [sp,#0]
add sp, sp, #0x10
func_epilogue
ret
41
__psci_cpu_off
common/psci/psci_entry.S
__psci_cpu_off:
func_prologue
sub sp, sp, #0x10
stp x19, x20, [sp, #0]
mov x19, sp
bl read_mpidr
bl platform_set_coherent_stack
bl psci_cpu_off << do work of switching off
mov x1, #PSCI_E_SUCCESS
cmp x0, x1
b.eq final_wfi
mov sp, x19
ldp x19, x20, [sp,#0]
add sp, sp, #0x10
func_epilogue
ret
42
psci_cpu_off
common/psci/psci_main.c
int psci_cpu_off(void)
{
…
int target_afflvl = get_max_afflvl();
mpidr = read_mpidr();
/*
* Traverse from the highest to the lowest affinity level. When the
* lowest affinity level is hit, all the locks are acquired. State
* management is done immediately followed by cpu, cluster ...
* ..target_afflvl specific actions as this function unwinds back.
*/
rc = psci_afflvl_off(mpidr, target_afflvl, MPIDR_AFFLVL0);
if (rc != PSCI_E_SUCCESS) {
assert(rc == PSCI_E_DENIED);
}
return rc;
}
43
Affinity Level 1/Cluster
Affinity Level 0/CPU
Affinity Level 1/ClusterTake Cluster Node Lock
Not at AffinityLevel0 so recurse down a level
Take CPU Node Lock
cpustate = CPU_OFF
if last man cluster state = OFF
psci_afflvl0_off(…) //bl31/common/psci_afflvl_off.c
flush PoU
fvp_affinst_off(..,AffLevel0,…) //plat/fvp/Fvp_pm.c
take core out of coherency
prevent IRQ spurious wakeups
power power controller
Release CPU Node Lock
psci_afflvl1_off(…) //bl31/common/psci_afflvl_off.c
if cluster_state == OFF
flush to PoC
fvp_affinst_off(..,AffLevel1,…) //plat/fvp/Fvp_pm.c
disable cci
power power controller
Release Cluster Node Lock
psci_afflvl_off – FVP success
44
psci_afflvl_off
common/psci/psci_aff_lvl_off.c
int psci_afflvl_off(unsigned long mpidr,int
cur_afflvl, int tgt_afflvl)
{
…
bakery_lock_get(mpidr, &aff_node->lock);
/* Keep the old state and the next one handy */
prev_state = psci_get_state(aff_node->state);
next_state = PSCI_STATE_OFF;
/*
* We start from the highest affinity level
* and work our way
* downwards to the lowest i.e. MPIDR_AFFLVL0.
*/
if (aff_node->level == tgt_afflvl) {
psci_change_state(mpidr,
tgt_afflvl,
get_max_afflvl(),
next_state);
} else {
rc = psci_afflvl_off(mpidr, level - 1,
tgt_afflvl);
if (rc != PSCI_E_SUCCESS) {
psci_set_state(aff_node->state,
prev_state);
goto exit;
}
}
…
rc = psci_afflvl_off_handlers[level](mpidr,
aff_node);
if (rc != PSCI_E_SUCCESS) {
psci_set_state(aff_node->state, prev_state);
goto exit;
}
…
exit:
bakery_lock_release(mpidr, &aff_node->lock);
return rc;
}
When we get to CPU
Level (Aff0) we set state.
This also sets higher
affinity level states if last
man
45
psci_afflvl_off
common/psci/psci_aff_lvl_off.c
int psci_afflvl_off(unsigned long mpidr,int
cur_afflvl, int tgt_afflvl)
{
…
bakery_lock_get(mpidr, &aff_node->lock);
/* Keep the old state and the next one handy */
prev_state = psci_get_state(aff_node->state);
next_state = PSCI_STATE_OFF;
/*
* We start from the highest affinity level
* and work our way
* downwards to the lowest i.e. MPIDR_AFFLVL0.
*/
if (aff_node->level == tgt_afflvl) {
psci_change_state(mpidr,
tgt_afflvl,
get_max_afflvl(),
next_state);
} else {
rc = psci_afflvl_off(mpidr, level - 1,
tgt_afflvl);
if (rc != PSCI_E_SUCCESS) {
psci_set_state(aff_node->state,
prev_state);
goto exit;
}
}
…
rc = psci_afflvl_off_handlers[level](mpidr,
aff_node);
if (rc != PSCI_E_SUCCESS) {
psci_set_state(aff_node->state, prev_state);
goto exit;
}
…
exit:
bakery_lock_release(mpidr, &aff_node->lock);
return rc;
}
Handlers do actual
powering down
46
__psci_cpu_off
common/psci/psci_entry.S
__psci_cpu_off:
func_prologue
sub sp, sp, #0x10
stp x19, x20, [sp, #0]
mov x19, sp
bl read_mpidr
bl platform_set_coherent_stack
bl psci_cpu_off
mov x1, #PSCI_E_SUCCESS
cmp x0, x1
b.eq final_wfi << all OK final WFI
mov sp, x19
ldp x19, x20, [sp,#0]
add sp, sp, #0x10
func_epilogue
ret
47
__psci_cpu_off
common/psci/psci_entry.S
__psci_cpu_off:
func_prologue
sub sp, sp, #0x10
stp x19, x20, [sp, #0]
mov x19, sp
bl read_mpidr
bl platform_set_coherent_stack
bl psci_cpu_off
mov x1, #PSCI_E_SUCCESS
cmp x0, x1
b.eq final_wfi
mov sp, x19 << else switch stack back and return
ldp x19, x20, [sp,#0]
add sp, sp, #0x10
func_epilogue
ret
48
psci_cpu_on
common/psci/psci_main.c
int psci_cpu_on(unsigned long target_cpu, unsigned long entrypoint,
unsigned long context_id)
{
int rc;
unsigned int start_afflvl, target_afflvl;
/* Determine if the cpu exists of not */
rc = psci_validate_mpidr(target_cpu, MPIDR_AFFLVL0);
if (rc != PSCI_E_SUCCESS) {
goto exit;
}
start_afflvl = get_max_afflvl();
target_afflvl = MPIDR_AFFLVL0;
rc = psci_afflvl_on(target_cpu,
entrypoint,
context_id,
start_afflvl,
target_afflvl);
exit:
return rc;
}
Basic error checking
Heavy lifting
49
Affinity Level 0/CPU
Affinity Level 1/Cluster
Affinity Level 0/CPU
Affinity Level 1/Cluster
Affinity Level 0/CPU
Affinity Level 1/Cluster
Affinity Level 0/CPU
Affinity Level 1/ClusterTake Cluster Node Lock
Take CPU Node Lock
psci_afflvl1_on(…) //bl31/common/psci_afflvl_on.c
fvp_affinst_on(..,AffLevel1,…) //plat/fvp/Fvp_pm.c
basic validation
psci_afflvl0_on(…) //bl31/common/psci_afflvl_on.c
store ns_entry_point and context_id (passed from by OSPM)
fvp_affinst_on(..,AffLevel0,…) //plat/fvp/Fvp_pm.c
Wait for any pending off to complete (CPU that you are turning ON,
could have been turning itself OFF)
Set up a mailbox so booting core takes warm boot path
program power controller
Change CPU Node state to ON_PENDING
Change Cluster Node state to ON_PENDING
Release CPU Node Lock
Release Cluster Node Lock
psci_afflvl_on – FVP success
50
psci_afflvl_on
common/psci/psci_aff_lvl_on.c
int psci_afflvl_on(unsigned long target_cpu,
unsigned long entrypoint,
unsigned long context_id,
int current_afflvl,
int target_afflvl)
{
…
for (level = current_afflvl;
level >= target_afflvl; level--) {
aff_node = psci_get_aff_map_node…
if (aff_node)
bakery_lock_get(mpidr, &aff_node->lock);
}
…
for (level = current_afflvl;
level >= target_afflvl; level--) {
… psci_afflvl_on_handlers[level](target_cpu,
aff_node,
entrypoint,
context_id);
}
/*
* State management: Update the states …
*/
psci_change_state(target_cpu,
target_afflvl,
get_max_afflvl(),
PSCI_STATE_ON_PENDING);
exit:
…
for (level = target_afflvl;
level <= current_afflvl; level++) {
…
bakery_lock_release(mpidr,
&aff_node->lock);
…
}
Unlock
Call each level’s on handler
Set Hierarchy to ON_PENDING
Lock
51
fvp_affinst_on
plat/fvp/Fvp_pm.c
int fvp_affinst_on(unsigned long mpidr,
unsigned long sec_entrypoint,
unsigned long ns_entrypoint,
unsigned int afflvl,
unsigned int state)
{
…
if (ns_entrypoint < DRAM_BASE) {
rc = PSCI_E_INVALID_PARAMS;
goto exit;
}
…
if (afflvl != MPIDR_AFFLVL0)
goto exit;
…
/*
* Ensure that we do not cancel an inflight
* power off request
* for the target cpu. That would leave
* it in a zombie wfi…
*/
do {
psysr = fvp_pwrc_read_psysr(mpidr);
} while (psysr & PSYSR_AFF_L0);
linear_id = platform_get_core_pos(mpidr);
fvp_mboxes = (mailbox *) (TZDRAM_BASE +
MBOX_OFF);
fvp_mboxes[linear_id].value = sec_entrypoint;
flush_dcache_range(
(unsigned long) &fvp_mboxes[linear_id],
sizeof(unsigned long));
fvp_pwrc_write_pponr(mpidr);
…
Ensure entry point is valid
Deal with potential race with CPU_OFF
Set up warm boot
Program power controller
52
./bl1/aarch64/bl1_entrypoint.S
If PSCI provided an entrypoint, then jump to BL31
(either hotplug or resume from idle)
53
common/psci/psci_entry.S
Call psci_afflvl_power_on_finish
on coherent stacks
Jump into OSPM entry point
54
Affinity Level 0/CPU
Affinity Level 1/Cluster
Affinity Level 0/CPU
Affinity Level 1/Cluster
Affinity Level 1/Cluster
Affinity Level 0/CPU
psci_afflvl_power_on_finish
Take CPU Node Lock //bl31/common/psci_common.c
psci_afflvl0_on_finish(…) //bl31/common/psci_afflvl_on.c
fvp_affinst_on_finish(..,AffLevel0,…) //plat/fvp/Fvp_pm.c
turn on intra cluster coherency
zero out mailbox, enable GIC, enable access to system counter
install exception handlers,enable mmu and caching, EL3 setup
psci_get_ns_entry_info() //bl31/common/psci_common.c
set up return non-secure Exception Level
Take Cluster Node Lock
psci_afflvl1_on_finish(…) //bl31/common/psci_afflvl_on.c
fvp_affinst_on_finish(..,AffLevel1,…) //plat/fvp/Fvp_pm.c
enable CCI
update CPU Node State to ON
update cluster Node State to ON
Release Cluster Node Lock
Release CPU Node Lock
55
Further reading…
 GitHub
https://github.com/ARM-software/arm-trusted-firmware
 Usage Guide
https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/user-
guide.md
 Porting Guide
https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/porting-
guide.md
 SMC Calling Convention
http://infocenter.arm.com/help/topic/com.arm.doc.den0028a/index.html
 PSCI spec
http://infocenter.arm.com/help/topic/com.arm.doc.den0022b/index.html
 ARMv8 ARM
http://infocenter.arm.com/help/topic/com.arm.doc.ddi0487a.a_errata1/index.html

More Related Content

What's hot

U Boot or Universal Bootloader
U Boot or Universal BootloaderU Boot or Universal Bootloader
U Boot or Universal Bootloader
Satpal Parmar
 
Introduction to Optee (26 may 2016)
Introduction to Optee (26 may 2016)Introduction to Optee (26 may 2016)
Introduction to Optee (26 may 2016)
Yannick Gicquel
 
Lcu14 107- op-tee on ar mv8
Lcu14 107- op-tee on ar mv8Lcu14 107- op-tee on ar mv8
Lcu14 107- op-tee on ar mv8
Linaro
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernelEmbedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
Emertxe Information Technologies Pvt Ltd
 
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
The Linux Foundation
 
Android Things : Building Embedded Devices
Android Things : Building Embedded DevicesAndroid Things : Building Embedded Devices
Android Things : Building Embedded Devices
Emertxe Information Technologies Pvt Ltd
 
Basic Linux Internals
Basic Linux InternalsBasic Linux Internals
Basic Linux Internals
mukul bhardwaj
 
LCA13: Power State Coordination Interface
LCA13: Power State Coordination InterfaceLCA13: Power State Coordination Interface
LCA13: Power State Coordination Interface
Linaro
 
Linux device drivers
Linux device drivers Linux device drivers
RISC-V Boot Process: One Step at a Time
RISC-V Boot Process: One Step at a TimeRISC-V Boot Process: One Step at a Time
RISC-V Boot Process: One Step at a Time
Atish Patra
 
Linux Kernel Overview
Linux Kernel OverviewLinux Kernel Overview
Linux Kernel Overview
Anil Kumar Pugalia
 
A practical guide to buildroot
A practical guide to buildrootA practical guide to buildroot
A practical guide to buildroot
Emertxe Information Technologies Pvt Ltd
 
Embedded Android : System Development - Part III
Embedded Android : System Development - Part IIIEmbedded Android : System Development - Part III
Embedded Android : System Development - Part III
Emertxe Information Technologies Pvt Ltd
 
Embedded_Linux_Booting
Embedded_Linux_BootingEmbedded_Linux_Booting
Embedded_Linux_Booting
Rashila Rr
 
Linux : PSCI
Linux : PSCILinux : PSCI
Linux : PSCI
Mr. Vengineer
 
USB Drivers
USB DriversUSB Drivers
USB Drivers
Anil Kumar Pugalia
 
Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0
Emertxe Information Technologies Pvt Ltd
 
U boot porting guide for SoC
U boot porting guide for SoCU boot porting guide for SoC
U boot porting guide for SoC
Macpaul Lin
 
Linux Audio Drivers. ALSA
Linux Audio Drivers. ALSALinux Audio Drivers. ALSA
Linux Audio Drivers. ALSA
GlobalLogic Ukraine
 

What's hot (20)

U Boot or Universal Bootloader
U Boot or Universal BootloaderU Boot or Universal Bootloader
U Boot or Universal Bootloader
 
Introduction to Optee (26 may 2016)
Introduction to Optee (26 may 2016)Introduction to Optee (26 may 2016)
Introduction to Optee (26 may 2016)
 
Lcu14 107- op-tee on ar mv8
Lcu14 107- op-tee on ar mv8Lcu14 107- op-tee on ar mv8
Lcu14 107- op-tee on ar mv8
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
 
Embedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernelEmbedded Linux Kernel - Build your custom kernel
Embedded Linux Kernel - Build your custom kernel
 
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall...
 
Android Things : Building Embedded Devices
Android Things : Building Embedded DevicesAndroid Things : Building Embedded Devices
Android Things : Building Embedded Devices
 
Basic Linux Internals
Basic Linux InternalsBasic Linux Internals
Basic Linux Internals
 
LCA13: Power State Coordination Interface
LCA13: Power State Coordination InterfaceLCA13: Power State Coordination Interface
LCA13: Power State Coordination Interface
 
Linux device drivers
Linux device drivers Linux device drivers
Linux device drivers
 
RISC-V Boot Process: One Step at a Time
RISC-V Boot Process: One Step at a TimeRISC-V Boot Process: One Step at a Time
RISC-V Boot Process: One Step at a Time
 
Linux Kernel Overview
Linux Kernel OverviewLinux Kernel Overview
Linux Kernel Overview
 
A practical guide to buildroot
A practical guide to buildrootA practical guide to buildroot
A practical guide to buildroot
 
Embedded Android : System Development - Part III
Embedded Android : System Development - Part IIIEmbedded Android : System Development - Part III
Embedded Android : System Development - Part III
 
Embedded_Linux_Booting
Embedded_Linux_BootingEmbedded_Linux_Booting
Embedded_Linux_Booting
 
Linux : PSCI
Linux : PSCILinux : PSCI
Linux : PSCI
 
USB Drivers
USB DriversUSB Drivers
USB Drivers
 
Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0Linux Internals - Interview essentials 4.0
Linux Internals - Interview essentials 4.0
 
U boot porting guide for SoC
U boot porting guide for SoCU boot porting guide for SoC
U boot porting guide for SoC
 
Linux Audio Drivers. ALSA
Linux Audio Drivers. ALSALinux Audio Drivers. ALSA
Linux Audio Drivers. ALSA
 

Viewers also liked

ARM Trusted FirmwareのBL31を単体で使う!
ARM Trusted FirmwareのBL31を単体で使う!ARM Trusted FirmwareのBL31を単体で使う!
ARM Trusted FirmwareのBL31を単体で使う!
Mr. Vengineer
 
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
SFO15-TR9: PSCI, ACPI (and UEFI to boot)SFO15-TR9: PSCI, ACPI (and UEFI to boot)
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
Linaro
 
LCU14 302- How to port OP-TEE to another platform
LCU14 302- How to port OP-TEE to another platformLCU14 302- How to port OP-TEE to another platform
LCU14 302- How to port OP-TEE to another platform
Linaro
 
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARMSFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
Linaro
 
Linux Porting to a Custom Board
Linux Porting to a Custom BoardLinux Porting to a Custom Board
Linux Porting to a Custom Board
Patrick Bellasi
 
U-Boot presentation 2013
U-Boot presentation  2013U-Boot presentation  2013
U-Boot presentation 2013
Wave Digitech
 
LCU14-103: How to create and run Trusted Applications on OP-TEE
LCU14-103: How to create and run Trusted Applications on OP-TEELCU14-103: How to create and run Trusted Applications on OP-TEE
LCU14-103: How to create and run Trusted Applications on OP-TEE
Linaro
 
BKK16-300 Benchmarking 102
BKK16-300 Benchmarking 102BKK16-300 Benchmarking 102
BKK16-300 Benchmarking 102
Linaro
 
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
Linaro
 
Q2.12: Power Management Across OSs
Q2.12: Power Management Across OSsQ2.12: Power Management Across OSs
Q2.12: Power Management Across OSs
Linaro
 
Lcu14 306 - OP-TEE Future Enhancements
Lcu14 306 - OP-TEE Future EnhancementsLcu14 306 - OP-TEE Future Enhancements
Lcu14 306 - OP-TEE Future Enhancements
Linaro
 
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
Mr. Vengineer
 
Note - (EDK2) Acpi Tables Compile and Install
Note - (EDK2) Acpi Tables Compile and InstallNote - (EDK2) Acpi Tables Compile and Install
Note - (EDK2) Acpi Tables Compile and Install
boyw165
 
Xilinx SDSoC(2016.3)解体新書ソフトウェア編
Xilinx SDSoC(2016.3)解体新書ソフトウェア編Xilinx SDSoC(2016.3)解体新書ソフトウェア編
Xilinx SDSoC(2016.3)解体新書ソフトウェア編
Mr. Vengineer
 

Viewers also liked (14)

ARM Trusted FirmwareのBL31を単体で使う!
ARM Trusted FirmwareのBL31を単体で使う!ARM Trusted FirmwareのBL31を単体で使う!
ARM Trusted FirmwareのBL31を単体で使う!
 
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
SFO15-TR9: PSCI, ACPI (and UEFI to boot)SFO15-TR9: PSCI, ACPI (and UEFI to boot)
SFO15-TR9: PSCI, ACPI (and UEFI to boot)
 
LCU14 302- How to port OP-TEE to another platform
LCU14 302- How to port OP-TEE to another platformLCU14 302- How to port OP-TEE to another platform
LCU14 302- How to port OP-TEE to another platform
 
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARMSFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
SFO15-205: OP-TEE Content Decryption with Microsoft PlayReady on ARM
 
Linux Porting to a Custom Board
Linux Porting to a Custom BoardLinux Porting to a Custom Board
Linux Porting to a Custom Board
 
U-Boot presentation 2013
U-Boot presentation  2013U-Boot presentation  2013
U-Boot presentation 2013
 
LCU14-103: How to create and run Trusted Applications on OP-TEE
LCU14-103: How to create and run Trusted Applications on OP-TEELCU14-103: How to create and run Trusted Applications on OP-TEE
LCU14-103: How to create and run Trusted Applications on OP-TEE
 
BKK16-300 Benchmarking 102
BKK16-300 Benchmarking 102BKK16-300 Benchmarking 102
BKK16-300 Benchmarking 102
 
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
Q2.12: Idling ARMs in a busy world: Linux Power Management for ARM Multiclust...
 
Q2.12: Power Management Across OSs
Q2.12: Power Management Across OSsQ2.12: Power Management Across OSs
Q2.12: Power Management Across OSs
 
Lcu14 306 - OP-TEE Future Enhancements
Lcu14 306 - OP-TEE Future EnhancementsLcu14 306 - OP-TEE Future Enhancements
Lcu14 306 - OP-TEE Future Enhancements
 
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
ZynqMPのブートとパワーマネージメント : (ZynqMP Boot and Power Management)
 
Note - (EDK2) Acpi Tables Compile and Install
Note - (EDK2) Acpi Tables Compile and InstallNote - (EDK2) Acpi Tables Compile and Install
Note - (EDK2) Acpi Tables Compile and Install
 
Xilinx SDSoC(2016.3)解体新書ソフトウェア編
Xilinx SDSoC(2016.3)解体新書ソフトウェア編Xilinx SDSoC(2016.3)解体新書ソフトウェア編
Xilinx SDSoC(2016.3)解体新書ソフトウェア編
 

Similar to Trusted firmware deep_dive_v1.0_

Bootloaders (U-Boot)
Bootloaders (U-Boot) Bootloaders (U-Boot)
Bootloaders (U-Boot)
Omkar Rane
 
Slimline Open Firmware
Slimline Open FirmwareSlimline Open Firmware
Slimline Open Firmware
Heiko Joerg Schick
 
Juniper Trouble Shooting
Juniper Trouble ShootingJuniper Trouble Shooting
Juniper Trouble Shooting
Mike(Haobin) Zheng
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New Hardware
RuggedBoardGroup
 
Information Gathering 2
Information Gathering 2Information Gathering 2
Information Gathering 2
Aero Plane
 
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner FischerOSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
NETWAYS
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogic
Aleem Shariff
 
Когда предрелизный не только софт
Когда предрелизный не только софтКогда предрелизный не только софт
Когда предрелизный не только софт
CEE-SEC(R)
 
Rloader, alternative tech to achieve fast boot time for ARM Linux
Rloader, alternative tech to achieve fast boot time for ARM LinuxRloader, alternative tech to achieve fast boot time for ARM Linux
Rloader, alternative tech to achieve fast boot time for ARM Linux
matt_hsu
 
Embedding Linux On The Encore Simputer
Embedding Linux On The Encore SimputerEmbedding Linux On The Encore Simputer
Embedding Linux On The Encore Simputer
Satpal Parmar
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eve
chiportal
 
Advanced Root Cause Analysis
Advanced Root Cause AnalysisAdvanced Root Cause Analysis
Advanced Root Cause Analysis
Eric Sloof
 
Analisis_avanzado_vmware
Analisis_avanzado_vmwareAnalisis_avanzado_vmware
Analisis_avanzado_vmware
virtualizacionTV
 
Ch04
Ch04Ch04
Ch04 system administration
Ch04 system administration Ch04 system administration
Ch04 system administration
Raja Waseem Akhtar
 
Nvidia tegra K1 Presentation
Nvidia tegra K1 PresentationNvidia tegra K1 Presentation
Nvidia tegra K1 Presentation
ANURAG SEKHSARIA
 
1.2 boot the system v2
1.2 boot the system v21.2 boot the system v2
1.2 boot the system v2
Acácio Oliveira
 
Linux Booting Process
Linux Booting ProcessLinux Booting Process
Linux Booting Process
Rishabh5121993
 
Network Drivers
Network DriversNetwork Drivers
Network Drivers
Anil Kumar Pugalia
 
LCA13: CPUIDLE: One driver to rule them all?
LCA13: CPUIDLE: One driver to rule them all?LCA13: CPUIDLE: One driver to rule them all?
LCA13: CPUIDLE: One driver to rule them all?
Linaro
 

Similar to Trusted firmware deep_dive_v1.0_ (20)

Bootloaders (U-Boot)
Bootloaders (U-Boot) Bootloaders (U-Boot)
Bootloaders (U-Boot)
 
Slimline Open Firmware
Slimline Open FirmwareSlimline Open Firmware
Slimline Open Firmware
 
Juniper Trouble Shooting
Juniper Trouble ShootingJuniper Trouble Shooting
Juniper Trouble Shooting
 
U-Boot Porting on New Hardware
U-Boot Porting on New HardwareU-Boot Porting on New Hardware
U-Boot Porting on New Hardware
 
Information Gathering 2
Information Gathering 2Information Gathering 2
Information Gathering 2
 
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner FischerOSMC 2014: Server Hardware Monitoring done right | Werner Fischer
OSMC 2014: Server Hardware Monitoring done right | Werner Fischer
 
Armboot process zeelogic
Armboot process zeelogicArmboot process zeelogic
Armboot process zeelogic
 
Когда предрелизный не только софт
Когда предрелизный не только софтКогда предрелизный не только софт
Когда предрелизный не только софт
 
Rloader, alternative tech to achieve fast boot time for ARM Linux
Rloader, alternative tech to achieve fast boot time for ARM LinuxRloader, alternative tech to achieve fast boot time for ARM Linux
Rloader, alternative tech to achieve fast boot time for ARM Linux
 
Embedding Linux On The Encore Simputer
Embedding Linux On The Encore SimputerEmbedding Linux On The Encore Simputer
Embedding Linux On The Encore Simputer
 
Track c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eveTrack c-High speed transaction-based hw-sw coverification -eve
Track c-High speed transaction-based hw-sw coverification -eve
 
Advanced Root Cause Analysis
Advanced Root Cause AnalysisAdvanced Root Cause Analysis
Advanced Root Cause Analysis
 
Analisis_avanzado_vmware
Analisis_avanzado_vmwareAnalisis_avanzado_vmware
Analisis_avanzado_vmware
 
Ch04
Ch04Ch04
Ch04
 
Ch04 system administration
Ch04 system administration Ch04 system administration
Ch04 system administration
 
Nvidia tegra K1 Presentation
Nvidia tegra K1 PresentationNvidia tegra K1 Presentation
Nvidia tegra K1 Presentation
 
1.2 boot the system v2
1.2 boot the system v21.2 boot the system v2
1.2 boot the system v2
 
Linux Booting Process
Linux Booting ProcessLinux Booting Process
Linux Booting Process
 
Network Drivers
Network DriversNetwork Drivers
Network Drivers
 
LCA13: CPUIDLE: One driver to rule them all?
LCA13: CPUIDLE: One driver to rule them all?LCA13: CPUIDLE: One driver to rule them all?
LCA13: CPUIDLE: One driver to rule them all?
 

More from Linaro

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Linaro
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Linaro
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Linaro
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
Linaro
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
Linaro
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
Linaro
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
Linaro
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Linaro
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Linaro
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
Linaro
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
Linaro
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
Linaro
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
Linaro
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
Linaro
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
Linaro
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
Linaro
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
Linaro
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
Linaro
 

More from Linaro (20)

Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea GalloDeep Learning Neural Network Acceleration at the Edge - Andrea Gallo
Deep Learning Neural Network Acceleration at the Edge - Andrea Gallo
 
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta VekariaArm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
Arm Architecture HPC Workshop Santa Clara 2018 - Kanta Vekaria
 
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua MoraHuawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
Huawei’s requirements for the ARM based HPC solution readiness - Joshua Mora
 
Bud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qaBud17 113: distribution ci using qemu and open qa
Bud17 113: distribution ci using qemu and open qa
 
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
OpenHPC Automation with Ansible - Renato Golin - Linaro Arm HPC Workshop 2018
 
HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018HPC network stack on ARM - Linaro HPC Workshop 2018
HPC network stack on ARM - Linaro HPC Workshop 2018
 
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
It just keeps getting better - SUSE enablement for Arm - Linaro HPC Workshop ...
 
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
Intelligent Interconnect Architecture to Enable Next Generation HPC - Linaro ...
 
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
Yutaka Ishikawa - Post-K and Arm HPC Ecosystem - Linaro Arm HPC Workshop Sant...
 
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
Andrew J Younge - Vanguard Astra - Petascale Arm Platform for U.S. DOE/ASC Su...
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening KeynoteHKG18-100K1 - George Grey: Opening Keynote
HKG18-100K1 - George Grey: Opening Keynote
 
HKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP WorkshopHKG18-318 - OpenAMP Workshop
HKG18-318 - OpenAMP Workshop
 
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainlineHKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
HKG18-501 - EAS on Common Kernel 4.14 and getting (much) closer to mainline
 
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and allHKG18-315 - Why the ecosystem is a wonderful thing, warts and all
HKG18-315 - Why the ecosystem is a wonderful thing, warts and all
 
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse HypervisorHKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
HKG18- 115 - Partitioning ARM Systems with the Jailhouse Hypervisor
 
HKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMUHKG18-TR08 - Upstreaming SVE in QEMU
HKG18-TR08 - Upstreaming SVE in QEMU
 
HKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8MHKG18-113- Secure Data Path work with i.MX8M
HKG18-113- Secure Data Path work with i.MX8M
 
HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation HKG18-120 - Devicetree Schema Documentation and Validation
HKG18-120 - Devicetree Schema Documentation and Validation
 
HKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted bootHKG18-223 - Trusted FirmwareM: Trusted boot
HKG18-223 - Trusted FirmwareM: Trusted boot
 

Trusted firmware deep_dive_v1.0_

  • 1. 1 Trusted Firmware Deep Dive Dan Handley Charles Garcia-Tobin
  • 2. 2 Agenda  Architecture overview  Memory usage  Code organisation  Cold boot deep dive  PSCI deep dive
  • 3. 3 Example System Architecture Guest A App 1 Guest A App 2 Guest Linux Kernel A Guest B App 1 Guest B App 2 Trusted App 1 Trusted App 2 Trusted OS Hypervisor EL0 EL1 EL2 EL3 Normal World Secure World Boot Firmware Boot Firmware Runtime EL3 Firmware (Secure Monitor) EL1 / EL0 Execution Secure EL1 / EL0 Execution EL2 Execution Key EL3 Execution UEFI Guest Linux Kernel B SMC Secure EL0 Secure EL1  SMC Calling Convention Interface (ARM DEN 0028A) is gateway to:  Runtime EL3 Firmware  Trusted OS / TEE services  Power State Coordination Interface (PSCI) (ARM DEN 0022B.b)  Transported by SMC calls  Also see ARMv8-A Architecture Manual (AR150-DA-70000)
  • 4. 4 Guest A App 1 Guest A App 2 Guest Linux Kernel A Guest B App 1 Guest B App 2 Trusted App 1 Trusted App 2 Trusted OS Hypervisor EL0 EL1 EL2 EL3 Normal World Secure World Boot Firmware Boot Firmware Runtime EL3 Firmware (Secure Monitor) EL1 / EL0 Execution Secure EL1 / EL0 Execution EL2 Execution Key EL3 Execution UEFI Guest Linux Kernel B SMC Secure EL0 Secure EL1 Example System Architecture  SMC Calling Convention Interface (ARM DEN 0028A) is gateway to:  Runtime EL3 Firmware  Trusted OS / TEE services  Power State Coordination Interface (PSCI) (ARM DEN 0022B.b)  Transported by SMC calls  Also see ARMv8-A Architecture Manual (AR150-DA-70000) ARM Trusted Firmware
  • 5. 5 ARM Trusted Firmware Architecture EL3 Firmware - BL31 (Secure Monitor) SMC Interface Service Router Other EL3 Interfaces Interrupt Handler World Switcher PSCI Pwr Ctrl Driver EL3 Arch Context Save/Restore Normal World Secure World Interface Usage External Interface EL1 Execution Secure EL1 Execution EL2 Execution KeyGlossary BL - Boot Loader EDK2 - EFI Development Kit 2 EL - Exception Level NV - Non-Volatile PSCI - Power State Control Interface SMC - Secure Monitor Call UEFI - Unified Enhanced Firmware Interface EL3 Execution Potential Interface UEFI - BL33 UEFI Secure Boot EDK2 Core I/O Drivers Boot ROM - BL1 Trusted Board Boot 1 Trusted Boot Firmware - BL2 Trusted Board Boot 2 Cold/Warm Boot Detection NV Storage Driver Boot Time Arch + Platform Init Temp SMC Handler Boot Time Arch + Platform Init Test Secure EL1 Payload - BL32 PSCI Test Service Router Other Test S-EL1 Arch Context Save/Restore Interrupt Handler Runtime Arch + Platform Init Test Suite – BL33_ALT PSCI Tests EL1 Arch Context Save/Restore EL2 Arch Context Save/Restore Other Tests Interrupt Handler Runtime Arch + Platform InitException Trapper
  • 6. 6 EL3 Firmware - BL31 (Secure Monitor) SMC Interface Service Router Other EL3 Interfaces Interrupt Handler World Switcher PSCI Pwr Ctrl Driver EL3 Arch Context Save/Restore Normal World Trusted World Interface Usage External Interface EL1 Execution Secure EL1 Execution EL2 Execution KeyGlossary BL - Boot Loader EDK2 - EFI Development Kit 2 EL - Exception Level NV - Non-Volatile PSCI - Power State Control Interface SMC - Secure Monitor Call UEFI - Unified Enhanced Firmware Interface EL3 Execution Potential Interface UEFI - BL33 UEFI Secure Boot EDK2 Core I/O Drivers Boot ROM - BL1 Trusted Board Boot 1 Trusted Boot Firmware - BL2 Trusted Board Boot 2 Cold/Warm Boot Detection NV Storage Driver Boot Time Arch + Platform Init Temp SMC Handler Boot Time Arch + Platform Init Test Secure EL1 Payload - BL32 PSCI Test Service Router Other Test S-EL1 Arch Context Save/Restore Interrupt Handler Runtime Arch + Platform Init Test Suite – BL33_ALT PSCI Tests EL1 Arch Context Save/Restore EL2 Arch Context Save/Restore Other Tests Interrupt Handler Runtime Arch + Platform InitException Trapper ARM Trusted Firmware Architecture Not Available Yet Partially Available
  • 7. 7 EL3 Firmware - BL31 (Secure Monitor) SMC Interface Service Router Other EL3 Interfaces Interrupt Handler World Switcher PSCI Pwr Ctrl Driver EL3 Arch Context Save/Restore Normal World Trusted World Interface Usage External Interface EL1 Execution Secure EL1 Execution EL2 Execution KeyGlossary BL - Boot Loader EDK2 - EFI Development Kit 2 EL - Exception Level NV - Non-Volatile PSCI - Power State Control Interface SMC - Secure Monitor Call UEFI - Unified Enhanced Firmware Interface EL3 Execution Potential Interface UEFI - BL33 UEFI Secure Boot EDK2 Core I/O Drivers Boot ROM - BL1 Trusted Board Boot 1 Trusted Boot Firmware - BL2 Trusted Board Boot 2 Cold/Warm Boot Detection NV Storage Driver Boot Time Arch + Platform Init Temp SMC Handler Boot Time Arch + Platform Init Test Secure EL1 Payload - BL32 PSCI Test Service Router Other Test S-EL1 Arch Context Save/Restore Interrupt Handler Runtime Arch + Platform Init Test Suite – BL33_ALT PSCI Tests EL1 Arch Context Save/Restore EL2 Arch Context Save/Restore Other Tests Interrupt Handler Runtime Arch + Platform InitException Trapper ARM Trusted Firmware Boot Flow BL33 BL33 (Alternative) BL32 BL2 BL1 BL31 RESET 2nd level Boot Loader (BL2) loads all 3rd level images 1st level Boot Loader (BL1) loads 2nd level image To Hypervisor / Linux Kernel
  • 8. 8 Current Memory Usage on FVP  Secure ROM  BL1 code  Semi-hosting  BL1, BL2, BL31 and Linux kernel code.  NOR  UEFI code  Virtio Block  Linux file system  Intend to use this for BL images instead of semi- hosting Storage
  • 9. 9 Current Memory Usage on FVP  Secure ROM  BL1 code  Semi-hosting  BL1, BL2, BL31 and Linux kernel code.  NOR  UEFI code  Virtio Block  Linux file system  Intend to use this for BL images instead of semi- hosting  Secure ROM  BL1 code  Secure SRAM  BL1 data (bottom)  BL2 code & data (top)  BL31 code & data (middle, above BL1)  Secure DRAM  Hand-off structures between BL images  When secure_memory parameter is defined, will need to program TZC to access DRAM  NOR  Early UEFI code  DRAM  UEFI code and data  Linux code and data Storage Execution
  • 10. 10 Code Organisation  arch - Architecture specific code (just AArch64 for now)  plat - Platform specific code (i.e. porting layer)  bl<X> - BL specific code  common - architecture/platform neutral code used by all BLs  lib - library functionality common to all other code  drivers - e.g. CCI, UART, FVP power controller  include  docs  fdts - not necessarily the final home for these!  Conforms to Linux coding standard
  • 12. 12 CPU States  Running. CPU is executing normally  Idle. States where OS is aware of CPU being idle  Execution resumes if new OS thread needs scheduling or interrupt is received  Idle Standby. Typically entered via WFI/WFE and exited via wake-up event  All caches and memories required for execution remain powered on and coherent  All CPU state (context) is preserved  Execution resumes from line after WFI/WFE  Idle Retention. Similar to standby except cannot access debug registers  Idle Power Down. CPU is powered off  CPU state at each EL must be saved  Execution starts at the reset vector, after which CPU state at each EL must be restored  Local caches may be off  Off. States where the OS is not using the CPU for scheduling  Execution starts from the reset vector and no CPU state is restored  Enabling a CPU in this state requires a Hotplug
  • 15. 15 ./bl1/aarch64/bl1_entrypoint.S If PSCI provided an entrypoint, then jump to BL31 (either hotplug or resume from idle)
  • 16. 16 ./bl1/aarch64/bl1_entrypoint.S If there's no entrypoint and this is the primary CPU, then continue with cold boot
  • 17. 17 ./bl1/aarch64/bl1_entrypoint.S Otherwise put the secondary CPU in a safe state (e.g. On FVP, power off CPU)
  • 19. 19 ./plat/fvp/aarch64/bl1_plat_helpers.S Initialize the secure memory used by BL1 (On FVP, just zero out the entrypoint mailboxes)
  • 20. 20 ./plat/fvp/aarch64/bl1_plat_helpers.S Create a small stack in "always uncached" memory to allow "C" code execution as soon as possible
  • 21. 21 ./plat/fvp/aarch64/bl1_plat_helpers.S "early" means "before MMU is enabled" In this case, just calculate extents of memory
  • 22. 22 ./plat/fvp/aarch64/bl1_plat_helpers.S Now create simple page tables and enable MMU / caches
  • 26. 26 ./bl1/bl1_main.c Do remaining architectural / platform setup now we are executing in normal memory with MMU / caches enabled E.g. control registers, generic timer, CCI snoops, console, ...
  • 27. 27 ./bl1/bl1_main.c Calculate where to load BL2, load it, then run it
  • 28. 28 BL2 and BL31  Entrypoints are similar to BL1's platform_cold_boot_init()  Create a small stack in coherent / always uncached memory)  Early platform setup (e.g. unpack image hand-off information)  Platform-specific architectural setup (e.g. enable MMU / caches)  Create a stack in normal memory  Branch to main "C" function for remaining platform / arch setup  BL2 loads BL3<X> images similarly to how BL1 loads BL2  BL2 passes information about BL33 (e.g. UEFI) to BL31, before running BL31  Means that BL31 can jump to BL33 without going back to BL2  Has to go via BL1 SMC handler to jump from Secure EL1 to EL3  See SynchronousExceptionA64 in ./bl1/aarch64/early_exceptions.S
  • 29. 29 BL31 Initialization  Can override any BL1 initialization  Reinitializes exception vectors, MMU, control registers, etc ...  Installs runtime SMC handler  Initializes platform for normal world software  Initializes GIC, runtime services (e.g. PSCI)  Returns from exception into normal world boot loader, BL33 (e.g UEFI)
  • 31. 31 PSCI Status  Work in progress. Key functions for boot and hotplug are functional. Idle is next on the radar PSCI Function Implementation Status PSCI_VERSION OK CPU_ON OK CPU_SUSPEND NOK (code present not ready) CPU_OFF OK AFFINTY_INFO OK MIGRATE Not present MIGRATE_INFO_TYPE Not present SYSTEM_OFF Not present SYSTEM_RESET Not present
  • 32. 32 PSCI Topology  PSCI needs to build a map of system topology  How many clusters, cores per cluster etc  Used for last man tracking  Topology information is provided by the platform using the following functions: int plat_get_max_afflvl() int plat_get_aff_count(unsigned int aff_lvl, unsigned long mpidr) unsigned int plat_get_aff_state(unsigned int aff_lvl, unsigned long mpidr)
  • 33. 33 PSCI Topology plat_get_max_afflvl() Returns highest affinity level implemented by the platform e.g. For FVP models: int plat_get_max_afflvl() { return MPIDR_AFFLVL1; }
  • 34. 34 PSCI Topology plat_get_aff_count(aff_lvl, mpidr) Given an MPIDR and a level of affinity return how many instances are implemented at that affinity level For example consider 2x3 system: 2 clusters, 2 cores in cluster 0, and 3 in cluster 1 mpidr aff_lvl Return Value 0000 (Cluster 0, Core 0) 0001 (Cluster 0, Core 1) 0 2 (two affinity level 0 instances, or cores, in cluster 0) 1 2 (There are two affinity level 1 instances or clusters in the system) 0100 (Cluster 1, Core 0) 0101 (Cluster 1, Core 1) 0 3 (three affinity level 0 instances, or cores, in cluster 1) 1 2 (There are two affinity level 1 instances or clusters in the system)
  • 35. 35 PSCI Topology plat_get_aff_state(aff_lvl, mpidr)  Returns whether an affinity instance is present or absent  You can use it to deal with hierarchies that are asymmetric  For example a cluster and a single core sharing an interconnect  Saves on having to take locks for affinity levels that don’t exist
  • 36. 36 bl31_entrypoint bl31_main bl31_platform_setup plat_setup_topology runtime_svc_init psci_setup FVP topology  FVP model sets up topology information as part of cold boot path (called from primary CPU)  plat_setup_topology() sets up necessary data to allow the following to work  plat_get_max_aff_lvl(), plat_get_aff_count(), plat_get_aff_state()  BL31 then moves on to set up PSCI Bl31/AArch64/bl31_entrypoint.S Bl31/bl31_main.c Plat/fvp/bl31_plat_setup.c Plat/fvp/Fvp_topology.c
  • 37. 37 PSCI Deep Dive  After cold boot path calls plat_setup_topology(), it calls psci_setup()  This functions create a topology map for the system based on the platform specific functions  Map is an array of aff_map_node pointers typedef struct { unsigned long mpidr; //mpidr of node unsigned char state; //[present||absent]|[PSCI state] char level; //aff level unsigned int data; //cookie (holds index into //non-secure data for CPU_ON/CPU_SUSPEND bakery_lock lock; //Lock for node } aff_map_node; (bakery_lock will be replaced with a more abstract lock API) See psci_setup.c/psci_common.c/psci_private.h
  • 38. 38 PSCI Topology  psci_aff_map holds topology tree (array of aff_map_node (s))  Held in device ordered memory  Array is populated in a breadth first way Aff 3 entities Aff 2 entities Aff 1 entities Aff 0 entities  e.g. for a 2x2 system Cluster 0 Cluster 1 CPU 0.0 CPU 0.1 CPU 1.0 CPU 1.1  Additional affinity information arrays: psci_aff_limits : indices to start and end of each affinty level in topology tree psci_ns_entry_info: array of ns_entry_info. Structure to hold entry information into EL[2|1] for CPU_ON/CPU_SUSPEND psci_secure_context: array of secure_context. Structure to hold secure context information that needs saving when powering down
  • 39. 39 PSCI Topology  psci_setup_up completes by:  Initialising the state of the primary CPU (and containing higher affinity levels, e.g cluster) to PSCI_STATE_ON  All others default to PSCI_STATE_OFF  Calls into platform to set up platform specific PSCI operations platfvpFvp_pm.c int platform_setup_pm(plat_pm_ops **plat_ops) { *plat_ops = &fvp_plat_pm_ops; return 0; } static plat_pm_ops fvp_plat_pm_ops = { 0, // standby (not used in FVP) fvp_affinst_on, // cpu_on will come here fvp_affinst_off, // cpu_off will come here fvp_affinst_suspend, // cpu_suspend fvp_affinst_on_finish, // called on wake up path of cpu being turned on fvp_affinst_suspend_finish, // called on wake up path of cpu waking from suspend };
  • 40. 40 __psci_cpu_off common/psci/psci_entry.S __psci_cpu_off: func_prologue sub sp, sp, #0x10 stp x19, x20, [sp, #0] mov x19, sp bl read_mpidr bl platform_set_coherent_stack << Switch stack bl psci_cpu_off mov x1, #PSCI_E_SUCCESS cmp x0, x1 b.eq final_wfi mov sp, x19 ldp x19, x20, [sp,#0] add sp, sp, #0x10 func_epilogue ret
  • 41. 41 __psci_cpu_off common/psci/psci_entry.S __psci_cpu_off: func_prologue sub sp, sp, #0x10 stp x19, x20, [sp, #0] mov x19, sp bl read_mpidr bl platform_set_coherent_stack bl psci_cpu_off << do work of switching off mov x1, #PSCI_E_SUCCESS cmp x0, x1 b.eq final_wfi mov sp, x19 ldp x19, x20, [sp,#0] add sp, sp, #0x10 func_epilogue ret
  • 42. 42 psci_cpu_off common/psci/psci_main.c int psci_cpu_off(void) { … int target_afflvl = get_max_afflvl(); mpidr = read_mpidr(); /* * Traverse from the highest to the lowest affinity level. When the * lowest affinity level is hit, all the locks are acquired. State * management is done immediately followed by cpu, cluster ... * ..target_afflvl specific actions as this function unwinds back. */ rc = psci_afflvl_off(mpidr, target_afflvl, MPIDR_AFFLVL0); if (rc != PSCI_E_SUCCESS) { assert(rc == PSCI_E_DENIED); } return rc; }
  • 43. 43 Affinity Level 1/Cluster Affinity Level 0/CPU Affinity Level 1/ClusterTake Cluster Node Lock Not at AffinityLevel0 so recurse down a level Take CPU Node Lock cpustate = CPU_OFF if last man cluster state = OFF psci_afflvl0_off(…) //bl31/common/psci_afflvl_off.c flush PoU fvp_affinst_off(..,AffLevel0,…) //plat/fvp/Fvp_pm.c take core out of coherency prevent IRQ spurious wakeups power power controller Release CPU Node Lock psci_afflvl1_off(…) //bl31/common/psci_afflvl_off.c if cluster_state == OFF flush to PoC fvp_affinst_off(..,AffLevel1,…) //plat/fvp/Fvp_pm.c disable cci power power controller Release Cluster Node Lock psci_afflvl_off – FVP success
  • 44. 44 psci_afflvl_off common/psci/psci_aff_lvl_off.c int psci_afflvl_off(unsigned long mpidr,int cur_afflvl, int tgt_afflvl) { … bakery_lock_get(mpidr, &aff_node->lock); /* Keep the old state and the next one handy */ prev_state = psci_get_state(aff_node->state); next_state = PSCI_STATE_OFF; /* * We start from the highest affinity level * and work our way * downwards to the lowest i.e. MPIDR_AFFLVL0. */ if (aff_node->level == tgt_afflvl) { psci_change_state(mpidr, tgt_afflvl, get_max_afflvl(), next_state); } else { rc = psci_afflvl_off(mpidr, level - 1, tgt_afflvl); if (rc != PSCI_E_SUCCESS) { psci_set_state(aff_node->state, prev_state); goto exit; } } … rc = psci_afflvl_off_handlers[level](mpidr, aff_node); if (rc != PSCI_E_SUCCESS) { psci_set_state(aff_node->state, prev_state); goto exit; } … exit: bakery_lock_release(mpidr, &aff_node->lock); return rc; } When we get to CPU Level (Aff0) we set state. This also sets higher affinity level states if last man
  • 45. 45 psci_afflvl_off common/psci/psci_aff_lvl_off.c int psci_afflvl_off(unsigned long mpidr,int cur_afflvl, int tgt_afflvl) { … bakery_lock_get(mpidr, &aff_node->lock); /* Keep the old state and the next one handy */ prev_state = psci_get_state(aff_node->state); next_state = PSCI_STATE_OFF; /* * We start from the highest affinity level * and work our way * downwards to the lowest i.e. MPIDR_AFFLVL0. */ if (aff_node->level == tgt_afflvl) { psci_change_state(mpidr, tgt_afflvl, get_max_afflvl(), next_state); } else { rc = psci_afflvl_off(mpidr, level - 1, tgt_afflvl); if (rc != PSCI_E_SUCCESS) { psci_set_state(aff_node->state, prev_state); goto exit; } } … rc = psci_afflvl_off_handlers[level](mpidr, aff_node); if (rc != PSCI_E_SUCCESS) { psci_set_state(aff_node->state, prev_state); goto exit; } … exit: bakery_lock_release(mpidr, &aff_node->lock); return rc; } Handlers do actual powering down
  • 46. 46 __psci_cpu_off common/psci/psci_entry.S __psci_cpu_off: func_prologue sub sp, sp, #0x10 stp x19, x20, [sp, #0] mov x19, sp bl read_mpidr bl platform_set_coherent_stack bl psci_cpu_off mov x1, #PSCI_E_SUCCESS cmp x0, x1 b.eq final_wfi << all OK final WFI mov sp, x19 ldp x19, x20, [sp,#0] add sp, sp, #0x10 func_epilogue ret
  • 47. 47 __psci_cpu_off common/psci/psci_entry.S __psci_cpu_off: func_prologue sub sp, sp, #0x10 stp x19, x20, [sp, #0] mov x19, sp bl read_mpidr bl platform_set_coherent_stack bl psci_cpu_off mov x1, #PSCI_E_SUCCESS cmp x0, x1 b.eq final_wfi mov sp, x19 << else switch stack back and return ldp x19, x20, [sp,#0] add sp, sp, #0x10 func_epilogue ret
  • 48. 48 psci_cpu_on common/psci/psci_main.c int psci_cpu_on(unsigned long target_cpu, unsigned long entrypoint, unsigned long context_id) { int rc; unsigned int start_afflvl, target_afflvl; /* Determine if the cpu exists of not */ rc = psci_validate_mpidr(target_cpu, MPIDR_AFFLVL0); if (rc != PSCI_E_SUCCESS) { goto exit; } start_afflvl = get_max_afflvl(); target_afflvl = MPIDR_AFFLVL0; rc = psci_afflvl_on(target_cpu, entrypoint, context_id, start_afflvl, target_afflvl); exit: return rc; } Basic error checking Heavy lifting
  • 49. 49 Affinity Level 0/CPU Affinity Level 1/Cluster Affinity Level 0/CPU Affinity Level 1/Cluster Affinity Level 0/CPU Affinity Level 1/Cluster Affinity Level 0/CPU Affinity Level 1/ClusterTake Cluster Node Lock Take CPU Node Lock psci_afflvl1_on(…) //bl31/common/psci_afflvl_on.c fvp_affinst_on(..,AffLevel1,…) //plat/fvp/Fvp_pm.c basic validation psci_afflvl0_on(…) //bl31/common/psci_afflvl_on.c store ns_entry_point and context_id (passed from by OSPM) fvp_affinst_on(..,AffLevel0,…) //plat/fvp/Fvp_pm.c Wait for any pending off to complete (CPU that you are turning ON, could have been turning itself OFF) Set up a mailbox so booting core takes warm boot path program power controller Change CPU Node state to ON_PENDING Change Cluster Node state to ON_PENDING Release CPU Node Lock Release Cluster Node Lock psci_afflvl_on – FVP success
  • 50. 50 psci_afflvl_on common/psci/psci_aff_lvl_on.c int psci_afflvl_on(unsigned long target_cpu, unsigned long entrypoint, unsigned long context_id, int current_afflvl, int target_afflvl) { … for (level = current_afflvl; level >= target_afflvl; level--) { aff_node = psci_get_aff_map_node… if (aff_node) bakery_lock_get(mpidr, &aff_node->lock); } … for (level = current_afflvl; level >= target_afflvl; level--) { … psci_afflvl_on_handlers[level](target_cpu, aff_node, entrypoint, context_id); } /* * State management: Update the states … */ psci_change_state(target_cpu, target_afflvl, get_max_afflvl(), PSCI_STATE_ON_PENDING); exit: … for (level = target_afflvl; level <= current_afflvl; level++) { … bakery_lock_release(mpidr, &aff_node->lock); … } Unlock Call each level’s on handler Set Hierarchy to ON_PENDING Lock
  • 51. 51 fvp_affinst_on plat/fvp/Fvp_pm.c int fvp_affinst_on(unsigned long mpidr, unsigned long sec_entrypoint, unsigned long ns_entrypoint, unsigned int afflvl, unsigned int state) { … if (ns_entrypoint < DRAM_BASE) { rc = PSCI_E_INVALID_PARAMS; goto exit; } … if (afflvl != MPIDR_AFFLVL0) goto exit; … /* * Ensure that we do not cancel an inflight * power off request * for the target cpu. That would leave * it in a zombie wfi… */ do { psysr = fvp_pwrc_read_psysr(mpidr); } while (psysr & PSYSR_AFF_L0); linear_id = platform_get_core_pos(mpidr); fvp_mboxes = (mailbox *) (TZDRAM_BASE + MBOX_OFF); fvp_mboxes[linear_id].value = sec_entrypoint; flush_dcache_range( (unsigned long) &fvp_mboxes[linear_id], sizeof(unsigned long)); fvp_pwrc_write_pponr(mpidr); … Ensure entry point is valid Deal with potential race with CPU_OFF Set up warm boot Program power controller
  • 52. 52 ./bl1/aarch64/bl1_entrypoint.S If PSCI provided an entrypoint, then jump to BL31 (either hotplug or resume from idle)
  • 54. 54 Affinity Level 0/CPU Affinity Level 1/Cluster Affinity Level 0/CPU Affinity Level 1/Cluster Affinity Level 1/Cluster Affinity Level 0/CPU psci_afflvl_power_on_finish Take CPU Node Lock //bl31/common/psci_common.c psci_afflvl0_on_finish(…) //bl31/common/psci_afflvl_on.c fvp_affinst_on_finish(..,AffLevel0,…) //plat/fvp/Fvp_pm.c turn on intra cluster coherency zero out mailbox, enable GIC, enable access to system counter install exception handlers,enable mmu and caching, EL3 setup psci_get_ns_entry_info() //bl31/common/psci_common.c set up return non-secure Exception Level Take Cluster Node Lock psci_afflvl1_on_finish(…) //bl31/common/psci_afflvl_on.c fvp_affinst_on_finish(..,AffLevel1,…) //plat/fvp/Fvp_pm.c enable CCI update CPU Node State to ON update cluster Node State to ON Release Cluster Node Lock Release CPU Node Lock
  • 55. 55 Further reading…  GitHub https://github.com/ARM-software/arm-trusted-firmware  Usage Guide https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/user- guide.md  Porting Guide https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/porting- guide.md  SMC Calling Convention http://infocenter.arm.com/help/topic/com.arm.doc.den0028a/index.html  PSCI spec http://infocenter.arm.com/help/topic/com.arm.doc.den0022b/index.html  ARMv8 ARM http://infocenter.arm.com/help/topic/com.arm.doc.ddi0487a.a_errata1/index.html