Presented by Valerio Schiavoni @vschiavoni at DAIS 19 The TrustZone technology, available in the vast majority of recent Arm processors, allows the execution of code inside a so-called secure world. It effectively provides hardware-isolated areas of the processor for sensitive data and code, i.e., a trusted execution environment (TEE). The Op-Tee framework provides a collection of toolchain, opensource libraries and secure kernel specifically geared to develop applications for TrustZone. This paper presents an in-depth performance- and energy-wise study of TrustZone using the Op-Tee framework, including secure storage and the cost of switching between secure and unsecure worlds, using emulated and hardware measurements.

1. On the Performance of ARM TrustZone
Julien Amacher, Valerio Schiavoni
University of Neuchâtel, Switzerland
19th International Conference on Distributed Applications and Interoperable Systems
DAIS 2019 - DTU, Denmark
20/06/2019
The research leading to these results has received funding from the European Union’s Horizon 2020 research and
innovation programme under the LEGaTO Project (legato-project.eu), grant agreement No 780681.
(Practical Experience Report)

2. On the Performance of ARM TrustZone
Julien Amacher, Valerio Schiavoni
University of Neuchâtel, Switzerland
19th International Conference on Distributed Applications and Interoperable Systems
DAIS 2019 - DTU, Denmark
20/06/2019
The research leading to these results has received funding from the European Union’s Horizon 2020 research and
innovation programme under the LEGaTO Project (legato-project.eu), grant agreement No 780681.
(Practical Experience Report)
Expect more plots than usual…at least in the paper

3. On the Performance of ARM TrustZone
Julien Amacher, Valerio Schiavoni
University of Neuchâtel, Switzerland
19th International Conference on Distributed Applications and Interoperable Systems
DAIS 2019 - DTU, Denmark
20/06/2019
The research leading to these results has received funding from the European Union’s Horizon 2020 research and
innovation programme under the LEGaTO Project (legato-project.eu), grant agreement No 780681.
as implemented by Broadcom BCM2837

4. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Motivation
• Powerful processing devices at the edge nowadays pervasive
• Off-loading at the edge, rather than on an untrusted cloud
• Technologies to provide privacy and secure processing at the
edge exist since many years
• In this talk, we look at ARM TrustZone
0
5
10
15
20
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018
ARMv6ZTrustZone
ARMv7TrustZone
ARMv8−ATZ/EL3
ARMv8−Mw/TZ
Unitssold[bn]
Year
4

5. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
TrustZone in one slide
• TrustZone is a Trusted Execution Environment (TEE) for ARM processors
• Intel SGX (2015), AMD SEV (2017-ish), RISC-V KeyStone (2018)
• Runtime split in two: untrusted area (the Rich Execution Environment or
REE) and the trusted area, the Trusted Execution Environment
5
❽
Kernel mode
User mode
Rich application Trusted application (TA)
Secure
kernel
tee-supplicant
GlobalPlatform
TEE Client API
OP-TEE
driver
Rich kernel
GlobalPlatform
TEE Internal API
Storage
Shared
memory
Secure Monitor
& TEE core
Internal TEE
utility functions
OP-TEE OS
OP-TEE Client
➊
➋ ➌➍ ➎ ➏
➐
REE (non-secure) TEE (secure)
User mode
Kernel mode
❾
➓
REE TEE

6. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
TrustZone in 2 slides…
• TEE and REE share the same core
• TEE has unrestricted access to
memory regions, hardware and devices
• Physical memory dedicated to the REE
• The Translation Lookaside Buffer (TLB) is
TEE-aware, secure and unsecure
descriptors stored side-by-side
• TA (trusted application) in theory is
allowed 3-5 Mb, in practice <= 1Mb
6

7. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
TrustZone in 3 slides!
• World-switching
• Move from TEE to REE and back via a special hardware
instruction, SMC (Secure Monitor Call)
• Secure storage leveraging a transparent encryption layer from the TEE
• TAs can store encrypted objects on disk
• Signed against anti-tampering
• Secure (fast) interrupts
• But there is even more: key management, protection against
attacks (stack canaries), covert data channels, …
7
© Disney

8. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Problem
• Difficult (if not impossible) to find public studies (scientific papers,
books, or even blog entries) regarding the performance of TrustZone
• By performance we mean:
• Throughput of operations, loading time of trusted applications, etc.
• But also: energy-related costs
• Without these informations, it’s difficult to predict the expected overhead
that programs encounter when inside a TEE
• Essentially, we were missing a baseline
8

9. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Contributions
1. In-depth study of performance and energy of TrustZone
features
2. Insights on the programmability of trusted applications
3. Extensions of the kernel running inside the TEE to securely
extract measurements, as well as methodology to implement
new syscalls
4. Results on emulation and real hardware, i.e., Raspberry Pi
9
Not in this talkIn this talk In the paper
or in 60min

10. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
The OP-TEE runtime
• Native support for TrustZone
• Open-source, sponsored by Linaro
• Supports many devices, including cheap
Raspberry PI 3B
• Attempted support newer RaPI 3B+ failed
• Easy to use, and compliant with existing
standards
• SDK with clearly defined APIs
10
RaPi 3B, <=50 $

11. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Experimental Methodology
• Applications implemented in C/C++
• Initial plan to port stress-ng failed
• Many syscalls not available inside TEE
• What is the one thing that is always needed in benchmarks ?
11
Start/stop
recording
Export CSV
Parse CSV
Markers,
duration
Host computerKM001 official application Raspberry Pi
Benchmark
applicationsBenchmark
applicationsBenchmark
applications
Power supply
Execute
benchmark
JTAG
Power
consumption
KM001
Monitoring
program
Time inside TEE
problematic

12. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Time inside TEE
• clock_gettime() supports 4 different types:
• CLOCK_REALTIME
• CLOCK_MONOTONIC
• CLOCK_PROCESS_CPUTIME_ID
• CLOCK_THREAD_CPUTIME_ID
• To read the monitonic counter from the TA, we extended the secure
kernel with new system calls
12
https://github.com/vschiavoni/on-the-performance-of-arm-trustzone
and temperature

13. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
After all, it’s about
Performance
• We benchmarked several aspects:
• Power-consumption, idle and burn
and for 3 different CPU governors
• Load and unload of TAs
• Context (World) switching
• Operations on volatile memory
• Operations on persistent (secure)
storage
• CPU benchmarking
• Thermal
benchmarking

14. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
After all, it’s about
Performance
• We benchmarked several aspects:
• Power-consumption, idle and burn
and for 3 different CPU governors
• Load and unload of TAs
• Context (World) switching
• Operations on volatile memory
• Operations on persistent (secure)
storage
• CPU
benchmarking
• Thermal
benchmarking
In the paper

15. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Load and Unload TAs
Basic TA operations
0
500
1000
1500
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Executiontime[ms]
Large TA (517kB)
Qemu on ESXi
rpi3b ondemand
rpi3b performance
rpi3b powersave
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Small TA (102kB)
4 platforms
(1 virt+3 hw)

16. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Load and Unload TAs
Basic TA operations
0
500
1000
1500
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Executiontime[ms]
Large TA (517kB)
Qemu on ESXi
rpi3b ondemand
rpi3b performance
rpi3b powersave
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Small TA (102kB)
Bigger than
L2 cache

17. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Load and Unload TAs
Basic TA operations
0
500
1000
1500
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Executiontime[ms]
Large TA (517kB)
Qemu on ESXi
rpi3b ondemand
rpi3b performance
rpi3b powersave
Em
pty
function
TA
loadFirstTA
load
TA
unload
FirstTA
unload
Small TA (102kB)
avg/stdev
over 10k run

18. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Context SwitchingContext switching
0
50
100
150
200
250
300
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Executiontime[µs]
REE to TEE TEE to REE
0
10
20
30
40
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Instrumentationdelay[µs]
rpi3b ondemand rpi3b performance rpi3b powersave
REE TEE
9
10
11
12
13
Calling empty
TA function
Energy[nWh]
• To account for caching effects, we measure the first call (left)
and the following calls (right)

19. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Context SwitchingContext switching
0
50
100
150
200
250
300
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Executiontime[µs]
REE to TEE TEE to REE
0
10
20
30
40
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Instrumentationdelay[µs]
rpi3b ondemand rpi3b performance rpi3b powersave
REE TEE
9
10
11
12
13
Calling empty
TA function
Energy[nWh]
• To account for caching effects, we measure the first call (left-
side whiskers) and the following calls (right-side whiskers)
• Both directions (REE to TEE, and TEE to REE)

20. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Context SwitchingContext switching
0
50
100
150
200
250
300
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Executiontime[µs]
REE to TEE TEE to REE
0
10
20
30
40
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Instrumentationdelay[µs]
rpi3b ondemand rpi3b performance rpi3b powersave
REE TEE
9
10
11
12
13
Calling empty
TA function
Energy[nWh]
• Instrumentation delay: difference between two consecutive
calls to the time measurement function
• Inside the TEE there are more context switches

21. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Context SwitchingContext switching
0
50
100
150
200
250
300
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Executiontime[µs]
REE to TEE TEE to REE
0
10
20
30
40
1
stcall
Follow
ing
calls
1
stcall
Follow
ing
calls
Instrumentationdelay[µs]
rpi3b ondemand rpi3b performance rpi3b powersave
REE TEE
9
10
11
12
13
Calling empty
TA function
Energy[nWh]
• Interestingly, the ondemand governor is the most energy-eager
• Expensive to continuously change CPU frequency!

22. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Thermal Benchmarking
• We measure the thermal envelope of the SoC
• Two metrics: software measurements (from our syscalls)…
CPU temperature during prime benchmark
20
30
40
50
60
70
80
90
0 4 8 12 16
Temperature[°C]
ondemand
External sensor
Thermals API
External sensor (w/ fan)
Thermals API (w/ fan)
#1
#2
Time [min]
0 4 8 12 16
powersave
#3
#4
0 4 8 12 16
Max
Ambient
performance
#5
#6

23. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
Thermal Benchmarking
ondemand
MAX 66.7 °C 90.5°C
18.2°C
90.5°C
18.2°C
MAX 52.4 °C
MAX 87 °C 90.5°C
19.9°C
MAX 64.3 °C 90.5°C
19.6°C
MAX 67.5 °C 90.5°C
18.2°C
MAX 90.3 °C 90.5°C
19.1°C
powersave performance
fan on fan on fan on
fan off fan off fan off
Broadcom BCM2837

24. OnThe Performance of ARMTrustZone -V. Schiavoni - DAIS’19
What’s left?
• The mentioned experimental
results, plus:
• Lessons learned
• Considerations on the
memory limitations
• Standard compliancy
• OP-TEE developer friendliness
• How to extend the secure kernel