1. [ARM] OP-TEE Mediator in Xen
Volodymyr Babchuk, Embedded Engineer
at EPAM Systems

2. Agenda
•What is Trusted Execution Environment (TEE)
•OP-TEE
•Implementation details of virtualization in OP-TEE
•Why we need mediator
•Implementation of OP-TEE mediator in XEN
•Current status (Jul 2019)

3. Trusted Execution Environment
• Basically, this is minimalistic secure OS designed to run small
Trusted Applications (TAs)
• Interfaces are standardized by GlobalPlatform
(https://globalplatform.org/)
• Indented to run apps like embedded SIM, contactless payments,
cryptographic services, DRM and so on

4. TEE and REE

5. OP-TEE
• Open source TEE implementation for ARMv7/ARMv8
• Initially developed by ST-Ericsson as closed source TEE
• Was open sourced in 2014
• Now is being maintained by Linaro’s Security Working Group
• Upstream branch supports more than 20 platforms (including
Renesas RCAR Gen3)
• Implements GP specifications for TEE and passes GP Test Suite
• Now has virtualization support

6. Virtualization in OP-TEE
• OP-TEE was designed without virtualization support in mind
• State is stored in global variables across object files and in heap
• OP-TEE code can be divided into two parts:
• “nexus” - kernel - drivers, low-level code, MMU management and so on
• “tee” - provides value - TA handling, implementation of GP APIs and other
“business logic”
• We need one “nexus” instance and multiple “tee” instances - one
instance per VM/guest.

7. Virtualization in OP-TEE - obvious way
1. Introduce struct vm_context (like struct domain in XEN)
2. Move all global state to that struct
3. Add struct vm_context *vmctx parameter to each function
or introduce current variable.
+ Explicit state
+ Easy to understand
- Requires changes to all “tee” parts of OP-TEE
- Provides poor isolation
- Tricky situations - e.g. VM dies during the call

8. Virtualization in OP-TEE - our way
Banked TEE state .data, .bss and
.heap sections.
“Nexus” in own .nex_data, .nex_bss,
.nex_heap sections
Nexus maps needed sections when it
receives call from the given VM

9. Virtualization in OP-TEE - our way
+ VM states are isolated - code can access only one VM data in the
same time
+ No changes to the TEE part
+ Minor changes to Nexus - MMU, entry/exit code, move variables
to .nex_* sections
- Implicit, unorthodox approach - harder to understand
- Unclear how to share hardware resources, so no support for HW
crypto accelerators, external secure storage, etc right now

10. TEE mediator
1. Inform TEE about guest lifecycle
2. Tell TEE which guest is calling it
3. Decide if guest has access to TEE at all
4. Translate memory address (IPA->PA)
5. Track shared memory
6. Tell guest about TEE
7. (Optionally) do this in transparent manner

11. tee mediator framework in hypervisor
framework is located at arch/arm/tee/
Calls:
- probe for TEE
- domain created
- domain is being destroyed
- handle HVC/SMC from guest
- mangle DTB for Dom0 (planned)

12. Mediator in XEN

13. OP-TEE mediator
- First (and only one for now) mediator in XEN
- Fully functional - OP-TEE passes all tests:
24045 subtests of which 0 failed
93 test cases of which 0 failed
0 test cases were skipped
TEE test application done!
- Still in EXPERIMENTAL state

14. OP-TEE mediator
Basic flow:
1. Receive HVC (or SMC) from guest
2. Check command ID
3. Alter arguments
4. Call OP-TEE (if needed)
5. Check return values
6. Return result back to guest

15. OP-TEE mediator - challenges 1
Two types of calls (as described in SMC Calling Convention):
- Fast call: one SMC from guest to OP-TEE
- Standard call: multiple SMCs per call. We want interrupt call to:
- Allow NW to handle interrupt request
- Because we are blocked on mutex
- To ask NW for some service: read/write file, send/receive network packet,
get time, allocate shared memory, read TA from disk
This leads to quite complex handling of standard calls in the
mediator, because we need to consider all possible states of
OP-TEE and guest.

16. OP-TEE mediator: fast vs std call

17. OP-TEE mediator - challenges 2
GlobalPlatform API provides a call to share memory between
application in EL0 and TA in S-EL0.
Guest sends scatter-gather list of IPAs to OP-TEE. We need to
translate each IPA to PA, so OP-TEE can map shared memory
region into TA address space.
Mediator needs to pin shared pages and maintain list of them.

18. Current state (JUL 2019)
- Virtualization support merged in OP-TEE 3.5
- Hypervisor part is merged to XEN (as experimental/preview
feature)
- Patches for toolstack are still under review. So right now only
Dom0 can talk to OP-TEE

19. Future Plans - OP-TEE
Design sharing of hardware resources:
- Some generic approach to share HW
- Support for crypto HW
- Secure storage on external flash:
- needs extension in Normal World <-> Secure World APIs to manage
partitions
- we need some way to authenticate guests before authorizing them to
access secure storage

20. Future Plans - XEN
Maturize OP-TEE mediator:
- Limit number of P2M lookups per call
- Limit XEN’s memory usage per guest
- Handle machine DTB
- Improve handling of some tricky state during RPC handling
New features:
- Add support for secure storage partition API (when it will be
implemented in OP-TEE)
- Add some persistent guest UUIDs (?)