SGX Trusted Execution Environment

SGX Trusted Execution Environment
Linux Kernel Meetup
Tel-Aviv, May 10, 2018
Ofir Weisse

Linux Kernel Meetup, Tel-Aviv, May 10, 2018
Cloud Computing Attack Surface
Service Hosting
2
Medical Records
Intellectual Property
Private Data

Cloud Computing Attack Surface
 To lower costs - computation and storage
are moved to third party machines
 This implies trust
Cloud provider
employees
3
OS
Virtualization
Software
SMM code
(firmware)
Hardware
The attack surface is large

SGX Secure Execution
 What is the impact on overall application’s performance?
 What creates the bottlenecks? Can we alleviate them?
 How can the kernel attack SGX?
 How can we defend against a malicious kernel? 4
Authenticated code
Malicious environment
Is it practical?
No SGX With SGX
Throughput degradation

Outline
 Part 1 – Performance Optimization
Intel SGX background
Measuring SGX performance bottlenecks
Improving SGX performance with HotCalls
 Part 2 – Attacks on SGX
5

SGX in a nutshell
User Space ddddddd d
OS Kernel
VMM
SMM
RAM HW CPU
Enclave
6
Attestation
Remote
Client

SGX – Memory Organization
Physical Memory
Enclave Page Cache (EPC)
EPC metadata
Encrypted by
Memory Encryption Engine
(MEE)
7

SGX – Memory Organization
No roll back 8
Physical Memory
EPC metadata
No roll-back

SGX Encrypted Memory Management
9
Virtual Address space (>4GB): code data
Physical memory:

SGX Instructions
Supervisor Instructions (Ring 0)
 ECREATE
 EADD – copy to EPC
 EEXTEND – add to SHA256
 EINIT
 EDBGRD
 EDBGWR
 EINIT
 EWB – evict from EPC
 ELD – load to EPC
 ETRACK
10
User Instructions (Ring 3)
 EENTER
 EEXIT
 ERESUME
 EGETKEY
 EREPORT

SGX Encrypted Memory Management
11
Virtual Address space (>4GB): code data
Physical memory:
EADD

Application memory address spacePlaintext Shared Memory
SGX - Secure Enclave Life-cycle
12
Enclave –
Trusted Code
Application –
Untrusted Code
Encrypted Memory
ocall
ecall
• Can access all memory
• No access to system
calls
• Can call system API
functions
(send, fread, etc.)
External
Verifier
SGX operations may become a bottleneck

Outline
13

 Accessing encrypted memory
Read
Write
 Control transfers
Ecalls (EENTER+EEXIT)
Ocalls (EEXIT+ERESUME)
 SDK inefficiencies
What are the Potential Bottlenecks?
14

Cost of Accessing Encrypted Memory
15
102%
overhead
Write Latency Read Latency
6% overhead
(Cache-miss: 30%) (Cache-miss: 20%)
Encrypted memory is a potential bottleneck

Ecalls
SDK code
EENTER
EEXIT
Ocalls
SDK code
EEXIT
ERESUME
Cost of Secure Context Switch
16
 Gathering required
enclave information
 Defensive checks of
pointers
 Legal destination
 No overlaps

Cost of Secure Context Switch (CDF)
17
8,600 cycles = 2.16 𝝁𝒔𝒆𝒄
8,200 cycles = 2.06 𝝁𝒔𝒆𝒄
Ecalls
SDK code
EENTER
EEXIT
Ocalls
SDK code
EEXIT
ERESUME
8,600 cycles
= 2.16 𝝁𝒔𝒆𝒄
8,200 cycles
= 2.06 𝝁𝒔𝒆𝒄
Cycles (x1000)
14,100 cycles
= 3.52 𝝁𝒔𝒆𝒄
14,100 cycles
= 3.52 𝝁𝒔𝒆𝒄
100%
50%
0%
100%
50%
0%

Call Type Cycles
Linux System Call 150 x1 (OSDI 2010)
KVM Hypercall 1,300 x8 (ISCA 2016)
Call Type Cycles
Linux System Call 150 x1 (OSDI 2010)
KVM Hypercall 1,300 x8 (ISCA 2016)
SGX calls (warm cache) 8,600 x57
SGX calls (cold cache, median) 14,100 x94
SGX calls (cold cache top 5%) 16,000 x106
Context Switch in Perspective
18
#Calls per
second
Cycles overhead
@ 4GHz
10,000 2.15%
50,000 10.75%
100,000 21.5%
200,000 43%
Application
# Calls
/second
Core
spending
Memcached 200,000 43%
OpenVPN 275,000 57%
Lighttpd 270,000 56%
Real Applications

Outline
19

Fast EcallsFast OcallsHotCalls
HotCalls – New Calling Interface
EnclaveApplication
20
Properties:
 Not dependent on OS mutexes, semaphores or signals
 Maintains security properties of SGX
No context switch

Fast EcallsFast OcallsHotCalls
HotCalls – New Calling Interface
ResponderRequester
21
Shared Memory
void *dataSpinlock call_ID Go | DoneSpinlockSpinlock void *data call_ID Go | DoneSpinlock Go | Done
Shared Memory
void *dataSpinlock call_ID Go | DoneSpinlock void *data call_ID
Shared Memory
void *dataSpinlock call_ID Go | DoneSpinlock call_ID
Additional thread
Go | Donevoid *data
Shared memory
No context switch

HotCalls vs. SDK Calls (CDF)
23
8,200 cycles
14,100 cycles
600 cycles
1,300 cycles
580 cycles
1,300 cycles
Cycles (x1000)
100%
50%
0%
100%
50%
0%

OS CallsOS Calls
HotCalls in Practice
24
OS Calls
- Porting strategy similar to Haven & SCONE
- Developed an SGX porting framework to automate the process

Lost Cycles Estimation
25
Application
Frequent Calls
(Calls x1000 / second)
Total
Calls
Core
Time
Memcached
read(66.5), sendmsg(66.5)
RunEnclaveFucntion(66.5)
200K 42%
OpenVPN
poll(87), time(87), getpid(13.6),
write(30), recvfrom(30),
read(13.6) sendto(13.6)
275K 57%
Lighttpd
read(49),fcntl(25),
epoll_ctl(25), close(25),
setsockopt(25), __fxstat64(25)
inet_ntop(12),accept(12),
inet_addr(12),ioctl(12),
__open64_2(12), sendfile64(12)
shutdown(12),writev(12)
270K 56%
#Calls per
second
Core
overhead
10,000 2.15%
50,000 10.75%
100,000 21.5%
200,000 43%
Context switches consume up to 57% of the cycles

-64% -79% -77%x2.66 x2.8 x3.7
HotCalls in Action
26
317866 53.4
OpenVPN
(MB / sec)
Memcached
(K-ops / sec)
Lighttpd
(K-requests / sec)
66
309
12.1
185
823
44.8
Normalized Throughput
Improvement

Part 1 Conclusion
 Naively porting applications may derail
performance
 Memory access may be expensive
 Interaction with the OS may be costly
 Can optimize performance with HotCalls
 Request latency is reduced by up to 13X
 Throughput can be boosted to near-native
performance
27

Attacks on SGX
 Controlled Channel Attacks – with Page Faults
 Controlled Channel Attacks – with Page Table Side Channels
 Branch Shadowing
 Defense Mechanisms
28

Virtual to Physical Mapping 101
Virtual Memory: code data
29
Physical memory
The OS can induce a page fault on every memory access
Enclave Page Cache
(EPC)

Controlled Channel Attack
(The Original)
30Source: “Controlled Channel Attacks”, IEEE S&P 2015

(The Original)
31Image Source: “Controlled Channel Attacks”, IEEE S&P 2015

32Image Source: “Controlled Channel Attacks”, IEEE S&P 2015

33Source: “Telling Your Secrets Without Page Faults”, USENIX Security 2017

Without Page Faults
 Dirty bits in PTEs
 Cache side channels
34

Attacks on SGX
 Controlled Channel Attacks – with Page Faults
 Controlled Channel Attacks – with Page Table Side Channels
 Branch Shadowing
 Defense Mechanisms
35

Branch Shadowing Attacks
 Branch prediction and BTB
36
Source: “Inferring Fine-grained Control Flow Inside SGX Enclaves with Branch Shadowing”,
USENIX Security 2017

Last Branch Record (LBR)
 A trace of all recently taken branches and branch mispredictions
 Alas, LBR is disabled when SGX enclaves are executing
37

Branch Prediction and
Branch Target Buffer (BTB)
Current RIP [Bits 31:0] Taken/Not-taken Predicted Destination
0x7FADEA1050DE0000 Taken 0x7FADEA1050DE0300
0x7FADEA1050DE0100 Taken 0x7FADEA1050DE0200
0x7FADEA1050DE0200 Not Taken 0x7FADEA1050DE0100
0x7FADEA1050DE0300 Not Taken 0x7FADEA1050DE0500
38
 What happens if RIP=0x0000000050DE0000?

Branch Shadowing
39
Address Code
0x7FADEA1050DE0000 If( keyBit == 0 ) {
0x7FADEA1050DE0010 multiply(…);
0x7FADEA1050DE0018 ……….
0x7FADEA1050DE0020 } else if(…) {
0x7FADEA1050DE0028 add( ….. );
0x7FADEA1050DE0030 …….
0x7FADEA1050DE0038 }
Inside Enclave Branch Shadow
Address Code
0x0000000050DE0000 If( dummy != 5 ) {
0x0000000050DE0010 nop;
0x0000000050DE0018 ……….
0x0000000050DE0020 } else if(…) {
0x0000000050DE0028 nop;
0x0000000050DE0030 …….
0x0000000050DE0038 }

Single step(ish)
 How can we pause execution after every branch?
 Clock Interrupt causes Asynchronous Exit (AEX)
 Then single step branch shadow code
 Observe LBR
40

Branch Shadowing Results
 66% of the bits of 1024 RSA key were recovered using a
single run
 With 10 runs – all the bits were recovered
41

Defense Mechanisms
 T-SGX – using Transactional Memory Extensions (TSX)
 SGX-Shield – ASLR for SGX
 Racing in Hyperspace
42

TSX –
Transactional Synchronization eXtensions
 Computation performed in cache
 Complete rollback upon abort
 Faults are supressed
43
Transaction
XBEGIN
XEND
TSX Abort Transaction complete

 Page-fault -> TSX abort
T-SGX
44
XBEGIN
XEND
Image Source:
“Controlled Channel Attacks”, IEEE S&P 2015

T-SGX (2)
45
XBEGIN
XEND
 Timer Interrupt -> TSX abort
Image Source: “Inferring Fine-grained Control Flow Inside SGX Enclaves
with Branch Shadowing”, USENIX Security 2017

 Secure execution alone is only the first step for secure
systems
 Performance impact may be prohibitive but can be
optimized with HotCalls
 Including the OS/VMM in the threat model presents new
challenges
Conclusion
46
www.OfirWeisse.com
github.com/oweisse/hot-calls

SGX Trusted Execution Environment

More Related Content

What's hot

Similar to SGX Trusted Execution Environment

More from Kernel TLV

Recently uploaded

In this document

SGX Trusted Execution Environment