Takaaki Fukai, Satoru Takekoshi, Kohei Azuma, Takahiro Shinagawa, Kazuhiko Kato. In Proceedings of the 9th IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2017), Dec 2017. http://dx.doi.org/10.1109/CloudCom.2017.43

1. BMCArmor: A Hardware Protection Scheme
for Bare-Metal Clouds
Takaaki Fukai, Satoru Takekoshi (University of Tsukuba, Japan);
Kohei Azuma, Takahiro Shinagawa (The University of Tokyo, Japan);
Kazuhiko Kato (University of Tsukuba, Japan)

2. Bare-metal clouds
= IaaS providing physical machine
2
Internet
User
Data Center
Physical machines
E.g. IBM Cloud, Oracle Cloud, AWS

3. Virtual machine vs. Physical machine
3
OS
Direct
Access
OS
VMM
virt Hardware
Access
Translated
Access
Physical machineVirtual machine

4. Virtual machine vs. Physical machine
4
OS
Direct
Access
OS
VMM
virt Hardware
Access
Translated
Access
Physical machineVirtual machine
Have No virtualization overhead

5. Virtual machine vs. Physical machine
5
OS
Direct
Access
OS
VMM
virt Hardware
Access
Translated
Access
Physical machineVirtual machine
Have No virtualization overhead
Expose all hardware functions

6. Virtual machine vs. Physical machine
6
OS
Direct
Access
OS
VMM
virt Hardware
Access
Translated
Access
Physical machineVirtual machine
Have No virtualization overhead
Expose all hardware functions

7. Direct access to physical hardware
7
OS
Internet
Data Center
User
Direct
Access

8. Direct access to physical hardware
8
OS
Internet
Data Center
User
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Direct
Access

9. Direct access to physical hardware
9
OS
Internet
Data Center
User
Install
Rootkit
Break
Firmware
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware

10. Attack hardware by malicious user
10
OS
Internet
Data Center
User
Break
Firmware
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Hardware become
unworkable

11. Attack hardware by malicious user
11
OS
Internet
Data Center
User
Install
Rootkit
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware

12. Attack hardware by malicious user
12
OS
Data Center
User
Install
Rootkit
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Internet
User

13. Attack hardware by malicious user
13
OS
Internet
Data Center
User
Install
Rootkit
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Attack OS
User

14. Attack hardware by malicious user
14
OS
Internet
Data Center
User
Install
Rootkit
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Attack OS
Steal data
User

15. Attack hardware by malicious user
15
OS
Internet
Data Center
User
Install
Rootkit
Malicious
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware
Attack OS
Steal data
Break data
User

16. Existing counter methods
Protection of NVM by hardware
• May have vulnerability [Kallenberg et al. 2015]
• Not enabled in some real machines
• Many peripheral devices has no protection of NVM
Restoration of NVM after the machine is returned
• The hardware may not work enough to restore the NVM
• The rootkit may block the restoration
16

17. Related works
about hardware security
• VIPER [Yanlin., et al CCS 2011]
• Detecting malware in devices by measuring response time
• By the OS
• IOCheck[Fengwei., et al ESORICS 2014]
• Check the devices and firmware by BIOS in SMM
17
Detecting malware in hardware for protecting OS

18. Related works
about hardware security
• VIPER [Yanlin., et al CCS 2011]
• Detecting malware in devices by measuring response time
• By the OS
• IOCheck[Fengwei., et al ESORICS 2014]
• Check the devices and firmware by BIOS in SMM
18
Detecting malware in hardware for protecting OS
Not prevent from breaking firmware
Not remove installed malware

19. Our goal: Protect all of NVMs
• Even if the hardware does not have protection of itself
• Prevent modification of NVM
• Prevention is better than cure
19
Install
Rootkit
Break
Firmware
NVM
UEFI/BIOS
NVM NVM
Firmware Firmware

20. System requirement
in bare-metal clouds
• OS-independency
• Any OS will run on the machines
(including any version and customized OS)
• Almost zero performance degradation
• To keep performance advantage of the bare-metal clouds
20

21. Proposal: BMCArmor
21
Hardware
Hypervisor
Guest OS
NVMOther Functions Protection of
NVM
Enabling
Protect NVM by thin hypervisor
= Read access = Write access
Protect
Interrupt
DMA

22. Proposal: BMCArmor
22
Hardware
Hypervisor
Guest OS
NVMOther Functions Protection of
NVM
Enabling
Protect NVM by thin hypervisor
= Read access = Write access
OS-independent
Protect
Interrupt
DMA

23. Proposal: BMCArmor
23
Hardware
Hypervisor
Guest OS
NVMOther Functions Protection of
NVM
Enabling
Protect NVM by thin hypervisor
= Read access = Write access
OS-independent Pass-through
Protect
Interrupt
DMA

24. Proposal: BMCArmor
24
Hardware
Hypervisor
Guest OS
NVMOther Functions Protection of
NVM
Enabling
Protect NVM by thin hypervisor
= Read access = Write access
OS-independent Pass-through
Block writing
to NVM
Protect
Interrupt
DMA

25. Proposal: BMCArmor
25
Hardware
Hypervisor
Guest OS
NVMOther Functions Protection of
NVM
Enabling
Protect NVM by thin hypervisor
= Read access = Write access
OS-independent Pass-through
Block writing
to NVM
Keep protection enabled
Protect
Interrupt
DMA

26. Types of write accesses to NVM
26
NVM
OS
I/O
instruction
Registers
I/O space Physical memory space
Memory access
MMIO registers Memory mapped
NVM data

27. • BMCArmor uses the CPU’s function to intercept I/O
instructions issued by the guest OS
How to block the accesses via I/O spaces
27
NVM
Register
I/O space
Cause VMExit on read/write
 Translate the control to Hypervisor
RegisterRegister
OS
=Read/Write access
Hypervisor Read: Emulate, Write: Discard
Pass-through
Intercept

28. How to block the memory accesses
Intercept by using Nested-paging mechanism
28
Host Physical
Address
No write-permission
VMExit on writes
=Write access
NVM
Register
All permissions
 Pass-through
Guest Physical
Address
OS
Hypervisor Write: Discard
InterceptIntercept

29. Prototype Implementation
• Based on BitVisor [Shinagawa et al. VEE 2009]
• Enable protections of BIOS ROM by chipset
• Block write accesses to the BIOS ROM and NVM of NIC
29

30. Evaluation
Security Evaluation
• Does the hypervisor enable the protections?
• Does the hypervisor block the write accesses?
Performance Evaluation
• Is the overhead low?
30

31. Setup for the evaluation
• CPU: Intel Xeon E5-2603 v4 (1.70GHz)
• Memory: 16GB
• Mother board: ASRock X99 Extreme4
• NIC: Intel 82574L (1 GbE)
• OS: Ubuntu 16.04 LTS (Linux 4.4.0)
31

32. The machine does not enable the
protections
The results of CHIPSEC w/o BMCArmor : 3 “FAILED”s
32
# chipsec_main
[...]
[!] None of the SPI protected ranges write-protect BIOS region
[...]
[CHIPSEC] Modules failed 2:
[-] FAILED: chipsec.modules.common.bios_wp
[-] FAILED: chipsec.modules.common.spi_lock
[...]

33. Enabling protection functions
The results of CHIPSEC w/ BMCArmor :
3 “FAILED”s  0 “FAILED”s
33
# chipsec_main
[...]
[+] PASSED: BIOS is write protected (by SMM and SPI Protected
Ranges)
[...]
[CHIPSEC] Modules failed 0:
[...]
[+] PASSED: chipsec.modules.common.bios_wp
[+] PASSED: chipsec.modules.common.spi_lock
[...]

34. Modification NVM of NIC
34
˜# ethtool -e enp3s0 offset 0 length 6
Offset Values
------ ------
0x0000: 00 1b 21 53 84 3f
˜# ethtool -E enp3s0 magic 0x10d38086 value 0x11 offset 0x0
˜# ethtool -e enp3s0 offset 0 length 6
Offset Values
------ ------
0x0000: 11 1b 21 53 84 3f
˜#
On bare metal: Success the modification

35. Modification NVM of NIC
35
˜# ethtool -e enp3s0 offset 0 length 6
Offset Values
------ ------
0x0000: 00 1b 21 53 84 3f
˜# ethtool -E enp3s0 magic 0x10d38086 value 0x11 offset 0x0
Cannot set EEPROM data: Operation not permitted
˜# ethtool -e enp3s0 offset 0 length 6
Offset Values
------ ------
0x0000: 00 1b 21 53 84 3f
˜#
On BMCArmor: Blocked the modification

36. Network performance (Latency)
36
TCP
UDP
Latency (us)

37. Network performance (Latency)
37
TCP
UDP
< 1% overhead
≈ 24% overhead
Latency (us)

38. Number of VMExits
BMCArmor KVM
Write to MSR - 39854.7
External Interrupt - 33094.5
I/O instruction - 10473.5
EPT Violation 28239.3 -
Others 36.4 -
Total 28275.7 83422.3
38
Number of VMExits for 1 second during netperf workload
KVM has ≈ x3 VMExits
For timer
interrupt

39. Conclusion
BMCArmor protects hardware in bare-metal clouds
= Prevents OS’s writing to NVM by using a thin hypervisor
• Block write accesses to NVM
• Enable hardware’s protections of NVM
• Be OS-independent (b/c it is based on hypervisor)
• Have almost zero overhead (no device virtualization)
• Network latency increase is < 1%
39

40. Future work
• Support more devices (NVMe, 10 GbE)
• Performance evaluation on real applications
• KVS, SQL server, etc
• Evaluation on real services
• IBM Cloud, Oracle Cloud, AWS, etc
40