The new Nitro architecture is fundamental to the Amazon EC2 virtual machine service. With Nitro, each host in the core compute platform is built with trusted computers that simulate the outside world and surround an untrusted CPU and memory computer that runs workloads. Those trusted Nitro computers appear to the customer workload computer as I/O devices that are accessible across the PCIe bus. Most of the traditional virtualization work is done via hardware emulation. The Nitro computers carefully control the workload computer access, providing a layer of protection. Learn about the security properties of this powerful architecture, which significantly increase cloud reliability and performance.

1. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security benefits of the
Nitro architecture
Mark Ryland
Director, Office of the CISO
AWS
S E P 4 0 1 - R 1

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Traditional virtualization and its limitations
The Amazon EC2 Nitro architecture
Security benefits of Nitro
Questions

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts and previous talks
Tuesday, June 25
SEP402: Encrypting everything with AWS
re:Invent 2017
C5 Instances and the Evolution of Amazon EC2 Virtualization
re:Invent 2017
Amazon EC2 Bare Metal Instances
re:Invent 2018
Powering Next-Gen EC2 Instances: Deep Dive into the Nitro System

4. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Intel mainboard
Amazon EC2 CR1: January
2013
Traditional software-based
virtualization
Virtual machine monitor is
relatively simple
Device models and privileged OS
are not
Amazon Linux
(dom0)
cr1.8xlarge
(domU/guest)
Amazon EBS volumes
DM
Instance storage
VPC networking
Othersoftware
DM
DM
DM
Xen

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
VMM
Virtualization: standard instructions

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
ERROR
Virtualization: privileged instructions

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
TRAP
VMM
Trap & emulate: virtual machine monitor

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
VMM
EMULATE
Trap & emulate : virtual machine monitor

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
VMM
Continue

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
<_start>:
e9 59 e1 17 00 jmpq ffff82d08037e15e
0f 1f 00 nopl (%rax)
<multiboot1_header_start>:
02 b0 ad 1b 03 00 add 0x31bad(%rax),%dh
00 00 add %al,(%rax)
fb sti
4f 52 rex.WRXB push %r10
e4 0f in $0xf,%al
<multiboot1_header_end>:
0f 1f 40 00 nopl 0x0(%rax)
<multiboot2_header_start>:
d6 (bad)
50 push %rax
52 push %rdx
e8 00 00 00 00 callq ffff82d080200020
88 00 mov %al,(%rax)
EMULATE
VMM
TRAP
Device
Model
Device
Model
Device
model
Trap & emulate: devices and I/O

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Beginning the Nitro journey
(Re)invent and simplify; can we do better?
Data center CPUs are powerful and expensive; it is wasteful to use them as accelerators
Device models compete for CPU and system resources; jitter is hard to avoid
Dom0 OS is a big, complicated piece of software and a convenient landing zone
Apply microservices and building block concepts to simplify development,
enhance quality, and speed up innovation?
Use specialized hardware for acceleration and increased security?

13. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Intel mainboard
Step by step
Multi-year process to decompose
the system
Device model by device model
Learnings along the way
Amazon EBS volumes
DM
Instance storage
VPC networking
DM
Othersoftware
DM
DM
Amazon Linux
(dom0)
cr1.8xlarge
(domU/guest)
Xen

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PCIe bus
Amazon
RDS
m5d.24xlarge
Nitro hypervisor (KVM-based)
Amazon EBS
volumes
Instance
storage
Intel mainboard
Nitro architecture
ENA
Private network
Nitro controller & other
Nitro computers

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PCIe bus
Amazon
RDS
Nitro hypervisor (KVM-based)
Instance
storage
Intel mainboard
Nitro architecture
ENA
Private network
m5d.4xlarge
m5d.4xlarge
m5d.4xlarge
m5d.4xlarge
m5d.4xlarge
m5d.4xlarge
Nitro controller & other
Nitro computers
Amazon EBS
volumes

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PCIe bus
Amazon
RDS
M5d.metal
instance type
Instance
storage
Intel mainboard
Nitro architecture
ENA
Private network
Nitro controller & other
Nitro computers
Amazon EBS
volumes

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PCIe bus
Amazon
RDS
Instance
storage
Intel mainboard
Nitro architecture
ENA
Private network
Nitro controller & other
Nitro computers
Amazon EBS
volumes

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PCIe bus
Amazon
RDS
Instance
storage
Intel mainboard
Amazon EBS attach volume API
User calls Amazon EC2 API endpoint
Internal microservices send command to control plane
Control plane sends command to Nitro controller
Nitro controller sends command to EBS controller
EBS controller sends hot-plug event for PCIe device
NVMe device (emulated) shows up on the bus
ENA
Nitro controller & other
Nitro computers
Amazon EBS
volumes

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key aspects of Nitro
Software device models replaced by (software-defined) hardware devices on the
system bus
Amazon EC2 dynamic system changes modeled as hardware events (e.g., NVMe
and ENA hot-plug, ACPI power state changes)
Extension of microservice architecture into hardware
ENA, NVMe protocols are hardened APIs behind which we can innovate
Data hiding and service decomposition
Apply (relatively) cheap hardware acceleration to a range of issues
Software elements are also microservices, all dynamically updatable
No virtual machine (VM) downtime required even for major updates

21. © 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security (CIA) benefits
Confidentiality
Encryption features
Passive communications design
Additional benefits
Integrity
Secure boot process of Nitro controller (and other Nitro computers)
Secure boot process of mainboard (“customer workload co-processor”)
Availability
Systems can be fully updated without VM downtime

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Integrity: Nitro system
Nitro controller is the root of trust
Nitro controller boots from completely private SSD
Boot process formally verified by AWS Automated Reasoning Group:
https://link.springer.com/chapter/10.1007/978-3-319-96142-2_28
Conducts various integrity checks of
Nitro computers
Continues on with mainboard boot
When necessary, secure software
updates for all components using
secure channels, signed binaries

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
QPI
Integrity: M5d boot process
Mainboard cannot
update firmware
But…
Hold mainboard in reset
during power-up
Validate all firmware;
if valid, continue
Either inject known-good
hypervisor
Or boot customer
OS/hypervisor AMI from
pseudo-NVMe (EBS)
volume
Intel mainboard
Nitro computers &
private network
Nitro controller

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Encryption—performance
Hardware acceleration allows for line-rate AES-256 encryption of EBS,
instance storage, and network without performance penalty
Instance storage: All data
EBS: Now enforceable (for all types) at the account level
Network: Beginning with N types, all direct inter-N customer traffic
Same VPC and across VPC peering, same region
All at up to 100 GB/s
Cf: Project Lever; VPC x-region peering
Caveats

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Encryption—key management
EBS: Volumes have independent lifetimes (plus snapshots); therefore, key
management via AWS KMS
Instance storage: Locally generated, used, and deleted (instance lifecycle)
VPC:
Seed materials regionally generated and managed in AWS KMS
Seeds distributed, not actual secrets; rotated frequently
(previous, current, next)
In all cases, plaintext data keys are
cached/ used only on Nitro computers
Protected from “customer workload co-processor”

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Passive communications design
Hypervisor awaits commands from Nitro controller
Sent via trusted communications channel
Never initiates communications with the controller
Not connected to the network at all
Nitro controller awaits commands from the external control plane
Listens on network substrate awaiting encrypted, authenticated API calls
Never initiates outbound connections
Outbound communications from either layer
are a clear sign of compromise and are
treated accordingly

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional confidentiality benefits
No Dom0 in Nitro hypervisor—greater simplicity and safety
No SSH or other interactive modes anywhere—no direct human access
All access via 100% AuthN/AuthZ APIs with logging/auditing
—no APIs for memory access
Only the Nitro controller has access to the physical Amazon EC2 network; the
mainboard does not
End-to-end Nitro system is developed,
deployed, and managed by DevSecOps
process

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In sum
Big benefits from applying microservice concepts to hardware and full-
system design
Unneeded functionality not present
What remains is better-defined and easier to reason about: build/test/validate
Stronger single root of trust and greater separation of concerns (and code,
and teams) along every dimension
Nitro building blocks will continue to be
applied in Amazon EC2 and beyond
Firecracker, Outposts, etc.
Lots of security value already, and there’s
more that we can do!

30. Thank you!
© 2019,Amazon Web Services, Inc. or its affiliates. All rights reserved.
Mark Ryland
markry@amazon.com