The AWS Global Network provides a secure, highly available, and high- performance infrastructure for customers. In this session, we walk through the architecture of various parts of the AWS network such as Availability Zones, AWS Regions, our Global Network connecting AWS Regions to each other and our Edge Network which provides Internet connectivity. We explain how AWS services such as AWS Direct Connect and Amazon CloudFront integrate with our Global Network to provide the best experience for our customers. We also dive into how the AWS Global Network connects to the rest of the Internet through peering at a global scale. If you are curious about how AWS network infrastructure can support large-scale cat photo distribution or how Internet routing works, this session answers those questions. Please join us for a speaker meet-and-greet following this session at the Speaker Lounge (ARIA East, Level 1, Willow Lounge). The meet-and-greet starts 15 minutes after the session and runs for half an hour.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Behind the Scenes:
Exploring the AWS Global Network
Tom Scholl
Sr. Principal Network Engineer
Amazon Web Services
N E T 3 0 5
Steve Seymour
Principal Solutions Architect
Amazon Web Services
N E T 3 0 5

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Key themes in the AWS network
• AWS Regions
• Global network backbone
• Edge POPs

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key themes
• Security & availability
• Strong isolation from failures
• Cellular architectures
• Scale
• Performance

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example of a customer traffic flow
VPC
Availability Zone
AWS Region
Internet
Cat Photos

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A bit more detailed…
AWS Region
Availability Zone
Datacenter
Datacenter
Availability Zone
Datacenter
Datacenter
Availability Zone
Datacenter
Datacenter
Backbone
Edge POPEdge POP
VPC
Cat Photos
Transit Center Transit Center
Internet Internet

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
• Failure isolation from other Availability Zones
• Directly connects to other Availability Zones
• Can include multiple data centers
• Low-latency & close proximity
• Scalability
Availability Zone
Region
Availability Zone Availability Zone
us-east-1 (N.Virginia)
us-east-1a us-east-1b us-east-1c

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zones
AWS Region
Availability Zone #1 Availability Zone #2 Availability Zone #3
Datacenter Datacenter
Datacenter Datacenter Datacenter Datacenter Datacenter Datacenter
Datacenter DatacenterDatacenter Datacenter
Transit Center #1 Transit Center #2

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data centers within Availability Zones
• Two main traffic flow types:
• Side-to-side (host to host)
• Up-and-down (to/from the internet, other AWS Regions)
• Data centers need to be elastic:
• Scaling up intra-AZ capacity
• Scaling up inter-AZ capacity
• Scaling up internet & inter AWS Region capacity

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a scalable data center
• What do you need?
• Network building blocks
• Make it easy to scale in right-sized increments
• Strong isolation boundaries
• Large amounts of network capacity
• Networking technology
• Routers
• Connectivity
• Control-plane

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building cellular data center architectures
Transit Centers
Other AZ
Local AZ
Datacenter
Hosts Hosts Hosts Hosts
Access Cell Access Cell Access Cell Access Cell
Other AZ
Local AZ
Datacenter
Core
Intra-AZ
Cell
Core Access Cell Core Access Cell
Core Edge Cell Core Edge Cell
Core
Inter-AZ
Cell
Core
Intra-AZ
Cell
Core
Inter-AZ
Cell

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Large-chassis vs. single chip routers
• Large-chassis routers
• More ports, larger failure domain
• Flexibility of port types with linecards
• Fewer devices to manage
• Multiple-stage forwarding architecture
• Single chip routers
• Fewer ports, contained failure domain
• Fixed-ports
• Many devices to manage
• Simpler forwarding architecture

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Large-chassis-based platforms
Large-Chassis Network Platform
Linecard
Linecard
Linecard
Linecard
Linecard
Linecard
Linecard
Switching
Fabric
Switching
Fabric
Switching
Fabric
Switching
Fabric
Linecard
Route Processor /
Supervisor (CPU)
Route Processor /
Supervisor (CPU)
PSU / Fans PSU / Fans PSU / Fans PSU / Fans
Routing
ASIC
Routing
ASIC
Routing
ASIC
Routing
ASIC
Routing
ASIC
Routing
ASIC
Routing
ASIC
Routing
ASIC

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Single-Chip Network Platform
Routing ASIC
PSU / Fans PSU / Fans
Route Processor /
Supervisor (CPU)
Single-chip-based platforms

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Operating a network of many devices
• More devices, more links—network monitoring
• Active data-plane probing
• Exercise traffic over all available paths
• Statistical deviations & anomaly detection
• What bits go in a device, must come out
• Extracting signal from devices
• Syslog, ASIC messages, registers,
table sizes
• Deep component monitoring
• Hardware
• Power
• Temperature
• Device lifecycle
• Automation is key for all stages
• Programmatic configuration

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Intra & inter-AZ connectivity
• Dark fiber “spans”
• Optimized for low-latency & physical diversity
• Amazon controlled infrastructure
• Geospatial coordinates
• Dense wavelength division multiplexing (DWDM)

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transit centers
• Provide internet and inter-Region (backbone) connectivity
• All Availability Zones are connected redundantly
• Located in facilities with dense internet inter-connection

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Global network backbone
• Multiple AWS services traverse it:

22. AWS Global
backbone
• From 2017
• Go to Peter DeSantis Monday
Night Live for the latest version

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why have a backbone network?
• Security
• Traffic traverses our infrastructure
rather than the internet
• Availability
• Controlling scaling and redundancy
• Traffic operates over Amazon-
controlled infrastructure
• Reliable performance
• Controlling specific paths customer
traffic traverses
• Connecting closer to
customers
• Avoiding internet “hot spots” or sub-
optimal external connectivity
All commercial Region-to-Region traffic
traverses the backbone except China

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a global backbone network
• Extreme auditing of fiber paths
• End-to-end latency
• Path hazards
• Repair expectations
• Path diversity
• Understanding shared risk link groups (SRLGs)
• Capacity/Scale
• Underlying optical transport capabilities

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building a backbone (cont’d)
• Latency matters
• Optimal in normalized situations
• Minimize additional latency during path failures
• 100G the new normal for backbone links
• Similar design patterns and operations to the data centers

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Global network backbone fabric
Transit Center / Edge POP Transit Center / Edge POP
Backbone Cell Backbone Cell
Backbone
Cell
Backbone
Cell
Backbone
Cell
Backbone
Cell
Remote POP
Remote POP
Remote POP
Remote POP
Remote POP Remote POP

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Backbone path additions
• Three new cable additions to the AWS Global network:

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How does Amazon connect to the internet?
• AWS Region transit centers
• Edge POPs via the AWS Global network

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Internet interconnection history
• Interconnection facilities/carrier hotels
• Internet exchanges

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POPs
• Extends the AWS Global network to the internet edge
• Increased network scaling
• Optimal interconnection with external networks

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP
• Multiple AWS services:
• AWS Direct Connect
• Amazon CloudFront (CDN)
• Amazon Route 53 (DNS)
• AWS Shield (DDoS Protection)
• AWS Global network access
• External internet connectivity

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP
AWS Global Network
Internet
External
Networks
External
Networks
External
Networks
External
Networks
External Internet Cell External Internet Cell
AWS
Shield
DDoS
Scrubbing
Backbone Cell Backbone Cell
Direct
Connect
Route 53
CloudFront

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—Amazon CloudFront
• At the Amazon Global network perimeter
• Low-latency to external networks
• Origin fetches traverse the AWS network backbone

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—Route 53
• Low-latency to external internet networks
• IPv4 and IPv6 DNS anycast services

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—AWS Direct Connect
• Low-latency access into AWS
• Access to all AWS Regions
• Multiple customer-facing edge routers for redundancy
• Multiple Edge POPs for redundancy

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—AWS Shield
• Traffic scrubbing platforms to protect
customers automatically
• Stopped at the internet edge before traffic
reaches the backbone

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—Internet connectivity
• Two types of external connectivity: transit and peering
• Transit: provides you the entire internet
• Peering: provides you connectivity to an external network and their downstream customers
• Thousands of peering relationships established globally

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inside an Edge POP—Internet connectivity
• Private peering/private network interconnection (PNI)
• Public peering/internet exchanges

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PNIs & internet exchanges
AWS
Global Network
Internet
Exchange
Switch
External
Network
Router
External Network
#2
External
Network
Router
External Network
#3
External
Network
Router
External Network
#1
BGP Sessions
Amazon
Router
AWS
Global Network
External Network
PNI
BGP Session
Amazon
Router
External
Network
Router

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BGP—The internet control-plane
• TCP session established between routers
• Routers exchange messages containing routing information over BGP
• BGP provides reachability information for internet prefixes

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BGP messages
Amazon
Router
External
Network
Router
AWS
Global Network
(AS16509)
External Network
#1
(AS23456)
BGP Update
Prefix: 3.0.0.0/15
AS-Path: AS16509
BGP Update
Prefix: 13.232.0.0/14
AS-Path: AS23456
External Network
#2
(AS65535)
BGP Update
Prefix: 52.220.0.0/15
AS-Path: AS23456 AS65535
3.0.0.0/15 13.232.0.0/14 52.220.0.0/15

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POP internet edge—Design
• Similar design patterns to the data center
• Go wide—stripe external connections across multiple routers
• Understanding dependencies: extracting BGP peer-id to understand
our peer network failure domains

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Going wide at the internet edge
AWS
Edge POP
Router
Router
Router
Router
Router
Router
External Network #1
Router
Router
External Network #2Router
External Network #3
Router
Router
Router

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POP internet edge—Monitoring
• Active data-plane monitoring
• Collecting internet performance data from AWS service logs

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POP internet edge—Auto-remediation
• Internet traffic-engineering
• Traffic management/congestion avoidance
• Rerouting around upstream faults
• Inbound
• Outbound

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POP internet edge—Outbound
AWS
Edge POP
Router
Router
Router
External Network #1
Router
Router
External Network #2Router
External Network #3
Router
Router Router
Router
Destination

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge POP internet edge—Inbound
AWS
Edge POP
Router
Router
Router
External Network #1
Router
Router
External Network #2Router
External Network #3
Router
Router Router
Router
Source
BGP Updates

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Conclusions
• Strong isolation from failures
• Extensive network monitoring & auto-remediation systems
• Large amounts of redundancy and over-provisioning
• Easily scalable at every layer
• Custom hardware and end-to-end control

50. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please join us for a speaker meet-and-greet following this session at the Speaker
Lounge (ARIA East, Level 1, Willow Lounge). The meet-and-greet starts 15 minutes
after the session and runs for half an hour.
Tom Scholl
Steve Seymour

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.