AWS Direct Connect provides low latency and high performance connectivity to the AWS cloud by allowing the provision of physical fiber from the customer’s location or data center into AWS Direct Connect points of presence. This session covers design considerations around AWS Direct Connect solutions. We will discuss how to design and configure physical and logical redundancy using both physically redundant fibers and logical VPN connectivity, and includes a live demo showing both the configuration and the failure of a doubly redundant connectivity solution. This session is for network engineers/architects, technical professionals, and infrastructure managers who have a working knowledge of Amazon VPC, Amazon EC2, general networking, and routing protocols.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Matt Lehwess, AWS
October 2015
ARC402
Double Redundancy
With AWS Direct Connect

2. Agenda
• Building network foundations in AWS
• Connecting your onsite deployment to AWS
• Adding some redundancy into the mix
• Demo: Taking our environment live and
introducing some failures!

3. Foundations: Amazon VPC
Your own private, isolated section of the AWS cloud

4. VPC CIDR 10.1.0.0/16
Availability Zone A Availability Zone B
Public Subnet Public Subnet
Private Subnet Private Subnet
Instance A
10.1.1.11 /24
Instance B
10.1.2.22 /24
Instance C
10.1.3.33 /24
Instance D
10.1.4.44 /24
10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
Only 1 IGW and 1 VGW per VPC

5. Foundations: Other Services
Lets add some AWS services outside of VPC

6. AWS Region - eg: US-WEST1
Our VPC from Earlier
AWS Region
AWS Region Level Services (plus many more)
AWS VPC Internal Services (e.g. Amazon EMR,
Elastic Load Balancing, Amazon RDS)
IGW, gateway between AWS region level
services and internal VPC services
Instance A
10.1.1.11 /24
Availability Zone A Availability Zone B
Public Subnet Public Subnet
Private Subnet Private Subnet
Instance B
10.1.2.22 /24
Instance C
10.1.3.33 /24
Instance D
10.1.4.44 /24
10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
Amazon SNS
Amazon SQS
Amazon SWF
Amazon SES
Amazon S3
Amazon Glacier
Amazon DynamoDB
AWS Lambda

7. Connectivity: AWS to On-Premises
Using AWS Direct Connect

8. 10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
Customer DCColocation Facility - e.g. Equinix SV1
VPC CIDR 10.1.0.0/16
Customer Subnet
192.168.0.0/16
Direct Connect POP
Colocation Facility
Customer or Partner Device
AWS Direct Connect
Point of Presence
Customer Gateway
Cross Connect
Customer Data Center
Service Provider Backhaul
Anatomy of AWS Direct Connect
Private Virtual Interface
Configure Customer Gateway
VPC VGW

9. Standard Interface & BGP Configuration…
interface GigabitEthernet0/1
no ip address
interface GigabitEthernet0/1.807
description "Direct Connect to your Amazon VPC or AWS Cloud"
encapsulation dot1Q 807
ip address 172.16.7.5 255.255.255.252
router bgp 65001
neighbor 172.16.7.6 remote-as 7224
neighbor 172.16.7.6 password 7 $1$zVOvlUSp$UrqWP2awtiG8ZbXo9BwcB
network 0.0.0.0
exit
Physical Interface that fiber is plugged into
Sub-interface (Generally matches VLAN)
VLAN Association
/30 Private P2P address
BGP ASN
Route Advertisement to AWS
Just a description
BGP MD5 Password
Neighbor Peer Address

10. 10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16
Customer DCColocation Facility - e.g. Equinix SV1
Customer Subnet
192.168.0.0/16
Configure Customer Gateway
Customer Gateway
BGP Comes up, prefixes are advertised.
%BGP-5-ADJCHANGE: neighbor 172.16.6.6 Up
AWS Direct Connect
Point of Presence
Anatomy of AWS Direct Connect continued...

11. 10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16
Customer DCColocation Facility - e.g. Equinix SV1
Customer Subnet
172.160.0.0/16
Anatomy of AWS Direct Connect continued...
Customer Gateway
AWS Direct Connect
Point of Presence
My Private Virtual Interface is up, now what?
What about my S3 bucket or DynamoDB? – in comes Public Virtual Interfaces!

12. 10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16
Amazon SNS
Amazon SQS
Amazon SWF
Amazon SES
Amazon S3 Amazon DynamoDB
AWS Region - eg: US-WEST1
AWS LambdaAmazon Glacier
Customer DCColocation Facility - e.g. Equinix SV1
Customer Subnet
172.160.0.0/16
Customer Gateway
AWS Regions much larger than just what’s inside a VPC
Create Public Virtual Interface
Configure Customer Gateway
BGP Comes up, prefixes are advertised (Public only!).
%BGP-5-ADJCHANGE: neighbor 203.50.24.5
Anatomy of AWS Direct Connect continued...
AWS Direct Connect
Point of Presence

13. Adding Redundancy
“Everything fails, all the time.” – Werner Vogels

14. Anatomy of a redundant AWS Direct Connect
Customer Subnet
172.160.0.0/16
Double connectivity
The standard connectivity we built earlierVPC VGW
Redundant DX POP LocationOther AWS Services
10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16
Amazon SNS
Amazon SQS
Amazon SWF
Amazon SES
Amazon S3 Amazon DynamoDB
AWS Region - eg: US-WEST1
AWS LambdaAmazon Glacier

15. Amazon SNS
Amazon SQS
Amazon SWF
Amazon SES
Amazon S3 Amazon DynamoDB
AWS Region - eg: US-WEST1
AWS LambdaAmazon Glacier
Anatomy of a redundant AWS Direct Connect
Customer Subnet
172.160.0.0/16
How do we configure redundant BGP?
And here too!
10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16

16. Standard Interface & BGP Configuration…
#Active Passive deployment:
router bgp 65001
neighbor 10.1.0.2 remote-as 65200
neighbor 10.1.0.2 description Backup
neighbor 10.1.0.2 route-map prepend out
route-map prepend permit 10
set as-path prepend 65001 65001 65001
Using one link as the primary, and the
other “Prepended” as the secondary
and less preferred route

17. Autonomous System (AS) Path Prepending?
Origin NetworkPrepended ASNPrepended ASNPrepended ASN
Verses.
Origin Network
Metric 4
Metric 1
Less Preferred
More Preferred
0%
100%

18. Standard Interface & BGP Configuration…
#Active Active deployment:
router bgp 1
maximum-paths 4 Usually reserved for a single customer router scenario,
can be configured at the service provider level as well.
Note: By default we “Multi-path” outbound from VGW over equal cost paths
unless you set a metric such as AS PATH on one route.

19. Autonomous System (AS) Equal Paths
Origin Network
Vs.
Origin Network
Metric 1
Metric 1
Both Preferred
Both Preferred
50%
50%

20. Did I hear Double Redundancy?
You can use VPN as your backup of backups

21. Amazon SNS
Amazon SQS
Amazon SWF
Amazon SES
Amazon S3 Amazon DynamoDB
AWS Region - eg: US-WEST1
AWS LambdaAmazon Glacier
10.1.1.0/16
10.1.2.0/16
10.1.3.0/16
VPC CIDR 10.1.0.0/16
Anatomy of a redundant AWS Direct Connect
Customer Subnet
172.160.0.0/16
Most MPLS Providers can “trunk”
you an internet circuitOur VGW’s are also used as VPN
connection points remember!
Dual VPN tunnels providing
connectivity and encryption.

22. VPN & BGP Redundancy Configuration…
#Direct Connect Interface:
interface GigabitEthernet0/0/0.259
description "Direct Connect to your Amazon VPC or AWS Cloud"
encapsulation dot1Q 259
ip address 169.254.254.2 255.255.255.252
bfd interval 300 min_rx 300 multiplier 3
!
Subinterface
VLAN ID
Local IP Address
BFD Configuration

23. VPN & BGP Redundancy Configuration…
#Inter Router Interface:
interface GigabitEthernet0/1
description ** Internal Interface - SW2 Gi2/0/1 **
ip address 192.168.51.253 255.255.255.0
ip virtual-reassembly in
standby 1 ip 192.168.51.254
standby 1 timers msec 300 msec 900
standby 1 priority 110
standby 1 preempt
duplex auto
speed auto
!
Local LAN IP
HSRP Configuration
HSRP sub second hello
This router is primary
Preempt primary if not active

24. VPN & BGP Redundancy Configuration…
BGP Configuration:
router bgp 65501
bgp log-neighbor-changes
neighbor 169.254.254.1 remote-as 9059
neighbor 169.254.254.1 password 7 124B36F51
neighbor 169.254.254.1 fall-over bfd
neighbor 192.168.51.252 remote-as 65501
!
Direct Connect neighbor
BFD Configuration
Inter router neighbor

25. VPN & BGP Redundancy Configuration…
Secondary router BGP and route-map assignment:
router bgp 65501
bgp log-neighbor-changes
neighbor 169.254.254.37 remote-as 9059
neighbor 169.254.254.37 route-map LOCAL-PREF in
neighbor 169.254.254.37 route-map AS-PREPEND out
Secondary Direct Connect
neighbor
Inbound route-map
Outbound route-map

26. VPN & BGP Redundancy Configuration…
Secondary router route-map:
ip prefix-list LOCAL-ROUTES seq 10 permit 192.168.0.0/16 le 32
route-map AS-PREPEND permit 10
match ip address prefix-list LOCAL-ROUTES
set as-path prepend 65501 65501
!
route-map LOCAL-PREF permit 10
set local-preference 90
!
Match local routes for AS prepending
Match above prefix list
Add ASN x 2 to AS Path
Set local preference to 90 (for secondary)

27. Now adding VPN….
VPN Tunnel interface (Straight forward):
interface Tunnel1
ip address 169.254.20.62 255.255.255.252
ip virtual-reassembly in
ip tcp adjust-mss 1387
tunnel source 62.216.229.132
tunnel mode ipsec ipv4
tunnel destination 52.17.141.73
tunnel protection ipsec profile ipsec-vpn-946e19df-0
!

28. Now adding VPN….
VPN Tunnel interface (Straight forward):
interface Tunnel2
ip address 169.254.20.162 255.255.255.252
ip virtual-reassembly in
ip tcp adjust-mss 1387
tunnel source 62.216.229.132
tunnel mode ipsec ipv4
tunnel destination 52.18.219.193
tunnel protection ipsec profile ipsec-vpn-946e19df-1
!
Plus your other VPN goodness like crypto-maps…

29. Now adding VPN….
VPN BGP Configuration (Still standard..)
Router BGP 65501
neighbor 169.254.20.61 remote-as 9059
neighbor 169.254.20.61 timers 10 30 30
!
Address-family ipv4
network 192.168.51.0
neighbor 169.254.20.61 activate
neighbor 169.254.20.61 route-map LOCAL-PREF-VPN in
neighbor 169.254.20.61 route-map AS-PREPEND-VPN out
!
Standard BGP Configuration
Where it gets interesting…

30. Now adding VPN….
#Where we add our metrics:
route-map AS-PREPEND-VPN permit 10
match ip address prefix-list LOCAL-ROUTES
set as-path prepend 65501 65501 65501
!
route-map LOCAL-PREF-VPN permit 10
set local-preference 80
!
An additional ASN beyond our backup direct connect link
Local Preference is 10 lower than our backup
Direct Connect link

31. Our real life environment

32. Demo
Let’s see how our use case was built on AWS

33. Our real life environment

34. In summary
• Built our network foundations in AWS
• Connected your onsite deployment to AWS
• Added some redundancy into the mix
• Demo: Took our environment live and
introduced some failures!

35. Related Sessions
NET406 - Deep Dive: AWS Direct Connect and VPNs
• Thursday, Oct 8, 2:45 PM - 3:45 PM – Palazzo C
ISM403 - How Amazon.com is Moving to Amazon WorkSpaces
• Thursday, Oct 8, 1:30 PM - 2:30 PM – Titian 2306

36. Thank you!

37. Remember to complete
your evaluations!