Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS re:Invent 2016: Another Day, Another Billion Packets (NET401)

1,058 views

Published on

In this session, we walk through the Amazon VPC network presentation and describe the problems we were trying to solve when we created it. Next, we walk through how these problems are traditionally solved, and why those solutions are not scalable, inexpensive, or secure enough for AWS. Finally, we provide an overview of the solution that we've implemented and discuss some of the unique mechanisms that we use to ensure customer isolation, get packets into and out of the network, and support new features like VPC endpoints.

Published in: Technology
  • Be the first to comment

AWS re:Invent 2016: Another Day, Another Billion Packets (NET401)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Steve Mueller, Senior Product Manager, VPC, EC2 Networking November 30, 2016 Another Day, Another Billion Packets NET401
  2. 2. We Have the Cloud Amazon EBS Amazon RDS Amazon ElastiCache Amazon Redshift Amazon EC2 Elastic Load Balancing
  3. 3. Customers Have Datacenters
  4. 4. Whiteboard Engineering Amazon EBS Amazon RDS Amazon ElastiCache Amazon Redshift Amazon EC2 Elastic Load Balancing
  5. 5. EC2 as It Was 10.44.12.4 10.44.12.5 10.44.92.17 10.44.12.27 10.108.6.4
  6. 6. Why That Doesn’t Work 192.168.0.0/16 Routing Table • 192.168.0.0/16: stay here • 10.44.12.4/32: AWS • 10.44.92.17/32: AWS • 10.108.6.4/32: AWS 10.44.0.0/16 10.44.12.4 10.44.12.5 10.44.92.17 10.44.12.27 10.108.6.4
  7. 7. Requirements Customer selected IP addresses Route aggregation for external connectivity Conformance with existing network designs
  8. 8. 172.31.0.0/18 192.168.0.0/16 Routing Table • 192.168.0.0/16: stay here • 172.31.0.0/18: AWS 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.1.8 172.31.1.9 172.31.2.12 172.31.2.51 Amazon Virtual Private Cloud (Amazon VPC)
  9. 9. This Is Just Virtual Networking! Subnet ~= VLAN VPC ~= VRF (virtual routing and forwarding) But…
  10. 10. Scaling Challenges VLAN ID space is constrained • 12 bits => 4096 total VLANs VRF support is constrained • Large routers => 1-2 thousand VRFs Fixed ratio of VLANs:VRFs
  11. 11. Router and Capacity Dimensions Big Router Data Plane Control Plane Big Router Data Plane Control Plane
  12. 12. An Example Average router configuration line: 50 chars Config per VPC: 10 lines Subnets per VPC: 4 Config per subnet: 5 lines Total VPCs: 2,000 Config size: 3 MB
  13. 13. But… Doesn’t scale • 12 bit VLAN ID = 4096 VLANs (not enough) • BIG routers support 4000 VRFs ($200k+) Large VLANs make NEs cry Tied to vendor bugfix cycles (6 months +) We want commodity, fungible network gear • BIG virtual routers are built by few companies • Interoperability of advanced features is marginal
  14. 14. Silos of Capacity A C B FE D G A AA A B C B B B B C D F FF D D B G G /4 /4 /40 /40 0 0 0 0 1324 132 C G G 3 27 D DD 9910 F F F F F 1815 40 BB B B B BB B B B BB B B B B B
  15. 15. Implementation Requirements Scale to millions of environments the size of Amazon.com Any server, anywhere in a region can host an instance attached to any subnet in any VPC
  16. 16. Concepts Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 … 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service Server: Physical host in an Amazon data center Instance: Amazon EC2 instance owned by a customer VPC: Amazon Virtual Private Cloud owned by a customer VPC ID: Identifier for a VPC such as vpc- 1a2b3c4d Mapping Service: Distributed lookup service. Maps VPC + Instance IP to server
  17. 17. Layer 2 (L2): Ethernet 10.0.0.2 10.0.0.3 L2 Src: MAC(10.0.0.2) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.3? The switch floods the ARP request out all ports Ethernet Switch L2 Src: MAC(10.0.0.3) L2 Dst: MAC(10.0.0.2) ARP 10.0.0.3 is at MAC(10.0.0.3) The switch snoops the ARP response and learns the port for MAC(10.0.0.3). L2 Src: MAC(10.0.0.2) L2 Dst: MAC(10.0.0.3) L3 Src: 10.0.0.2 L3 Dst: 10.0.0.3 ICMP/TCP/UDP/…
  18. 18. Layer 2 (L2): VPC Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service L2 Src: MAC(10.0.0.2) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.3? L2 Src: MAC(10.0.0.3) L2 Dst: MAC(10.0.0.2) ARP 10.0.0.3 is at MAC(10.0.0.3) Src: 192.168.0.3 Dst: Mapping Service Query: Blue 10.0.0.3 Src: Mapping Service Dst: 192.168.0.3 Reply: Host: 192.168.1.4 MAC: MAC(10.0.0.3) 10.0.0.2
  19. 19. Server 192.168.0.3 Server 192.168.0.4 Server 192.168.1.3 Server 192.168.1.4 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service 10.0.0.2 … L2 Src: MAC(10.0.0.2) L2 Dst: MAC(10.0.0.3) L3 Src: 10.0.0.2 L3 Dst: 10.0.0.3 ICMP/TCP/UDP/… VPC: Blue Src: 192.168.0.3 Dst: 192.168.1.4 Src: 192.168.1.4 Dst: Mapping Service Validate: Blue 10.0.0.2 is at 192.168.0.3 Src: Mapping Service Dst: 192.168.1.4 Mapping valid: Blue10.0.0.2 is at 192.168.0.3 Layer 2 (L2): VPC …
  20. 20. VPC Isolation Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service 10.0.0.2 Src: 192.168.0.4 Dst: Mapping Service Query: Grey 10.0.0.3 L2 Src: MAC(10.0.0.4) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.3?
  21. 21. VPC Isolation Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service 10.0.0.2 Src: 192.168.0.4 Dst: Mapping Service Query: Blue 10.0.0.3 L2 Src: MAC(10.0.0.4) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.3? 192.168.0.4 is not hosting any instances in VPC Blue. Mapping Denied Alarm Raised
  22. 22. VPC Isolation Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service 10.0.0.2 … L2 Src: MAC(10.0.0.4) L2 Dst: MAC(10.0.0.3) L3 Src: 10.0.0.4 L3 Dst: 10.0.0.3 ICMP/TCP/UDP/… VPC: Blue Src: 192.168.0.4 Dst: 192.168.1.4 Src: 192.168.1.4 Dst: Mapping Service Validate: Blue 10.0.0.4 is at 192.168.0.4 Src: Mapping Service Dst: 192.168.1.4 Mapping invalid! 192.168.1.4 does not deliver the packet to the instance. Alarm Raised.
  23. 23. Layer 3 (L3): IP Routing 10.0.0.2 10.0.1.3 L2 Src: MAC(10.0.0.2) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.1? Ethernet Switch L2 Src: MAC(10.0.0.1) L2 Dst: MAC(10.0.0.2) ARP 10.0.0.1 is at MAC(10.0.0.1) L2 Src: MAC(10.0.0.2) L2 Dst: MAC(10.0.0.1) L3 Src: 10.0.0.2 L3 Dst: 10.0.1.3 ICMP/TCP/UDP/… Router Ethernet Switch L2 Src: MAC(10.0.1.1) L2 Dst: MAC(10.0.1.3) L3 Src: 10.0.0.2 L3 Dst: 10.0.1.3 ICMP/TCP/UDP/…
  24. 24. Layer 3 (L3): VPC Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.1.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service L2 Src: MAC(10.0.0.2) L2 Dst: ff:ff:ff:ff:ff:ff ARP Who has 10.0.0.1? L2 Src: MAC(10.0.0.1) L2 Dst: MAC(10.0.0.2) ARP 10.0.0.1 is at MAC(10.0.0.1) Src: 192.168.0.3 Dst: Mapping Service Query: Blue 10.0.0.1 Src: Mapping Service Dst: 192.168.0.3 Reply: Host: Gateway MAC: MAC(10.0.0.1) 10.0.0.2
  25. 25. Layer 3 (L3): VPC Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 10.0.1.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service Src: 192.168.0.3 Dst: Mapping Service Query: Blue 10.0.1.3 Src: Mapping Service Dst: 192.168.0.3 Reply: Host: 192.168.1.4 MAC: MAC(10.0.1.3) 10.0.0.2 L2 Src: MAC(10.0.0.2) L2 Dst: MAC(10.0.0.1) L3 Src: 10.0.0.2 L3 Dst: 10.0.1.3 ICMP/TCP/UDP/… VPC: Blue Src: 192.168.0.3 Dst: 192.168.1.4 Src: 192.168.1.4 Dst: Mapping Service Validate: Blue 10.0.0.2 is at 192.168.0.3 Src: Mapping Service Dst: 192.168.1.4 Mapping valid: Blue 10.0.0.2 is at 192.168.0.3 L2 Src: MAC(10.0.1.1) L2 Dst: MAC(10.0.1.3) L3 Src: 10.0.0.2 L3 Dst: 10.0.1.3 ICMP/TCP/UDP/…
  26. 26. Caching Server 192.168.0.3 Server 192.168.0.4 … Server 192.168.1.3 Server 192.168.1.4 … 10.0.0.2 10.0.0.3 10.0.0.4 10.0.0.4 10.0.0.2 10.0.0.5 10.0.0.3 Mapping Service L2 Src: MAC(10.0.1.1) L2 Dst: MAC(10.0.1.3) L3 Src: 10.0.0.2 L3 Dst: 10.0.1.3 ICMP/TCP/UDP/…
  27. 27. 10.0.0.0/18 172.16.0.0/16 10.0.0.0/24 10.0.1.0/24 10.0.0.7 10.0.0.8 10.0.0.9 10.0.1.12 10.0.1.51 VPC: Blue Src: 192.168.0.3 Dst: ??? L3 Src: 10.0.0.7 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… Getting Home – or Anywhere, Really
  28. 28. Edges Server 192.168.0.3 Server 192.168.0.4 … Edge 192.168.4.3 Edge 192.168.4.4 10.0.1.3 10.0.0.4 10.0.0.2 Mapping Service 10.0.0.2 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… Host 10.0.0.4  192.168.0.4 Host 10.0.1.4  192.168.0.4 … 172.16.0.0/16  Edge 192.168.4.3 …
  29. 29. Edges: VPN Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… IPSEC Stuff Src: 54.68.100.245 Dst: 205.251.242.54 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… VPN
  30. 30. Edges: Direct Connect Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… 802.1Q VLAN Tag Src: 54.68.100.245 Dst: 205.251.242.54 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… AWS Direct Connect
  31. 31. Edges: Internet (IGW) Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 176.32.96.190 ICMP/TCP/UDP/… L3 Src: 10.0.0.2 L3 Dst: 176.32.96.190 ICMP/TCP/UDP/… Internet 54.148.157.46
  32. 32. Edges: Recap VPN Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… IPSEC Stuff Src: 54.68.100.245 Dst: 205.251.242.54 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… AWS Direct Connect Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… 802.1Q VLAN Tag Src: 54.68.100.245 Dst: 205.251.242.54 L3 Src: 10.0.0.2 L3 Dst: 172.16.14.17 ICMP/TCP/UDP/… Internet Edge 192.168.4.3 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.3 L3 Src: 10.0.0.2 L3 Dst: 176.32.96.190 ICMP/TCP/UDP/… L3 Src: 54.148.157.46 L3 Dst: 176.32.96.190 ICMP/TCP/UDP/…
  33. 33. Image credit: Wikipedia https://en.wikipedia.org/wiki/1918_Eighth_Avenue A Brief Diversion
  34. 34. VPC Pricing Cost per VPC: $0.00 Cost per subnet: $0.00 Upcharge per instance: $0.00
  35. 35. Nov 10, 2010
  36. 36. 172.31.0.0/18 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.1.8 172.31.2.12 172.31.2.51 VPC as a Platform
  37. 37. VPC as a Platform VPN and Direct Connect Security group egress filtering Network ACLs Routing tables Elastic Network Interfaces (ENIs) Multiple IPs
  38. 38. Amazon S3 Endpoints 172.31.0.0/18 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.2.12
  39. 39. Amazon S3 Endpoints 172.31.0.0/18 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.2.12
  40. 40. Server 192.168.0.3 Server 192.168.0.4 … Edge 192.168.4.3 Edge 192.168.4.4 10.0.1.3 10.0.0.4 10.0.0.2 10.0.0.2 L3 Src: 10.0.0.2 L3 Dst: 54.231.33.89 TCP/HTTP/… VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.4 L3 Src: 10.0.0.2 L3 Dst: 54.231.33.89 TCP/HTTP/… Edges Mapping Service Host 10.0.0.4  192.168.0.4 Host 10.0.1.4  192.168.0.4 … 172.16.0.0/16  Edge 192.168.4.3 S3.us-east-1  Edge 192.168.4.4 …
  41. 41. A New Edge: S3 Edge 192.168.4.4 VPC: Blue Src: 192.168.0.3 Dst: 192.168.4.4 L3 Src: 10.0.0.2 L3 Dst: 54.231.33.89 TCP/HTTP/… VPC Endpoint 1a2b3c4d Src: 54.68.100.245 Dst: 54.231.33.89 L3 Src: 10.0.0.2 L3 Dst: 54.231.33.89 TCP/HTTP/… S3 endpoint
  42. 42. S3 Endpoints and Policy 172.31.0.0/18 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.2.12 { "Statement": [ { "Sid": "Access-to-specific-bucket-only", "Principal": "*", "Action": [ "s3:GetObject", "s3:PutObject" ], "Effect": "Allow", "Resource": ["arn:aws:s3:::my_secure_bucket", "arn:aws:s3:::my_secure_bucket/*"] } ] } { "Statement": [ { "Sid": "Access-to-specific-VPC-only", "Principal": "*", "Action": "s3:*", "Effect": "Deny", "Resource": ["arn:aws:s3:::my_secure_bucket", "arn:aws:s3:::my_secure_bucket/*"], "Condition": { "StringNotEquals": { "aws:sourceVpc": "vpc-111bbb22" } } } ] }
  43. 43. Simple Complex Limited Flexible EC2 VPC
  44. 44. 172.31.0.0/18 172.31.1.0/24 172.31.2.0/24 172.31.1.7 172.31.1.8 172.31.1.9 172.31.2.12 172.31.2.51 Default VPC
  45. 45. Simple Complex Limited Flexible EC2 - VPC
  46. 46. Related Sessions NET201 - Creating Your Virtual Data Center: VPC Fundamentals and Connectivity Options NET303 - NextGen Networking: New Capabilities for Amazon’s Virtual Private Cloud NET304 - Moving Mountains: Netflix's Migration into VPC NET402 - Deep Dive: AWS Direct Connect and VPNs NET404 - Making Every Packet Countr
  47. 47. Remember to complete your evaluations!
  48. 48. Thank you!

×