DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013

1. DDoS Resilience with Amazon Web Services nated@amazon.com November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. Agenda • • • • Anatomy of DDoS Things We Do So You Don’t Have To Designing for Availability Attack Response

3. DDoS Facts • Yes, DDoS attacks are on the rise and the big ones are getting bigger • …although those attacks average out to ~14Gbps* and target services owners ~1 per year *source: Arbor Networks

4. DDoS Facts Percentile Max Gbps 10 20 30 40 50 60 70 80 90 95 99 2.39 4.28 6.55 8.27 10.49 11.85 13.97 17.38 25.45 35.74 84.90 Max 299.43 Average Duration (minutes) 5.87 7.68 9.00 10.53 13.23 16.80 23.12 35.87 66.13 141.74 906.80 13.81 *source: Arbor Networks

5. DDoS Anatomy Application Exhaustion /search.php?expensive-params service attacker

6. DDoS Anatomy Host Exhaustion attacker service attacker

7. DDoS Anatomy attacker Traditional Datacenter Exhaustion attacker traditional datacenter transit attacker

8. DDoS Anatomy attacke r attacke r Intermediary Exhaustion attacke r transit traditional datacenter attacke r transit transit attacke r transit attacke r attacke r

9. DDoS Anatomy • Large enough attacks consume the capacity of application layer, host, datacenter connectivity, Internet connectivity, or intermediary networks

10. How can we help you? • Scale and Diversity of AWS • Resilient Service Designs • Business or Enterprise Support

11. Things We Do So You Don’t Have To

12. model credit: Scale

13. Scale traditional datacenter transit

14. Scale More Bandwidth transit AWS region transit transit

15. Scale More Compute transit AWS region transit transit

16. Scale More Points of Presence transit AWS region AWS edge AWS edge transit transit AWS edge

17. Scale Attack Absorbed attacker transit AWS region AWS edge transit transit attacker AWS edge AWS edge attacker

18. Diversity transit peer Internet Transit and Peering peer peer peer transit AWS region peer transit

19. Diversity

20. Diversity

21. Diversity Amazon Route 53 Example - Anycast Striping • Leverages Resolver Behavior • Edge Location Diversity • Network Path Diversity

22. Delegation Set [nated@xyz ~]$ dig NS internetkitties.com ;; QUESTION SECTION: ;internetkitties.com. ;; ANSWER SECTION: internetkitties.com. internetkitties.com. internetkitties.com. internetkitties.com. IN NS 172800 172800 172800 172800 IN IN IN IN NS NS NS NS ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net.

23. Delegation Set [nated@xyz ~]$ dig NS internetkitties.com ;; QUESTION SECTION: ;internetkitties.com. ;; ANSWER SECTION: internetkitties.com. internetkitties.com. internetkitties.com. internetkitties.com. IN NS 172800 172800 172800 172800 IN IN IN IN NS NS NS NS ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net.

24. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.

29. Network Path Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net. [nated@xyz ~]$ traceroute ns-1131.awsdns-13.org. [nated@xyz ~]$ traceroute ns-1751.awsdns-26.co.uk. traceroute to ns-1131.awsdns-13.org (205.251.196.107), 64 hops max, 52 byte packets traceroute to ~]$ traceroute ns-340.awsdns-42.com. 1 (192.168.1.1) 1.748 ms 0.830 ms 0.750 ms [nated@xyz ns-1751.awsdns-26.co.uk (205.251.198.215), 64 hops max, 52 byte packets 1 *traceroute to ns-340.awsdns-42.com (205.251.193.84), 64 hops max, 52 byte packets (192.168.1.1) 1.298 ms 0.755 ms 0.694 ms 2 ** [nated@xyz ~]$ traceroute ns-952.awsdns-55.net. 2 *** 1 (192.168.1.1) 2.444 ms 1.676 (68.85.255.255) 14.634 ms 12.822 ms 10.774 ms 3 cat.seattle.wa.seattle.comcast.net ms 1.028 ms (205.251.195.184), 64 hops max, 52 byte packets traceroute to ns-952.awsdns-55.net 3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 9.254 ms 24.156 ms 19.167 ms 2 *** 4 ae-20-0-ar03.burien.wa.seattle.comcast.net (69.139.164.125) 31.766 ms 13.898 ms 1 (192.168.1.1) 1.352 ms 0.642 ms 0.630 ms 4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 17.281 ms 18.580 ms 17.906 3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 19.842 ms 23.018 ms 26.469 ms 5 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 20.108 ms 2 *** 5 he-1-5-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.94.65) 20.753 ms 29.955 ms 4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 24.366 ms 20.842 ms 3 cat.seattle.wa.seattle.comcast.net (68.86.93.5) 18.781 ms 16.253 6 he-1-7-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.85.255.255)34.612 ms 30.382 ms 17.851 ms 6 ae12.edge2.seattle3.level3.net (4.68.63.65)(68.86.93.173) 30.211 ms ms 17.221 ms 5 he-1-12-0-0-10-cr01.seattle.wa.ibone.comcast.net 38.159 ms 4 be-1-ur08.seattle.wa.seattle.comcast.net (69.139.164.134) ms 13.561 ms 7 ae12.edge2.seattle3.level3.net (4.68.63.65) 34.371 ms 36.504 msms 49.457 ms 49.945 ms 7 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 31.948 ms 29.775 ms 6 ae12.edge2.seattle3.level3.net (4.68.63.65) 33.596 ms 48.510 27.301 5 ae-1-0-ar03.seattle.wa.seattle.comcast.net (68.85.240.94) 21.009 ms ms 8 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 48.557 ms 60.610 43.456 ms ms 8 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 45.286167.112 ms 161.82143.219 7 ae-32-52.ebr2.seattle1.level3.net (4.69.147.182) 162.580 ms ms ms 56.751 ms 6 he-1-12-0-0-11-cr01.seattle.wa.ibone.comcast.net 9 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 58.662 ms 46.830 ms 62.458 ms 9 ae-62-62.csw1.sanjose1.level3.net (4.69.153.18) 44.181(68.86.93.177) 17.366 ms 19.162 ms 8 ae-2-2.ebr2.denver1.level3.net (4.69.132.54) 163.723 ms 159.037 ms 174.670 ms ms 7 be-12-pe03.seattle.wa.ibone.comcast.net (68.86.84.106) ae-3-80.edge1.sanjose3.level3.net (4.69.152.144) 46.817 ms 19.949 ms 22.968 ms 24.976 ms 10 10 9 ae-3-3.ebr1.chicago2.level3.net (4.69.132.62) 60.700 ms 47.997 ms 54.477 ms ae-2-2.ebr2.sanjose5.level3.net (4.69.148.141) 169.379 ms 167.307 ms 168.454 ms 8 *** 4.53.208.22 (4.53.208.22) 54.634 ms 60.111 ms 44.187 ms 11 11 10 ae-6-6.ebr1.chicago1.level3.net (4.69.148.201)166.002 ms 168.125 ms 164.232 ms ae-6-6.ebr2.losangeles1.level3.net (4.69.140.189) 55.190 ms 58.829 ms 55.751 ms 9 *** 205.251.229.155 (205.251.229.155) 47.758 ms 12 12 11 ae-2-2.ebr2.newyork2.level3.net (4.69.132.66) 167.861 ms ms ae-92-92.csw4.losangeles1.level3.net (4.69.137.30) 49.261 167.893 ms 160.681 ms 10 * 65-122-235-178.dia.static.qwest.net (65.122.235.178) 40.707 ms 30.916 ms 205.251.230.91 (205.251.230.91) 52.714 ms 43.560 53.091 ms 13 13 12 ae-1-100.ebr1.newyork2.level3.net (4.69.144.139)163.919 ms ms ae-3-80.edge5.losangeles1.level3.net (4.69.135.253) 58.707 ms166.782 ms 161.686 ms 11 14 13 4.69.201.45 (4.69.201.45) 164.023 ms SFO5 205.251.225.22 (205.251.225.22) 85.275 ms 14 amazon.com.edge5.losangeles1.level3.net (205.129.4.26) 46.477 ms 36.525 ms 42.110 ms 12 205.251.225.122 (205.251.225.122) 35.017 160.461 ms 14 ae-42-42.ebr2.london1.level3.net (4.69.137.69) 165.560 msms 38.568 ms 15 LAX3 13 205.251.226.136 (205.251.226.136) 36.560 ms 15 16 17 18 19 ae-46-46.ebr2.amsterdam1.level3.net (4.69.143.73) 165.627 ms 14 SEA50 ae-59-224.csw2.amsterdam1.level3.ne (t4.69.153.214) 172.909 ms 166.052 ms 4.69.162.154 (4.69.162.154) 166.353 ms 212.72.41.162 (212.72.41.162) 171.714 ms 174.033 ms 179.219 ms AMS50

30. Striping in Action

31. Striping in Action

32. Striping in Action awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.

35. Diversity client AWS edge AWS edge AWS region AWS edge transit client AWS edge attacker

36. Diversity • Amazon Route 53 - Anycast Striping • Amazon CloudFront Edge Locations • AWS Regions

37. How can we help you? • Scale and Diversity of AWS • Resilient Service Designs • Business or Enterprise Support

38. How can we help you? • Amazon Route 53 and Amazon CloudFront • Resilient Service Designs • Business or Enterprise Support

39. Designing for Resilience

40. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding

42. N+1 Failover • Scale Out, Plus Redundancy

43. N+1 Failover • Scale Out, Plus Redundancy • Failure of 1/100 < Failure of 1/10

44. N+1 Failover • Scale Out, Plus Redundancy • Failure of 1/100 < Failure of 1/10 • Automatic Failover with Health Checked DNS

45. N+1 Failover attacker client

46. N+1 Failover attacker client

47. N+1 Failover Check out Amazon Route 53 Health Checks

49. Resilient Clients • Use multi-record RRSets • Randomize the record on connect retry • Popular HTTP clients already do this!

50. Resilient Clients [nated@xyz ~]$ dig www.internetkitties.com ;; QUESTION SECTION: ;www.internetkitties.com. IN ;; ANSWER SECTION: www.internetkitties.com. 32 IN d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. A CNAME 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A d3g5kqnbrlf3fg.cloudfront.net. 54.230.69.190 54.230.71.141 54.230.71.172 54.230.71.233 54.240.188.66 54.230.68.41 54.230.68.212 54.230.69.141

51. Resilient Clients Browser Packet Capture Num 4 5 6 7 8 9 11 12 13 14 15 16 Time 2.535515 2.736659 2.93782 3.138996 3.339767 3.540963 3.541123 3.742296 3.824502 3.824515 4.024809 4.225094 Source 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 Destination 54.230.69.141 54.230.69.190 54.230.71.141 54.230.71.172 54.230.71.233 54.240.188.66 54.230.68.41 54.230.68.212 54.230.69.190 54.230.69.141 54.230.71.141 54.230.71.172 [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN]

52. Client Retry Behavior, SYN Timeout Browser OS Rotates IPs Time to Rotation Chrome 30.0.1599 Windows 7 Yes 12 Internet Explorer 8 Windows 7 Yes 12 Firefox 25 Windows 7 Yes 20 Safari 5.0.5 Windows 7 Yes 20 Safari 6.0.5 OSX 10.7.5 Yes <1 Firefox 25 OSX 10.7.5 Yes (2) <1 Chrome 32.0.1678 OSX 10.7.5 Yes (2) DNS TTL, or Refresh

53. Resilient Clients attacker service client

54. Resilient Clients

56. Capped Workloads • Protect Application Layer Capacity • Strive for Sameness • Throttle or Sample Request Workloads

57. Strive for Sameness Application Exhaustion /search.php?expensive-params service attacker

58. Strive for Sameness /search.php?expensive-params attacker service Search_Result_Page_1

59. Capped Workloads AppLayer ~1K to ~10K rps Host/OS ~500K to 5M pps

60. Capped Workloads AppLayer ~1K to ~10K rps Host/OS ~500K to 5M pps Core DAL Auth Logging

61. Capped Workloads AppLayer ~1K to ~10K rps Throttle ~10 to ~100K rps Core DAL Auth Logging Host/OS ~500K to 5M pps

62. Capped Workloads AppLayer ~1K to ~10K rps Throttle ~10 to ~100K rps Host/OS ~500K to 5M pps Core DAL Auth Logging 1,000 samples / sec

64. Process Isolation • Isolate application components across processes • Let the OS protect critical resources

65. Process Isolation Core DAL Auth Logging

66. Process Isolation Core DAL Auth Logging

68. Evolution of Resilience client client

77. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations

78. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations • 8 endpoints 2 AZs = 64

79. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations • 8 endpoints 2 AZs = 64 • 8 endpoints 3 AZs = 512

80. Shuffle Sharding – Amazon Route 53 • Define Availability Lattice • Stripes – Edge Location • Braids – Host Isolation • Assign Endpoints to the Lattice • Virtual Name Servers • Allocate Endpoints to Resources • Hosted Zone Delegate Set

81. Non-Overlapping Delegation Sets ;; QUESTION SECTION: ;gray.internetkitties.com. IN NS ;; QUESTION SECTION: ;orange.internetkitties.org. IN NS ;; ANSWER SECTION: ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net. ;; ANSWER SECTION: ns-1140.awsdns-14.org. ns-1773.awsdns-29.co.uk. ns-290.awsdns-36.com. ns-989.awsdns-59.net.

82. Shuffle Sharding .com .net .co.uk .org

83. Shuffle Sharding .com .net .co.uk .org ns-1773.awsdns-29.co.uk. ns-1140.awsdns-14.org.

84. Shuffle Sharding A B C D .com .net .co.uk .org ns-1773.awsdns-29.co.uk. ns-1140.awsdns-14.org.

85. Shuffle Sharding A .com .net .co.uk .org B gray.internetkitties.com orange.internetkitties.org C D

86. Shuffle Sharding A .com .net .co.uk .org B gray.internetkitties.com orange.internetkitties.org C D

87. Non-Overlapping Delegation Sets ;; QUESTION SECTION: ;gray.internetkitties.com. IN NS ;; QUESTION SECTION: ;orange.internetkitties.org. IN NS ;; ANSWER SECTION: ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net. ;; ANSWER SECTION: ns-1140.awsdns-14.org. ns-1773.awsdns-29.co.uk. ns-290.awsdns-36.com. ns-989.awsdns-59.net.

88. Shuffle Sharding Resilience attacke r .co.uk A B C D client .org A B C D gray.internetkitties.com orange.internetkitties.org

89. Shuffle Sharding Resilience attacke r .co.uk A B C D client .org A B C D gray.internetkitties.com orange.internetkitties.org

90. Shuffle Sharding Toolkit • • • • Define a Lattice of Availability Allocate Service Resources to the Lattice Assign Customers Isolated Resources https://github.com/awslabs/route53-infima

91. Lattice Configuration // Create a 1-D lattice with "AvailabilityZone” as the dimension OneDimensionalLattice<HealthCheckedRecordSet> myServiceLayout = new OneDimensionalLattice<HealthCheckedRecordSet>("AvailabilityZone”);

92. Lattice Configuration // Add endpoints in the us-west-1a Availability zone myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.1")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.2")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.3")); … // Add endpoints in the us-west-1b Availability zone myServiceLayout.addEndpoint("us-west-1b” new HealthCheckedRecordSet("192.0.2.11")); …

93. Lattice Configuration // Add endpoints in the us-west-1a Availability zone myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.1")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.2")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.3")); … // Add endpoints in the us-west-1b Availability zone myServiceLayout.addEndpoint("us-west-1b” new HealthCheckedRecordSet("192.0.2.11")); …

94. Shuffle Shard // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);

97. Vulcanized Lattice // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1); // Create a RubberTree of DNS records Route53RubberTree rubberTree = new Route53RubberTree(”v123543234.video.internetkitties.com", shard); List rrsets = rubberTree.vulcanize();

98. Lattice Shard RRSet [nated@xyz ~]$ dig v123543234.video.internetkitties.com ;; QUESTION SECTION: ; v123543234.video.internetkitties.com. IN A ;; ANSWER SECTION: v123543234.video.internetkitties.com. 60 IN A v123543234.video.internetkitties.com. 60 IN A v123543234.video.internetkitties.com. 60 IN A 192.0.2.12 192.0.1.45 192.0.3.24 us-west-1b us-west-1a us-west-1c

100. Attack Response

101. Attack Response • Detection • Src-IP Blocking • Engaging Customer Support

103. Detect • Traffic Spikes, Drops • CPU Utilization • Network Stats

104. Detect • Use Resilience Patterns to Access Logs • X-Forwarded-For • Sort and Sum

105. X-Forwarded-For • Use a trusted load balancer or proxy

106. X-Forwarded-For • Use a trusted load balancer or proxy • Enable logging

107. X-Forwarded-For • Use a trusted load balancer or proxy • Enable logging – IIS7 • Install ‘IIS Advanced Logging’ • Configure X-Forwarded-For field

108. X-Forwarded-For Enable Logging if($http_x_forwarded_for !='-’) { nginx: log_format main '$http_x_forwarded_for - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$remote_addr"'; } else { log_format main '$remote_addr - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; }

109. X-Forwarded-For • Use a trusted load balancer or proxy • Enable X-Forwarded-For logging

110. Sort & Sum • Used to identify “top talkers” [nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' | sort | uniq -c | tail 2 10.54.4.1 3 10.63.34.1 5 10.23.97.212 1182 10.54.0.183

111. Sort & Sum • Used to identify “top talkers” [nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' | sort | uniq -c | tail 2 10.54.4.1 3 10.63.34.1 5 10.23.97.212 1182 10.54.0.183

112. Src-IP Blacklisting • • • • Host-Level Firewalling Web-Server Configuration VPC Network ACLs Web Application Firewall

113. Src-IP Blacklisting • • • • Host-Level Firewalling (IPTables) Web-Server Configuration (Nginx / Apache, IIS) VPC Network ACLs Web Application Firewall

114. Src-IP Blacklisting • • • • Host-Level Firewalling Web-Server Configuration VPC Network ACLs Web Application Firewall

115. VPC Network ACLs • Apply to a VPC subnet • Supports DENY rules

116. VPC Network ACLs • Enter each source IP • Set DENY

117. Src-IP Blacklisting • Host-Level Firewalling • VPC Network ACLs • Web Application Firewall

118. Web Application Firewall • • • • Src-IP Blacklist HTTP Headers (X-Forwarded-For) URI-Based Filtering Advanced Throttling

120. Engaging Customer Support http://aws.amazon.com/premiumsupport/

121. Summary How can we help? Resilient Design • Scale and Diversity • Route 53 and CloudFront • Business and Enterprise Support • • • • • • Attack Response • • • • Enable X-Forwarded-For Logging Detect, Sum and Sort Src-IP Blacklist Engage Customer Support Availability Lattice Shuffle Sharding N+1 Failover Resilient Clients Capped Workloads Process Isolation

122. Summary How can we help? Resilient Design • Scale and Diversity • Route 53 and CloudFront • Business and Enterprise Support • • • • • • Attack Response • • • • Enable X-Forwarded-For Logging Detect, Sum and Sort Src-IP Blacklist Engage Customer Support Availability Lattice Shuffle Sharding N+1 Failover Resilient Clients Capped Workloads Process Isolation

123. Please give us your feedback on this presentation SEC305 As a thank you, we will select prize winners daily for completed surveys!