DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013

24,241 views

Published on

It's a rough world out there, filled with mega bot nets that threaten the availability of your web service. How do you keep your service running in the event of a 10,000x increase in traffic? Maximizing service availability under DDoS conditions requires thoughtful service architecture, and at times, fast acting operations teams. This presentation covers best practices for DDoS-resilient services.

Published in: Technology

DDoS Resiliency with Amazon Web Services (SEC305) | AWS re:Invent 2013

  1. 1. DDoS Resilience with Amazon Web Services nated@amazon.com November 14, 2013 © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. Agenda • • • • Anatomy of DDoS Things We Do So You Don’t Have To Designing for Availability Attack Response
  3. 3. DDoS Facts • Yes, DDoS attacks are on the rise and the big ones are getting bigger • …although those attacks average out to ~14Gbps* and target services owners ~1 per year *source: Arbor Networks
  4. 4. DDoS Facts Percentile Max Gbps 10 20 30 40 50 60 70 80 90 95 99 2.39 4.28 6.55 8.27 10.49 11.85 13.97 17.38 25.45 35.74 84.90 Max 299.43 Average Duration (minutes) 5.87 7.68 9.00 10.53 13.23 16.80 23.12 35.87 66.13 141.74 906.80 13.81 *source: Arbor Networks
  5. 5. DDoS Anatomy Application Exhaustion /search.php?expensive-params service attacker
  6. 6. DDoS Anatomy Host Exhaustion attacker service attacker
  7. 7. DDoS Anatomy attacker Traditional Datacenter Exhaustion attacker traditional datacenter transit attacker
  8. 8. DDoS Anatomy attacke r attacke r Intermediary Exhaustion attacke r transit traditional datacenter attacke r transit transit attacke r transit attacke r attacke r
  9. 9. DDoS Anatomy • Large enough attacks consume the capacity of application layer, host, datacenter connectivity, Internet connectivity, or intermediary networks
  10. 10. How can we help you? • Scale and Diversity of AWS • Resilient Service Designs • Business or Enterprise Support
  11. 11. Things We Do So You Don’t Have To
  12. 12. model credit: Scale
  13. 13. Scale traditional datacenter transit
  14. 14. Scale More Bandwidth transit AWS region transit transit
  15. 15. Scale More Compute transit AWS region transit transit
  16. 16. Scale More Points of Presence transit AWS region AWS edge AWS edge transit transit AWS edge
  17. 17. Scale Attack Absorbed attacker transit AWS region AWS edge transit transit attacker AWS edge AWS edge attacker
  18. 18. Diversity transit peer Internet Transit and Peering peer peer peer transit AWS region peer transit
  19. 19. Diversity
  20. 20. Diversity
  21. 21. Diversity Amazon Route 53 Example - Anycast Striping • Leverages Resolver Behavior • Edge Location Diversity • Network Path Diversity
  22. 22. Delegation Set [nated@xyz ~]$ dig NS internetkitties.com ;; QUESTION SECTION: ;internetkitties.com. ;; ANSWER SECTION: internetkitties.com. internetkitties.com. internetkitties.com. internetkitties.com. IN NS 172800 172800 172800 172800 IN IN IN IN NS NS NS NS ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net.
  23. 23. Delegation Set [nated@xyz ~]$ dig NS internetkitties.com ;; QUESTION SECTION: ;internetkitties.com. ;; ANSWER SECTION: internetkitties.com. internetkitties.com. internetkitties.com. internetkitties.com. IN NS 172800 172800 172800 172800 IN IN IN IN NS NS NS NS ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net.
  24. 24. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  25. 25. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  26. 26. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  27. 27. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  28. 28. Edge Location Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  29. 29. Network Path Diversity awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net. [nated@xyz ~]$ traceroute ns-1131.awsdns-13.org. [nated@xyz ~]$ traceroute ns-1751.awsdns-26.co.uk. traceroute to ns-1131.awsdns-13.org (205.251.196.107), 64 hops max, 52 byte packets traceroute to ~]$ traceroute ns-340.awsdns-42.com. 1 (192.168.1.1) 1.748 ms 0.830 ms 0.750 ms [nated@xyz ns-1751.awsdns-26.co.uk (205.251.198.215), 64 hops max, 52 byte packets 1 *traceroute to ns-340.awsdns-42.com (205.251.193.84), 64 hops max, 52 byte packets (192.168.1.1) 1.298 ms 0.755 ms 0.694 ms 2 ** [nated@xyz ~]$ traceroute ns-952.awsdns-55.net. 2 *** 1 (192.168.1.1) 2.444 ms 1.676 (68.85.255.255) 14.634 ms 12.822 ms 10.774 ms 3 cat.seattle.wa.seattle.comcast.net ms 1.028 ms (205.251.195.184), 64 hops max, 52 byte packets traceroute to ns-952.awsdns-55.net 3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 9.254 ms 24.156 ms 19.167 ms 2 *** 4 ae-20-0-ar03.burien.wa.seattle.comcast.net (69.139.164.125) 31.766 ms 13.898 ms 1 (192.168.1.1) 1.352 ms 0.642 ms 0.630 ms 4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 17.281 ms 18.580 ms 17.906 3 cat.seattle.wa.seattle.comcast.net (68.85.255.255) 19.842 ms 23.018 ms 26.469 ms 5 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 20.108 ms 2 *** 5 he-1-5-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.86.94.65) 20.753 ms 29.955 ms 4 ae-20-0-ar03.seattle.wa.seattle.comcast.net (69.139.164.129) 24.366 ms 20.842 ms 3 cat.seattle.wa.seattle.comcast.net (68.86.93.5) 18.781 ms 16.253 6 he-1-7-0-0-11-cr01.seattle.wa.ibone.comcast.net (68.85.255.255)34.612 ms 30.382 ms 17.851 ms 6 ae12.edge2.seattle3.level3.net (4.68.63.65)(68.86.93.173) 30.211 ms ms 17.221 ms 5 he-1-12-0-0-10-cr01.seattle.wa.ibone.comcast.net 38.159 ms 4 be-1-ur08.seattle.wa.seattle.comcast.net (69.139.164.134) ms 13.561 ms 7 ae12.edge2.seattle3.level3.net (4.68.63.65) 34.371 ms 36.504 msms 49.457 ms 49.945 ms 7 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 31.948 ms 29.775 ms 6 ae12.edge2.seattle3.level3.net (4.68.63.65) 33.596 ms 48.510 27.301 5 ae-1-0-ar03.seattle.wa.seattle.comcast.net (68.85.240.94) 21.009 ms ms 8 ae-31-51.ebr1.seattle1.level3.net (4.69.147.150) 48.557 ms 60.610 43.456 ms ms 8 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 45.286167.112 ms 161.82143.219 7 ae-32-52.ebr2.seattle1.level3.net (4.69.147.182) 162.580 ms ms ms 56.751 ms 6 he-1-12-0-0-11-cr01.seattle.wa.ibone.comcast.net 9 ae-7-7.ebr2.sanjose1.level3.net (4.69.132.49) 58.662 ms 46.830 ms 62.458 ms 9 ae-62-62.csw1.sanjose1.level3.net (4.69.153.18) 44.181(68.86.93.177) 17.366 ms 19.162 ms 8 ae-2-2.ebr2.denver1.level3.net (4.69.132.54) 163.723 ms 159.037 ms 174.670 ms ms 7 be-12-pe03.seattle.wa.ibone.comcast.net (68.86.84.106) ae-3-80.edge1.sanjose3.level3.net (4.69.152.144) 46.817 ms 19.949 ms 22.968 ms 24.976 ms 10 10 9 ae-3-3.ebr1.chicago2.level3.net (4.69.132.62) 60.700 ms 47.997 ms 54.477 ms ae-2-2.ebr2.sanjose5.level3.net (4.69.148.141) 169.379 ms 167.307 ms 168.454 ms 8 *** 4.53.208.22 (4.53.208.22) 54.634 ms 60.111 ms 44.187 ms 11 11 10 ae-6-6.ebr1.chicago1.level3.net (4.69.148.201)166.002 ms 168.125 ms 164.232 ms ae-6-6.ebr2.losangeles1.level3.net (4.69.140.189) 55.190 ms 58.829 ms 55.751 ms 9 *** 205.251.229.155 (205.251.229.155) 47.758 ms 12 12 11 ae-2-2.ebr2.newyork2.level3.net (4.69.132.66) 167.861 ms ms ae-92-92.csw4.losangeles1.level3.net (4.69.137.30) 49.261 167.893 ms 160.681 ms 10 * 65-122-235-178.dia.static.qwest.net (65.122.235.178) 40.707 ms 30.916 ms 205.251.230.91 (205.251.230.91) 52.714 ms 43.560 53.091 ms 13 13 12 ae-1-100.ebr1.newyork2.level3.net (4.69.144.139)163.919 ms ms ae-3-80.edge5.losangeles1.level3.net (4.69.135.253) 58.707 ms166.782 ms 161.686 ms 11 14 13 4.69.201.45 (4.69.201.45) 164.023 ms SFO5 205.251.225.22 (205.251.225.22) 85.275 ms 14 amazon.com.edge5.losangeles1.level3.net (205.129.4.26) 46.477 ms 36.525 ms 42.110 ms 12 205.251.225.122 (205.251.225.122) 35.017 160.461 ms 14 ae-42-42.ebr2.london1.level3.net (4.69.137.69) 165.560 msms 38.568 ms 15 LAX3 13 205.251.226.136 (205.251.226.136) 36.560 ms 15 16 17 18 19 ae-46-46.ebr2.amsterdam1.level3.net (4.69.143.73) 165.627 ms 14 SEA50 ae-59-224.csw2.amsterdam1.level3.ne (t4.69.153.214) 172.909 ms 166.052 ms 4.69.162.154 (4.69.162.154) 166.353 ms 212.72.41.162 (212.72.41.162) 171.714 ms 174.033 ms 179.219 ms AMS50
  30. 30. Striping in Action
  31. 31. Striping in Action
  32. 32. Striping in Action awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  33. 33. Striping in Action awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  34. 34. Striping in Action awsdns-13.org. awsdns-26.co.uk. awsdns-42.com. awsdns-55.net.
  35. 35. Diversity client AWS edge AWS edge AWS region AWS edge transit client AWS edge attacker
  36. 36. Diversity • Amazon Route 53 - Anycast Striping • Amazon CloudFront Edge Locations • AWS Regions
  37. 37. How can we help you? • Scale and Diversity of AWS • Resilient Service Designs • Business or Enterprise Support
  38. 38. How can we help you? • Amazon Route 53 and Amazon CloudFront • Resilient Service Designs • Business or Enterprise Support
  39. 39. Designing for Resilience
  40. 40. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  41. 41. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  42. 42. N+1 Failover • Scale Out, Plus Redundancy
  43. 43. N+1 Failover • Scale Out, Plus Redundancy • Failure of 1/100 < Failure of 1/10
  44. 44. N+1 Failover • Scale Out, Plus Redundancy • Failure of 1/100 < Failure of 1/10 • Automatic Failover with Health Checked DNS
  45. 45. N+1 Failover attacker client
  46. 46. N+1 Failover attacker client
  47. 47. N+1 Failover Check out Amazon Route 53 Health Checks
  48. 48. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  49. 49. Resilient Clients • Use multi-record RRSets • Randomize the record on connect retry • Popular HTTP clients already do this!
  50. 50. Resilient Clients [nated@xyz ~]$ dig www.internetkitties.com ;; QUESTION SECTION: ;www.internetkitties.com. IN ;; ANSWER SECTION: www.internetkitties.com. 32 IN d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. d3g5kqnbrlf3fg.cloudfront.net. A CNAME 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A 30 IN A d3g5kqnbrlf3fg.cloudfront.net. 54.230.69.190 54.230.71.141 54.230.71.172 54.230.71.233 54.240.188.66 54.230.68.41 54.230.68.212 54.230.69.141
  51. 51. Resilient Clients Browser Packet Capture Num 4 5 6 7 8 9 11 12 13 14 15 16 Time 2.535515 2.736659 2.93782 3.138996 3.339767 3.540963 3.541123 3.742296 3.824502 3.824515 4.024809 4.225094 Source 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 10.61.60.17 Destination 54.230.69.141 54.230.69.190 54.230.71.141 54.230.71.172 54.230.71.233 54.240.188.66 54.230.68.41 54.230.68.212 54.230.69.190 54.230.69.141 54.230.71.141 54.230.71.172 [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN] [SYN]
  52. 52. Client Retry Behavior, SYN Timeout Browser OS Rotates IPs Time to Rotation Chrome 30.0.1599 Windows 7 Yes 12 Internet Explorer 8 Windows 7 Yes 12 Firefox 25 Windows 7 Yes 20 Safari 5.0.5 Windows 7 Yes 20 Safari 6.0.5 OSX 10.7.5 Yes <1 Firefox 25 OSX 10.7.5 Yes (2) <1 Chrome 32.0.1678 OSX 10.7.5 Yes (2) DNS TTL, or Refresh
  53. 53. Resilient Clients attacker service client
  54. 54. Resilient Clients
  55. 55. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  56. 56. Capped Workloads • Protect Application Layer Capacity • Strive for Sameness • Throttle or Sample Request Workloads
  57. 57. Strive for Sameness Application Exhaustion /search.php?expensive-params service attacker
  58. 58. Strive for Sameness /search.php?expensive-params attacker service Search_Result_Page_1
  59. 59. Capped Workloads AppLayer ~1K to ~10K rps Host/OS ~500K to 5M pps
  60. 60. Capped Workloads AppLayer ~1K to ~10K rps Host/OS ~500K to 5M pps Core DAL Auth Logging
  61. 61. Capped Workloads AppLayer ~1K to ~10K rps Throttle ~10 to ~100K rps Core DAL Auth Logging Host/OS ~500K to 5M pps
  62. 62. Capped Workloads AppLayer ~1K to ~10K rps Throttle ~10 to ~100K rps Host/OS ~500K to 5M pps Core DAL Auth Logging 1,000 samples / sec
  63. 63. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  64. 64. Process Isolation • Isolate application components across processes • Let the OS protect critical resources
  65. 65. Process Isolation Core DAL Auth Logging
  66. 66. Process Isolation Core DAL Auth Logging
  67. 67. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  68. 68. Evolution of Resilience client client
  69. 69. Evolution of Resilience client client
  70. 70. Evolution of Resilience client client
  71. 71. Evolution of Resilience client client
  72. 72. Evolution of Resilience client client
  73. 73. Evolution of Resilience client client
  74. 74. Evolution of Resilience client client
  75. 75. Evolution of Resilience client client
  76. 76. Evolution of Resilience client client
  77. 77. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations
  78. 78. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations • 8 endpoints 2 AZs = 64
  79. 79. N Choose M Isolation • 2 endpoints 2 AZs = 4 permutations • 8 endpoints 2 AZs = 64 • 8 endpoints 3 AZs = 512
  80. 80. Shuffle Sharding – Amazon Route 53 • Define Availability Lattice • Stripes – Edge Location • Braids – Host Isolation • Assign Endpoints to the Lattice • Virtual Name Servers • Allocate Endpoints to Resources • Hosted Zone Delegate Set
  81. 81. Non-Overlapping Delegation Sets ;; QUESTION SECTION: ;gray.internetkitties.com. IN NS ;; QUESTION SECTION: ;orange.internetkitties.org. IN NS ;; ANSWER SECTION: ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net. ;; ANSWER SECTION: ns-1140.awsdns-14.org. ns-1773.awsdns-29.co.uk. ns-290.awsdns-36.com. ns-989.awsdns-59.net.
  82. 82. Shuffle Sharding .com .net .co.uk .org
  83. 83. Shuffle Sharding .com .net .co.uk .org ns-1773.awsdns-29.co.uk. ns-1140.awsdns-14.org.
  84. 84. Shuffle Sharding A B C D .com .net .co.uk .org ns-1773.awsdns-29.co.uk. ns-1140.awsdns-14.org.
  85. 85. Shuffle Sharding A .com .net .co.uk .org B gray.internetkitties.com orange.internetkitties.org C D
  86. 86. Shuffle Sharding A .com .net .co.uk .org B gray.internetkitties.com orange.internetkitties.org C D
  87. 87. Non-Overlapping Delegation Sets ;; QUESTION SECTION: ;gray.internetkitties.com. IN NS ;; QUESTION SECTION: ;orange.internetkitties.org. IN NS ;; ANSWER SECTION: ns-1131.awsdns-13.org. ns-1751.awsdns-26.co.uk. ns-340.awsdns-42.com. ns-952.awsdns-55.net. ;; ANSWER SECTION: ns-1140.awsdns-14.org. ns-1773.awsdns-29.co.uk. ns-290.awsdns-36.com. ns-989.awsdns-59.net.
  88. 88. Shuffle Sharding Resilience attacke r .co.uk A B C D client .org A B C D gray.internetkitties.com orange.internetkitties.org
  89. 89. Shuffle Sharding Resilience attacke r .co.uk A B C D client .org A B C D gray.internetkitties.com orange.internetkitties.org
  90. 90. Shuffle Sharding Toolkit • • • • Define a Lattice of Availability Allocate Service Resources to the Lattice Assign Customers Isolated Resources https://github.com/awslabs/route53-infima
  91. 91. Lattice Configuration // Create a 1-D lattice with "AvailabilityZone” as the dimension OneDimensionalLattice<HealthCheckedRecordSet> myServiceLayout = new OneDimensionalLattice<HealthCheckedRecordSet>("AvailabilityZone”);
  92. 92. Lattice Configuration // Add endpoints in the us-west-1a Availability zone myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.1")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.2")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.3")); … // Add endpoints in the us-west-1b Availability zone myServiceLayout.addEndpoint("us-west-1b” new HealthCheckedRecordSet("192.0.2.11")); …
  93. 93. Lattice Configuration // Add endpoints in the us-west-1a Availability zone myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.1")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.2")); myServiceLayout.addEndpoint("us-west-1a”, new HealthCheckedRecordSet("192.0.2.3")); … // Add endpoints in the us-west-1b Availability zone myServiceLayout.addEndpoint("us-west-1b” new HealthCheckedRecordSet("192.0.2.11")); …
  94. 94. Shuffle Shard // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);
  95. 95. Shuffle Shard // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);
  96. 96. Shuffle Shard // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1);
  97. 97. Vulcanized Lattice // Create a shuffle sharder SimpleSignatureShuffleSharder shuffleSharder = new SimpleSignatureShuffleSharder(5353L); Lattice shard = shuffleSharder.shuffleShard(myServiceLayout, "v123543234", 1); // Create a RubberTree of DNS records Route53RubberTree rubberTree = new Route53RubberTree(”v123543234.video.internetkitties.com", shard); List rrsets = rubberTree.vulcanize();
  98. 98. Lattice Shard RRSet [nated@xyz ~]$ dig v123543234.video.internetkitties.com ;; QUESTION SECTION: ; v123543234.video.internetkitties.com. IN A ;; ANSWER SECTION: v123543234.video.internetkitties.com. 60 IN A v123543234.video.internetkitties.com. 60 IN A v123543234.video.internetkitties.com. 60 IN A 192.0.2.12 192.0.1.45 192.0.3.24 us-west-1b us-west-1a us-west-1c
  99. 99. Designing for Resilience • • • • • N+1 Failover Resilient Clients Capped Workloads Process Isolation Shuffle Sharding
  100. 100. Attack Response
  101. 101. Attack Response • Detection • Src-IP Blocking • Engaging Customer Support
  102. 102. Attack Response • Detection • Src-IP Blocking • Engaging Customer Support
  103. 103. Detect • Traffic Spikes, Drops • CPU Utilization • Network Stats
  104. 104. Detect • Use Resilience Patterns to Access Logs • X-Forwarded-For • Sort and Sum
  105. 105. X-Forwarded-For • Use a trusted load balancer or proxy
  106. 106. X-Forwarded-For • Use a trusted load balancer or proxy • Enable logging
  107. 107. X-Forwarded-For • Use a trusted load balancer or proxy • Enable logging – IIS7 • Install ‘IIS Advanced Logging’ • Configure X-Forwarded-For field
  108. 108. X-Forwarded-For Enable Logging if($http_x_forwarded_for !='-’) { nginx: log_format main '$http_x_forwarded_for - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$remote_addr"'; } else { log_format main '$remote_addr - $remote_user [$time_local] $status ' '"$request" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; }
  109. 109. X-Forwarded-For • Use a trusted load balancer or proxy • Enable X-Forwarded-For logging
  110. 110. Sort & Sum • Used to identify “top talkers” [nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' | sort | uniq -c | tail 2 10.54.4.1 3 10.63.34.1 5 10.23.97.212 1182 10.54.0.183
  111. 111. Sort & Sum • Used to identify “top talkers” [nated@xyz.com ~]$ grep 'expensive-param' ./access.log | awk '{print $1}' | sort | uniq -c | tail 2 10.54.4.1 3 10.63.34.1 5 10.23.97.212 1182 10.54.0.183
  112. 112. Src-IP Blacklisting • • • • Host-Level Firewalling Web-Server Configuration VPC Network ACLs Web Application Firewall
  113. 113. Src-IP Blacklisting • • • • Host-Level Firewalling (IPTables) Web-Server Configuration (Nginx / Apache, IIS) VPC Network ACLs Web Application Firewall
  114. 114. Src-IP Blacklisting • • • • Host-Level Firewalling Web-Server Configuration VPC Network ACLs Web Application Firewall
  115. 115. VPC Network ACLs • Apply to a VPC subnet • Supports DENY rules
  116. 116. VPC Network ACLs • Enter each source IP • Set DENY
  117. 117. Src-IP Blacklisting • Host-Level Firewalling • VPC Network ACLs • Web Application Firewall
  118. 118. Web Application Firewall • • • • Src-IP Blacklist HTTP Headers (X-Forwarded-For) URI-Based Filtering Advanced Throttling
  119. 119. Attack Response • Detection • Src-IP Blocking • Engaging Customer Support
  120. 120. Engaging Customer Support http://aws.amazon.com/premiumsupport/
  121. 121. Summary How can we help? Resilient Design • Scale and Diversity • Route 53 and CloudFront • Business and Enterprise Support • • • • • • Attack Response • • • • Enable X-Forwarded-For Logging Detect, Sum and Sort Src-IP Blacklist Engage Customer Support Availability Lattice Shuffle Sharding N+1 Failover Resilient Clients Capped Workloads Process Isolation
  122. 122. Summary How can we help? Resilient Design • Scale and Diversity • Route 53 and CloudFront • Business and Enterprise Support • • • • • • Attack Response • • • • Enable X-Forwarded-For Logging Detect, Sum and Sort Src-IP Blacklist Engage Customer Support Availability Lattice Shuffle Sharding N+1 Failover Resilient Clients Capped Workloads Process Isolation
  123. 123. Please give us your feedback on this presentation SEC305 As a thank you, we will select prize winners daily for completed surveys!

×