Your SlideShare is downloading. ×
0
Scalability and Availability
in the Real World
Cupertino, CA – October 2, 2013
Cory von Wallenstein
Chief Technologist,
Dy...
Pg. 2 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
What do we care about?
• Achieving high(er...
Pg. 3 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
How can we do it?
• Know Thy Enemy: DDoS
•...
4 | © 2013 Infoblox Inc. All Rights Reserved.4 | © 2013 Infoblox Inc. All Rights Reserved.
Cricket Liu
DNS-based DDoS Atta...
5 | © 2013 Infoblox Inc. All Rights Reserved.5 | © 2013 Infoblox Inc. All Rights Reserved.
What You’ll Learn (or Your Mone...
6 | © 2013 Infoblox Inc. All Rights Reserved.6 | © 2013 Infoblox Inc. All Rights Reserved.
DDoS and DNS
• DDoS attacks are...
7 | © 2013 Infoblox Inc. All Rights Reserved.7 | © 2013 Infoblox Inc. All Rights Reserved.
DDoS Attacks Target Name Server...
8 | © 2013 Infoblox Inc. All Rights Reserved.8 | © 2013 Infoblox Inc. All Rights Reserved.
And DDoS Attacks Use Name Serve...
9 | © 2013 Infoblox Inc. All Rights Reserved.9 | © 2013 Infoblox Inc. All Rights Reserved.
DDoS Illustrated
Open recursive...
10 | © 2013 Infoblox Inc. All Rights Reserved.10 | © 2013 Infoblox Inc. All Rights Reserved.
% dig any isc.org. +dnssec
; ...
11 | © 2013 Infoblox Inc. All Rights Reserved.11 | © 2013 Infoblox Inc. All Rights Reserved.
A Little Math
• Say each bot ...
12 | © 2013 Infoblox Inc. All Rights Reserved.12 | © 2013 Infoblox Inc. All Rights Reserved.
The Scourge of the Open Recur...
13 | © 2013 Infoblox Inc. All Rights Reserved.13 | © 2013 Infoblox Inc. All Rights Reserved.
Why Should I Worry?
• More ba...
14 | © 2013 Infoblox Inc. All Rights Reserved.14 | © 2013 Infoblox Inc. All Rights Reserved.
What Can I Do to Protect Myse...
15 | © 2013 Infoblox Inc. All Rights Reserved.15 | © 2013 Infoblox Inc. All Rights Reserved.
Overprovision
• (Yes, I know,...
16 | © 2013 Infoblox Inc. All Rights Reserved.16 | © 2013 Infoblox Inc. All Rights Reserved.
Anycast
• Anycast allows mult...
17 | © 2013 Infoblox Inc. All Rights Reserved.17 | © 2013 Infoblox Inc. All Rights Reserved.
Anycast in Action
Router 2
Ro...
18 | © 2013 Infoblox Inc. All Rights Reserved.18 | © 2013 Infoblox Inc. All Rights Reserved.
Anycast in Action
Router 2
Ro...
19 | © 2013 Infoblox Inc. All Rights Reserved.19 | © 2013 Infoblox Inc. All Rights Reserved.
How Does Anycast Address DDoS...
20 | © 2013 Infoblox Inc. All Rights Reserved.20 | © 2013 Infoblox Inc. All Rights Reserved.
Screen Traffic to Your Name S...
21 | © 2013 Infoblox Inc. All Rights Reserved.21 | © 2013 Infoblox Inc. All Rights Reserved.
Monitor Traffic to Your Name ...
22 | © 2013 Infoblox Inc. All Rights Reserved.22 | © 2013 Infoblox Inc. All Rights Reserved.
Monitoring Aggregate Query Ra...
23 | © 2013 Infoblox Inc. All Rights Reserved.23 | © 2013 Infoblox Inc. All Rights Reserved.
Setting an Alert on Aggregate...
24 | © 2013 Infoblox Inc. All Rights Reserved.24 | © 2013 Infoblox Inc. All Rights Reserved.
Monitoring Top Clients
25 | © 2013 Infoblox Inc. All Rights Reserved.25 | © 2013 Infoblox Inc. All Rights Reserved.
Don’t Be a Part of the Proble...
26 | © 2013 Infoblox Inc. All Rights Reserved.26 | © 2013 Infoblox Inc. All Rights Reserved.
Rate-limit Traffic from Your ...
27 | © 2013 Infoblox Inc. All Rights Reserved.27 | © 2013 Infoblox Inc. All Rights Reserved.
How RRL Works
isc.org/ANY
[33...
28 | © 2013 Infoblox Inc. All Rights Reserved.28 | © 2013 Infoblox Inc. All Rights Reserved.
Thank you!
28
Pg. 29 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Going from one datacenter to five:
The Io...
30© 2012 iovation Inc.
What iovation Does
Recognize devices connecting to
websites
Understands how these devices are relat...
31© 2012 iovation Inc.
Driving Factors
•



•


•

32© 2012 iovation Inc.
Design Criteria
•

•



•

33© 2012 iovation Inc.
Service Oriented Architecture
Consumer Facing
Subscriber Facing
Internal Service
Real-time
Asynchro...
34© 2012 iovation Inc.
Datacenter Types
•


•


•


35© 2012 iovation Inc.
Network Design
BB1
AMS
MIA
10g
20g
10g
Internet
consumers
Content
Delivery
#3
Data StoragePDX
MIADa...
36© 2012 iovation Inc.
Portland to Seattle
http://www.zayo.com/sites/default/files/images/Zayo-US-Network-EXTERNAL-11-1-20...
Pg. 37 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
How does Iovation use Dyn?
• API Interfac...
Pg. 38 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
• DNS cache
poisoning, DNSSEC and
general...
Pg. 39 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Anatomy of a HTTP
connection
Pg. 40 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
How does DNS load
balancing work?
Pg. 41 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Simple active/passive example
• Primary l...
Pg. 42 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 43 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 44 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 45 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Active/Active Load Balancing
(Global Serv...
Pg. 46 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Traffic management with
Dyn Traffic Direc...
Pg. 47 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 48 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Dyn Traffic Director
• Fast Anycast netwo...
Pg. 49 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 50 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 51 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Favor performance over network topology?
...
Pg. 52 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 53 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 54 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 55 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Favor geopolitical boundaries above all?
...
Pg. 56 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 57 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Traffic Management Recap
• Active/Passive...
Pg. 58 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Pg. 59 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
Dyn Delivers Internet Performance
•Traffi...
Pg. 60 Presentation Title (edit from Slide Master) @twitterhandle
Scalability and Availability
in the Real World
Cory von ...
Upcoming SlideShare
Loading in...5
×

Dyn Roadshow: Cricket Liu & Cory von Wallenstein On Scalability & Availability

596

Published on

From the recent Dyn roadshow event in Cupertino, CA, here is the deck put together by Dyn Chief Technologist Cory von Wallenstein and Infoblox Chief Infrastructure Officer Cricket Liu.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
596
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Dyn Roadshow: Cricket Liu & Cory von Wallenstein On Scalability & Availability"

  1. 1. Scalability and Availability in the Real World Cupertino, CA – October 2, 2013 Cory von Wallenstein Chief Technologist, Dyn Inc. cvw@dyn.com @cvwdyn Cricket Liu, Chief Infrastructure Officer, Infoblox cricket@infoblox.com @cricketondns
  2. 2. Pg. 2 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns What do we care about? • Achieving high(er) availability • Resilience in disaster (DDoS) • Flexibility to change infrastructure without downtime • Ability to expand infrastructure beyond current 4 walls • And of course, performance!
  3. 3. Pg. 3 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns How can we do it? • Know Thy Enemy: DDoS • Understanding DNS-based DDoS, and what you can do • The Iovation Technical Story • Going from one datacenter to five • How Dyn Helps • Anycast DNS and DDoS resilience • Global load balancing & traffic management
  4. 4. 4 | © 2013 Infoblox Inc. All Rights Reserved.4 | © 2013 Infoblox Inc. All Rights Reserved. Cricket Liu DNS-based DDoS Attacks 4
  5. 5. 5 | © 2013 Infoblox Inc. All Rights Reserved.5 | © 2013 Infoblox Inc. All Rights Reserved. What You’ll Learn (or Your Money Back!) • What is a DNS-based DDoS Attack? • Why should I worry? • What should I worry about? • How can I defend myself?
  6. 6. 6 | © 2013 Infoblox Inc. All Rights Reserved.6 | © 2013 Infoblox Inc. All Rights Reserved. DDoS and DNS • DDoS attacks are twice the threat to DNS ̶ DDoS attacks target name servers ̶ DDoS attacks use name servers
  7. 7. 7 | © 2013 Infoblox Inc. All Rights Reserved.7 | © 2013 Infoblox Inc. All Rights Reserved. DDoS Attacks Target Name Servers • Authoritative name servers are obviously a critical resource ̶ Without them, your customers can’t get to your web site, send you email • Authoritative name servers are easy to find –dig ns company.example. • Recent attack against a Prolexic customer: 167 Gbps
  8. 8. 8 | © 2013 Infoblox Inc. All Rights Reserved.8 | © 2013 Infoblox Inc. All Rights Reserved. And DDoS Attacks Use Name Servers • Why? ̶ Because name servers make surprisingly good amplifiers This one goes to eleven…
  9. 9. 9 | © 2013 Infoblox Inc. All Rights Reserved.9 | © 2013 Infoblox Inc. All Rights Reserved. DDoS Illustrated Open recursive name servers Evil resolver Target Response to spoofed address Spoofed query
  10. 10. 10 | © 2013 Infoblox Inc. All Rights Reserved.10 | © 2013 Infoblox Inc. All Rights Reserved. % dig any isc.org. +dnssec ; <<>> DiG 9.8.3-P1 <<>> any isc.org. +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57121 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 29, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4096 ;; QUESTION SECTION: ;isc.org. IN ANY ;; ANSWER SECTION: isc.org. 7200 IN RRSIG SPF 5 2 7200 20130719232951 20130619232951 50012 isc.org. Q8n5F9ZucnRaYw762EghVeq9NLLFN4tuAvJZTue/spQJUnRKcM5WuwR4 F8FuEh55EbIs5YxnrG2LbDmEJDOBh0aER+lE6Ts8TdCyZoTVylSf0kmr tmzf0r80Q5xBOdPMfsSARNxWrFDQr03r69IU0Lsp4EbneiM6wIiI7oyJ bz0= isc.org. 7200 IN SPF "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all" isc.org. 3600 IN RRSIG NSEC 5 2 3600 20130719232951 20130619232951 50012 isc.org. r9HtzBqbh52z37xEleIZfNY6gK7SU/6BvlQiSmv8d5bGjyW21vW1zT4N +nUXcd2TJCIJqYRMveZttOom4PgR/6HNq06vS67fn+9YlB/PtWbtvoh/ X1fAU107U+5u7s5EATiGKLcY/7hxPT6UcJd7RvInCyG8BrnxegilRqxG qq4= isc.org. 3600 IN NSEC _adsp._domainkey.isc.org. A NS SOA MX TXT AAAA NAPTR RRSIG NSEC DNSKEY SPF isc.org. 7200 IN RRSIG NAPTR 5 2 7200 20130719232951 20130619232951 50012 isc.org. Sv3chyUtJk8h6G4x/GXAtnV/owBxIsnRKV+FFJBdAyI0BJjwaIW8lCVE 5ntEfn/CbuyAj/nhEUZ7pwhIAKiY8sApkNwnRAlUFB2kJDxKZwyQ2F5R Bas4BbauN/yIyrEeQupIafsc88B7Hy3dl2GJKifPxocUJDvGQTWm8tsU mWk= isc.org. 7200 IN NAPTR 20 0 "S" "SIP+D2U" "" _sip._udp.isc.org. isc.org. 60 IN RRSIG AAAA 5 2 60 20130719232951 20130619232951 50012 isc.org. jPhFaMBwgJckbh6F27bYrr+28xUvurUlE0g75EsRLpAZ55b3di0F3jdX fmpBd/1YoBOR7UcOdSg7Uq596kewRFOeNGILPJW8V69Hb1CLL/JLnUOp x5hX8y9mduN0INm3wvyImH/GB6NHJ0/RKkEh3hHFVXgXTl9z284HHrkH Ba0= isc.org. 60 IN AAAA 2001:4f8:0:2::69 isc.org. 7200 IN RRSIG TXT 5 2 7200 20130719232951 20130619232951 50012 isc.org. Bl7lhqWAPJcSB6lFlITQ8AB74bxxHJ6Pm02pKh9dtDvOQn/0FFPT6Y5U YsqQCbyfZZPH8cVEH5+VFrE76cWH4WoOzz7urd9DrjGh+o41pu2ersPn C8dp7cY81O6s9v66y8pb0CISYDAAhzdIi5Sasx4nKtPXZXlXjWJTWRZj 5r0= isc.org. 7200 IN TXT "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all" isc.org. 7200 IN TXT "$Id: isc.org,v 1.1824 2013-06-18 00:33:44 bind Exp $" isc.org. 7200 IN RRSIG MX 5 2 7200 20130719232951 20130619232951 50012 isc.org. PQrHeMs1C/vuOeklOHA92Ls1mzRgJrE7SY8Yg2gk5IGylMmHKH68gaaa rhdhG6tpmA8X20mMUwFP2YIPkMEecjRatDoSzKctH1YaPwRJJl3QLG4z Fiy0NSGS/qaHHCdoMiQ64KNm88p3xK2vsFa27WyI8pjSXqWOcbvPLvW8 FqU= isc.org. 7200 IN MX 10 mx.pao1.isc.org. isc.org. 60 IN RRSIG A 5 2 60 20130719232951 20130619232951 50012 isc.org. Tu753SDWWqAlfFQSzqJ0vEFF9cweMkvHC2MSK7VU1pntWcdUngwXBgLO DtPJLds7nZ1eUyVgos+WlsWtENw8PMRrYuNlwRxW9PRWpT8jIZTN1ieh fDOu/Y9JNeCqauE54eMfMluc+GH3R2Lh06513yaZB0G/Zn4dSJF6E0XI rCs= isc.org. 60 IN A 149.20.64.69 isc.org. 7200 IN RRSIG SOA 5 2 7200 20130719232951 20130619232951 50012 isc.org. UtICg7Is/C+8NHjYoN79iuI+tgc/Wn1AaqTBkcdGn2NY6XL5KEY5iwdo TZiN8VHyLObwBtwyn3W4tMRH0ETEf6SzSETnvFYf6NbRkrz4snIcvBIa Vj9HhJ3UwYqfOpJCA7EWxb1QvvVhdDYidm60WBEiohMDoVHZdJ5Ol4DK VHo= isc.org. 7200 IN SOA ns-int.isc.org. hostmaster.isc.org. 2013062000 7200 3600 24796800 3600 isc.org. 83390 IN RRSIG DS 7 2 86400 20130708155016 20130617145016 2373 org. OFtHIU34tL9lYvSoe7uLlQCyvHOrY5ldFbK+WM48av0FScRCqEWyjXYg 0vEpojvzR6CPxJ/Lh41HFlCb3ZevRn8ETykiNEgGwViFznPhBsrz0gdT ONmJMHAQgmVt8Lar0GwsjjjI2J6k5gCTwzQyZjkI31V6RiRNoKe8M1iA k3w= isc.org. 83390 IN DS 12892 5 1 982113D08B4C6A1D9F6AEE1E2237AEF69F3F9759 isc.org. 83390 IN DS 12892 5 2 F1E184C0E1D615D20EB3C223ACED3B03C773DD952D5F0EB5C777586D E18DA6B5 isc.org. 7200 IN RRSIG DNSKEY 5 2 7200 20130719230127 20130619230127 12892 isc.org. RMMZLopr6bX0u3MureNVdNPGUjtv1V6fFxyXVAlD9EOLRz9ND0fFoKr4 YnX2W70i2llvlg1uA0vMUUeUKaEM8RtR5olCChNBSLIurU/SwzsjKDG3 jfovHzwhEOF18Na9Fzd701jkn3q3rqqXsMSUFRA5MOiIfPBSplzlqtLS fJ2rF0MHgZzy5lzmsVNX2FPcbWG5lf+p3doxoGkLrYaBYCBCMVKZNw9f QFTRgvju2shpfNUodq7Jur958lmTbPV/BG8xQ2tFSUuJnVojIJQpD3Kz v6EnnjPDKP2djNS8fr3xsc4KxZPfHQ1MUGCJBROVGaxxpGP4TglG15XD WJjfJQ== isc.org. 7200 IN RRSIG DNSKEY 5 2 7200 20130719230127 20130619230127 50012 isc.org. TfbYfiP8bq6k89EudcS69xRB7DDuWhEmedUdq30/DNmWi1omAfNz1lrC iXL8OQHvO88YG0p0IuPrpQqYZMw7FYxVe913KydhlbozR83T6nLdpHwZ /TeYTm9zrGWDubbhlFW2OP/cgETIbcj7w3flFs4MNlkIu4ur38ALWoaZ Zdo= isc.org. 7200 IN DNSKEY 257 3 5 BEAAAAOhHQDBrhQbtphgq2wQUpEQ5t4DtUHxoMVFu2hWLDMvoOMRXjGr hhCeFvAZih7yJHf8ZGfW6hd38hXG/xylYCO6Krpbdojwx8YMXLA5/kA+ u50WIL8ZR1R6KTbsYVMf/Qx5RiNbPClw+vT+U8eXEJmO20jIS1ULgqy3 47cBB1zMnnz/4LJpA0da9CbKj3A254T515sNIMcwsB8/2+2E63/zZrQz Bkj0BrN/9Bexjpiks3jRhZatEsXn3dTy47R09Uix5WcJt+xzqZ7+ysyL KOOedS39Z7SDmsn2eA0FKtQpwA6LXeG2w+jxmw3oA8lVUgEf/rzeC/bB yBNsO70aEFTd isc.org. 7200 IN DNSKEY 256 3 5 BQEAAAABwuHz9Cem0BJ0JQTO7C/a3McR6hMaufljs1dfG/inaJpYv7vH XTrAOm/MeKp+/x6eT4QLru0KoZkvZJnqTI8JyaFTw2OM/ItBfh/hL2lm Cft2O7n3MfeqYtvjPnY7dWghYW4sVfH7VVEGm958o9nfi79532Qeklxh x8pXWdeAaRU= isc.org. 7200 IN RRSIG NS 5 2 7200 20130719232951 20130619232951 50012 isc.org. YaKIWDJdbioSHJ7XBShYxVvvSFHn4cFJLfbW+fUjtXTRRF+ezR2B0FXI wd1ItCOya2k//JGkQ9dxQmM9+lgIwrBUJLi4QuR5uVTAhbPLyZAqoCvW adNa2qmQQeubOpalMYRjqVI8Pf42D6Rcq0FQvXJDKLv4LEKmYygti2XG vso= Amplification: They Go Past Eleven… Query for isc.org/ANY 36 bytes sent, 3284 bytes received ~91x amplification!
  11. 11. 11 | © 2013 Infoblox Inc. All Rights Reserved.11 | © 2013 Infoblox Inc. All Rights Reserved. A Little Math • Say each bot has a measly 1 Mbps connection to the Internet ̶ It can send 1Mbps/36B =~ 28K qps ̶ That generates 28K * 3284B =~ 736 Mbps • So 14 bots =~ 10 Gbps
  12. 12. 12 | © 2013 Infoblox Inc. All Rights Reserved.12 | © 2013 Infoblox Inc. All Rights Reserved. The Scourge of the Open Recursor • Open recursors are like the AK-47s the Soviets left all over the world, just waiting to be used for no good But just how common are they? = 33 million resolvers
  13. 13. 13 | © 2013 Infoblox Inc. All Rights Reserved.13 | © 2013 Infoblox Inc. All Rights Reserved. Why Should I Worry? • More bad news about DDoS attacks ̶ Average attack bandwidth up 718% to 48 Gbps from Q4 2012 to Q1 2013 ̶ Average attack packet rate now 32.4 Mpps ̶ Average attack duration up 7% to 34.5 hours ̶ 6.97% of attacks were DNS-based - An increase of over 200% in the last year *Source: Prolexic Quartlerly Global DDoS Attack Report, Q1 2013
  14. 14. 14 | © 2013 Infoblox Inc. All Rights Reserved.14 | © 2013 Infoblox Inc. All Rights Reserved. What Can I Do to Protect Myself? 1. Overprovision 2. Use anycast 3. Screen traffic to your name servers 4. Monitor traffic to your name servers
  15. 15. 15 | © 2013 Infoblox Inc. All Rights Reserved.15 | © 2013 Infoblox Inc. All Rights Reserved. Overprovision • (Yes, I know, it seems primitive) • Overprovisioning is one of the simplest ways to resist a DDoS attack ̶ Run authoritative name servers with more capacity than you need ̶ Run a widely distributed set of authoritative name servers ̶ Augment your authoritative name servers with cloud-based secondary name servers - Make sure the provider uses anycast
  16. 16. 16 | © 2013 Infoblox Inc. All Rights Reserved.16 | © 2013 Infoblox Inc. All Rights Reserved. Anycast • Anycast allows multiple, distributed name servers to share a single virtual IP address • Each name server advertises a route to that address to its neighbors • Queries sent to that address are routed to the closest name server instance
  17. 17. 17 | © 2013 Infoblox Inc. All Rights Reserved.17 | © 2013 Infoblox Inc. All Rights Reserved. Anycast in Action Router 2 Router 4Router 3 Router 1 Server instance A Server instance B Client DNS query to 10.0.0.1 Routing table from Router 1: Destination Mask Next-Hop Distance 192.168.0.0 /29 127.0.0.1 0 10.0.0.1 /32 192.168.0.1 1 10.0.0.1 /32 192.168.0.2 2 192.168.0.1 192.168.0.2 10.0.0.1 10.0.0.1
  18. 18. 18 | © 2013 Infoblox Inc. All Rights Reserved.18 | © 2013 Infoblox Inc. All Rights Reserved. Anycast in Action Router 2 Router 4Router 3 Router 1 Server instance A Server instance B Client Routing table from Router 1: Destination Mask Next-Hop Distance 192.168.0.0 /29 127.0.0.1 0 10.0.0.1 /32 192.168.0.1 1 10.0.0.1 /32 192.168.0.2 2 192.168.0.1 192.168.0.2 10.0.0.1 10.0.0.1
  19. 19. 19 | © 2013 Infoblox Inc. All Rights Reserved.19 | © 2013 Infoblox Inc. All Rights Reserved. How Does Anycast Address DDoS Attacks? • From any one location on the Internet, you can only see (and hence attack) a single member of an anycast group at once • If you succeed in taking out that replica, routing will shift traffic to another ̶ The first replica will probably recover ̶ It’s like Whac-A-Mole
  20. 20. 20 | © 2013 Infoblox Inc. All Rights Reserved.20 | © 2013 Infoblox Inc. All Rights Reserved. Screen Traffic to Your Name Servers • Take advantage of any anti-DDoS features built into devices on the path between your name servers and the Internet, such as ̶ Internet firewalls ̶ Load balancers • For example ̶ SYN flood mitigation, such as rate limiting SYN frames ̶ Router traffic shaping of UDP
  21. 21. 21 | © 2013 Infoblox Inc. All Rights Reserved.21 | © 2013 Infoblox Inc. All Rights Reserved. Monitor Traffic to Your Name Servers • Monitor traffic to your name servers, including ̶ Aggregate query rate ̶ Top queriers
  22. 22. 22 | © 2013 Infoblox Inc. All Rights Reserved.22 | © 2013 Infoblox Inc. All Rights Reserved. Monitoring Aggregate Query Rate
  23. 23. 23 | © 2013 Infoblox Inc. All Rights Reserved.23 | © 2013 Infoblox Inc. All Rights Reserved. Setting an Alert on Aggregate Query Rate
  24. 24. 24 | © 2013 Infoblox Inc. All Rights Reserved.24 | © 2013 Infoblox Inc. All Rights Reserved. Monitoring Top Clients
  25. 25. 25 | © 2013 Infoblox Inc. All Rights Reserved.25 | © 2013 Infoblox Inc. All Rights Reserved. Don’t Be a Part of the Problem 1. Use ingress filtering 2. Apply ACLs to your recursive name servers 3. Rate-limit traffic or responses from your name servers
  26. 26. 26 | © 2013 Infoblox Inc. All Rights Reserved.26 | © 2013 Infoblox Inc. All Rights Reserved. Rate-limit Traffic from Your Name Servers • If you can, rate-limit traffic from your name servers ̶ Using Response Rate Limiting, for example - A patch to BIND 9 by Paul Vixie and Vernon Schryver - Applies to authoritative name servers used in DDoS attacks against others - Prevents these name servers from sending the same response to the same client too frequently - Implemented in – NSD (3.2.15) – Knot (1.2-RC3) – As patches to BIND 9.8 and later - See www.redbarn.org/dns/ratelimits
  27. 27. 27 | © 2013 Infoblox Inc. All Rights Reserved.27 | © 2013 Infoblox Inc. All Rights Reserved. How RRL Works isc.org/ANY [3335 byte response] token bucket
  28. 28. 28 | © 2013 Infoblox Inc. All Rights Reserved.28 | © 2013 Infoblox Inc. All Rights Reserved. Thank you! 28
  29. 29. Pg. 29 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Going from one datacenter to five: The Iovation Story
  30. 30. 30© 2012 iovation Inc. What iovation Does Recognize devices connecting to websites Understands how these devices are related to each other Block devices that are known to be associated with fraud or abuse and flag anomalies
  31. 31. 31© 2012 iovation Inc. Driving Factors •    •   • 
  32. 32. 32© 2012 iovation Inc. Design Criteria •  •    • 
  33. 33. 33© 2012 iovation Inc. Service Oriented Architecture Consumer Facing Subscriber Facing Internal Service Real-time Asynchronous Web Service APIs Device Recognition Service Association & Reputation Service Business Rules Service Admin Console UI Reporting Message Bus Web Device Print Distribution Geo Service Velocity Service Analytics WWW Internet iovation subscribers consumers
  34. 34. 34© 2012 iovation Inc. Datacenter Types •   •   •  
  35. 35. 35© 2012 iovation Inc. Network Design BB1 AMS MIA 10g 20g 10g Internet consumers Content Delivery #3 Data StoragePDX MIAData StorageSEA iovation subscribers Data Storage Content Delivery Data Processing Subscriber Queries Private Network Content Downloads
  36. 36. 36© 2012 iovation Inc. Portland to Seattle http://www.zayo.com/sites/default/files/images/Zayo-US-Network-EXTERNAL-11-1-2012.kmz
  37. 37. Pg. 37 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns How does Iovation use Dyn? • API Interface • Active/Active between two sites • Admin Console • Active/Active between two sites • Content Distribution • GSLB among four sites
  38. 38. Pg. 38 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns • DNS cache poisoning, DNSSEC and general DNS security
  39. 39. Pg. 39 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Anatomy of a HTTP connection
  40. 40. Pg. 40 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns How does DNS load balancing work?
  41. 41. Pg. 41 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Simple active/passive example • Primary location assumes 100% traffic • In event of disaster, swing 100% of traffic to a standby location • Could be a “we’ll be back soon” or “status” page • Could be a backup copy of your app • We call this Active Failover
  42. 42. Pg. 42 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  43. 43. Pg. 43 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  44. 44. Pg. 44 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  45. 45. Pg. 45 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Active/Active Load Balancing (Global Server Load Balancing, GSLB) (Hot/Hot Load Balancing) (High Availability Load Balancing)
  46. 46. Pg. 46 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Traffic management with Dyn Traffic Director
  47. 47. Pg. 47 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  48. 48. Pg. 48 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Dyn Traffic Director • Fast Anycast network enables low TTLs • Monitor endpoints for health • Globally load balance among 7 regions • Use Anycast to gauge “where is the user?”
  49. 49. Pg. 49 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  50. 50. Pg. 50 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  51. 51. Pg. 51 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Favor performance over network topology? • Real-time monitoring of endpoints • Always serve the fastest endpoint for each user, regardless of network topology • That’s real-time traffic management with Dyn’s Traffic Director
  52. 52. Pg. 52 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  53. 53. Pg. 53 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  54. 54. Pg. 54 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  55. 55. Pg. 55 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Favor geopolitical boundaries above all? • Per query lookup on source address • Geopolitical IP mapping database • State by state and country by country granularity • That’s geo traffic management with Dyn’s Traffic Director
  56. 56. Pg. 56 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  57. 57. Pg. 57 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Traffic Management Recap • Active/Passive with health checks • Dyn Active Failover • Active/Active with health checks • 7 global regions by network topology -> Dyn Traffic Director • Add in real-time latency measurement -> Dyn Traffic Director with real-time traffic management • Add in geopolitical granularity -> Dyn Traffic Director with geo traffic management
  58. 58. Pg. 58 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns
  59. 59. Pg. 59 Scalability and Availability in the Real World -- @cvwdyn & @cricketondns Dyn Delivers Internet Performance •Traffic management and managed DNS •Message management and email delivery •Remote access and domain services
  60. 60. Pg. 60 Presentation Title (edit from Slide Master) @twitterhandle Scalability and Availability in the Real World Cory von Wallenstein Chief Technologist, Dyn Inc. cvw@dyn.com @cvwdyn Thank You! Cricket Liu, Chief Infrastructure Officer, Infoblox cricket@infoblox.com @cricketondns
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×