Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TLD Anycast DNS servers to ISPs

728 views

Published on

Presentation by Shoji Noguchi at APRICOT 2017 on Tuesday, 28 February 2017.

Published in: Internet
  • Be the first to comment

  • Be the first to like this

TLD Anycast DNS servers to ISPs

  1. 1. APRICOT 2017 Shoji Noguchi (JPRS), Yoshibumi Suematsu (QTNet) TLD Anycast DNS servers to ISPs - to Create a More Resilient DNS Environment - 1Copyright © 2017 Japan Registry Services Co., Ltd., and Kyushu Telecommunication Network Co., Inc.
  2. 2. Agenda 1. Concept of “.jprs” 2. Background of Joint Research 3. Overview of Joint Research 4. Joint Research Report by JPRS 5. Joint Research Report by QTNet Copyright © 2017 Japan Registry Services Co., Ltd. 2
  3. 3. 1. Concept of “.jprs” Copyright © 2017 Japan Registry Services Co., Ltd. 3
  4. 4. “.jprs” R&D Platform Concept of “.jprs” In order for the Internet to keep growing, as a registry operator, we will need an environment in which to create innovations … .jprs TLD can provide experimental environment for domain names and DNS. For demonstration of experiments – For innovative technologies and productions – Difficult to implement in a production environment For collaborative R&D with research and business partners – Autonomous research pertaining to the Internet Copyright © 2017 Japan Registry Services Co., Ltd. 4
  5. 5. 2. Background of Joint Research Copyright © 2017 Japan Registry Services Co., Ltd. 5
  6. 6. Japan is a Disaster-prone Country Natural disasters in/around Japan Earthquake, tsunami, typhoon, and volcanic eruption… Copyright © 2017 Japan Registry Services Co., Ltd. 6 The importance of being prepared for natural disasters Collapsed highway The Great Hanshin/Awaji Earthquake on Jan 17, 1995 [*1] [*1] http://sciencewindow.jst.go.jp/html/sw23/sp-003 [*2] http://www.bousai.go.jp/kohou/kouhoubousai/h23/63/special_01.html Tsunami Surging to town The Great East Japan Earthquake on Mar 11, 2011 [*2]
  7. 7. Physical Geography of Japan Characteristics of national land Japan is an unexpectedly large land. Copyright © 2017 Japan Registry Services Co., Ltd. 7 From <http://thetruesize.com/#/aboutModal?borders=1~!MTE3ODk5MzY.NzQyNzMzNw*MzMzOTgyNDc(MjQ5NjA3NDc~!JP*Mzg5NTA4MA.NDc5NjA5Mw(MTQw)OQ> * Geographical latitude of Japan is not same as that of Southeast Asia. Approx. 3,000 km Geographical features do not always pose potential danger to all regions of Japan but specific regions in many cases.
  8. 8. Logical Structure of Internet in Japan Characteristic of Internet structure Internet resources are concentrated in Tokyo and Osaka. Internet Exchanges (IXs), Transit connections, Data Centers, and so on Copyright © 2017 Japan Registry Services Co., Ltd. 8 Tokyo Osaka Approx. 400km Natural disasters that occur in/around Tokyo/Osaka can intensely affect Internet connectivity.
  9. 9. Goal - Enhancing the DNS Resiliency By locations of DNS servers Install DNS servers in several regions other than Tokyo and Osaka. Mitigation/distribution of DNS traffic concentration Stabilization/reduction of RTT Continuous provision of DNS/Internet services In case of an outage of the DNS servers in a particular region Copyright © 2017 Japan Registry Services Co., Ltd. 9
  10. 10. 3. Overview of Joint Research Copyright © 2017 Japan Registry Services Co., Ltd. 10
  11. 11. .jprs DNS servers Installation sites of .jprs DNS servers The above locations resemble those of .jp DNS. .jprs DNS operations .jprs DNS servers in Tokyo/Osaka are operated by JPRS. Copyright © 2017 Japan Registry Services Co., Ltd. 11 Hostname Location Remarks tld1.nic.jprs Tokyo tld2.nic.jprs Osaka tld3.nic.jprs Worldwide tld4.nic.jprs Tokyo For R&D tld5.nic.jprs Worldwide tld2 Osaka Tokyo tld1 tld4
  12. 12. Approach Joint research effort with 8 domestic ISPs Each of their service area covers designated geographical areas without overlapping. How to direct DNS query to Local Node Install tld4.nic.jprs DNS servers into their networks as Local Node. A) Routing configuration BGP B) Full resolver configuration “static-stub” zone type for BIND – Specifying DNS servers’ IP addresses Copyright © 2017 Japan Registry Services Co., Ltd. 12 Global Node (JPRS) Local Node (ISP)
  13. 13. Measurement Environment Model Copyright © 2017 Japan Registry Services Co., Ltd. 13 AS18149 AS2914 AS12041 AS131905 tld1.nic.jprs tld2.nic.jprs tld3.nic.jprs tld4.nic.jprs Collecting Data Source Destination Tools Interval [min.] Continuous ISP’s Internet service Stub Resolver 2LD App server – Web wget 1 Continuous Name Resolution Stub Resolver 2LD App server – DNS dig 1 Reachability of .jprs DNS servers Full Resolver tld[1-5].nic.jp (JPRS) + tld4.nic.jprs (ISP) ping 1 traceroute 1 tcpdump without a pause tld5.nic.jprs Internet ISP 1 - AS XXX1 .jprs DNS - Global Nodes AS131905 tld4.nic.jprs Full Resolver Stub Resolver 2LD App server - Web & DNS ISP network www.example.jprs ISP 8 – AS XXX8 Local Node
  14. 14. ISPs’ Measurement Environment ISPs used routing/full resolver configuration and DNS software. Copyright © 2017 Japan Registry Services Co., Ltd. 14 ISP Configuration (A) Routing (B) Full resolver DNS Software (BIND) version Full Resolver Local Node HOTnet (A) BGP 9.9.7 9.9.8-P4 TOHKnet (A) BGP 9.9.4-RedHat-9.9.4-29.el7_2.3 HTNet (B) “static-stub” 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 K-OPT (A) BGP 9.9.4-RedHat-9.9.4-29.el7_2.2 Enecom (B) “static-stub” 9.9.8-P4 STNet (B) “static-stub” 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6 QTNet (A) BGP 9.9.8-P4 OTNet (A) BGP N/A (Non-disclosure for business use)
  15. 15. Evaluation Methods  Continuous Internet service availability and reachability of .jprs DNS Compare traffic behavior with/without Local Node. Copyright © 2017 Japan Registry Services Co., Ltd. 15 .jprs DNS are located only outside ISP network. .jprs DNS are located inside/outside ISP network. < Without Local Node > < With Local Node > Global Nodes Global Nodes Local Nodes
  16. 16. Evaluation Methods  Continuous Internet service availability and reachability of .jprs DNS Compare traffic behavior by turning on/off Internet connection (hereinafter called Normal/Disaster). Copyright © 2017 Japan Registry Services Co., Ltd. 16 All .jprs DNS are unreachable from inside ISP network. Only Local Node inside ISP is reachable from inside ISP network. < Without Local Node > < With Local Node > Global Nodes Global Nodes Local Nodes
  17. 17. 4. Joint Research Report by JPRS Copyright © 2017 Japan Registry Services Co., Ltd. 17
  18. 18. Enhancing the DNS Resiliency Reachability of .jprs DNS confirmed at full resolvers Destination of .jprs DNS queries from full resolver had inclined toward Local Node on their ISP network. Copyright © 2017 Japan Registry Services Co., Ltd. 18 Queries/ hour Measured using tcpdump command at ISPs’ full resolvers. Full resolver  .jprs DNS servers Execute on a minute-by-minute basis at stub resolver (dig command) Number of DNS queries to each .jprs DNS (IPv4+IPv6) * GN: Global Node, LN: Local Node 0 40 80 120 6/26 13:00 6/26 19:00 6/27 1:00 6/27 7:00 6/27 13:00 6/27 19:00 6/28 1:00 6/28 7:00 GN (tld1) GN (tld2) GN (tld3) LN (tld4) GN (tld5) Local Node W/O With Normal - Disaster - <Normal> <Disaster> by K-OPT
  19. 19. Findings at Full Resolver - Normal & W/O Local Node Geographic distribution of DNS traffic Over concentration of DNS query to Tokyo and Osaka Copyright © 2017 Japan Registry Services Co., Ltd. Tokyo Osaka ISP Global Node (Tokyo, Osaka) 828km 401km 486km 292km 281km DNS query by Destination and ISP at ISPs’ full resolvers Dest. ISP Tokyo tld[14] Osaka tld2 Worldwide tld[35] Total HOTnet @Sapporo 3,646 (72.7) 170 (3.4) 1,200 (23.9) 5,016 (100.0) HTNet @Kanazawa 1,476 (79.6) 154 (8.3) 224 (12.1) 1,854 (100.0) K-OPT @Osaka 222 (10.8) 1,488 (72.4) 346 (16.8) 2,056 (100.0) Enecom @Hiroshima 812 (44.9) 678 (37.5) 319 (17.6) 1,809 (100.0) QTNet @Fukuoka 2,242 (27.9) 3,286 (40.9) 2,514 (31.3) 8,042 (100.0) Total 8,398 (44.7) 5,776 (30.8) 4,603 (24.5) 18,777 (100.0) [queries (ratio)] Measurement period: 2016/6/23 13:00 ~ 6/24 13:00 (JST) 19 Local Node W/O With Normal - - Disaster - - Worldwide 24.5% Tokyo 44.7% Osaka 30.8%
  20. 20. Findings at Full Resolver - Disaster & With Local Node De-concentration of DNS query to Tokyo and Osaka DNS query addressed to other than Local Node ≠ Zero Affected by NS selection algorithm in BIND Copyright © 2017 Japan Registry Services Co., Ltd. DNS query by Destination and ISP at ISPs’ full resolvers [queries (ratio)] Measurement period: 2016/6/23 13:00 ~ 6/24 13:00 (JST) 20 Local Node W/O With Normal - - Disaster - - Dest. ISP Tokyo tld[14] Osaka tld2 World tld[35] Local Node Total HOTnet 102* (2.2) 96* (1.3) 162* (2.4) 5,108 (94.1) 5,468 (100.0) HTNet 36* (6.0) 34* (1.0) 80* (5.2) 2,066 (87.8) 2,216 (100.0) K-OPT 112* (4.5) 96* (3.9) 158* (6.4) 2,102 (85.2) 2,468 (100.0) Enecom - - - 1,638 1,638 QTNet - - - 8,236 8,236 Total 250* 226* 400* 19,150 20,026 *: packet loss Tokyo Osaka ISP Global Node (Tokyo, Osaka) 828km 401km 486km 292km 281km Worldwide 3.9%*Tokyo 2.5%* Osaka 2.2%* Local Node 91.4%
  21. 21. Findings at Stub Resolver Changes in RTT: Normal  Disaster Compared between “(A) BGP” and “(B) static-stub.” Case “(A) BGP” - 5 ISPs selected S.D. of RTT affected by loss was 4~10 times larger. Setting initial timeout to 800ms from BIND 9.6.0a1. Copyright © 2017 Japan Registry Services Co., Ltd. 21 Measurement period: 2016/6/26 13:00 ~ 6/28 13:00 (JST) 20 40 60 80 Local Node W/O With Normal - Disaster - RTT of DNS query Avg. + S.D. [ms] HOTnet @Sapporo 0.7 + 6.7  1.9 + 31.2 K-OPT @Osaka 7.2 + 5.4  8.2 + 54.8 QTNet @Fukuoka 3.7 + 17.6  8.9 + 69.0
  22. 22. Findings at Stub Resolver Case “(B) static-stub” - 3 ISPs selected This function is provided by BIND 9.8.0 or above. Forcing DNS queries for a zone to go to specified IP addresses S.D. of RTT affected by loss was ~2 times larger. S.D. of RTT(B) was 2~8 times smaller than that of (A). Copyright © 2017 Japan Registry Services Co., Ltd. 22 Measurement period: 2016/6/26 13:00 ~ 6/28 13:00 (JST) 20 40 60 80 Local Node W/O With Normal - Disaster - RTT of DNS query Avg. + S.D. [ms] HTNet @Kanazawa 2.5 + 11.3  2.6 + 14.8 Enecom @Hiroshima 2.5 + 3.4  2.6 + 7.7 HOTnet : 31.2 K-OPT : 54.8 QTNet : 69.0 In a particular (B), DNS query inclines toward a destination with a shorter RTT. It depends on how we use either or both of (A) and (B).
  23. 23. RTT of Enecom was shorter than that of HTNet. Confirmed the trend of DNS queries. Enecom - BIND 9.9.8 – Sent one DNS query to all .jprs DNS servers only once a day at 0:00 AM, and sent all the other DNS queries to Local Node. HTNet - BIND 9.8.2 – Sent DNS queries to all .jprs DNS servers approximately every 6 hours, and sent all the other DNS queries to Local Node. Secondary Findings - Behavior of “static-stub” 23 Assumed that the difference is a load balancing algorithm. 5 10 15 20 Local Node W/O With Normal - Disaster - - Copyright © 2017 Japan Registry Services Co., Ltd. Full Resolver DNS Software RTT of DNS query [ms] Avg. + S.D. Enecom BIND 9.9.8-P4 0.9 + 0.8 HTNet BIND 9.8.2rc1 5.1 + 4.6
  24. 24. Continuity of Internet services Effect of installing Local Node into ISP network All 8 ISPs were able to continue offering their Internet service inside their own network.  Copyright © 2017 Japan Registry Services Co., Ltd. 24  Measured using wget command at ISPs’ stub resolvers.  Stub resolver  .jprs 2LD Web server  Execute on a minute-by-minute basis  All 8 ISPs could continue receiving the results of success! Number of Success to access .jprs 2LD Web Server by K-OPTSuccesses/ hour 0 10 20 30 40 50 60 6/26 13:00 6/26 19:00 6/27 01:00 6/27 07:00 6/27 13:00 6/27 19:00 6/28 01:00 6/28 07:00 6/28 13:00 <Normal> <Disaster> 100%! Local Node W/O With Normal - Disaster -
  25. 25. Future Works Relating to this activity Sharing ISPs’ .jprs DNS servers, or Local Nodes, among themselves Evaluation of geographical dispersal of DNS Root, TLD, and 2nd level domain DNS servers  Vertical integration Full resolver and authoritative DNS  Horizontal integration Copyright © 2017 Japan Registry Services Co., Ltd. 25
  26. 26. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. 5. Joint Research Report by QTNet 26
  27. 27. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Kyushu Telecommunication Network 27 Company Name Kyushu Telecommunication Network Co., Inc. (QTNet) Telecommunications carrier in Kyushu , Japan Services Wide-Area Ethernet FTTH Internet Accsess, VoIP, and TV AS7679 Our coverage area: Area: 36.750km2 (10% of Japan) Population: 13.2Mil. (10% of Japan) Kyushu is one of the 8 regions of Japan
  28. 28. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.  Internet connectivity in Japan concentrates in Tokyo and Osaka. →The network of Kyushu depends on these areas. What are problems?  If the large disasters simultaneously hit in/around Tokyo and Osaka Isolated from other regions of Japan Cannot provide our Internet services in Kyushu IX Full Resolver(QTNet) Internet in Kyushu 28 Approx. 486km Approx. 401km
  29. 29. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. The task of immediate importance Earthquake in Kyusyu (Kumamoto) Felt earthquakes have been approximately 4000 times since Oct. 10, 2016 04/15 06:00 04/15 08:20 04/15 10:40 04/15 13:00 04/15 15:20 04/15 17:40 04/15 20:00 04/15 22:20 04/16 00:40 04/16 03:00 04/16 05:20 DNS Query received at Full Resolver requests received Recent Natural Disasters in Kyushu 29 In any situation, we must provide our customers with our Internet services! Date & Time(JST) Magnitude 14 April 21:26 6.5 14 April 22:07 5.8 15 April 00:03 6.4 16 April 01:25 7.3 16 April 01:45 5.9 16 April 03:55 5.8 16 April 09:48 5.4 [qps] M7.3 M5.8
  30. 30. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Damage by Earthquake Kumamoto Earthquakes in Apr. 14, 2016 30 https://ja-jp.facebook.com/kyuden.jp/posts/940170829434491
  31. 31. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Background 31 com jp net arpa org other TLD ranking of request for DNS query jp com net The advantage of using “.jprs”  ".jprs" registry operator is same as ".jp," which is ccTLD for Japan  Ratio of DNS queries by TLD in our FTTH service: “.jp” is 2nd place. There are many DNS queries for .jp. Many important customers have used .jp.
  32. 32. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Simulating the isolation of Kyushu. TLD Anycast DNS servers to QTNet 32 0 20 40 60 80 01/08 13:00 01/09 03:00 01/09 17:00 01/10 07:00 01/10 21:00 01/11 11:00 01/12 01:00 01/12 15:00 01/13 05:00 01/13 19:00 01/14 09:00 01/14 23:00 01/15 13:00 01/16 03:00 01/16 17:00 01/17 07:00 01/17 21:00 Number of DNS queries to each .jprs DNS GN(tld1) GN(tld2) GN(tld3) GN/LN(tld4) GN(tld5) Process #1 #2 #3 #4 #5 [Queries/Hour] (Full resolver  .jprs DNS servers) LN GN 1 GN 2 GN 3 GN 4 GN 5 #1 #2 #3 #4 #5 Disconnect DNS service could be provided even under the condition that Kyushu had been isolated.
  33. 33. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Other results  Other interesting results by setting local node to ISP. 33 0.0 5.0 10.0 15.0 20.0 25.0 30.0 0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 04/25 05:00 04/25 07:00 04/25 09:00 04/25 11:00 04/25 13:00 04/25 15:00 04/25 17:00 04/25 19:00 04/25 21:00 Number of RTT & hop to each .jprs DNS tld1(RTT) tld2(RTT) tld3(RTT) tld4(RTT) tld5(RTT) tld1(hop) tld2(hop) tld3(hop) tld4(hop) tld5(hop) 0 10 20 30 40 50 60 70 04/25 05:00 04/25 07:00 04/25 09:00 04/25 11:00 04/25 13:00 04/25 15:00 04/25 17:00 04/25 19:00 04/25 21:00 Number of DNS queries to each .jprs DNS tld1v4 tld2v4 tld3v4 tld4v4 tld5v4 The full resolver(BIND) preferentially selects .jprs DNS with shorter RTT. [ms] [hop] [Queries /Hour] <Normal> <Disaster> Collecting Data Source Destination Tools Interval [min.] RTT & hop Stub Resolver .jprs DNS traceroute 1 DNS query Stub Resolver .jprs DNS dig 1 <Normal> <Disaster>
  34. 34. Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved. Conclusion Installing TLD Anycast DNS server in QTNet. Providing DNS service could be continued under the conditions such as Kyushu is isolated. Reducing RTT by installing .jprs local node. Installing local node is effective both in normal times and in large-scale disasters. Future work To install .jp local node in Kyushu! :-) 34
  35. 35. Contact Us Email: dotjprstestbed-sec@jprs.co.jp URI: https://nic.jprs/ Copyright © 2017 Japan Registry Services Co., Ltd. 35
  36. 36. APPENDICES Copyright © 2017 Japan Registry Services Co., Ltd. 36
  37. 37. Sample of “static-stub” zone type named.conf for BIND Copyright © 2017 Japan Registry Services Co., Ltd. 37 % cat /etc/named-without-localnode.conf (snip) zone "jprs." { type static-stub; server-addresses { // Global Nodes (JPRS) 103.47.2.1; // tld1.nic.jprs 2001:dda::1; // tld1.nic.jprs 117.104.133.16; // tld2.nic.jprs 2001:218:3001::1; // tld2.nic.jprs 65.22.40.1; // tld3.nic.jprs 2a01:8840:1ba::1; // tld3.nic.jprs 103.198.210.1; // tld4.nic.jprs 2403:2880::1; // tld4.nic.jprs 65.22.40.129; // tld5.nic.jprs 2a01:8840:1ba::129; // tld5.nic.jprs }; }; (snip) % cat /etc/named-with-localnode.conf (snip) zone "jprs." { type static-stub; server-addresses { // Global Nodes (JPRS) 103.47.2.1; // tld1.nic.jprs 2001:dda::1; // tld1.nic.jprs 117.104.133.16; // tld2.nic.jprs 2001:218:3001::1; // tld2.nic.jprs 65.22.40.1; // tld3.nic.jprs 2a01:8840:1ba::1; // tld3.nic.jprs // 103.198.210.1; // tld4.nic.jprs // 2403:2880::1; // tld4.nic.jprs 65.22.40.129; // tld5.nic.jprs 2a01:8840:1ba::129; // tld5.nic.jprs // Local Nodes (ISP) 192.0.2.53; // tld4.nic.jprs 2001:db8::53; // tld4.nic.jprs }; }; (snip) Add Rem
  38. 38. Available to Local Node Mitigation of DNS traffic De-concentration of DNS query to Tokyo and Osaka Copyright © 2017 Japan Registry Services Co., Ltd. DNS query by Destination and ISPs at ISPs’ full resolvers Measurement period: 2016/6/24 13:00 ~ 6/25 13:00 (JST) 38 Local Node W/O With Normal - - Disaster - - Dest. ISP Tokyo tld[14] Osaka tld2 World tld[35] Local Node Total HOTnet 114 (2.2) 70 (1.3) 128 (2.4) 4,954 (94.1) 5,266 (100.0) HTNet 128 (6.0) 22 (1.0) 112 (5.2) 1,880 (87.8) 2,142 (100.0) K-OPT 122 (5.5) 176 (8.0) 224 (10.2) 1,678 (76.3) 2,200 (100.0) Enecom 18 (1.1) 11 (0.7) 13 (0.8) 1,592 (97.4) 1,634 (100.0) QTNet 972 (16.4) 890 (15.1) 1,854 (31.4) 2,988 (50.6) 5,894 (100.0) Total 1,696 (7.5) 2,067 (6.5) 3,071 (13.0) 13,484 (73.0) 20,318 (100.0) Tokyo Osaka ISP Global Node (Tokyo, Osaka) 828km 401km 486km 292km 281km [queries (ratio)] Worldwide 13.0% Tokyo 7.5% Osaka 6.5% Local Node 73.0%

×