TLD Anycast DNS servers to ISPs

APRICOT 2017
Shoji Noguchi (JPRS),
Yoshibumi Suematsu (QTNet)
TLD Anycast DNS servers to ISPs
- to Create a More Resilient DNS Environment -
1Copyright © 2017 Japan Registry Services Co., Ltd., and Kyushu Telecommunication Network Co., Inc.

Agenda
1. Concept of “.jprs”
2. Background of Joint Research
3. Overview of Joint Research
4. Joint Research Report by JPRS
5. Joint Research Report by QTNet
Copyright © 2017 Japan Registry Services Co., Ltd. 2

1. Concept of “.jprs”

“.jprs” R&D Platform
Concept of “.jprs”
In order for the Internet to keep growing, as a registry
operator, we will need an environment in which to
create innovations …
.jprs TLD can provide experimental environment for
domain names and DNS.
For demonstration of experiments
– For innovative technologies and productions
– Difficult to implement in a production environment
For collaborative R&D with research and business partners
– Autonomous research pertaining to the Internet

2. Background of Joint Research

Japan is a Disaster-prone Country
Natural disasters in/around Japan
Earthquake, tsunami, typhoon, and volcanic eruption…
The importance of being prepared for
natural disasters
Collapsed highway
The Great Hanshin/Awaji Earthquake on Jan 17, 1995 [*1]
[*1] http://sciencewindow.jst.go.jp/html/sw23/sp-003 [*2] http://www.bousai.go.jp/kohou/kouhoubousai/h23/63/special_01.html
Tsunami Surging to town
The Great East Japan Earthquake on Mar 11, 2011 [*2]

Physical Geography of Japan
Characteristics of national land
Japan is an unexpectedly large land.
From <http://thetruesize.com/#/aboutModal?borders=1~!MTE3ODk5MzY.NzQyNzMzNw*MzMzOTgyNDc(MjQ5NjA3NDc~!JP*Mzg5NTA4MA.NDc5NjA5Mw(MTQw)OQ>
* Geographical latitude of Japan is
not same as that of Southeast Asia.
Approx.
3,000 km
Geographical features do not always pose potential danger
to all regions of Japan but specific regions in many cases.

Logical Structure of Internet in Japan
Characteristic of Internet structure
Internet resources are concentrated in
Tokyo and Osaka.
Internet Exchanges (IXs),
Transit connections,
Data Centers, and so on
Tokyo
Osaka
Approx.
400km
Natural disasters that occur in/around Tokyo/Osaka
can intensely affect Internet connectivity.

Goal - Enhancing the DNS Resiliency
By locations of DNS servers
Install DNS servers in several regions other than Tokyo
and Osaka.
Mitigation/distribution of DNS traffic concentration
Stabilization/reduction of RTT
Continuous provision of DNS/Internet services
In case of an outage of the DNS servers in a particular region

3. Overview of Joint Research

.jprs DNS servers
Installation sites of .jprs DNS servers
The above locations resemble those of .jp DNS.
.jprs DNS operations
.jprs DNS servers in Tokyo/Osaka are operated by JPRS.
Hostname Location Remarks
tld1.nic.jprs Tokyo
tld2.nic.jprs Osaka
tld3.nic.jprs Worldwide
tld4.nic.jprs Tokyo For R&D
tld5.nic.jprs Worldwide
tld2
Osaka Tokyo
tld1
tld4

Approach
Joint research effort with 8 domestic ISPs
Each of their service area covers designated
geographical areas without overlapping.
How to direct DNS query to Local Node
Install tld4.nic.jprs DNS servers into
their networks as Local Node.
A) Routing configuration
BGP
B) Full resolver configuration
“static-stub” zone type for BIND
– Specifying DNS servers’ IP addresses
Global Node (JPRS)
Local Node (ISP)

Measurement Environment Model
AS18149
AS2914
AS12041
AS131905
tld1.nic.jprs
tld2.nic.jprs
tld3.nic.jprs
tld4.nic.jprs
Collecting Data Source Destination Tools Interval [min.]
Continuous ISP’s Internet service Stub Resolver 2LD App server – Web wget 1
Continuous Name Resolution Stub Resolver 2LD App server – DNS dig 1
Reachability of .jprs DNS servers Full Resolver tld[1-5].nic.jp (JPRS)
+
tld4.nic.jprs (ISP)
ping 1
traceroute 1
tcpdump without a pause
tld5.nic.jprs
Internet
ISP 1 - AS XXX1
.jprs DNS - Global Nodes
AS131905
tld4.nic.jprs
Full Resolver
Stub Resolver
2LD App server
- Web & DNS
ISP network
www.example.jprs
ISP 8 – AS XXX8
Local Node

ISPs’ Measurement Environment
ISPs used routing/full resolver configuration and
DNS software.
ISP
Configuration
(A) Routing
(B) Full resolver
DNS Software (BIND) version
Full Resolver Local Node
HOTnet (A) BGP 9.9.7
9.9.8-P4
TOHKnet (A) BGP 9.9.4-RedHat-9.9.4-29.el7_2.3
HTNet (B) “static-stub” 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6
K-OPT (A) BGP 9.9.4-RedHat-9.9.4-29.el7_2.2
Enecom (B) “static-stub” 9.9.8-P4
STNet (B) “static-stub” 9.8.2rc1-RedHat-9.8.2-0.37.rc1.el6_7.6
QTNet (A) BGP 9.9.8-P4
OTNet (A) BGP N/A (Non-disclosure for business use)

Evaluation Methods
 Continuous Internet service availability and reachability
of .jprs DNS
Compare traffic behavior with/without Local Node.
.jprs DNS are located
only outside ISP network.
.jprs DNS are located
inside/outside ISP network.
< Without Local Node > < With Local Node >
Global Nodes Global Nodes Local Nodes

Evaluation Methods
 Continuous Internet service availability and reachability
of .jprs DNS
Compare traffic behavior by turning on/off Internet
connection (hereinafter called Normal/Disaster).
All .jprs DNS are unreachable
from inside ISP network.
Only Local Node inside ISP is
reachable from inside ISP network.
< Without Local Node > < With Local Node >
Global Nodes Global Nodes Local Nodes

4. Joint Research Report
by JPRS

Enhancing the DNS Resiliency
Reachability of .jprs DNS confirmed at full resolvers
Destination of .jprs DNS queries from full resolver
had inclined toward Local Node on their ISP network.
Queries/
hour
Measured using tcpdump
command at ISPs’ full resolvers.
Full resolver  .jprs DNS servers
Execute on a minute-by-minute basis
at stub resolver (dig command)
Number of DNS queries
to each .jprs DNS (IPv4+IPv6)
* GN: Global Node, LN: Local Node
0
40
80
120
6/26
13:00
6/26
19:00
6/27
1:00
6/27
7:00
6/27
13:00
6/27
19:00
6/28
1:00
6/28
7:00
GN
(tld1)
GN
(tld2)
GN
(tld3)
LN
(tld4)
GN
(tld5)
Local Node
W/O With
Normal -
Disaster -
<Normal> <Disaster>
by K-OPT

Findings at Full Resolver
- Normal & W/O Local Node
Geographic distribution of DNS traffic
Over concentration of DNS query to Tokyo and Osaka
Copyright © 2017 Japan Registry Services Co., Ltd.
Tokyo
Osaka
ISP
Global Node
(Tokyo, Osaka)
828km
401km
486km
292km
281km
DNS query by Destination and ISP at ISPs’ full resolvers
Dest.
ISP
Tokyo
tld[14]
Osaka
tld2
Worldwide
tld[35] Total
HOTnet
@Sapporo
3,646
(72.7)
170
(3.4)
1,200
(23.9)
5,016
(100.0)
HTNet
@Kanazawa
1,476
(79.6)
154
(8.3)
224
(12.1)
1,854
(100.0)
K-OPT
@Osaka
222
(10.8)
1,488
(72.4)
346
(16.8)
2,056
(100.0)
Enecom
@Hiroshima
812
(44.9)
678
(37.5)
319
(17.6)
1,809
(100.0)
QTNet
@Fukuoka
2,242
(27.9)
3,286
(40.9)
2,514
(31.3)
8,042
(100.0)
Total 8,398
(44.7)
5,776
(30.8)
4,603
(24.5)
18,777
(100.0)
[queries (ratio)]
Measurement period:
2016/6/23 13:00 ~ 6/24 13:00
(JST)
19
Local Node
W/O With
Normal - -
Disaster - -
Worldwide
24.5%
Tokyo
44.7%
Osaka
30.8%

Findings at Full Resolver
- Disaster & With Local Node
De-concentration of DNS query to Tokyo and Osaka
DNS query addressed to other than Local Node ≠ Zero
Affected by NS selection algorithm in BIND
DNS query by Destination and ISP at ISPs’ full resolvers
[queries (ratio)]
Measurement period:
2016/6/23 13:00 ~ 6/24 13:00
(JST)
20
Local Node
W/O With
Normal - -
Disaster - -
Dest.
ISP
Tokyo
tld[14]
Osaka
tld2
World
tld[35]
Local
Node
Total
HOTnet 102*
(2.2)
96*
(1.3)
162*
(2.4)
5,108
(94.1)
5,468
(100.0)
HTNet 36*
(6.0)
34*
(1.0)
80*
(5.2)
2,066
(87.8)
2,216
(100.0)
K-OPT 112*
(4.5)
96*
(3.9)
158*
(6.4)
2,102
(85.2)
2,468
(100.0)
Enecom - - - 1,638 1,638
QTNet - - - 8,236 8,236
Total 250* 226* 400* 19,150 20,026
*: packet loss
Tokyo
Osaka
ISP
Global Node
(Tokyo, Osaka)
828km
401km
486km
292km
281km
Worldwide
3.9%*Tokyo
2.5%*
Osaka
2.2%*
Local Node
91.4%

Findings at Stub Resolver
Changes in RTT: Normal  Disaster
Compared between “(A) BGP” and “(B) static-stub.”
Case “(A) BGP” - 5 ISPs selected
S.D. of RTT affected by loss was 4~10 times larger.
Setting initial timeout to 800ms from BIND 9.6.0a1.
Measurement period:
2016/6/26 13:00 ~ 6/28 13:00
(JST)
20 40 60 80
Local Node
W/O With
Normal -
Disaster -
RTT of DNS query
Avg. + S.D. [ms]
HOTnet
@Sapporo
0.7 + 6.7
 1.9 + 31.2
K-OPT
@Osaka
7.2 + 5.4
 8.2 + 54.8
QTNet
@Fukuoka
3.7 + 17.6
 8.9 + 69.0

Findings at Stub Resolver
Case “(B) static-stub” - 3 ISPs selected
This function is provided by BIND 9.8.0 or above.
Forcing DNS queries for a zone to go to specified IP addresses
S.D. of RTT affected by loss was ~2 times larger.
S.D. of RTT(B) was 2~8 times smaller than that of (A).
Measurement period:
2016/6/26 13:00 ~ 6/28 13:00
(JST)
20 40 60 80
Local Node
W/O With
Normal -
Disaster -
RTT of DNS query
Avg. + S.D. [ms]
HTNet
@Kanazawa
2.5 + 11.3
 2.6 + 14.8
Enecom
@Hiroshima
2.5 + 3.4
 2.6 + 7.7
HOTnet : 31.2
K-OPT : 54.8
QTNet : 69.0
In a particular (B), DNS query inclines toward a destination
with a shorter RTT.
It depends on how we use either or both of (A) and (B).

RTT of Enecom was shorter than that of HTNet.
Confirmed the trend of DNS queries.
Enecom - BIND 9.9.8
– Sent one DNS query to all .jprs DNS servers only once a day
at 0:00 AM, and sent all the other DNS queries to Local Node.
HTNet - BIND 9.8.2
– Sent DNS queries to all .jprs DNS servers approximately every 6
hours, and sent all the other DNS queries to Local Node.
Secondary Findings
- Behavior of “static-stub”
23
Assumed that the difference is a load balancing algorithm.
5 10 15 20
Local Node
W/O With
Normal -
Disaster - -
Full Resolver
DNS Software
RTT of DNS query [ms]
Avg. + S.D.
Enecom BIND 9.9.8-P4 0.9 + 0.8
HTNet BIND 9.8.2rc1 5.1 + 4.6

Continuity of Internet services
Effect of installing Local Node into ISP network
All 8 ISPs were able to continue offering their Internet
service inside their own network. 
 Measured using wget command at
ISPs’ stub resolvers.
 Stub resolver  .jprs 2LD Web server
 Execute on a minute-by-minute basis
 All 8 ISPs could continue receiving
the results of success!
Number of Success to access .jprs
2LD Web Server by K-OPTSuccesses/
hour
0
10
20
30
40
50
60
6/26
13:00
6/26
19:00
6/27
01:00
6/27
07:00
6/27
13:00
6/27
19:00
6/28
01:00
6/28
07:00
6/28
13:00
<Normal> <Disaster>
100%!
Local Node
W/O With
Normal -
Disaster -

Future Works
Relating to this activity
Sharing ISPs’ .jprs DNS servers, or Local Nodes, among
themselves
Evaluation of geographical dispersal of DNS
Root, TLD, and 2nd level domain DNS servers
 Vertical integration
Full resolver and authoritative DNS
 Horizontal integration

Copyright © 2017 Kyushu Telecommunication Network Co., Inc. All rights reserved.
5. Joint Research Report
by QTNet
26

Kyushu Telecommunication Network
27
Company Name
Kyushu Telecommunication Network Co., Inc. (QTNet)
Telecommunications carrier in Kyushu , Japan
Services
Wide-Area Ethernet
FTTH
Internet Accsess, VoIP, and TV
AS7679
Our coverage area:
Area: 36.750km2 (10% of Japan)
Population: 13.2Mil. (10% of Japan)
Kyushu is one of the 8 regions of Japan

 Internet connectivity in Japan concentrates in Tokyo and Osaka.
→The network of Kyushu depends on these areas.
What are problems?
 If the large disasters simultaneously
hit in/around Tokyo and Osaka
Isolated from other regions of Japan
Cannot provide our Internet services in Kyushu
IX
Full Resolver(QTNet)
Internet in Kyushu
28
Approx.
486km
Approx.
401km

The task of immediate importance
Earthquake in Kyusyu (Kumamoto)
Felt earthquakes have been approximately 4000 times since Oct. 10, 2016
04/15
06:00
04/15
08:20
04/15
10:40
04/15
13:00
04/15
15:20
04/15
17:40
04/15
20:00
04/15
22:20
04/16
00:40
04/16
03:00
04/16
05:20
DNS Query received at Full Resolver
requests received
Recent Natural Disasters in Kyushu
29
In any situation, we must provide our customers with our Internet services!
Date & Time（JST） Magnitude
14 April 21:26 6.5
14 April 22:07 5.8
15 April 00:03 6.4
16 April 01:25 7.3
16 April 01:45 5.9
16 April 03:55 5.8
16 April 09:48 5.4
[qps]
M7.3
M5.8

Damage by Earthquake
Kumamoto Earthquakes in Apr. 14, 2016
30
https://ja-jp.facebook.com/kyuden.jp/posts/940170829434491

Background
31
com
jp
net
arpa
org
other
TLD ranking of request for DNS query
jp
com
net
The advantage of using “.jprs”
 ".jprs" registry operator is same as ".jp," which is ccTLD for Japan
 Ratio of DNS queries by TLD in our FTTH service: “.jp” is 2nd place.
There are many DNS queries for .jp.
Many important customers have used .jp.

Simulating the isolation of Kyushu.
TLD Anycast DNS servers to QTNet
32
0
20
40
60
80
01/08
13:00
01/09
03:00
01/09
17:00
01/10
07:00
01/10
21:00
01/11
11:00
01/12
01:00
01/12
15:00
01/13
05:00
01/13
19:00
01/14
09:00
01/14
23:00
01/15
13:00
01/16
03:00
01/16
17:00
01/17
07:00
01/17
21:00
Number of DNS queries to each .jprs DNS
GN(tld1) GN(tld2) GN(tld3) GN/LN(tld4) GN(tld5)
Process
#1 #2 #3 #4 #5
[Queries/Hour] (Full resolver  .jprs DNS servers)
LN GN
1
GN
2
GN
3
GN
4
GN
5
#1
#2
#3
#4
#5
Disconnect
DNS service could be provided
even under the condition that Kyushu had been isolated.

Other results
 Other interesting results by setting local node to ISP.
33
0.0
5.0
10.0
15.0
20.0
25.0
30.0
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
04/25
05:00
04/25
07:00
04/25
09:00
04/25
11:00
04/25
13:00
04/25
15:00
04/25
17:00
04/25
19:00
04/25
21:00
Number of RTT & hop to each .jprs DNS
tld1(RTT) tld2(RTT) tld3(RTT) tld4(RTT) tld5(RTT)
tld1(hop) tld2(hop) tld3(hop) tld4(hop) tld5(hop)
0
10
20
30
40
50
60
70
04/25
05:00
04/25
07:00
04/25
09:00
04/25
11:00
04/25
13:00
04/25
15:00
04/25
17:00
04/25
19:00
04/25
21:00
Number of DNS queries to each .jprs DNS
tld1v4 tld2v4 tld3v4 tld4v4 tld5v4
The full resolver(BIND) preferentially selects .jprs DNS with shorter RTT.
[ms] [hop] [Queries
/Hour] <Normal> <Disaster>
Collecting Data Source Destination Tools Interval [min.]
RTT & hop Stub Resolver .jprs DNS traceroute 1
DNS query Stub Resolver .jprs DNS dig 1
<Normal> <Disaster>

Conclusion
Installing TLD Anycast DNS server in QTNet.
Providing DNS service could be continued under the
conditions such as Kyushu is isolated.
Reducing RTT by installing .jprs local node.
Installing local node is effective both in normal
times and in large-scale disasters.
Future work
To install .jp local node in Kyushu! :-)
34

Contact Us
Email: dotjprstestbed-sec@jprs.co.jp
URI: https://nic.jprs/

APPENDICES

Sample of “static-stub” zone type
named.conf for BIND
% cat /etc/named-without-localnode.conf
(snip)
zone "jprs." {
type static-stub;
server-addresses {
// Global Nodes (JPRS)
103.47.2.1; // tld1.nic.jprs
2001:dda::1; // tld1.nic.jprs
117.104.133.16; // tld2.nic.jprs
2001:218:3001::1; // tld2.nic.jprs
65.22.40.1; // tld3.nic.jprs
2a01:8840:1ba::1; // tld3.nic.jprs
103.198.210.1; // tld4.nic.jprs
2403:2880::1; // tld4.nic.jprs
65.22.40.129; // tld5.nic.jprs
2a01:8840:1ba::129; // tld5.nic.jprs
};
};
(snip)
% cat /etc/named-with-localnode.conf
(snip)
zone "jprs." {
type static-stub;
server-addresses {
// Global Nodes (JPRS)
103.47.2.1; // tld1.nic.jprs
2001:dda::1; // tld1.nic.jprs
117.104.133.16; // tld2.nic.jprs
2001:218:3001::1; // tld2.nic.jprs
65.22.40.1; // tld3.nic.jprs
2a01:8840:1ba::1; // tld3.nic.jprs
// 103.198.210.1; // tld4.nic.jprs
// 2403:2880::1; // tld4.nic.jprs
65.22.40.129; // tld5.nic.jprs
2a01:8840:1ba::129; // tld5.nic.jprs
// Local Nodes (ISP)
192.0.2.53; // tld4.nic.jprs
2001:db8::53; // tld4.nic.jprs
};
};
(snip)
Add
Rem

Available to Local Node
Mitigation of DNS traffic
De-concentration of DNS query to Tokyo and Osaka
DNS query by Destination and ISPs at ISPs’ full resolvers
Measurement period:
2016/6/24 13:00 ~ 6/25 13:00
(JST)
38
Local Node
W/O With
Normal - -
Disaster - -
Dest.
ISP
Tokyo
tld[14]
Osaka
tld2
World
tld[35]
Local
Node
Total
HOTnet 114
(2.2)
70
(1.3)
128
(2.4)
4,954
(94.1)
5,266
(100.0)
HTNet 128
(6.0)
22
(1.0)
112
(5.2)
1,880
(87.8)
2,142
(100.0)
K-OPT 122
(5.5)
176
(8.0)
224
(10.2)
1,678
(76.3)
2,200
(100.0)
Enecom 18
(1.1)
11
(0.7)
13
(0.8)
1,592
(97.4)
1,634
(100.0)
QTNet 972
(16.4)
890
(15.1)
1,854
(31.4)
2,988
(50.6)
5,894
(100.0)
Total 1,696
(7.5)
2,067
(6.5)
3,071
(13.0)
13,484
(73.0)
20,318
(100.0)
Tokyo
Osaka
ISP
Global Node
(Tokyo, Osaka)
828km
401km
486km
292km
281km
[queries (ratio)]
Worldwide
13.0%
Tokyo
7.5%
Osaka
6.5%
Local Node
73.0%

TLD Anycast DNS servers to ISPs

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to TLD Anycast DNS servers to ISPs

Similar to TLD Anycast DNS servers to ISPs (20)

More from APNIC

More from APNIC (20)

Recently uploaded

Recently uploaded (9)

TLD Anycast DNS servers to ISPs