Importance of data
Jay Daley, APNIC Vietnam 2017
• Small registry - Big data pioneer
22 server Hadoop cluster installed in 2012
Full packet capture of DNS servers
• Data science oriented research team
18% of total staff
• World class registrar data portal
Main business priority for new development
• Actively building data products
Recent success with broadbandmap.nz
• Open data portal for .nz data
Why is .nz presenting?
APNIC Vietnam 2017 2
• Evidence based policy
• Organisational development
• Detecting badness
• Business - more, new, better
• Public trust
Why data matters
APNIC Vietnam 2017 3
Evidence based policy
APNIC Vietnam 2017 4
• Data is not public and so …
No reproducibility
Agenda set by those who pay the analysts
Very low throughput of research
• Some people have their own data
Gives them real power in the debate
• Open Data means Open Debate
Use of data in policy
APNIC Vietnam 2017 5
Case study – hoarding?
APNIC Vietnam 2017 6
1
10
100
1000
10000
100000
1000000
1
2
3
4
5
6
7
8
9
10
11-20
21-30
31-40
41-50
51-60
61-70
71-80
81-90
91-100
101-200
201-300
301-400
401-500
501-600
601-700
801-900
901-1000
1001-2000
2001-3000
Numberofregistrants
Portfolio size
Number of registrants by portfolio size
Detecting badness
APNIC Vietnam 2017 7
• Highly developed use of data
Multiple research teams, cooperative forums and
service providers
Multiple data resources, extensive data sharing
• Data collection – source evidence
Passive monitoring (e.g. DNSDB, PassiveTotal)
Hand produced by threat researchers
• Strong sharing culture – via data feeds
40+ feeds available (both NFP and commercial)
Track domains, IPs, credentials, URLs, etc
Very active area
APNIC Vietnam 2017 8
Business – more, new, better
APNIC Vietnam 2017 9
10APNIC Vietnam 2017
11
Search by
address
Search by
dropping a pin
Switch to
aerial view
Browse
APNIC Vietnam 2017
12APNIC Vietnam 2017
13APNIC Vietnam 2017
14APNIC Vietnam 2017
• Over 20 providers
• Some very sensitive about their data
• Awkward formats
• High cost for them to provide
• Our approach
• Ask nicely – no payment
• Demonstrate security – gain trust
• Sell the vision
• Be patient
• Help with tools
• Built a tool to generate wireless coverage
• Adopted another tool to convert files to right
format
The data
15APNIC Vietnam 2017
• Only be used for this purpose:
• Provide info to owner/occupier or
prospective owner/occupier on availability
• Missing providers
• Missing technologies
• No satellite (coming soon)
• No cellular
Limitations on the data
16APNIC Vietnam 2017
IP Topology Map
APNIC Vietnam 2017 17
Public trust
18APNIC Vietnam 2017
“Sunlight is the best
disinfectant”
• Open data can prove:
Effectiveness of markets and regulators
Social benefit
Where problems remain
Open data and public trust
APNIC Vietnam 2017 19
Case study – Web Index
APNIC Vietnam 2017 20
Data enabled community &
industry
• Data openly shared by all market
participants
• Data is easy to find and use and can be
trusted
• Strong community use of data
• Authoritative publications
A goal to consider
APNIC Vietnam 2017 21
• Begin cultural change
Broaden the principle of openness to include data
Set the vision of the benefits
• Engage community – “social license”
Community expectations of openness vs privacy
• Put legal framework in place
Adjust contracts, policies and processes to
support open data
Determine privacy protection rules
Steps to make it happen
APNIC Vietnam 2017 22
• National Broadband Map
• broadbandmap.nz
• Internet Data Portal
• idp.nz
• IP topology map
• ip.topology.net.nz
• BGP topology map
• bgp.topology.net.nz
Finding our services
23APNIC Vietnam 2017
Contact:
www.nzrs.net.nz
jay@nzrs.net.nz
Thanks
Any questions?

The importance of data

  • 1.
    Importance of data JayDaley, APNIC Vietnam 2017
  • 2.
    • Small registry- Big data pioneer 22 server Hadoop cluster installed in 2012 Full packet capture of DNS servers • Data science oriented research team 18% of total staff • World class registrar data portal Main business priority for new development • Actively building data products Recent success with broadbandmap.nz • Open data portal for .nz data Why is .nz presenting? APNIC Vietnam 2017 2
  • 3.
    • Evidence basedpolicy • Organisational development • Detecting badness • Business - more, new, better • Public trust Why data matters APNIC Vietnam 2017 3
  • 4.
  • 5.
    • Data isnot public and so … No reproducibility Agenda set by those who pay the analysts Very low throughput of research • Some people have their own data Gives them real power in the debate • Open Data means Open Debate Use of data in policy APNIC Vietnam 2017 5
  • 6.
    Case study –hoarding? APNIC Vietnam 2017 6 1 10 100 1000 10000 100000 1000000 1 2 3 4 5 6 7 8 9 10 11-20 21-30 31-40 41-50 51-60 61-70 71-80 81-90 91-100 101-200 201-300 301-400 401-500 501-600 601-700 801-900 901-1000 1001-2000 2001-3000 Numberofregistrants Portfolio size Number of registrants by portfolio size
  • 7.
  • 8.
    • Highly developeduse of data Multiple research teams, cooperative forums and service providers Multiple data resources, extensive data sharing • Data collection – source evidence Passive monitoring (e.g. DNSDB, PassiveTotal) Hand produced by threat researchers • Strong sharing culture – via data feeds 40+ feeds available (both NFP and commercial) Track domains, IPs, credentials, URLs, etc Very active area APNIC Vietnam 2017 8
  • 9.
    Business – more,new, better APNIC Vietnam 2017 9
  • 10.
  • 11.
    11 Search by address Search by droppinga pin Switch to aerial view Browse APNIC Vietnam 2017
  • 12.
  • 13.
  • 14.
  • 15.
    • Over 20providers • Some very sensitive about their data • Awkward formats • High cost for them to provide • Our approach • Ask nicely – no payment • Demonstrate security – gain trust • Sell the vision • Be patient • Help with tools • Built a tool to generate wireless coverage • Adopted another tool to convert files to right format The data 15APNIC Vietnam 2017
  • 16.
    • Only beused for this purpose: • Provide info to owner/occupier or prospective owner/occupier on availability • Missing providers • Missing technologies • No satellite (coming soon) • No cellular Limitations on the data 16APNIC Vietnam 2017
  • 17.
    IP Topology Map APNICVietnam 2017 17
  • 18.
  • 19.
    “Sunlight is thebest disinfectant” • Open data can prove: Effectiveness of markets and regulators Social benefit Where problems remain Open data and public trust APNIC Vietnam 2017 19
  • 20.
    Case study –Web Index APNIC Vietnam 2017 20
  • 21.
    Data enabled community& industry • Data openly shared by all market participants • Data is easy to find and use and can be trusted • Strong community use of data • Authoritative publications A goal to consider APNIC Vietnam 2017 21
  • 22.
    • Begin culturalchange Broaden the principle of openness to include data Set the vision of the benefits • Engage community – “social license” Community expectations of openness vs privacy • Put legal framework in place Adjust contracts, policies and processes to support open data Determine privacy protection rules Steps to make it happen APNIC Vietnam 2017 22
  • 23.
    • National BroadbandMap • broadbandmap.nz • Internet Data Portal • idp.nz • IP topology map • ip.topology.net.nz • BGP topology map • bgp.topology.net.nz Finding our services 23APNIC Vietnam 2017
  • 24.