The engineering report summarizes ARIN's staffing, accomplishments, operational challenges, and systems. It notes that the QA manager passed away and positions were filled. Accomplishments included improvements to DNS updates, DNSSEC, the user interface, and software integration. Operational challenges involved a UPS outage and DOS attacks. Systems discussed included expanded use of RPKI, RDAP development, and high uptime of services.

1. Engineering Report
Mark Kosters

2. Staffing
• Tim Christensen
QA Manager
– Passed away August 5,
2014
– Worked for ARIN for 14
years
• DBA
• System Architect
2

3. Staffing
• Operations
– 6 operations engineers + 2 managers (One vacancy in
operations)
• Development
– 8 programmers + manager
• New Software Integration head taken from engineering
• New hire – filled vacancy created by transfer to SI
• Software Integration formerly known as Quality Assurance
– Leadership Change
– 5 engineers, 1 contractor + manager
• Project Management
– 1
• CTO
– 1
3

4. Accomplishments since ARIN 33
• DNS (and DNSSEC) now have near-real time updates
– TTL’s added for NS’s and DS’s
– Hardening of signing infrastructure
• DNSSEC enabled for ARIN’s forward zones (and reverse)
• Shared tickets
• Display agreements associated with organizations
• User interface improvements for payment processing
• Transfers
– 8.3 Released
– 8.2 and 8.4 underway
• Movement away from ARIN HQ to Colo for back office
production underway
• Movement from EMC to NetApp underway
4

5. Accomplishments Cont…
• Fault Tolerance Improvements
– More efficient system backups
– Moving Production Systems from ARIN HQ to Colo
– Moving backend services to physical hardware when
merited
• Corporate Help Desk and IT Support
• ARIN Member Meeting Support
• Care and Feeding of Servers & Network
• OT&E
5

6. OT&E
• Operational Test & Evaluation
– Place to test code
– Place to test process
– All services now under ote.arin.net
• Replicated Core services
– Reg-RWS (provisioning API)
– Whois-RWS (directory API)
– Web Interface
– RPKI suite (up/down and hosted)
• Participation
– 30 new requests since February
– 152 networks registered to access OT&E
6

7. YTD Efforts Cont…
• IETF Participation
– SIDR (RPKI), WEIRDS (RDAP)
• ICANN Participation
– SSAC
– RSSAC
– Technical Advisory Group
7

8. Operational Challenges
• UPS incident
• DOS attacks
• ISP availability
8

9. UPS Incident
• ARIN HQ is in a cheap location – however suffers from
– Lack of power diversity
– Lack of connectivity options
• ARIN hosts the provisioning systems at HQ
– Mail, web, and reg-rws
– Long running project to move gear out to colocation site
• UPS outage
– Complicated and long-running fix
– Resulted in “almost” new UPS
• Lessons learned
– We did a pretty good job recovering gear that had not
suffered power cycles for a long time
– Renewed emphasis on moving to colo for production services
9

10. Other Operational Concerns
• Periodic DOS attacks
– Hitting our provisioning network
– Need to implement DOS mitigation with
upstreams
• ISP Availability
– Multiple connectivity outages with ARIN HQ
• Unannounced maintenance
• Fiber availability
– Issues with west coast PFS site
10

11. Operational Highlights
• 100% uptime on our public facing sites
over the past 6 years
• Services include
– Whois
– Whois-RWS
– DNS
– Mailing lists
– FTP
• Will do the same with RPKI
11

12. ARIN Online Usage
• 81,984 accounts activated since
inception through Q3 of 2014
12
2014*
2013
2012
2011
2010
2009
2008
Number of Accounts Activated
5000 10000 15000 20000
* Through Q3 of 2014

13. Active Usage of ARIN Online
13
• Logins from inception through Q3 of 2014
40000
30000
20000
10000
0
Logins
0 1 2 - 5 6 - 10 11 - 15 >16
# of Users
Times logged in

14. Reg-RWS Transactions
5000000
4500000
4000000
3500000
3000000
2500000
2000000
1500000
1000000
500000
14
ARIN
29
ARIN
30
ARIN
31
ARIN
32
ARIN
33
ARIN
34
0
Template 408383 595858 846943 1066037 1311403 1498204
REST 40374 320197 841105 3524124 4296734 4715231
Template
REST

15. Reports Via REST
ARIN 33 ARIN 34
Associations 176 9,445
Reassignments 25,219 69,320
WhoWas 253,135 382,619
15
• Requests since inception

16. DNSSEC
ARIN 34
DNSSEC Secured Zones 648
Number of Orgs with DNSSEC 94
Total Number of Delegations 552,329
16

17. RPKI Usage
ARIN XXX ARIN XXXI ARIN XXXII ARIN33 ARIN34
RPAs Signed 27 72 130 162 208
Certified
Orgs
47 68 108 153
ROAs 19 60 106 162 239
Covered
Resources
30 82 147 258 332
Web
Delegated
(REMOVED)
0 0 0 0
Up/Down
Delegated
0 0 0
17

18. Whois Queries Per Second
18
0
500
1000
1500
2000
2500
3000
3500
4000
2001-07
2001-11
2002-03
2002-07
2002-11
2003-03
2003-07
2003-11
2004-03
2004-07
2004-11
2005-03
2005-07
2005-11
2006-03
2006-07
2006-11
2007-03
2007-07
2007-11
2008-03
2008-07
2008-11
2009-03
2009-07
2009-11
2010-03
2010-07
2010-11
2011-03
2011-07
2011-11
2012-03
2012-07
2012-11
2013-03
2013-07
2013-11
2014-03
2014-07
RESTful
Port 43

19. Whois via IPv6
19
Percentage of traffic over IPv6
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
2009-01
2009-03
2009-05
2009-07
2009-09
2009-11
2010-01
2010-03
2010-05
2010-07
2010-09
2010-11
2011-01
2011-03
2011-05
2011-07
2011-09
2011-11
2012-01
2012-03
2012-05
2012-07
2012-09
2012-11
2013-01
2013-03
2013-05
2013-07
2013-09
2013-11
2014-01
2014-03
2014-05
2014-07
V6 Percentage
V6 Percentage

20. IRR Maintainers
2011 2012 2013 2014
2500
2000
1500
1000
500
0
Maintainers 1726 1850 1951 2102
20

21. IRR Route / Route6
Route
Route6
2011 2012 2013 2014
100000
10000
1000
100
10
1
Route 18636 19969 21204 23535
Route6 242 527 698 1072
21

22. IRR InetNum / Inet6Num
InetNum
Inet6Num
2011 2012 2013 2014
1000
100
10
1
InetNum 419 481 531 621
Inet6Num 13 25 38 51
22

23. Systems at the forefront
• RPKI
– Up/Down available – no takers so far
– Removed web delegated
– Upgraded the HSM’s to IBM 4765
• RDAP (IETF WEIRDS)
– “Soon” will be an RFC
– Public testbed
http://rdappilot.arin.net/restfulwhois/rdap
– Work is underway to make it production
– ARIN has open source software at
http://projects.arin.net
– Other RIRs are also deploying RDAP
23

24. Systems (cont..)
• We are a small engineering shop
– Lots of demands
– Attempting to provide exceptional service
• Creating API’s to core services
– Allows YOU to create tools
– Allows YOU to follow your timeline
• projects.arin.net (ACSP completed years ago)
– If you find your tool is cool
– Way to allow others to come find and use it
– We had one taker so far since ARIN 33 with a new
tool (ArinWhois.NET)
24

25. What we have accomplished
since ARIN 33
• Finished up more ACSPs
• DNSSEC on forward zones (arin.net/arin.com)
• Moved RPKI to a new HSM
• Making DNS changes near real-time
• Started automation on transfers
• Moving core production from ARIN HQ to colo
• Moving SAN from EMC to NetApp
25

26. What we are working on through
2015 Q2 (provisional)
• Moving the RDAP pilot into production *
• Further automation on transfers *
• Complete core production from ARIN HQ to colo *
• Complete migration of SAN from EMC to NetApp *
• Add Links to Whois Query Responses (ACSP 2011.29)
• Change Whois Output for Certain /8 Records (ACSP
2013.4)
• Start “SWIP Easy” – a web-based tool to send in
reassignment information
• Deploy Two-Factor Authentication (ACSP 2013.8)
* Part of 2014 Board-approved Operating Plan
26

27. 27
Comments?