Engineering Report
Mark Kosters
Staffing
• Operations
– 7 operations engineers + 2 managers (AT FULL
STRENGTH)
• Development
– 8 programmers + manager (AT FULL STRENGTH)
• New PM taken from engineering
• New hire - filled vacancy going to PM
• Quality Assurance
– 4 engineers, 1 contractor + manager (One
vacancy)
• Project Management – 1 (AT FULL STRENGTH)
• CTO – 1 (Working more on weights to be FULL
STRENGTH)
2
YTD Efforts
• Focus on ACSPs
• Work underway for sharing ticket information
– Important for transfers
• RPKI
– Mopping up work
– Migration from IBM 4764 to IBM 4765 HSMs
• Migration from Oracle to PostgreSQL complete!
• Movement from EMC to NetApp underway
3
YTD Efforts Cont…
• DNSSEC
– Making updates near realtime
– Hardening of key management
• Fault Tolerance Improvements
– More efficient system backups
– Moving Production Systems from ARIN HQ to Colo
– Moving backend services to real hardware when
merited
• Corporate Help Desk and IT Support
• ARIN Member Meeting Support
• Care and Feeding of Servers & Network
– Includes new systems provisioning with Puppet and
Foreman
• OT&E
4
OT&E
• Operational Test & Evaluation
– Place to test code
– Place to test process
• Replicated Core services
– Reg-RWS (provisioning API)
– Whois-RWS (directory API)
– Web Interface
– RPKI suite
5
YTD Efforts Cont…
• Security Audit by Foreground Security
• IETF Participation
– SIDR, RPKI GTA, WEIRDS (RDAP)
• ICANN Participation
– SSAC
– RSSAC
– Technical Advisory Group
6
YTD Efforts PostgreSQL
• We had a successful conversion
• Validation of 100% of all data from Oracle to PostgreSQL
– Hope to make the tool publically available for other parties
– Data integrity was paramount
• Noted in the PostgreSQL community
– High Availability (HA) talk at PGConf NYC 2014
– Exercised HA in the first week with a hardware failure on
the production DB node – no issues
• Did have one failure
– Installed rsyslog for centralized logging
– Ran into a buffering problem that occurred after a long run
– Resulted in short PostgreSQL outages on 2/15 and 2/25
7
ARIN Online Usage
• 81,984 accounts activated since
inception through Q1 of 2014
8
2008
2009
2010
2011
2012
2013
2014*
Number of Accounts Activated
5000 10000 15000 20000
* Through Q1 of 2014
Active Usage of ARIN Online
9
0
10000
20000
30000
40000
0 1 2 - 5 6 - 10 11 - 15 >16
Logins
#ofUsers
Times logged in
• Logins from inception through Q1 of 2014
Reg-RWS Transactions
10
ARIN 29 ARIN 30 ARIN 31 ARIN 32 ARIN 33
Template 658853 980068 1373933 1730163 2175889
REST 28373 319865 835914 3500958 4270946
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
4000000
4500000
Template
REST
Reports Via REST
Via REST
Associations 176
Reassignments 25,219
WhoWas 253,135
11
• Requests from inception through Q1 of 2014
RPKI Usage
ARIN XXX ARIN XXXI ARIN XXXII ARIN33
RPAs Signed 27 72 130 162
Certified Orgs 47 68 108
ROAs 19 60 106 162
Covered
Resources
30 82 147 258
Web
Delegated
0 0 0
Up/Down
Delegated
0 0
12
Whois Queries Per Second
13
0.00
500.00
1000.00
1500.00
2000.00
2500.00
3000.00
3500.00
4000.00
RESTful
Port 43
Whois via IPv6
14
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
2009-01
2009-03
2009-05
2009-07
2009-09
2009-11
2010-01
2010-03
2010-05
2010-07
2010-09
2010-11
2011-01
2011-03
2011-05
2011-07
2011-09
2011-11
2012-01
2012-03
2012-05
2012-07
2012-09
2012-11
2013-01
2013-03
2013-05
2013-07
2013-09
2013-11
2014-01
2014-03
Percentage of traffic over IPv6
IRR Maintainers
2011 2012 2013 2014
Maintainers 1726 1850 1951 2029
1550
1600
1650
1700
1750
1800
1850
1900
1950
2000
2050
2100
15
IRR Route / Route6
2011 2012 2013 2014
Route 18636 19969 21204 22370
Route6 242 527 698 871
1
10
100
1000
10000
100000
Route
Route6
16
IRR InetNum / Inet6Num
2011 2012 2013 2014
InetNum 419 481 531 556
Inet6Num 13 25 38 43
1
10
100
1000
InetNum
Inet6Num
17
Interops
• RPKI
– Up/Down now available – first use will probably be
between the RIR’s
– Will begin interop using Up/Down for ERX space
when APNIC is ready
• RDAP (IETF WEIRDS)
– Participated in public interop with APNIC, RIPE NCC,
LACNIC, Afilias, VeriSign, CNNIC at IETF 89
– ARIN has open source software at
http://projects.arin.net
– Public testbed at
http://rdappilot.arin.net/restfulwhois/rdap
– Other RIRs are following suit
18
RDAP
• Started at ARIN
• Other RIR’s found it interesting
• ICANN immensely interested
– Solves internationalized character problem
– Structured data (no complicated parsing
needed to get what you need)
– Navigation (no need to remember all these
whois sites)
– Ability to run over a validated channel (https)
– Ability to provide access control (allows for
partitioning of data and more privacy controls)
19
One of our Focuses
• We are a small engineering shop
– Lots of demands
– Attempting to provide exceptional service
• Creating API’s to core services
– Allows YOU to create tools
– Allows YOU to follow your timeline
• projects.arin.net (ACSP completed years ago)
– If you find your tool is cool
– Way to allow others to come find and use it
20
What we are working on
• Finish up more ACSPs
• DNSSEC on forward zones (arin.net/arin.com)
• Making DNS changes near real-time
• Moving the RDAP pilot into production
• Further automation on transfers
• Moving core production from ARIN HQ to colo
• Moving SAN from EMC to NetApp
21
22
Comments?

ARIN Engineering Department Report

  • 1.
  • 2.
    Staffing • Operations – 7operations engineers + 2 managers (AT FULL STRENGTH) • Development – 8 programmers + manager (AT FULL STRENGTH) • New PM taken from engineering • New hire - filled vacancy going to PM • Quality Assurance – 4 engineers, 1 contractor + manager (One vacancy) • Project Management – 1 (AT FULL STRENGTH) • CTO – 1 (Working more on weights to be FULL STRENGTH) 2
  • 3.
    YTD Efforts • Focuson ACSPs • Work underway for sharing ticket information – Important for transfers • RPKI – Mopping up work – Migration from IBM 4764 to IBM 4765 HSMs • Migration from Oracle to PostgreSQL complete! • Movement from EMC to NetApp underway 3
  • 4.
    YTD Efforts Cont… •DNSSEC – Making updates near realtime – Hardening of key management • Fault Tolerance Improvements – More efficient system backups – Moving Production Systems from ARIN HQ to Colo – Moving backend services to real hardware when merited • Corporate Help Desk and IT Support • ARIN Member Meeting Support • Care and Feeding of Servers & Network – Includes new systems provisioning with Puppet and Foreman • OT&E 4
  • 5.
    OT&E • Operational Test& Evaluation – Place to test code – Place to test process • Replicated Core services – Reg-RWS (provisioning API) – Whois-RWS (directory API) – Web Interface – RPKI suite 5
  • 6.
    YTD Efforts Cont… •Security Audit by Foreground Security • IETF Participation – SIDR, RPKI GTA, WEIRDS (RDAP) • ICANN Participation – SSAC – RSSAC – Technical Advisory Group 6
  • 7.
    YTD Efforts PostgreSQL •We had a successful conversion • Validation of 100% of all data from Oracle to PostgreSQL – Hope to make the tool publically available for other parties – Data integrity was paramount • Noted in the PostgreSQL community – High Availability (HA) talk at PGConf NYC 2014 – Exercised HA in the first week with a hardware failure on the production DB node – no issues • Did have one failure – Installed rsyslog for centralized logging – Ran into a buffering problem that occurred after a long run – Resulted in short PostgreSQL outages on 2/15 and 2/25 7
  • 8.
    ARIN Online Usage •81,984 accounts activated since inception through Q1 of 2014 8 2008 2009 2010 2011 2012 2013 2014* Number of Accounts Activated 5000 10000 15000 20000 * Through Q1 of 2014
  • 9.
    Active Usage ofARIN Online 9 0 10000 20000 30000 40000 0 1 2 - 5 6 - 10 11 - 15 >16 Logins #ofUsers Times logged in • Logins from inception through Q1 of 2014
  • 10.
    Reg-RWS Transactions 10 ARIN 29ARIN 30 ARIN 31 ARIN 32 ARIN 33 Template 658853 980068 1373933 1730163 2175889 REST 28373 319865 835914 3500958 4270946 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000 Template REST
  • 11.
    Reports Via REST ViaREST Associations 176 Reassignments 25,219 WhoWas 253,135 11 • Requests from inception through Q1 of 2014
  • 12.
    RPKI Usage ARIN XXXARIN XXXI ARIN XXXII ARIN33 RPAs Signed 27 72 130 162 Certified Orgs 47 68 108 ROAs 19 60 106 162 Covered Resources 30 82 147 258 Web Delegated 0 0 0 Up/Down Delegated 0 0 12
  • 13.
    Whois Queries PerSecond 13 0.00 500.00 1000.00 1500.00 2000.00 2500.00 3000.00 3500.00 4000.00 RESTful Port 43
  • 14.
  • 15.
    IRR Maintainers 2011 20122013 2014 Maintainers 1726 1850 1951 2029 1550 1600 1650 1700 1750 1800 1850 1900 1950 2000 2050 2100 15
  • 16.
    IRR Route /Route6 2011 2012 2013 2014 Route 18636 19969 21204 22370 Route6 242 527 698 871 1 10 100 1000 10000 100000 Route Route6 16
  • 17.
    IRR InetNum /Inet6Num 2011 2012 2013 2014 InetNum 419 481 531 556 Inet6Num 13 25 38 43 1 10 100 1000 InetNum Inet6Num 17
  • 18.
    Interops • RPKI – Up/Downnow available – first use will probably be between the RIR’s – Will begin interop using Up/Down for ERX space when APNIC is ready • RDAP (IETF WEIRDS) – Participated in public interop with APNIC, RIPE NCC, LACNIC, Afilias, VeriSign, CNNIC at IETF 89 – ARIN has open source software at http://projects.arin.net – Public testbed at http://rdappilot.arin.net/restfulwhois/rdap – Other RIRs are following suit 18
  • 19.
    RDAP • Started atARIN • Other RIR’s found it interesting • ICANN immensely interested – Solves internationalized character problem – Structured data (no complicated parsing needed to get what you need) – Navigation (no need to remember all these whois sites) – Ability to run over a validated channel (https) – Ability to provide access control (allows for partitioning of data and more privacy controls) 19
  • 20.
    One of ourFocuses • We are a small engineering shop – Lots of demands – Attempting to provide exceptional service • Creating API’s to core services – Allows YOU to create tools – Allows YOU to follow your timeline • projects.arin.net (ACSP completed years ago) – If you find your tool is cool – Way to allow others to come find and use it 20
  • 21.
    What we areworking on • Finish up more ACSPs • DNSSEC on forward zones (arin.net/arin.com) • Making DNS changes near real-time • Moving the RDAP pilot into production • Further automation on transfers • Moving core production from ARIN HQ to colo • Moving SAN from EMC to NetApp 21
  • 22.