The document discusses things the author wishes they had known about IPv6 before starting to implement it for their small provider network. It covers IPv6 justification in terms of IPv4 address scarcity and rising costs, advice on IPv6 addressing plans and transition technologies, and gotchas like IPv6 neighbor discovery exhaustion issues. The author advocates for embracing IPv6 to avoid expensive IPv4 solutions and make the most of the large IPv6 allocations provided.

1. THINGS I WISH
I HAD KNOWN ABOUT IPv6
BEFORE I STARTED
MAREK ISALSKI — FAELIX — UKWISPA MEMBERS' MEETING 2019-06-13
https://faelix.link/ukwispa201906

2. THINGS I WISH
I HAD KNOWN ABOUT IPv6
BEFORE I STARTED
WARNING: MIGHT CONTAIN A BIT OF VENDOR BASHING
https://faelix.link/ukwispa201906

3. About Marek
Stuff I do:
CTO @FAELIX – https://faelix.net/
PC @uknof – https://uknof.uk/
PC @net_mcr – https://www.netmcr.uk/
Trail of SSIDs in my wake: "AS41495 Faelix Limited"
Me — @maznu – @NetworkMoose – @IPv6HULK

4. About Marek
Stuff I do:
CTO @FAELIX – https://faelix.net/
PC @uknof – https://uknof.uk/
PC @net_mcr – https://www.netmcr.uk/
Trail of SSIDs in my wake: "AS41495 Faelix Limited"
Me — @maznu – @NetworkMoose – @IPv6HULK

5. This Talk
IPv6 Justification
IPv6 Advice
IPv6 Gotchas
IPv6 Evangelism

6. “This memo documents the
fundamental truths of networking for
the Internet community.”
– RFC1925

7. RFC1925 #10 (redux)
ONE SIZE NEVER FITS ALL

8. Problem Statement
Small provider network (100-10000 customers)
Mixed B2B and B2C
With its own AS, /22 IPv4, /29-/32 IPv6 (i.e. an LIR)
Even where incumbents cannot provide access,
customer expectations of price/performance remain

9. Problem Statement
Small provider network (100-10000 customers)
Mixed B2B and B2C
With its own AS, /22 IPv4, /29-/32 IPv6 (i.e. an LIR)
Even where incumbents cannot provide access,
customer expectations of price/performance remain

10. Problem Statement
Small provider network (100-10000 customers)
Mixed B2B and B2C
With its own AS, /22 IPv4, /29-/32 IPv6 (i.e. an LIR)
Even where incumbents cannot provide access,
customer expectations of price/performance remain

11. IPv6 JUSTIFICATION

12. RFC1925 #9 (redux)
FOR ALL RESOURCES, WHATEVER
IT IS, YOU NEED MORE

13. Credit: @lunarsynthesis

14. “Grant me the optimism of this lorry,
attempting to slowly advance beneath
a low bridge, but the wisdom to know
when it's time to reverse.”
– @lunarsynthesis

15. IPv4 Market Group,
UKNOF33, January 2016
IPv4 Life Cycle
• Events
– ARIN Runout
– RIPE Inter-RIR Transfers
– /8s come to market
– IPv6 Adoption
2

16. IPv4 Market Group,
UKNOF33, January 2016

17. IPv4 Market Group,
Website, July 2018

18. IPv4 Market Group,
Website, July 2018
$18
$10

19. IPv4 Market Group,
UKNOF33, January 2016
are we here yet?

20. “…a plateau before $20 seems
unlikely because demand is currently
greater than supply, and there are not
as many sellers coming to market…”
– IPv4 Market Group

21. “…a plateau before $20 seems
unlikely because demand is currently
greater than supply, and there are not
as many sellers coming to market…”
– IPv4 Market Group
per-customer capex

22. Trade-Off
Engineering time vs capital expenditure
Training, tooling, support
Existing backhaul/infrastructure
Backfilling existing customers
Transition technology costs vs legacy costs

23. End-to-end connectivity
IPv4 + NAT is not security
NAT gives you stateful firewalling
CG-NAT puts that state in the core of the network
IPv6 has stateful firewalling too!
But now the state is in each customer's router

24. Where is the state?
CGNAT
router
customer
routers
transit
router

25. Where is the state?
customer
router
transit
router

26. Why I think WISPs should
embrace IPv6
"Small" networks
"Local" networks
Exist to fill gaps in competitive market
No "triple-play" offer (but increasingly irrelevant?)
Does 5G feel like a threat?

27. Why I think WISPs should
embrace IPv6
IPv4 CG-NAT is stateful and therefore expensive
More IPv4 addresses are also expensive
IPv6 already deployed on the big content networks
And plenty of eyeball devices support IPv6
Your IPv6 allocation is mindbogglingly big
"Less bullshit, more engineering"

28. HAVE A PLAN

29. /22 vs /29
000000000000000000000000001024 IPv4
633825300114114700748351602688 IPv6

30. Address Plan BCOPs
RFC 7381
RFC 5375
ripe-690
e.g. Veronika McKillop at UKNOF35
Chair of UK IPv6 Council

31. MAREK'S
"STARTER FOR TEN"

32. Resi IPv6 Address Plan
/29
is 8 /32s
each of which is 16777216 /56s
each of which is 256 /64s

33. Resi IPv6 Address Plan
/29
is 8 /32s
each of which is 16777216 /56s
each of which is 256 /64s
who has this many
connected homes?

34. Resi IPv6 Address Plan
/29
is 8 /32s
each of which is 16777216 /56s
each of which is 256 /64s
who has a residential customer
with this many subnets?

35. Biz IPv6 Address Plan
/29
is 8 /32s
each of which is 65536 /48s
each of which is 65536 /64s

36. Biz IPv6 Address Plan
/29
is 8 /32s
each of which is 65536 /48s
each of which is 65536 /64s
who has this
many business
customers?

37. Biz IPv6 Address Plan
/29
is 8 /32s
each of which is 65536 /48s
each of which is 65536 /64s
please find me the "IT Network
Manager" hero who
configured this many subnets

38. IPv6 = THE NETWORK
IPv4 = NECESSARY LEGACY

39. Dual Stack

40. Transition Technology

41. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
startrek.com?

42. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
64:ff9b::9765:41bc

43. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
64:ff9b::9765:41bc

44. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
64:ff9b::9765:41bc

45. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
64:ff9b::9765:41bc

46. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
64:ff9b::9765:41bc
151 . 101 . 65 . 188

47. NAT64 + DNS64
NAT64
router
IPv6-only
customer
IPv4+IPv6
transit router
151.101.65.188

48. WHERE CAN I GET IT?

49. NAT64 Vendor Support
MikroTik = nope
Ubiquiti = nope
VyOS = nope
Quagga/FRR or BIRD = DIY
Cisco ASA / ASR = yes!
Juniper = yes!

50. NAT64 Vendor Support
MikroTik = nope
Ubiquiti = nope
VyOS = nope
Quagga/FRR or BIRD = DIY
Cisco ASA / ASR = yes!
Juniper = yes!
DIY:
TAYGA
WrapSix
Ecdysis
Jool
OpenBSD
BIND

51. RFC1925 #8
IT IS MORE COMPLICATED
THAN YOU THINK

52. Transition Technologies
NAT64 + DNS64
4rd
464XLAT
DS-lite

53. Transition Technologies
End customer doesn't have a static IPv4
Can't do port forwarding, DMZ, etc
CCTV/etc might be a challenge
Some protocols signal IPv4 address literals within
the packet payloads
ALGs for SIP
Skype, others

54. IPv6 GOTCHAS

55. 2017-09-17: UNM.EDU

58. Actions
Emailed abuse
Blocked their scanner
Went back to sleep

59. 2018-03-31: BERKELEY

60. "We've blocked your /48 from our
network because the IPv6 scanning
you are performing against
2a01:9e00::/32 is aggressive."
– email to cesr-scanning@eecs.berkeley.edu, 2018-03-31

61. Actions
Emailed abuse and project contact in WHOIS
Blocked their scanner
Went back to sleep

62. Actions
Emailed abuse and project contact in WHOIS
Blocked their scanner
Went back to sleep
…got an email back from them!

63. Discussed with Berkeley
AM: "smart scanning techniques […] measurement
research […] probe a large set of hosts on the
Internet "
MI: "slipshod IPv6 implementations"
AM: "based on RFC7707 and RFC6583 we have
decided to add a module to our scanner that will rate
control probes sent to each /64 in addition to each
routed prefix"

64. IPv6 SCANNING EXISTS!

65. Scanning Projects/Papers
E Vyncke, UK IPv6 Council, 2014: https://www.ipv6.org.uk/
wp-content/uploads/2018/10/evyncke-UK-council-IPv6-
security.pdf
Chris Grundemann, 2015: https://
www.internetsociety.org/blog/2015/02/ipv6-security-
myth-4-ipv6-networks-are-too-big-to-scan/
RIPE74, 2017: http://www.entropy-ip.com/
6Gen, Berkeley: https://conferences.sigcomm.org/imc/
2017/papers/imc17-final245.pdf
ZMapv6: https://ipv6hitlist.github.io/

66. CVE-2018-19298

67. IPv6 Neighbour Discovery
transit
router
customer
equipment
security
project

68. Neighbour Solicitation
transit
router
customer
equipment
security
project
NS
135

69. Neighbour Advertisement
transit
router
customer
equipment
security
project
NA
136

70. Massive Address Space
/64
transit
router
customer
equipment
security
project

71. Neighbour Solicitation
/64
transit
router
customer
equipment
security
project
NS
135

72. Neighbour Solicitations
/64
transit
router
customer
equipment
security
project
NS
135
NS
135
NS
135
NS
135
NS
135

73. Neighbour Table Fills Up
/64
transit
router
customer
equipment
security
project
NS
135
NS
135
NS
135
NS
135
NS
135

74. Oldest Entry is Dropped
/64
transit
router
customer
equipment
security
project
???

75. Loss of Connectivity…
/64
transit
router
customer
equipment
security
project
???

76. Re-Discover Neighbour
/64
transit
router
customer
equipment
security
project
NS
135

77. Neighbour Advertisement
/64
transit
router
customer
equipment
security
project
NA
136

78. …until next time!
/64
transit
router
customer
equipment
security
project
NS
135
NS
135
NS
135
NS
135
NS
135
???

79. CVE-2018-19298: IPv6
NEIGHBOUR EXHAUSTION
(MIKROTIK ROUTEROS v6)

80. CVE-2018-19298 Timeline
2018-04-08 — reported to vendor
2018-06-29 — "not yet fixed"
2018-11-15 — CVE assigned
2019-01-15 — "can not give you any ETA for the fix"
2019-02-14 — discussion at NetMcr
2019-03-31 — lots of stuff happens
2019-04-09 — wider disclosure

81. Nothing to See Here
CVE-2018-19298 is not that new, fundamentally
Most vendors have fixes for NDP exhaustion
Could just not use /64 subnets
…except for Android not having DHCPv6
…so you rely on IPv6 RA
…and so you probably have /64 subnets (RFC7421)
But at least not having /64 linknets would save core
routers from short-lived loss of adjacency (RFC6164)

82. Nothing to See Here
CVE-2018-19298 is not that new, fundamentally
Most vendors have fixes for NDP exhaustion
Could just not use /64 subnets
…except for Android not having DHCPv6
…so you rely on IPv6 RA
…and so you probably have /64 subnets (RFC7421)
But at least not having /64 linknets would save core
routers from short-lived loss of adjacency (RFC6164)
another
vendor
"gotcha"

83. “simplistic implementations of [ND]
can be vulnerable to deliberate or
accidental [DoS], whereby they attempt
to perform address resolution for
large numbers of unassigned […]”
– RFC6583, Operational Neighbor Discovery Problems, 2012

84. “simplistic implementations of [ND]
can be vulnerable to deliberate or
accidental [DoS], whereby they attempt
to perform address resolution for
large numbers of unassigned […]”
– RFC6583, Operational Neighbor Discovery Problems, 2012

85. Conclusion
IPv6 support in RouterOS needs some love:
>6 year old ops-experience RFCs unaddressed
MikroTik RouterOS v6 is (very patched) Linux 3.3.5

86. THE END!

87. “…only it wasn't the end…”
– Narrator

88. Voyage of Discovery
While trying to test IPv6 ND exhaustion…
Using a VPS and some "l33t t00lz"…
Aimed at a /64 subnet behind a MikroTik hAP…

89. IPv6 Neighbour Discovery
transit
router
victim
router
Attack
VPS
target
/64
ICMP
v6
ICMP
v6
NS
135

90. IPv6 Neighbour Discovery
transit
router
victim
router
Attack
VPS
target
/64
ICMP
v6
ICMP
v6
NS
135

91. IPv6 Neighbour Discovery
transit
router
victim
router
Attack
VPS
target
/64
ICMP
v6
ICMP
v6

92. IPv6 Neighbour Discovery
transit
router
victim
router
Attack
VPS
target
/64
ICMP
v6

93. transit
router
victim
router
Attack
VPS
target
/64
ICMP
v6
?? W
? T
F!!!

94. CVE-2018-19299

95. CVE-2018-19299 Timeline
2018-04-16 — reported to vendor
2018-04-17 — [this is ND exhaustion]
2018-04-17 — no, it isn't
2018-04-17 — [yes it is]
2018-04-17 — no, it isn't
2018-04-17 — [it is! you used an NDP exhaust tool!]

96. CVE-2018-19299 Timeline
2018-04-16 — reported to vendor
2018-04-17 — [this is ND exhaustion]
2018-04-17 — no, it isn't
2018-04-17 — [yes it is]
2018-04-17 — no, it isn't
2018-04-17 — [it is! you used an NDP exhaust tool!]
2018-04-17 — …no! I'm begging you! It isn't NDP!

97. CVE-2018-19299 Timeline
2018-04-17 — "Sorry for confusion, dst is two hops
away. We will test this scenario."

98. …

99. CVE-2018-19299 Timeline
2018-04-19 — "forwarding of ipv6 traffic eats all the
memory"

100. YES, REALLY.

101. The worries…
multiple problems?

102. The response…

103. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"

104. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "we accept this as a bug, but we
would not call it a vulnerability"

105. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "we accept this as a bug, but we
would not call it a vulnerability"
2018-11-15 — "with our development team"
UKWISPA

106. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "we accept this as a bug, but we
would not call it a vulnerability"
2018-11-15 — "with our development team"
2019-01-15 — "can not give you any ETA for the fix"

107. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "we accept this as a bug, but we
would not call it a vulnerability"
2018-11-15 — "with our development team"
2019-01-15 — "can not give you any ETA for the fix"
NetMcr discuss

108. NetMcr 2019-02-14
Explained IPv6 NDP exhaustion
Spoke about how this is CVE-2018-19298
Presented initial version of first half of this talk
Did not give details of exploit of CVE-2018-19299
Asked the audience, "What next?"
Continue to aggro the vendor?
Publish full details in MITRE?
Make noise in the technology press?

109. NetMcr 2019-02-14
Decided plan for way forward:
Notify vendor of publication date (2019-04-09)
Get word out (notify NCSC, CISP, CERTs, etc)
Prepare for move to full disclosure

110. I shall be discussing IPv6 neighbor discovery exhaustion, and also
how RouterOS will crash when routing IPv6 packets, i.e. both
vulnerabilities I have disclosed to MikroTik in April 2018,
currently unpublished as CVE-2018-19298 and CVE-2018-19299.
Do you think that MikroTik will have an update about these
vulnerabilities that I can include in my presentation on April 9th?
– email to MikroTik support, 2019-03-04

111. "At the moment there is no news,
but I will definitely let you know
as soon as there will be an update
regarding this matter."
– email from MikroTik support, 2019-03-11

112. CVE-2018-19299 Timeline
2018-04-19 — "ipv6 traffic eats all the memory"
2018-04-19 — "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "we accept this as a bug, but we
would not call it a vulnerability"
2018-11-15 — "with our development team"
2019-01-15 — "can not give you any ETA for the fix"
2019-03-11 — "there is no news"

113. MEANWHILE…

114. CVE-2018-19299 in Wild?
mar/09/2019 06:58:04 system,error,critical router was rebooted
without proper shutdown, probably kernel failure
mar/09/2019 06:58:04 system,error,critical kernel failure in previous
boot
mar/09/2019 06:58:04 system,error,critical out of memory condition
was detected
mar/10/2019 16:56:18 system,error,critical router was rebooted
without proper shutdown, probably kernel failure
mar/10/2019 16:56:18 system,error,critical kernel failure in previous
boot
mar/10/2019 16:56:18 system,error,critical out of memory condition
was detected

115. production MikroTik router at AS41495 edge
graph of free memory vs time
first two weeks of March 2019
scraped every 30 seconds by API into Prometheus

118. 230Mb RAM eaten
in ~20 mins

119. 230Mb RAM eaten
in ~20 mins
110Mb
in <4 mins

120. 230Mb RAM eaten
in ~20 mins
110Mb
in <4 mins
BGP
reloading

121. Notify vendor of exploits?

122. “Yes, it is highly possible,
however, we would prefer to not
jump to conclusions without
seeing an actual file.”
– email from MikroTik support, 2019-03-21

123. “Sadly, I will not be able to provide any
supouts showing IPv6 crashes - we are
removing MikroTik from our IPv6 transit
network entirely, because you have
not taken this bug seriously.”
– email to MikroTik support, 2019-03-21

124. “Sadly, I will not be able to provide any
supouts showing IPv6 crashes - we are
removing MikroTik from our IPv6 transit
network entirely, because you have
not taken this bug seriously.”
– email to MikroTik support, 2019-03-21

125. UBNT TO THE RESCUE?

126. One Mitigation We Adopted
IPv6
Ubiquiti
destinationupstream
IPv4
MikroTik

127. How EdgeOS Fared
Some MikroTik/Ubiquiti interop issues (OSPFv3)
Does it have fastpath?
[cannot] enable IPv6 offloading for PPPoE and
VLANs simultaneously.
Some self-made gotchas involving IPv6 RA
IPv6 firewall maybe not as efficient for BCP38
Doesn't suffer some of RouterOS' IPv6 routing bugs
On the whole: pretty good!

128. BACK TO MIKROTIK…

129. CVE-2018-19299 Timeline
2018-04-16 — reported to vendor
2018-04-19 — acknowledged by vendor as "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "not […] a vulnerability"
2018-11-15 — CVE assigned; "with our development team"
2019-01-15 — "can not give you any ETA for the fix"
2019-02-14 — V-Day 0-day discussion @net_mcr
2019-03-11 — "there is no news"
2019-03-14 — last ditch scrabble around for CERTs/etc
2019-04-09 — full disclosure @uknof

130. CVE-2018-19299 Timeline
2018-04-16 — reported to vendor
2018-04-19 — acknowledged by vendor as "not a security vulnerability"
2018-06-29 — "not yet fixed"
2018-10-10 — "not […] a vulnerability"
2018-11-15 — CVE assigned; "with our development team"
2019-01-15 — "can not give you any ETA for the fix"
2019-02-14 — V-Day 0-day discussion @net_mcr
2019-03-11 — "there is no news"
2019-03-14 — last ditch scrabble around for CERTs/etc
2019-04-09 — full disclosure @uknof
then things got busy

131. CVE-2018-19299 Timeline
2019-03-27 14:08 — "UKNOF 43 CVE" topic starts to MikroTik forum
2019-03-28 11:37 — TechRepublic starts coverage
2019-03-28 11:57 — another thread starts on forum; multiple Reddits
2019-03-28 11:50 — MikroTik: "we are aware […] working on it"
2019-03-29 07:56 — MikroTik: "we aim to fix before [UKNOF]"
2019-03-29 08:02 — MI: "contact me privately" (via forum)
2019-03-29 11:00 — @mikrotik_build: "version 6.45beta22" claims fix
2019-03-29 11:23 — @maznu: "not fixed"
2019-03-29 11:35 — MI: "not fixed" (via forum)
2019-03-29 12:17 — MikroTik: "please clarify" (via forum)
then things got weird

132. “It’s MikroTik’s fault that this was filed
as yet another ipv6 bug […] The issue
is now fixed, the memory exhaustion
is also fixed, build is coming Monday.”
– @normis on Twitter, 2019-03-30 12:36

133. Condensed Timeline
2019-03-29 11:00 — 6.45beta22 (not a fix)
2019-03-29 14:46 — workaround for other issues
2019-03-29 14:09 — "next beta version"
2019-03-30 12:36 — "build is coming Monday"
2019-04-01 ??:?? — release fix for CVE-2018-19299

134. Condensed Timeline
2019-03-29 11:00 — 6.45beta22 (not a fix)
2019-03-29 14:46 — workaround for other issues
2019-03-29 14:09 — "next beta version"
2019-03-30 12:36 — "build is coming Monday"
2019-04-01 ??:?? — release fix for CVE-2018-19299
what goes here?

135. “RouterOS IPv6 route cache max size
by default is 1 million. […] If you have
device that does not have such
resources, it will reboot itself.”
– forum post by MikroTik, 2019-03-31 13:28

136. A Customer's Reaction

137. “However, it can not be considered as
a bug or vulnerability. […]
This is not a bug.”
– forum post by MikroTik, 2019-03-31 13:28

138. WHAT CAME OF –19299?

140. Changelog
v6.44.2 — v6.45beta23 — v6.43.14
ipv6 - fixed soft lockup when forwarding IPv6
packets
ipv6 - fixed soft lockup when processing large IPv6
Neighbor table
ipv6 - adjust IPv6 route cache max size based on
total RAM memory

142. MikroTik Security
"Responsible disclosure of discovered
vulnerabilities"
"If you have found such a security flaw, we would
like to hear more about it to be able to correct the
problem as soon as possible."
"We promise you that: […]"
When contacting MikroTik about vulnerabilities,
please use the e-mail address
security@mikrotik.com

143. IPv6 EVANGELISM

144. IPv6 is here to stay

145. IPv6 is here to stay

146. IPv4 is expensive legacy
$18
$10

147. IPv4 is almost all gone

148. IPv6 is in production
Sky IPv6 Roll-Out, I. Dickinson, UKNOF33
EE IPv6 Only, N. Heatley, UKNOF36
Virgin Media, L. Olopade, UK IPv6 Council (Dec 2018)
Psychology of IPv6, V. McKillop, UKNOF43

149. CHAT OVER
A BREW?
E: marek @ faelix . net
T: @maznu
T: @faelix
W: https://faelix.net/
https://faelix.link/ukwispa201906