PLNOG16: Mix 2-in-1: IPv6 troubleshooting for helpdesks - and –DANE/DNSSEC/TLS experiments from Go6la, Jan Zorz

1. Mix 2-in-1:
IPv6 troubleshooting for helpdesks
- and –
DANE/DNSSEC/TLS experiments from Go6lab
Jan Žorž <zorz@isoc.org>

2. About Me – A Quick History
Name: Jan Žorž <zorz@isoc.org>
Founder of Slovenian Go6 Institute
Worked in Internet operations for
20+ years
18 years of IPv6 experience
Active and contributing member of RIPE and IETF
communities
Primary co-author of RIPE-501/RIPE-554 IPv6 procurement
BCP document and RIPE-631 IPv6 helpdesk document
Co-author of RFC 6346 (A+P approach to IPv4 depletion)
Joined Internet Society DO team in December 2012

3. Walls-of-text warning
Despite long contemplations on how to make this slide-
pack without walls-of-text – we could not find any better
way how to tell our story.
We apologize for any inconvenience. 
Please, bear with us anyway…

4. Misconceptions and reality ;)
• Disclaimer: This document is *not* an ISOC
document. This is a product of a group of a brilliant
experts from the community (list with names comes
later in the slide pack).
• I just happen to be the initiator and one of the editors
of the document
• I would like to thanks ISOC for dedicating some of
my working time to run this effort.

5. Removing one of the next IPv6 speedbumps
• One of the first speed-bumps was addressed by
RIPE-554
• Next speed-bump is lack of IPv6 knowledge at ISP
helpdesks

6. IT helpdesk staff can be… difficult sometimes 

7. Tools used?
• Fact 1.: We need to build a short and simple set of
detect/explain/action scenarios that would help
people at help desks identify and fix the issue
• Fact 2.: We need a simple online tool to detect the
state of connectivity on the users computer
• Fact 3.: Test-ipv6.com is a very useful tool that
detects the state of connectivity on the users
computer
• So the idea emerged to bring Jason Fesler in the
group and talk him into creating a special version of
the tool meant specially for ISP helpdesks 

8. Tools used?

9. Removing one of the next IPv6 speedbumps
Title: “IPv6 Troubleshooting for Residential ISP
Helpdesks (Using test-ipv6.com)”
Contributors and authors:
Lee Howard (Time Warner Cable),
John Jason Brzozowski (Comcast),
David Freedman (ClaraNET),
Jason Fesler (Yahoo!),
Tim Chown (University of Southampton),
Sander Steffann (SJM Steffann),
Chris Grundemann (ISOC),
Jen Linkova (Google),
Chris Tuska (Comcast),
Daniel Breuer (Comcast),
Jan Žorž (ISOC)

10. Table of content
1. What is a BCOP?
2. Summary
3. Background / History
4. Using This Document - Note for Helpdesk Managers
5. IPv6 Troubleshooting
5.1 Basic IPv6 Test
5.2 Test Connectivity
5.3 Test DNS
5.4 Check Home Router
5.5 Escalate
6. Explanation of Help Desk Codes on http://isp.test-
ipv6.com
112 - IPv4, plus Broken IPv6
4 - IPv4 only
4t - IPv4 plus Teredo
46 - IPv4 + IPv6
46t - Dual Stack, Possible Tunnel
624 - 6to4
64 - NAT64

11. Table of content
64t - NAT64, possible tunnel
“slow”
“mtu” - “Possible MTU issues” Warning
“Site(s) with failed connectivity” Warning
7. IPv6 training for helpdesk
8. Conclusion
9. Operator’s specifics
Appendix A: Acknowledgements
Appendix B. Basic troubleshooting flowchart
Appendix C. Collecting Data for Escalation

12. Example: one of the possible generic situations
Help desk code: 624
6to4
IPv4: Good, AS65536, CableCo
IPv6: Good, 6to4, Preferred
IPv4 address: 192.0.2.1
IPv6 address: 2001:db8::1

13. Example (continued):
Interpretation: “6to4” was used to provide an IPv6 address;
and the host was configured to actively take advantage of
this service. Any web site that has an IPv6 presence, will be
reached using 6to4 instead of native IPv4. Modern
operating systems do not prefer these kinds of tunnels by
default. Be aware that the user might have a very old
operating system or a non-default configuration.
Action: 1. Have the user disable any automatic tunneling
mechanisms that are active. 6to4 is a protocol that tries to
get IPv6 traffic through a public relay, using IPv4 to reach the
public relay. Public 6to4 relays offer no SLA; and published
studies show approximately 15% failure rates. Windows:
Disable tunnel interfaces using
http://support.microsoft.com/kb/929852 (for example,
Microsoft Fix it 50412).

14. Example (Action continued):
2. If IPv6 is desired, configure IPv6 (verify the user has an
IPv6 address, and a default route). and test again. If user is
still experiencing access issues, follow the troubleshooting
steps for the corresponding code returned by http://isp.test-
ipv6.com/

15. Example (hardest and longest one):
112 - IPv4, plus Broken IPv6
Help desk code: 112
IPv4: Good, AS65536, CableCo
IPv6: broken
IPv4 address: 192.0.2.1

16. Example (hardest and longest one):
Interpretation: IPv6 network connectivity somewhere
between the user and the website is broken. IPv6
connections are timing out instead of succeeding (or failing
fast to IPv4). The user experience visiting major web sites
may be suffering, and some applications completely failing.
Assumption: User has already power cycled home router,
modem, and device, as part of your standard troubleshooting
procedure.
Action: 1. Determine whether IPv6 is offered to this
customer, based on company documentation. If yes,
continue to the next step.
2. …

17. Very quick reminder: What’s a BCOP?
Best Current Operational Practice
•A living document describing the best
operational practices currently agreed on by
subject matter experts
•Vetted and periodically reviewed by the global
network engineering community (GNEC)

18. Quick facts in case you are interested:
BCOP activity around the world
http://www.internetsociety.org/deploy360/about/bcop/
• Africa region: A BCOP group was started under AfNOG,
lead by Douglas Onyango and Fiona Asonga
• Asia: BCOP Task Force started at JANOG, co-chaired
by Seiichi Kawamura and Yoshinobu Matsuzaki,
NZNOG BCOP starting up, lead by Dean Pemberton
• BoF at Apricot lately…
• Europe: RIPE BCOP Task Force created, co-chaired by
Benno Overeider and Jan Žorž
• Latin America: A BCOP Task Force was started under
LACNOG, lead by Luis Balbinot and Pedro R Torres Jr.
• North America: NANOG BCOP Committee established,
lead by Aaron Hughes and Chris Grundemann

19. Status and future work?
Document is now an official RIPE BCOP document – RIPE-
631
• Done in RIPE BCOP TF and subsequently to RIPE IPv6 WG
Join the mailing lists and contribute to discussion:
RIPE BCOP TF: https://www.ripe.net/mailman/listinfo/bcop
RIPE IPv6 WG ML: http://www.ripe.net/mailman/listinfo/ipv6-wg/
Issues/comments/ideas tracker URL:https://git.steffann.nl/go6/ipv6-
troubleshooting-for-helpdesks/issues
https://www.ripe.net/ripe/docs/ripe-631

20. https://www.ripe.net/ripe/docs/ripe-631

21. DANE/DNSSEC/TLS
Testing in the Go6lab
Jan Žorž, ISOC/Go6 Institute, Slovenia
jan@go6.si
zorz@isoc.org

22. Acknowledgement
I would like to thank Internet Society to let me
spend some of my ISOC working time in go6lab
and test all this new and exciting protocols and
mechanisms that makes Internet a bit better
and more secure place…

23. DNSSEC implementation in go6lab
• Powerdns server (used as primary for non-
signed domains) as “hidden” primary DNS
server
• OpenDNSSEC platform for signing domains
• BIND9 DNS servers as secondaries to
OpenDNSSEC to serve signed zones
• Virtualization used: PROXMOX 3.4
• OS templates: fedora-20, Centos6/7

24. DNSSEC implementation in go6lab
• “Bump in a wire”
• Two public “primary” servers
• Concept:

25. DNSSEC in go6lab
• That was fairly easy and it works very well.
• Implementation document used from Matthijs
Mekking:
http://go6.si/docs/opendnssec-start-guide-draft.pdf

26. DANE experiment
• When DNSSEC was set up and functioning we
started to experiment with DANE (DNS
Authenticated Name Entities).
• Requirements:
– DNSSEC signed domains
– Postfix server with TLS support > 2.11
• We decided on Postfix 3.0.1

27. DANE
• TLSA record for mx.go6lab.si
_25._tcp.mx.go6lab.si. IN TLSA 3 0 1
B4B7A46F9F0DFEA0151C2E07A5AD7908F4C8B0050E7CC
25908DA05E2 A84748ED
It’s basically a hash of TLS certificate on mx.go6lab.si
More about DANE:
http://www.internetsociety.org/deploy360/resources/da
ne/

28. What is DANE and how does it work

31. DANE verification
• Mx.go6lab.si was able to verify TLS cert to T-2
mail server and nlnet-labs and some others…
mx postfix/smtp[31332]: Verified TLS connection established to
smtp-good-in-2.t-2.si[2a01:260:1:4::24]:25: TLSv1 with cipher
DHE-RSA-AES256-SHA (256/256 bits)
dicht postfix/smtp[29540]: Verified TLS connection established to
mx.go6lab.si[2001:67c:27e4::23]:25: TLSv1.2 with cipher
ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)

32. Postfix config
smtpd_use_tls = yes
smtpd_tls_security_level = may
smtpd_tls_key_file = /etc/postfix/ssl/server.pem
smtpd_tls_cert_file = /etc/postfix/ssl/server.pem
smtpd_tls_auth_only = no
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
smtp_tls_security_level = dane
smtp_use_tls = yes
smtp_tls_note_starttls_offer = yes
smtp_tls_loglevel = 1
tls_random_exchange_name = /var/run/prng_exch
tls_random_source = dev:/dev/urandom
tls_smtp_use_tls = yes

33. Malformed TLSA record
• We created a TLSA record with a bad hash (one
character changed)
• Postfix failed to verify it and refused to send a message
mx postfix/smtp[1765]: Untrusted TLS connection established to
mail-bad.go6lab.si[2001:67c:27e4::beee]:25: TLSv1.2 with
cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
mx postfix/smtp[1765]: 3A4BE8EE5C: Server certificate not
trusted

34. 1M top Alexa domains and DANE
• We fetched top 1 million Alexa domains and
created a script that sent an email to each of
them ( test-dnssec-dane@[domain] )
• After some tweaking of the script we got
some good results
• Then we built a script that parsed mail log file
and here are the results:

35. Results
• Out of 1 million domains, 992,232 of them had
MX record and mail server.
• Nearly 70% (687,897) of all attempted SMTP
sessions to Alexa top 1 million domains MX
records were encrypted with TLS
• Majority of TLS connections (60%) were
established with trusted certificate
• 1,382 connections where remote mail server
announced TLS capability failed with "Cannot
start TLS: handshake failure"

36. More results
TLS established connections ratios are:
Anonymous: 109.753
Untrusted: 167.063
Trusted: 410.953
Verified: 128
Quick guide: Anonymous (opportunistic TLS with no
signature), Untrusted (peer certificate not signed by trusted
CA), Trusted (peer certificate signed by trusted CA) and
Verified (verified with TLSA by DANE).

37. DANE Verified
Verified: 128 !!!

38. Mail distribution
Mail Servers # Domains Handled TLS State
google.com 125,422 Trusted
secureserver.net 35,759 Some Trusted, some
no TLS at all
qq.com 11,254 No TLS
Yandex.ru 9,268 Trusted
Ovh.net 8.531 Most Trusted, with
redirect servers
having no TLS at all

39. Mail distribution
Mail Servers # Domains Handled TLS State
Emailsrvr.com 8,262 Trusted
Zohomail.com 2.981 Trusted
Lolipop.jp 1.685 No TLS
Kundenserver.de 2,834 Trusted
Gandi.net 2,200 Anonymous

40. DNSSEC? DANE?
None of these “big” mail servers (and
their domains) are DNSSEC signed
(that meant no DANE for them
possible up to January 2016).

41. • Of course, with wrong certificate hash in TLSA
record (refuses to send mail)
• If domain where MX record resides is not
DNSSEC signed (can’t trust the data in MX, so
no verification)
• If TLSA record published in non-DNSSEC zone
(can’t trust the data in TLSA, so no
verification)
When do DANE things fail?

42. • go6lab.si zone is signed, so is mx.go6lab.si
• there is TLSA for mx.go6lab.si, also signed
• Domain signed.si is signed and MX points to
mx.go6lab.si
• Domain not-signed.si is not signed and MX
points to mx.go6lab.si
• We send email to jan@signed.si and jan@not-
signed.si (signed.si and not-signed.si are used
just as examples)
When do things fail? (example)

43. When I send email to jan@signed.si (signed domain):
Verified TLS connection established to
mx.go6lab.si[2001:67c:27e4::23]:25:
When I send email to jan@not-signed.si (not signed
domain):
Anonymous TLS connection established to
mx.go6lab.si[2001:67c:27e4::23]:25:
When do things fail? (example)

44. • Let’s try to point MX record from signed domain to
A/AAAA record in not-signed domain with TLSA that
is also not signed (obviously) – mail.not-signed.si
Send mail to jan@signed.si when MX for signed.si
points to mail.not-signed.si – DANE verification is not
even started as chain of trust is broken
When do DANE verification also fail?

45. postfix-3.1-20160103/HISTORY:
20160103
Feature: enable DANE policies when an MX host has a secure
TLSA DNS record, even if the MX DNS record was obtained
with insecure lookups. The existence of a secure TLSA record
implies that the host wants to talk TLS and not plaintext.
This behavior is controlled with smtp_tls_dane_insecure_mx_policy
(default: "dane", other settings: "encrypt" and "may"; the
latter is backwards-compatible with earlier Postfix releases).
Viktor Dukhovni.
Postfix latest improvements 

46. Conclusions
• 70% of email can be encrypted in some way,
you just need to enable TLS on your server
• Low number of DNSSEC signed
domains/servers
• Even lower number of DANE/TLSA verified
servers/connections
• It’s easy, go and do it – it’s not the end of the
world and it helps with verifying who are you
sending emails to – and vice versa ;)

47. Conclusions II.
• DANE verification failed (or was aborted) if
DNSSEC chain of trust is not fully established and
complete along the whole way.
• TLSA in not-signed DNS zones would not help you
much preventing your correspondents sending
emails to server-in-the-middle (if you are not
running latest bleeding edge development
version of Postfix)
• DNSSEC/DANE is easy, but please understand
what are you doing before implementing it in
production…

48. Q&A
Questions? Protests? Suggestions?
Complaints?
zorz@isoc.org