Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PLNOG16: Mix 2-in-1: IPv6 troubleshooting for helpdesks - and – DANE/DNSSEC/TLS experiments from Go6la, Jan Zorz


Published on

PLNOG16: Mix 2-in-1: IPv6 troubleshooting for helpdesks - and – DANE/DNSSEC/TLS experiments from Go6la, Jan Zorz

Published in: Internet
  • Be the first to comment

  • Be the first to like this

PLNOG16: Mix 2-in-1: IPv6 troubleshooting for helpdesks - and – DANE/DNSSEC/TLS experiments from Go6la, Jan Zorz

  1. 1. Mix 2-in-1: IPv6 troubleshooting for helpdesks - and – DANE/DNSSEC/TLS experiments from Go6lab Jan Žorž <>
  2. 2. About Me – A Quick History Name: Jan Žorž <> Founder of Slovenian Go6 Institute Worked in Internet operations for 20+ years 18 years of IPv6 experience Active and contributing member of RIPE and IETF communities Primary co-author of RIPE-501/RIPE-554 IPv6 procurement BCP document and RIPE-631 IPv6 helpdesk document Co-author of RFC 6346 (A+P approach to IPv4 depletion) Joined Internet Society DO team in December 2012
  3. 3. Walls-of-text warning Despite long contemplations on how to make this slide- pack without walls-of-text – we could not find any better way how to tell our story. We apologize for any inconvenience.  Please, bear with us anyway…
  4. 4. Misconceptions and reality ;) • Disclaimer: This document is *not* an ISOC document. This is a product of a group of a brilliant experts from the community (list with names comes later in the slide pack). • I just happen to be the initiator and one of the editors of the document • I would like to thanks ISOC for dedicating some of my working time to run this effort.
  5. 5. Removing one of the next IPv6 speedbumps • One of the first speed-bumps was addressed by RIPE-554 • Next speed-bump is lack of IPv6 knowledge at ISP helpdesks
  6. 6. IT helpdesk staff can be… difficult sometimes 
  7. 7. Tools used? • Fact 1.: We need to build a short and simple set of detect/explain/action scenarios that would help people at help desks identify and fix the issue • Fact 2.: We need a simple online tool to detect the state of connectivity on the users computer • Fact 3.: is a very useful tool that detects the state of connectivity on the users computer • So the idea emerged to bring Jason Fesler in the group and talk him into creating a special version of the tool meant specially for ISP helpdesks 
  8. 8. Tools used?
  9. 9. Removing one of the next IPv6 speedbumps Title: “IPv6 Troubleshooting for Residential ISP Helpdesks (Using” Contributors and authors: Lee Howard (Time Warner Cable), John Jason Brzozowski (Comcast), David Freedman (ClaraNET), Jason Fesler (Yahoo!), Tim Chown (University of Southampton), Sander Steffann (SJM Steffann), Chris Grundemann (ISOC), Jen Linkova (Google), Chris Tuska (Comcast), Daniel Breuer (Comcast), Jan Žorž (ISOC)
  10. 10. Table of content 1. What is a BCOP? 2. Summary 3. Background / History 4. Using This Document - Note for Helpdesk Managers 5. IPv6 Troubleshooting 5.1 Basic IPv6 Test 5.2 Test Connectivity 5.3 Test DNS 5.4 Check Home Router 5.5 Escalate 6. Explanation of Help Desk Codes on http://isp.test- 112 - IPv4, plus Broken IPv6 4 - IPv4 only 4t - IPv4 plus Teredo 46 - IPv4 + IPv6 46t - Dual Stack, Possible Tunnel 624 - 6to4 64 - NAT64
  11. 11. Table of content 64t - NAT64, possible tunnel “slow” “mtu” - “Possible MTU issues” Warning “Site(s) with failed connectivity” Warning 7. IPv6 training for helpdesk 8. Conclusion 9. Operator’s specifics Appendix A: Acknowledgements Appendix B. Basic troubleshooting flowchart Appendix C. Collecting Data for Escalation
  12. 12. Example: one of the possible generic situations Help desk code: 624 6to4 IPv4: Good, AS65536, CableCo IPv6: Good, 6to4, Preferred IPv4 address: IPv6 address: 2001:db8::1
  13. 13. Example (continued): Interpretation: “6to4” was used to provide an IPv6 address; and the host was configured to actively take advantage of this service. Any web site that has an IPv6 presence, will be reached using 6to4 instead of native IPv4. Modern operating systems do not prefer these kinds of tunnels by default. Be aware that the user might have a very old operating system or a non-default configuration. Action: 1. Have the user disable any automatic tunneling mechanisms that are active. 6to4 is a protocol that tries to get IPv6 traffic through a public relay, using IPv4 to reach the public relay. Public 6to4 relays offer no SLA; and published studies show approximately 15% failure rates. Windows: Disable tunnel interfaces using (for example, Microsoft Fix it 50412).
  14. 14. Example (Action continued): 2. If IPv6 is desired, configure IPv6 (verify the user has an IPv6 address, and a default route). and test again. If user is still experiencing access issues, follow the troubleshooting steps for the corresponding code returned by http://isp.test-
  15. 15. Example (hardest and longest one): 112 - IPv4, plus Broken IPv6 Help desk code: 112 IPv4: Good, AS65536, CableCo IPv6: broken IPv4 address:
  16. 16. Example (hardest and longest one): Interpretation: IPv6 network connectivity somewhere between the user and the website is broken. IPv6 connections are timing out instead of succeeding (or failing fast to IPv4). The user experience visiting major web sites may be suffering, and some applications completely failing. Assumption: User has already power cycled home router, modem, and device, as part of your standard troubleshooting procedure. Action: 1. Determine whether IPv6 is offered to this customer, based on company documentation. If yes, continue to the next step. 2. …
  17. 17. Very quick reminder: What’s a BCOP? Best Current Operational Practice •A living document describing the best operational practices currently agreed on by subject matter experts •Vetted and periodically reviewed by the global network engineering community (GNEC)
  18. 18. Quick facts in case you are interested: BCOP activity around the world • Africa region: A BCOP group was started under AfNOG, lead by Douglas Onyango and Fiona Asonga • Asia: BCOP Task Force started at JANOG, co-chaired by Seiichi Kawamura and Yoshinobu Matsuzaki, NZNOG BCOP starting up, lead by Dean Pemberton • BoF at Apricot lately… • Europe: RIPE BCOP Task Force created, co-chaired by Benno Overeider and Jan Žorž • Latin America: A BCOP Task Force was started under LACNOG, lead by Luis Balbinot and Pedro R Torres Jr. • North America: NANOG BCOP Committee established, lead by Aaron Hughes and Chris Grundemann
  19. 19. Status and future work? Document is now an official RIPE BCOP document – RIPE- 631 • Done in RIPE BCOP TF and subsequently to RIPE IPv6 WG Join the mailing lists and contribute to discussion: RIPE BCOP TF: RIPE IPv6 WG ML: Issues/comments/ideas tracker URL: troubleshooting-for-helpdesks/issues
  20. 20.
  21. 21. DANE/DNSSEC/TLS Testing in the Go6lab Jan Žorž, ISOC/Go6 Institute, Slovenia
  22. 22. Acknowledgement I would like to thank Internet Society to let me spend some of my ISOC working time in go6lab and test all this new and exciting protocols and mechanisms that makes Internet a bit better and more secure place…
  23. 23. DNSSEC implementation in go6lab • Powerdns server (used as primary for non- signed domains) as “hidden” primary DNS server • OpenDNSSEC platform for signing domains • BIND9 DNS servers as secondaries to OpenDNSSEC to serve signed zones • Virtualization used: PROXMOX 3.4 • OS templates: fedora-20, Centos6/7
  24. 24. DNSSEC implementation in go6lab • “Bump in a wire” • Two public “primary” servers • Concept:
  25. 25. DNSSEC in go6lab • That was fairly easy and it works very well. • Implementation document used from Matthijs Mekking:
  26. 26. DANE experiment • When DNSSEC was set up and functioning we started to experiment with DANE (DNS Authenticated Name Entities). • Requirements: – DNSSEC signed domains – Postfix server with TLS support > 2.11 • We decided on Postfix 3.0.1
  27. 27. DANE • TLSA record for IN TLSA 3 0 1 B4B7A46F9F0DFEA0151C2E07A5AD7908F4C8B0050E7CC 25908DA05E2 A84748ED It’s basically a hash of TLS certificate on More about DANE: ne/
  28. 28. What is DANE and how does it work
  29. 29. DANE verification • was able to verify TLS cert to T-2 mail server and nlnet-labs and some others… mx postfix/smtp[31332]: Verified TLS connection established to[2a01:260:1:4::24]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) dicht postfix/smtp[29540]: Verified TLS connection established to[2001:67c:27e4::23]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
  30. 30. Postfix config smtpd_use_tls = yes smtpd_tls_security_level = may smtpd_tls_key_file = /etc/postfix/ssl/server.pem smtpd_tls_cert_file = /etc/postfix/ssl/server.pem smtpd_tls_auth_only = no smtpd_tls_loglevel = 1 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s smtp_tls_security_level = dane smtp_use_tls = yes smtp_tls_note_starttls_offer = yes smtp_tls_loglevel = 1 tls_random_exchange_name = /var/run/prng_exch tls_random_source = dev:/dev/urandom tls_smtp_use_tls = yes
  31. 31. Malformed TLSA record • We created a TLSA record with a bad hash (one character changed) • Postfix failed to verify it and refused to send a message mx postfix/smtp[1765]: Untrusted TLS connection established to[2001:67c:27e4::beee]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) mx postfix/smtp[1765]: 3A4BE8EE5C: Server certificate not trusted
  32. 32. 1M top Alexa domains and DANE • We fetched top 1 million Alexa domains and created a script that sent an email to each of them ( test-dnssec-dane@[domain] ) • After some tweaking of the script we got some good results • Then we built a script that parsed mail log file and here are the results:
  33. 33. Results • Out of 1 million domains, 992,232 of them had MX record and mail server. • Nearly 70% (687,897) of all attempted SMTP sessions to Alexa top 1 million domains MX records were encrypted with TLS • Majority of TLS connections (60%) were established with trusted certificate • 1,382 connections where remote mail server announced TLS capability failed with "Cannot start TLS: handshake failure"
  34. 34. More results TLS established connections ratios are: Anonymous: 109.753 Untrusted: 167.063 Trusted: 410.953 Verified: 128 Quick guide: Anonymous (opportunistic TLS with no signature), Untrusted (peer certificate not signed by trusted CA), Trusted (peer certificate signed by trusted CA) and Verified (verified with TLSA by DANE).
  35. 35. DANE Verified Verified: 128 !!!
  36. 36. Mail distribution Mail Servers # Domains Handled TLS State 125,422 Trusted 35,759 Some Trusted, some no TLS at all 11,254 No TLS 9,268 Trusted 8.531 Most Trusted, with redirect servers having no TLS at all
  37. 37. Mail distribution Mail Servers # Domains Handled TLS State 8,262 Trusted 2.981 Trusted 1.685 No TLS 2,834 Trusted 2,200 Anonymous
  38. 38. DNSSEC? DANE? None of these “big” mail servers (and their domains) are DNSSEC signed (that meant no DANE for them possible up to January 2016).
  39. 39. • Of course, with wrong certificate hash in TLSA record (refuses to send mail) • If domain where MX record resides is not DNSSEC signed (can’t trust the data in MX, so no verification) • If TLSA record published in non-DNSSEC zone (can’t trust the data in TLSA, so no verification) When do DANE things fail?
  40. 40. • zone is signed, so is • there is TLSA for, also signed • Domain is signed and MX points to • Domain is not signed and MX points to • We send email to and jan@not- ( and are used just as examples) When do things fail? (example)
  41. 41. When I send email to (signed domain): Verified TLS connection established to[2001:67c:27e4::23]:25: When I send email to (not signed domain): Anonymous TLS connection established to[2001:67c:27e4::23]:25: When do things fail? (example)
  42. 42. • Let’s try to point MX record from signed domain to A/AAAA record in not-signed domain with TLSA that is also not signed (obviously) – Send mail to when MX for points to – DANE verification is not even started as chain of trust is broken When do DANE verification also fail?
  43. 43. postfix-3.1-20160103/HISTORY: 20160103 Feature: enable DANE policies when an MX host has a secure TLSA DNS record, even if the MX DNS record was obtained with insecure lookups. The existence of a secure TLSA record implies that the host wants to talk TLS and not plaintext. This behavior is controlled with smtp_tls_dane_insecure_mx_policy (default: "dane", other settings: "encrypt" and "may"; the latter is backwards-compatible with earlier Postfix releases). Viktor Dukhovni. Postfix latest improvements 
  44. 44. Conclusions • 70% of email can be encrypted in some way, you just need to enable TLS on your server • Low number of DNSSEC signed domains/servers • Even lower number of DANE/TLSA verified servers/connections • It’s easy, go and do it – it’s not the end of the world and it helps with verifying who are you sending emails to – and vice versa ;)
  45. 45. Conclusions II. • DANE verification failed (or was aborted) if DNSSEC chain of trust is not fully established and complete along the whole way. • TLSA in not-signed DNS zones would not help you much preventing your correspondents sending emails to server-in-the-middle (if you are not running latest bleeding edge development version of Postfix) • DNSSEC/DANE is easy, but please understand what are you doing before implementing it in production…
  46. 46. Q&A Questions? Protests? Suggestions? Complaints?