Troubleshoo*ng	
  IPv6	
  

         Rolf	
  Schärer,	
  CCIE	
  #17218	
  
HSR	
  Hochschule	
  für	
  Technik	
  Rapperswil	
  
        Swiss	
  IPv6	
  Council	
  Member	
  
Off-­‐Topic	
  


•  The	
  sad	
  thing	
  about	
  IPv6	
  jokes	
  is	
  that	
  almost	
  no	
  one	
  
   understands	
  them	
  and	
  no	
  one	
  is	
  using	
  them	
  yet.	
  
                                                                                           unknown,	
  Internet	
  




                            Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Agenda	
  


•    Real	
  life	
  IPv6	
  issues	
  
•    DemonstraNon	
  
•    Tools	
  
•    Conclusions	
  and	
  RecommendaNons	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Agenda	
  


•    Real	
  life	
  IPv6	
  issues	
  
•    DemonstraNon	
  
•    Tools	
  
•    Conclusions	
  and	
  RecommendaNons	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Real	
  life	
  IPv6	
  issues	
  
Host	
  

•  Wrong	
  IPv6	
  address/subnetmask/default	
  gateway	
  
     ð IPv4/IPv6	
  fallback	
  problem	
  
     ð Check	
  reachability	
  of	
  your	
  prefix	
  with	
  looking	
  glass	
  
           •  ping	
  /	
  traceroute	
  




                                  Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Real	
  life	
  IPv6	
  issues	
  
Network	
  

•  Broken	
  connecNvity	
  on	
  the	
  path	
  
     ð traceroute	
  
     ð check	
  reachability	
  of	
  your	
  prefix	
  and	
  the	
  desNnaNon	
  via	
  
        looking	
  glass	
  /	
  verificaNon	
  service	
  
•  Broken	
  Path	
  MTU	
  discovery	
  (RFC1981)	
  
     ð No	
  fragmentaNon	
  allowed	
  on	
  path	
  
        -­‐>	
  node	
  must	
  adjust	
  the	
  MTU	
  
•  Firewall	
  blocks	
  traffic	
  
     ð On	
  today’s	
  products,	
  IPv4	
  and	
  IPv6	
  rules	
  have	
  to	
  be	
  
        configured	
  separately	
  


                                Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Real	
  life	
  IPv6	
  issues	
  
IPv6	
  to	
  IPv4	
  fallback	
  problem	
  

                                               IPv4/IPv6 enabled network

               IPv6                                                                            IPv6
                                                   Connected via IPv6
               IPv4                                                                            IPv4

   www.test.com
   A: 192.0.2.10
   AAAA: 2001:db8::10

                                                                                  ca. 20-30s
               IPv6             Connection FAILS via IPv6                                      IPv6
               IPv4                                Connected via IPv4                          IPv4

   www.test.com
   A: 192.0.2.10
   AAAA: 2001:db8::10


               IPv6
                                                   Connected via IPv4                          IPv4
               IPv4

   www.test.com
   A: 192.0.2.10



                                Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Agenda	
  


•    Real	
  life	
  IPv6	
  issues	
  
•    Demonstra*on	
  
•    Tools	
  
•    Conclusions	
  and	
  RecommendaNons	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Demonstra*on	
  


•  IPv4/IPv6	
  performance	
  measurement	
  
    ð via	
  online	
  service	
  –	
  test	
  your	
  internet	
  uplink	
  


•  DNS	
  behavior	
  with	
  and	
  without	
  dualstack	
  
    ð v4.ipv6now.ch	
  –	
  A	
  record	
  only	
  
    ð v6.ipv6now.ch	
  –	
  AAAA	
  record	
  only	
  
    ð dualstack.ipv6now.ch	
  –	
  A	
  and	
  AAAA	
  record	
  


•  IPv6/IPv4	
  fallback	
  


                              Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Agenda	
  


•    Real	
  life	
  IPv6	
  issues	
  
•    DemonstraNon	
  
•    Tools	
  
•    Conclusions	
  and	
  RecommendaNons	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
End	
  Host	
  Tools	
  and	
  Plugins	
  

•  Firefox	
  /	
  Chrome	
  
    ð Show	
  IP	
  Plugin	
  
            •  only	
  show’s	
  the	
  response	
  of	
  the	
  DNS	
  server,	
  not	
  the	
  used	
  IP	
  address	
  
•  ping/ping6	
  
•  tracert/traceroute6	
  




                                    Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
End	
  Host	
  Commands	
  (I)	
  

•  Address	
  verificaNon	
  
    ipconfig
    netsh interface ipv6 show addresses

•  DHCP	
  
    ipconfig /release6 ; ipconfig /renew6

•  Default	
  Gateway	
  
    netstat –nr

•  Neighbor	
  Table	
  
    netsh interface ipv6 show neighbors

•  Ping	
  /	
  Traceroute	
  
    ping <X:X:X:X:X:X:X:X> ; ping -6 <hostname>
    tracert <X:X:X:X:X:X:X:X>; tracert -6 <hostname>




                         Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
End	
  Host	
  Commands	
  (II)	
  

•  Address	
  verificaNon	
  
    ifconfig en0
    ifconfig en0 | grep inet6      	
  
•  Default	
  Gateway	
  
    netstat –nr | grep default

•  Neighbor	
  Table	
  
    ndp -a   	
  
•  Ping	
  /	
  Traceroute	
  
    ping6 <X:X:X:X:X:X:X:X> | <hostname>
    traceroute6 <X:X:X:X:X:X:X:X> | <hostname>




                         Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
End	
  Host	
  Commands	
  (III)	
  

•  Useful	
  Commands	
  
    ð Verify	
  IPv6	
  Address	
  
            •  All	
  sedngs:	
  ip addr list eth0	
  
            •  Address	
  only:	
  ip addr list eth0 | grep inet6	
  
•  Default	
  Gateway	
  
    ip –6 route list
    netstat -6 -rn

•  Neighbor	
  Table	
  
    ip -6 neigh show




                            Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Network	
  Equipment	
  Commands	
  (cisco)	
  

•  IPv6	
  unicast-­‐rouNng	
  has	
  to	
  be	
  enabled	
  
   separately	
  
    Router# config t
    Router(config)# ipv6 unicast-routing

•  RouNng	
  Table	
  
     show ipv6 route
     show ipv6 route X:X:X:X::/64

•  Interfaces	
  
     show ipv6 interface brief




                             Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Network	
  Connec*vity	
  Tools	
  

•  BGP	
  looking	
  glasses	
  
    ð e.g.	
  Switch	
  IPv6	
  looking	
  glass	
  
        hfp://www.switch.ch/network/tools/ipv6lookingglass/index.html	
  



    ð shows	
  the	
  view	
  of	
  the	
  
       internet	
  to	
  your	
  	
  
       network	
  




                                Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Server	
  Verifica*on	
  Tools	
  

•  e.g.	
  hfp://www.checked.by.iks-­‐jena.de/	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
World	
  IPv6	
  Day	
  –	
  Connec*vity	
  Chart	
  

•  hfp://ipv6eyechart.ripe.net/	
  
    ð overview	
  of	
  sites	
  parNcipate	
  at	
  the	
  World	
  IPv6	
  Day	
  




                             Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
DNS	
  verifica*on	
  –	
  on	
  client	
  

•  Windows:	
  
    nslookup –q=AAAA hostname




•  Linux/OS	
  X:	
  
    dig AAAA hostname




                         Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
DNS	
  verifica*on	
  –	
  external	
  services	
  




                       Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Path	
  MTU	
  discovery	
  

•  hfp://www.ipv6chicken.com	
  
    ð shows	
  an	
  incomplete	
  image	
  if	
  the	
  MTU	
  	
  




                             Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Simple	
  IPv6	
  performance	
  measurement	
  

•  iperf	
  
    ð hfp://sourceforge.net/projects/iperf/	
  




•  Jperf	
  (graphical	
  extension	
  to	
  iperf)	
  
    ð hfp://www.nwlab.net/know-­‐how/JPerf/	
  




                         Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Online	
  IPv4/IPv6	
  Performance	
  Test	
  

•  With	
  online	
  speedtest’s	
  you	
  can	
  measure	
  your	
  internet	
  
   connecNvity	
  
    ð  Bad	
  performance	
  does	
  not	
  automaNcally	
  mean	
  your	
  internet	
  uplink	
  is	
  bad,	
  it	
  also	
  
        can	
  be	
  the	
  server	
  or	
  the	
  network	
  between	
  

•  hfp://ipv6-­‐test.com/speedtest/	
  
                                                                                                               Internet




                                    Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Tools	
  
Wireshark	
  

•  hfp://www.wireshark.org	
  




                 Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Agenda	
  


•    Real	
  life	
  IPv6	
  issues	
  
•    DemonstraNon	
  
•    Tools	
  
•    Conclusions	
  and	
  Recommenda*ons	
  




                     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Conclusions	
  and	
  Recommenda*ons	
  (I)	
  


•  Update	
  your	
  exisNng	
  troubleshooNng	
  tools	
  AND	
  skills	
  
     	
  
OR	
  
	
  
•  Look	
  for	
  new	
  tools	
  if	
  your	
  exisNng	
  tools	
  do	
  not	
  support	
  
     IPv6	
  




                           Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Conclusions	
  and	
  Recommenda*ons	
  (II)	
  


•  Monitor	
  your	
  IPv6	
  infrastructure	
  
   	
  


AND	
  
	
  
	
  
•  Document	
  your	
  IPv6	
  environment	
  




                       Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Conclusions	
  and	
  Recommenda*ons	
  (III)	
  


•  Align	
  your	
  IPv6	
  structure	
  with	
  your	
  exisNng	
  IPv4	
  
   environment	
  
•  A	
  good	
  design	
  speeds	
  up	
  your	
  troubleshooNng	
  and	
  
   secures	
  your	
  environment	
  
    ð datacenter	
  
         •  EUI-­‐64	
  and	
  router	
  adverNsements	
  not	
  necessary	
  
            –	
  use	
  staNc	
  allocaNon	
  only	
  
    ð clients	
  
         •  design	
  depends	
  on	
  your	
  environment	
  
            (full	
  DHCPv6,	
  eui-­‐64/DHCPv6,	
  pure	
  eui-­‐64)	
  
         •  enable	
  IPv6	
  privacy	
  extension	
  on	
  all	
  your	
  EUI-­‐64	
  
            clients!	
  

                                Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Conclusions	
  and	
  Recommenda*ons	
  (IV)	
  


•  Train	
  yourself	
  (and	
  your	
  other	
  IT	
  staff)	
  in	
  IPv6	
  
   troubleshooNng	
  
   	
  
   it’s	
  all	
  about	
  pracNce...	
  
   	
  
   ...	
  but	
  not	
  that	
  
   different	
  to	
  IPv4!	
  




                            Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
Conclusions	
  and	
  Recommenda*ons	
  (V)	
  


•  Update	
  your	
  internal	
  processes	
  for	
  IPv6	
  
    ð VerificaNon	
  aler	
  a	
  change	
  
    ð Deployment	
  of	
  new	
  equipment	
  
    ð Helpdesk	
  
    ð ...	
  




                         Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
5. Troubleshooting IPv6 - Rolf Schärer

5. Troubleshooting IPv6 - Rolf Schärer

  • 1.
    Troubleshoo*ng  IPv6   Rolf  Schärer,  CCIE  #17218   HSR  Hochschule  für  Technik  Rapperswil   Swiss  IPv6  Council  Member  
  • 2.
    Off-­‐Topic   •  The  sad  thing  about  IPv6  jokes  is  that  almost  no  one   understands  them  and  no  one  is  using  them  yet.   unknown,  Internet   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 3.
    Agenda   •  Real  life  IPv6  issues   •  DemonstraNon   •  Tools   •  Conclusions  and  RecommendaNons   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 4.
    Agenda   •  Real  life  IPv6  issues   •  DemonstraNon   •  Tools   •  Conclusions  and  RecommendaNons   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 5.
    Real  life  IPv6  issues   Host   •  Wrong  IPv6  address/subnetmask/default  gateway   ð IPv4/IPv6  fallback  problem   ð Check  reachability  of  your  prefix  with  looking  glass   •  ping  /  traceroute   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 6.
    Real  life  IPv6  issues   Network   •  Broken  connecNvity  on  the  path   ð traceroute   ð check  reachability  of  your  prefix  and  the  desNnaNon  via   looking  glass  /  verificaNon  service   •  Broken  Path  MTU  discovery  (RFC1981)   ð No  fragmentaNon  allowed  on  path   -­‐>  node  must  adjust  the  MTU   •  Firewall  blocks  traffic   ð On  today’s  products,  IPv4  and  IPv6  rules  have  to  be   configured  separately   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 7.
    Real  life  IPv6  issues   IPv6  to  IPv4  fallback  problem   IPv4/IPv6 enabled network IPv6 IPv6 Connected via IPv6 IPv4 IPv4 www.test.com A: 192.0.2.10 AAAA: 2001:db8::10 ca. 20-30s IPv6 Connection FAILS via IPv6 IPv6 IPv4 Connected via IPv4 IPv4 www.test.com A: 192.0.2.10 AAAA: 2001:db8::10 IPv6 Connected via IPv4 IPv4 IPv4 www.test.com A: 192.0.2.10 Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 8.
    Agenda   •  Real  life  IPv6  issues   •  Demonstra*on   •  Tools   •  Conclusions  and  RecommendaNons   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 9.
    Demonstra*on   •  IPv4/IPv6  performance  measurement   ð via  online  service  –  test  your  internet  uplink   •  DNS  behavior  with  and  without  dualstack   ð v4.ipv6now.ch  –  A  record  only   ð v6.ipv6now.ch  –  AAAA  record  only   ð dualstack.ipv6now.ch  –  A  and  AAAA  record   •  IPv6/IPv4  fallback   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 10.
    Agenda   •  Real  life  IPv6  issues   •  DemonstraNon   •  Tools   •  Conclusions  and  RecommendaNons   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 11.
    Tools   End  Host  Tools  and  Plugins   •  Firefox  /  Chrome   ð Show  IP  Plugin   •  only  show’s  the  response  of  the  DNS  server,  not  the  used  IP  address   •  ping/ping6   •  tracert/traceroute6   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 12.
    Tools   End  Host  Commands  (I)   •  Address  verificaNon   ipconfig netsh interface ipv6 show addresses •  DHCP   ipconfig /release6 ; ipconfig /renew6 •  Default  Gateway   netstat –nr •  Neighbor  Table   netsh interface ipv6 show neighbors •  Ping  /  Traceroute   ping <X:X:X:X:X:X:X:X> ; ping -6 <hostname> tracert <X:X:X:X:X:X:X:X>; tracert -6 <hostname> Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 13.
    Tools   End  Host  Commands  (II)   •  Address  verificaNon   ifconfig en0 ifconfig en0 | grep inet6   •  Default  Gateway   netstat –nr | grep default •  Neighbor  Table   ndp -a   •  Ping  /  Traceroute   ping6 <X:X:X:X:X:X:X:X> | <hostname> traceroute6 <X:X:X:X:X:X:X:X> | <hostname> Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 14.
    Tools   End  Host  Commands  (III)   •  Useful  Commands   ð Verify  IPv6  Address   •  All  sedngs:  ip addr list eth0   •  Address  only:  ip addr list eth0 | grep inet6   •  Default  Gateway   ip –6 route list netstat -6 -rn •  Neighbor  Table   ip -6 neigh show Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 15.
    Tools   Network  Equipment  Commands  (cisco)   •  IPv6  unicast-­‐rouNng  has  to  be  enabled   separately   Router# config t Router(config)# ipv6 unicast-routing •  RouNng  Table   show ipv6 route show ipv6 route X:X:X:X::/64 •  Interfaces   show ipv6 interface brief Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 16.
    Tools   Network  Connec*vity  Tools   •  BGP  looking  glasses   ð e.g.  Switch  IPv6  looking  glass   hfp://www.switch.ch/network/tools/ipv6lookingglass/index.html   ð shows  the  view  of  the   internet  to  your     network   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 17.
    Tools   Server  Verifica*on  Tools   •  e.g.  hfp://www.checked.by.iks-­‐jena.de/   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 18.
    Tools   World  IPv6  Day  –  Connec*vity  Chart   •  hfp://ipv6eyechart.ripe.net/   ð overview  of  sites  parNcipate  at  the  World  IPv6  Day   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 19.
    Tools   DNS  verifica*on  –  on  client   •  Windows:   nslookup –q=AAAA hostname •  Linux/OS  X:   dig AAAA hostname Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 20.
    Tools   DNS  verifica*on  –  external  services   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 21.
    Tools   Path  MTU  discovery   •  hfp://www.ipv6chicken.com   ð shows  an  incomplete  image  if  the  MTU     Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 22.
    Tools   Simple  IPv6  performance  measurement   •  iperf   ð hfp://sourceforge.net/projects/iperf/   •  Jperf  (graphical  extension  to  iperf)   ð hfp://www.nwlab.net/know-­‐how/JPerf/   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 23.
    Tools   Online  IPv4/IPv6  Performance  Test   •  With  online  speedtest’s  you  can  measure  your  internet   connecNvity   ð  Bad  performance  does  not  automaNcally  mean  your  internet  uplink  is  bad,  it  also   can  be  the  server  or  the  network  between   •  hfp://ipv6-­‐test.com/speedtest/   Internet Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 24.
    Tools   Wireshark   • hfp://www.wireshark.org   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 25.
    Agenda   •  Real  life  IPv6  issues   •  DemonstraNon   •  Tools   •  Conclusions  and  Recommenda*ons   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 26.
    Conclusions  and  Recommenda*ons  (I)   •  Update  your  exisNng  troubleshooNng  tools  AND  skills     OR     •  Look  for  new  tools  if  your  exisNng  tools  do  not  support   IPv6   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 27.
    Conclusions  and  Recommenda*ons  (II)   •  Monitor  your  IPv6  infrastructure     AND       •  Document  your  IPv6  environment   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 28.
    Conclusions  and  Recommenda*ons  (III)   •  Align  your  IPv6  structure  with  your  exisNng  IPv4   environment   •  A  good  design  speeds  up  your  troubleshooNng  and   secures  your  environment   ð datacenter   •  EUI-­‐64  and  router  adverNsements  not  necessary   –  use  staNc  allocaNon  only   ð clients   •  design  depends  on  your  environment   (full  DHCPv6,  eui-­‐64/DHCPv6,  pure  eui-­‐64)   •  enable  IPv6  privacy  extension  on  all  your  EUI-­‐64   clients!   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 29.
    Conclusions  and  Recommenda*ons  (IV)   •  Train  yourself  (and  your  other  IT  staff)  in  IPv6   troubleshooNng     it’s  all  about  pracNce...     ...  but  not  that   different  to  IPv4!   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011
  • 30.
    Conclusions  and  Recommenda*ons  (V)   •  Update  your  internal  processes  for  IPv6   ð VerificaNon  aler  a  change   ð Deployment  of  new  equipment   ð Helpdesk   ð ...   Troubleshooting IPv6 – Swiss IPv6 Day Zürich – 08. Juni 2011