PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Packet-Tracer
•Download as PPTX, PDF•
1 like•644 views
Piotr Kupisiewicz – Technical Expert in Krakow’s TAC VPN team. In IT for more than 10 years, out of which 5 years is mostly software engineering experience. Last 5 years spent mostly in networking area interested mostly in Network Security. His hobby are drums and very heavy music. CCIE Security 39762. Olivier Pelerin – as a key member of the escalation team at Cisco’s Technical Assistance Center, he handles world-wide escalations on VPN technologies pertaining to IPSEC, DMVPN, EzVPN, GetVPN, FlexVPN, PKI. Olivier has spent years troubleshooting and diagnosing issues on some of largest, and most complex VPN deployments Olivier have a CCIE in security #20306 Topic of Presentation: Make IOS-XE Troubleshooting Easy – Packet-Tracer Language: English Abstract: “IOS-XE is operating system running on Service Provider devices like ASR series and ISR-4451. Aim of this session is to show how very complicated Service Provider’s configurations can be easily troubleshoted using packet-tracer tool.”
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (6)
Similar to PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Packet-Tracer
Similar to PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Packet-Tracer (20)
Recently uploaded
Recently uploaded (20)
PLNOG 13: P. Kupisiewicz, O. Pelerin: Make IOS-XE Troubleshooting Easy – Packet-Tracer
- 1. Make IOS XE Troubleshooting Easy: Packet tracer PLNOG 2014 Breakout Piotr Kupisiewicz -- pkupisie@cisco.com VPN Lead, Cisco TAC Krakow CCIE Security #39762 Olivier Pelerin – olpeleri@cisco.com VPN Escalation, Cisco TAC Brussels CCIE Security #20306 September 30th, 2014
- 2. Session Agenda System architecture Day in life of normal packet Debugging strategies Packet tracer and conditional filters Live Demo Wrap up © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
- 3. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
- 4. System Architecture Forwarding Plane ESP FECP QFP Crypto Assist. PPE BQS interconn. ESP FECP QFP Crypto Assist. PPE BQS interconn. IOCP interconn. SPA Aggreg. interconn. SIP IOCP SPA Aggreg. IOCP interconn. SPA Aggreg. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 4 RP CPU interconn. GE switch SIP SPA SPA RP CPU interconn. GE switch Midplane SIP SPA SPA SPA SPA Active Active Stby Stby Hypertransport 10 Gbps Ethernet Embedded Service Interconnect aka ESI Bus 11.2 – 40 Gbps Forwarding Bus Centralized Architecture All traffic flows through ESP
- 5. CPU RP ESP FECP ESP ASR1K Software Architecture Chassis Manager CPU IOS Forwarding Manager interconn. GE switch Linux Kernel Chassis Manager FECP Forwarding Manager Linux Kernel QFP Drivers Drivers Drivers Crypto Assist. Crypto Assist. BQS interconn. μ μ μ μ SIP Chassis Manager IOCP QFP μ interconn. SPA Driver SPA Driver SPA Driver SPA Aggreg. Linux Kernel SPA SPA RP EOBC (1 Gbps) ESI (10-40 Gbps) ESI (10-40 Gbps) μ SIP IOCP SPA SPA SPA I2C © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
- 6. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
- 7. SIP intercon. SPA Aggreg. Ingress Packet Through SIP IOCP SPA SPA … ESPs … Interconnect Egress buffers C2W EV-FC EV-RP In ref clocks Network clock distribution Network clocks Ingress Scheduler Egress Buffer Status SPA Aggregation ASIC (Marmot) Ingress Classifier SPA Agg. (per port) Reset / Pwr Ctrl Temp Sensor EEPROM IOCP (SC854x SOC) … Ingress buffers (per port) DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 SPA
- 8. ESP FECP Ingress Packet Through ESP Crypto QFP Assist. PPE BQS intercon . Reset / Pwr Ctrl Temp Sensor EEPROM FECP Crypto TCAM Resource DRAM Packet Buffer DRAM QFP Complex SA table DRAM Packet Processor Engine PPE1 PPE3 PPE4 PPE5 … PPE2 PPE6 PPE7 PPE8 PPEN Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl Interconnect SPI Mux RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
- 9. Reset / Pwr Ctrl Temp Sensor EEPROM FECP Crypto ESP FECP Packet Dispatched to PPE Core TCAM Resource DRAM Packet Buffer DRAM QFP Complex SA table DRAM PPE2 Packet Processor Engine PPE1 PPE3 PPE4 PPE5 … PPE2 Thread 2 Thread 3 Thread 4 PPE6 PPE7 PPE8 PPEN Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPI Mux Interconnect Thread 1 RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS Crypto QFP Assist. PPE BQS intercon . © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
- 10. Reset / Pwr Ctrl Temp Sensor EEPROM FECP Crypto ESP FECP Packet Dispatched to PPE Thread TCAM Resource DRAM Packet Buffer DRAM QFP Complex SA table DRAM PPE2 Packet Processor Engine PPE1 PPE3 PPE4 PPE5 … PPE2 Thread 2 Thread 4 Thread 3 PPE6 PPE7 PPE8 PPEN Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPI Mux Interconnect Thread 1 RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS Crypto QFP Assist. PPE BQS intercon . © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
- 11. X-Connect L2 Switch IPv4 IPv6 MPLS Reset / Pwr Ctrl Temp Sensor EEPROM Netflow Input ACL NBAR Classify FECP MQC Classify … NAT PBR Dialer IDLE Rst URD Crypto ESP FECP FIA’s Applied on Packet by PPE Thread TCAM Resource DRAM Packet Buffer DRAM Input FIA Output FIA QFP Complex SA table DRAM PPE2 Netflow Packet Processor Engine NAT PPE1 PPE3 PPE4 PPE5 … PPE2 Thread 2 Thread 4 Thread 3 IP Unicast NBAR Classify … PPE6 PPE7 PPE8 PPEN IP Multicast Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPI Mux Packet For Us Interconnect Thread 1 MQC Policing MAC Accounting Output ACL RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS Crypto QFP Assist. PPE BQS intercon . PPE2 Thread 3 © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
- 12. X-Connect L2 Switch IPv4 IPv6 MPLS Reset / Pwr Ctrl Temp Sensor EEPROM FECP Crypto ESP FECP Leaving the PPE Thread TCAM Resource DRAM Packet Buffer DRAM Input FIA Output FIA QFP Complex SA table DRAM PPE2 Packet Processor Engine PPE1 PPE2 PPE3 PPE4 PPE5 … Thread 2 Thread 4 Thread 3 PPE6 PPE7 PPE8 PPEN Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPI Mux Thread 1 Interconnect RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS Crypto QFP Assist. PPE BQS intercon . PPE2 Thread 3 © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
- 13. Reset / Pwr Ctrl Temp Sensor EEPROM FECP Crypto Packet proceeding to BQS then SIP TCAM Resource DRAM Packet Buffer DRAM QFP Complex SA table DRAM Packet Processor Engine PPE1 PPE3 PPE4 PPE5 … PPE2 PPE6 PPE7 PPE8 PPEN Dispatcher Packet Buffer DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl Interconnect SPI Mux RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS ESP FECP Crypto QFP Assist. PPE BQS intercon . © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
- 14. Egress Packet Through SIP ESPs … Interconnect Egress buffers C2W EV-FC EV-RP SIP IOCP intercon. SPA Aggreg. SPA SPA In ref clocks Network clock distribution Network clocks Ingress Scheduler Egress Buffer Status SPA Aggregation ASIC (Marmot) Ingress Classifier SPA Agg. (per port) Reset / Pwr Ctrl Temp Sensor EEPROM IOCP (SC854x SOC) … Ingress buffers (per port) DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPA © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
- 15. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
- 16. Everyday situations IPsec ZBF NAT Routing WAAS OTV SNMP © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 18 Which feature went wrong ?
- 17. Using statistics for troubleshooting packet drops • SPA • show interfaces <interface-name> • show interfaces <interface-name> accounting • show interfaces <interface-name> stats • SIP • show platform hardware port <slot/card/port> plim statistics • show platform hardware subslot {slot/card} plim statistics • show platform hardware slot {slot} plim statistics • show platform hardware slot {0|1|2} plim status internal • show platform hardware slot {0|1|2} serdes statistics • RP • show platform hardware slot {r0|r1} serdes statistics • show platform software infrastructure lsmpi • ESP Not easy… not very practical either. Let’s dig deeper before making it simpler • show platform hardware slot {f0|f1} serdes statistics • show platform hardware slot {f0|f1} serdes statistics internal • show platform hardware qfp active bqs 0 ipm mapping • show platform hardware qfp active bqs 0 ipm statistics channel all • show platform hardware qfp active bqs 0 opm mapping • show platform hardware qfp active bqs 0 opm statistics channel all • show platform hardware qfp active statistics drop [detail] • show platform hardware qfp active interface if-name <Interface-name> statistics • show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_ • show platform hardware qfp active infrastructure punt statistics type punt-drop | exclude _0_ • show platform hardware qfp active infrastructure punt statistics type inject-drop | exclude _0_ • show platform hardware qfp active infrastructure punt statistics type global-drop | exclude _0_ • show platform hardware qfp active infrastructure bqs queue output default all © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 • show platform hardware qfp active infrastructure bqs queue output recycle all
- 18. Debugging Strategies to Date IOS Control Plane • ACL + show access-list,… • show interface / ip route / bgp … Platform Control Plane • ESP “stuff” • e.g. show platform … hard to remember Data Plane • ESP “stuff” • More arcane show platform … Well Known Rock bottom © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 20 Top Down Very Difficult Let’s change that!!
- 19. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
- 20. X-Connect L2 Switch IPv4 IPv6 MPLS Reset / Pwr Ctrl Condition determines packets Temp Sensor EEPROM Pak Match ? Input ACL FECP MQC Classify NAT PBR Crypto The Packet Tracer and FIA Debugger TCAM Resource DRAM Packet Buffer DRAM Input FIA Output FIA Optionally match on the QFP Complex SA table DRAM PPE2 Packet Processor Engine Output ACL PPE1 PPE3 PPE4 PPE5 … PPE2 Thread 2 Thread 4 Thread 3 NAT PPE6 PPE7 PPE8 PPEN IP Unicast Dispatcher Packet Buffer to be traced DDRAM Boot Flash (OBFL,…) JTAG Ctrl Reset / Pwr Ctrl SPI Mux Interconnect Thread 1 Encaps egress FIA Statistics and final action will be Crypto collected (matched packets dropped, punted to RP, forwarded to output RPs RPs ESP RPs SIPs Part Len / BW SRAM BQS PPE2 Thread 3 Packet # 16 Input ACL MQC Classify NAT PBR Output ACL NAT Encaps Crypto interface …) Optionally, FIA actions can logged per packet System can capture several packets flows Packet flows can be reviewed in show commands IOS 3.10+ © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
- 21. Conditionally Matching Packets Identifying Interesting Packets asr-1k# debug platform condition ? debug platform condition ? both Simultaneous ingress and egress debug egress Egress only debug … ingress Ingress only debug interface Set interface for conditional debug ipv4 Debug IPv4 conditions ipv6 Debug IPv6 conditions mpls Debug MPLS conditions … asr-1k#debug platform condition ingress Match all ingress packets asr-1k#debug platform condition interface gig0/0/3 ingress asr-1k#debug platform condition ipv4 10.0.0.1/32 both asr-1k#debug platform condition ipv4 access-list 100 egress asr-1k#debug platform condition mpls 10 1 ingress Match MPLS packets with © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 top ingress label 10 Match all ingress packets on interface gig0/0/3 Match in & out packets with source or destination 10.0.0.1 Match egress packets passing access-list 100
- 22. Activating the Packet Tracer Following packets through IOS-XE – Basic Statistics asr-1k# debug platform packet-trace ? copy Copy packet data drop Trace drops only enable Enable packet trace packet Packet count The packet tracer follows a set of packets in details through the FIA asr-1k# debug platform condition interface gig0/0/0 ingress asr-1k# debug platform condition start asr-1k# debug platform packet-trace enable asr-1k# … !send traffic asr-1k# show platform packet-trace statistics Packets Summary Matched 102 Traced 0 Packets Received Ingress 12 Inject 90 102 packets were matched by the condition Count Code Cause 90 9 QFP ICMP generated packet Packets Processed Forward 12 Punt 0 Drop 90 12 packets were forwarded 90 packets were dropped 13 packets were dropped Count Code Cause 13 92 Ipv4Null0 17 47 FirewallInvalidZone 60 184 FirewallL4 Consume 0 due to no route 17 packets were dropped due to 60 packets dropped by L4 absence of zone pair © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 inspection (e.g. receiving window)
- 23. Packet Tracer – Tracing Packets… The fate of 16 packets asr-1k# debug platform condition interface gig0/0/0 ingress asr-1k# debug platform condition start asr-1k# debug platform packet-trace packet 16 asr-1k# debug platform packet-trace enable asr-1k# … !send traffic asr-1k# show platform packet-trace summary Automatically stops tracing Pkt Input Output State Reason after 16 packets 0 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) 1 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) 2 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) 3 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) 4 INJ.7 Gi0/0/2 FWD 5 INJ.7 Gi0/0/2 FWD 6 Gi0/0/2 internal0/0/rp:0 PUNT 55 (For-us control) 7 INJ.7 Gi0/0/2 FWD 8 … 16 packets were traced; we can zoom in INJ.7: Packet injected by the RP internal0/0/rp:0: Packet punted to the RP © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
- 24. Packet Tracer – Tracing Packets… The fate of an individual packet asr-1k# show platform packet-trace packet 1 Packet: 1 CBUG ID: 109056985 Summary Input : GigabitEthernet0/0/2 Output : internal0/0/rp:0 State : PUNT 55 (For-us control) Timestamp Start : 334771580191282 ns (04/29/2014 08:01:38.017738 UTC) Stop : 334771580487612 ns (04/29/2014 08:01:38.018035 UTC) Path Trace Feature: IPV4 Source : 17.0.0.196 Destination : 172.18.0.1 Protocol : 50 (ESP) Feature: IPSec Action : DECRYPT SA Handle : 753 SPI : 0x30ba5940 Peer Addr : 17.0.0.196 Feature specific details are displayed © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Local Addr: 172.18.0.1 Zooming on packet 1 Only major features are shown
- 25. Packet Tracer – Focus on Drops Dropped packets – nothing else asr-1k# debug platform condition interface gig0/0/0 ingress asr-1k# debug platform condition start asr-1k# debug platform packet-trace packet 16 For drops, condition is optional… Only save dropped packets asr-1k# debug platform packet-trace drop [code <dropcode>] asr-1k# debug platform packet-trace enable asr-1k# … !send traffic asr-1k# debug platform condition stop asr-1k# show platform packet-trace summary Stop tracing before dumping the Pkt Input Output State Reason summary (code limitation) 0 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 1 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 2 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 3 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 4 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 5 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 6 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 7 Gi0/0/2 Gi0/0/2 DROP 53 (IpsecInput) 8 … Focus on specific drop codes (find codes in packet-trace statistics) Admire dropped packets… real close asr-1k#show platform packet-trace packet 1 Packet: 1 CBUG ID: 148787639 Summary Input : GigabitEthernet0/0/2 Output : GigabitEthernet0/0/2 State : DROP 53 (IpsecInput) Timestamp Start : 361426338620013 ns (04/29/2014 15:25:52.785406 UTC) Stop : 361426338684993 ns (04/29/2014 15:25:52.785471 UTC) Path Trace Feature: IPV4 Source : 17.0.1.34 Destination : 172.18.0.1 Protocol : 50 (ESP) Packet Copy Out © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 002304bb 72020007 7dfbe301 080045c0 0088d135 0000fe32 2c191100 0122ac12 0001085e 1d620000 00c8172c e8010c3e 44726e6f 3eb231d5 166298c1 f519313c IOS 3.11+
- 26. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
- 27. Demo Network Diagram leased MPLS Internet Spoke 2 Spoke 3 Spoke … © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 29 ASR1000 DMZ Spoke 1 Spoke 11 GE 2 GE 1 GE 3 This Internet based client PC can not connect to the server in the DMZ. 192.168.1.0/24 192.168.11.0/24 172.16.0.11 10.1.1.71 172.16.0.1 192.168.0.254
- 28. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 30
- 29. New Debugging Strategy IOS Control Plane • show interface, show ip route, show bgp … • Feature debugging Platform Control Plane • Unified show commands • Platform show commands • Future: control plane conditional debugging Data Plane • Packet Tracer • Forwarding plane conditional debugging • Embedded Packet Capture Well Known © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 31 Still Difficult (not overly)
- 30. Title Goes Here © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
- 31. N7K-M148GS-11 © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
- 32. © 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
- 33. Thank you.