Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400

Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 1
Внутренняя архитектура IOS-
XE: Средства траблшутинга
предачи трафика наASR1k и
ISR4400
Oleg Tipisov
Customer Support Engineer, Cisco TAC
Apr, 2015. Revision 1.0
Cisco Public

© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
Сегодня на семинаре эксперт Cisco TAC Олег Типисов
расскажет об особенностях аппаратной и программной архитектуры
платформ ASR1k и ISR4400/ISR4300. Также будут рассмотрены
диагностические средства IOS-XE, используемые для траблшутинга передачи
трафика
Олег Типисов
Инженер центра
технической поддержки
Cisco TAC, Москва

Технические эксперты
Сергей Василенко
Дмитрий Леонтьев
Дата проведения вебинара – 22 апреля 2015г.

• Сегодняшняя презентация включает опросы аудитории
• Пожалуйста, участвуйте!

Скачать презентацию Вы можете по ссылке:
https://supportforums.cisco.com/ru/document/12483586

Уважаемые пользователи мы предлагаем Вам принять
участие в конкурсе после проведения вебкаста,
который так и будет называться «Внутренняя
архитектура IOS-XE: Средства траблшутинга
предачи трафика на ASR1k и ISR4400».
• Первые три победителя получат фирменный куб Cisco-TAC
• Ответы присылайте на csc-russian@external.cisco.com
• Задание конкурса будет размещено сегодня после проведения
вебкаста (14-00мск)

• Используйте панель Q&A, чтобы задать вопрос
• Наши эксперты ответят на них

• Hardware and Software Architecture
• Conditional Debugging
• Packet Tracer
• Embedded Packet Capture

RP1
(in slots
“r0”&“r1”)
ESP10
(in slots
“f0” & “f1”)
SIP10
SPAs

SPACarrierCard
SPASPA
IOCPMarmot
…
Scooby
SPA-SPI
SPI4.2
Route
Processor
(active)
RP
Scooby
HT-DP
Route
Processor
(standby)
RP
Scooby
HT-DP
ESI
SPACarrierCard
SPASPA
IOCPMarmot
…
Scooby
SPA-SPI
SPI4.2
ESI
SPACarrierCard SPASPA
IOCPMarmot
…
Scooby
SPA-SPI
SPI4.2
ESI
Forwarding Processor
(active)
FECP
HT-DP
Scooby
QFP subsystemCrypto
assist
Fwding
engine
Scooby
Forwarding Processor
(standby)
FECP
HT-DP
Scooby
QFP subsystemCrypto
assist
Fwding
engine
Scooby
11.5Gbps 11.5Gbps 11.5Gbps
11.5Gbps
11.5Gbps 11.5Gbps 11.5Gbps
11.5Gbps 11.5Gbps
Other (e.g. CPP client IPC)
Punt/Inject/ctl pkts
Network pkts
HT-DP – DMA pkt protocol over HT
State sync pkts
Other pkts

• MCP – Midrange Convergence Platform
Initial name for the ASR1k project, replacement platform for C7200 / C7300 /
C10K routers
• ESP (aka FP) – Embedded Services Processor (or Forwarding
Processor)
Board that integrates QFP subsystem, hardware crypto engine (Nitrox II in
classic ASR1k models), control processor in classic models (FECP), TCAM,
interconnect ASICs, DRAM, etc.
• QFP – Quantum Flow Processor (aka CPP - Cisco Packet
Processor)
Forwarding engine that integrates PPE matrix, BQS ASIC, packet buffers, etc.
• PPE – Packet Processing Element
Processor core that implements ASR1k datapath

• FECP – Forwarding Engine Control Processor
Control processor for ESP
• RP – Route Processor
Implements control plane and handles legacy protocols
• IOSd – IOS daemon
IOS code running on RP under Linux (linux_iosd_image RP process)
• BQS – Buffering, Queuing and Scheduling ASIC
Data plane QoS ASIC
• SIP (or CC) – SPA Interface Processor or Carrier Card
• SPA – Shared Port Adapter
• IOCP – I/O Control Processor

http://www.cisco.com/cdc_content_elements/flash/netsol/sp/quantum_flow/demo.html

show platform hardware slot ?
0 SPA-Inter-Processor slot 0
F0 Embedded-Service-Processor slot 0
P0 Power-Supply slot 0
P1 Power-Supply slot 1
R0 Route-Processor slot 0
show platform hardware qfp ?
active Active instance
standby Standby instance
show platform software ipsec ?
FP Embedded-Service-Processor
RP Route-Processor
show platform software ipsec fp ?
active Active instance
standby Standby instance

• First generation ASR1000 routers: ASR1000 (ESP5, ESP10,
ESP20, ESP40; RP1/RP2), ASR1001
asr1000rp1-advipservicesk9.03.13.02.S.154-3.S2-ext.bin
asr1001-universalk9.03.13.02.S.154-3.S2-ext.bin
• Second generation ASR1000 routers: ASR1000 (ESP100,
ESP200), ASR1001-X, ASR1002-X
asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
asr1002x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
IOS-XE Version
IOS Version
Extended Lifetime Release
Platform
RP
Feature Set

• Virtual router: CSR1000V
csr1000v-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
• New generation ISR routers: ISR4300 (ISR4351, ISR4331,
ISR4321), ISR4400 (ISR4451, ISR4431)
isr4300-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
isr4400-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
• Routers for mobile backhaul: ASR900, ASR903, ASR920

IOS-XE Platforms Family
ISR
ISR4400
ISR4300
ASR1K
(1001/1001-X/1002-X/1004/1006/1013)
CPP10/10+
Cavium Nitrox II
Yoda / Luke
Cavium Octeon
CSR
(Ultra)
VMware
XEN
Hyper V
ESP10 & ESP20 – CPP10 ASIC
ESP40 – CPP10+ ASIC
ESP100 & ESP200 – 2x or 4x Yoda ASIC
ASR1002-X – Yoda ASIC
ASR1001-X – Luke ASIC
ISR4400 – Octeon processor
ISR4300 – RP cores
Data path implementation

Embedded Services
Processor
Route Processor (RP)
SPA Interface Processor
Control Messaging
Linux Kernel Linux Kernel
Linux Kernel
QFP
Client/Driver
Chassis
Manager
Forwarding
Manager
SPA
Driver
SPA
Driver
SPA
Driver
SPA
Driver
IOS
(Standby)
Forwarding
Manager
Chassis
Manager
IOS
(Active)
IOS-XE Platform Abstraction Layer (PAL)
Chassis
Manager
• IOS-XE (BinOS) – Linux OS running
multiple processes
• IOS runs as its own Linux process
• IOS-XE design goals:
Modularity
Preemptive scheduling of processes
Fault isolation and containment via
memory protection
Software infrastructure designed for high
availability
Operational consistency – same look and
feel as IOS router
Rapid feature development and built-in
development and diagnostic tools

ESP FECP
Interconn.
Crypto assist
RP
Chassis Mgr.
Forwarding Mgr.
Chassis Mgr.
Forwarding Mgr.
QFP
Client /
Driver
Interconn.
Interconn.
SIP
SPASPA
IOCP
SPA Agg.
…
Interconn.
Kernel (incl. utilities)
Chassis Mgr.SPA
drive
r
SPA
drive
r
SPA
drive
r
SPA
driver
IOSd
QFP subsystem
QFP microcode
• Runs Control Plane
• Generates configurations
• Populates and maintains routing tables (RIB, FIB…)
• Implements forwarding plane for all features
• Executes egress QoS in hardware
• Communicates with Forwarding manager on RP
• Provides interface to QFP Client / Driver
• Maintains copy of FIB
• Programs QFP forwarding plane and QFP DRAM
• Statistics collection and communication to RP
• Process scheduling, memory management, interrupts
• Suite of low-level applications (OBFL, debugging...)
• Provides IPC to other system components
• Provides abstraction layer between hardware and IOS
• Manages ESP redundancy
• Maintains copy of FIB and interface list
• Communicates FIB status to active & standby ESP (or
bulk-download state info in case of restart)

• IOSd is a user-level process scheduled by the Linux kernel
• IOSd runs in a protected address space so it is isolated from
other components on the RP
• IOSd preserves the run-to-completion scheduler model for IOS
processes, but IOSd itself can be preempted by the Linux
scheduler
• Internally, IOSd provides an IOS environment controlled by the
traditional IOS process scheduler
• IOSd consists of several pthreads:
IOS processes (BGP, OSPF, etc.) run in the main IOS thread
Fastpath IOS thread handles punted packets and IPC messages

• IOSd has no direct access to any hardware
• IOSd interacts with the rest of the system through platform-
dependent shims but all of the hardware-specific processing
occurs in other modules
• The shims communicate with the other processes running on the
RP via IPC messages and via regions of shared memory with
per-process access controls
• IOSd has access to an isolated “container” filesystem, which is
within the Linux filesystem space. IOSd views this filesystem as
the root (“/”) directory and has no means to climb “higher” in the
path

• IOSd is responsible for processing of:
Locally-addressed packets
Legacy protocol packets
Exception packets (e.g. packets with Router Alert IP option)
Glean packets (e.g. when ARP request needs to be sent)
• IOSd does not execute any code in the context of an interrupt
handler or at interrupt level
• When a packet is sent to the RP, the interconnect ASIC generates
an interrupt which is handled by a Linux kernel driver
• The driver sends an event to the IOSd punt path handler which is
implemented within IOSd as a high priority fastpath thread
• If the IOSd process is blocked waiting for an event, it is marked
as runnable and scheduled by the Linux

• So, the punt path handler in IOSd is the replacement for the
interrupt handler in IOS
• Packets are received and transmitted by IOS from a virtual ring-
based packet interface
show platform software infrastructure lsmpi
...
Lsmpi0 is up, line protocol is up
Hardware is LSMPI
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Unknown, Unknown, media type is unknown media type
...
Input queue: 0/1500/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
22373606 packets input, 0 bytes, 0 no buffer
...
1276902 packets output, 119357659 bytes, 0 underruns
...
Linux Shared Memory Punt Interface
LSMPI a module in Linux kernel to support
zero-copy transfer of packets between the
IOSd and QFP using Linux memory mapping

• If the packet cannot be forwarded in the IOSd fast path, it gets
punted in the usual IOS manner to an IOS process for process
switching
• Remember that most transit traffic is processed by QFP running
its own code and IOSd doesn’t see it
• Although statistics is updated in IOSd via IPC messages, e.g.:
• But statistics for process-switched packets is not correct:
• CEF forwarding runs on QFP and this statistics is always zero:
show interfaces
show interfaces summary
show interfaces stats
show interfaces switching
show ip cef switching statistics
show ip cef switching statistics feature

• In this test we send continuous ping (timeout 0) from telnet
session opened to ASR1k (ESP10/RP1)
show platform software status control-processor brief
...
CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
RP0 0 44.24 16.81 0.00 36.93 1.90 0.10 0.00
ESP0 0 2.30 18.40 0.00 79.30 0.00 0.00 0.00
ESP1 0 3.09 17.28 0.00 79.62 0.00 0.00 0.00
SIP0 0 1.70 1.00 0.00 97.30 0.00 0.00 0.00
Total RP CPU utilization

• This is an IOS interface to Linux ‘top’ tool
• It can display per-process CPU utilization for processes running
on RP, FECP, IOCP
show platform software process slot r0 monitor cycles 10 interval 5 lines 10
top - 00:06:30 up 10 days, 7:44, 0 users, load average: 0.25, 0.17, 0.06
Tasks: 152 total, 3 running, 149 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.3%us, 3.3%sy, 0.0%ni, 93.2%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 2009376k total, 1874704k used, 134672k free, 144276k buffers
Swap: 0k total, 0k used, 0k free, 1055620k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3223 root 20 0 979m 552m 208m S 51.6 28.1 370:39.81 linux_iosd-imag
8201 root 15 -5 0 0 0 S 1.9 0.0 3:53.05 lsmpi-xmit
8202 root 15 -5 0 0 0 R 1.9 0.0 4:17.45 lsmpi-rx
This statistics is not correct
show platform software process slot {f0 | f1 | fp active | r0 | r1 | rp active | 0 | 1 | 2} ...
IOSd process

• CPU utilization inside IOSd process (16 + 19.75 + 9.43 = 45)
show proc cpu sorted 1m | ex _0.00%_
CPU utilization for five seconds: 45%/16%; one minute: 32%; five minutes: 16%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
614 28167 141868 198 19.75% 13.89% 6.72% 2 Virtual Exec
114 295382 5653468 52 9.43% 6.20% 3.01% 0 IOSXE-RP Punt Se
15 1101101 6322367 174 0.15% 0.08% 0.08% 0 ARP Input
68 661399 3599770 183 0.07% 0.07% 0.08% 0 IOSD ipc task
Total utilization Fastpath thread utilization. The thread handles
punted packets and IPC messages
Utilization due to processes running
within the main IOS thread
“IOSXE-RP Punt Service Process” is the process
that handles IPv4 punt queue inside IOSd, analyzes
“punt cause” in the punt header and enqueues the
packet into the respective IOS process queue.
We also have “IOSXE-RP Punt IPV6 Service Process”.

Embedded Services
Processor
Route Processor (RP)
SPA Interface Processor
Control Messaging
Linux Kernel Linux Kernel
Linux Kernel
QFP
Client/Driver
Chassis
Manager
Forwarding
Manager
SPA
Driver
SPA
Driver
SPA
Driver
SPA
Driver
IOS
(Standby)
Forwarding
Manager
Chassis
Manager
IOS
(Active)
IOS-XE Platform Abstraction Layer (PAL)
Chassis
Manager
• RP processes
Chassis Manager (cmand)
Host Manager (hman)
Forwarding Manager (fman_rp)
Interface Manager (imand)
Shell Manager (smand)
Logging Manager (plogd)
• FP processes
Chassis Manager (cman_fp)
Forwarding Manager (fman_fp_image)
Logging Manager (plogd)
QFP Client Control Process (cpp_cp_svr)
QFP Client Service Process (cpp_sp_svr)
QFP Driver Process (cpp_driver)
show platform software process list {rp |
fp} active [sort memory]

• Each software layer has its own diagnostic commands, but most
of them are only used by TAC and development team
! IOS layer
{show | debug} crypto ...
! IOSd shim layer
{show | debug} platform software ipsec ...
! FMAN-RP layer
show platform software ipsec rp active ...
! FMAN-FP layer
show platform software ipsec fp active ...
! CPP client layer
{show | debug} platform hardware qfp active feature ipsec ...
! CPP µcode (datapath)
{show | debug} platform hardware qfp active feature ipsec datapath ...
! Crypto hardware (only “statistics” is available on ISR4k routers)
show platform hardware crypto-device ...

• IPSec SA at different software layers
• IOS layer (PI)
show crypto ipsec sa | i interface|ident|esp|spi|flow
interface: Tunnel1
local ident (addr/mask/prot/port): (192.168.1.1/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (192.168.2.2/255.255.255.255/47/0)
current outbound spi: 0x6721A788(1730258824)
inbound esp sas:
spi: 0x9E6410A3(2657357987)
transform: esp-aes esp-sha-hmac ,
conn id: 2003, flow_id: HW:3, sibling_flags 80004008, crypto map: Tunnel1-head-0
outbound esp sas:
spi: 0x6721A788(1730258824)
transform: esp-aes esp-sha-hmac ,
conn id: 2004, flow_id: HW:4, sibling_flags 80004008, crypto map: Tunnel1-head-0

• FMAN-FP layer (PD)
show platform software ipsec fp active flow id 3
=========== Flow id: 3
mode: transport
direction: inbound
protocol: esp
SPI: 0x9e6410a3
local IP addr: 192.168.1.1
remote IP addr: 192.168.2.2
crypto device id: 0
crypto map id: 1
SPD id: 1
ACE line number: 1
QFP SA handle: 5
IOS XE interface id: 19
interface name: Tunnel1
Crypto SA ctx id: 0x000000002e03bffd
cipher: AES-128
auth: SHA1
...
...
show platform software ipsec fp active flow id 4
=========== Flow id: 4
mode: transport
direction: outbound
protocol: esp
SPI: 0x6721a788
local IP addr: 192.168.1.1
remote IP addr: 192.168.2.2
crypto device id: 0
crypto map id: 1
SPD id: 1
ACE line number: 1
QFP SA handle: 6
IOS XE interface id: 19
interface name: Tunnel1
use path MTU: 1500
Crypto SA ctx id: 0x000000002e03bffc
cipher: AES-128
auth: SHA1
...

• CPP Client layer (PD)
show platform hardware qfp active feature ipsec sa 5
QFP ipsec sa Information
QFP sa id: 5
pal sa id: 3
QFP spd id: 1
QFP sp id: 2
QFP spi: 0x9e6410a3(2147483647)
crypto ctx: 0x000000002e03bffd
flags: 0xc000800 (Details below)
: src:IKE valid:True soft-life-expired:False hard-life-expired:False
: replay-check:True proto:0 mode:0 direction:0
: qos_preclassify:False qos_group:False
: frag_type:BEFORE_ENCRYPT df_bit_type:COPY
: sar_enable:False getvpn_mode:SNDRCV_SA
: doing_translation:False assigned_outside_rport:False
: inline_tagging_enabled:False
...
Inbound IPsec SA, which means that anti-replay check is
important, but fragmentation type (before/after encryption),
or QoS pre-classify is not.

• CPP Client layer (PD)
show platform hardware qfp active feature ipsec sa 6
QFP ipsec sa Information
QFP sa id: 6
pal sa id: 4
QFP spd id: 1
QFP sp id: 2
QFP spi: 0x6721a788(1730258824)
crypto ctx: 0x000000002e03bffc
flags: 0x4240040 (Details below)
: src:IKE valid:Yes soft-life-expired:No hard-life-expired:No
: replay-check:No proto:0 mode:0 direction:1
: qos_preclassify:No qos_group:No
: frag_type:AFTER_ENCRYPT df_bit_type:COPY
: sar_enable:No getvpn_mode:SNDRCV_SA
: doing_translation:No assigned_outside_rport:No
: inline_tagging_enabled:No
...
Outbound IPSec SA, which means that frag_type is important,
but anti-replay check is not. We always fragment after encryption
if “tunnel protection ipsec profile …” is applied to the tunnel,
hence always configure “ip mtu” on mGRE interfaces (for p2p
GRE system can set it automatically as of CSCtq09372 fix).

• ASR1k crypto hardware layer (PD)
show platform software ipsec fp active encryption-processor 0 context 2e03bffd
show platform software ipsec fp active encryption-processor 0 context 2e03bffc

• In XE3.7 several handy macro commands were introduced to
make troubleshooting of IPSec control plane easier
show crypto ipsec sa peer 10.48.67.149 platform | i --- show
------------------ show platform software ipsec fp active flow identifier 19
------------------ show platform hardware qfp active feature ipsec sa 7
------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfed
------------------ show platform software ipsec fp active encryption-processor 0 context 2dc3bfec
show crypto ipsec sa interface tunnel1 platform | i --- show
------------------ show platform software ipsec fp active interface name Tunnel1
------------------ show platform hardware qfp active feature ipsec interface Tunnel1
------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfdd
------------------ show platform software ipsec fp active encryption-processor 0 context 2e03bfdc
! Use with caution, because the output can be huge in a scaled setup!
show tech-support ipsec platform

• Here we send “show tech” output to FTP server
show tech | redirect ftp://<ip>/<file>.txt
show processes cpu sorted 5sec | ex _0.00%_
CPU utilization for five seconds: 14%/0%; one minute: 7%; five minutes: 2%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
614 16392 127450 128 9.57% 3.99% 0.93% 3 Virtual Exec
612 1132 16114 70 2.59% 1.27% 0.28% 3 FTP Write Proces
613 2056 7633 269 1.21% 0.09% 0.02% 2 Virtual Exec
show platform software process slot r0 monitor cycles 10 interval 5 lines 10
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5800 root 20 0 145m 132m 7608 R 54.4 6.7 3:13.29 smand
3263 root 20 0 979m 543m 205m S 21.4 27.7 20:58.75 linux_iosd-imag
2217 root 20 0 47980 20m 5800 S 13.6 1.0 14:21.85 hman
...
CPU Utilization
RP0 0 84.59 15.00 0.00 0.00 0.19 0.19 0.00

• In customer case we observed that IPSec SVTI tunnels may go
down on ASR1k (RP1) when “show tech” is copied to external FTP
server, if periodic DPD is configured with aggressive 10/3 timers on
several hundred spokes and on the ASR
show platform resources slot r0
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
-----------------------------------------------------------------------
RP0 (ok, active) C
Control Processor 100.00% 100% 90% 95% C
DRAM 1813MB(92%) 1962MB 90% 95% W
...
show processes cpu platform sorted 5sec location r0 | ex _0%_
CPU utilization for five seconds: 99%, one minute: 26%, five minutes: 10%
Pid PPid 5Sec 1Min 5Min Status Size Name
--------------------------------------------------------------------------------
5800 4756 59% 6% 1% R 152535040 smand
3263 2650 13% 10% 4% S 1027596288 linux_iosd-imag
2217 997 4% 1% 1% R 49135616 hman

• ASR1k RP and FECP memory utilization
• Linux memory management is complicated…
• The “free” memory includes “cached” memory which can be
reused, so low “free” doesn’t mean that the system memory is low
• Refer to ASR1k Troubleshooting TechNotes and CSCuc40262
http://www.cisco.com/c/en/us/support/routers/asr-1000-series-aggregation-
services-routers/products-tech-notes-list.html
...
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
RP0 Healthy 2009376 1873508 (93%) 135868 ( 7%) 1553268 (77%)
ESP0 Healthy 2009400 702804 (35%) 1306596 (65%) 490840 (24%)
ESP1 Healthy 2009400 693428 (35%) 1315972 (65%) 491144 (24%)
SIP0 Healthy 471804 318548 (68%) 153256 (32%) 245744 (52%)
The “committed” is the sum of all malloc().
This doesn’t mean that all this memory was
really allocated… “Committed” can be
more than 100%.

• QFP datapath utilization reflects how many PPE's/threads are
busy with packets at a given point of time
• Calculated as an exponentially damped moving average
• Output collected on a very busy BRAS router doing NAT (ESP40)
show platform hardware qfp active datapath utilization
CPP 0: Subdev 0 5 secs 1 min 5 min 60 min
Input: Priority (pps) 939 931 977 806
(bps) 2888288 2953600 3122040 1787376
Non-Priority (pps) 1601727 1606945 1586457 1541474
(bps) 10671107208 10668441928 10514528440 10342623728
Total (pps) 1602666 1607876 1587434 1542280
(bps) 10673995496 10671395528 10517650480 10344411104
Output: Priority (pps) 572 557 551 574
(bps) 380912 360048 353688 376280
Non-Priority (pps) 1550452 1555896 1535883 1490399
(bps) 10149855856 10148858160 9996408704 9819515880
Total (pps) 1551024 1556453 1536434 1490973
(bps) 10150236768 10149218208 9996762392 9819892160
Processing: Load (pct) 58 59 58 56

• QFP memory utilization
• Output collected on ASR1k ESP20 doing NAT (2.3M PAT
translations)
show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics
Type: Name: DRAM, QFP: 0
Total: 1073741824
InUse: 793689088
Free: 280052736
Lowest free water mark: 208302080
Type: Name: IRAM, QFP: 0
Total: 134217728
InUse: 118105088
Free: 16112640
Type: Name: SRAM, QFP: 0
Total: 32768
InUse: 14848
Free: 17920
1GB PPE RLDRAM2 (RDRAM or Resource DRAM)
- NAT sessions
- NetFlow cache
- Firewall sessions / hash tables
- IPSec SA
- QoS marking / policing
128MB instruction RAM
- Used for QFP code (FIA array)
- Can also store data
32KB SRAM
- High speed traffic management functions
- E.g. virtual reassembly

• ASR1k QFP TCAM utilization
• ASR1k BQS resources (queues, etc.) and packet buffers
show platform hardware qfp active tcam resource usage
QFP TCAM Usage Information
...
Total TCAM Cell Usage Information
----------------------------------
Name : TCAM #0 on CPP #0
Total number of regions : 3
Total tcam used cell entries : 104332
Total tcam free cell entries : 944244
Threshold status : below critical limit
show platform hardware qfp active infrastructure bqs status
show platform hardware qfp active bqs 0 packet-buffer utilization
This means that everything is fine 
Unavailable on ISR4k routers, because
they use software TCAM and CACE –
Cisco Adaptive Classification Engine
BQS ASIC is unavailable on ISR4k routers. QoS is implemented on a separate Octeon core.
Software QoS uses same control plane code as ASR1k BQS, except the hardware layer (RM).

• ISR4451: single control plane CPU – Intel Crystal Forest Gladden
CPU 4C/8T @2.0MHz, universal data plane DDR3 memory
• QFP is emulated on Cavium Octeon 6645 (10 cores, one thread
per core, 1 core runs QoS code)
Load Average
Slot Status 1-Min 5-Min 15-Min
RP0 Healthy 0.00 0.00 0.00
Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
RP0 Healthy 3970904 3142812 (79%) 828092 (21%) 2384508 (60%)
CPU Utilization
RP0 0 1.80 1.40 0.00 96.30 0.00 0.50 0.00
1 4.80 0.90 0.00 94.29 0.00 0.00 0.00
2 0.20 4.80 0.00 95.00 0.00 0.00 0.00
3 0.80 3.70 0.00 95.49 0.00 0.00 0.00
4 0.70 0.70 0.00 98.59 0.00 0.00 0.00
5 0.20 1.20 0.00 98.59 0.00 0.00 0.00
6 1.60 1.40 0.00 97.00 0.00 0.00 0.00
7 4.09 0.89 0.00 95.00 0.00 0.00 0.00
show platform hardware qfp active
infrastructure exmem statistics
QFP exmem statistics
Type: Name: DRAM, QFP: 0
Total: 2147483648
InUse: 1713403904
Free: 434079744
Type: Name: IRAM, QFP: 0
Total: 0
InUse: 0
Free: 0
Type: Name: SRAM, QFP: 0
Total: 0
InUse: 0
Free: 0

• Integrated view of platform resources – XE3.13
show platform resources slot [f0 | f1 | r0 | r1 | 0 | ...]
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
--------------------------------------------------------------------------------------------------
RP0 (ok, active) W
Control Processor 6.30% 100% 90% 95% H
DRAM 1797MB(91%) 1962MB 90% 95% W
ESP0(ok, active) H
DRAM 657MB(33%) 1962MB 90% 95% H
QFP H
TCAM 14cells(0%) 131072cells 45% 55% H
DRAM 125263KB(23%) 524288KB 80% 90% H
IRAM 9941KB(7%) 131072KB 80% 90% H
ESP1(ok, standby) H
DRAM 669MB(34%) 1962MB 90% 95% H
QFP H
TCAM 14cells(0%) 131072cells 45% 55% H
DRAM 125263KB(23%) 524288KB 80% 90% H
IRAM 9941KB(7%) 131072KB 80% 90% H
SIP0 H
DRAM 293MB(63%) 460MB 90% 95% H

• New commands for CPU and memory monitoring – XE3.14
• CLI interface to Linux ‘top’ tool – XE3.14
show processes memory platform [sorted] location {rp active | fp active | r0 | r1 | f0 | f1 | 0 | 1
| 2 | ...}
show processes cpu platform [sorted [5sec | 1min | 5min]] location {rp active | fp active | r0 | r1
| f0 | f1 | 0 | 1 | 2 | ...}
show processes cpu platform monitor [cycles <N> [[interval <M>] [lines <K>]]] [location ...]

PPE
ASIC
BQS
ASIC
FECP
R0 R1
GE
EOBC
Serdes Serdes
SPI4.2
SPI
MuxCrypto SPI4.2
SPI4.2
SPI4.2
HT
Packet
Memory
128M
CC0 CC1 CC2RP0 RP1 FP-stby
TCAM
Resource
DRAM
DRAM
Data Path
ESI Links
Control
Path
PPE
ASIC +
BQS
ASIC = QFP

• Implements data plane on PPEs
• Feature Invocation Array (FIA) determines feature ordering
show platform hardware qfp active interface if-name GigabitEthernet0/0/1.99
…
Protocol 0 - ipv4_input
FIA handle - CP:0x1091ed50 DP:0x8091f680
IPV4_INPUT_DST_LOOKUP_ISSUE (M)
IPV4_INPUT_ARL_SANITY (M)
IPV4_INPUT_DST_LOOKUP_CONSUME (M)
IPV4_INPUT_FOR_US_MARTIAN (M)
IPV4_INPUT_VFR
IPV4_NAT_INPUT_FIA
IPV4_INPUT_LOOKUP_PROCESS (M)
IPV4_INPUT_IPOPTIONS_PROCESS (M)
IPV4_INPUT_GOTO_OUTPUT_FEATURE (M)
Protocol 1 - ipv4_output
FIA handle - CP:0x1091ed1c DP:0x8091ff00
IPV4_OUTPUT_VFR
IPV4_NAT_OUTPUT_FIA
IPV4_OUTPUT_THREAT_DEFENSE
IPV4_VFR_REFRAG (M)
IPV4_OUTPUT_L2_REWRITE (M)
IPV4_OUTPUT_FRAG (M)
IPV4_OUTPUT_DROP_POLICY (M)
MARMOT_SPA_D_TRANSMIT_PKT
DEF_IF_DROP_FIA (M)
show run int g0/0/1.99
Current configuration : 115 bytes
!
interface GigabitEthernet0/0/1.99
encapsulation dot1Q 99
ip address 1.1.1.1 255.255.255.0
ip nat outside
End

• Feature processing order follows the 12.0S data path implementation
L2/L3
Classify
IPv4 Validation
Netflow
BGP Accounting
NBAR Classify
MQC Classify
LI
Firewall / IDS / Proxy
Security ACL
RPF
MQC Marking
MQC Policing
MAC Accounting
Prec. Accounting
NAT
PBR
WCCP
Server LB
Dialer IDLE Rst
URD
Firewall / CBAC
TCP Intercept
MQC Marking
IP Accounting
RSVP
MQC Policing
MAC Accounting
Prec Accounting
URDIP Frag
Netflow
Firewall / IDS / Proxy
WCCP
NAT
NBAR Classify
BGP Accounting
LI
Crypto
MQC Classify
FW ACL & Pregen Check
Security ACL
WRED
Queuing
F
F
F
F
F
Forwarding
• IP Unicast
• Loadbalancing
• IP Multicast
• MPLS Imposit.
• MPLS Dispos.
• MPLS Switch.
• FRR
• AToM Dispos.
• MPLSoGRE
IPv6 IPv4 MPLS XConnect L2 Switch

GPM &
Packet Distribution / Gather
IPM HT
i/f OPM
Pkt
Memory
FECP
SERDES SERDES
On chip packet memory
CC0 CC1 CC2RP0 RP1 FP-Stby
CRYPTO SPI Mux
Recycle
PPEs & HW Assists
PPE ASIC BQS ASIC
FE
…

• Frame is received and classified (‘hi’ / ‘lo’) by either SPA or SIP
• Frames are scheduled based on priority and sent to QFP over
ESI ‘hi’ or ‘lo’ priority channel
• Entire L2 frame is received by QFP Input Packet Module (IPM)
and stored in Global Packet Memory (GPM)
• A free PPE thread is assigned to process the packet
• Packet remains in on chip memory (GPM) while it is processed by
one of the PPEs
• The PPE thread runs through a Feature Chain in software. It can
access resources like the HW-assists and TCAM and perform
deep packet inspection, e.g. NBAR

• When processed, the PPE thread releases the packet to the
Traffic Manager and its own packet buffer for placement into an
output queue for scheduling
• The Output Packet Module (OPM) pulls the selected packet for
transmission
• The packet is either transmitted out a physical interface
• Or transmitted back to another PPE thread for further processing
(Recycle Path)

• From OPM traffic can be sent to a SIP module, punted to RP, sent to
crypto co-processor for encryption or decryption or recycled back to QFP
• This command displays default interface queues (QoS can create its own
queues)
show platform hardware qfp active infrastructure bqs queue output default all | i Interface
Interface: internal0/0/recycle:0 QFP: 0.0 if_h: 1 Num Queues/Schedules: 0
Interface: internal0/0/rp:0 QFP: 0.0 if_h: 2 Num Queues/Schedules: 2
Interface: internal0/0/crypto:0 QFP: 0.0 if_h: 4 Num Queues/Schedules: 2
Interface: CPP_Null QFP: 0.0 if_h: 5 Num Queues/Schedules: 0
Interface: Null0 QFP: 0.0 if_h: 6 Num Queues/Schedules: 0
Interface: GigabitEthernet0/0/0 QFP: 0.0 if_h: 7 Num Queues/Schedules: 1
Interface: Loopback0 QFP: 0.0 if_h: 12 Num Queues/Schedules: 0
Interface: Tunnel1 QFP: 0.0 if_h: 17 Num Queues/Schedules: 0
Interface: GigabitEthernet0/0/1.75 QFP: 0.0 if_h: 18 Num Queues/Schedules: 0
Interface: Virtual-Template1 QFP: 0.0 if_h: 21 Num Queues/Schedules: 0
Interface: DmvpnSpoke16908304 QFP: 0.0 if_h: 22 Num Queues/Schedules: 0
RP and crypto chip have
two queues: ‘hi’ / ‘lo’.
There are many recycle
queues (see next slides).

• After PPE has finished processing a packet, it is gathered from
the GPM and written to a queue in BQS
• The queue may be used to recycle the packet back to the GPM
for further processing. E.g. fragmentation or reassembly
show platform hardware qfp active infrastructure bqs queue output recycle summary
Recycle Queue Summary Table (Total Recycle Queues: 73)
ID Name ParentID Prio Bandwidth RateType Mode Limit
=============================================================================================
0x0003 MulticastLeafHigh 0x0002 01 0 00 00 0
0x0004 MulticastLeafLow 0x0002 00 100 01 00 0
0x0005 L2MulticastLeafHigh 0x0002 01 0 00 00 0
0x0006 L2MulticastLeafLow 0x0002 00 100 01 00 0
0x0007 LSMMulticastLeafHigh 0x0002 01 0 00 00 0
0x0008 LSMMulticastLeafLow 0x0002 00 100 01 00 0
0x0009 SBCMMOHLeafHigh 0x0002 01 0 00 00 0
0x000a SBCMMOHLeafLow 0x0002 00 100 01 00 0
0x000b IPFragHi 0x0002 01 0 00 00 0
0x000c IPFragLo 0x0002 00 100 01 00 0
0x000d IPReassemblyHi 0x0002 01 0 00 00 0
0x000e IPReassemblyLo 0x0002 00 100 01 00 0
…

show platform hardware qfp active infrastructure bqs queue output recycle summary
Recycle Queue Summary Table (Total Recycle Queues: 73)
ID Name ParentID Prio Bandwidth RateType Mode Limit
=============================================================================================
…
0x000f IPv6ReassemblyHi 0x0002 01 0 00 00 0
0x0010 IPv6ReassemblyLo 0x0002 00 100 01 00 0
0x0011 IPv4vasi 0x0002 00 100 01 00 0
0x0012 IPv6vasi 0x0002 00 100 01 00 0
…
0x001e MulticastReplicationHigh 0x001d 01 0 00 00 0
0x001f MulticastReplicationLow 0x001d 00 100 01 00 0
…
0x003e ICMPRecycleQ 0x0037 00 100 01 00 0
…
0x0042 FwallRecycleHi 0x0037 01 0 00 00 0
0x0043 FwallRecycleLo 0x0037 00 100 01 00 0
…
0x0047 SSLVPNRecycleQ 0x0037 01 100 01 00 0
0x0048 TcpRecycle 0x0037 01 100 01 00 0
…
0x0057 MetaPkt_Hi 0x0056 01 0 00 00 0
0x0058 MetaPkt_Lo 0x0056 00 100 01 00 0

• Statistics is available for recycle queues
show platform hardware qfp active infrastructure bqs queue output recycle id 12
Recycle Queue Object ID:0xc Name:IPFragLo (Parent Object ID: 0x2)
plevel: 0, bandwidth: 100 , rate_type: 1
queue_mode: 0, queue_limit: 0, num_queues: 1
Queue specifics:
Index 0 (Queue ID:0x11, Name: IPFragLo)
Software Control Info:
(cache) queue id: 0x00000011, wred: 0x88b160f0, qlimit (pkts ): 8192
parent_sid: 0x208, debug_name: IPFragLo
sw_flags: 0x00010001, sw_state: 0x00000c01, port_uidb: 0
orig_min : 0 , min: 0
min_qos : 0 , min_dflt: 0
orig_max : 0 , max: 0
max_qos : 0 , max_dflt: 0
share : 1
plevel : 0, priority: 65535
defer_obj_refcnt: 0
Statistics:
tail drops (bytes): 0 , (packets): 0
total enqs (bytes): 79591976 , (packets): 379948
queue_depth (pkts ): 0
show platform hardware qfp active infrastructure bqs queue output recycle {all | id <number>}
This is a bug CSCut83283.
We increment a counter for each and every packet that
needs to be encrypted on a tunnel interface with tunnel
protection applied, even if the packet is small. This is a
counter issue. Packets are sent to IPFragLo(Hi) recycle
queue only if they need be fragmented.
“all” gives incomplete info – bug CSCub11524

• Mechanism to send a packet from QFP to either RP, or (back to)
QFP for further processing
• Why punt to RP? Basically this is where all the packets QFP can’t
process go: control plane protocols, traffic to router IP, legacy
protocols
• Why punt to (back to) QFP? This is analogous to RP injecting a
packet to QFP. For example, ICMP echo request/response. When
QFP receives an echo request, it will create the echo reply packet
and use the Punt/Inject path to transmit the packet

QFP
LSMPI/
IOS-shim
IOS process
QFP
Punt packet to RP
Punt packet back to QFP
1
2
3
1 2
1. Receive pkt from network
2. Packet marked for punting
to RP. Transmit packet out
Packet is processed by PD
LSMPI/IOS-shim and sent
to IOS PI for processing
to QFP. Packet is formatted
w/ an inject header and recycled
back to QFP.
3. QFP internal interface FIA processes
packet and packet will be transmitted
out appropriate physical interface.

• Mechanism for RP (or QFP) to transmit packets out of ASR1k. RP
will inject packets to QFP for transmission
• Injects from RP: There’s a few flavors. We can break these down
into either fully formatted packets (ie: L2+L3+payload) or L3
packets (ie: IP, IPv6, MPLS)
• Injects from QFP? Ditto what we said w/ punt… A feature needs
to transmit a new (generated) packet out. The feature uses the
CPP inject path to route and transmit the packet

QFPIOS-shim
IOS
process
QFP
Inject packet from RP
Inject packet from QFP
1
2
3
1 2
1. IOS PI sends packet via IOS-shim
IOS-shim formats the CPP inject
headers
2. Inject infra processes inject header
QFP internal interface FIA processes
to QFP. Packet is formatted
w/ an inject header and recycled
back to QFP.
3. Inject infra processes inject header
QFP internal interface FIA processes

• Punt/Inject to/from RP is easy to understand…
• Punt/Inject to/from QFP is complicated…
• Example: Single ICMP Ping to the router IP:
show platform hardware qfp active infrastructure punt statistics type per-cause | exclude _0_
Per Punt Cause Statistics
Packets Packets
Counter ID Punt Cause Name Received Transmitted
--------------------------------------------------------------------------------------
026 QFP ICMP generated packet 1 1
Per Inject Cause Statistics
Packets Packets
Counter ID Inject Cause Name Received Transmitted
--------------------------------------------------------------------------------------

• Router received 1 echo request and generated 1 reply, but, as
you can see, three packets were captured by PACTRAC 
show platform packet-trace statistics
Packets Summary
Matched 3
Traced 3
Packets Received
Ingress 2
Inject 1
Count Code Cause
1 9 QFP ICMP generated packet
Packets Processed
Forward 1
Punt 1
Count Code Cause
1 26 QFP ICMP generated packet
Drop 0
Consume 1
show platform packet-trace summary
Pkt Input Output State Reason
0 Gi0/0/1 Gi0/0/1 CONS Packet Consumed
1 Gi0/0/1 internal0/0/recycle:0 PUNT 26 (QFP ICMP generated packet)
2 INJ.9 Gi0/0/1 FWD
0: ICMP Echo Request
1, 2: ICMP Echo Reply

• There are many commands for Punt Path troubleshooting
• Major punt statistics
show platform software infrastructure punt
...
IOSXE-RP Punt packet causes:
1874682 Layer2 control and legacy packets
1918031 ARP request or response packets
57 Reverse ARP request or repsonse packets
64429 For-us data packets
125191 RP<->QFP keepalive packets
2 Glean adjacency packets
7856 Subscriber session control packets
1577645 For-us control packets
268613 IP subnet or broadcast packet packets
FOR_US Control IPv4 protcol stats:
19101 TCP packets
228855 UDP packets
2505 GRE packets
58177 EIGRP packets
1252125 OSPF packets
16882 PIM packets
...

• Aggregated punt statistics for RP0 low and high priority queues
show platform hardware qfp active infrastructure bqs queue out default interface-string internal0/0/rp:0
Queue specifics:
Index 0 (Queue ID:0x86, Name: i2l_if_2_cpp_0_prio0)
(cache) queue id: 0x00000086, wred: 0x88b16862, qlimit (bytes): 6250048
parent_sid: 0x25c, debug_name: i2l_if_2_cpp_0_prio0
...
Statistics:
queue_depth (bytes): 0
Queue specifics:
Index 1 (Queue ID:0x87, Name: i2l_if_2_cpp_0_prio1)
(cache) queue id: 0x00000087, wred: 0x88b16872, qlimit (bytes): 6250048
parent_sid: 0x25c, debug_name: i2l_if_2_cpp_0_prio1
...
Statistics:
queue_depth (bytes): 0

• Per-cause punt/inject statistics
show platform hardware qfp active infrastructure punt statistic type per-cause | ex _0_
Global Per Cause Statistics
Number of punt causes = 106
Packets Packets
--------------------------------------------------------------------------------------
003 Layer2 control and legacy 1877032 1876909
007 ARP request or response 1977106 1920808
008 Reverse ARP request or repsonse 57 57
011 For-us data 64519 64519
021 RP<->QFP keepalive 125351 125351
024 Glean adjacency 2 2
027 Subscriber session control 7867 7866
055 For-us control 1615501 1579662
060 IP subnet or broadcast packet 268677 268677

• Используете ли вы маршрутизаторы, работающие под
управлением IOS-XE, и для чего?
 Для BGP, как граничный роутер моей AS
 Как PE для организации MPLS VPN
 Как Internet Gateway для выполнения NAT
 Для Broadband Aggregation (BRAS)
 В качестве Cisco Unified Border Element (CUBE)
 Для организации Site-to-Site VPN
 Для организации Remote Access VPN
 В качестве Firewall
 Для Mobile Backhaul
 Использую также, как и маршрутизаторы ISR G2, для решения
различных мелких задач
 Для обогрева серверной комнаты

• System-wide conditions can be used by Packet Tracer tool for data
path troubleshooting and by various features to limit the scope of
the debug
• In this presentation we will not talk about feature debugs
• Implemented in XE3.10
• http://www.cisco.com/c/en/us/td/docs/routers/asr1000/troubleshooti
ng/guide/Tblshooting-xe-3s-asr-1000-book.html

• Conditional Debug configuration
• Global and interface conditions cannot be enabled simultaneously
• Special interfaces:
Internal-RP Dataplane Punt/Inject interface
Internal-Recycle Dataplane Recycle interface
• The “<ipv4-addr[/mask]>” condition matches traffic bi-directionally
• The “access-list <name>” condition is unidirectional
debug platform condition [interface <name>] ipv4 [access-list <name> | <ipv4-addr>[/mask]] {ingress
| egress | both}
debug platform condition [interface <name>] ipv6 [access-list <name> | <ipv6-addr>[/mask]] {ingress
| egress | both}
debug platform condition [interface <name>] mpls [<label-ID>] {ingress | egress | both}
debug platform condition {ingress | egress | both}

• Ingress Conditional Debug in the packet processing path
• Egress Conditional Debug in the packet processing path
show platform hardware qfp active interface if-name <interface-name>
...
Protocol 0 - ipv4_input
FIA handle - CP:0x1091f05c DP:0x80917700
IPV4_INPUT_DST_LOOKUP_ISSUE (M)
IPV4_INPUT_ARL_SANITY (M)
CBUG_INPUT_FIA
DEBUG_COND_INPUT_PKT
...
show platform hardware qfp active interface if-name <interface-name>
...
FIA handle - CP:0x108db890 DP:0x80791c80
CBUG_OUTPUT_FIA
IPV4_OUTPUT_VFR
IPV4_OUTPUT_NAT
IPV4_VFR_REFRAG (M)
DEBUG_COND_OUTPUT_PKT
DEF_IF_DROP_FIA (M)
Conditional Debug also notifies
Packet Tracer on “match”
Packet Tracer packet copy

• This command displays all configured conditions
• “Show debug” includes above output
show platform conditions
Conditional Debug Global State: Start
Conditions Direction
------------------------------------------------------------------------------------|---------
GigabitEthernet0/0/1.75 & IPV4 ACL [145] ingress
GigabitEthernet0/0/1.99 & IPV4 ACL [144] ingress
Feature Condition Type Value
-----------------------|-----------------------|--------------------------------
Feature Type Submode Level
------------|-------------|---------------------------------------------------------|----------
show debug

• Conditions can be removed or cleared
• Next command doesn’t clear conditions, but it stops all debugs
including conditional debug
• Next command starts/stops conditional debug
• Without conditions it enables debug for all packets
no debug platform condition <exact command needs to be entered here>
clear platform condition all
debug platform condition {start | stop}
no debug all

• XE3.11 – Drop Tracing support
• XE3.11 – Recycle Enhancements
• XE3.11 – "decode" Option
• XE3.12 – CSCug38748 – PACTRAC: packet-trace summary output
should print timestamp in datetime
• XE3.13 – Punt/Inject Tracing
• XE3.13 – VASI support
• http://www.cisco.com/c/en/us/td/docs/routers/asr1000/configuration/
guide/chassis/asrswcfg/Packet_Trace.html

• This example provides a quick overview of using Packet Tracer with
a simple IPv4 address condition
! Step1: Define a condition
debug platform condition ipv4 address 172.27.1.1/32 ingress
! Step2: Enable Packet Tracer
debug platform packet-trace packet 2048
debug platform packet-trace enable
! Step3: Start Conditional Debugging (this also starts Packet Tracer)
debug platform condition start
! Step4: Display Packet Tracer configuration, accounting and summary data
show platform packet-trace configuration
! Step5: Stop Conditional Debugging (this also stops Packet Tracer)
debug platform condition stop
! Step6: Clear all information collected by Packet Tracer (optional, requires “stop”)
clear platform packet-trace statistics
! Step7: Clear Packet Trace configuration
clear platform packet-trace configuration

• This example illustrates how to use FIA trace to understand where
certain features live in the packet processing path
policy-map inner
class Prec5
priority percent 20
class Prec3
bandwidth percent 50
policy-map outer
class class-default
shape average 32000
service-policy inner
interface Tunnel0
nhrp map group TEST service-policy output outer
tunnel source GigabitEthernet0/0/2
tunnel mode gre multipoint
tunnel protection ipsec profile prof1
…

• Conditional Debug
• Packet Tracer
access-list 166 permit ip host 192.168.1.1 host 192.168.2.2
debug platform condition interface tunnel0 ipv4 access-list 166 egress
Conditional Debug Global State: Stop
------------------------------------------------------------------------------------|---------
Tunnel0 & IPV4 ACL [166] egress
debug platform packet-trace packet 256 fia-trace
debug platform condition start

• After sending 100 continuous pings (timeout 0) we see that 35
packets were dropped by QoS
show policy-map multipoint Tunnel0
Interface Tunnel0 <--> 1.1.1.2
Service-policy output: outer
Class-map: class-default (match-any)
166 packets, 106384 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/35/0
...
show platform hardware qfp active statistics drop
-------------------------------------------------------------------------
Global Drop Stats Packets Octets
-------------------------------------------------------------------------
TailDrop 35 37790

• Accounting info (statistics)
• Summary info
Packets Summary
Matched 100
Traced 100
Packets Received
Ingress 100
Inject 0
Packets Processed
Forward 65
Punt 0
Drop 35
Count Code Cause
35 22 TailDrop
Consume 0
0 Gi0/0/0.27 Gi0/0/2 FWD
...
64 Gi0/0/0.27 Gi0/0/2 FWD
65 Gi0/0/0.27 Gi0/0/2 DROP 22 (TailDrop)
...

• Path info for forwarded packet #64 (part 1)
show platform packet-trace packet 64
Packet: 64 CBUG ID: 64
Summary
Input : GigabitEthernet0/0/0.27
Output : GigabitEthernet0/0/2
State : FWD
Timestamp
Start : 1398207324379 ns (01/19/2000 04:49:22.995458 UTC)
Stop : 1398207470896 ns (01/19/2000 04:49:22.995604 UTC)
Path Trace
Feature: IPV4
Source : 192.168.1.1
Destination : 192.168.2.2
Protocol : 1 (ICMP)
...
Feature: FIA_TRACE
Entry : 0x8200ed80 - IPV4_OUTPUT_QOS
Lapsed time: 3164 ns
...
Feature: FIA_TRACE
Entry : 0x80128400 - IPV4_OUTPUT_TUNNEL_PROTECTION_ENCRYPT
Lapsed time: 657 ns
Feature: IPSec
Result : IPSEC_RESULT_SA
Action : ENCRYPT
SA Handle : 4
Peer Addr : 1.1.1.2
Local Addr: 1.1.1.1
...
Lapsed time is displayed for each FIA element.
Can be used for datapath profiling!
QoS classification
(output FIA of interface tunnel)
Tunnel protection
We leave tunnel output FIA
at this point and the packet
is sent to crypto engine for
encryption

• Path info for forwarded packet #64 (part 2)
• The packet is received from crypto engine and the processing
continues
...
Feature: FIA_TRACE
Entry : 0x80424e18 - IPV4_IPSEC_FEATURE_RETURN
Lapsed time: 497 ns
Feature: FIA_TRACE
Entry : 0x80126c3c - IPV4_TUNNEL_GOTO_OUTPUT
...
Feature: FIA_TRACE
Entry : 0x8062fc68 - IPV4_TUNNEL_ENCAP_GOTO_OUTPUT_FEATURE
...
Feature: FIA_TRACE
Entry : 0x8200e480 - IPV4_OUTPUT_DROP_POLICY
Feature: FIA_TRACE
Entry : 0x82016c80 - MARMOT_SPA_D_TRANSMIT_PKT
We enter egress physical interface
output FIA at this point
Packet is transmitted

• Path info for dropped packet #65 (part 1)
Summary
State : DROP 22 (TailDrop)
Timestamp
Start : 1398207410699 ns (01/19/2000 04:49:22.995544 UTC)
Stop : 1398207589076 ns (01/19/2000 04:49:22.995722 UTC)
Path Trace
Feature: IPV4
Source : 192.168.1.1
Protocol : 1 (ICMP)
...
Feature: FIA_TRACE
Entry : 0x8200ed80 - IPV4_OUTPUT_QOS
...
Feature: FIA_TRACE
Entry : 0x80128400 - IPV4_OUTPUT_TUNNEL_PROTECTION_ENCRYPT
Lapsed time: 977 ns
Feature: IPSec
Result : IPSEC_RESULT_SA
Action : ENCRYPT
SA Handle : 4
Peer Addr : 1.1.1.2
Local Addr: 1.1.1.1
...
Lapsed time is displayed for each FIA element.
Can be used for datapath profiling!
QoS classification
Tunnel protection
We leave tunnel output FIA
at this point and the packet
is sent to crypto engine for
encryption

• Path info for dropped packet #65 (part 2)
• The packet is received from crypto engine and the processing
continues, but the packet is dropped by QoS code
...
Feature: FIA_TRACE
Entry : 0x8062fc68 - IPV4_TUNNEL_ENCAP_GOTO_OUTPUT_FEATURE
...
Feature: QOS
Direction : Egress
Action : DROP
Drop Cause : TailDrop
Policy : Tail drop
Pak Priority : FALSE
Priority : FALSE
Queue ID : 145 (0x91)
PAL Queue ID : 1073741829 (0x40000005)
Queue Limit : 64
WRED enabled : FALSE
Inst Queue len: n/a
Avg Queue len : n/a
Feature: FIA_TRACE
Entry : 0x806c1acc - OUTPUT_DROP
Lapsed time: 302 ns
Feature: FIA_TRACE
Entry : 0x8200e480 - IPV4_OUTPUT_DROP_POLICY
We enter egress physical interface
output FIA at this point
Packet is dropped. Important point
here is that it’s dropped after IPSec
encapsulation, which can cause
IPSec anti-replay drops on the
receiver side.

• Packet Tracer relies on the Conditional Debug to determine which
packets are interesting. The condition infra provides the ability to
filter by protocol, IP address and mask, ACL, interface and direction
• Conditions define what the filters are and when the filters are
applied to a packet. For example, “debug platform condition
interface g0/0/0 egress” means that a packet will be identified as a
match when it reaches the output FIA on interface g0/0/0 so any
packet-processing that took place from ingress up to that point is
missed
• It is recommended to use ingress conditions for Packet Tracer to
get the most complete and meaningful data. Egress conditions can
be used, but just be aware of the limitation above

• Packet Trace captures different levels of packet processing detail
and provides commands to display the captured data
• Four detail levels:
1) Accounting
2) Packet summary
3) Packet details
4) Packet details with FIA trace and optional packet copy
• Packet details, FIA trace and packet copy are collected per packet
when the packet is processed in data path. The detailed information
collected is commonly referred to as “Path Data”

• Accounting (or statistics) level is always enabled if Packet Tracer is
enabled. Per-packet info is not collected in this mode. Performance
impact is low
Packets Summary
Matched 31
Traced 2
Packets Received
Ingress 31
Inject 0
Packets Processed
Forward 0
Punt 31
Count Code Cause
10 3 Layer2 control and legacy
3 7 ARP request or response
7 11 For-us data
9 21 RP<->QFP keepalive
2 27 Subscriber session control
Drop 0
Consume 0
Packets matched by conditional debug
Packets traced:
- limited by the max number of traced packets configured
- or PACTRAC can set additional creteria (e.g. PUNT code #27)
Forward – “ready to go to SIP/SPA”
Punt and drop codes are printed for
punted and dropped packets
Packets consumed by data path code
This command is required for all detail levels

• Per-packet info is collected: input and output interfaces, final packet
state, punt/inject/drop codes and tracing start and stop timestamps
• Collecting summary data uses little performance over the normal
packet processing
• An example usage may be to isolate which interfaces are dropping
traffic so more detailed inspection can be used after applying
interface specific conditions
debug platform packet-trace packet <16-8192> [circular] summary-only
0 Gi0/0/0.27 Gi0/0/2 FWD
...
64 Gi0/0/0.27 Gi0/0/2 FWD
...
Punt and drop codes are printed for
punted and dropped packets
What happened with each packet

• Summary information is always collected whenever any per packet
data is collected. The summary information is displayed by the
“summary” command and also the “per packet” command
0 Gi0/0/0 internal0/0/rp:0 PUNT 27 (Subscriber session control
1 Gi0/0/0 internal0/0/rp:0 PUNT 27 (Subscriber session control
Summary
Input : GigabitEthernet0/0/0
Output : internal0/0/rp:0
State : PUNT 27 (Subscriber session control
Timestamp
Start : 4994905059758 ns (12/13/2014 19:23:54.523840 UTC)
Stop : 4994905077772 ns (12/13/2014 19:23:54.523858 UTC)
Summary info for specified packet

• Path data may be collected per packet for a limited number of
packets and is made up of different types of data as follows:
Common path data (e.g. IP tuple)
Feature specific data (major features only, e.g. NAT, QoS, VPN, ZBF, etc.)
Feature Invocation Array (FIA) trace – if enabled
Packed dump – if enabled
• Capturing per packet data requires the use of QPF DRAM
• Capturing path data has the greatest impact on packet processing
capability specifically FIA trace and packet copy
FIA tracing creates many path data entries costing instructions and DRAM writes
Packet copy creates many DRAM read/writes

• The “data-size” option allows the user to specify the size of the path
data buffers used to store per feature and FIA-trace data. The
default value is currently 2048 and need not be changed
• Using circular mode means that all matching packets are traced
until Packet Trace is halted so it has a greater impact on system
resources
• Packet copy:
“input” - copy the packet when the packet is injected or seen on ingress interface
“output” - copy the packet at the moment of drop, punt or forward
“both” - copy the packet twice
start the copy from l2/l3/l4 header
the default packet size is 64
debug platform packet-trace packet <16-8192> [circular] [data-size <2048-16384>] [fia-trace]
debug platform packet-trace copy packet {input | output | both} [size <16-2048>] {l2 | l3 | l4}

• User config affects µcode performance and QFP DRAM usage based on
the type and amount of tracing requested
• Packet Tracer statistics
Always tracked if PACTRAC enabled (“debug platform packet-trace enable”)
Least performance impact
• Per packet summary data
Always collected if per packet enabled (“debug platform packet-trace packet ...”)
Minor performance impact
• Per packet feature path data
Enabled by default when per packet enabled, can be disabled with “summary-only”
Variable performance impact – totally depends on feature mix
• Per packet ingress/egress packet copy
Enabled when per packet and packet copy enabled
Noticeable performance impact

• XE3.11 – Drop Tracing, XE3.13 – Punt/Inject Tracing
• XE3.14 – List of Drop/Punt/Inject codes
• Drop and Punt tracing can be enabled with and without conditions
• When enabled with conditions, the per-packet data is collected for
all packets matched, but then collected data is discarded if the
packet wasn’t dropped (or punted) – performance impact similar to
“circular” mode
• When enabled without conditions, only the drop event is traced –
very low performance impact, but information collected is limited
• “debug platform condition start” is still required
debug platform packet-trace {punt | inject | drop} [code <0-65534>]
show platform packet-trace code {drop | punt | inject}

• XE3.11: You can use embedded decoder, but only few protocol
dissectors are currently supported (CSCul62487)
• This simple script can help decode a single packet 
show platform packet-trace packet {<number> | all} [decode]
#!/usr/bin/perl
foreach (<>) {
s/[^a-fA-F0-9]//g;
print join("", pack("H*", $_));
}
cat packet.txt | hex2der.pl | od -t x1 | text2pcap -o oct - packet.pcap
Create this script, save file as hex2der.pl
Don’t forget to run “chmod 700 ./hex2der.pl”
To add fake Ethernet header run text2pcap with -e 0x0800

• This simple example illustrates the interactions between NAT and
output packet copy
Conditional Debug Global State: Start
----------------------------------------------------------------------|---------
GigabitEthernet0/0/0 & IPV4 [10.1.75.2/32] egress
debug platform packet-trace packet 16 fia-trace data-size 2048
debug platform packet-trace copy packet output size 2048 L2
interface GigabitEthernet0/0/0
ip address 10.48.66.159 255.255.254.0
ip nat outside
interface GigabitEthernet0/0/1.75
encapsulation dot1Q 75
ip address 10.1.75.1 255.255.255.0
ip nat inside
We’re going to capture packets on
NAT outside interface on “output”.

• Packet Tracer will start tracing packets as soon as they reach
egress interface FIA, but packet copy will happen after NAT when
the packets are about to be transmitted to a SIP module
show platform hardware qfp active interface if-name g0/0/0
...
FIA handle - CP:0x108db890 DP:0x80791c80
CBUG_OUTPUT_FIA
IPV4_OUTPUT_VFR
IPV4_OUTPUT_NAT
IPV4_VFR_REFRAG (M)
DEBUG_COND_OUTPUT_PKT
DEF_IF_DROP_FIA (M)
“match” by inside IP,
but “copy” after NAT

show platform packet-trace packet 0 decode
Summary
State : FWD
Timestamp
Start : 461570571226
Stop : 461570727146
Path Trace
Feature: IPV4
Source : 10.1.75.2
Protocol : 1 (ICMP)
Feature: FIA_TRACE
Entry : 0x803550d8 - IPV4_OUTPUT_VFR
Timestamp : 461570576503
Feature: FIA_TRACE
Entry : 0x802a7f40 - IPV4_OUTPUT_NAT
Timestamp : 461570577819
Feature: NAT
Direction : IN to OUT
Action : Translate Source
Old Address : 10.1.75.2 00013
New Address : 10.48.66.159 00002
...

...
Packet Copy Out
0006f62a c4a30021 d89a0600 08004500 0064003d 0000fe01 235c0a30 429f0a30
42010800 33eb0002 00000000 000009f1 406cabcd abcdabcd abcdabcd abcdabcd
abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd abcdabcd
abcdabcd abcdabcd abcdabcd abcdabcd abcd
Ethernet
Destination MAC : 0006f62ac4a3
Source MAC : 0021d89a0600
Type : 0x0800 (IPV4)
IPv4
Version : 4
Header Length : 5
ToS : 0x00
Total Length : 100
Identifier : 0x003d
IP Flags : 0x0
Frag Offset : 0
TTL : 254
Protocol : 1 (ICMP)
Header Checksum : 0x235c
Source Address : 10.48.66.159
Destination Address : 10.48.66.1
ICMP
Type : 8
Code : 0x00
Checksum : 0x33eb
Identifier : 0x0002
Sequence : 0x0000
Translated IP address

• Что по вашему мнению необходимо сделать в первую очередь,
чтобы улучшить мнение пользователей о платформах ASR1k и
ISR4400/4300?
 Выпустить еще больше бессмысленных маркетинговых брошюр
 Написать наконец нормальную документацию
 Выпустить в Cisco Press пару книжек о них
 Повысить надежность работы ПО
 Повысить надежность аппаратуры
 Отказаться от IOS-XE. Нам ни к чему все эти сложности

• Embedded Packet Capture (EPC) is a powerful troubleshooting and
tracing tool, it allows for network administrators to capture data
packets flowing through, to, and from a Cisco router
• EPC is a software feature consisting of infrastructure to allow for
packet data to be captured at various points. The network
administrator may define the capture buffer to save capture and
capture filter to customize the capture rules
• http://www.cisco.com/c/en/us/td/docs/ios-
xml/ios/epc/configuration/xe-3s/epc-xe-3s-book.html

IP cloud
ASR1000
SPA
Driver
SPA
Driver
SPA
Driver
SPA
Driver
IOSd
QFP ESP
SIP
RP
Replicate with
classification
Punt
Data
Data
Data

• Configuration
• Supported interfaces
• Up to 8 concurrent sessions (captures)
• More than one interface in each session
• Classification by ACL (only named ACLs supported!), class-map or
inline “match”
monitor capture <name> {interface <name> | control-plane} {in | out | both} {access-list <name> |
class-map <name> | match {any | ipv4 | ipv6 | mac} <criteria>} [<options>]
For control-plane:
“in” – Inject
“out” – Punt
monitor capture cap1 interface ?
GigabitEthernet GigabitEthernet IEEE 802.3z
Multilink Multilink-group interface
Port-channel Ethernet Channel of interfaces
Tunnel Tunnel interface

• Capture options
• Defaults:
linear buffer
10MB buffer
40,000pps max
no sampling
entire packets are captured
monitor capture cap1 [buffer size <1-2000 MB>] [circular]
monitor capture cap1 [limit [packets <1-100000>] [duration <sec>] [every <Nth>] [packet-len <64-
9500>] [pps <pps>]]

• Configuration
ip access-list extended A198
permit ip host 192.168.2.1 host 192.168.1.1
monitor capture cap1 interface tunnel 1 in access-list A198
show monitor capture cap1
Status Information for Capture cap1
Target Type:
Interface: Tunnel1, Direction: in
Status : Inactive
Filter Details:
Access-list: A198
Buffer Details:
Buffer Type: LINEAR (default)
Limit Details:
Number of Packets to capture: 0 (no limit)
Packet Capture duration: 0 (no limit)
Packet Size to capture: 0 (no limit)
Packet sampling rate: 0 (no sampling)
show monitor capture cap1 parameter
monitor capture cap1 interface Tunnel1 in
monitor capture cap1 access-list A198

• Capture buffer
monitor capture cap1 start
show monitor capture cap1 buffer
buffer size (KB) : 10240
buffer used (KB) : 128
packets in buf : 5
packets dropped : 0
packets per sec : 113
show monitor capture cap1 buffer brief
-------------------------------------------------------------
# size timestamp source destination protocol
-------------------------------------------------------------
0 114 0.000000 192.168.2.1 -> 192.168.1.1 ICMP
1 114 0.001999 192.168.2.1 -> 192.168.1.1 ICMP
2 114 0.014999 192.168.2.1 -> 192.168.1.1 ICMP
3 114 0.016998 192.168.2.1 -> 192.168.1.1 ICMP
4 114 0.044996 192.168.2.1 -> 192.168.1.1 ICMP

• Capture buffer
show monitor capture cap1 buffer detailed
-------------------------------------------------------------
# size timestamp source destination protocol
-------------------------------------------------------------
0 114 0.000000 192.168.2.1 -> 192.168.1.1 ICMP
0000: 00000000 00000000 00000000 08004500 ..............E.
0010: 006486F5 0000FF01 B050C0A8 0201C0A8 .d.......P......
0020: 01010800 AC410018 00000000 00008404 .....A..........
0030: 4DECABCD ABCDABCD ABCDABCD ABCDABCD M...............
…
show monitor capture cap1 buffer dump
0
0000: 00000000 00000000 00000000 08004500 ..............E.
0010: 006486F5 0000FF01 B050C0A8 0201C0A8 .d.......P......
0020: 01010800 AC410018 00000000 00008404 .....A..........
0030: 4DECABCD ABCDABCD ABCDABCD ABCDABCD M...............
0040: ABCDABCD ABCDABCD ABCDABCD ABCDABCD ................
0070: ABCD

• Other commands
! Stop Capture session
monitor capture cap1 stop
! Export capture buffer
monitor capture cap1 export <URL>
! Clear capture buffer
monitor capture cap1 clear
! Clear configuration
no monitor capture cap1

• EPC per-cause punt policer
show platform hardware qfp active infrastructure punt statistics type per-cause | i Punt
Cause|Packets|Counter ID|075
Packets Packets
075 EPC 5 5
show platform software punt-policer | i ^ 75|pps|Cause
Per Punt-Cause Policer Configuration and Packet Counters
Punt Configured (pps) Conform Packets Dropped Packets
Cause Description Normal High Normal High Normal High
75 EPC 40000 1000 5 0 0 0
conf t
platform punt-policer 75 <new-value> [high]

• http://www.cisco.com/c/en/us/support/routers/asr-1000-series-
aggregation-services-routers/products-tech-notes-list.html
• http://www.ciscolive.com/global/

• Standard support releases
18 months lifetime, 3 scheduled rebuilds
3.11S, 3.12S, 3.14S, 3.15S, etc.
• Extended support releases
48 months lifetime, 8 scheduled rebuilds
3.10S, 3.13S, 3.16S, etc.
• http://www.cisco.com/c/en/us/products/collateral/routers/asr-1000-
series-aggregation-services-routers/product_bulletin_c25-
726436.html

• Используйте панель Q&A, чтобы задать вопрос
• Наши эксперты ответят на них

Получить дополнительную информацию, а также задать
вопросы эксперту в рамках данной темы Вы можете на
странице, доступной по ссылке:
https://supportforums.cisco.com/community/russian/expert-corner
Вы можете получить видеозапись данного семинара и текст
сессии Q&A в течении ближайших 5 дней по следующей ссылке
https://supportforums.cisco.com/community/russian/expert-corner/

Тема: VoLTE – технологии передачи голоса в LTE
сети
в среду, 20 мая, в 12.00 мск
Присоединяйтесь к эксперту Cisco
Владимиру Суконкину
Во время презентации эксперт Cisco Владимир Суконкин
рассмотрим архитектуру голосовых сервисов поверх LTE
сети (VoLTE), а так же технологии для поэтапного перехода
от существующей традиционной 2G/3G сети к VoLTE
архиетектуре.

• http://www.facebook.com/CiscoSupportCommunity
• http://twitter.com/#!/cisco_support
• http://www.youtube.com/user/ciscosupportchannel
• https://plus.google.com/110418616513822966153?prsrc=3#110418616513822
966153/posts
• http://itunes.apple.com/us/app/cisco-technical-support/id398104252?mt=8
• https://play.google.com/store/apps/details?id=com.cisco.swtg_android
• http://www.linkedin.com/groups/CSC-Cisco-Support-Community-3210019
• Newsletter Subscription:
https://tools.cisco.com/gdrp/coiga/showsurvey.do?surveyCode=589&keyCode=
146298_2&PHYSICAL%20FULFILLMENT%20Y/N=NO&SUBSCRIPTION%20
CENTER=YES

Если вы говорите на Испанском, Португальском или Японском,
мы приглашаем вас принять участие в сообществах:
Русский язык:
https://supportforums.cisco.com/community/russian
Испанский язык:
https://supportforums.cisco.com/community/5591/comunidad-de-soporte-de-cisco-en-
espanol
Португальский язык:
https://supportforums.cisco.com/community/5141/comunidade-de-suporte-cisco-em-
portugues
Японский язык:
http://www.csc-china.com.cn/

Спасибо за
Ваше время
Пожалуйста, участвуйте в опросе

Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400

Similar to Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400 (20)

More from Cisco Russia

More from Cisco Russia (20)

Внутренняя архитектура IOS-XE: средства траблшутинга предачи трафика на ASR1k и ISR4400