Troubleshooting Firewalls (2012 San Diego)

38,647 views
38,326 views

Published on

This presentation focuses on preemptive measures and reactive techniques that can be used to troubleshoot, secure, and maintain the Cisco Adaptive Security Appliance Products and the Cisco Firewall Services Module (FWSM). Providing an in-depth understanding of the packet flow through the firewall device, as well as how to effectively utilize the available commands and on-board tools to troubleshoot connectivity problems are the main goals of this presentation. Knowledge is assumed of security fundamentals and firewall technology at the level presented in the Cisco Networkers Online Introduction to Firewalls and Deploying Firewalls.

Cisco Live 365: https://www.ciscolive365.com/connect/sessionDetail.ww?SESSION_ID=4377

Published in: Technology, Business
0 Comments
44 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
38,647
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
8
Comments
0
Likes
44
Embeds 0
No embeds

No notes for slide

Troubleshooting Firewalls (2012 San Diego)

  1. 1. Troubleshooting Firewalls Session BRKSEC-3020BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public
  2. 2. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
  3. 3. Packet Flow
  4. 4. Understanding the Packet Flow To effectively troubleshoot a problem, one must first understand the packet path through the network Attempt to isolate the problem down to a single device Then perform a systematic walk of the packet path through the device to determine where the problem could be For problems relating to the Cisco ASA/FWSM, always ‒ Determine the flow: SRC IP, DST IP, SRC port, DST port, and protocol ‒ Determine the interfaces through which the flow passes Note: All Firewall Issues Can Be Simplified to Two Interfaces (Ingress and Egress) and the Rules Tied to BothBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
  5. 5. Example Flow Flow SRC IP: 10.1.1.9 SRC Port: 11030 Protocol: TCP DST IP: 198.133.219.25 DST Port: 80 Interfaces Source: Inside Destination: Outside Client: 10.1.1.9 With the Flow Servers Packet Flow Defined, Examination of Configuration Eng Accounting Issues Boils Down to Just the Two Outside Interfaces: Inside and Outside Server: 198.133.219.25BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
  6. 6. Packet Processing: Ingress Interface CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Packet arrives on ingress interface Input counters incremented Software input queue is an indicator of load No buffers and overruns indicates packet drops (usually traffic bursts) ASA# show interface GigabitEthernet0/1 Interface GigabitEthernet0/1 "", is up, line protocol is up Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps) Input flow control is unsupported, output flow control is off Active member of Port-channel10 MAC address 4055.3980.0e67, MTU 1500 IP address unassigned 410179 packets input, 42620992 bytes, 0 no buffer Received 138749 broadcasts, 0 runts, 0 giants …BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
  7. 7. Packet Processing: Locate Connection CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Check first for existing connection If connection exists, flow is matched: bypass ACL check If no existing connection ‒ TCP non-SYN packet, drop and log ‒ TCP SYN or UDP packet, pass to ACL checks Established Connection: ASA-5540# show conn TCP out 198.133.219.25:80 in 10.1.1.9:11030 idle 0:00:04 Bytes 1293 flags UIO Syslog Because of No existing connection, and non-SYN Packet: ASA-6-106015: Deny TCP (no connection) from 10.1.1.9/11031 to 198.133.219.25/80 flags PSH ACK on interface insideBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
  8. 8. Packet Processing: NAT Un-Translate CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Incoming packet is checked against NAT rules Starting version 8.3+, packet is un-translated first, before ACL check ‒ In version 8.2, incoming packet was subjected to ACL check prior to un-translation More on this in the NAT section of this presentation…BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
  9. 9. Packet Processing: ACL Check CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP First packet in flow is processed through interface ACLs ACLs are first match First packet in flow matches ACE, incrementing hit count by one Denied packets are dropped and logged Packet Permitted by ACL: ASA-5540B# show access-list inside access-list inside line 10 permit ip 10.1.1.0 255.255.255.0 any (hitcnt=1) Syslog when Packet Is Denied by ACL: ASA-4-106023: Deny tcp src inside:10.1.1.9/11034 dst outside:198.133.219.25/80 by access- group "inside"BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
  10. 10. Packet Processing: Inspections/Sec Checks CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Inspections are applied to ensure protocol compliance  (Optional) customized AIC inspections  NAT-embedded IPs in payload  Additional security checks are applied to the packet  (Optional) packets passed to Content Security and Control (CSC) module Syslog from Packets Denied by Security Check: ASA-4-406002: FTP port command different address: 10.2.252.21(192.168.1.21) to 209.165.202.130 on interface inside ASA-4-405104: H225 message received from outside_address/outside_port to inside_address/inside_port before SETUP BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
  11. 11. Packet Processing: NAT IP Header CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Translate the IP address in the IP header  Translate the port if performing PAT  Update checksums  (Optional) Following the above, pass packet to IPS (AIP) module BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
  12. 12. Packet Processing: Egress Interface CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Packet is virtually forwarded to egress interface (i.e., not forwarded to the driver yet)  Egress interface is determined first by translation rules NOT the routing table  If translation rules do not specify egress interface (e.g., outbound initial packet) the results of a global route lookup are used to determine egress interface  Example: Inbound Packets to Inside Outside 192.168.12.4 Get Routed to 172.16.0.0/16 Inside Based on Order of Statics DMZ 172.16.12.0/24 172.16.12.4 nat (inside,outside) source static 172.16.0.0-net 192.168.0.0-net nat (dmz,outside) source static 172.16.12.0-net 192.168.12.0-net BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
  13. 13. Packet Processing: L3 Route Lookup CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Once on egress interface, an interface route lookup is performed  Only routes pointing out the egress interface are eligible  Remember: NAT rule can forward the packet to the egress interface, even though the routing table may point to a different interface Syslog from Packet on Egress Interface with No Route Pointing Out Interface: %ASA-6-110003: Routing failed to locate next hop for TCP from inside:192.168.103.220/59138 to dmz:172.18.124.76/23 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
  14. 14. Packet Processing: L2 Address Lookup CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Once a Layer 3 route has been found, and next hop identified, Layer 2 resolution is performed  Layer 2 rewrite of MAC header  If Layer 2 resolution fails—no syslog  show arp will not display an entry for the L3 next hop  debug arp will indicate if we are not receiving an ARP reply arp-req: generating request for 10.1.2.33 at interface outside arp-req: request for 10.1.2.33 still pending BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
  15. 15. Packet Processing: Transmit Packet CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 Yes RX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route Addr Pkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP  Packet is transmitted on wire  Interface counters will increment on interface ASA# show interface GigabitEthernet 0/2 Interface GigabitEthernet0/2 "", is up, line protocol is up Hardware is bcm56800 rev 01, BW 1000 Mbps, DLY 10 usec Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps) Input flow control is unsupported, output flow control is off Active member of Port-channel11 MAC address 4055.3980.0e68, MTU 1500 IP address unassigned … 101315 packets output, 13086040 bytes, 0 underruns 0 pause/resume output 0 output errors, 0 collisions, 0 interface resets 0 late collisions, 0 deferred 0 rate limit drops 0 switch egress policy drops 0 input reset drops, 0 output reset drops BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
  16. 16. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
  17. 17. Cisco ASA — Understanding the Architecture ASA processes all packets in software (via the central CPU) All packets are processed first in… usually also first out ASA platforms have software imposed connection limits Multi-CPU / Multi-Core systems hash packets in the same flow to the same CPU/core. 10 Gig interfaces hash flow to same RX ring. Architecture optimized for multi-flow traffic patterns ASASM packet processing is also done in software, unlike FWSMBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
  18. 18. Cisco ASA — Understanding the Architecture Global Shared Memory Pool ‒ Global Shared Pool of Memory used by ASA processes Show free memory with CLI, ASDM, or ‗show memory‘ and ‗show memory detail‘ commands ASA# show memory Free memory: 250170904 bytes (47%) Used memory: 286700008 bytes (53%) ------------- ------------------ Total memory: 536870912 bytes (100%) ASA# Use ASDM to graph free memory available • If available memory trends down over time, call Cisco TAC %ASA-3-211001: Memory allocation Error • For 64-bit software (8.4+) use CISCO-ENHANCED-MEMPOOL-MIB.my for accurate countersBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
  19. 19. Cisco ASA — Understanding the Architecture Memory Blocks ‒ Fixed-size blocks of memory allocated at startup, used for packet processing, VPN, etc Current number of free ASA# show blocks blocks available SIZE MAX LOW CNT 0 400 397 400 4 100 99 99 80 403 379 401 256 1200 1190 1195 1550 6511 803 903 2048 1200 1197 1200 2560 264 264 264 4096 100 100 100 8192 100 100 100 Use ASDM to graph free blocks available 16384 102 102 102 65536 16 16 16 ASA# 1550 and 2048 byte blocks are used for processing ethernet frames %ASA-3-321007: System is low on free memory blocks of size 1550 (10 CNT out of 7196 MAX)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
  20. 20. Maximum ACL Limits  No hard-coded limit on the number of elements (ACEs) in an ACL. Bound only by Memory.  Each ACE uses a minimum of 212 bytes of RAM  However, maximum performance may decrease (typically 10- 15%) as you reach or exceed the Max Recommended ACEs. 5585 ASA 5505 5510 5512-X 5515-X 5520 5525-X 5540 5545-X 5550 5555-X 5580 10/20/40/6 SM 0 Max 500k / 750kRecommended 25k 80k 100k 100k 200k 200k 500k 300k 700k 500k 750k 2 million 1 / 2 million ACEs 1 500k / 750k Tested ACEs 80k 300k 700k 700k 2 million million+ 1 / 2 millionMax Observed 15.959 3.5 million 2.74 million(from customers) million (SSP-40) Note: Issue show access-list | include elements to see how many ACEs you have BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
  21. 21. Warning - ACE Explosion Object-groups: Single line ACL • Sources (10 addresses) explodes to • Destinations (21 addresses) • Ports (33 ports) • Result: 10x21x33 = 6,930 rules Nested object-groups: • Assume you add a SRC object-group to the above, which contains 25 additional sources • Result: (10+25)x21x33 = 24,255 rules (ACEs)• New command to reduce ACL memory impact for large ACLs. Available starting in 8.3(1) ASA-5585(config)# object-group-search access-controlBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
  22. 22. Global ACLs ASAInterface Independent Policies Only Global ACLs introduced in version 8.3 Best used for new installations, or config migration from other vendors ASA(config)# access-group <access_list> global Policy Ordering Interface Specific access-list Global access-list Default (implicit) deny ip any anyBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
  23. 23. Object-NAT (Auto-NAT) (version 8.3+) Object NAT is the simplest form of NAT, and is defined within an object Host NAT object network obj-WebServer host 10.3.19.50 nat (inside,outside) static 198.51.100.50 Network NAT object network Servers subnet 10.0.54.0 255.255.255.0 nat (inside,outside) static 203.0.113.0 Dynamic PAT (interface overload) object network InternalUsers subnet 192.168.2.0 255.255.255.0 nat (inside,outside) dynamic interface BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
  24. 24. Manual-NAT (Twice-NAT) (version 8.3+) Manual NAT can specify the source and the destination translation Network Objects object network 10.10.10.0-net subnet 10.10.10.0 255.255.255.0 ! object network 192.168.1.0-net subnet 192.168.1.0 255.255.255.0 Manual NAT Config nat (inside,outside) source static 10.10.10.0-net 10.10.10.0-net destination static 192.168.1.0-net 192.168.1.0-net BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
  25. 25. NAT Order of Operation version 8.3+ For your reference The ASA configuration is built into the NAT table The NAT Table is based on First Match (top to bottom) The show nat command will display the NAT table in order NAT Table Static NAT Manual NAT Policies First Match Longest Prefix (Section 1) (in config) Shortest Prefix Auto NAT Policies Dynamic NAT (Section 2) Longest Prefix Manual NAT [after auto] Policies First Match Shortest Prefix (Section 3) (in config) BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
  26. 26. Network Address Translation in ASA 8.3+Optional NAT Parameters in Manual NAT For your reference no-proxy-arp 8.4.(2)+/8.5(1) ‒ For static NAT, disables proxy ARP for incoming packets to the mapped IP addresses. route-lookup 8.4.(2)+/8.5(1) ‒ For identity NAT in routed mode, determines the egress interface using a route lookup instead of using the interface specified in the NAT command. If you do not specify interfaces in the NAT command, a route lookup is used by default. pat-pool <obj> 8.4.(2)+/8.5(1) ‒ Enables a PAT pool of addresses; all addresses in the object are used as PAT addresses and not dynamic 1- to-1 round-robin 8.4.(2)+/8.5(1) ‒ Enables round-robin address allocation for a PAT pool. By default, all ports for a PAT address will be allocated before the next PAT address is used. Round Robin will pick one address from each PAT address extended 8.4.(3) ‒ Enables extended PAT for a PAT pool. Extended PAT uses 65535 ports per service, as opposed to per IP address, by including the destination address and port in the translation information.BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
  27. 27. Real-IP (version 8.3+) Finally, a reminder that with 8.3+ Real-IPs are used in ACLs ACL contains REAL object network obj-WebServer (local) IP of server host 10.3.19.50 nat (inside,outside) static 198.51.100.50 ! access-list allowIn permit tcp any host 10.3.19.50 eq 80 ! access-group allowIn in interface outside BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
  28. 28. FWSM—Understanding the Architecture FWSM Processes most packets in hardware, with some packets needing to be processed in software—via the Control Point (CP) Packets processed in hardware have zero impact on CPU Similarly, if the CPU is pegged at 100%, this has zero impact on packets processed in hardware Note that FWSM packet processing is different from ASABRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
  29. 29. FWSM Architectural Overview Control Point (CP) Control Point Central CPU ACL Compilation, Software Fixups, Syslog, AAA, IPv6 in Hardware Software Session Manager Session Manager NP 3 Session Establishment and Teardown, AAA Cache, ACLs Fast Path Fast Path NP 1 Fast Path NP 2 Flow Identification Security Checks and FWSM C6K Backplane Interface NAT in hardwareBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
  30. 30. FWSM—Hardware Limits See Appendix FWSM has several hardware limits that should be considered in your network design Limits are hard set, but vary based on single or multimode Some limits include: Increase over 2.3 Increase over 3.2 3.2 / 4.0 /4.1 2.3 (Multimode) 3.1/3.2 (Multimode) 4.0/4.1 (Multimode) Configurable ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X Global Statements 1K (1K) 4K (4K) 4K (4K) Static NAT Statements 2K (2K) 2K (2K) 2K (2K) Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X NAT Translations 256K (256K) 256K (256K) 256K (256K) Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990) Route Table Entries 32K (32K) 32K (32K) 32K (32K) Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X Filter Statements 3942 (606) 2764 (425) 3747 (576) X *Complete List in FWSM Docs, Appendix A (Specifications)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
  31. 31. show np blocks (FWSM Only) The show np blocks command is used to see if the FWSM is over subscribed Data and Control Data packets Warning packets dropped dropped FWSM# show np blocks MAX FREE THRESH_0 THRESH_1 THRESH_2 NP1 (ingress) 32768 32768 0 0 550 (egress) 521206 521206 0 0 0 NP2 (ingress) 32768 32768 0 0 92 (egress) 521206 521206 0 0 0 NP3 (ingress) 32768 32768 13 460417 4427509 (egress) 521206 521206 0 0 0 BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
  32. 32. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
  33. 33. Failover Basics Internet  Active/Standby vs. Primary/Secondary  Stateful failover (optional)  A failover only occurs when either firewall determines the standby firewall is healthier than the active firewall  Both firewalls swap MAC Stateful and IP addresses when a failover occurs LAN Link  Level 1 syslogs will give reason of failover Secondary Primary (Standby) (Active) Corp BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
  34. 34. Verifying Failover Configuration ASA# show failover Failover On Failover unit Primary Failover LAN Interface: failover Redundant5 (up) Unit Poll frequency 200 milliseconds, holdtime 1 seconds Interface Poll frequency 500 milliseconds, holdtime 5 seconds Interface Policy 1 Monitored Interfaces 2 of 250 maximum Version: Ours 8.2(2), Mate 8.2(1) Last Failover at: 10:37:11 UTC May 14 2010 This host: Primary - Active Interface Monitoring Active time: 1366024 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys) Interface outside (10.8.20.241): Normal Interface inside (10.89.8.29): Normal Other host: Secondary - Standby Ready Active time: 0 (sec) slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys) Interface outside (10.8.20.242): Normal Interface inside (10.89.8.30): Normal Stateful Failover Logical Update Statistics Link : stateful Redundant6 (up) Stateful Obj xmit xerr rcv rerr General 424525 0 424688 0 sys cmd 423182 0 423182 0BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
  35. 35. What Triggers a Failover? Power loss/reload (this includes crashes) on the Active firewall SSM interface/module failure The Standby becoming healthier than the Active firewall In LAN based Failover, what happens if the LAN interface communication is severed?BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
  36. 36. What Triggers a Failover? (Con’t) Two consecutive hello messages missed on any monitored interface forces the interface into testing mode Both units first verify the link status on the interface Next, both units execute the following tests Network activity test ARP test Broadcast ping test The first test passed causes the interface on that unit to be marked healthy; only if all tests fail will the interface be marked failedBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
  37. 37. How Well do you Understand Failover?What Happens When… You disable failover? (By issuing no failover) You RMA/Replace the Primary unit? You don‘t define Standby IP addresses on interfaces?BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
  38. 38. What to Do After a Failover Always check the syslogs to determine root cause ‒ Example: switch port failed on inside interface of active firewall Syslogs from Primary (Active) ASA ASA-4-411002: Line protocol on Interface inside, changed state to down ASA-1-105007: (Primary) Link status ‗Down‘ on interface 1 ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier Syslogs from Secondary (Standby) ASA ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me ActiveBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
  39. 39. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
  40. 40. Troubleshooting Tools Syslogs Debug commands Show commands Packet capture Packet tracer TCP PingBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
  41. 41. Uses of Syslogs Primary mechanism to record traffic to and through the firewall The best troubleshooting tool available Archival Purposes Debugging Purposes Console Syslog/FTP Server ASDM flash Local Trap Syslog. Buffer SNMP Server Monitor (ssh, telnet)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
  42. 42. ASA Syslog Level vs. Number of Messages Number of Messages (SUM) Log Description Level Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1 Ver. 8.2 Ver. 8.3 Ver. 8.4 0 Emergencies 0 0 0 0 0 0 0 1 Alerts 62 (62) 77 (77) 78 (78) 87 (87) 87 (87) 95 (95) 109 (109) 2 Critical 29 (91) 35 (112) 49 (127) 50 (137) 56 (143) 57 (152) 63 (172) More messages 3 Errors 274 (365) 334 (446) 361 (488) 363 (500) 384 (527) 408 (560) 448 (620) 4 Warnings 179 (544) 267 (713) 280 (768) 281 (781) 315 (842) 324 (884) 357 (997) 5 Notifications 161 (705) 206 (919) 216 (984) 218 (999) 237 (1079) 246 (1130) 265 (1242) 6 Informational 234 (939) 302 (1221) 335 (1319) 337 (1336) 368 (1447) 377 (1507) 395 (1637) 217 7 Debugging 258 (1479) 266 (1585) 267 (1603) 269 (1716) 269 (1776) 276 (1913) (1156)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
  43. 43. What Are Modifiable Syslog Levels? [no] logging message <syslog_id> level <level>  Modifiable syslog levels Levels Allows one to move any syslog message to any level 0—Emergency 1—Alert  Problem 2—Critical You want to record what exec commands are being executed on the firewall; syslog ID 111009 3—Errors records this information, but by default it is at 4—Warnings level seven (debug) 5—Notifications %ASA-7-111009: User ‗johndoe‘ 6—Informational executed cmd: show run 7—Debugging The problem is we don‘t want to log all 1775 other syslogs that are generated at debug level BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
  44. 44. How to Create Modifiable Syslog Levels  Lower syslog message 111009 to level 3 (error) ASA(config)# logging message 111009 level 3 Ifyou were syslog looks as follows Now our only interested in logging one syslog message, how could you do it? ASA-3-111009: User ‗johndoe‘ executed cmd: show run  To restore the default syslog level ASA(config)# no logging message 111009 level 3 Tip: Use show logging message all to see the default level for any message BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
  45. 45. Logging – Common Issues  logging flash-bufferwrap – should only be used when logging to buffer at Level 1  logging history – should only be used when you really have an SNMP server that you want to receive all syslogs  logging console – should only be enabled while actively troubleshooting on the Console  logging standby – should only be used if you want to receive double the syslogs  logging permit-hostdown – should always be used with TCP syslogging BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
  46. 46. Debug Commands1. Debugs should not be the first choice to troubleshoot a problem2. Debugs can negatively impact the CPU of the box, and also the performance of it; use with caution3. Debugs are not conditional*4. Know how much traffic, of the specified type, is passing through the firewall before enabling the respective debugBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
  47. 47. Show Output Filters See Appendix show <cmd> | begin|include|exclude|grep [-v] <regular_exp>  Use output filters to filter the output of show command to only the information you want to see  To use them, at the end of show <Command>, use the pipe character ―|‖ followed by ‒begin Start displaying the output beginning at the first match of the RegEx, and continue to display the remaining output ‒include Display any line that matches the RegEx ‒exclude Display any line that does not match the RegEx ‒grep Same as include ‒grep –v Same as exclude BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
  48. 48. show cpu usage Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled The show cpu usage command displays the CPU over time as a running average ASA# show cpu usage CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
  49. 49. show processes cpu-usage The show processes cpu-usage command displays the amount of CPU used on a per-process basis for the last 5sec, 1min, and 5min ASA# show process cpu-usage sorted non-zero PC Thread 5Sec 1Min 5Min Process 0x08dc4f6c 0xc81abd38 14.4% 8.2% 8.0% SNMP Notify Thread 0x087798cc 0xc81b0658 6.8% 5.0% 4.9% esw_stats 0x081daca1 0xc81bcf70 1.3% 1.1% 1.0% Dispatch Unit 0x08e7b225 0xc81a28f0 1.2% 0.1% 0.0% ssh 0x08ebd76c 0xc81b5db0 0.6% 0.3% 0.3% Logger 0x087b4c65 0xc81aaaf0 0.1% 0.1% 0.1% MFIB 0x086a677e 0xc81ab928 0.1% 0.1% 0.1% ARP Thread ASA# *First Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
  50. 50. Show traffic The show traffic command displays the traffic received and transmitted out each interface of the firewall since last ‗clear traffic ‘ ASA# show traffic outside: received (in 124.650 secs): 295468 packets 167218253 bytes 2370 pkts/sec 1341502 bytes/sec transmitted (in 124.650 secs): 260901 packets 120467981 bytes 2093 pkts/sec 966449 bytes/sec inside: received (in 124.650 secs): 261478 packets 120145678 bytes 2097 pkts/sec 963864 bytes/sec transmitted (in 124.650 secs): 294649 packets 167380042 bytes 2363 pkts/sec 1342800 bytes/secBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
  51. 51. show xlate and show xlate debug The show xlate command displays information about the translations through the firewall You can limit the output to just the local or global IP ASA-5585# show xlate 5014 in use, 5772 most used TCP PAT from inside:192.168.103.220/57762 to outside:10.2.1.2/43756 flags ri idle 0:00:00 timeout 0:00:30 TCP PAT from inside:192.168.103.220/57761 to outside:10.2.1.2/54464 flags ri idle 0:00:00 timeout 0:00:30 ASA-5585# show nat pool TCP PAT pool outside, address 10.2.1.2, range 1-511, allocated 1 TCP PAT pool outside, address 10.2.1.2, range 512-1023, allocated 0 TCP PAT pool outside, address 10.2.1.2, range 1024-65535, allocated 2321 ASA-5585# Added in version 8.3BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
  52. 52. show nat detail The show nat command displays information about the nat table of the firewall The detail keyword will display object definitions ASA-5585# show nat detail Manual NAT Policies (Section 1) 1 (inside) to (outside) source static science-obj science-obj destination static vpn-obj vpn-obj translate_hits = 0, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 192.168.0.0/16 Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24 Auto NAT Policies (Section 2) 1 (dmz) to (outside) source static webserver-obj 14.36.103.83 translate_hits = 0, untranslate_hits = 3232 Source - Origin: 192.168.22.32/32, Translated: 14.36.103.83/32 2 (inside) to (outside) source dynamic science-obj interface translate_hits = 37723, untranslate_hits = 0 Source - Origin: 192.168.0.0/16, Translated: 14.36.103.96/16 ASA-5585/admin#BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
  53. 53. Show Conn and Show Conn Detail Real Interface Names Added in Idle Time, Connection 7.2(4), 8.0(4) Bytes Transferred Flags ASA# show conn 2 in use, 64511 most used TCP outside 198.133.219.25:80 dmz 10.9.9.3:4101, idle 0:00:06, Bytes 127, flags UIO UDP outside 172.18.124.1:123 dmz 10.1.1.9:123 idle 0:00:13 flags – detail Adds Uptime ASA# show conn detail and Timeout in 7.2(4), 2 in use, 64511 most used 8.0(4) Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN, B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump, E - outside back connection, F - outside FIN, f - inside FIN, G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data, i - incomplete, J - GTP, j - GTP data, K - GTP t3-response k - Skinny media, M - SMTP data, m - SIP media, n - GUP O - outbound data, P - inside back connection, q - SQL*Net data, R - outside acknowledged FIN, R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN, s - awaiting outside SYN, T - SIP, t - SIP transient, U - up, W - WAAS, X - inspected by service module TCP outside:198.133.219.25/80 dmz:10.9.9.3/4101, flags UIO, idle 8s, uptime 10s, timeout 1h, bytes 127 UDP outside:172.18.124.1/123 dmz:10.1.1.9/123, flags -, idle 15s, uptime 16s, timeout 2m, bytes 1431BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
  54. 54. Example—Connection Build Up  Firewall receives an initial SYN packet from the inside; the SYN is permitted by the access-list, a translation (xlate) is built up, and the connection is also created with the flags saA  The outside device responds to the SYN packet with a SYN+ACK; the connection flags are updated to reflect this, and now show A  The inside device responds to the SYN+ACK with an ACK and this completes the TCP three-way handshake, and the connection is now considered up (U flag)  The outside device sends the first data packet; the connection is updated and an I is added to the flags to indicate the firewall received Inbound data on that connection  Finally, the inside device has sent a data packet and the connection is updated to include the O flag Connection Flags SYN+ACK SYN Data ACK 1 5 3 42 UI UIO s A UaA Inside Outside Client ServerBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
  55. 55. Example—Connection Teardown  Firewall receives a FIN packet from the inside; as the FIN passes through the firewall, it updates the connection flags by adding an f to indicate that the FIN was received on the Inside interface  The outside device immediately responds to the FIN packet with a FIN+ACK; the connection flags are updated to reflect this, and now show UfFR  The inside device responds to the FIN+ACK with a final ACK and the firewall tears down the connection; thus, there are no more connection flags, because the connection no longer exists Connection Flags FIN+ACK ACK FIN 3 1 2 UfFR Uf r UfFR Inside Outside Client ServerBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
  56. 56. Connection Flags—Quick For your reference Reference Outbound Connection Inbound ConnectionBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
  57. 57. TCP Connection Termination Reasons  If a TCP connection is built through the firewall, it will always have a teardown reason  The TCP teardown syslog is logged at level six  If you are having problems with connections abnormally What does the Reset-O Termination reason mean in the closing, temporally increase your logging level (or move Teardown TCP connection syslog? the syslog down), and check the teardown reason ASA-6-302014: Teardown TCP connection number for intf_name:real_IP/real_port to intf_name:real_IP/real_port duration time bytes number [reason] [(user)]BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
  58. 58. TCP Connection Termination Reasons—Quick Reference For your reference Reason Description Connection Ended Because It Was Idle Longer Than the Conn-Timeout Configured Idle Timeout Deny Terminate Flow Was Terminated by Application Inspection The Standby Unit in a Failover Pair Deleted a Connection Failover Primary Closed Because of a Message Received from the Active Unit Force Termination After Ten Minutes Awaiting the Last ACK or FIN Timeout After Half-Closed Timeout Flow Closed by Inspection Flow Was Terminated by Inspection Feature Flow Terminated by IPS Flow Was Terminated by IPS Flow Reset by IPS Flow Was Reset by IPS Flow Terminated by Flow Was Terminated by TCP Intercept TCP Intercept Invalid SYN SYN Packet Not Valid Connection Timed Out Because It Was Idle Longer than the Idle Timeout Timeout Value IPS Fail-Close Flow Was Terminated Due to IPS Card Down SYN Control Back Channel Initiation from Wrong Side BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
  59. 59. TCP Connection Termination Reasons—Quick Reference(Cont.) For your Reason Description reference Force Termination After Two Minutes Awaiting SYN Timeout Three-Way Handshake Completion TCP Bad Retransmission Connection Terminated Because of Bad TCP Retransmission TCP Fins Normal Close Down Sequence TCP Invalid SYN Invalid TCP SYN Packet TCP Reset-I TCP Reset Was Sent From the Inside Host TCP Reset-O TCP Reset Was Sent From the Outside Host TCP Segment Partial Overlap Detected a Partially Overlapping Segment TCP Unexpected Window Connection Terminated Due to a Variation in the Size Variation TCP Window Size Tunnel Has Been Torn Down Flow Terminated Because Tunnel Is Down Unauth Deny Connection Denied by URL Filtering Server Unknown Catch-All Error Xlate Clear User Executed the ‗Clear Xlate‘ Command BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
  60. 60. show local-host command A local-host entry is created for any IP tracked through the firewall It groups the xlates, connections, and AAA information Very useful for seeing the connections terminating on servers ASA# show local-host detail connection tcp 50 Interface dmz: 0 active, 0 maximum active, 0 denied Interface inside: 1 active, 1 maximum active, 0 denied local host: <192.168.103.220>, TCP flow count/limit = 798/unlimited TCP embryonic count to host = 0 TCP intercept watermark = unlimited UDP flow count/limit = 0/unlimited Conn: TCP outside:172.18.124.76/80 inside:192.168.103.220/34078, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 TCP outside:172.18.124.76/80 inside:192.168.103.220/34077, flags UO, idle 0s, uptime 0s, timeout 30s, bytes 0 (output truncated)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
  61. 61. show service-policy The show service-policy command is used to quickly see what inspection policies are applied and the packets matching them ASA-5585/admin# show service-policy Global policy: Service-policy: global_policy Class-map: inspection_default Inspect: ftp, packet 0, lock fail 0, drop 0, reset-drop 0 … Inspect: sqlnet, packet 0, lock fail 0, drop 0, reset-drop 0 Inspect: http, packet 1215927, lock fail 0, drop 0, reset-drop 0 Inspect: icmp, packet 57, lock fail 0, drop 0, reset-drop 0 ASA-5585/admin# …BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
  62. 62. show service-policy flow Use to determine what policies a given flow will match in the Modular Policy Framework (MPF) ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521 Global policy: Service-policy: global_policy Interface outside: Service-policy: outside Class-map: oracle-dcd Match: access-list oracle-traffic Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet Action: Input flow: set connection timeout dcdBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
  63. 63. show asp drop Packets dropped in the Accelerated Security Path (ASP) will increment a counter FWSM – applies only to traffic sent to the control-point Frame drop counters are per packet, flow drops are per flow Some counters have corresponding syslogs ASA# show asp drop Frame drop: Invalid encapsulation (invalid-encap) 10897 Invalid tcp length (invalid-tcp-hdr-length) 9382 Invalid udp length (invalid-udp-length) 10 No valid adjacency (no-adjacency) 5594 No route to host (no-route) 1009 Reverse-path verify failed (rpf-violated) 15 Flow is denied by access rule (acl-drop) 25247101 First TCP packet not SYN (tcp-not-syn) 36888 Bad TCP flags (bad-tcp-flags) 67148 TCP option list invalid (tcp-bad-option-list) 731 TCP MSS was too large (tcp-mss-exceeded) 10942 Bad TCP Checksum (bad-tcp-cksum) 893 *Drop Counters Are Documented in the CMD Ref, Under show asp dropBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
  64. 64. Packet Capture  Capture command first introduced in Cisco 7.0; FWSM need to use 3.1.5 or later  ASA 7.2(3) and 8.0(3) added a real-time option  ASDM 6.0 adds a capture wizard  Capture sniffs packets on an interface that match an ACL, or match line  Key steps ‒ Use the ‗match‘ keyword to specify what traffic to capture (implicitly bi-directional) ‒ Define the capture and bind it to an access-list and interface ‒ View the capture on the firewall, or copy it off in .pcap format ASA# capture outcap interface outside match ip any host 10.1.2.3 ASA# capture incap interface inside match ip any host 10.1.2.3 ASA# ASA# show cap outcap 2 packets captured 1: 16:04:28.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request 2: 16:04:29.023741 802.1Q vlan#10 P0 172.18.254.139 > 10.1.2.3: icmp: echo request BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
  65. 65. Packet Capture (Cont.) See Appendix  Traffic can be captured both before and after it passes through the firewall; one capture on the inside interface, one capture on the outside interface  Capture buffer saved in RAM (default size 512 KB)  Default is to stop capturing when buffer is full  Default packet length is 1518 bytes  Copy captures off via TFTP or HTTPS Inside Capture Outside Capture Inside Outside Capture In Capture OutBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
  66. 66. Where Packets Are Captured in Packet Flow CSC IPS Modul Module e Yes No Yes Inspections Yes L3 Yes L2 YesRX Ingress Existing NAT ACL NAT IP Egress TX Permit Sec Route AddrPkt Interface Conn Untranslate Header Interface Pkt Checks No No No No No DROP DROP DROP DROP DROP Ingress Packets Egress Packets Captured Captured  Packets are captured at the first and last points they can be in the flow  Ingress packets are captured before any packet processing has been done on them  Egress packets are captured after all processing (including L2 source MAC rewrite) BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
  67. 67. Capturing Packets Dropped by the ASP ASA Only  Capture all packets dropped by the ASP ASA# capture drops type asp-drop all  Capture on a specific drop reason ASA# capture drops type asp-drop tcp-not-syn  Applies to both ASA and FWSM ASA# capture drop type asp-drop ? acl-drop Flow is denied by configured rule all All packet drop reasons bad-crypto Bad crypto return in packet bad-ipsec-natt Bad IPSEC NATT packet bad-ipsec-prot IPSEC not AH or ESP bad-ipsec-udp Bad IPSEC UDP packet bad-tcp-cksum Bad TCP checksum bad-tcp-flags Bad TCP flagsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
  68. 68. Packet Tracer - Overview ASA Only Introduced in ASA version 7.2 A packet tagged with the trace option is injected into the interface, and processed in the data-plane Each action taken on the packet is recorded in the packet itself When the packet reaches the egress interface, or is dropped, it is punted to the control-plane The control-plane reads and displays the actions taken on the packet, along with the associated lines in the configurationBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
  69. 69. Packet Tracer: ASDM (Located off Tools Menu) Define Packet Action Matching Link Back Config to Edit Rule Final ResultBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
  70. 70. Packet Tracer: Example Output ASA# packet-tracer input outside tcp 172.18.124.66 1234 172.18.254.139 3389 Phase: 1 Type: CAPTURE Subtype: Result: ALLOW Config: Additional Information: MAC Access list Phase: 2 Type: ACCESS-LIST Subtype: Result: ALLOW Config: Implicit Rule Additional Information: MAC Access list Phase: 3 Type: UN-NAT Subtype: static Result: ALLOW Config: nat (outside,dmz) source dynamic any interface destination static interface Win7-vm service rdp-outside rdp-outside Additional Information: NAT divert to egress interface dmz Untranslate 172.18.254.139/3389 to 192.168.103.221/3389 …….BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
  71. 71. Packet Tracer: Example Output Phase: 4 Type: ACCESS-LIST Subtype: log Result: ALLOW Config: access-group outside_in in interface outside access-list outside_in extended permit tcp any any eq 3389 Additional Information: …… Phase: 8 Type: NAT Subtype: Result: ALLOW Config: nat (outside,dmz) source dynamic any interface destination static interface jajohnstWin7-vm service rdp-outside rdp- outside Additional Information: Dynamic translate 172.18.124.66/1234 to 192.168.103.221/1234 …… Phase: 12 Type: FLOW-CREATION Subtype: Result: ALLOW Config: Additional Information: New flow created with id 16538274, packet dispatched to next moduleBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
  72. 72. Packet Tracer: Tracing Captured Packet Important!  Create a capture using the trace option ASA# capture inside access-list web interface inside trace  Find the packet in the capture you want traced ASA# show capture inside 68 packets captured 1: 15:22:47.581116 10.1.1.2.31746 > 198.133.219.25.80: S 2: 15:22:47.583465 198.133.219.25.80 > 10.1.1.2.31746: S ack 3: 15:22:47.585052 10.1.1.2.31746 > 198.133.219.25.80: . ack 4: 15:22:49.223728 10.1.1.2.31746 > 198.133.219.25.80: P ack 5: 15:22:49.223758 198.133.219.25.80 > 10.1.1.2.31746: . Ack ...  Then select that packet to be traced ASA# show capture inside trace packet-number 4 .BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
  73. 73. TCP Ping  New troubleshooting tool added in ASA ver 8.4.1  Why is it needed??? Consider the following… www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
  74. 74. TCP Ping  Previously – limited reachability tools: Ping and Traceroute  Access to client machine? Attempts to about the path What validate NAT and/or PAT? …but with ICMP ICMP Echo Request ICMP Echo Request ICMP Echo Reply ICMP Echo Reply www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
  75. 75. TCP Ping  Sources TCP SYN packet with Client’s IP and injects it into Client’s interface of the ASA Packet with SRC of 10.1.1.7 injected TCP SYN sent on Inside interface to server Internal hosts are PATed to 198.51.100.2 inside outside www server 10.1.1.7 (209.165.200.225) TCP SYN+ACK ASA Datapath Packet PATed to sent from server Validated 198.51.100.2 (NAT, ACLs, etc) on EgressBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
  76. 76. TCP Ping – The Big Picture  Validates 2 of the 3 legs of the connection from client to server TCP path from client side of ASA to Server through the cloud -Validated- inside outside www server 10.1.1.7 (209.165.200.225) 1st Leg 2nd Leg 3rd LegBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
  77. 77. TCP Ping - Example Specify Client‘s asa# ping tcp Interface: inside source Interface Target IP address: 209.165.200.225 Target IP port: 80 Specify Client‘s Specify source? [n]: y Source IP address: 10.1.1.7 real IP Address Source IP port: [0] Repeat count: [5] Timeout in seconds: [2] Type escape sequence to abort. Sending 5 TCP SYN requests to 209.165.200.225 port 80 from 10.1.1.7 starting port 3465, timeout is 5 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms inside outside www server 10.1.1.7 (209.165.200.225)BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
  78. 78. Agenda Packet Flow Understanding the Architecture Failover Troubleshooting Case Studies Online Resources Best PracticesBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
  79. 79. Case StudyLeveraging Smart Call Home
  80. 80. Case Study: Smart Call Home ASA Only Email CMD Output to You  Objective – Send the output of a command directly to your e-mail.  This is easily accomplished with SCH. Use the command: call-home send <―cmd‖> email <email_addr> Example: call-home send ―show run‖ email userid@cisco.com  This will send a plain-text e-mail with the output of the command to the e- mail address specified, with the command in the subject line. ‒ Example: Subject: CLI ‗show run‘ outputBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
  81. 81. Case Study: Smart Call Home Collecting Memory Diagnostics over Time  Objective – Memory appears to be depleting over time on your ASA. Use SCH to collect the detailed memory output hourly, for further investigation.  This is easily accomplished with SCH. Setting a ‖snapshot‖ alert- group to e-mail commands at a specified interval  Snapshot will contain the following command: show conn count show memory detailBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
  82. 82. Case Study: Smart Call Home Example Config service call-home call-home alert-group-config snapshot add-command ―show conn count‖ add-command "show memory detail― contact-email-addr user@cisco.com sender from user@cisco.com sender reply-to user@cisco.com mail-server smtp-server.cisco.com priority 1 profile SENDCMD active destination address email user@cisco.com destination preferred-msg-format long-text destination transport-method email subscribe-to-alert-group snapshot periodic hourlyBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
  83. 83. Case StudyIntermittent Access to Web Server
  84. 84. Case Study: Intermittent Access to Web Server Problem  Most external clients are not able to load company‘s web page NATed to 10.1.1.50 HTTP Requests to 192.168.1.50 Internet Web Server ASA-5510 10.1.1.50 ClientsBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
  85. 85. Case Study: Intermittent Access to Web Server Traffic SpikeBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
  86. 86. Case Study: Intermittent Access to Web Server  show perfmon indicates high number of embryonic connections ASA-5510# show perfmon PERFMON STATS: Current Average Xlates 0/s 0/s Connections 2059/s 299/s TCP Conns 2059/s 299/s UDP Conns 0/s 0/s URL Access 0/s 0/s URL Server Req 0/s 0/s TCP Fixup 0/s 0/s TCP Intercept Established Conns 0/s 0/s TCP Intercept Attempts 0/s 0/s TCP Embryonic Conns Timeout 1092/s 4/s HTTP Fixup 0/s 0/s FTP Fixup 0/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s VALID CONNS RATE in TCP INTERCEPT: Current Average N/A 95.00%BRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
  87. 87. Case Study: Intermittent Access to Web Server  Issue show conn to see ‗who‘ is creating the connections Random Sources Embryonic Conns ASA-5510# show conn 54764 in use, 54764 most used TCP outside 17.24.101.118:26093 inside 10.1.1.50:80, idle 0:00:23, bytes 0, flags aB TCP outside 111.76.36.109:23598 inside 10.1.1.50:80, idle 0:00:13, bytes 0, flags aB TCP outside 24.185.110.202:32729 inside 10.1.1.50:80, idle 0:00:25, bytes 0, flags aB TCP outside 130.203.2.204:56481 inside 10.1.1.50:80, idle 0:00:29, bytes 0, flags aB TCP outside 39.142.106.205:18073 inside 10.1.1.50:80, idle 0:00:02, bytes 0, flags aB TCP outside 75.27.223.63:51503 inside 10.1.1.50:80, idle 0:00:03, bytes 0, flags aB TCP outside 121.226.213.239:18315 inside 10.1.1.50:80, idle 0:00:04, bytes 0, flags aB TCP outside 66.187.75.192:23112 inside 10.1.1.50:80, idle 0:00:06, bytes 0, flags aBBRKSEC-3020 © 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

×