Troubleshooting WirelessLANs with CentralizedControllersBRKEWN-3011Wesley Terry      BRKEWN-3011   © 2011 Cisco and/or its...
Troubleshooting Wireless LANs Supportability Software and Support Model Troubleshooting Basics The Client Debug WLC C...
Supportability BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   3
Supportability WLC Supportability        Methods of Management        Using the GUI        Important Show Commands (CLI) ...
WLC SupportabilityMethods of Management                                                                                   ...
WLC SupportabilityUsing the GUI Monitor        AP/Radio Statistics        WLC Statistics        Client Details        Tra...
WLC SupportabilityUsing the GUI Wireless > All APs        AP list shows AP Physical UP Time        APs are sorted by Cont...
WLC SupportabilityUsing the GUI Management        SNMP Config        Logs        Tech SupportBRKEWN-3011    © 2011 Cisco ...
WLC SupportabilityImportant Show Commands (CLI) Show run-config        Must have! No exceptions!        “show run-config ...
WLC SupportabilityImportant Debugs (CLI) Debug client <client mac address>        Client Involved? Must Have! No Exceptio...
WLC SupportabilityBest Practices Change default SNMP Parameters Configure Syslog for WLC and AP Enable Coredump for WLC...
AP Supportability Methods of Accessing the AP                                                                            ...
AP SupportabilityAP Remote Commands (WLC CLI) Debug AP enable <AP name>        Enables AP Remote Debug        AP Must be ...
AP SupportabilityShow Commands (AP CLI or WLC Remote Cmd) Show controller Do[0/1] (or Show Tech)        Must have! Before...
Software andSupport ModelBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   15
Software and Support Model Opening a TAC Service Request Cisco Support Model        TAC vs Business Unit        What to ...
Opening a TAC Service Request What should I have ready?        Clear problem description        Always: Show run-config  ...
Opening a TAC Service Request Potential reasons to slow a TAC SR‟s resolution        Information about the problem is mis...
Cisco Support Model – TAC vs. BU TAC        Customer advocate        Technology focused with cross technology collaborati...
Cisco Support Model – Expectations What not to expect from TAC        Design and deployment        Complete configuration...
Cisco Support Model - Escalation TAC Escalation Process        Multi-Tier support resources within a technology        TA...
WLC Software Trains CCO - Cisco.com release        6.0.202.0, 7.0.116.0, etc…        Full test cycle        Classified as...
WLC Software Trains - ES vs. Escalation Engineering Special        Development “special” image for fix validation or limi...
Troubleshooting BasicsBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   24
The 10-Point Capture                                                                           EAP                        ...
Troubleshooting Basics Troubleshooting 101                                                                     Problem   ...
Troubleshooting 101 Troubleshooting is an art with no right or wrong  procedure, but best with a logical methodology. St...
Troubleshooting 101 Step 2: Understand any possible triggers        If something previously worked but no longer works, t...
Troubleshooting 101 Step 4: Reproducibility        Any problem that has a known procedure to reproduce (or        frequen...
Recommended Tools Wireless Sniffer        Example: Linksys USB600N with Omnipeek              TAC can publish Omnipeek-RA...
The Client DebugBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   31
Steps to Building an 802.11 Connection    802.11                              1. Listen for Beacons    State 1:Unauthentic...
The Client Debugdebug client <mac address> A multi-debug macro        (Cisco Controller) >debug client 00:16:EA:B2:04:36 ...
Understanding the Client StateName                                                                                 Descrip...
The Client Debug - Walkthrough Association (Start) L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD) ...
Client Debug - Association BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   36
Association(Cisco Controller) >debug client 00:16:EA:B2:04:36(Cisco Controller) >(Cisco Controller) >Association received ...
AssociationAssociation received from mobile on AP 00:26:cb:94:44:c00.0.0.0 START (0) Changing ACL none (ACL ID 0) ===> non...
AssociationSTA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0Processing RSN IE type 48, length 22 for mobi...
Association0.0.0.0 START (0) Initializing policy0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)0....
AssociationSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0        Slot 0 = B/G(2...
Association - FSRProcessing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36CCKM: Mobile is using CCKMCCKM: Process...
Association - Takeaway Association vs. Reassociation Debug shows        AP, Slot, AP-Group, WLAN ID, Interface, Data Rat...
Client Debug –L2 Authentication BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   44
802.1X AuthenticationSupplicant                                                                               Authenticato...
WPA2-AES-802.1XSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Station 00:16:ea:b2...
Common EAP Types 1 – Identity 2 – Notification 3 – NAK                  Sending EAP Request from AAA to mobile 00:16:ea...
802.1X (Cont.) (WPA2-AES-PSK)Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Creat...
WPA2-AES-PSK - FailedStarting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be droppedSending EAPOL-Key Mess...
L2 Authentication - Takeaway  8021X_REQD means L2 Authentication pending          Authentication/Encryption has not be es...
Client Debug –IP Learning State BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   51
Client DHCP00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state00:16:ea:b2:04:36 apfMs1xStateInc00:16:ea:b2:04...
Client DHCP                                                                                               Client State = ...
DHCP Proxy Enabled – DHCP Discover*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x032.151...
DHCP Proxy Enabled – DHCP Offer34.166: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0x...
DHCP Proxy Enabled – DHCP Request38.169: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, enca...
DHCP Proxy Enabled – DHCP Ack38.172: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec...
DHCP Proxy Disabled – Discover/Offer*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*00:...
DHCP Proxy Disabled – Request/Ack*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec0...
Learning IP without DHCP*Orphan Packet from 10.99.76.147 on mobile*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule ...
Client DHCP - Takeway DHCP_REQD means Learning IP State        Only “Required” if enabled on WLC If Proxy is enabled    ...
Client Debug –L3 Authentication BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   62
Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)*pemReceiveTa...
Webauth Redirect                                                                                  Webauth                 ...
ARP and DNS FunctionConfirm ARP and DNS FunctionBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   C...
3-Way Handshake                                                                                                           ...
Webauth - Takeaway If WEBAUTH_REQD, then not authenticated      Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv...
Client Debug - RunBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   68
Run State10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)10.10.3.82 RUN (20) Reached PLUMBFASTPATH: f...
Client Debug –Deauth/DisassocBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   70
Deauthenticated Client Idle Timeout       Occurs after no traffic received from Client       Default Duration is 300 seco...
Deauthenticated Client WLAN Change       Modifying a WLAN in anyway Disables and Renables WLANapfSendDisAssocMsgDebug (ap...
Deauthenticated Client Authentication Timeout     Auth or Key Exchange max-retransmissions reachedRetransmit failure for ...
Deauthentication - Takeaway Client can be removed for numerous reasons        WLAN change, AP change, configured interval...
Client Debug – Tips and Tricks BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   75
Tips and Tricks Collect a client debug for an extended duration        Several roams, deauths, failures, etc… Use an enh...
Tips and TricksBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   77
Tips and TricksBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   78
Client Debug – SummaryBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   79
Client Connectivity Unified Wireless Network: Troubleshoot Client  Issues Document ID: 107585 Configuration Issues      ...
802.11n Speeds Troubleshoot 802.11n Speeds Document ID: 112055 Configuration Issues        11n Support Enabled        WM...
802.11n A-MPDU/A-MSDU Aggregation methods used could impact interop or  performance                                      ...
WLC Config Analyzer (WLCCA)BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   83
What Is the WLCCA? It is a Post Sales tool Main objective: Save time while analyzing configuration files  from WLCs Sec...
Where? Support Forums DOC-1373 BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   85
Input Needed Complete config output from WLC     Show run-config It does not work with old “show running-config” or with...
Functionality Overview - Checks Audit Checks     More than 100 config detail verifications     Based on TAC/Escalation ca...
Functionality Overview Audit Checks BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public...
Functionality Overview Config View BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public ...
WLCCA – High RF Index APsBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   90
Reducing CCI Turn off excess 2.4 radios. May want to do this  gradually, e.g. turn off 20% of radios per attempt After t...
2.4GHz – Target Coverage Most all 2.4GHz radios are at power 2 - 5 (dont want 7  or 8) In all locations, you have covera...
5 GHz – Target Coverage  Most all 5GHz radios are at power 1 – 3 (at   least 14dBm)      Consider the RRM min power setti...
Additional TroubleshootingBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   94
Additional Troubleshooting Wireshark Tutorial Clean Air SE-Connect / AP Sniffer Mode AP Join RRM Multicast/Broadcast...
Wireshark Tutorial BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   96
Wireshark Tutorial Default Wireshark view might look like this:BRKEWN-3011   © 2011 Cisco and/or its affiliates. All righ...
Wireshark Tutorial Newer versions of Wireshark have a feature for  “Apply as Column”      This will take any decodable pa...
Wireshark Tutorial Within seconds your wireshark can also have:BRKEWN-3011   © 2011 Cisco and/or its affiliates. All righ...
Wireshark Tutorial Filtering data is just as easyBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.  ...
Wireshark Tutorial - CAPWAP User data is encapsulated in CAPWAPBRKEWN-3011   © 2011 Cisco and/or its affiliates. All righ...
Wireshark Tutorial Wireshark can also de-encapsulate CAPWAP DATA       Edit > Preference > Protocols > CAPWAPBRKEWN-3011 ...
Wireshark Tutorial With CAPWAP de-encapsulated you can see all the  packets to/from client (between AP and WLC)BRKEWN-301...
SE-Connect – Clean AirAP Sniffer ModeBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public...
SE-Connect and Sniffer Mode Clean Air APs can be used in lieu of Spectrum Card  for Spectrum Analysis        AP can be pl...
Spectrum Expert with Clean Air Obtain Spectrum Key Connect to Remote Sensor  BRKEWN-3011   © 2011 Cisco and/or its affil...
Spectrum Expert with Clean AirBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   107
Sniffer Mode AP Select channel to Sniff Select destination for traffic    BRKEWN-3011   © 2011 Cisco and/or its affiliat...
Sniffer Mode AP Omnipeek has a Remote Adapter to capture this data Wireshark, just capture network adapter   NOTE: Wires...
Sniffer Mode AP With wireshark, filter !icmp.type == 3 Data (UDP 5000) still not intelligible yet   Decode as Airopeek  ...
Sniffer Mode APBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   111
AP Discover/JoinBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   112
AP Discover/Join                                                                                         AP Runs Hunting  ...
AP - Discover Process AP Discovery Req to known and learned WLCs Broadcast       Reaches WLCs with MGMT Interface in loc...
AP - Discover Process                       broadcast                                                                     ...
AP – WLC Selection/Join WLCs send Discovery Response back to AP    Name, Capacity, AP Count, Master?, AP-MGR, Load per AP...
Troubleshooting AP Discovery/Join “Lightweight AP (LAP) Registration to a Wireless LAN  Controller (WLC)”, Document ID 70...
RRMBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   118
RRM There are usually only two common scenarios or  issues involving RRM APs not changing channel        Check if other ...
RRM Debugs WLC – debug airewave-director <?> AP       debug capwap rm mesurements       debug capwap rm rogueBRKEWN-3011...
RRM Show AP Auto-RF (In Run-Config) show ap auto-rf [802.11a/b] <AP Name> Load Information        Receive Utilization.. ...
Broadcast/MulticastBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   122
Broadcast/MulticastBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   123
Broadcast/Multicast AP Multicast Mode – Multicast     Address must be unique among WLCs Broadcast Traffic is delivered v...
Broadcast/Multicast AP Show Commands     Show capwap mcast     Show capwap mcast mgid all BRKEWN-3011   © 2011 Cisco and/...
Client Mobility BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   126
Mobility—Intra-Controller Client roams between two APs on the same controller BRKEWN-3011   © 2011 Cisco and/or its affil...
Mobility—Inter-Controller (Layer 2)BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public  ...
Mobility—Layer 3 Layer 3 roaming (a.k.a. anchor/foreign)        New WLC does not have an interface on the subnet the clie...
Mobility—Messaging Flow When a client connects to a WLC for the first time,  the following happens:        New WLC sends ...
Debug Client <Mac Address>Mobility— L2 Inter WLC                                                                  Debug Mo...
Debug Client <Mac Address>Mobility— L3 Inter WLC                                                                  Debug Mo...
Debug Client <Mac Address>Mobility— L3 Inter WLC                                                                  Debug Mo...
Mobility— L3 Handoff Ignored*mmListen: Mobility packet received from:*mmListen: 10.4.22.55, port 16666*mmListen: type: 3(M...
Mobility Group vs. Mobility Domain Mobility Group - WLCs with the same group name      L2/L3 Handoff      Auto Anchoring ...
Mobility Data/Control Path Sent between all WLCs, by member with lowest MAC      Control Path = UDP 16666 (30 Seconds)   ...
Voice over WiFi BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   137
VoWiFi Wireless IP Phone Deployment Guide        http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/79        25g/7_0/e...
VoWiFi - Troubleshooting Must know if problem occurs during roaming events  or when no association change takes place If...
VoWiFi - Troubleshooting Define a reproducible area where you believe you  have perfect voice coverage but have problems...
VoWiFi - Debugs Phone can Trace (debug) to file or syslog       Recommend USB Connection and SYSLOG       Configured via ...
SummaryBRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   142
SummaryClient       WLC - show run-config, debug client <mac>, debug dhcp message enable,             debug dot1x <?> enab...
Summary Links:        Understanding Debug Client on Wireless LAN Controllers        (WLCs) Document ID: 100260        Uni...
Complete Your OnlineSession Evaluation Receive 25 Cisco Preferred Access points for each session  evaluation you complete...
Visit the Cisco Store for               Related Titles        http://theciscostores.comBRKEWN-3011   © 2011 Cisco and/or i...
BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   147
Thank you.BRKEWN-3011   © 2011 Cisco and/or its affiliates. All rights reserved.   Cisco Public   148
Upcoming SlideShare
Loading in...5
×

Troubleshooting Wireless LANs with Centralized Controllers

42,965

Published on

Best practices for troubleshooting your wireless LAN issues prior and during TAC engagement. Learn More: http://www.cisco.com/go/wireless

Published in: Technology, Education
2 Comments
13 Likes
Statistics
Notes
No Downloads
Views
Total Views
42,965
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
1,534
Comments
2
Likes
13
Embeds 0
No embeds

No notes for slide

Troubleshooting Wireless LANs with Centralized Controllers

  1. 1. Troubleshooting WirelessLANs with CentralizedControllersBRKEWN-3011Wesley Terry BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 1
  2. 2. Troubleshooting Wireless LANs Supportability Software and Support Model Troubleshooting Basics The Client Debug WLC Config Analyzer (WLCCA) Additional TroubleshootingBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
  3. 3. Supportability BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
  4. 4. Supportability WLC Supportability Methods of Management Using the GUI Important Show Commands (CLI) Important Debugs (CLI) Best Practices AP Supportability Methods of Accessing the AP Important Show CommandsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
  5. 5. WLC SupportabilityMethods of Management Default Mode GUI (E)=Enabled (D)=Disabled HTTPS (E) / HTTP (D) CLI Console SSH (E) / Telnet (D) SNMP V1 (D) / V2 (E) – Change me! V3 (E) – Change meNote: Management Via Wireless Clients (D)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
  6. 6. WLC SupportabilityUsing the GUI Monitor AP/Radio Statistics WLC Statistics Client Details Trap LogBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
  7. 7. WLC SupportabilityUsing the GUI Wireless > All APs AP list shows AP Physical UP Time APs are sorted by Controller Associated Time Check bottom of AP list for any recent AP disruptions Select AP to see Controller Associated Time (duration)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
  8. 8. WLC SupportabilityUsing the GUI Management SNMP Config Logs Tech SupportBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
  9. 9. WLC SupportabilityImportant Show Commands (CLI) Show run-config Must have! No exceptions! “show run-config commands” (like IOS show running-config) “show run-config no-ap” (no AP information added) Show tech-support CLI Tip Log all output Config Paging DisableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
  10. 10. WLC SupportabilityImportant Debugs (CLI) Debug client <client mac address> Client Involved? Must Have! No Exceptions Debug capwap <event/error/detail/info> enable CLI Tips Log all output Debugs are session based, they end when session ends “Config session timeout 60”, sets 60 minute idle timeout Debug mac addr <mac address> Used to filter debugs on specific Mac Address Debug disable-all (Disables all debugs)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
  11. 11. WLC SupportabilityBest Practices Change default SNMP Parameters Configure Syslog for WLC and AP Enable Coredump for WLC and AP Configure NTP Server for Date/TimeBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
  12. 12. AP Supportability Methods of Accessing the AP Default Mode Console (E)=Enabled (D)=Disabled Telnet (D) / SSH (D) No GUI support AP Remote Commands Enabling Telnet/SSH WLC CLI: config ap [telnet/ssh] enable <ap name> WLC GUI: Wireless > All APs > Select AP > Advanced Select [telnet/ssh] > ApplyBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
  13. 13. AP SupportabilityAP Remote Commands (WLC CLI) Debug AP enable <AP name> Enables AP Remote Debug AP Must be associated to WLC Redirects AP Console output to WLC session Debug AP command “<command>” <AP name> Output is redirected to WLC session AP runs IOS, numerous generic IOS commands availableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
  14. 14. AP SupportabilityShow Commands (AP CLI or WLC Remote Cmd) Show controller Do[0/1] (or Show Tech) Must have! Before/During/After event Show log WLC: show ap eventlog <ap name> Show capwap client <?> CLI Tips Debug capwap console client Debug capwap client no-reloadBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
  15. 15. Software andSupport ModelBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
  16. 16. Software and Support Model Opening a TAC Service Request Cisco Support Model TAC vs Business Unit What to expect from TAC How does escalation work? WLC Software Trains CCO (ED/MD/AW) “Engineering Special” vs “Escalation”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
  17. 17. Opening a TAC Service Request What should I have ready? Clear problem description Always: Show run-config If client involved, always: “debug client <mac address>” Your analysis of any data provided Expectations for customer involvement TAC SR severity level descriptions state that You and Cisco will commit necessary resources according to severity You must set correct expectation of timeline and severityBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
  18. 18. Opening a TAC Service Request Potential reasons to slow a TAC SR‟s resolution Information about the problem is missing The severity level was not set appropriately Data, such as traces or logs, has not been forwarded to the engineer The scope or time requirements are not well understood by the engineer The problem cannot be reproduced in the Cisco Technical Assistance Center lab Access to the affected equipment for debugging purposes is not availableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
  19. 19. Cisco Support Model – TAC vs. BU TAC Customer advocate Technology focused with cross technology collaboration Escalation path within TAC exists Business Unit - Escalation Work in conjunction with TAC during specific engagements Product specific focus Engages development resources when necessaryBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
  20. 20. Cisco Support Model – Expectations What not to expect from TAC Design and deployment Complete configuration Sales related information What to expect from TAC Configuration assistance Problem analysis / bug isolation Workarounds or fixes Action plan to resolve SR Hardware replacement Engage BU when appropriateBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
  21. 21. Cisco Support Model - Escalation TAC Escalation Process Multi-Tier support resources within a technology TAC to engage resources (TAC/BU) when appropriate SR ownership might not change hands Customer Escalation Process Raise SR priority (S1/S2) Engage account team Your satisfaction is important to the Cisco TAC. If you have concerns about the progress of your case, please contact your regional TAC.BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
  22. 22. WLC Software Trains CCO - Cisco.com release 6.0.202.0, 7.0.116.0, etc… Full test cycle Classified as ED when posted AssureWave AW is no longer tagged on CCO, but AW validation results are available at: http://www.cisco.com/go/assurewave Results available 4 weeks after CCO MD MD tag represents stable releases for mass adoption MD tag will be considered on CCO after AW release validation, 10 weeks in field and TAC/Escalation signoffBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
  23. 23. WLC Software Trains - ES vs. Escalation Engineering Special Development “special” image for fix validation or limited use Sanity tested “As-is” Escalation Code Escalation is a post-CCO maintenance release with specific/minimal customer impacting SW fixes Fix must be fully committed to the next CCO MR Sanity + focus tested Fully TAC+BU supported “Running-Master” so each release builds upon the previousBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
  24. 24. Troubleshooting BasicsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
  25. 25. The 10-Point Capture EAP Radio chan. 1 Driver Supp. IP RADIUS ACS 802.11 Data WLC 802.11 Management CAPWAP EOIP IP IP DHCP 802.11 Management CAPWAP WLCSupplicant Wired Logs AP Sniff Wired Debugs Sniff Driver Debugs/ DHCP Wireless Spectrum WLC Logs ACS Adapter Sniff Analysis Debugs Logs Capture NTP BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
  26. 26. Troubleshooting Basics Troubleshooting 101 Problem Definition Clearly define the problem Understand any possible triggers Know the expected behavior Questions Reproducibility Recommended Tools Tests Spectrum Analyzer Wireless Sniffer and Wired Captures Analysis Solution(s)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
  27. 27. Troubleshooting 101 Troubleshooting is an art with no right or wrong procedure, but best with a logical methodology. Step 1: Define the problem It is crucial to understand all possible details of a problem Knowing what is and is not working will go a long way With a proper understanding of the problem description you can skip many steps Bad description: “Client slow to connect” Good description: “Client associations are rejected with Status17 several times before they associate successfully.”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
  28. 28. Troubleshooting 101 Step 2: Understand any possible triggers If something previously worked but no longer works, there should be an identifiable trigger Understanding any and all configuration or environmental changes could help pinpoint a trigger Step 3: Know the expected behavior If you know the order of expected behavior that is failing, defining where the behavior breaks down (Problem Description) is better than defining the end result. Example: “One way audio between Phone A and B, because Phone A does not get an ARP Response for Phone B”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
  29. 29. Troubleshooting 101 Step 4: Reproducibility Any problem that has a known procedure to reproduce (or frequently randomly occurs) should be easy to diagnose Being able to easily validate or disprove a potential solution saves time by being able to quickly move on to the next theory If a problem is reproducible in other environments with a known procedure, TAC/BU can facilitate internal testing and proposed fix/workaround verification Debugs and Captures of working scenarios can help pin point where exactly the difference isBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
  30. 30. Recommended Tools Wireless Sniffer Example: Linksys USB600N with Omnipeek TAC can publish Omnipeek-RA if you have compatible HW Wired Packet Capture Example: Wireshark Use for spanned switchports of AP/WLC or client side data Spectrum Analyzer Spectrum Expert with Card or Clean-Air APBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
  31. 31. The Client DebugBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
  32. 32. Steps to Building an 802.11 Connection 802.11 1. Listen for Beacons State 1:Unauthenticated, 2. Probe Request Unassociated 3. Probe Response AP 4. Authentication Request 5. Authentication Response State 2: Authenticated, 6. Association Request Unassociated 7. Association Response WLC State 3: 8. (Optional: EAPOL Authentication) Authenticated, Associated 9. (Optional: Encrypt Data) 10. Move User Data BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
  33. 33. The Client Debugdebug client <mac address> A multi-debug macro (Cisco Controller) >debug client 00:16:EA:B2:04:36 (Cisco Controller) >show debug MAC address ................................ 00:16:ea:b2:04:36 Debug Flags Enabled: dhcp packet enabled dot11 mobile enabled dot11 state enabled dot1x events enabled dot1x states enabled pem events enabled pem state enabled CCKM client debug enabledBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
  34. 34. Understanding the Client StateName Description8021X_REQD 802.1x (L2) Authentication PendingDHCP_REQD IP Learning StateWEBAUTH_REQD Web (L3) Authentication PendingRUN Client Traffic Forwarding (Cisco Controller) >show client detail 00:16:ea:b2:04:36 Client MAC Address............................... 00:16:ea:b2:04:36 ….. Policy Manager State............................. WEBAUTH_REQD 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
  35. 35. The Client Debug - Walkthrough Association (Start) L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD) L3 Authentication (WEBAUTH_REQD) Client Fully Connected (RUN) Deauth/Disassoc Tips and TricksBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
  36. 36. Client Debug - Association BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
  37. 37. Association(Cisco Controller) >debug client 00:16:EA:B2:04:36(Cisco Controller) >(Cisco Controller) >Association received from mobile on AP 00:26:cb:94:44:c00.0.0.0 START (0) Changing ACL none (ACL ID 0) ===> none (ACL ID 255) --- (caller apf_policy.c:1621)Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site default-group, interface 3Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface 3„STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:360.0.0.0 START (0) Initializing policy0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1apfMsAssoStateIncapfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to AssociatedScheduling deletion of Mobile Station: (callerId: 49) in 1800 secondsSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
  38. 38. AssociationAssociation received from mobile on AP 00:26:cb:94:44:c00.0.0.0 START (0) Changing ACL none (ACL ID 0) ===> none (ACL ID 255) --- (caller apf_policy.c:1621)Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site default-group, interface 3Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface 3  Association received Association Request, client did not “Roam” (Reassociate) AP Base Radio = 00:26:cb:94:44:c0  vapId 1, site default-group, interface 3„ vapId = WLAN # (Wlan 1) site = AP Group (default-group) Interface = Dynamic Interface name (3)  vlan 3 Vlan = Vlan # of Dynamic Interface BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
  39. 39. AssociationSTA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36  STA - rates Madatory Rates (>128) = (#-128)/2 Supported Rates (<128) = #/2 1m,2m,5.5m,11m,6s,9s,12s,18s,24s,36s,48s,54s  Processing RSN IE type 48 WPA2-AES Processing WPA IE type 221 = WPA-TKIP BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
  40. 40. Association0.0.0.0 START (0) Initializing policy0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1apfMsAssoStateIncapfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to AssociatedScheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds  0.0.0.0 START 0.0.0.0 = IP we know for client (In this case nothing)  Change state to 8021X_REQD Passed association, moving client to next state: 8021X_REQD  Scheduling deletion Session Time on WLAN (1800 seconds in this case) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
  41. 41. AssociationSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0  Slot 0 = B/G(2.4) Radio Slot 1 = A(5) Radio  Sending Assoc Response Status 0 = Success Anything other than Status 0 is Failure Common Assoc Response Failures: 1 – Unknown Reason – Anything not matching defined reason codes 12 – Unknown or Disabled SSID 17 – AP cannot handle any more associations 18 – Client is using a datarate that is not allowed 35 – WLAN requires the use of WMM and client does not support it 201 – Voice client attempting to connect to a non-platinum WLAN 202 – Not enough available bandwidth to handle a new voice call (CAC Rejection) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
  42. 42. Association - FSRProcessing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36CCKM: Mobile is using CCKMCCKM: Processing REASSOC REQ IEIncluding CCKM Response IE (length 62) in Assoc Resp to mobileSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot 1ORProcessing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36Received PMKID: (16) [0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8Found an entry in the global PMK cache for stationComputed a valid PMKID from global PMK cache for mobile FSR aIOS CUWN CCKM - WPA yes yes CCKM - WPA2 yes yes WPA2 PKC no yes WPA2 "Sticky" yes no* BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
  43. 43. Association - Takeaway Association vs. Reassociation Debug shows AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type Association Response Confirms if Client is associated Defines reason if denied Further troubleshooting May require Wireless Sniffer or capture at AP Switchport If not sending Assoc Request, must know why from Client Trying disabling WLAN features to “dumb it down”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
  44. 44. Client Debug –L2 Authentication BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
  45. 45. 802.1X AuthenticationSupplicant Authenticator Server EAPOL-START EAP-ID-Request EAP-ID-Response RADIUS (EAP-ID_Response) Rest of the EAP Conversation Radius-Access-Accept EAP-Success (Key) The Supplicant Derives the Session Key from User Password or Session Certificate and Authentication Key Exchange BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
  46. 46. WPA2-AES-802.1XSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Username entry (cisco) created for mobile Received Identity Response (count=1) from mobile 00:16:ea:b2:04:36 EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36 dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state………………….. Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36 Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)...........................Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 10, EAP Type 25) Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36 Processing Access-Challenge for mobile 00:16:ea:b2:04:36 Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36 Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 11) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 11, EAP Type 25) Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36 Processing Access-Accept for mobile 00:16:ea:b2:04:36***OR*** Processing Access-Reject for mobile 00:16:ea:b2:04:36BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
  47. 47. Common EAP Types 1 – Identity 2 – Notification 3 – NAK Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 4 – MD5 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25) 5 – OTP 6 – Generic Token 13 – EAP TLS 17 – LEAP 18 – EAP SIM 21 – EAP TTLS 25 – PEAP 43 – EAP-FAST BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
  48. 48. 802.1X (Cont.) (WPA2-AES-PSK)Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36New PMKID: (16) [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cdInitiating RSN PSK to mobile 00:16:ea:b2:04:36dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth stateSkipping EAP-Success to mobile 00:16:ea:b2:04:36Including PMKID in M1 (16) [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cdStarting key exchange to mobile 00:16:ea:b2:04:36, data packets will be droppedSending EAPOL-Key Message to mobile 00:16:ea:b2:04:36 state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00Received EAPOL-Key from mobile 00:16:ea:b2:04:36Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36Stopping retransmission timer for mobile 00:16:ea:b2:04:36Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36 state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01Received EAPOL-Key from mobile 00:16:ea:b2:04:36Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36apfMs1xStateInc0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
  49. 49. WPA2-AES-PSK - FailedStarting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be droppedSending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57 state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57…………………802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb deauth count 3Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
  50. 50. L2 Authentication - Takeaway  8021X_REQD means L2 Authentication pending Authentication/Encryption has not be established  PSK is 802.1X, key is derived from PSK not AAA  If “Processing Access-Reject” AAA/RADIUS Rejected the user (not the WLC)  If “Processing Access-Accept” AAA/Radius Accepted the user M1-M4 should follow  Further Troubleshooting Debug aaa [all/event/detail/packet] enable Debug dot1x [aaa/packet] enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
  51. 51. Client Debug –IP Learning State BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
  52. 52. Client DHCP00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state00:16:ea:b2:04:36 apfMs1xStateInc00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3for this client00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 3 apVapId 300:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0...................00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)...................00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)...................00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
  53. 53. Client DHCP Client State = Client is in DHCP_REQD state “DHCP_REQD“ Proxy Enabled: DHCP Proxy Enabled DHCP Proxy Disabled DHCP Relay/Proxy Between WLC and Server Client DHCP Discover Client DHCP Discover Is Unicast to DHCP Bridged to DS Required for Internal DHCP Servers Proxy Disabled: DHCP Offer from Server Between Client and Server DHCP is broadcast out VLAN Client DHCP Request IP helper or other means required DHCP ACK from Server IP Address Learned BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
  54. 54. DHCP Proxy Enabled – DHCP Discover*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x032.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings: dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 032.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1 (local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 132.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 032.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3632.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.032.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.432.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.14732.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)32.152: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings: dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 032.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
  55. 55. DHCP Proxy Enabled – DHCP Offer34.166: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)34.166: 00:16:ea:b2:04:36 DHCP setting server from OFFER (server 10.10.1.3, yiaddr 10.10.1.103)34.167: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)34.167: 00:16:ea:b2:04:36 DHCP transmitting DHCP OFFER (2)34.167: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 034.167: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 034.167: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3634.167: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.10334.167: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.034.168: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
  56. 56. DHCP Proxy Enabled – DHCP Request38.169: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)38.169: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings: dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 038.169: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.3 (local address 10.10.1.4, gateway 10.10.1.3, VLAN 0, port 29)38.169: 00:16:ea:b2:04:36 DHCP transmitting DHCP REQUEST (3)38.169: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 138.170: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 038.170: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3638.170: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.038.170: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.438.170: 00:16:ea:b2:04:36 DHCP requested ip: 10.10.1.10338.170: 00:16:ea:b2:04:36 DHCP server id: 10.10.1.3 rcvd server id: 1.1.1.138.170: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.3 (len 354, port 29, vlan 0)38.170: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings: dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 038.171: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
  57. 57. DHCP Proxy Enabled – DHCP Ack38.172: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)38.173: 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Reached PLUMBFASTPATH: from line 527338.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Replacing Fast Path rule38.173: 00:16:ea:b2:04:36 Assigning Address 10.10.1.103 to mobile38.173: 00:16:ea:b2:04:36 DHCP success event for client. Clearing dhcp failure count for interface management38.174: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)38.174: 00:16:ea:b2:04:36 DHCP transmitting DHCP ACK (5)38.174: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 038.174: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 038.174: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3638.174: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.10338.174: 00:16:ea:b2:04:36 DHCP siaddr: 10.10.1.30, giaddr: 0.0.0.038.174: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.338.179: 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
  58. 58. DHCP Proxy Disabled – Discover/Offer*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)*00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)*00:16:ea:b2:04:36 DHCP processing DHCP OFFER (2)*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
  59. 59. DHCP Proxy Disabled – Request/Ack*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)*00:16:ea:b2:04:36 DHCP processing DHCP REQUEST (3)*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)*00:16:ea:b2:04:36 DHCP processing DHCP ACK (5)*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)*00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA*00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
  60. 60. Learning IP without DHCP*Orphan Packet from 10.99.76.147 on mobile*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)*Installing Orphan Pkt IP address 10.99.76.147 for station*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) Client IP can be learned by ways other than DHCP Client sends gratuitous ARP or ARP Request (Static Client) Client sends IP packet (Orphan Packet), we learn IP DS sends packet to client, we learn IP from DS Seen with mobile devices that talk before validating DHCP Up to client to realize their address is not valid for the subnet DHCP Required on WLAN for prevent this BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
  61. 61. Client DHCP - Takeway DHCP_REQD means Learning IP State Only “Required” if enabled on WLC If Proxy is enabled Confirm DHCP Server on Interface (or Wlan) is correct DHCP Server may not respond to WLC Proxy (Firewalls?) If Proxy is disabled, DHCP is similar to wired client Further Troubleshooting Check DHCP Server for what it believes is happening If WLC does not show a BOOTREQUEST, confirm the client request arrives to the WLC and leaves in the configured way If still believed to be on WLC: debug dhcp message enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
  62. 62. Client Debug –L3 Authentication BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
  63. 63. Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)……………………………...*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state toWEBAUTH_REQD (8) last state WEBAUTH_REQD (8)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP rule*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL ID 255)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile*pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile………………………………*emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last stateWEBAUTH_NOL3SEC (14)*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) laststate RUN (20)*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID =5006 IPv6 Vlan = 3, IPv6 intf id = 8*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
  64. 64. Webauth Redirect Webauth Client State =  Client in WEBAUTH_REQD state “WEBAUTH_REQD“  ARP and DNS must be functional ARP and DNS Function Client attempts to browse internet 3-Way Handshake HTTP HTTP GET  WLC “Hijacks” the handshake 200 Response Client redirects to Virtual Interface 3-Way Handshake HTTP(S) GET Certificate negotiation if applicable Webauth Page Displayed  Webauth page is displayed  Client authenticates Successful Authentication Client State = “RUN“ BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
  65. 65. ARP and DNS FunctionConfirm ARP and DNS FunctionBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
  66. 66. 3-Way Handshake HTTP GET 200 Response 3-Way Handshake HTTP(S) GET Capture from Wireless Adapter Webauth Page DisplayedWebauth Redirect WLC Responding with SYN, ACK Redirect to Virtual Interface Comes from Here WLC Responding with SYN, ACK Client Is Talking to Webauth…. Address for Client to Redirect to (Virtual IP/Name) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
  67. 67. Webauth - Takeaway If WEBAUTH_REQD, then not authenticated Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6* If not redirected, can client browse to virtual IP? Cert issue? Consider disabling HTTPS for HTTP webauth Most common scenario involves ARP/DNS failure Must confirm that client actually sends TCP SYN (http) to IP If proven that TCP SYN is sent and WLC does not SYN ACK, then there may be a WLC side problem Debug webauth enable <client ip address> debug client <MAC Address> debug pm ssh-appgw enable debug pm ssh-tcp enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
  68. 68. Client Debug - RunBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
  69. 69. Run State10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 527310.10.3.82 Added NPU entry of type 1, dtlFlags 0x0OR10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)Session Timeout is 1800 - starting session timer for the mobile10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 506310.10.3.86 Added NPU entry of type 1, dtlFlags 0x0 RUN State is the Client Traffic Forwarding State Client is Connected and should be functional BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
  70. 70. Client Debug –Deauth/DisassocBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
  71. 71. Deauthenticated Client Idle Timeout Occurs after no traffic received from Client Default Duration is 300 secondsReceived Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4Scheduling deletion of Mobile Station: (callerId: 30) in 1 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094) Session Timeout Occurs at scheduled duration (default 1800 seconds) Will force WEBAUTH user to WEBAUTH againapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedScheduling deletion of Mobile Station: (callerId: 45) in 10 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
  72. 72. Deauthenticated Client WLAN Change Modifying a WLAN in anyway Disables and Renables WLANapfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedSent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094) Manual Deauth From GUI: Remove Client From CLI: config client deauthenticate <mac address>apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1Scheduling deletion of Mobile Station: (callerId: 30) in 1 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedSent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
  73. 73. Deauthenticated Client Authentication Timeout Auth or Key Exchange max-retransmissions reachedRetransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3,mscb deauth count 0Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)  AP Radio Reset (Power/Channel) AP disasassociates clients but WLC does not delete entry Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0) apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
  74. 74. Deauthentication - Takeaway Client can be removed for numerous reasons WLAN change, AP change, configured interval Start with Client Debug to see if there is a reason for a client‟s deauthentication Further Troubleshooting Client debug should give some indication of what kind of deauth is happening Packet capture or client logs may be require to see exact reasonBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
  75. 75. Client Debug – Tips and Tricks BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
  76. 76. Tips and Tricks Collect a client debug for an extended duration Several roams, deauths, failures, etc… Use an enhanced text editor with filter or “find all” I use Notepad++ Find All “Association Received” (will also pull reassociations) “Assoc Resp” “Access-Reject” “timeoutEvt”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
  77. 77. Tips and TricksBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
  78. 78. Tips and TricksBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
  79. 79. Client Debug – SummaryBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
  80. 80. Client Connectivity Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585 Configuration Issues SSID Mismatch Security Mismatch Disabled WLAN Unsupported Data-Rates Disabled Clients Radio Preambles Cisco Features - Issues with Third Party Clients Aironet IE MFPBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
  81. 81. 802.11n Speeds Troubleshoot 802.11n Speeds Document ID: 112055 Configuration Issues 11n Support Enabled WMM is Allowed or Required Open or WPA2-AES 5Ghz Channel Width 2.4Ghz does not support 40-Mhz ChannelsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
  82. 82. 802.11n A-MPDU/A-MSDU Aggregation methods used could impact interop or performance 802.11n Status: WLC Default 11n Config: A-MPDU Tx: Priority 0............................... Enabled Priority 1............................... Disabled Priority 2............................... Disabled Priority 3............................... Disabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... Disabled A-MSDU Tx: Priority 0............................... Enabled Priority 1............................... Enabled Priority 2............................... Enabled Priority 3............................... Enabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... DisabledBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
  83. 83. WLC Config Analyzer (WLCCA)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
  84. 84. What Is the WLCCA? It is a Post Sales tool Main objective: Save time while analyzing configuration files from WLCs Secondary objective: Carry out RF analysis It is NOT a management or monitoring tool Focused to work off-line to the WLC Not TAC supported Development: wlc-conf-app-dev@cisco.com General internal alias:wlc-conf-app@cisco.com “Pet project”: no official Cisco product. BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
  85. 85. Where? Support Forums DOC-1373 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
  86. 86. Input Needed Complete config output from WLC Show run-config It does not work with old “show running-config” or with TFTP backup, or with show tech The show run-config acts as “snapshot” of current config + RF state Likely best to obtain config from SSH with config paging disable BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
  87. 87. Functionality Overview - Checks Audit Checks More than 100 config detail verifications Based on TAC/Escalation cases experience Some obvious, some hard to catch No “change this” messages, some need “contextualization” BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
  88. 88. Functionality Overview Audit Checks BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
  89. 89. Functionality Overview Config View BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
  90. 90. WLCCA – High RF Index APsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
  91. 91. Reducing CCI Turn off excess 2.4 radios. May want to do this gradually, e.g. turn off 20% of radios per attempt After turning off excess radios, could set DCA sensitivity to high Let DCA/power settings settle down overnight. See how things look in the morning Repeat till you see the desired coverage in 2.4GHz BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
  92. 92. 2.4GHz – Target Coverage Most all 2.4GHz radios are at power 2 - 5 (dont want 7 or 8) In all locations, you have coverage that looks like this (take these as guidelines, not gospel): Hottest channels AP is at least -67dBm Next hottest AP on that channel is at least 19 dB below the hottest Next hottest channels AP is at least -67dBm OK if next hottest AP on that channel is less than 19 dB below the hottest BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
  93. 93. 5 GHz – Target Coverage  Most all 5GHz radios are at power 1 – 3 (at least 14dBm) Consider the RRM min power setting in 6.0 Consider a radically high tx-power-threshold, like -55 dBm  8 – 12 channels in use (20 seem to be too many for the 792x to scan)  In all locations, seek this: Hottest channels AP is at least -67dBm Next hottest AP on that channel is at least 19 dB below the hottest Next hottest channels AP is at least -67dBm OK if next hottest AP on that channel is less than 19 dB below the hottest BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
  94. 94. Additional TroubleshootingBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
  95. 95. Additional Troubleshooting Wireshark Tutorial Clean Air SE-Connect / AP Sniffer Mode AP Join RRM Multicast/Broadcast Mobility VoWiFiBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
  96. 96. Wireshark Tutorial BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
  97. 97. Wireshark Tutorial Default Wireshark view might look like this:BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
  98. 98. Wireshark Tutorial Newer versions of Wireshark have a feature for “Apply as Column” This will take any decodable parameter and make a columnBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
  99. 99. Wireshark Tutorial Within seconds your wireshark can also have:BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
  100. 100. Wireshark Tutorial Filtering data is just as easyBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
  101. 101. Wireshark Tutorial - CAPWAP User data is encapsulated in CAPWAPBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
  102. 102. Wireshark Tutorial Wireshark can also de-encapsulate CAPWAP DATA Edit > Preference > Protocols > CAPWAPBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
  103. 103. Wireshark Tutorial With CAPWAP de-encapsulated you can see all the packets to/from client (between AP and WLC)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
  104. 104. SE-Connect – Clean AirAP Sniffer ModeBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
  105. 105. SE-Connect and Sniffer Mode Clean Air APs can be used in lieu of Spectrum Card for Spectrum Analysis AP can be placed in SE-Connect mode for full functionality AP in local mode can be used now for Spectrum Analysis of current channel AP Sniffer Mode can be used in lieu of Wireless Sniffer Packets can be sent from either radio upstream to a packet capture software (Wireshark or Omnipeek for example)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
  106. 106. Spectrum Expert with Clean Air Obtain Spectrum Key Connect to Remote Sensor BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
  107. 107. Spectrum Expert with Clean AirBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 107
  108. 108. Sniffer Mode AP Select channel to Sniff Select destination for traffic BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
  109. 109. Sniffer Mode AP Omnipeek has a Remote Adapter to capture this data Wireshark, just capture network adapter NOTE: Wireshark does not open the port UDP 5000 PC will send ICMP Unreachables BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
  110. 110. Sniffer Mode AP With wireshark, filter !icmp.type == 3 Data (UDP 5000) still not intelligible yet Decode as Airopeek BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
  111. 111. Sniffer Mode APBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
  112. 112. AP Discover/JoinBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
  113. 113. AP Discover/Join AP Runs Hunting Algorithm to Find Candidate Controllers to JoinBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
  114. 114. AP - Discover Process AP Discovery Req to known and learned WLCs Broadcast Reaches WLCs with MGMT Interface in local subnet of AP Use “ip helper-address <ip>” with “ip forward-protocol udp” Dynamic DNS: cisco-capwap-controller DHCP: Option 43 Configured (nvram) High Availability WLCs – Pri/Sec/Ter/Backup Last WLC All WLCs in same mobility group as last WLC Manual from AP - “capwap ap controller ip address <ip>”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
  115. 115. AP - Discover Process broadcast X Discover Request sent to all methods the AP knows Discover Response sent from all WLCs that received the Discovery RequestBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
  116. 116. AP – WLC Selection/Join WLCs send Discovery Response back to AP Name, Capacity, AP Count, Master?, AP-MGR, Load per AP- MGR AP selects the single best WLC candidate from High Availability Config: Primary/Secondary/Tertiary/Backup Master Controller Greatest available capacity Ratio of total capacity to available capacity AP sends single Join Request to best candidate WLC responds with Join Response AP joins and receives config (or downloads image if not correct) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
  117. 117. Troubleshooting AP Discovery/Join “Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC)”, Document ID 70333 Make sure time on WLC is accurate! From AP: Debug ip udp Debug capwap client events From WLC Debug mac addr <AP ethernet mac> Debug capwap [event/error/packet] enable Debug pm pki enable BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
  118. 118. RRMBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
  119. 119. RRM There are usually only two common scenarios or issues involving RRM APs not changing channel Check if other APs are in each others neighbor list APs not changing power Nearby APs list meets the general rule of RSSI from 3rd closest AP is better than TPC ThresholdBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
  120. 120. RRM Debugs WLC – debug airewave-director <?> AP debug capwap rm mesurements debug capwap rm rogueBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
  121. 121. RRM Show AP Auto-RF (In Run-Config) show ap auto-rf [802.11a/b] <AP Name> Load Information Receive Utilization.. 0 % Rx load to Radio Transmit Utilization.. 2 % Tx load from Radio Channel Utilization.. 12 % % Busy Nearby APs AP 00:16:9c:4b:c4:c0 slot 0.. -28 dBm on 11 (10.10.1.5) AP 00:26:cb:94:44:c0 slot 0.. -32 dBm on 11 (10.10.1.4)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
  122. 122. Broadcast/MulticastBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
  123. 123. Broadcast/MulticastBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 123
  124. 124. Broadcast/Multicast AP Multicast Mode – Multicast Address must be unique among WLCs Broadcast Traffic is delivered via the Multicast Mode AP/WLC/Client Subnets must be Multicast enabled For Multicast Mode - Multicast Quick check for Multicast is to confirm that Multicast- Unicast mode works BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
  125. 125. Broadcast/Multicast AP Show Commands Show capwap mcast Show capwap mcast mgid all BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
  126. 126. Client Mobility BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
  127. 127. Mobility—Intra-Controller Client roams between two APs on the same controller BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
  128. 128. Mobility—Inter-Controller (Layer 2)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
  129. 129. Mobility—Layer 3 Layer 3 roaming (a.k.a. anchor/foreign) New WLC does not have an interface on the subnet the client is on New WLC will tell the old WLC to forward all client traffic to the new WLC Asymmetric traffic path established (deprecated) Symmetric traffic pathBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
  130. 130. Mobility—Messaging Flow When a client connects to a WLC for the first time, the following happens: New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when client connects Old WLC sends HANDOFF_REQUEST New WLC sends HANDOFF_REPLYBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
  131. 131. Debug Client <Mac Address>Mobility— L2 Inter WLC Debug Mobility Handoff EnableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
  132. 132. Debug Client <Mac Address>Mobility— L3 Inter WLC Debug Mobility Handoff EnableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 132
  133. 133. Debug Client <Mac Address>Mobility— L3 Inter WLC Debug Mobility Handoff EnableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 133
  134. 134. Mobility— L3 Handoff Ignored*mmListen: Mobility packet received from:*mmListen: 10.4.22.55, port 16666*mmListen: type: 3(MobileAnnounce) subtype: 0 version: 1 xid: 783 seq: 1453 len 116 flags 0*mmListen: group id: e42cb3a9 87f62b45 57c0f8a3 92747b23*mmListen: mobile MAC: 00:23:33:41:71:10, IP: 0.0.0.0, instance: 0*mmListen: VLAN IP: 10.4.23.97, netmask: 255.255.255.0*mmListen: Switch IP: 10.4.22.55*mmListen: Handoff Virtual IP Mismatch, Local = 1010101, Request = 1020304 **** Handoff Request Ignored*apfReceiveTask: 10.4.122.127 RUN (20) State Update from Mobility-Complete to Mobility-Incomplete*apfReceiveTask: Mobile 00:23:33:41:71:10 associated with another AP elsewhere, delete mobile*apfReceiveTask: 10.4.122.127 RUN (20) mobility role update request from Local to Handoff Peer = 0.0.0.0, Old Anchor = 10.4.130.70, New Anchor = 0.0.0.0*apfReceiveTask: Clearing Address 10.4.122.127 on mobile*apfReceiveTask: apfMsRunStateDec*apfReceiveTask: 10.4.122.127 RUN (20) Change state to DHCP_REQD (7) last state RUN (20)*apfReceiveTask: apfMmProcessDeleteMobile (apf_mm.c:548) Expiring Mobile!*apfReceiveTask: Mobility Response: IP 0.0.0.0 code Handoff Indication (2), reason Client handoff successful - anchor retained (0), PEM State DHCP_REQD, Role Handoff(6)*apfReceiveTask: apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:23:33:41:71:10 on AP 10:8c:cf:eb:69:80 from Associated to Disassociated*apfReceiveTask: Deleting mobile on AP 10:8c:cf:eb:69:80(1)*pemReceiveTask: 0.0.0.0 Removed NPU entry. BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 134
  135. 135. Mobility Group vs. Mobility Domain Mobility Group - WLCs with the same group name L2/L3 Handoff Auto Anchoring Fast Secure Roaming APs get all of these as a Discover candidate Mobility Domain - WLCs in the mobility list L2/L3 Handoff Auto AnchoringBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
  136. 136. Mobility Data/Control Path Sent between all WLCs, by member with lowest MAC Control Path = UDP 16666 (30 Seconds) Data Path = EoIP Protocol 97 (10 Seconds) debug mobility keep-alive enable <IP Address>BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 136
  137. 137. Voice over WiFi BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 137
  138. 138. VoWiFi Wireless IP Phone Deployment Guide http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/79 25g/7_0/english/deployment/guide/7925dply.pdf Best Practices -67 dBm signal with 20-30% cell overlap 802.11A CCKM for Fastest Roaming Avoid designs where AP is seen at superb signal, but drops off instantlyBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 138
  139. 139. VoWiFi - Troubleshooting Must know if problem occurs during roaming events or when no association change takes place If no change in connection Interference Coverage loss with no other candidate End to End QOS missing/problem If during roaming event How long did the roam take? Does the client associate to another AP again within seconds? Does the client associate to the same AP again? Is the phone roaming to the designed next candidate?BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 139
  140. 140. VoWiFi - Troubleshooting Define a reproducible area where you believe you have perfect voice coverage but have problems Place phone in Neighbor List Mode (On a call) Real Time current AP RSSI and candidate list Confirm AP as next best candidate is realistically a good candidate Confirm devices roams to correct candidate where the intended design specifies Watch out for sudden drops in coverageBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 140
  141. 141. VoWiFi - Debugs Phone can Trace (debug) to file or syslog Recommend USB Connection and SYSLOG Configured via GUI Enable Debug level for Kernel, WLAN MGR, WLAN Driver WLC Debugs Debug client <mac> Debug cac all enable Wireless Packet CapturesBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 141
  142. 142. SummaryBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
  143. 143. SummaryClient WLC - show run-config, debug client <mac>, debug dhcp message enable, debug dot1x <?> enable, debug aaa <?> enable, AP - Show tech, show controller D<0/1> Data - Driver/Supplicant Logs, Wireless Capture, AAA Logs, DHCP LogsWebauth WLC - (Client debugs), debug webauth enable <IP>, debug pm ssh-appgw enable, debug pm ssh-tcp enable Client - local captureMobility WLC - debug mobility handoff enable, debug mobility keepalive enable <IP> Data - Wired captureAP Join WLC - debug capwap [events/error/packet] enable AP - debug capwap client events, debug ip udp Data - Wired captureRRM WLC - show run-config, debug airewave-director <?> AP - debug capwap rm measurements, debug capwap rm rogueMulticast/Broadcast AP - show capwap mcast, show capwap mcast mgid all Data - Infrastructure ConfigurationVoice WLC - (Client debugs), debug cac all enable Data – Wireless capture, Phone traces BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 143
  144. 144. Summary Links: Understanding Debug Client on Wireless LAN Controllers (WLCs) Document ID: 100260 Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585 Troubleshoot 802.11n Speeds Document ID: 112055 Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller Document ID: 99948BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 144
  145. 145. Complete Your OnlineSession Evaluation Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don‟t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 145
  146. 146. Visit the Cisco Store for Related Titles http://theciscostores.comBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
  147. 147. BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 147
  148. 148. Thank you.BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×