Troubleshooting Wireless LANs with Centralized Controllers

Troubleshooting Wireless LANs Supportability Software and Support Model Troubleshooting Basics The Client Debug WLC Config Analyzer (WLCCA) Additional TroubleshootingBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 2

Supportability WLC Supportability Methods of Management Using the GUI Important Show Commands (CLI) Important Debugs (CLI) Best Practices AP Supportability Methods of Accessing the AP Important Show CommandsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 4

WLC SupportabilityMethods of Management Default Mode GUI (E)=Enabled (D)=Disabled HTTPS (E) / HTTP (D) CLI Console SSH (E) / Telnet (D) SNMP V1 (D) / V2 (E) – Change me! V3 (E) – Change meNote: Management Via Wireless Clients (D)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

WLC SupportabilityUsing the GUI Wireless > All APs AP list shows AP Physical UP Time APs are sorted by Controller Associated Time Check bottom of AP list for any recent AP disruptions Select AP to see Controller Associated Time (duration)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 7

WLC SupportabilityImportant Show Commands (CLI) Show run-config Must have! No exceptions! “show run-config commands” (like IOS show running-config) “show run-config no-ap” (no AP information added) Show tech-support CLI Tip Log all output Config Paging DisableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 9

WLC SupportabilityImportant Debugs (CLI) Debug client <client mac address> Client Involved? Must Have! No Exceptions Debug capwap <event/error/detail/info> enable CLI Tips Log all output Debugs are session based, they end when session ends “Config session timeout 60”, sets 60 minute idle timeout Debug mac addr <mac address> Used to filter debugs on specific Mac Address Debug disable-all (Disables all debugs)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

WLC SupportabilityBest Practices Change default SNMP Parameters Configure Syslog for WLC and AP Enable Coredump for WLC and AP Configure NTP Server for Date/TimeBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 11

AP Supportability Methods of Accessing the AP Default Mode Console (E)=Enabled (D)=Disabled Telnet (D) / SSH (D) No GUI support AP Remote Commands Enabling Telnet/SSH WLC CLI: config ap [telnet/ssh] enable <ap name> WLC GUI: Wireless > All APs > Select AP > Advanced Select [telnet/ssh] > ApplyBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 12

AP SupportabilityAP Remote Commands (WLC CLI) Debug AP enable <AP name> Enables AP Remote Debug AP Must be associated to WLC Redirects AP Console output to WLC session Debug AP command “<command>” <AP name> Output is redirected to WLC session AP runs IOS, numerous generic IOS commands availableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 13

AP SupportabilityShow Commands (AP CLI or WLC Remote Cmd) Show controller Do[0/1] (or Show Tech) Must have! Before/During/After event Show log WLC: show ap eventlog <ap name> Show capwap client <?> CLI Tips Debug capwap console client Debug capwap client no-reloadBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

Software and Support Model Opening a TAC Service Request Cisco Support Model TAC vs Business Unit What to expect from TAC How does escalation work? WLC Software Trains CCO (ED/MD/AW) “Engineering Special” vs “Escalation”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

Opening a TAC Service Request What should I have ready? Clear problem description Always: Show run-config If client involved, always: “debug client <mac address>” Your analysis of any data provided Expectations for customer involvement TAC SR severity level descriptions state that You and Cisco will commit necessary resources according to severity You must set correct expectation of timeline and severityBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

Opening a TAC Service Request Potential reasons to slow a TAC SR‟s resolution Information about the problem is missing The severity level was not set appropriately Data, such as traces or logs, has not been forwarded to the engineer The scope or time requirements are not well understood by the engineer The problem cannot be reproduced in the Cisco Technical Assistance Center lab Access to the affected equipment for debugging purposes is not availableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

Cisco Support Model – TAC vs. BU TAC Customer advocate Technology focused with cross technology collaboration Escalation path within TAC exists Business Unit - Escalation Work in conjunction with TAC during specific engagements Product specific focus Engages development resources when necessaryBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 19

Cisco Support Model – Expectations What not to expect from TAC Design and deployment Complete configuration Sales related information What to expect from TAC Configuration assistance Problem analysis / bug isolation Workarounds or fixes Action plan to resolve SR Hardware replacement Engage BU when appropriateBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

Cisco Support Model - Escalation TAC Escalation Process Multi-Tier support resources within a technology TAC to engage resources (TAC/BU) when appropriate SR ownership might not change hands Customer Escalation Process Raise SR priority (S1/S2) Engage account team Your satisfaction is important to the Cisco TAC. If you have concerns about the progress of your case, please contact your regional TAC.BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

WLC Software Trains CCO - Cisco.com release 6.0.202.0, 7.0.116.0, etc… Full test cycle Classified as ED when posted AssureWave AW is no longer tagged on CCO, but AW validation results are available at: http://www.cisco.com/go/assurewave Results available 4 weeks after CCO MD MD tag represents stable releases for mass adoption MD tag will be considered on CCO after AW release validation, 10 weeks in field and TAC/Escalation signoffBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 22

WLC Software Trains - ES vs. Escalation Engineering Special Development “special” image for fix validation or limited use Sanity tested “As-is” Escalation Code Escalation is a post-CCO maintenance release with specific/minimal customer impacting SW fixes Fix must be fully committed to the next CCO MR Sanity + focus tested Fully TAC+BU supported “Running-Master” so each release builds upon the previousBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

The 10-Point Capture EAP Radio chan. 1 Driver Supp. IP RADIUS ACS 802.11 Data WLC 802.11 Management CAPWAP EOIP IP IP DHCP 802.11 Management CAPWAP WLCSupplicant Wired Logs AP Sniff Wired Debugs Sniff Driver Debugs/ DHCP Wireless Spectrum WLC Logs ACS Adapter Sniff Analysis Debugs Logs Capture NTP BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 25

Troubleshooting Basics Troubleshooting 101 Problem Definition Clearly define the problem Understand any possible triggers Know the expected behavior Questions Reproducibility Recommended Tools Tests Spectrum Analyzer Wireless Sniffer and Wired Captures Analysis Solution(s)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 26

Troubleshooting 101 Troubleshooting is an art with no right or wrong procedure, but best with a logical methodology. Step 1: Define the problem It is crucial to understand all possible details of a problem Knowing what is and is not working will go a long way With a proper understanding of the problem description you can skip many steps Bad description: “Client slow to connect” Good description: “Client associations are rejected with Status17 several times before they associate successfully.”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

Troubleshooting 101 Step 2: Understand any possible triggers If something previously worked but no longer works, there should be an identifiable trigger Understanding any and all configuration or environmental changes could help pinpoint a trigger Step 3: Know the expected behavior If you know the order of expected behavior that is failing, defining where the behavior breaks down (Problem Description) is better than defining the end result. Example: “One way audio between Phone A and B, because Phone A does not get an ARP Response for Phone B”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 28

Troubleshooting 101 Step 4: Reproducibility Any problem that has a known procedure to reproduce (or frequently randomly occurs) should be easy to diagnose Being able to easily validate or disprove a potential solution saves time by being able to quickly move on to the next theory If a problem is reproducible in other environments with a known procedure, TAC/BU can facilitate internal testing and proposed fix/workaround verification Debugs and Captures of working scenarios can help pin point where exactly the difference isBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

Recommended Tools Wireless Sniffer Example: Linksys USB600N with Omnipeek TAC can publish Omnipeek-RA if you have compatible HW Wired Packet Capture Example: Wireshark Use for spanned switchports of AP/WLC or client side data Spectrum Analyzer Spectrum Expert with Card or Clean-Air APBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 30

Steps to Building an 802.11 Connection 802.11 1. Listen for Beacons State 1:Unauthenticated, 2. Probe Request Unassociated 3. Probe Response AP 4. Authentication Request 5. Authentication Response State 2: Authenticated, 6. Association Request Unassociated 7. Association Response WLC State 3: 8. (Optional: EAPOL Authentication) Authenticated, Associated 9. (Optional: Encrypt Data) 10. Move User Data BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 32

The Client Debugdebug client <mac address> A multi-debug macro (Cisco Controller) >debug client 00:16:EA:B2:04:36 (Cisco Controller) >show debug MAC address ................................ 00:16:ea:b2:04:36 Debug Flags Enabled: dhcp packet enabled dot11 mobile enabled dot11 state enabled dot1x events enabled dot1x states enabled pem events enabled pem state enabled CCKM client debug enabledBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 33

Understanding the Client StateName Description8021X_REQD 802.1x (L2) Authentication PendingDHCP_REQD IP Learning StateWEBAUTH_REQD Web (L3) Authentication PendingRUN Client Traffic Forwarding (Cisco Controller) >show client detail 00:16:ea:b2:04:36 Client MAC Address............................... 00:16:ea:b2:04:36 ….. Policy Manager State............................. WEBAUTH_REQD 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

The Client Debug - Walkthrough Association (Start) L2 Authentication (8021X_REQD) Client Address Learning (DHCP_REQD) L3 Authentication (WEBAUTH_REQD) Client Fully Connected (RUN) Deauth/Disassoc Tips and TricksBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 35

Association(Cisco Controller) >debug client 00:16:EA:B2:04:36(Cisco Controller) >(Cisco Controller) >Association received from mobile on AP 00:26:cb:94:44:c00.0.0.0 START (0) Changing ACL none (ACL ID 0) ===> none (ACL ID 255) --- (caller apf_policy.c:1621)Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site default-group, interface 3Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface 3„STA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:360.0.0.0 START (0) Initializing policy0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1apfMsAssoStateIncapfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to AssociatedScheduling deletion of Mobile Station: (callerId: 49) in 1800 secondsSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

AssociationAssociation received from mobile on AP 00:26:cb:94:44:c00.0.0.0 START (0) Changing ACL none (ACL ID 0) ===> none (ACL ID 255) --- (caller apf_policy.c:1621)Applying site-specific IPv6 override for station 00:16:ea:b2:04:36 - vapId 1, site default-group, interface 3Applying IPv6 Interface Policy for station 00:16:ea:b2:04:36 - vlan 3, interface id 8, interface 3  Association received Association Request, client did not “Roam” (Reassociate) AP Base Radio = 00:26:cb:94:44:c0  vapId 1, site default-group, interface 3„ vapId = WLAN # (Wlan 1) site = AP Group (default-group) Interface = Dynamic Interface name (3)  vlan 3 Vlan = Vlan # of Dynamic Interface BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

AssociationSTA - rates (12): 130 132 139 150 12 18 24 36 48 72 96 108 0 0 0 0Processing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36  STA - rates Madatory Rates (>128) = (#-128)/2 Supported Rates (<128) = #/2 1m,2m,5.5m,11m,6s,9s,12s,18s,24s,36s,48s,54s  Processing RSN IE type 48 WPA2-AES Processing WPA IE type 221 = WPA-TKIP BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

Association0.0.0.0 START (0) Initializing policy0.0.0.0 START (0) Change state to AUTHCHECK (2) last state AUTHCHECK (2)0.0.0.0 AUTHCHECK (2) Change state to 8021X_REQD (3) last state 8021X_REQD (3)0.0.0.0 8021X_REQD (3) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1for this client0.0.0.0 8021X_REQD (3) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 1 apVapId 1apfMsAssoStateIncapfPemAddUser2 Changing state for mobile 00:16:ea:b2:04:36 on AP 00:26:cb:94:44:c0 from Idle to AssociatedScheduling deletion of Mobile Station: (callerId: 49) in 1800 seconds  0.0.0.0 START 0.0.0.0 = IP we know for client (In this case nothing)  Change state to 8021X_REQD Passed association, moving client to next state: 8021X_REQD  Scheduling deletion Session Time on WLAN (1800 seconds in this case) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 40

AssociationSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0  Slot 0 = B/G(2.4) Radio Slot 1 = A(5) Radio  Sending Assoc Response Status 0 = Success Anything other than Status 0 is Failure Common Assoc Response Failures: 1 – Unknown Reason – Anything not matching defined reason codes 12 – Unknown or Disabled SSID 17 – AP cannot handle any more associations 18 – Client is using a datarate that is not allowed 35 – WLAN requires the use of WMM and client does not support it 201 – Voice client attempting to connect to a non-platinum WLAN 202 – Not enough available bandwidth to handle a new voice call (CAC Rejection) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

Association - FSRProcessing WPA IE type 221, length 22 for mobile 00:16:ea:b2:04:36CCKM: Mobile is using CCKMCCKM: Processing REASSOC REQ IEIncluding CCKM Response IE (length 62) in Assoc Resp to mobileSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) Vap Id 6 Slot 1ORProcessing RSN IE type 48, length 22 for mobile 00:16:ea:b2:04:36Received RSN IE with 1 PMKIDs from mobile 00:16:ea:b2:04:36Received PMKID: (16) [0000] cb bc 27 82 88 14 92 fd 3b 88 de 6a eb 49 be c8Found an entry in the global PMK cache for stationComputed a valid PMKID from global PMK cache for mobile FSR aIOS CUWN CCKM - WPA yes yes CCKM - WPA2 yes yes WPA2 PKC no yes WPA2 "Sticky" yes no* BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 42

Association - Takeaway Association vs. Reassociation Debug shows AP, Slot, AP-Group, WLAN ID, Interface, Data Rates, Encryption type Association Response Confirms if Client is associated Defines reason if denied Further troubleshooting May require Wireless Sniffer or capture at AP Switchport If not sending Assoc Request, must know why from Client Trying disabling WLAN features to “dumb it down”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 43

802.1X AuthenticationSupplicant Authenticator Server EAPOL-START EAP-ID-Request EAP-ID-Response RADIUS (EAP-ID_Response) Rest of the EAP Conversation Radius-Access-Accept EAP-Success (Key) The Supplicant Derives the Session Key from User Password or Session Certificate and Authentication Key Exchange BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

WPA2-AES-802.1XSending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Station 00:16:ea:b2:04:36 setting dot1x reauth timeout = 1800dot1x - moving mobile 00:16:ea:b2:04:36 into Connecting state Sending EAP-Request/Identity to mobile 00:16:ea:b2:04:36 (EAP Id 1) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Username entry (cisco) created for mobile Received Identity Response (count=1) from mobile 00:16:ea:b2:04:36 EAP State update from Connecting to Authenticating for mobile 00:16:ea:b2:04:36 dot1x - moving mobile 00:16:ea:b2:04:36 into Authenticating state………………….. Entering Backend Auth Req state (id=3) for mobile 00:16:ea:b2:04:36 Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25)...........................Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 10, EAP Type 25) Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36 Processing Access-Challenge for mobile 00:16:ea:b2:04:36 Entering Backend Auth Req state (id=11) for mobile 00:16:ea:b2:04:36 Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 11) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 11, EAP Type 25) Entering Backend Auth Response state for mobile 00:16:ea:b2:04:36 Processing Access-Accept for mobile 00:16:ea:b2:04:36***OR*** Processing Access-Reject for mobile 00:16:ea:b2:04:36BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

Common EAP Types 1 – Identity 2 – Notification 3 – NAK Sending EAP Request from AAA to mobile 00:16:ea:b2:04:36 (EAP Id 3) Received EAPOL EAPPKT from mobile 00:16:ea:b2:04:36 4 – MD5 Received EAP Response from mobile 00:16:ea:b2:04:36 (EAP Id 3, EAP Type 25) 5 – OTP 6 – Generic Token 13 – EAP TLS 17 – LEAP 18 – EAP SIM 21 – EAP TTLS 25 – PEAP 43 – EAP-FAST BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

802.1X (Cont.) (WPA2-AES-PSK)Sending Assoc Response to station on BSSID 00:26:cb:94:44:c0 (status 0) ApVapId 1 Slot 0Creating a PKC PMKID Cache entry for station 00:16:ea:b2:04:36 (RSN 2)Adding BSSID 00:26:cb:94:44:c0 to PMKID cache for station 00:16:ea:b2:04:36New PMKID: (16) [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cdInitiating RSN PSK to mobile 00:16:ea:b2:04:36dot1x - moving mobile 00:16:ea:b2:04:36 into Force Auth stateSkipping EAP-Success to mobile 00:16:ea:b2:04:36Including PMKID in M1 (16) [0000] 31 d5 5b 0b 64 28 2b be c5 8d d5 4c 03 30 c7 cdStarting key exchange to mobile 00:16:ea:b2:04:36, data packets will be droppedSending EAPOL-Key Message to mobile 00:16:ea:b2:04:36 state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00Received EAPOL-Key from mobile 00:16:ea:b2:04:36Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36Received EAPOL-key in PTK_START state (message 2) from mobile 00:16:ea:b2:04:36Stopping retransmission timer for mobile 00:16:ea:b2:04:36Sending EAPOL-Key Message to mobile 00:16:ea:b2:04:36 state PTKINITNEGOTIATING (message 3), replay counter 00.00.00.00.00.00.00.01Received EAPOL-Key from mobile 00:16:ea:b2:04:36Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:16:ea:b2:04:36Received EAPOL-key in PTKINITNEGOTIATING state (message 4) from mobile 00:16:ea:b2:04:36apfMs1xStateInc0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4) last state L2AUTHCOMPLETE (4) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

WPA2-AES-PSK - FailedStarting key exchange to mobile 00:1e:8c:0f:a4:57, data packets will be droppedSending EAPOL-Key Message to mobile 00:1e:8c:0f:a4:57 state INITPMK (message 1), replay counter 00.00.00.00.00.00.00.00Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit 1 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57Received EAPOL-Key from mobile 00:1e:8c:0f:a4:57Ignoring invalid EAPOL version (1) in EAPOL-key message from mobile 00:1e:8c:0f:a4:57Received EAPOL-key in PTK_START state (message 2) from mobile 00:1e:8c:0f:a4:57Received EAPOL-key M2 with invalid MIC from mobile 00:1e:8c:0f:a4:57802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit 2 of EAPOL-Key M1 (length 121) for mobile 00:1e:8c:0f:a4:57…………………802.1x timeoutEvt Timer expired for station 00:1e:8c:0f:a4:57Retransmit failure for EAPOL-Key M1 to mobile 00:1e:8c:0f:a4:57, retransmit count 3, mscb deauth count 3Blacklisting (if enabled) mobile 00:1e:8c:0f:a4:57apfBlacklistMobileStationEntry2 (apf_ms.c:4192) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:16:9c:4b:c4:c0 from Associated to Exclusion-list (1) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 49

L2 Authentication - Takeaway  8021X_REQD means L2 Authentication pending Authentication/Encryption has not be established  PSK is 802.1X, key is derived from PSK not AAA  If “Processing Access-Reject” AAA/RADIUS Rejected the user (not the WLC)  If “Processing Access-Accept” AAA/Radius Accepted the user M1-M4 should follow  Further Troubleshooting Debug aaa [all/event/detail/packet] enable Debug dot1x [aaa/packet] enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 50

Client DHCP00:16:ea:b2:04:36 Received EAPOL-key in PTKINITNEGOTIATING state00:16:ea:b2:04:36 apfMs1xStateInc00:16:ea:b2:04:36 0.0.0.0 8021X_REQD (3) Change state to L2AUTHCOMPLETE (4)00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) DHCP Not required on AP 00:26:cb:94:44:c0 vapId 3 apVapId 3for this client00:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Plumbed mobile LWAPP rule on AP 00:26:cb:94:44:c0 vapId 3 apVapId 300:16:ea:b2:04:36 0.0.0.0 L2AUTHCOMPLETE (4) Change state to DHCP_REQD (7)00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) pemAdvanceState2 4755, Adding TMP rule00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)00:16:ea:b2:04:36 Stopping retransmission timer for mobile 00:16:ea:b2:04:36*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0...................00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 308,vlan 0, port 29, encap 0xec03)...................00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)...................00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 52

Client DHCP Client State = Client is in DHCP_REQD state “DHCP_REQD“ Proxy Enabled: DHCP Proxy Enabled DHCP Proxy Disabled DHCP Relay/Proxy Between WLC and Server Client DHCP Discover Client DHCP Discover Is Unicast to DHCP Bridged to DS Required for Internal DHCP Servers Proxy Disabled: DHCP Offer from Server Between Client and Server DHCP is broadcast out VLAN Client DHCP Request IP helper or other means required DHCP ACK from Server IP Address Learned BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 53

DHCP Proxy Enabled – DHCP Discover*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x032.151: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)32.151: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings: dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 0.0.0.0 VLAN: 032.151: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.1 (local address 10.10.1.4, gateway 10.10.1.1, VLAN 0, port 29)32.151: 00:16:ea:b2:04:36 DHCP transmitting DHCP DISCOVER (1)32.151: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 132.151: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 032.152: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3632.152: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.032.152: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.432.152: 00:16:ea:b2:04:36 DHCP requested ip: 10.99.76.14732.152: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.1 (len 346, port 29, vlan 0)32.152: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings: dhcpServer: 0.0.0.0, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 032.152: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 54

DHCP Proxy Enabled – DHCP Offer34.166: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)34.166: 00:16:ea:b2:04:36 DHCP setting server from OFFER (server 10.10.1.3, yiaddr 10.10.1.103)34.167: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)34.167: 00:16:ea:b2:04:36 DHCP transmitting DHCP OFFER (2)34.167: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 034.167: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 034.167: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3634.167: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.10334.167: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.034.168: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.3 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

DHCP Proxy Enabled – DHCP Request38.169: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)38.169: 00:16:ea:b2:04:36 DHCP selecting relay 1 - control block settings: dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 038.169: 00:16:ea:b2:04:36 DHCP selected relay 1 - 10.10.1.3 (local address 10.10.1.4, gateway 10.10.1.3, VLAN 0, port 29)38.169: 00:16:ea:b2:04:36 DHCP transmitting DHCP REQUEST (3)38.169: 00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 138.170: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 038.170: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3638.170: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.038.170: 00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 10.10.1.438.170: 00:16:ea:b2:04:36 DHCP requested ip: 10.10.1.10338.170: 00:16:ea:b2:04:36 DHCP server id: 10.10.1.3 rcvd server id: 1.1.1.138.170: 00:16:ea:b2:04:36 DHCP sending REQUEST to 10.10.1.3 (len 354, port 29, vlan 0)38.170: 00:16:ea:b2:04:36 DHCP selecting relay 2 - control block settings: dhcpServer: 10.10.1.3, dhcpNetmask: 0.0.0.0, dhcpGateway: 0.0.0.0, dhcpRelay: 10.10.1.4 VLAN: 038.171: 00:16:ea:b2:04:36 DHCP selected relay 2 - NONE BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

DHCP Proxy Enabled – DHCP Ack38.172: 00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 0, port 29, encap 0xec00)38.173: 00:16:ea:b2:04:36 10.10.1.103 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)38.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Reached PLUMBFASTPATH: from line 527338.173: 00:16:ea:b2:04:36 10.10.1.103 RUN (20) Replacing Fast Path rule38.173: 00:16:ea:b2:04:36 Assigning Address 10.10.1.103 to mobile38.173: 00:16:ea:b2:04:36 DHCP success event for client. Clearing dhcp failure count for interface management38.174: 00:16:ea:b2:04:36 DHCP sending REPLY to STA (len 414, port 29, vlan 0)38.174: 00:16:ea:b2:04:36 DHCP transmitting DHCP ACK (5)38.174: 00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 038.174: 00:16:ea:b2:04:36 DHCP xid: 0x91014db0 (2432781744), secs: 0, flags: 038.174: 00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:3638.174: 00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.1.10338.174: 00:16:ea:b2:04:36 DHCP siaddr: 10.10.1.30, giaddr: 0.0.0.038.174: 00:16:ea:b2:04:36 DHCP server id: 1.1.1.1 rcvd server id: 10.10.1.338.179: 00:16:ea:b2:04:36 10.10.1.103 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 57

DHCP Proxy Disabled – Discover/Offer*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)*00:16:ea:b2:04:36 DHCP processing DHCP DISCOVER (1)*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)*00:16:ea:b2:04:36 DHCP processing DHCP OFFER (2)*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

DHCP Proxy Disabled – Request/Ack*00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 316,vlan 0, port 29, encap 0xec03)*00:16:ea:b2:04:36 DHCP processing DHCP REQUEST (3)*00:16:ea:b2:04:36 DHCP op: BOOTREQUEST, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 1024, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP requested ip: 10.10.3.86*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 DHCP successfully bridged packet to DS*00:16:ea:b2:04:36 DHCP received op BOOTREPLY (2) (len 308,vlan 3, port 29, encap 0xec00)*00:16:ea:b2:04:36 DHCP processing DHCP ACK (5)*00:16:ea:b2:04:36 DHCP op: BOOTREPLY, htype: Ethernet, hlen: 6, hops: 0*00:16:ea:b2:04:36 DHCP xid: 0x18a596d9 (413505241), secs: 0, flags: 0*00:16:ea:b2:04:36 DHCP chaddr: 00:16:ea:b2:04:36*00:16:ea:b2:04:36 DHCP ciaddr: 0.0.0.0, yiaddr: 10.10.3.86*00:16:ea:b2:04:36 DHCP siaddr: 0.0.0.0, giaddr: 0.0.0.0*00:16:ea:b2:04:36 DHCP server id: 10.10.3.3 rcvd server id: 10.10.3.3*00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)*00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile*00:16:ea:b2:04:36 DHCP successfully bridged packet to STA*00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1, dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 59

Learning IP without DHCP*Orphan Packet from 10.99.76.147 on mobile*0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)*Installing Orphan Pkt IP address 10.99.76.147 for station*10.99.76.147 DHCP_REQD (7) Change state to RUN (20) last state RUN (20) Client IP can be learned by ways other than DHCP Client sends gratuitous ARP or ARP Request (Static Client) Client sends IP packet (Orphan Packet), we learn IP DS sends packet to client, we learn IP from DS Seen with mobile devices that talk before validating DHCP Up to client to realize their address is not valid for the subnet DHCP Required on WLAN for prevent this BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

Client DHCP - Takeway DHCP_REQD means Learning IP State Only “Required” if enabled on WLC If Proxy is enabled Confirm DHCP Server on Interface (or Wlan) is correct DHCP Server may not respond to WLC Proxy (Firewalls?) If Proxy is disabled, DHCP is similar to wired client Further Troubleshooting Check DHCP Server for what it believes is happening If WLC does not show a BOOTREQUEST, confirm the client request arrives to the WLC and leaves in the configured way If still believed to be on WLC: debug dhcp message enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

Webauth*apfReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 DHCP_REQD (7) Successfully plumbed mobile rule (ACL ID 255)*pemReceiveTask: 00:16:ea:b2:04:36 0.0.0.0 Added NPU entry of type 9, dtlFlags 0x0*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 DHCP received op BOOTREQUEST (1) (len 312,vlan 0, port 29, encap 0xec03)……………………………...*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 DHCP_REQD (7) Change state toWEBAUTH_REQD (8) last state WEBAUTH_REQD (8)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) pemAdvanceState2 5170, Adding TMP rule*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Successfully plumbed mobile rule (ACL ID 255)*DHCP Proxy DTL Recv Task: 00:16:ea:b2:04:36 Assigning Address 10.10.3.86 to mobile*pemReceiveTask: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 2, dtlFlags 0x0*pemReceiveTask: 00:16:ea:b2:04:36 Sent an XID frame*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile*apfReceiveTask: 00:16:ea:b2:04:36 Orphan Packet from 10.10.3.86 on mobile………………………………*emWeb: 00:16:ea:b2:04:36 Username entry (cisco) created for mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14) last stateWEBAUTH_NOL3SEC (14)*emWeb: 00:16:ea:b2:04:36 10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) laststate RUN (20)*emWeb: 00:16:ea:b2:04:36 Session Timeout is 1800 - starting session timer for the mobile*emWeb: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 5063*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Fast Path rule (contd...) 802.1P = 0, DSCP = 0, TokenID =5006 IPv6 Vlan = 3, IPv6 intf id = 8*emWeb: May 17 22:25:16.564: 00:16:ea:b2:04:36 10.10.3.86 RUN (20) Successfully plumbed mobile rule (ACL ID 255)*pemReceiveTask: May 17 22:25:16.578: 00:16:ea:b2:04:36 10.10.3.86 Added NPU entry of type 1,dtlFlags 0x0 BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

Webauth Redirect Webauth Client State =  Client in WEBAUTH_REQD state “WEBAUTH_REQD“  ARP and DNS must be functional ARP and DNS Function Client attempts to browse internet 3-Way Handshake HTTP HTTP GET  WLC “Hijacks” the handshake 200 Response Client redirects to Virtual Interface 3-Way Handshake HTTP(S) GET Certificate negotiation if applicable Webauth Page Displayed  Webauth page is displayed  Client authenticates Successful Authentication Client State = “RUN“ BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 64

3-Way Handshake HTTP GET 200 Response 3-Way Handshake HTTP(S) GET Capture from Wireless Adapter Webauth Page DisplayedWebauth Redirect WLC Responding with SYN, ACK Redirect to Virtual Interface Comes from Here WLC Responding with SYN, ACK Client Is Talking to Webauth…. Address for Client to Redirect to (Virtual IP/Name) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

Webauth - Takeaway If WEBAUTH_REQD, then not authenticated Only traffic allowed is DHCP, ARP, DNS, Pre-Auth ACL, IPv6* If not redirected, can client browse to virtual IP? Cert issue? Consider disabling HTTPS for HTTP webauth Most common scenario involves ARP/DNS failure Must confirm that client actually sends TCP SYN (http) to IP If proven that TCP SYN is sent and WLC does not SYN ACK, then there may be a WLC side problem Debug webauth enable <client ip address> debug client <MAC Address> debug pm ssh-appgw enable debug pm ssh-tcp enableBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

Run State10.10.3.82 DHCP_REQD (7) Change state to RUN (20) last state RUN (20)10.10.3.82 RUN (20) Reached PLUMBFASTPATH: from line 527310.10.3.82 Added NPU entry of type 1, dtlFlags 0x0OR10.10.3.86 WEBAUTH_REQD (8) Change state to WEBAUTH_NOL3SEC (14)10.10.3.86 WEBAUTH_NOL3SEC (14) Change state to RUN (20) last state RUN (20)Session Timeout is 1800 - starting session timer for the mobile10.10.3.86 RUN (20) Reached PLUMBFASTPATH: from line 506310.10.3.86 Added NPU entry of type 1, dtlFlags 0x0 RUN State is the Client Traffic Forwarding State Client is Connected and should be functional BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 69

Deauthenticated Client Idle Timeout Occurs after no traffic received from Client Default Duration is 300 secondsReceived Idle-Timeout from AP 00:26:cb:94:44:c0, slot 0 for STA 00:1e:8c:0f:a4:57apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 4, reasonCode 4Scheduling deletion of Mobile Station: (callerId: 30) in 1 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094) Session Timeout Occurs at scheduled duration (default 1800 seconds) Will force WEBAUTH user to WEBAUTH againapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedScheduling deletion of Mobile Station: (callerId: 45) in 10 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

Deauthenticated Client WLAN Change Modifying a WLAN in anyway Disables and Renables WLANapfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedSent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983)Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094) Manual Deauth From GUI: Remove Client From CLI: config client deauthenticate <mac address>apfMsDeleteByMscb Scheduling mobile for deletion with deleteReason 6, reasonCode 1Scheduling deletion of Mobile Station: (callerId: 30) in 1 secondsapfMsExpireCallback (apf_ms.c:608) Expiring Mobile!apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to DisassociatedSent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller apf_ms.c:5094)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

Deauthenticated Client Authentication Timeout Auth or Key Exchange max-retransmissions reachedRetransmit failure for EAPOL-Key M3 to mobile 00:1e:8c:0f:a4:57, retransmit count 3,mscb deauth count 0Sent Deauthenticate to mobile on BSSID 00:26:cb:94:44:c0 slot 0(caller 1x_ptsm.c:534)  AP Radio Reset (Power/Channel) AP disasassociates clients but WLC does not delete entry Cleaning up state for STA 00:1e:8c:0f:a4:57 due to event for AP 00:26:cb:94:44:c0(0) apfSendDisAssocMsgDebug (apf_80211.c:1855) Changing state for mobile 00:1e:8c:0f:a4:57 on AP 00:26:cb:94:44:c0 from Associated to Disassociated Sent Disassociate to mobile on AP 00:26:cb:94:44:c0-0 (reason 1, caller apf_ms.c:4983) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

Deauthentication - Takeaway Client can be removed for numerous reasons WLAN change, AP change, configured interval Start with Client Debug to see if there is a reason for a client‟s deauthentication Further Troubleshooting Client debug should give some indication of what kind of deauth is happening Packet capture or client logs may be require to see exact reasonBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

Tips and Tricks Collect a client debug for an extended duration Several roams, deauths, failures, etc… Use an enhanced text editor with filter or “find all” I use Notepad++ Find All “Association Received” (will also pull reassociations) “Assoc Resp” “Access-Reject” “timeoutEvt”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

Client Connectivity Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585 Configuration Issues SSID Mismatch Security Mismatch Disabled WLAN Unsupported Data-Rates Disabled Clients Radio Preambles Cisco Features - Issues with Third Party Clients Aironet IE MFPBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

802.11n Speeds Troubleshoot 802.11n Speeds Document ID: 112055 Configuration Issues 11n Support Enabled WMM is Allowed or Required Open or WPA2-AES 5Ghz Channel Width 2.4Ghz does not support 40-Mhz ChannelsBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 81

802.11n A-MPDU/A-MSDU Aggregation methods used could impact interop or performance 802.11n Status: WLC Default 11n Config: A-MPDU Tx: Priority 0............................... Enabled Priority 1............................... Disabled Priority 2............................... Disabled Priority 3............................... Disabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... Disabled A-MSDU Tx: Priority 0............................... Enabled Priority 1............................... Enabled Priority 2............................... Enabled Priority 3............................... Enabled Priority 4............................... Enabled Priority 5............................... Enabled Priority 6............................... Disabled Priority 7............................... DisabledBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

What Is the WLCCA? It is a Post Sales tool Main objective: Save time while analyzing configuration files from WLCs Secondary objective: Carry out RF analysis It is NOT a management or monitoring tool Focused to work off-line to the WLC Not TAC supported Development: wlc-conf-app-dev@cisco.com General internal alias:wlc-conf-app@cisco.com “Pet project”: no official Cisco product. BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 84

Input Needed Complete config output from WLC Show run-config It does not work with old “show running-config” or with TFTP backup, or with show tech The show run-config acts as “snapshot” of current config + RF state Likely best to obtain config from SSH with config paging disable BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 86

Functionality Overview - Checks Audit Checks More than 100 config detail verifications Based on TAC/Escalation cases experience Some obvious, some hard to catch No “change this” messages, some need “contextualization” BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 87

Reducing CCI Turn off excess 2.4 radios. May want to do this gradually, e.g. turn off 20% of radios per attempt After turning off excess radios, could set DCA sensitivity to high Let DCA/power settings settle down overnight. See how things look in the morning Repeat till you see the desired coverage in 2.4GHz BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 91

2.4GHz – Target Coverage Most all 2.4GHz radios are at power 2 - 5 (dont want 7 or 8) In all locations, you have coverage that looks like this (take these as guidelines, not gospel): Hottest channels AP is at least -67dBm Next hottest AP on that channel is at least 19 dB below the hottest Next hottest channels AP is at least -67dBm OK if next hottest AP on that channel is less than 19 dB below the hottest BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 92

5 GHz – Target Coverage  Most all 5GHz radios are at power 1 – 3 (at least 14dBm) Consider the RRM min power setting in 6.0 Consider a radically high tx-power-threshold, like -55 dBm  8 – 12 channels in use (20 seem to be too many for the 792x to scan)  In all locations, seek this: Hottest channels AP is at least -67dBm Next hottest AP on that channel is at least 19 dB below the hottest Next hottest channels AP is at least -67dBm OK if next hottest AP on that channel is less than 19 dB below the hottest BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 93

Additional Troubleshooting Wireshark Tutorial Clean Air SE-Connect / AP Sniffer Mode AP Join RRM Multicast/Broadcast Mobility VoWiFiBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 95

Wireshark Tutorial Newer versions of Wireshark have a feature for “Apply as Column” This will take any decodable parameter and make a columnBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 98

SE-Connect and Sniffer Mode Clean Air APs can be used in lieu of Spectrum Card for Spectrum Analysis AP can be placed in SE-Connect mode for full functionality AP in local mode can be used now for Spectrum Analysis of current channel AP Sniffer Mode can be used in lieu of Wireless Sniffer Packets can be sent from either radio upstream to a packet capture software (Wireshark or Omnipeek for example)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 105

Sniffer Mode AP Omnipeek has a Remote Adapter to capture this data Wireshark, just capture network adapter NOTE: Wireshark does not open the port UDP 5000 PC will send ICMP Unreachables BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 109

Sniffer Mode AP With wireshark, filter !icmp.type == 3 Data (UDP 5000) still not intelligible yet Decode as Airopeek BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 110

AP - Discover Process AP Discovery Req to known and learned WLCs Broadcast Reaches WLCs with MGMT Interface in local subnet of AP Use “ip helper-address <ip>” with “ip forward-protocol udp” Dynamic DNS: cisco-capwap-controller DHCP: Option 43 Configured (nvram) High Availability WLCs – Pri/Sec/Ter/Backup Last WLC All WLCs in same mobility group as last WLC Manual from AP - “capwap ap controller ip address <ip>”BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

AP - Discover Process broadcast X Discover Request sent to all methods the AP knows Discover Response sent from all WLCs that received the Discovery RequestBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

AP – WLC Selection/Join WLCs send Discovery Response back to AP Name, Capacity, AP Count, Master?, AP-MGR, Load per AP- MGR AP selects the single best WLC candidate from High Availability Config: Primary/Secondary/Tertiary/Backup Master Controller Greatest available capacity Ratio of total capacity to available capacity AP sends single Join Request to best candidate WLC responds with Join Response AP joins and receives config (or downloads image if not correct) BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 116

Troubleshooting AP Discovery/Join “Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC)”, Document ID 70333 Make sure time on WLC is accurate! From AP: Debug ip udp Debug capwap client events From WLC Debug mac addr <AP ethernet mac> Debug capwap [event/error/packet] enable Debug pm pki enable BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 117

RRM There are usually only two common scenarios or issues involving RRM APs not changing channel Check if other APs are in each others neighbor list APs not changing power Nearby APs list meets the general rule of RSSI from 3rd closest AP is better than TPC ThresholdBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 119

RRM Show AP Auto-RF (In Run-Config) show ap auto-rf [802.11a/b] <AP Name> Load Information Receive Utilization.. 0 % Rx load to Radio Transmit Utilization.. 2 % Tx load from Radio Channel Utilization.. 12 % % Busy Nearby APs AP 00:16:9c:4b:c4:c0 slot 0.. -28 dBm on 11 (10.10.1.5) AP 00:26:cb:94:44:c0 slot 0.. -32 dBm on 11 (10.10.1.4)BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 121

Broadcast/Multicast AP Multicast Mode – Multicast Address must be unique among WLCs Broadcast Traffic is delivered via the Multicast Mode AP/WLC/Client Subnets must be Multicast enabled For Multicast Mode - Multicast Quick check for Multicast is to confirm that Multicast- Unicast mode works BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 124

Mobility—Layer 3 Layer 3 roaming (a.k.a. anchor/foreign) New WLC does not have an interface on the subnet the client is on New WLC will tell the old WLC to forward all client traffic to the new WLC Asymmetric traffic path established (deprecated) Symmetric traffic pathBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 129

Mobility—Messaging Flow When a client connects to a WLC for the first time, the following happens: New WLC sends MOBILE_ANNOUNCE to all controllers in the mobility group when client connects Old WLC sends HANDOFF_REQUEST New WLC sends HANDOFF_REPLYBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 130

Mobility— L3 Handoff Ignored*mmListen: Mobility packet received from:*mmListen: 10.4.22.55, port 16666*mmListen: type: 3(MobileAnnounce) subtype: 0 version: 1 xid: 783 seq: 1453 len 116 flags 0*mmListen: group id: e42cb3a9 87f62b45 57c0f8a3 92747b23*mmListen: mobile MAC: 00:23:33:41:71:10, IP: 0.0.0.0, instance: 0*mmListen: VLAN IP: 10.4.23.97, netmask: 255.255.255.0*mmListen: Switch IP: 10.4.22.55*mmListen: Handoff Virtual IP Mismatch, Local = 1010101, Request = 1020304 **** Handoff Request Ignored*apfReceiveTask: 10.4.122.127 RUN (20) State Update from Mobility-Complete to Mobility-Incomplete*apfReceiveTask: Mobile 00:23:33:41:71:10 associated with another AP elsewhere, delete mobile*apfReceiveTask: 10.4.122.127 RUN (20) mobility role update request from Local to Handoff Peer = 0.0.0.0, Old Anchor = 10.4.130.70, New Anchor = 0.0.0.0*apfReceiveTask: Clearing Address 10.4.122.127 on mobile*apfReceiveTask: apfMsRunStateDec*apfReceiveTask: 10.4.122.127 RUN (20) Change state to DHCP_REQD (7) last state RUN (20)*apfReceiveTask: apfMmProcessDeleteMobile (apf_mm.c:548) Expiring Mobile!*apfReceiveTask: Mobility Response: IP 0.0.0.0 code Handoff Indication (2), reason Client handoff successful - anchor retained (0), PEM State DHCP_REQD, Role Handoff(6)*apfReceiveTask: apfMsExpireMobileStation (apf_ms.c:5009) Changing state for mobile 00:23:33:41:71:10 on AP 10:8c:cf:eb:69:80 from Associated to Disassociated*apfReceiveTask: Deleting mobile on AP 10:8c:cf:eb:69:80(1)*pemReceiveTask: 0.0.0.0 Removed NPU entry. BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 134

Mobility Group vs. Mobility Domain Mobility Group - WLCs with the same group name L2/L3 Handoff Auto Anchoring Fast Secure Roaming APs get all of these as a Discover candidate Mobility Domain - WLCs in the mobility list L2/L3 Handoff Auto AnchoringBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 135

Mobility Data/Control Path Sent between all WLCs, by member with lowest MAC Control Path = UDP 16666 (30 Seconds) Data Path = EoIP Protocol 97 (10 Seconds) debug mobility keep-alive enable <IP Address>BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 136

VoWiFi Wireless IP Phone Deployment Guide http://www.cisco.com/en/US/docs/voice_ip_comm/cuipph/79 25g/7_0/english/deployment/guide/7925dply.pdf Best Practices -67 dBm signal with 20-30% cell overlap 802.11A CCKM for Fastest Roaming Avoid designs where AP is seen at superb signal, but drops off instantlyBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 138

VoWiFi - Troubleshooting Must know if problem occurs during roaming events or when no association change takes place If no change in connection Interference Coverage loss with no other candidate End to End QOS missing/problem If during roaming event How long did the roam take? Does the client associate to another AP again within seconds? Does the client associate to the same AP again? Is the phone roaming to the designed next candidate?BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 139

VoWiFi - Troubleshooting Define a reproducible area where you believe you have perfect voice coverage but have problems Place phone in Neighbor List Mode (On a call) Real Time current AP RSSI and candidate list Confirm AP as next best candidate is realistically a good candidate Confirm devices roams to correct candidate where the intended design specifies Watch out for sudden drops in coverageBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 140

VoWiFi - Debugs Phone can Trace (debug) to file or syslog Recommend USB Connection and SYSLOG Configured via GUI Enable Debug level for Kernel, WLAN MGR, WLAN Driver WLC Debugs Debug client <mac> Debug cac all enable Wireless Packet CapturesBRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 141

SummaryClient WLC - show run-config, debug client <mac>, debug dhcp message enable, debug dot1x <?> enable, debug aaa <?> enable, AP - Show tech, show controller D<0/1> Data - Driver/Supplicant Logs, Wireless Capture, AAA Logs, DHCP LogsWebauth WLC - (Client debugs), debug webauth enable <IP>, debug pm ssh-appgw enable, debug pm ssh-tcp enable Client - local captureMobility WLC - debug mobility handoff enable, debug mobility keepalive enable <IP> Data - Wired captureAP Join WLC - debug capwap [events/error/packet] enable AP - debug capwap client events, debug ip udp Data - Wired captureRRM WLC - show run-config, debug airewave-director <?> AP - debug capwap rm measurements, debug capwap rm rogueMulticast/Broadcast AP - show capwap mcast, show capwap mcast mgid all Data - Infrastructure ConfigurationVoice WLC - (Client debugs), debug cac all enable Data – Wireless capture, Phone traces BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 143

Summary Links: Understanding Debug Client on Wireless LAN Controllers (WLCs) Document ID: 100260 Unified Wireless Network: Troubleshoot Client Issues Document ID: 107585 Troubleshoot 802.11n Speeds Document ID: 112055 Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller Document ID: 99948BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 144

Complete Your OnlineSession Evaluation Receive 25 Cisco Preferred Access points for each session evaluation you complete. Give us your feedback and you could win fabulous prizes. Points are calculated on a daily basis. Winners will be notified by email after July 22nd. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don‟t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.BRKEWN-3011 © 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public 145

Troubleshooting Wireless LANs with Centralized Controllers

by Cisco Wireless

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 14

Statistics

Troubleshooting Wireless LANs with Centralized Controllers Presentation Transcript