1. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
1 #airheadsconf
#airheadsconf
Top 10 Tips from Aruba TAC
Ken Peredia, Jeffrey Goff
November 2013
2. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
2 #airheadsconf
1. Debugging Client Disconnects
2. RF Dashboard
3. AMON and Airwave
4. Client Quirks
5. AirGroup Config Notes
6. Voice Over WiFI Tweaking
7. High CPU On Controller
8. Aruba Utilities for Android
9. AirRecorder
10.11ac sniffing
Agenda
3. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
3 #airheadsconf
#airheadsconf
3
Debugging Client Disconnects
4. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
4 #airheadsconf
⢠Problem could be in any number of places
â AP to controller stability
⢠BSS uptime, AP uptime
â Feature related
⢠ClientMatch, bandsteering, SLB, dot11k
â Auth/Association related
⢠Use the tracebuf and new logging to narrow it down
â RF related
⢠ARM changing channels
⢠Interference
Debugging Client Disconnects
5. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
5 #airheadsconf
⢠AP to Controller
â If the AP is not stable, clients will experience disconnects
â Common causes
⢠Network issues between AP and controller
⢠Controller load issues
â Debug these using
⢠âshow ap debug countersâ â look for APs with high reboot/straps
⢠âshow ap bss-tableâ â look for APs with low tot-t (bss uptime)
⢠Collect âshow ap tech-support ap-name <the_ap>â
⢠Collect âtar logs + tech supportâ
Debugging Client Disconnects
6. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
6 #airheadsconf
⢠AP to Controller
â In regards to network issues, common problems include
⢠ARP issues on AP vlan
⢠DHCP issues on AP vlan, including duplicate IP
⢠Packet loss in intermediate switch/router (especially RAPs)
⢠IDS/Firewall device interfering with connectivity
â If you are seeing AP rebootstraps, start to investigate the
quality of the link between controller and AP
⢠Setup a long term ping with large packet size (say 1400 bytes)
and monitor
⢠âshow ap debug system-status ap-name <the_ap>â will tell you
something about why the AP is unstable
Debugging Client Disconnects
7. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
7 #airheadsconf
⢠ClientMatch
â Continuously scans the wireless environment and shares
information about the clients amongst the APs
â Based on the dynamic data obtained, constantly optimizes
the client association by steering clients to the most suitable
AP
â In some cases, ClientMatch may be more aggressive than
needed for certain environments and can be slightly tuned to
be less active
Debugging Client Disconnects
12. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
12 #airheadsconf
⢠View which clients were put in the unsupported
list due to many failed steering attempts
Debugging Client Disconnects
13. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
13 #airheadsconf
⢠To make ClientMatch less aggressive with load
balancing the following configuration can be
used
Debugging Client Disconnects
14. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
14 #airheadsconf
⢠Per user auth logging is new in ArubaOS 6.3
⢠Simplified âper user auth tracebufâ
â No longer need to enable
âlogging level debugging user-debug <user mac address>â
â This information is now saved per-user when CLI command
âaaa logâ is enabled in configuration terminal.
â To view the log
⢠âshow user-table mac <mac-address> logâ
⢠or âshow user-table ip <ip-address> logâ
Debugging Client Disconnects
15. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
15 #airheadsconf
(Aruba7210_Sanya) #show user-table mac 68:a8:6d:32:ef:a2 log
1: At Tue Nov 12 05:20:29: [L] Type eap-req <- id 8 len 6, bssid 6c:f3:7f:f0:b7:50
2: At Tue Nov 12 05:20:29: [L] Type eap-resp -> id 8 len 172, bssid 6c:f3:7f:f0:b7:50
3: At Tue Nov 12 05:20:29: [L] Type rad-req -> id 9 len 387, bssid 6c:f3:7f:f0:b7:50
4: At Tue Nov 12 05:20:29: [L] Type rad-resp <- id 9 len 139, bssid 6c:f3:7f:f0:b7:50
5: At Tue Nov 12 05:20:29: [L] Type eap-req <- id 9 len 69, bssid 6c:f3:7f:f0:b7:50
6: At Tue Nov 12 05:20:29: [L] Type eap-resp -> id 9 len 6, bssid 6c:f3:7f:f0:b7:50
7: At Tue Nov 12 05:20:29: [L] Type rad-req -> id 10 len 221, bssid 6c:f3:7f:f0:b7:50
8: At Tue Nov 12 05:20:29: [L] Type rad-accept <- id 10 len 246, bssid 6c:f3:7f:f0:b7:50
9: At Tue Nov 12 05:20:29: [L] Type eap-success <- id 9 len 4, bssid 6c:f3:7f:f0:b7:50
10: At Tue Nov 12 05:20:29: [L] Type wpa2-key1 <- id 0 len 117, bssid 6c:f3:7f:f0:b7:50
11: At Tue Nov 12 05:20:29: [L] Type wpa2-key2 -> id 0 len 135, bssid 6c:f3:7f:f0:b7:50
12: At Tue Nov 12 05:20:29: [L] Type wpa2-key3 <- id 0 len 151, bssid 6c:f3:7f:f0:b7:50
13: At Tue Nov 12 05:20:29: [L] Type wpa2-key4 -> id 0 len 95, bssid 6c:f3:7f:f0:b7:50
Debugging Client Disconnects
16. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
16 #airheadsconf
⢠Per client association history also new in 6.3
⢠Easily view the activity of a clientâs connection
â why it was de-authenticated by the system
â what alerts/errors were encountered by the client
â What APs has it been associated to, and for how long
Debugging Client Disconnects
(Aruba7210_Sanya) #show ap client trail-info 68:a8:6d:1f:5a:9a
Client Trail Info
-----------------
MAC BSSID ESSID AP-name VLAN Deauth Reason Alert
--- ----- ----- ------- ---- ------------- -----
68:a8:6d:1f:5a:9a 6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 2 STA has roamed to another AP STA has roamed to another AP
17. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
17 #airheadsconf
Deauth Reason
-------------
Reason Timestamp
------ ---------
STA has roamed to another AP Nov 11 20:14:50
Client Match Nov 11 20:14:36
Rx Data Bytes 108 Mbps+ (Mon) Nov 11 19:29:29
STA has roamed to another AP Nov 11 18:44:09
APAE Disconnect Nov 11 18:28:47
STA has roamed to another AP Nov 11 18:28:14
Client Match Nov 11 18:27:31
Ptk Challenge Failed Nov 11 18:26:57
STA has roamed to another AP Nov 11 18:24:41
STA has roamed to another AP Nov 11 18:24:06
Num Deauths:10
Debugging Client Disconnects
18. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
18 #airheadsconf
Mobility Trail
--------------
BSSID ESSID AP-name Timestamp
----- ----- ------- ---------
6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 12 05:20:34
6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 20:14:50
6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 20:14:50
6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 20:14:39
6c:f3:7f:f0:c5:90 Diamond5 AP225-4 Nov 11 20:14:36
6c:f3:7f:f0:c5:90 Diamond5 AP225-4 Nov 11 20:13:02
6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 19:29:29
6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 18:44:09
6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 18:44:09
6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 18:28:50
Num Mobility Trails:10
Debugging Client Disconnects
19. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
19 #airheadsconf
⢠RF Related
â If the 2.4GHz is particularly noisy the default settings of ARM
may be causing too many channel changes
⢠âshow ap arm history ap-name <the_ap>â to review
â Need to check first for legitimate sources of interference
⢠Use spectrum analysis
⢠Use AMON / RF Dashboard
â But sometimes spectrum looks ok and the issues are coming
more from co-channel interference and hidden nodes etc.
Debugging Client Disconnects
20. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
20 #airheadsconf
⢠RF Related
â If it is observed that the 2.4GHz radio is changing channels
excessively due to noise or error
⢠Create a new ARM profile for 2.4GHz
⢠Attach it to the dot11g radio profile
⢠Increase the noise and error thresholds
â Increase gradually and monitor
â Donât increase too much
⢠if there is real interference, AP will take longer to react.
Debugging Client Disconnects
Error Rate Threshold 50 %
Error Rate Wait Time 30 sec
Noise Threshold 75 -dBm
Noise Wait Time 120 sec
21. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
21 #airheadsconf
#airheadsconf
21
Controller RF Dashboard
22. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
22 #airheadsconf
RF Dashboard
⢠Use the dashboard to quickly find issues
â Find APs with interference
â Find clients with RF problems
â Get a summary of potential issues
â View short term trending information
⢠Donât have to check each AP from CLI
â show ap debug radio-statsâŚâŚ hard to use and interpret
24. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
24 #airheadsconf
⢠Everything is clickable â including the graphs
RF Dashboard
25. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
25 #airheadsconf
⢠SNR
â Low SNR may mean client is far away from AP or perhaps
not enough coverage.
â Difficult to interpret in a hotspot environment
â In a high density environment where the users are distributed
around the APs, there should be very few red results
⢠May also be due to sticky clients, ClientMatch can help
RF Dashboard
26. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
26 #airheadsconf
⢠Noise Floor
â in-band non 802.11 interference
â These APs should always be investigated
â Common causes
⢠Video bridges, bluetooth, DECT, microwave ovens etc.
RF Dashboard
27. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
27 #airheadsconf
⢠Channel Busy
â This is an important item â it needs to be considered with
interference
â Channel busy = tx time + rx time + interference
â Possible causes for high values
⢠Genuine traffic usage
⢠Over density / too much 802.11 wifi / beacons not optimised
⢠Interference (in band and out of band)
RF Dashboard
28. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
28 #airheadsconf
⢠Channel Busy
â For 2.4GHz this will often be high
⢠Especially in hotspots
⢠Values of 30-40% are common
⢠Influenced by g-tx-rate/g-basic-rate/g-beacon-rate
â For 5Ghz, this is a very good indicator of load on the AP
RF Dashboard
29. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
29 #airheadsconf
⢠Interference
â This is the non 802.11 component of channel busy
â Interference value needs to be as low as possible
⢠Typical value is 0 ~ 3%
â Possible causes of high interference numbers
⢠Out-of-band (WiMax, cellular DAS antennas, DECT 1.8GHz)
⢠In-band from non-802.11 devices (video, microwave ovens,
2.4GHz DECT phones etc)
RF Dashboard
31. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
31 #airheadsconf
AMON and Airwave
⢠What is AMON
â A protocol used to stream information from the controller to
Airwave
⢠Also used by Arubaâs Analytic Location Engine (ALE)
⢠AMON is built on top of PAPI â the AP <-> controller protocol
⢠Aruba does not provide any public documentation or description about
the protocol definition.
â AMON sends radio stats to Airwave (7.5.x and higher)
⢠Controller dashboard has only 15 minutes worth of data
⢠Airwave plots longer term ď trending
⢠Use alert triggers to proactively flag when an AP is experiencing
interference or high utilisation etc.
32. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
32 #airheadsconf
AMON and Airwave
⢠Enable sending of AMON to Airwave on controller
â webUI Configuration ď Management ď General
⢠Also used by Arubaâs Analytic Location Engine (ALE)
⢠In Airwave ensure AMON is enabled
⢠AMP Setup ď General ď Additional AMP Services
33. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
33 #airheadsconf
⢠After a short while AMON radio stats will appear
â Access the stats via the AP (i.e. APs/Devices ď Monitor)
â Select the radio by clicking on the name
â Then select âChannel Utilizationâ
from the drop down
â If Channel Utilization is not an option
then AMON is not enabled
AMON and Airwave
34. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
34 #airheadsconf
⢠Same data as being shown on the controller
â Remember: channel busy == rx + tx + interference
â Can hover over a point to get the values
â Much more history available
via âTime Rangeâ
â Note that noise floor
is accessible via another
dropdown option
AMON and Airwave
35. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
35 #airheadsconf
⢠What can we determine from this ?
â Perhaps an interference source that comes and goes
â Or one that suddenly appears
⢠Someone just bought a 2.4GHz non 802.11 wireless device to the
office â interference comes and goes with office hours
⢠Wireless video camera installed nearby.
â Alternatively, letâs assume no interference
⢠Utilisation versus time â does this AP have a problem on some day
of the week?
⢠Is this AP unexpectedly underutilized ? (maybe on a rare channel)
⢠Constant ârx == busyâ at a high value may indicate that beacon rate
optimization on the APs would be beneficial
AMON and Airwave
36. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
36 #airheadsconf
⢠Can we be more proactive?
â The statistics that appear in the radio monitor page can be used
within alert triggers
â Actually, use triggers to monitor other critical resources like disk
space on the Airwave itself, or AP CPU etc.
⢠Use short term and long term durations
â Cover immediate problems with short durations
⢠âmore than 20% interference over 10 minsâ ď critical impact
â Keep an eye on the growth with long durations
⢠âchannel busy > 80% over 6 hoursâ ď Warning, capacity running out
AMON and Airwave
37. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
37 #airheadsconf
⢠Example
â To have 20% interference
on 5GHz is very unusual
â It would be symptomatic
of a device like a motion
detector
⢠Use radio type to be more/less aggressive by band
â Determine the baseline in each band and adjust accordingly
AMON and Airwave
39. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
39 #airheadsconf
⢠MacOS 802.1X Roaming Latency
â MacOS 10.8.4 or newer may intermittently experience 5 â 20
seconds latency in completing 802.1X authentication when
roaming between APs.
â This latency is usually seen during the middle of EAP
negotiation.
â This problem is only seen on a 802.1X SSID. Itâs not seen on
a PSK SSID.
Client Quirks
40. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
40 #airheadsconf
⢠MacOS 802.1X Roaming Latency
â The following is an example 10 sec delay for the client's eap-response
in "show auth-tracebuf" output:
Client Quirks
Sep 13 18:03:17 rad-req -> 10:40:f3:83:fe:6c 00:1a:1e:55:53:50/RADIUS_Server_1 65518 222
Sep 13 18:03:17 rad-resp <- 10:40:f3:83:fe:6c 00:1a:1e:55:53:50/RADIUS_Server_1 65518 287
Sep 13 18:03:17 eap-req <- 10:40:f3:83:fe:6c 00:1a:1e:55:53:50 7 203
Sep 13 18:03:27 eap-resp -> 10:40:f3:83:fe:6c 00:1a:1e:55:53:50 7 1276
Sep 13 18:03:27 rad-req -> 10:40:f3:83:fe:6c 00:1a:1e:55:53:50/RADIUS_Server_1 13 1502
Sep 13 18:03:27 rad-resp <- 10:40:f3:83:fe:6c 00:1a:1e:55:53:50/RADIUS_Server_1 13 90
Sep 13 18:03:27 eap-req <- 10:40:f3:83:fe:6c 00:1a:1e:55:53:50 8 6
Sep 13 18:03:27 eap-resp -> 10:40:f3:83:fe:6c 00:1a:1e:55:53:50
41. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
41 #airheadsconf
⢠MacOS 802.1X Roaming Latency
â Available workaround seems to greatly improve this
authentication latency
â Trust the 802.1X RADIUS server cert as root in the System
keychain while setting âAlways Trustâ for SSL, EAP, and X.509
basic policy.
Client Quirks
44. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
44 #airheadsconf
⢠MacOS Ping Latency
â In viewing continuous pings on a Macbook Pro/Air, you might
see a large variation of ping latency from 2ms to 100+ms
â while other devices on the same AP experience only 1- 4ms
latency
Client Quirks
45. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
45 #airheadsconf
⢠MacOS Ping Latency
Client Quirks
46. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
46 #airheadsconf
⢠MacOS Ping Latency
â The issue is due to the aggressive power save behavior seen
in newer MacOS (10.8.4+).
â Sometimes the Macbooks will sleep 1ms after a ping request
is sent and may not wake up until 90ms+ later.
â Therefore the Aruba AP will buffer the ping response until the
Macbook informs the AP it is awake which sometimes results
in large latency as shown in the following sniffer trace.
Client Quirks
48. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
48 #airheadsconf
⢠Dot11k
â Some interop issues seen â be careful if you enable this
â Example: 4.3.x Android device on wpa2-aes SSID
⢠Client associates, gets IP address but locks up shortly thereafter
⢠disabling dot11k inside the âwlan dot11k-profile defaultâ resolved
⢠Bandsteering vs Clients
â Some clients donât react well to bandsteering
â A common one is Android, which can report âauthentication
error occurredâ
â This error is thrown by Android when the Aruba AP sends a
âresource constrainedâ message during 802.11 auth
Client Quirks
50. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
50 #airheadsconf
⢠Common AirGroup configuration issues
⢠AirGroup domain controller IP must be switch IP
â Symptom: active domain is stuck in âexcludedâ state
â make sure that the specified controller IP is a switch IP and
not an interface IP
AirGroup
(sg-3200) # show airgroup active-domains
AirGroup Active-Domains
-----------------------
Domain Name Status
----------- ------
SomeDomain Excluded
Num active-domains:1
51. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
51 #airheadsconf
⢠Wireless based AirGroup âserverâ (i.e. Apple Tv)
â Symptom: AppleTv connected wirelessly keeps disappearing
from list of Airplay servers on the client device
â Controller does not by default send refresh requests to
wireless side
â We have a configuration option to force this behavior
âairgroup active-wireless-discovery enableâ
AirGroup
52. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
52 #airheadsconf
⢠Apple TV and EAP-PEAP
â Symptom: AppleTv fails to connect to PEAP SSID
â AppleTv has no real time clock
â Connect fails because the PEAP cert fails date validation
check â because the date/time is wrong
â Requires NTP to set the time â but needs access to network
to use NTP
â Found to be resolved in iOS 7
⢠exact method of resolution by Apple is unknown
⢠Verified on hardware v2 and v3
AirGroup
53. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
53 #airheadsconf
⢠NAT
â Airgroup is not supported with NAT
â Do not configure ip nat inside or use a NAT ACL on any
client that needs Airgroup.
⢠Disallow VLAN
â Can be configured globally or on a per service basis
â Disallowing a VLAN means that mDNS servers in that VLAN
cannot be discovered.
â However⌠mDNS clients in the disallowed VLAN can still be
discovered by servers in other VLANs
AirGroup
54. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
54 #airheadsconf
#airheadsconf
54
Voice optimizations
55. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
55 #airheadsconf
ArubaOS â VoWIFI
⢠Deployment tips
â Always try to use 5GHz for voice
⢠If AP coverage allows and devices are capable
⢠But be aware of differences in client ability to support all channels in
your regulatory domain.
â Make sure you have adequate coverage, rule of thumb is two cell-
overlap of -67dB
56. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
56 #airheadsconf
ArubaOS â VoWIFI
⢠Ensure QOS vs. DSCP is matched
â on both the wired and wireless infrastructure
â SSID configuration and user-role
â Real voice applications should be correctly identified by the
controller ALGs (sip, h323, SCCP, etc.)
⢠But something like Skype will not be â itâs just âdataâ
⢠Assumptions
â Dual use SSID (voice and data)
⢠More aggressive optimizations can be used on a voice-only SSID
â Office style coverage/density
â AP signal strength of -65 dBm at the client
â Client SNR 25 or better at AP (show ap debug client-table)
59. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
59 #airheadsconf
ArubaOS â VoWIFI
⢠Suggested ARM configuration
â Donât have more than 6 dB difference between âmin-tx-powerâ and
âmax-tx-powerâ to provide a more predictable roaming behavior
⢠Avoids large power differences between neighboring APs
â âclient-awareâ and âvoip-aware-scanâ *must* be enabled
⢠âps-aware-scanâ should not be enabled
â Donât raise âmin-tx-powerâ too high as it can cause sticky clients
⢠Typical high density office might use a âmin-tx-powerâ of 9 dBm
⢠Smart phone type devices have quite low roaming thresholds, often as
low as -75dBm. Setting tx power too high can cause client->AP to fail
well before device tries to roam.
60. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
60 #airheadsconf
#airheadsconf
60
Common high CPU issues
61. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
61 #airheadsconf
⢠The usual suspectsâŚ
â WMS (the AirMonitor/IDS function)
â HTTPd (the captive portal webserver)
⢠WMS
â Responsible for the collection of statistics about valid,
interfering and rogue devices (APs and clients)
â In hotspot deployments WMS may be tracking upwards of
50,000 devices
â Airports and large campuses also have large WMS
databases
High CPU on controller
62. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
62 #airheadsconf
â As the WMS database grows so do the related SNMP tables
â Devices like Airwave are polling these tables periodically
which causes CPU spikes
This kind of WMS database would return over 250,000 results in SNMP
â Symptoms of high CPU load in WMS include
⢠sluggish webUI, SNMP timeouts, missing data in Airwave graphs
⢠Can also impact captive portal as it checks for CPU load
High CPU on controller
(aruba) #show wms counters | include table
AP G-table 19841
AP A-table 40
STA G-table 28365
STA A-table 1231
63. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
63 #airheadsconf
â Unstable APs can also contribute to the WMS CPU load.
⢠âshow ap debug countersâ
⢠check for high reboot/rebootstraps
⢠âshow ap bss-tableâ
⢠check for low tot-t (uptime) of the BSSID
⢠Especially for RAPs deployments where the controller to AP
transits multiple networks beyond your control.
High CPU on controller
AP Counters
-----------
Name Group IP Address Configs Sent Configs Acked Bootstraps (Total) Reboots
---- ----- ---------- ------------ ------------- ------------------ -------
ap100 apgroup 1.1.2.2 4 4 1 (1 ) 0
ap101 apgroup 1.1.2.3 4 4 1 (1 ) 0
ap102 apgroup 1.1.2.4 8 8 120 (120 ) 0
ap103 apgroup 1.1.2.5 8 8 3 (3 ) 0
ap104 apgroup 1.1.2.9 8 8 3 (3 ) 0
64. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
64 #airheadsconf
â What to do ?
1. WMS Offload to Airwave
⢠Moves large portions of WMS processing to Airwave
⢠APs on local controllers now send direct to Airwave
⢠But⌠increases WMS CPU load on local controllers
2. Need to reduce SNMP polling of certain MIBs
3. Or - reduce the amount of data being held in WMS
â If not able to use WMS offload, two basic levels of
optimization can be done depending on the features being
used.
â If not able to perform #2 or #3 then WMS offload is the only
choice
High CPU on controller
65. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
65 #airheadsconf
1. If no IDS/WIPS functionality is being used, the following config
will prevent interfering clients from being entered into the WMS
database.
âids general-profile default wms-client-monitoring noneâ
⢠A warning is printed to remind you of the functionality that is
impacted by this command.
High CPU on controller
(sg-620) (config) #ids general-profile default wms-client-monitoring none
Warning: Enabling this option will cause the following features to not work as intended: Protect Valid Station,
Detect Valid Client Misassociation, Detect Unencrypted Valid Clients, Tarpit Containment of Non-valid
clients, Detect Bad WEP, Detect Disconnect Station Attack, Detect Power Save DoS Attack, Detect Block
ACK DoS, Detect TKIP replay Attack, Detect ChopChop Attack, Detect Omerta Attack, Detect FATA-Jack
Attack, Detect Overflow EAPOL Key, Detect Frame Rate Anomalies.
(sg-620) (config) #
66. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
66 #airheadsconf
2. If you donât require or care about rogue AP information in
Airwave, then you can disable âRogue AP and Device Location
data Polling Periodâ
â Note this will impact any location services or RAPIDS, and rogue list will no
longer be populated in Airwave
High CPU on controller
67. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
67 #airheadsconf
⢠WMS
â What if all that is done but problem not resolved?
â Contact support for assistance
⢠there are a few more settings that can be changed
⢠ButâŚthese are very specific to the deployment and need to be
carefully chosen.
High CPU on controller
68. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
68 #airheadsconf
⢠HTTPD
â High CPU with internal captive portal + large certificates
⢠Large certs are handled in software for all controllers before 72xx
⢠Large being 2048 or 4096
⢠Very hard to get a public signed 1024 bit cert anymore
â Many smart devices constantly trying to hit https locations
once associated
⢠This is becoming more common over last 1-2 years
⢠Even if they never authenticate with the captive portal
⢠iTunes, Play Store, Windows Update, antivirus updates, CDNs
â Also caused by image heavy internal captive portal page
High CPU on controller
69. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
69 #airheadsconf
⢠Solutions
â Move to an http based captive portal â but keep https for the
username and password
⢠Turn on allow http in captive portal profile
⢠Modify the HTML code of the captive portal page to use https for
the form action:
High CPU on controller
<div id="logins">
<div id="loginbox" style="">
<h1><span>Wi-Fi Login</span></h1>
<form action="/cgi-bin/login" id="regform" method="post" autocomplete="off" title="Login">
<div id="usernamebox">
<label for="user" accesskey="u">Username</label>
<form action="https://securelogin.arubanetworks.com/auth/index.html/u" id="regform" method="post" title="Login">
If not using the Aruba default certificate, make sure to replace securelogin.arubanetworks.com with the CN of the new cert
70. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
70 #airheadsconf
⢠Solutions
â Changing to http with https form action may not totally
recover the CPUâŚ
â Need to deal with the high rate of incoming https requests for
clients that have not authenticated
â CPU load due to the captive portal redirect can also impact
external captive portal
High CPU on controller
71. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
71 #airheadsconf
⢠Solutions
â One option is to prevent any https from redirecting
⢠Remove the âuser any svc-https dst-nat 8081â rule from the
captive portal ACL
⢠This has a downside â clients with https home pages will fail to
redirect to captive portal
High CPU on controller
Priority Source Destination Service Action TimeRange
-------- ------ ----------- ------- ------ ---------
1 user controller svc-https dst-nat 8081
2 user any svc-http dst-nat 8080
3 user any svc-https dst-nat 8081
4 user any svc-http-proxy1 dst-nat 8088
5 user any svc-http-proxy2 dst-nat 8088
6 user any svc-http-proxy3 dst-nat 8088
72. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
72 #airheadsconf
⢠Solutions
â If removing the https redirect is not possible, another option
is to create a âblacklistâ of sites
â Use the datapath session table to examine where clients are
connecting to
â Packet capture on an external captive portal and examine
the URLs that the clients were trying to connect to
High CPU on controller
https-drop-list
---------------
Position Type IP addr Mask-Len/Range
-------- ---- ------- --------------
1 name 0.0.0.1 *.facebook.com
2 name 0.0.0.3 *.edgekey.net
3 name 0.0.0.4 *.1e100.net
4 name 0.0.0.5 *.akamaitechnologies.com
5 name 0.0.0.2 *.amazonaws.com
6 name 0.0.0.6 *.tfbnw.net
73. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
73 #airheadsconf
⢠Please note that the Aruba controller factory
certificate is due to expire on 21 November, 2013
⢠A new cert is loaded into 6.1.3.9, 5.0.4.13, 7.2.3.1
and higher
⢠There is a support advisory about this on the
Aruba support portal and Airheads Social
â Contact your partner/reseller to get a copy if you cannot
access the portal and you are on an affected version
By the wayâŚ
74. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
74 #airheadsconf
#airheadsconf
74
Android Aruba Utilities
75. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
75 #airheadsconf
⢠Free app on Play Store
⢠Requires Android 3.0 or better
â There is a stripped down version that runs on 2.2 or better
⢠Features include
â Built in performance tester (aperf)
â Interacts with Airwave and the controller
â Logs signal info to text file
â Site survey tools (heatmap etc.)
â RSSI graphs
â Handover tracking
Aruba Utilities
80. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
80 #airheadsconf
⢠AirRecorder is a Java based tool developed by
Aruba Customer Engineering that will
periodically run several common CLI commands
for checking controller, AP, and wireless device
health for troubleshooting and monitoring.
â It can run on Windows, Linux/Unix, and MacOS platforms.
â It supports ArubaOS, Instant VC (IAP), and MeshOS (MSR)
CLI commands.
â All CLI command output is saved to auto-rotating log files
AirRecorder
81. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
81 #airheadsconf
⢠By default, a new log file is created every 100 MB
but can be modified with option â--max-log-sizeâ.
⢠AirRecorder is very flexible in gathering data
â It can automatically retrieve data from all APs and devices on
a Controller via pre-defined variables
â It can manually retrieve data from specific APs and devices
on a Controller
â It can be configured to run different CLI commands at
different intervals
AirRecorder
82. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
82 #airheadsconf
⢠Example CLI commands and intervals that can
exist in an Airrecorder script
10m,show ap active
5m,show ap association
5m,show user-table
5m,show ap arm rf-summary ap-name %{ap:name}
30m,show ap arm state ap-name %{ap:name}
30m,show ap bss-table
5m,show ap debug client-table ap-name %{ap:name}
5m,show ap debug client-stats %{user:mac} advanced
5m,show ap client trail-info %{user:mac}
5m,show ap arm client-match history client-mac %{user:mac}
AirRecorder
83. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
83 #airheadsconf
⢠Example running of AirRecorder on Win7
C:java -jar AirRecorder-1.2.8-release.jar 10.8.7.63
AirRecorder (c)2011-2012 Thomas Bastian, Aruba Networks
Username: admin
Password:
Enable password:
No enable password set, assuming enable mode when connected
A new log file will be created every: 100 Mbytes
Recording to file: air-recorder-10.8.7.63-20131112-091902-00.log
Connecting to controller with hostname: 10.8.7.63, protocol: ssh, port: 22, username: admin
Processing command: [show ap active] at schedule specification of 10m
Processing command: [show ap association] at schedule specification of 5m
Processing command: [show user-table] at schedule specification of 5m
Processing command: [show ap arm rf-summary ap-name %{ap:name}] at schedule specification of 5m
Processing command: [show ap arm state ap-name %{ap:name}] at schedule specification of 30m
Processing command: [show ap bss-table] at schedule specification of 30m
Processing command: [show ap debug radio-stats ap-name %{ap:name} radio 0 advanced] at schedule
specification of 5m
Commands (cur/max): 4/4, Rx Rate: 161 kbps, File size: 1 MB
AirRecorder
84. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
84 #airheadsconf
⢠Example output of Airrecorder
.....
/////
///// Message: RESULT
///// Status: 1937
///// LocalBeginTime: 1384276748945 (2013-11-12T09:19:08.945-0800)
///// LocalEndTime: 1384276749360 (2013-11-12T09:19:09.360-0800)
///// QueryTag: airrecorder.command=10m,show ap active
///// Command: show ap active
///// Section: Stdout
Active AP Table
---------------
Name Group IP Address 11g Clients 11g Ch/EIRP/MaxEIRP 11a Clients 11a
Ch/EIRP/MaxEIRP AP Type Flags Uptime Outer IP
---- ----- ---------- ----------- ------------------- ----------- -----------
-------- ------- ----- ------ --------
AP225-3 Renaissance_APs 10.8.7.115 0 AP:HT:6/12/21 0
225 Ada 15h:21m:59s N/A
AP225-6 Renaissance_APs 10.8.7.116 0 AP:HT:1/12/21 8
AP:VHT:149+/15/22 225 Ada 15h:22m:13s N/A
AirRecorder
85. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
85 #airheadsconf
#airheadsconf
85
802.11ac Capture Tools
86. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
86 #airheadsconf
⢠Limited options for Windows users
â Linksys AE6000 â but single stream
â No multi stream options at this point
⢠Aruba AP-22x packet capture
â Can decode as âPEEKREMOTEâ but the plugin is a bit buggy
⢠Wildpackets Omnipeek
â beta driver for Linksys AE6000 (1 stream 11ac) NIC available
Known 802.11ac Capture Tools
87. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
87 #airheadsconf
⢠New Macbook Air (2 stream 11ac) or Macbook Pro (3
stream 11ac)
â Enabled through Wireless Diagnosticsď Utilitiesď Frame Capture
â Wireless Diagnostics can be found by clicking on
âOptionâ ď âWi-Fi iconâ ď Wireless Diagnostics
â Saved to a .wcap file that can be opened by Wireshark
Known 802.11ac Capture Tools
89. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
89 #airheadsconf
In conclusion
⢠support@arubanetworks.com
â One email address for all products
â You can always request your ticket to be moved to another
time-zone
â Upload files directly to the case via the support portal
â Avoid unicasting emails/attachments to support staff
⢠Using reply-all will get more eyes on your issue
⢠Always call support for urgent issues
⢠Please exercise caution when making changes
â Always move your backups off the controllers/servers
â When tweaking, incrementally add changes
90. CONFIDENTIAL
Š Copyright 2013. Aruba Networks, Inc.
All rights reserved
90 #airheadsconf
Takeaways
TAC Quick Reference Guide
â https://support.arubanetworks.com/DOCUMENTATION/tabid/77/DMXModule/512/Comma
nd/Core_Download/Default.aspx?EntryId=1371
Validated Reference Designs (VRD)
â http://www.arubanetworks.com/technology/reference-design-guides/
Airheads Social
â http://community.arubanetworks.com/
Aruba Knowledge Base
â https://kb.arubanetworks.com/
Raise a ticket for any product, RMA, anything !
â support@arubanetworks.com
Requests for Enhancements (RFE)
â Please discuss with your SE/Sales team
Outdoor planner tool
â https://outdoorplanner.arubanetworks.com/