Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Breakout - Airheads Macau 2013 - Top 10 Tips from Aruba TAC

23,162
views

Published on

Breakout - Airheads Macau 2013 - Top 10 Tips from Aruba TAC …

Breakout - Airheads Macau 2013 - Top 10 Tips from Aruba TAC
Topics Including:

Debugging Client Disconnects
RF Dashboard
AMON and Airwave
Client Quirks
AirGroup Config Notes
Voice Over WiFI Tweaking
High CPU On Controller
Aruba Utilities for Android
AirRecorder
11ac sniffing

Published in: Travel, Business

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
23,162
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
272
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • 46:01
  • Transcript

    • 1. Top 10 Tips from Aruba TAC Ken Peredia, Jeffrey Goff November 2013 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 1 #airheadsconf
    • 2. Agenda 1. Debugging Client Disconnects 2. RF Dashboard 3. AMON and Airwave 4. Client Quirks 5. AirGroup Config Notes 6. Voice Over WiFI Tweaking 7. High CPU On Controller 8. Aruba Utilities for Android 9. AirRecorder 10. 11ac sniffing CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 2 #airheadsconf
    • 3. Debugging Client Disconnects CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 3 #airheadsconf
    • 4. Debugging Client Disconnects • Problem could be in any number of places – AP to controller stability • BSS uptime, AP uptime – Feature related • ClientMatch, bandsteering, SLB, dot11k – Auth/Association related • Use the tracebuf and new logging to narrow it down – RF related • ARM changing channels • Interference CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 4 #airheadsconf
    • 5. Debugging Client Disconnects • AP to Controller – If the AP is not stable, clients will experience disconnects – Common causes • Network issues between AP and controller • Controller load issues – Debug these using • “show ap debug counters” – look for APs with high reboot/straps • “show ap bss-table” – look for APs with low tot-t (bss uptime) • Collect “show ap tech-support ap-name <the_ap>” • Collect “tar logs + tech support” CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 5 #airheadsconf
    • 6. Debugging Client Disconnects • AP to Controller – In regards to network issues, common problems include • ARP issues on AP vlan • DHCP issues on AP vlan, including duplicate IP • Packet loss in intermediate switch/router (especially RAPs) • IDS/Firewall device interfering with connectivity – If you are seeing AP rebootstraps, start to investigate the quality of the link between controller and AP • Setup a long term ping with large packet size (say 1400 bytes) and monitor • “show ap debug system-status ap-name <the_ap>” will tell you something about why the AP is unstable CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 6 #airheadsconf
    • 7. Debugging Client Disconnects • ClientMatch – Continuously scans the wireless environment and shares information about the clients amongst the APs – Based on the dynamic data obtained, constantly optimizes the client association by steering clients to the most suitable AP – In some cases, ClientMatch may be more aggressive than needed for certain environments and can be slightly tuned to be less active CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 7 #airheadsconf
    • 8. Debugging Client Disconnects CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 8 #airheadsconf
    • 9. Debugging Client Disconnects CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 9 #airheadsconf
    • 10. Debugging Client Disconnects CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 10 #airheadsconf
    • 11. Debugging Client Disconnects CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 11 #airheadsconf
    • 12. Debugging Client Disconnects • View which clients were put in the unsupported list due to many failed steering attempts CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 12 #airheadsconf
    • 13. Debugging Client Disconnects • To make ClientMatch less aggressive with load balancing the following configuration can be used CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 13 #airheadsconf
    • 14. Debugging Client Disconnects • Per user auth logging is new in ArubaOS 6.3 • Simplified “per user auth tracebuf” – No longer need to enable “logging level debugging user-debug <user mac address>” – This information is now saved per-user when CLI command “aaa log” is enabled in configuration terminal. – To view the log • “show user-table mac <mac-address> log” • or “show user-table ip <ip-address> log” CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 14 #airheadsconf
    • 15. Debugging Client Disconnects (Aruba7210_Sanya) #show user-table mac 68:a8:6d:32:ef:a2 log 1: At Tue Nov 12 05:20:29: [L] Type eap-req <- id 8 len 6, bssid 6c:f3:7f:f0:b7:50 2: At Tue Nov 12 05:20:29: [L] Type eap-resp -> id 8 len 172, bssid 6c:f3:7f:f0:b7:50 3: At Tue Nov 12 05:20:29: [L] Type rad-req -> id 9 len 387, bssid 6c:f3:7f:f0:b7:50 4: At Tue Nov 12 05:20:29: [L] Type rad-resp <- id 9 len 139, bssid 6c:f3:7f:f0:b7:50 5: At Tue Nov 12 05:20:29: [L] Type eap-req <- id 9 len 69, bssid 6c:f3:7f:f0:b7:50 6: At Tue Nov 12 05:20:29: [L] Type eap-resp -> id 9 len 6, bssid 6c:f3:7f:f0:b7:50 7: At Tue Nov 12 05:20:29: [L] Type rad-req -> id 10 len 221, bssid 6c:f3:7f:f0:b7:50 8: At Tue Nov 12 05:20:29: [L] Type rad-accept <- id 10 len 246, bssid 6c:f3:7f:f0:b7:50 9: At Tue Nov 12 05:20:29: [L] Type eap-success <- id 9 len 4, bssid 6c:f3:7f:f0:b7:50 10: At Tue Nov 12 05:20:29: [L] Type wpa2-key1 <- id 0 len 117, bssid 6c:f3:7f:f0:b7:50 11: At Tue Nov 12 05:20:29: [L] Type wpa2-key2 -> id 0 len 135, bssid 6c:f3:7f:f0:b7:50 12: At Tue Nov 12 05:20:29: [L] Type wpa2-key3 <- id 0 len 151, bssid 6c:f3:7f:f0:b7:50 13: At Tue Nov 12 05:20:29: [L] Type wpa2-key4 -> id 0 len 95, bssid 6c:f3:7f:f0:b7:50 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 15 #airheadsconf
    • 16. Debugging Client Disconnects • Per client association history also new in 6.3 • Easily view the activity of a client‟s connection – why it was de-authenticated by the system – what alerts/errors were encountered by the client – What APs has it been associated to, and for how long (Aruba7210_Sanya) #show ap client trail-info 68:a8:6d:1f:5a:9a Client Trail Info ----------------MAC --68:a8:6d:1f:5a:9a BSSID ----6c:f3:7f:f0:b9:b4 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved ESSID ----Diamond5 AP-name ------AP225-6 VLAN ---2 Deauth Reason ------------STA has roamed to another AP 16 Alert ----STA has roamed to another AP #airheadsconf
    • 17. Debugging Client Disconnects Deauth Reason ------------Reason Timestamp ------ --------- STA has roamed to another AP Nov 11 20:14:50 Client Match Nov 11 20:14:36 Rx Data Bytes 108 Mbps+ (Mon) Nov 11 19:29:29 STA has roamed to another AP Nov 11 18:44:09 APAE Disconnect Nov 11 18:28:47 STA has roamed to another AP Nov 11 18:28:14 Client Match Nov 11 18:27:31 Ptk Challenge Failed Nov 11 18:26:57 STA has roamed to another AP Nov 11 18:24:41 STA has roamed to another AP Nov 11 18:24:06 Num Deauths:10 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 17 #airheadsconf
    • 18. Debugging Client Disconnects Mobility Trail -------------- BSSID ESSID AP-name Timestamp ----- ----- ------- --------- 6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 12 05:20:34 6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 20:14:50 6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 20:14:50 6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 20:14:39 6c:f3:7f:f0:c5:90 Diamond5 AP225-4 Nov 11 20:14:36 6c:f3:7f:f0:c5:90 Diamond5 AP225-4 Nov 11 20:13:02 6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 19:29:29 6c:f3:7f:f0:b7:50 Diamond5 AP225-1 Nov 11 18:44:09 6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 18:44:09 6c:f3:7f:f0:b9:b4 Diamond5 AP225-6 Nov 11 18:28:50 Num Mobility Trails:10 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 18 #airheadsconf
    • 19. Debugging Client Disconnects • RF Related – If the 2.4GHz is particularly noisy the default settings of ARM may be causing too many channel changes • “show ap arm history ap-name <the_ap>” to review – Need to check first for legitimate sources of interference • Use spectrum analysis • Use AMON / RF Dashboard – But sometimes spectrum looks ok and the issues are coming more from co-channel interference and hidden nodes etc. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 19 #airheadsconf
    • 20. Debugging Client Disconnects • RF Related – If it is observed that the 2.4GHz radio is changing channels excessively due to noise or error • Create a new ARM profile for 2.4GHz • Attach it to the dot11g radio profile • Increase the noise and error thresholds – Increase gradually and monitor – Don‟t increase too much • if there is real interference, AP will take longer to react. Error Rate Threshold Error Rate Wait Time Noise Threshold Noise Wait Time CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 50 % 30 sec 75 -dBm 120 sec 20 #airheadsconf
    • 21. Controller RF Dashboard CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 21 #airheadsconf
    • 22. RF Dashboard • Use the dashboard to quickly find issues – – – – Find APs with interference Find clients with RF problems Get a summary of potential issues View short term trending information • Don‟t have to check each AP from CLI – show ap debug radio-stats…… hard to use and interpret CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 22 #airheadsconf
    • 23. RF Dashboard CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 23 #airheadsconf
    • 24. RF Dashboard • Everything is clickable – including the graphs CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 24 #airheadsconf
    • 25. RF Dashboard • SNR – Low SNR may mean client is far away from AP or perhaps not enough coverage. – Difficult to interpret in a hotspot environment – In a high density environment where the users are distributed around the APs, there should be very few red results • May also be due to sticky clients, ClientMatch can help CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 25 #airheadsconf
    • 26. RF Dashboard • Noise Floor – in-band non 802.11 interference – These APs should always be investigated – Common causes • Video bridges, bluetooth, DECT, microwave ovens etc. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 26 #airheadsconf
    • 27. RF Dashboard • Channel Busy – This is an important item – it needs to be considered with interference – Channel busy = tx time + rx time + interference – Possible causes for high values • Genuine traffic usage • Over density / too much 802.11 wifi / beacons not optimised • Interference (in band and out of band) CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 27 #airheadsconf
    • 28. RF Dashboard • Channel Busy – For 2.4GHz this will often be high • Especially in hotspots • Values of 30-40% are common • Influenced by g-tx-rate/g-basic-rate/g-beacon-rate – For 5Ghz, this is a very good indicator of load on the AP CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 28 #airheadsconf
    • 29. RF Dashboard • Interference – This is the non 802.11 component of channel busy – Interference value needs to be as low as possible • Typical value is 0 ~ 3% – Possible causes of high interference numbers • Out-of-band (WiMax, cellular DAS antennas, DECT 1.8GHz) • In-band from non-802.11 devices (video, microwave ovens, 2.4GHz DECT phones etc) CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 29 #airheadsconf
    • 30. AMON and Airwave CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 30 #airheadsconf
    • 31. AMON and Airwave • What is AMON – A protocol used to stream information from the controller to Airwave • Also used by Aruba‟s Analytic Location Engine (ALE) • AMON is built on top of PAPI – the AP <-> controller protocol • Aruba does not provide any public documentation or description about the protocol definition. – AMON sends radio stats to Airwave (7.5.x and higher) • Controller dashboard has only 15 minutes worth of data • Airwave plots longer term  trending • Use alert triggers to proactively flag when an AP is experiencing interference or high utilisation etc. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 31 #airheadsconf
    • 32. AMON and Airwave • Enable sending of AMON to Airwave on controller – webUI Configuration  Management  General • Also used by Aruba‟s Analytic Location Engine (ALE) • In Airwave ensure AMON is enabled • AMP Setup  General  Additional AMP Services CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 32 #airheadsconf
    • 33. AMON and Airwave • After a short while AMON radio stats will appear – Access the stats via the AP (i.e. APs/Devices  Monitor) – Select the radio by clicking on the name – Then select “Channel Utilization” from the drop down – If Channel Utilization is not an option then AMON is not enabled CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 33 #airheadsconf
    • 34. AMON and Airwave • Same data as being shown on the controller – Remember: channel busy == rx + tx + interference – Can hover over a point to get the values – Much more history available via “Time Range” – Note that noise floor is accessible via another dropdown option CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 34 #airheadsconf
    • 35. AMON and Airwave • What can we determine from this ? – Perhaps an interference source that comes and goes – Or one that suddenly appears • Someone just bought a 2.4GHz non 802.11 wireless device to the office – interference comes and goes with office hours • Wireless video camera installed nearby. – Alternatively, let‟s assume no interference • Utilisation versus time – does this AP have a problem on some day of the week? • Is this AP unexpectedly underutilized ? (maybe on a rare channel) • Constant “rx == busy” at a high value may indicate that beacon rate optimization on the APs would be beneficial CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 35 #airheadsconf
    • 36. AMON and Airwave • Can we be more proactive? – The statistics that appear in the radio monitor page can be used within alert triggers – Actually, use triggers to monitor other critical resources like disk space on the Airwave itself, or AP CPU etc. • Use short term and long term durations – Cover immediate problems with short durations • “more than 20% interference over 10 mins”  critical impact – Keep an eye on the growth with long durations • “channel busy > 80% over 6 hours”  Warning, capacity running out CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 36 #airheadsconf
    • 37. AMON and Airwave • Example – To have 20% interference on 5GHz is very unusual – It would be symptomatic of a device like a motion detector • Use radio type to be more/less aggressive by band – Determine the baseline in each band and adjust accordingly CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 37 #airheadsconf
    • 38. Client Quirks CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 38 #airheadsconf
    • 39. Client Quirks • MacOS 802.1X Roaming Latency – MacOS 10.8.4 or newer may intermittently experience 5 – 20 seconds latency in completing 802.1X authentication when roaming between APs. – This latency is usually seen during the middle of EAP negotiation. – This problem is only seen on a 802.1X SSID. It‟s not seen on a PSK SSID. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 39 #airheadsconf
    • 40. Client Quirks • MacOS 802.1X Roaming Latency – The following is an example 10 sec delay for the client's eap-response in "show auth-tracebuf" output: Sep Sep Sep Sep Sep Sep Sep Sep 13 13 13 13 13 13 13 13 18:03:17 18:03:17 18:03:17 18:03:27 18:03:27 18:03:27 18:03:27 18:03:27 rad-req rad-resp eap-req eap-resp rad-req rad-resp eap-req eap-resp CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved -> <<-> -> <<-> 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 10:40:f3:83:fe:6c 40 00:1a:1e:55:53:50/RADIUS_Server_1 00:1a:1e:55:53:50/RADIUS_Server_1 00:1a:1e:55:53:50 00:1a:1e:55:53:50 00:1a:1e:55:53:50/RADIUS_Server_1 00:1a:1e:55:53:50/RADIUS_Server_1 00:1a:1e:55:53:50 00:1a:1e:55:53:50 65518 65518 7 7 13 13 8 222 287 203 1276 1502 90 6 #airheadsconf
    • 41. Client Quirks • MacOS 802.1X Roaming Latency – Available workaround seems to greatly improve this authentication latency – Trust the 802.1X RADIUS server cert as root in the System keychain while setting “Always Trust” for SSL, EAP, and X.509 basic policy. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 41 #airheadsconf
    • 42. Client Quirks CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 42 #airheadsconf
    • 43. Client Quirks CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 43 #airheadsconf
    • 44. Client Quirks • MacOS Ping Latency – In viewing continuous pings on a Macbook Pro/Air, you might see a large variation of ping latency from 2ms to 100+ms – while other devices on the same AP experience only 1- 4ms latency CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 44 #airheadsconf
    • 45. Client Quirks • MacOS Ping Latency CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 45 #airheadsconf
    • 46. Client Quirks • MacOS Ping Latency – The issue is due to the aggressive power save behavior seen in newer MacOS (10.8.4+). – Sometimes the Macbooks will sleep 1ms after a ping request is sent and may not wake up until 90ms+ later. – Therefore the Aruba AP will buffer the ping response until the Macbook informs the AP it is awake which sometimes results in large latency as shown in the following sniffer trace. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 46 #airheadsconf
    • 47. Client Quirks CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 47 #airheadsconf
    • 48. Client Quirks • Dot11k – Some interop issues seen – be careful if you enable this – Example: 4.3.x Android device on wpa2-aes SSID • Client associates, gets IP address but locks up shortly thereafter • disabling dot11k inside the “wlan dot11k-profile default” resolved • Bandsteering vs Clients – Some clients don‟t react well to bandsteering – A common one is Android, which can report “authentication error occurred” – This error is thrown by Android when the Aruba AP sends a „resource constrained‟ message during 802.11 auth CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 48 #airheadsconf
    • 49. AirGroup CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 49 #airheadsconf
    • 50. AirGroup • Common AirGroup configuration issues • AirGroup domain controller IP must be switch IP – Symptom: active domain is stuck in „excluded‟ state (sg-3200) # show airgroup active-domains AirGroup Active-Domains ----------------------Domain Name Status ---------------SomeDomain Excluded Num active-domains:1 – make sure that the specified controller IP is a switch IP and not an interface IP CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 50 #airheadsconf
    • 51. AirGroup • Wireless based AirGroup „server‟ (i.e. Apple Tv) – Symptom: AppleTv connected wirelessly keeps disappearing from list of Airplay servers on the client device – Controller does not by default send refresh requests to wireless side – We have a configuration option to force this behavior “airgroup active-wireless-discovery enable” CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 51 #airheadsconf
    • 52. AirGroup • Apple TV and EAP-PEAP – Symptom: AppleTv fails to connect to PEAP SSID – AppleTv has no real time clock – Connect fails because the PEAP cert fails date validation check – because the date/time is wrong – Requires NTP to set the time – but needs access to network to use NTP – Found to be resolved in iOS 7 • exact method of resolution by Apple is unknown • Verified on hardware v2 and v3 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 52 #airheadsconf
    • 53. AirGroup • NAT – Airgroup is not supported with NAT – Do not configure ip nat inside or use a NAT ACL on any client that needs Airgroup. • Disallow VLAN – Can be configured globally or on a per service basis – Disallowing a VLAN means that mDNS servers in that VLAN cannot be discovered. – However… mDNS clients in the disallowed VLAN can still be discovered by servers in other VLANs CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 53 #airheadsconf
    • 54. Voice optimizations CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 54 #airheadsconf
    • 55. ArubaOS – VoWIFI • Deployment tips – Always try to use 5GHz for voice • If AP coverage allows and devices are capable • But be aware of differences in client ability to support all channels in your regulatory domain. – Make sure you have adequate coverage, rule of thumb is two celloverlap of -67dB CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 55 #airheadsconf
    • 56. ArubaOS – VoWIFI • Ensure QOS vs. DSCP is matched – on both the wired and wireless infrastructure – SSID configuration and user-role – Real voice applications should be correctly identified by the controller ALGs (sip, h323, SCCP, etc.) • But something like Skype will not be – it‟s just „data‟ • Assumptions – Dual use SSID (voice and data) • More aggressive optimizations can be used on a voice-only SSID – Office style coverage/density – AP signal strength of -65 dBm at the client – Client SNR 25 or better at AP (show ap debug client-table) CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 56 #airheadsconf
    • 57. ArubaOS – VoWIFI • Suggested SSID configuration – – – – – – – a-basic-rates a-tx-rates g-basic-rates g-tx-rates g-beacon-rate local-probe-req-thresh eapol-rate-opt 12 24 12 18 24 36 48 54 6 12 6 9 12 18 24 36 48 54 6 25 • Suggest virtual AP configuration – broadcast-filter all – broadcast-filter arp CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved * (unless mcast based hold music) 57 #airheadsconf
    • 58. ArubaOS – VoWIFI • Suggested SSID configuration – – – – – – – a-basic-rates a-tx-rates g-basic-rates g-tx-rates g-beacon-rate local-probe-req-thresh eapol-rate-opt 12 24 12 18 24 36 48 54 6 12 6 9 12 18 24 36 48 54 6 25 • Suggest virtual AP configuration – broadcast-filter all – broadcast-filter arp CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved * (unless mcast based hold music) 58 #airheadsconf
    • 59. ArubaOS – VoWIFI • Suggested ARM configuration – Don‟t have more than 6 dB difference between “min-tx-power” and “max-tx-power” to provide a more predictable roaming behavior • Avoids large power differences between neighboring APs – “client-aware” and “voip-aware-scan” *must* be enabled • “ps-aware-scan” should not be enabled – Don‟t raise “min-tx-power” too high as it can cause sticky clients • Typical high density office might use a “min-tx-power” of 9 dBm • Smart phone type devices have quite low roaming thresholds, often as low as -75dBm. Setting tx power too high can cause client->AP to fail well before device tries to roam. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 59 #airheadsconf
    • 60. Common high CPU issues CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 60 #airheadsconf
    • 61. High CPU on controller • The usual suspects… – WMS – HTTPd (the AirMonitor/IDS function) (the captive portal webserver) • WMS – Responsible for the collection of statistics about valid, interfering and rogue devices (APs and clients) – In hotspot deployments WMS may be tracking upwards of 50,000 devices – Airports and large campuses also have large WMS databases CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 61 #airheadsconf
    • 62. High CPU on controller – As the WMS database grows so do the related SNMP tables – Devices like Airwave are polling these tables periodically which causes CPU spikes (aruba) #show wms counters | include table AP G-table 19841 AP A-table 40 STA G-table 28365 STA A-table 1231 This kind of WMS database would return over 250,000 results in SNMP – Symptoms of high CPU load in WMS include • sluggish webUI, SNMP timeouts, missing data in Airwave graphs • Can also impact captive portal as it checks for CPU load CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 62 #airheadsconf
    • 63. High CPU on controller – Unstable APs can also contribute to the WMS CPU load. • “show ap debug counters” • check for high reboot/rebootstraps • “show ap bss-table” • check for low tot-t (uptime) of the BSSID • Especially for RAPs deployments where the controller to AP transits multiple networks beyond your control. AP Counters ----------Name Group -------ap100 apgroup ap101 apgroup ap102 apgroup ap103 apgroup ap104 apgroup IP Address ---------1.1.2.2 1.1.2.3 1.1.2.4 1.1.2.5 1.1.2.9 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved Configs Sent -----------4 4 8 8 8 Configs Acked ------------4 4 8 8 8 63 Bootstraps (Total) -----------------1 (1 ) 1 (1 ) 120 (120 ) 3 (3 ) 3 (3 ) Reboots ------0 0 0 0 0 #airheadsconf
    • 64. High CPU on controller – What to do ? 1. WMS Offload to Airwave • Moves large portions of WMS processing to Airwave • APs on local controllers now send direct to Airwave • But… increases WMS CPU load on local controllers 2. Need to reduce SNMP polling of certain MIBs 3. Or - reduce the amount of data being held in WMS – If not able to use WMS offload, two basic levels of optimization can be done depending on the features being used. – If not able to perform #2 or #3 then WMS offload is the only choice CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 64 #airheadsconf
    • 65. High CPU on controller 1. If no IDS/WIPS functionality is being used, the following config will prevent interfering clients from being entered into the WMS database. “ids general-profile default wms-client-monitoring none” • A warning is printed to remind you of the functionality that is impacted by this command. (sg-620) (config) #ids general-profile default wms-client-monitoring none Warning: Enabling this option will cause the following features to not work as intended: Protect Valid Station, Detect Valid Client Misassociation, Detect Unencrypted Valid Clients, Tarpit Containment of Non-valid clients, Detect Bad WEP, Detect Disconnect Station Attack, Detect Power Save DoS Attack, Detect Block ACK DoS, Detect TKIP replay Attack, Detect ChopChop Attack, Detect Omerta Attack, Detect FATA-Jack Attack, Detect Overflow EAPOL Key, Detect Frame Rate Anomalies. (sg-620) (config) # CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 65 #airheadsconf
    • 66. High CPU on controller 2. If you don‟t require or care about rogue AP information in Airwave, then you can disable “Rogue AP and Device Location data Polling Period” – Note this will impact any location services or RAPIDS, and rogue list will no longer be populated in Airwave CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 66 #airheadsconf
    • 67. High CPU on controller • WMS – What if all that is done but problem not resolved? – Contact support for assistance • there are a few more settings that can be changed • But…these are very specific to the deployment and need to be carefully chosen. CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 67 #airheadsconf
    • 68. High CPU on controller • HTTPD – High CPU with internal captive portal + large certificates • Large certs are handled in software for all controllers before 72xx • Large being 2048 or 4096 • Very hard to get a public signed 1024 bit cert anymore – Many smart devices constantly trying to hit https locations once associated • This is becoming more common over last 1-2 years • Even if they never authenticate with the captive portal • iTunes, Play Store, Windows Update, antivirus updates, CDNs – Also caused by image heavy internal captive portal page CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 68 #airheadsconf
    • 69. High CPU on controller • Solutions – Move to an http based captive portal – but keep https for the username and password • Turn on allow http in captive portal profile • Modify the HTML code of the captive portal page to use https for the form action: <div id="logins"> <div id="loginbox" style=""> <h1><span>Wi-Fi Login</span></h1> <form action="/cgi-bin/login" id="regform" method="post" autocomplete="off" title="Login"> <div id="usernamebox"> <label for="user" accesskey="u">Username</label> <form action="https://securelogin.arubanetworks.com/auth/index.html/u" id="regform" method="post" title="Login"> If not using the Aruba default certificate, make sure to replace securelogin.arubanetworks.com with the CN of the new cert CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 69 #airheadsconf
    • 70. High CPU on controller • Solutions – Changing to http with https form action may not totally recover the CPU… – Need to deal with the high rate of incoming https requests for clients that have not authenticated – CPU load due to the captive portal redirect can also impact external captive portal CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 70 #airheadsconf
    • 71. High CPU on controller • Solutions – One option is to prevent any https from redirecting • Remove the “user any svc-https dst-nat 8081” rule from the captive portal ACL • This has a downside – clients with https home pages will fail to redirect to captive portal Priority -------1 2 3 4 5 6 Source -----user user user user user user Destination ----------controller any any any any any CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved Service ------svc-https svc-http svc-https svc-http-proxy1 svc-http-proxy2 svc-http-proxy3 71 Action -----dst-nat dst-nat dst-nat dst-nat dst-nat dst-nat TimeRange --------8081 8080 8081 8088 8088 8088 #airheadsconf
    • 72. High CPU on controller • Solutions – If removing the https redirect is not possible, another option is to create a “blacklist” of sites – Use the datapath session table to examine where clients are connecting to – Packet capture on an external captive portal and examine the URLs that the clients were trying to connect to https-drop-list --------------Position Type -------- ---1 name 2 name 3 name 4 name 5 name 6 name IP addr ------0.0.0.1 0.0.0.3 0.0.0.4 0.0.0.5 0.0.0.2 0.0.0.6 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved Mask-Len/Range -------------*.facebook.com *.edgekey.net *.1e100.net *.akamaitechnologies.com *.amazonaws.com *.tfbnw.net 72 #airheadsconf
    • 73. By the way… • Please note that the Aruba controller factory certificate is due to expire on 21 November, 2013 • A new cert is loaded into 6.1.3.9, 5.0.4.13, 7.2.3.1 and higher • There is a support advisory about this on the Aruba support portal and Airheads Social – Contact your partner/reseller to get a copy if you cannot access the portal and you are on an affected version CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 73 #airheadsconf
    • 74. Android Aruba Utilities CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 74 #airheadsconf
    • 75. Aruba Utilities • Free app on Play Store • Requires Android 3.0 or better – There is a stripped down version that runs on 2.2 or better • Features include – – – – – – Built in performance tester (aperf) Interacts with Airwave and the controller Logs signal info to text file Site survey tools (heatmap etc.) RSSI graphs Handover tracking CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 75 #airheadsconf
    • 76. Aruba Utilities CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 76 #airheadsconf
    • 77. Aruba Utilities CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 77 #airheadsconf
    • 78. Aruba Utilities CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 78 #airheadsconf
    • 79. AirRecorder CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 79 #airheadsconf
    • 80. AirRecorder • AirRecorder is a Java based tool developed by Aruba Customer Engineering that will periodically run several common CLI commands for checking controller, AP, and wireless device health for troubleshooting and monitoring. – It can run on Windows, Linux/Unix, and MacOS platforms. – It supports ArubaOS, Instant VC (IAP), and MeshOS (MSR) CLI commands. – All CLI command output is saved to auto-rotating log files CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 80 #airheadsconf
    • 81. AirRecorder • By default, a new log file is created every 100 MB but can be modified with option “--max-log-size”. • AirRecorder is very flexible in gathering data – It can automatically retrieve data from all APs and devices on a Controller via pre-defined variables – It can manually retrieve data from specific APs and devices on a Controller – It can be configured to run different CLI commands at different intervals CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 81 #airheadsconf
    • 82. AirRecorder • Example CLI commands and intervals that can exist in an Airrecorder script 10m,show ap active 5m,show ap association 5m,show user-table 5m,show ap arm rf-summary ap-name %{ap:name} 30m,show ap arm state ap-name %{ap:name} 30m,show ap bss-table 5m,show ap debug client-table ap-name %{ap:name} 5m,show ap debug client-stats %{user:mac} advanced 5m,show ap client trail-info %{user:mac} 5m,show ap arm client-match history client-mac %{user:mac} CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 82 #airheadsconf
    • 83. AirRecorder • Example running of AirRecorder on Win7 C:java -jar AirRecorder-1.2.8-release.jar 10.8.7.63 AirRecorder (c)2011-2012 Thomas Bastian, Aruba Networks Username: admin Password: Enable password: No enable password set, assuming enable mode when connected A new log file will be created every: 100 Mbytes Recording to file: air-recorder-10.8.7.63-20131112-091902-00.log Connecting to controller with hostname: 10.8.7.63, protocol: ssh, port: 22, username: admin Processing command: [show ap active] at schedule specification of 10m Processing command: [show ap association] at schedule specification of 5m Processing command: [show user-table] at schedule specification of 5m Processing command: [show ap arm rf-summary ap-name %{ap:name}] at schedule specification of 5m Processing command: [show ap arm state ap-name %{ap:name}] at schedule specification of 30m Processing command: [show ap bss-table] at schedule specification of 30m Processing command: [show ap debug radio-stats ap-name %{ap:name} radio 0 advanced] at schedule specification of 5m Commands (cur/max): 4/4, Rx Rate: 161 kbps, File size: 1 MB CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 83 #airheadsconf
    • 84. AirRecorder • Example output of Airrecorder ..... ///// ///// Message: RESULT ///// Status: 1937 ///// LocalBeginTime: 1384276748945 (2013-11-12T09:19:08.945-0800) ///// LocalEndTime: 1384276749360 (2013-11-12T09:19:09.360-0800) ///// QueryTag: airrecorder.command=10m,show ap active ///// Command: show ap active ///// Section: Stdout Active AP Table --------------Name Group Ch/EIRP/MaxEIRP --------------- ------AP225-3 225 AP Type ----- IP Address 11g Clients 11g Ch/EIRP/MaxEIRP Flags Uptime Outer IP 11a Clients 11a --------------- ----------- Renaissance_APs 10.8.7.115 Ada 15h:21m:59s N/A AP225-6 Renaissance_APs AP:VHT:149+/15/22 225 CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved ------------------ ------------------- ----------- 0 AP:HT:6/12/21 0 AP:HT:1/12/21 N/A 8 10.8.7.116 0 Ada 15h:22m:13s 84 #airheadsconf
    • 85. 802.11ac Capture Tools CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 85 #airheadsconf
    • 86. Known 802.11ac Capture Tools • Limited options for Windows users – Linksys AE6000 – but single stream – No multi stream options at this point • Aruba AP-22x packet capture – Can decode as “PEEKREMOTE” but the plugin is a bit buggy • Wildpackets Omnipeek – beta driver for Linksys AE6000 (1 stream 11ac) NIC available CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 86 #airheadsconf
    • 87. Known 802.11ac Capture Tools • New Macbook Air (2 stream 11ac) or Macbook Pro (3 stream 11ac) – Enabled through Wireless DiagnosticsUtilitiesFrame Capture – Wireless Diagnostics can be found by clicking on “Option”  ”Wi-Fi icon” Wireless Diagnostics – Saved to a .wcap file that can be opened by Wireshark CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 87 #airheadsconf
    • 88. In conclusion CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 88 #airheadsconf
    • 89. In conclusion • support@arubanetworks.com – One email address for all products – You can always request your ticket to be moved to another time-zone – Upload files directly to the case via the support portal – Avoid unicasting emails/attachments to support staff • Using reply-all will get more eyes on your issue • Always call support for urgent issues • Please exercise caution when making changes – Always move your backups off the controllers/servers – When tweaking, incrementally add changes CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 89 #airheadsconf
    • 90. Takeaways TAC Quick Reference Guide – https://support.arubanetworks.com/DOCUMENTATION/tabid/77/DMXModule/512/Comma nd/Core_Download/Default.aspx?EntryId=1371 Validated Reference Designs (VRD) – http://www.arubanetworks.com/technology/reference-design-guides/ Airheads Social – http://community.arubanetworks.com/ Aruba Knowledge Base – https://kb.arubanetworks.com/ Raise a ticket for any product, RMA, anything ! – support@arubanetworks.com Requests for Enhancements (RFE) – Please discuss with your SE/Sales team Outdoor planner tool – https://outdoorplanner.arubanetworks.com/ CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 90 #airheadsconf
    • 91. Thank You CONFIDENTIAL © Copyright 2013. Aruba Networks, Inc. All rights reserved 91 #airheadsconf