Troubleshooting Common Network Related Issues with NetScaler

1. Troubleshooting Common Network Related Issues with NetScaler Michael Dean & Raghu Varma Tirumalaraju Citrix Support Secrets Webinar Series NetScaler Support Escalation May 2014

2. © 2014 Citrix. Confidential.2 Agenda Mac Based Forwarding Trunking on Non-Link Aggregation Links MAC Moves & Layer2 Loops & Common L2 Loop Scenario Changing the NSVLAN High Availability - Best Practices GARP & VMACs Link Aggregation with LACP

3. © 2014 Citrix. Confidential.3 Mac Based Forwarding (MBF) Mode Design considerations for MBF Mode (Why turn it ON or OFF) Special-use feature L3 routing is not needed (e.g. Stub topology) Used when “split-brain” or asymmetric routing condition is present Common asymmetric routing conditions: (a) Three arm – Third for management (b) Lots of static routes to be maintained

5. © 2014 Citrix. Confidential.5 Common issues seen with MBF - Intermittent performance issues - Packet loss in high volume environments - Same packet loss can result in dropped ICA connections for ICA-Proxy through Netscaler Gateway vserver - Periodic UI access issues - Effectively disables L3 routing in favor of L2-type forwarding

6. © 2014 Citrix. Confidential.6 Trunking on Non-Link Aggregation Links Common issues seen when misconfigured: - Dropped packets due to promiscuous vlan drops. nsconmsg -K newnslog -s disptime=1 -g nic_ -d current | grep 1/2 | grep drop | more 18 0 6874714506 35162 5033 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:17 2014 25 0 6862206790 35106 5025 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:17 2014 73 0 6874743016 28510 4072 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:24 2014 80 0 6862235244 28454 4064 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:24 2014 127 0 6874773155 30139 4306 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:31 2014 134 0 6862265285 30041 4292 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:31 2014 185 0 6874837857 64702 9243 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:38 2014 The solution in this case is to ensure that the switchport settings only reflect the same allowed vlans that the netscaler is configured to send or receive traffic for

7. © 2014 Citrix. Confidential.7 Trunking on Non-Link Aggregation Links cont. Design considerations – vlan number assignments and tagging: Native vs. Default VLAN Trunking vs Tag All Tagging the native vlan - For differentiation - Tagging single vlan - Network policy might dictate a different native vlan.

8. © 2014 Citrix. Confidential.8 Changing the NSVLAN Changing the nsvlan is considered an extreme case and should be used as a last resort Common Reasons for changing nsvlan: a) Network operations policy b) Separate requirement for native non-tagged vlan Pitfalls of changing the nsvlan: a) Requires reboot b) NSIP VLAN is impacted – special considerations needed c) Special order of operations may be needed

9. © 2014 Citrix. Confidential.9 MAC Moves & Layer2 Loops “The Rule of Bridging” – ARPs will be flooded out all connected interfaces (on the same vlan) except the one on which the request was received. - L2 Mode considerations – Frames forwarded. “Very” rarely used! (see next slide) - Common issues seen resulting from MAC moves /L2 loops. - Poor performance - Application traffic throughput on the Netscaler ADC may not be as timely as expected - Frequent packet retransmissions causing latency - Dropped packets on interface(s) - Packet drops incrementing on the interface stats (show screenshot/text output) -Common topologies that cause these issues - Multiple interfaces connected into the same vlan -Design considerations to alleviate MAC Moves/L2 loops - Disable multiple interfaces that are connected to the same vlan - Implement Link-Aggregation (LA) channel to manage the traffic for multiple interfaces in same vlan

11. © 2014 Citrix. Confidential.11 Common L2 Loop Scenario Network Switch Netscaler Int 1/1 Int 1/2 fa 0/3 fa 0/4 1 2 3 1. Netscaler sends ARP for destination MAC for a given IP. Since both int 1/1 and 1/2 are connected, the request is flooded out of both interfaces 2. Switch/End device receives both requests and responds with destination MAC for given IP via both connected switchports 3. Netscaler receives both responses and caches the first response, then sends the request to the destination MAC for the given IP in that first response. 4 4. The Network switch, since it has the MAC address in it’s ARP table for the Netscaler SNIP as being accessible via fa0/4 due to a previous transaction and ARP table entry, it responds via this interface. The Netscaler drops the packet in the response since it is aware of the destination having been mapped via int 1/1 (Bridge Table)

12. © 2014 Citrix. Confidential.12 Detecting L2 Loops - Overview The scenario in the previous slide is very common when more than one interface in the same vlan is connected to the switch with both switch ports also configured on the same vlan. The result from the received packet arriving on the assumed incorrect interface is a packet drop. Packet drops are detectable in a few ways through debugging the Netscaler logs.

13. © 2014 Citrix. Confidential.13 Detecting L2 Loops – From Shell debugging The NSCONMSG utility is very powerful and frequently used by Citrix Support in gathering useful information to help determine root cause for system issues. In this example we see that we have used ‘grep’ to parse the current newnslog file for the strings “nic_err” and “drop” simultaneously. *Note that even in a healthy environment, packet drops occur all the time however when interesting counters such as ‘nic_err_vlan_promisc_tag_drop’ increment, ‘so do nic_err_dropped_pkts’ root@ns# nsconmsg -K newnslog -d stats | grep nic_err | grep drop 1854 0 91 nic_err_dropped_pkts interface(1/1) 1855 0 91 nic_err_dropped_pkts interface(1/2) 1866 0 0 nic_err_tx_dropped 1886 0 0 nic_err_congested_pkts_dropped 1887 0 0 nic_err_congestionlimit_pkts_dropped 1914 0 0 nic_err_la_frame_collect_drops 1915 0 0 nic_err_la_tagged_bpdu_drops 1916 0 73 nic_err_vlan_promisc_tag_drops 1917 0 0 nic_err_la_untagged_pkt_drops 1919 0 0 nic_err_rl_pkt_drops 1920 0 0 nic_err_rl_rate_pkt_drops 1921 0 0 nic_err_rl_pps_pkt_drops 4438 0 0 allnic_err_rl_rate_pkt_drops 4439 0 0 allnic_err_rl_pps_pkt_drops root@ns#

14. © 2014 Citrix. Confidential.14 More L2 Issues – From the Shell debugging The below counter tracks how often a given MAC is being detected via different ports. In normal operation a MAC address should only be seen only on one port. A MAC moving often is a sign of a loop – This can cause both failovers and performance issues. nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg 30433 0 25681657 nic_tot_bdg_mac_moved interface(1/3) 30434 0 25725446 nic_tot_bdg_mac_moved interface(1/2) 30435 0 25556085 nic_tot_bdg_mac_moved interface(1/1) 30436 0 28858167 nic_tot_bdg_mac_moved interface(0/1) 30437 0 27641933 nic_tot_bdg_mac_moved interface(0/2) This then results in the NetScaler eventually muting the interfaces to try and end the loop. nsconmsg -K newnslog -d statswt0 | grep nic_err 30438 0 467 nic_err_bdg_muted interface(1/3) 30439 0 445 nic_err_bdg_muted interface(1/2) 30440 0 447 nic_err_bdg_muted interface(1/1) 30441 0 605 nic_err_bdg_muted interface(0/1) 30442 0 578 nic_err_bdg_muted interface(0/2) *Generally issues can be seen with the ‘-d statswt0 | grep nic_err’ switch in nsconmsg utility

15. © 2014 Citrix. Confidential.15 High Availability What events trigger (an unexpected) failover? Network issues = Missed Heartbeats = failover Tight loops on peer/crash/reboot SSL Card failures Interface failures Heartbeats are sent/received on all interfaces by default Regardless of hamon setting Use UDP port 3003 Common network issues leading to missed heartbeats: VLAN Tagging Mismatch nsVLAN mismatch – Native vlan for heartbeats and NSIP

17. © 2014 Citrix. Confidential.17 Verifying Heartbeat exchange root@ns# nsconmsg -K newnslog -g ha_tot_pkt -s disptime=1 -d current | more Index rtim totalcount-val delta rate/sec symbol-name&device-no&time 0 3560 13 13 3 ha_tot_pkt_rx Sat May 17 13:48:48 2014 1 0 12 12 3 ha_tot_pkt_tx Sat May 17 13:48:48 2014 2 7000 50 37 5 ha_tot_pkt_rx Sat May 17 13:48:55 2014 3 0 49 37 5 ha_tot_pkt_tx Sat May 17 13:48:55 2014 4 6998 101 35 5 ha_tot_pkt_rx Sat May 17 13:49:06 2014 5 0 100 35 5 ha_tot_pkt_tx Sat May 17 13:49:06 2014 6 7000 136 35 4 ha_tot_pkt_rx Sat May 17 13:49:13 2014 7 0 135 35 4 ha_tot_pkt_tx Sat May 17 13:49:13 2014 8 7001 171 35 4 ha_tot_pkt_rx Sat May 17 13:49:20 2014 9 0 170 35 4 ha_tot_pkt_tx Sat May 17 13:49:20 2014 10 7000 206 35 5 ha_tot_pkt_rx Sat May 17 13:49:27 2014

18. © 2014 Citrix. Confidential.18 Tagging of HA packets - Scenarios Scenario 1 NSVLAN is 1 (default) interface 1/2 is bound to VLAN 2 Interface 1/3 is bound to VLAN 3 Result: High Availability packets flow as untagged on the 1/2 and 1/3 interfaces on the native VLAN. Scenario 2 NSVLAN is 1 (default) interface 1/2 is bound to VLAN 2, which is configured with –tagall ON Interface 1/3 is bound to VLAN 3, which is configured with –tagall OFF (the default) Results: HA packets flow on 1/2 as tagged with a VLAN ID of 2, and untagged on the 1/3 interface. Scenario 3 NSVLAN is 3 (non default) interface 1/2 is bound to VLAN 2 interface 1/3 is bound to VLAN 3 Result: HA packets flow as tagged packets on interface 1/3 only and don’t flow on Interface 1/2

19. © 2014 Citrix. Confidential.19 HA related events Secondary Primary root@70# nsconmsg -K newnslog -d event | grep node 0031 0 PPE-0 remote node 10.72.137.50: Primary Tue May 20 21:48:12 2014 1032 0 PPE-0 self node 10.72.137.70: SYNC start Tue May 20 21:48:15 2014 1203 0 PPE-0 self node 10.72.137.70: SYNC complete Tue May 20 21:48:20 2014 1210 0 PPE-0 remote node 10.72.137.50: DOWN Tue May 20 21:51:51 2014 1211 0 PPE-0 self node 10.72.137.70: Claiming Tue May 20 21:51:51 2014 1212 0 PPE-0 self node 10.72.137.70: Primary Tue May 20 21:51:51 2014 1221 0 PPE-0 remote node 10.72.137.50: INIT Tue May 20 21:53:01 2014 1223 0 PPE-0 remote node 10.72.137.50: UP Tue May 20 21:53:02 2014 root@50# nsconmsg -K newnslog -d event | grep node 946 0 PPE-0 remote node 10.72.137.70: UP Tue May 20 21:53:02 2014 1017 0 PPE-0 self node 10.72.137.50: UP Tue May 20 21:53:03 2014 1018 0 PPE-0 self node 10.72.137.50: HA license check result: MATCH Tue May 20 21:53:03 2014 1019 0 PPE-0 remote node 10.72.137.70: Primary Tue May 20 21:53:03 2014 1020 5 PPE-0 self node 10.72.137.50: SYNC start Tue May 20 21:53:06 2014 1191 0 PPE-0 self node 10.72.137.50: SYNC complete Tue May 20 21:53:10 2014 This is why Tech support asks for Collectors from both Primary and Secondary units for HA related issues. Having the correct date and time set on the devices is important to simplify correlation. #date +val YYMMDDHHMM

20. © 2014 Citrix. Confidential.20 HA Best Practices Use VMACs Disable unused interfaces (not just hamon on those interfaces) rpc passwords should match – Mismatch leads to sync/prop failures “Warning: Unable to establish connection with the secondary. Command propagation failed” Ensure certs and files match on peers – to avoid certificate binding failures >sync ha files Ensure both devices are running same Versions and use the same hardware/license Enable failsafe mode to avoid a complete outage when both nodes have an issue > set ha node -failSafe ON Use Failover Interface Sets (FIS) - if interface bandwidth is not an issue >add fis testfis >bind fis testfis 1/1 1/2

22. © 2014 Citrix. Confidential.22 VMACs - A better solution – Simple config > add vrID 1 > bind vrID 1 -ifnum 1/1 – Instantaneous failovers – Configuration propagated to secondary Considerations: Just be sure switch supports it You might need to flush arp once > send arp all

23. © 2014 Citrix. Confidential.23 LACP Aggregating Multiple ports into a High Speed Link • 802.3ad Compliant • Aggregation of throughput • Better Resiliency • Channels are dynamically created using the port key when lacp is enabled on an interface Sample configuration: >set interface 1/1 -lacpmode active -lacpKey 1 – lacpTimeout long lacpkey value forms the basis for the LA/x notation active/passive decides whether NetScaler sends LACPDUs or not lacptimeout decides how often they are sent Note: - LACP Configs are neither propagated nor synchronized - Once an interface is bound to a channel, channel parameters take precedence i.e. interface VLAN config no longer applies

24. © 2014 Citrix. Confidential.24 LACP continued.. LACP on SDX Relies on “NIC Bonding” feature of XS Switch and SDX are aware of the bond LACP PDUs are exchanged Recommended Versions  SDX: 10.1 or higher  NetScaler: 10.1 or higher  XenServer: 6.1 or higher What is now possible: • Sharing across instances • LA for Management Interfaces Channel creation will require reboot on instances if the member interfaces are not already provisioned. Best Practices – Only create Channels from SVM – Not from XenServer not from within Instance. Ensure of equal speed – or lowest speed will be chosen. Create channel first before plugging in cables. Similarly disconnect the cable before removing a port from the channel.

25. © 2014 Citrix. Confidential.25 Logs on XenServer tail –f /var/log/messages | grep bond LACP going down [root@localhost ~]# cat /var/log/messages | grep bond May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03991|bond|INFO|interface eth8: link state down May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03992|bond|WARN|interface eth8: disabled May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03993|bond|INFO|bond bond0: active interface is now eth6 May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03994|bond|INFO|interface eth6: link state down May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03995|bond|WARN|interface eth6: disabled May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03996|bond|WARN|bond bond0: all interfaces disabled May 17 09:38:02 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 0 up_old 2 LACP back up [root@localhost ~]# cat /var/log/messages | grep bond May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03997|bond|INFO|interface eth6: link state up May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03998|bond|INFO|interface eth6: will be enabled if it stays up for 31000 ms May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03999|bond|INFO|bond bond0: active interface is now eth6, skipping remaining 31000 ms updelay (since no interface was enabled) May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04000|bond|WARN|interface eth6: enabled May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04001|bond|INFO|interface eth8: link state up May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04002|bond|INFO|interface eth8: will be enabled if it stays up for 31000 ms May 17 09:38:57 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 1 up_old 0

26. © 2014 Citrix. Confidential.26 On the SVM /var/mps/log/mps_service.log LA/1 configured in the SVM consists of 10/1 and 10/3: Tuesday, 14 Jan 14 14:36:24.988 +0100 [Debug] Json Web Content: {"host_interface":{"mapped_port":"LA/1","sync_operation":false,"channel_type":"LACP","channel_throughput" :"0","channel_bandwidth_high":"0","channel_bandwidth_normal":"0","channel_interface_list":"10/1,10/3","st atic_channel_state":true,"lacp_channel_time":"SLOW","channel_ha_monitoring":true,"channel_tag_all_vlans": true,"channel_alias":""}}

27. © 2014 Citrix. Confidential.27 Logs – on Instance/vpx/mpx # tail -f ns.log | grep interface May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICMIGRATE 55384 0 : NIC "interface(10/1)" State Migrated May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55385 0 : Device "interface(10/1)" - RX state PORT_DISABLED May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55386 0 : Device "interface(10/1)" - RX state LACP_DISABLED May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55387 0 : Device "interface(10/2)" - State DOWN May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55388 0 : Device "interface(LA/1)" - State DOWN In a healthy case: root@VPX01# nsconmsg -K newnslog -g rx_lacp -s disptime=1 -d current Index rtime totalcount-val delta rate/sec symbol-name&device-no&time 0 1462998 737728 8 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:13 2014 1 0 737664 7 1 nic_tot_rx_lacpdus interface(10/2) Sat May 17 12:09:13 2014 2 7000 737735 7 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:20 2014 When it goes down: 36 77000 737913 6 0 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:13:25 2014

28. © 2014 Citrix. Confidential.28 Show channel – what to look at? > show channel LA/1 1) Interface LA/1 (802.3ad Link Aggregate) #5 flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, 802.1q> MTU=1500, native vlan=1, MAC=e2:08:d2:15:b2:05, uptime 0h00m07s Requested: media NONE, speed NONE, duplex NONE, fctl NONE, throughput 0 Actual: throughput 2000 RX: Pkts(5099181) Bytes(1302893502) Errs(0) Drops(115221) Stalls(0) TX: Pkts(7816907) Bytes(1266490416) Errs(0) Drops(742) Stalls(0) NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0) Bandwidth thresholds are not set. LA mode: AUTO 10/1: FIBER-1000-FULL-NONE UP 0h00m14s PortID=(32768,1), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED <Active, Long timeout, Agg, Sync, Collecting, Distributing> Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 266) <Active, Long timeout, Agg, Sync, Collecting, Distributing> 10/2: FIBER-1000-FULL-NONE UP 0h00m14s PortID=(32768,2), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED <Active, Long timeout, Agg, Sync, Collecting, Distributing> Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 267) <Active, Long timeout, Agg, Sync, Collecting, Distributing>

29. © 2014 Citrix. Confidential.29 Trace > show lacp Actor SystemID: (32768, 00:e0:ed:26:dc:cd) Verify if PDUs are being correctly exchanged Question: do keys need to match on both devices? No – But on a given devices all interfaces that part of the same LA channel need to have the same key

30. © 2014 Citrix. Confidential.30 Next Webinar: June 2014 Title: Best practices for implementing, administering, and troubleshooting XenDesktop 7.5 Description: Citrix XenDesktop introduced a number of new concepts and processes for Desktop Administrators. Understanding these advancements and their effect on is key to a stable XenDesktop environment. This session will discuss core deployment and configuration concepts and considerations and provide proven practices for troubleshooting the top three XenDesktop issues. When: June 24th & 25th June 24th – Register Now June 25th – Register Now

31. © 2014 Citrix. Confidential.31 Fuel your talent with continuous learning. Citrix Education offers the following technical training for Networking professionals: CNS-205: Citrix Netscaler 10 Essentials and Networking CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam CNS-301: Citrix NetScaler 10 Advanced Implementation Visit (bit.ly/05Webinar) to save 10% off through June 30* *Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.

32. © 2014 Citrix. Confidential.32 Build your Citrix skills in your personal virtual sandbox Play in your own Virtual Sandbox with Learning Labs from Citrix Education. With your purchase, you’ll receive your own dedicated server with access to the seven most popular Learning Labs from Synergy. Featured labs include: • NetScaler, the Enterprise Security Swiss Army Knife • Front-Ending and Load Balancing XenDesktop and XenApp with NetScaler • Enhancing Visibility of Applications with NetScaler Insight Center http://training.citrix.com/cms/education/promotions/learninglabs/

33. © 2014 Citrix. Confidential.33 New Citrix Practice Exams Accelerate Your Path to Certification Available on training.citrix.com ($39 each): CPE-350 – Citrix NetScaler 10 Essentials and Networking Practice Exam CPE-300 – Deploying XenDesktop 7 Solutions Practice Exam CPE-A22 – Citrix XenApp 6.5 Advanced Administration Practice Exam http://training.citrix.com/cms/index.php/promotions/prac ticeexams/

Troubleshooting Common Network Related Issues with NetScaler

David McGeough

Troubleshooting Common Network Related Issues with NetScaler