Webinar recording: https://www1.gotomeeting.com/register/737119097 As a NetScaler Administrator, you will need to understand how the NetScaler interacts with the network to ensure an optimally running environment for your applications. In this Webinar delivered by NetScaler Escalation Engineers you will learn some of the common network configuration issues, how to avoid them and when necessary how to troubleshoot them. You will learn how to troubleshoot: - HA issues - GARP issues - LA channel issues - Layer 2 issues

1. Troubleshooting Common Network
Related Issues with NetScaler
Michael Dean & Raghu Varma Tirumalaraju
Citrix Support Secrets Webinar Series
NetScaler Support Escalation
May 2014

2. © 2014 Citrix. Confidential.2
Agenda
Mac Based Forwarding
Trunking on Non-Link Aggregation Links
MAC Moves & Layer2 Loops & Common L2 Loop
Scenario
Changing the NSVLAN
High Availability - Best Practices
GARP & VMACs
Link Aggregation with LACP

3. © 2014 Citrix. Confidential.3
Mac Based Forwarding (MBF) Mode
Design considerations for MBF Mode (Why turn it ON or OFF)
Special-use feature
L3 routing is not needed (e.g. Stub topology)
Used when “split-brain” or asymmetric routing condition is present
Common asymmetric routing conditions:
(a) Three arm – Third for management
(b) Lots of static routes to be maintained

4. © 2014 Citrix. Confidential.4
Common “Three Arm” Deployment where MBF is used
Netscaler
ISP
Admin Enclave Server Farm
VIP
SNIPNSIP
VPN/LB traffic
DataAdmin

5. © 2014 Citrix. Confidential.5
Common issues seen with MBF
- Intermittent performance issues
- Packet loss in high volume environments
- Same packet loss can result in dropped ICA connections for ICA-Proxy through Netscaler Gateway vserver
- Periodic UI access issues
- Effectively disables L3 routing in favor of L2-type forwarding

6. © 2014 Citrix. Confidential.6
Trunking on Non-Link Aggregation Links
Common issues seen when misconfigured:
- Dropped packets due to promiscuous vlan drops.
nsconmsg -K newnslog -s disptime=1 -g nic_ -d current | grep 1/2 | grep drop | more
18 0 6874714506 35162 5033 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:17 2014
25 0 6862206790 35106 5025 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:17 2014
73 0 6874743016 28510 4072 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:24 2014
80 0 6862235244 28454 4064 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:24 2014
127 0 6874773155 30139 4306 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:31 2014
134 0 6862265285 30041 4292 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:31 2014
185 0 6874837857 64702 9243 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:38 2014
The solution in this case is to ensure that the switchport settings only reflect the same allowed vlans that the netscaler is
configured to send or receive traffic for

7. © 2014 Citrix. Confidential.7
Trunking on Non-Link Aggregation Links cont.
Design considerations – vlan number assignments and tagging:
Native vs. Default VLAN
Trunking vs Tag All
Tagging the native vlan
- For differentiation
- Tagging single vlan
- Network policy might dictate a different native vlan.

8. © 2014 Citrix. Confidential.8
Changing the NSVLAN
Changing the nsvlan is considered an extreme case and should be used as a last resort
Common Reasons for changing nsvlan:
a) Network operations policy
b) Separate requirement for native non-tagged vlan
Pitfalls of changing the nsvlan:
a) Requires reboot
b) NSIP VLAN is impacted – special considerations needed
c) Special order of operations may be needed

9. © 2014 Citrix. Confidential.9
MAC Moves & Layer2 Loops
“The Rule of Bridging” – ARPs will be flooded out all connected interfaces (on the same
vlan) except the one on which the request was received.
- L2 Mode considerations – Frames forwarded. “Very” rarely used! (see next slide)
- Common issues seen resulting from MAC moves /L2 loops.
- Poor performance
- Application traffic throughput on the Netscaler ADC may not be as timely as expected
- Frequent packet retransmissions causing latency
- Dropped packets on interface(s)
- Packet drops incrementing on the interface stats (show screenshot/text output)
-Common topologies that cause these issues
- Multiple interfaces connected into the same vlan
-Design considerations to alleviate MAC Moves/L2 loops
- Disable multiple interfaces that are connected to the same vlan
- Implement Link-Aggregation (LA) channel to manage the traffic for multiple interfaces in same
vlan

10. © 2014 Citrix. Confidential.10
L2 or L3?

11. © 2014 Citrix. Confidential.11
Common L2 Loop Scenario
Network
Switch
Netscaler
Int 1/1
Int 1/2
fa 0/3
fa 0/4
1
2
3
1. Netscaler sends ARP for destination MAC for a given
IP. Since both int 1/1 and 1/2 are connected, the
request is flooded out of both interfaces
2. Switch/End device receives both requests and
responds with destination MAC for given IP via both
connected switchports
3. Netscaler receives both responses and caches the first
response, then sends the request to the destination
MAC for the given IP in that first response.
4 4. The Network switch, since it has the MAC address in
it’s ARP table for the Netscaler SNIP as being accessible
via fa0/4 due to a previous transaction and ARP table
entry, it responds via this interface.
The Netscaler drops the packet in the response since it is
aware of the destination having been mapped via int 1/1
(Bridge Table)

12. © 2014 Citrix. Confidential.12
Detecting L2 Loops - Overview
The scenario in the previous slide is very common when more than one
interface in the same vlan is connected to the switch with both switch ports
also configured on the same vlan.
The result from the received packet arriving on the assumed incorrect
interface is a packet drop. Packet drops are detectable in a few ways
through debugging the Netscaler logs.

13. © 2014 Citrix. Confidential.13
Detecting L2 Loops – From Shell debugging
The NSCONMSG utility is very powerful and frequently used by Citrix Support in gathering
useful information to help determine root cause for system issues.
In this example we see that we have used ‘grep’ to parse the current newnslog file for the
strings “nic_err” and “drop” simultaneously.
*Note that even in a healthy environment, packet drops occur all the time however when
interesting counters such as ‘nic_err_vlan_promisc_tag_drop’ increment, ‘so do
nic_err_dropped_pkts’
root@ns# nsconmsg -K newnslog -d stats | grep nic_err | grep drop
1854 0 91 nic_err_dropped_pkts interface(1/1)
1855 0 91 nic_err_dropped_pkts interface(1/2)
1866 0 0 nic_err_tx_dropped
1886 0 0 nic_err_congested_pkts_dropped
1887 0 0 nic_err_congestionlimit_pkts_dropped
1914 0 0 nic_err_la_frame_collect_drops
1915 0 0 nic_err_la_tagged_bpdu_drops
1916 0 73 nic_err_vlan_promisc_tag_drops
1917 0 0 nic_err_la_untagged_pkt_drops
1919 0 0 nic_err_rl_pkt_drops
1920 0 0 nic_err_rl_rate_pkt_drops
1921 0 0 nic_err_rl_pps_pkt_drops
4438 0 0 allnic_err_rl_rate_pkt_drops
4439 0 0 allnic_err_rl_pps_pkt_drops
root@ns#

14. © 2014 Citrix. Confidential.14
More L2 Issues – From the Shell debugging
The below counter tracks how often a given MAC is being detected via different ports. In normal operation a MAC address should only be
seen only on one port. A MAC moving often is a sign of a loop – This can cause both failovers and performance issues.
nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg
30433 0 25681657 nic_tot_bdg_mac_moved interface(1/3)
30434 0 25725446 nic_tot_bdg_mac_moved interface(1/2)
30435 0 25556085 nic_tot_bdg_mac_moved interface(1/1)
30436 0 28858167 nic_tot_bdg_mac_moved interface(0/1)
30437 0 27641933 nic_tot_bdg_mac_moved interface(0/2)
This then results in the NetScaler eventually muting the interfaces to try and end the loop.
nsconmsg -K newnslog -d statswt0 | grep nic_err
30438 0 467 nic_err_bdg_muted interface(1/3)
30439 0 445 nic_err_bdg_muted interface(1/2)
30440 0 447 nic_err_bdg_muted interface(1/1)
30441 0 605 nic_err_bdg_muted interface(0/1)
30442 0 578 nic_err_bdg_muted interface(0/2)
*Generally issues can be seen with the ‘-d statswt0 | grep nic_err’ switch in nsconmsg utility

15. © 2014 Citrix. Confidential.15
High Availability
What events trigger (an unexpected) failover?
Network issues = Missed Heartbeats = failover
Tight loops on peer/crash/reboot
SSL Card failures
Interface failures
Heartbeats are sent/received on all interfaces by default
Regardless of hamon setting
Use UDP port 3003
Common network issues leading to missed heartbeats:
VLAN Tagging Mismatch
nsVLAN mismatch – Native vlan for heartbeats and NSIP

16. © 2014 Citrix. Confidential.16
What is exchanged in a Heartbeat?

17. © 2014 Citrix. Confidential.17
Verifying Heartbeat exchange
root@ns# nsconmsg -K newnslog -g ha_tot_pkt -s disptime=1 -d current | more
Index rtim totalcount-val delta rate/sec symbol-name&device-no&time
0 3560 13 13 3 ha_tot_pkt_rx Sat May 17 13:48:48 2014
1 0 12 12 3 ha_tot_pkt_tx Sat May 17 13:48:48 2014
2 7000 50 37 5 ha_tot_pkt_rx Sat May 17 13:48:55 2014
3 0 49 37 5 ha_tot_pkt_tx Sat May 17 13:48:55 2014
4 6998 101 35 5 ha_tot_pkt_rx Sat May 17 13:49:06 2014
5 0 100 35 5 ha_tot_pkt_tx Sat May 17 13:49:06 2014
6 7000 136 35 4 ha_tot_pkt_rx Sat May 17 13:49:13 2014
7 0 135 35 4 ha_tot_pkt_tx Sat May 17 13:49:13 2014
8 7001 171 35 4 ha_tot_pkt_rx Sat May 17 13:49:20 2014
9 0 170 35 4 ha_tot_pkt_tx Sat May 17 13:49:20 2014
10 7000 206 35 5 ha_tot_pkt_rx Sat May 17 13:49:27 2014

18. © 2014 Citrix. Confidential.18
Tagging of HA packets - Scenarios
Scenario 1
NSVLAN is 1 (default)
interface 1/2 is bound to VLAN 2
Interface 1/3 is bound to VLAN 3
Result: High Availability packets flow as untagged on the 1/2 and 1/3 interfaces on the native VLAN.
Scenario 2
NSVLAN is 1 (default)
interface 1/2 is bound to VLAN 2, which is configured with –tagall ON
Interface 1/3 is bound to VLAN 3, which is configured with –tagall OFF (the default)
Results: HA packets flow on 1/2 as tagged with a VLAN ID of 2, and untagged on the 1/3 interface.
Scenario 3
NSVLAN is 3 (non default)
interface 1/2 is bound to VLAN 2
interface 1/3 is bound to VLAN 3
Result: HA packets flow as tagged packets on interface 1/3 only and don’t flow on Interface 1/2

19. © 2014 Citrix. Confidential.19
HA related events
Secondary Primary
root@70# nsconmsg -K newnslog -d event | grep node
0031 0 PPE-0 remote node 10.72.137.50: Primary Tue May 20 21:48:12 2014
1032 0 PPE-0 self node 10.72.137.70: SYNC start Tue May 20 21:48:15 2014
1203 0 PPE-0 self node 10.72.137.70: SYNC complete Tue May 20 21:48:20 2014
1210 0 PPE-0 remote node 10.72.137.50: DOWN Tue May 20 21:51:51 2014
1211 0 PPE-0 self node 10.72.137.70: Claiming Tue May 20 21:51:51 2014
1212 0 PPE-0 self node 10.72.137.70: Primary Tue May 20 21:51:51 2014
1221 0 PPE-0 remote node 10.72.137.50: INIT Tue May 20 21:53:01 2014
1223 0 PPE-0 remote node 10.72.137.50: UP Tue May 20 21:53:02 2014
root@50# nsconmsg -K newnslog -d event | grep node
946 0 PPE-0 remote node 10.72.137.70: UP Tue May 20 21:53:02 2014
1017 0 PPE-0 self node 10.72.137.50: UP Tue May 20 21:53:03 2014
1018 0 PPE-0 self node 10.72.137.50: HA license check result: MATCH Tue May 20 21:53:03 2014
1019 0 PPE-0 remote node 10.72.137.70: Primary Tue May 20 21:53:03 2014
1020 5 PPE-0 self node 10.72.137.50: SYNC start Tue May 20 21:53:06 2014
1191 0 PPE-0 self node 10.72.137.50: SYNC complete Tue May 20 21:53:10 2014
This is why Tech support asks for Collectors from both Primary and Secondary units for HA related issues.
Having the correct date and time set on the devices is important to simplify correlation.
#date +val YYMMDDHHMM

20. © 2014 Citrix. Confidential.20
HA Best Practices
Use VMACs
Disable unused interfaces (not just hamon on those interfaces)
rpc passwords should match – Mismatch leads to sync/prop failures
“Warning: Unable to establish connection with the secondary. Command propagation
failed”
Ensure certs and files match on peers – to avoid certificate binding failures
>sync ha files
Ensure both devices are running same Versions and use the same hardware/license
Enable failsafe mode to avoid a complete outage when both nodes have an issue
> set ha node -failSafe ON
Use Failover Interface Sets (FIS) - if interface bandwidth is not an issue
>add fis testfis
>bind fis testfis 1/1 1/2

21. © 2014 Citrix. Confidential.21
Gratuitous ARP
Possible Firewall issues?
> set network L2param –garpReply enabled

22. © 2014 Citrix. Confidential.22
VMACs - A better solution
– Simple config
> add vrID 1
> bind vrID 1 -ifnum 1/1
– Instantaneous failovers
– Configuration propagated to secondary
Considerations:
Just be sure switch supports it
You might need to flush arp once
> send arp all

23. © 2014 Citrix. Confidential.23
LACP
Aggregating Multiple ports into a High Speed Link
• 802.3ad Compliant
• Aggregation of throughput
• Better Resiliency
• Channels are dynamically created using the port key when lacp is enabled on an interface
Sample configuration:
>set interface 1/1 -lacpmode active -lacpKey 1 – lacpTimeout long
lacpkey value forms the basis for the LA/x notation
active/passive decides whether NetScaler sends LACPDUs or not
lacptimeout decides how often they are sent
Note:
- LACP Configs are neither propagated nor synchronized
- Once an interface is bound to a channel, channel parameters take precedence i.e. interface VLAN config no longer applies

24. © 2014 Citrix. Confidential.24
LACP continued..
LACP on SDX
Relies on “NIC Bonding” feature of XS
Switch and SDX are aware of the bond
LACP PDUs are exchanged
Recommended Versions
 SDX: 10.1 or higher
 NetScaler: 10.1 or higher
 XenServer: 6.1 or higher
What is now possible:
• Sharing across instances
• LA for Management Interfaces
Channel creation will require reboot on instances if the member interfaces are not already provisioned.
Best Practices –
Only create Channels from SVM – Not from XenServer not from within Instance.
Ensure of equal speed – or lowest speed will be chosen.
Create channel first before plugging in cables.
Similarly disconnect the cable before removing a port from the channel.

25. © 2014 Citrix. Confidential.25
Logs on XenServer
tail –f /var/log/messages | grep bond
LACP going down
[root@localhost ~]# cat /var/log/messages | grep bond
May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03991|bond|INFO|interface eth8: link state down
May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03992|bond|WARN|interface eth8: disabled
May 17 09:38:00 netscaler-sdx ovs-vswitchd: 03993|bond|INFO|bond bond0: active interface is now eth6
May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03994|bond|INFO|interface eth6: link state down
May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03995|bond|WARN|interface eth6: disabled
May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03996|bond|WARN|bond bond0: all interfaces disabled
May 17 09:38:02 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2
up 0 up_old 2
LACP back up
[root@localhost ~]# cat /var/log/messages | grep bond
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03997|bond|INFO|interface eth6: link state up
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03998|bond|INFO|interface eth6: will be enabled if it stays up for 31000 ms
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 03999|bond|INFO|bond bond0: active interface is now eth6, skipping remaining 31000 ms
updelay (since no interface was enabled)
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04000|bond|WARN|interface eth6: enabled
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04001|bond|INFO|interface eth8: link state up
May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04002|bond|INFO|interface eth8: will be enabled if it stays up for 31000 ms
May 17 09:38:57 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2
up 1 up_old 0

26. © 2014 Citrix. Confidential.26
On the SVM
/var/mps/log/mps_service.log
LA/1 configured in the SVM consists of 10/1 and 10/3:
Tuesday, 14 Jan 14 14:36:24.988 +0100 [Debug] Json Web Content:
{"host_interface":{"mapped_port":"LA/1","sync_operation":false,"channel_type":"LACP","channel_throughput"
:"0","channel_bandwidth_high":"0","channel_bandwidth_normal":"0","channel_interface_list":"10/1,10/3","st
atic_channel_state":true,"lacp_channel_time":"SLOW","channel_ha_monitoring":true,"channel_tag_all_vlans":
true,"channel_alias":""}}

27. © 2014 Citrix. Confidential.27
Logs – on Instance/vpx/mpx
# tail -f ns.log | grep interface
May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICMIGRATE 55384 0 : NIC
"interface(10/1)" State Migrated
May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55385 0 : Device
"interface(10/1)" - RX state PORT_DISABLED
May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55386 0 : Device
"interface(10/1)" - RX state LACP_DISABLED
May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55387 0 : Device
"interface(10/2)" - State DOWN
May 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55388 0 : Device
"interface(LA/1)" - State DOWN
In a healthy case:
root@VPX01# nsconmsg -K newnslog -g rx_lacp -s disptime=1 -d current
Index rtime totalcount-val delta rate/sec symbol-name&device-no&time
0 1462998 737728 8 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:13 2014
1 0 737664 7 1 nic_tot_rx_lacpdus interface(10/2) Sat May 17 12:09:13 2014
2 7000 737735 7 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:20 2014
When it goes down:
36 77000 737913 6 0 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:13:25 2014

28. © 2014 Citrix. Confidential.28
Show channel – what to look at?
> show channel LA/1
1) Interface LA/1 (802.3ad Link Aggregate) #5
flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, 802.1q>
MTU=1500, native vlan=1, MAC=e2:08:d2:15:b2:05, uptime 0h00m07s
Requested: media NONE, speed NONE, duplex NONE, fctl NONE,
throughput 0
Actual: throughput 2000
RX: Pkts(5099181) Bytes(1302893502) Errs(0) Drops(115221) Stalls(0)
TX: Pkts(7816907) Bytes(1266490416) Errs(0) Drops(742) Stalls(0)
NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0)
Bandwidth thresholds are not set.
LA mode: AUTO
10/1: FIBER-1000-FULL-NONE UP 0h00m14s
PortID=(32768,1), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 266)
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
10/2: FIBER-1000-FULL-NONE UP 0h00m14s
PortID=(32768,2), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED
<Active, Long timeout, Agg, Sync, Collecting, Distributing>
Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 267)
<Active, Long timeout, Agg, Sync, Collecting, Distributing>

29. © 2014 Citrix. Confidential.29
Trace
> show lacp
Actor SystemID: (32768, 00:e0:ed:26:dc:cd)
Verify if PDUs are being correctly exchanged
Question: do keys need to match on both devices?
No – But on a given devices all interfaces that part of the same LA channel need to have the same
key

30. © 2014 Citrix. Confidential.30
Next Webinar: June 2014
Title: Best practices for implementing, administering, and troubleshooting
XenDesktop 7.5
Description: Citrix XenDesktop introduced a number of new concepts and
processes for Desktop Administrators. Understanding these advancements and
their effect on is key to a stable XenDesktop environment.
This session will discuss core deployment and configuration concepts and
considerations and provide proven practices for troubleshooting the top three
XenDesktop issues.
When: June 24th & 25th
June 24th – Register Now
June 25th – Register Now

31. © 2014 Citrix. Confidential.31
Fuel your talent with continuous learning.
Citrix Education offers the following technical training for Networking professionals:
CNS-205: Citrix Netscaler 10 Essentials and Networking
CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam
CNS-301: Citrix NetScaler 10 Advanced Implementation
Visit (bit.ly/05Webinar) to save 10% off through June 30*
*Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.

32. © 2014 Citrix. Confidential.32
Build your Citrix skills in your personal virtual sandbox
Play in your own Virtual Sandbox with Learning Labs from Citrix Education. With
your purchase, you’ll receive your own dedicated server with access to the seven
most popular Learning Labs from Synergy. Featured labs include:
• NetScaler, the Enterprise Security Swiss Army Knife
• Front-Ending and Load Balancing XenDesktop and XenApp with NetScaler
• Enhancing Visibility of Applications with NetScaler Insight Center
http://training.citrix.com/cms/education/promotions/learninglabs/

33. © 2014 Citrix. Confidential.33
New Citrix Practice Exams
Accelerate Your Path to Certification
Available on training.citrix.com ($39 each):
CPE-350 – Citrix NetScaler 10 Essentials and
Networking Practice Exam
CPE-300 – Deploying XenDesktop 7 Solutions Practice
Exam
CPE-A22 – Citrix XenApp 6.5 Advanced Administration
Practice Exam
http://training.citrix.com/cms/index.php/promotions/prac
ticeexams/

34. © 2014 Citrix. Confidential.34
WORK BETTER. LIVE BETTER.