Your SlideShare is downloading. ×
0
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
In-depth Troubleshooting on NetScaler using Command Line Tools
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

In-depth Troubleshooting on NetScaler using Command Line Tools

35,054

Published on

Webinar recording - https://www1.gotomeeting.com/register/753997104 …

Webinar recording - https://www1.gotomeeting.com/register/753997104

Citrix NetScaler has a rich Web-based management suite of tools available. To dig deep troubleshooting NetScaler, sometimes it’s best to roll up your sleeves and dig out the command line!
The goal of this session is to demystify some useful command line tools and provide a tactical approach to troubleshooting of NetScaler.
In this session we will demonstrate troubleshooting approaches using the command line and many tips for common issues seen in customer deployments.

In this session you will learn about:
· Differences between NetScaler kernel and BSD
· Processes and disk layout
· Look up stats and statuses
· Troubleshoot using various different logs
· Use counters to help identify issues

Published in: Technology
0 Comments
24 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
35,054
On Slideshare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
2,083
Comments
0
Likes
24
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Access to the appliance CLI is through the serial console, or by connecting with SSH to the NetScaler management IP /SNIP with Management Access enabled.This lands you in the NetScaler CLI – which has custom commands. Some TCSH-like shortcuts can apply.TCSH shortcuts like CTRL-A brings you to the start of the line, and CTRL-E brings you to the end. Most day to day NetScaler commands will begin with the following verbs: show Display information about an entity.add Create an entity.remove Delete an entity.set Change/modify an entity.enable Turn a feature or setting ON.disable Turn a feature or setting OFF.force Used in High Availability to Sync & Failover.bind Create a relationship between two entities.unbind Remove a relationship between two entities.
  • >set cli prompt %u@%h-%T>set cli mode -color ON>help set cli promptAutocomplete is your best friend… <tab> & ? %! - will be replaced by the history event number %u - will be replaced by the NetScaler user name %h - will be replaced by the NetScaler hostname %t - will be replaced by the current time %T - will be replaced by the current time (24 hr format) %d - will be replaced by the current date %s - will be replaced by the node state
  • There is another shell (BASH) that is only used for file handling – never to configure the NetScaler. Note the different prompts.> shell to enter BASH shell in BSD#Ctrl+D to exit BASH or type exit
  • All of the UNIX command goodness in FreeBSD!
  • Show feature provides an output that allows us to quickly identify which features are enabledNotice these common features are disabledLBICREWRITE This output almost certainly would indicate a misconfiguration2. At a bare minimum most deployments will have at LEAST “load balancing” enabled. If LB is off, you can see issues where a vserver wont come up, or it will only utilize 1 service. If you have any inexplicable errors where features just don’t work at all despite a proper config it is almost ALWAYS that the feature has been disabled. This is a common mistake, I make it frequently.
  • In this case we specified a vserver name, this is the most desirable way to execute this command as it shows you the most detail, namely the state of your bound services.Key difference between this example and the generic example is the bound service summaryWe now know that this vserver is down because the only service bound to it is down. A vserver will go down when all of its bound services are down.
  • Two different directories. One for cores and one for crashes
  • Pitboss controls the processes on a NetScalerIf the pitboss detects a process failing it will try to restart itThe nsppe process is in userland so can be “warm” startedIf a process fails 5 times, on the sixth failure the NetScaler will undergo a full reboot
  • Show commands primarily provide configuration and status information about the system or specified entityShow commands for SYSTEM infoShow node, talk about various bad states. Essential to troubleshooting HA issuesConsolidate show version, show feature, show mode into “show info”Show license3. Show commands for a vserver and service:Shlbvserver/shcsvserver, generic command vs specific referencing an entitySh service, just like shlbvserver, generic vs specific outputShpersistencesession, helpful for tracking a persistent session without a traceShconnectiontable, large output but also useful for connection tracking4. Other show commandsSh route, ship, shShdnsaddrec –type proxy, useful for debugging cached DNS records
  • Why is this node down?2. Things to notice here:Node State: NOT UPMaster State: Secondary3. Its down because unused interfaces are enabled and not receiving heartbeats. If we compare which interfaces are enabled to which interfaces are not receiving heartbeats we can determine that 1/7 is the only interface receiving heartbeats.4. We can correct this by disabling all interfaces except for 1/7 (which is the only interface in use).5. Notice the partner node, he is secondary but has only one interface enabled. Its *NOT* failing for the same reason, notice the Node State: STAYSECONDARY setting.6. So we know the node we are on is down because its interfaces are misconfigured, and the partner node is forced to stay secondary.
  • BSD/NS relationship:BSD controls disk/time slicing, primarily a bootloader for NS kernelLogs are written by BSDLogs are rolled by BSDNetScaler runs inside of FreeBSD, nsppeConsumes majority of user land processer (99%)NetScaler controls NICs/packet processing/etc.Key processes:NsppeNsaaadNsconfigdNsauthdNslog.shKey processes:NsnetsvcNsumondNsconmsgDynamic routing processes
  • Taken on an MPX 15500
  • 1. Things to notice here:Various hit countersClient conn vs Server connSpilloverService stats (ttfb, transactions)
  • A significant portion of the information we care about is already available from “stat lbvserver [name]”TTFBSurge QueueThings to notice:TTFBSurge queue
  • 1. Discuss an overview of each mount point “/var” contains historical data in the form of logs. This is one of the first places to look when trying to troubleshoot a NetScaler issue./var/log is the “traditional” location for logs in Unix/Linux operating systems/var/nslog contains NetScaler specific logs“/flash” contains configuration and customizationsrc.netscalerAny modified configs from /etcUser monitorsKernel itself“/” contains the OSRam driveAvoid writing anything to it, no reason to2. What happens if the components fail?Device can operate without /var, but will not be able to logDevice cannot boot without /flashDevice cannot boot without /
  • 1. Discuss an overview of each mount point “/var” contains historical data in the form of logs. This is one of the first places to look when trying to troubleshoot a NetScaler issue./var/log is the “traditional” location for logs in Unix/Linux operating systems/var/nslog contains NetScaler specific logs“/flash” contains configuration and customizationsrc.netscalerAny modified configs from /etcUser monitorsKernel itself“/” contains the OSRam driveAvoid writing anything to it, no reason to2. What happens if the components fail?Device can operate without /var, but will not be able to logDevice cannot boot without /flashDevice cannot boot without /
  • /var/nslog/dmesg.boot/var/nslog/dmesg.prev/var/nslog/dmesg.last
  • 1. THIS IS AN INCOMPLETE LIST, THESE ARE THE MOST COMMON FILESNs.log (INCLUDING BUT NOT LIMITED TO):contains NSCLI commandscontains syslog messages useful for reconstructing user input and event timelinesBy far the most informative file containing the most information in one placeMessages contains system events authentication messages system startup messages commands executed under shellconsole messages
  • 1. THIS IS AN INCOMPLETE LISTNewnslogCurrent live fileNewnslog.*.gzRolled log filesNs.log - contains newnslog related eventsNsumond.log – contains log output for user land monitors using KASNslog.nextfile – next newnslog file to be written
  • If a problem hasn’t been solved by what we have done so far:Logfile analysisShow commandsStat commandsThen the next step is to get even further debug information from Nsconmsg. 2. Nsconmsg logs all of the statistics we have seen so far, and additionally there are literally thousands of other counters we don’t see in the NSCLI which are logged. All of these counters are recorded every 7 seconds and written to the file in a binary format.Nsconmsg is one of the primary tools support uses to debug issues3. Some things we can get from newnslog are:Events: UP/DOWN messages for entities (vservers, services), HA events, interface events, etc. Most of these events are also logged to syslog.Console messages: Mostly BSD messages, disk write errors, etc
  • If a problem hasn’t been solved by what we have done so far:Logfile analysisShow commandsStat commandsThen the next step is to get even further debug information from Nsconmsg. 2. Nsconmsg logs all of the statistics we have seen so far, and additionally there are literally thousands of other counters we don’t see in the NSCLI which are logged. All of these counters are recorded every 7 seconds and written to the file in a binary format.Nsconmsg is one of the primary tools support uses to debug issues3. Some things we can get from newnslog are:Events: UP/DOWN messages for entities (vservers, services), HA events, interface events, etc. Most of these events are also logged to syslog.Console messages: Mostly BSD messages, disk write errors, etcSystem statistics: all counters are captured every 7 seconds. We can view lb statistics, system statistics.System counters: mostly for software debugLoad balancing counters: ConLb shows us detailed load balancing statistics
  • This is a freshly booted device so we see a variety of messages here:Service up/down eventsInternal services coming upBootup messagesCPUCONFIG STARTUbsec_0 UPInterface eventsDisabled interfacesHA eventsVersion mismatch messageRemote node UP
  • -d consmsg provides output on any BSD console messagesThis console output consists only of bootup output, but you may also see things like: IP conflicts here NIC errors (duplex issues) lack of file handlers OS errors, etc.
  • –d oldconmsg provides CPU and memory utilization outputI will use this to quickly establish trends in CPU/MEM utilization (just let the output scroll and watch mem/cpu and see if they increase steadily).
  • 1. –s ConLB=2 provides detailed debug output regarding load balancingSpecific detail on the lbvserver, the types of traffic its handling, Specific detail on services and the traffic THEY are subsequently handling.2. Most interested in the following sections: Hits, particularly Pers (persistence status to explain hits) Pkt (packet stats) Conn (Current server, Maximum server, Open Established, Established, Reuse Pool, Surge queue)
  • 1. HDD issues – primary failure is that logging fails, /var is missingCheck df, are any drives missing?Check dmesg, are there any drives missing or errors?Run fsck on the drive to check for errorsAttempt to re-mount the drive2. Flash issues – config save issues, sync fails, device fails to bootThe box wont boot without flash, so if the NetScaler is running the device mounted OK.Check df, is the /flash missing or full?Check dmesg, is flash missing or getting any errors?Run fsck on the drive to check for errors3. Memory starvation – dropped session, cant allocate memory for other tasks (CPU profile, etc.)Feature memory allocation, IC, APPFW, TCPBUFFERINGConMEM4. CPU overutilizationSNMP pollingNewnslogs roll USIP? CMP?Anything in userland?
  • http://support.citrix.com/article/CTX109304
  • Perl script/netscaler/showtechsupport.plAlso available in the UI: System > Diagnostics > Generate support file
  • Perl script/netscaler/showtechsupport.plAlso available in the UI: System > Diagnostics > Generate support file
  • If we haven't been able to solve the issue with stat and show commands, cant find anything in the logs, the next step might be to get a sniff. Personally I like to get a sniff first and work forward from there, assuming the problem warrants it.2 kinds of traces,Nstrace – preferable, gives extended NetScalerdataNstcpdump – all NIC wrapper for tcpdump. quick, familiar, uses tcpdump syntax. 2. Common syntax -sz 0, or with a filter. Truncated traces are almost always worthless.3. 9.0 adds filter support to nstrace. This is the preferred acquisition method as we write extended data to the capture useful for session tracking, NIC tracking, operation tracking, etc. Requires custom build of Wireshark to view.If you use a filter, use the –link option to capture all of the other related traffic on that session.Operators: ==, !=, etc.4. Nstcpdump good for looking at live traffic, or for when traces need to be viewed with a standard Wireshark build. All we are doing is an all NIC trace when invoking tcpdump via nstcpdump.sh. Since we now have filtering in nstrace, it is much more preferable to use that method to capture instead of nstcpdump. Filtering in tcpdump is expensive in userspace due to the way we provide the data to tcpdump.Common syntax –X to print payload in ASCIICommon syntax –w to write the capture to a fileTcpdump not effective due to the way packets are captured from the kernel
  • Cover various switches, the most common syntax will be “nstrace.sh –sz 0”-sz specifies size of data to be captured-nf number of files in a cycle-tcpdump, writes file in tcpdump format (doesn’t need special Wireshark build)-filter, specifies the filter to apply-link, link associated traffic from filter
  • type qualifiers say what kind of thing the id name or number refers to. Possible types are host, net and port. E.g., `host foo', `net 128.3', `port 20'. If there is no type qualifier, host is assumed. dir qualifiers specify a particular transfer direction to and/or from id. Possible directions are src, dst, srcordst and srcanddst. E.g., `src foo', `dst net 128.3', `src or dst port ftp-data'. If there is no dir qualifier, srcordst is assumed. For `null' link layers (i.e. point to point pro- tocols such as slip) the inbound and out-bound qualifiers can be used to specify a desired direction. proto qualifiers restrict the match to a particular protocol. Possible protos are: ether, fddi, ip, arp, rarp, decnet, lat, moprc, mopdl, tcp and udp. E.g., `ether src foo', `arp net 128.3', `tcp port 21'. If there is no proto qualifier, all protocols consistent with the type are assumed. E.g., `src foo' means `(ip or arp or rarp) src foo' (except the latter is not legal syntax), `net bar' means `(ip or arp or rarp) net bar' and `port 53' means `(tcp or udp) port 53'. The –r option must not be given to the script because internally we are supplying ‘-r –‘ option as a default entry, so that TCPDUMP reads the traces from the standard input. So it does look logical, not to supply a ‘-r’ option from the CLI.The –i option must be avoided too. This is because TCPDUMP listens only on 1 interface at a time. In our case we are dumping packets that arrive on all the interfaces on to the standard output. If you want to view per interface packets, ‘nstrace.sh’ can be used with the ‘-tcpdump 1 -nic 1’ option as input.The ‘-F’ option is not encouraged. If this is to be put to full use, use ‘nstrace.sh’ or ‘nstcpdump.sh –w <file>’ and do an offline filtering using TCPDUMP directly. This is because we felt that, since you are using the standard output to dump the traces, it would not be a wise idea to use ‘complex’ filter expressions. So the better idea would be to store in a file the entire trace and view it using Ethereal or graphical packet analyzer and based on the fields you are interested, generate your ‘expression’ file and use TCPDUMP directly on the shell with the ‘-F’ option to filter the trace captured.-sSnarfsnaplen bytes of data from each packet rather than the default of 68 (with SunOS's NIT, the mini- mum is actually 96). 68 bytes is adequate for IP, ICMP, TCP and UDP but may truncate protocol infor- mation from name server and NFS packets (see below). Packets truncated because of a limited snapshot are indicated in the output with ``[|proto]'', where proto is the name of the proto- col level at which the truncation has occurred. Note that taking larger snapshots both increases the amount of time it takes to process packets and, effectively, decreases the amount of packet buffering. This may cause packets to be lost. You should limit snaplen to the smallest number that will capture the protocol information you're interested in. -T Force packets selected by "expression" to be interpreted the specified type. Currently known types are rpc (Remote Procedure Call), rtp (Real-Time Applications protocol), rtcp (Real-Time Applications control protocol), vat (Visual Audio Tool), and wb (distributed White Board).
  • Nstcpdump is just a wrapper for tcpdump, standard syntax applies.
  • # nsapimgr-K <nstrace-file> -s tcpdump=1 -k <tcpdump-file>Offline conversion of traces, those are in our NSTRACE format, to TCPDUMP format.<nstrace-file>: the file, which is in the NSTRACE format.<tcpdump-file>: the file into which the traces are to be converted and dumped in the TCPDUMP format.0=nstrace-format (default)
  • Two different directories. One for cores and one for crashes
  • Pitboss controls the processes on a NetScalerIf the pitboss detects a process failing it will try to restart itThe nsppe process is in userland so can be “warm” startedIf a process fails 5 times, on the sixth failure the NetScaler will undergo a full reboot
  • Don’t hurt meCaslon Ampersand!
  • Transcript

    • 1. Andrew Sandford Senior Readiness Specialist, Worldwide Support Readiness EMEA Citrix Support Secrets Webinar Series In-depth Troubleshooting on NetScaler using Command Line Tools 27 March 2014
    • 2. © 2014 Citrix | Confidential – Do Not Distribute
    • 3. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 4. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 5. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 6. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 7. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 8. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 9. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 10. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 11. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 12. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute netscaler.descr: NetScaler Virtual Appliance 3G netscaler.num_pe_running: 1 netscaler.version: NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27 netscaler.model: 3000 netscaler.vmpe_max_cpus: 2 netscaler.nCore: 1
    • 13. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 14. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 15. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 16. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 17. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 18. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 19. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 20. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute 50185 root 1 44 -52 814M 815M CPU1 1 20.2H 100.00% NSPPE-00
    • 21. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 22. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 23. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 24. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 25. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS State: UP Client Idle Timeout: 180 sec Down state flush: ENABLED Disable Primary Vserver On Down : DISABLED Appflow logging: ENABLED Port Rewrite : DISABLED No. of Bound Services : 1 (Total) 1 (Active) Configured Method: ROUNDROBIN Mode: IP Persistence: NONE Vserver IP and Port insertion: OFFWarning: Feature(s) not enabled [LB]
    • 26. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute 1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1
    • 27. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute 1) Monitor Name: mon-http-ecv State: UP Weight: 1 Passive: 0 Probes: 14887 Failed [Total: 124 Current: 0] Last response: Success - Pattern found in response. Response Time: 10.220 millisec State: UP
    • 28. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 29. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 30. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 31. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 32. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 33. © 2014 Citrix | Confidential – Do Not Distribute Show commands System show node show info show license Vserver/Service show lb vserver show cs vserver show service show persistencesession show connectiontable IP related show route show ip show dns addrec -type proxy Diagnostic show techsupport
    • 34. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute 2) Node ID: 1 IP: 192.168.1.45 Node State: STAYSECONDARY Master State: Secondary Fail-Safe Mode: OFF INC State: DISABLED Sync State: ENABLED Propagation: ENABLED Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1 Disabled Interfaces : None HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1 Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1 Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1 SSL Card Status: UP Local node information: Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1 > Node State: NOT UP Master State: Secondary Master Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1 Disabled Interfaces : None HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1 Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1 Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1 Node State: STAYSECONDARY Master State: Secondary
    • 35. © 2014 Citrix | Confidential – Do Not Distribute NetScaler Processes Process Description Process Description nsppe NetScaler Packet Engine nsfsyncd Sync bookmarks and SSL certificates nsaaad RBA and SSL VPN External Auth nsnetsvc Used by the GUI for config changes nsconf Writes the ns.conf file nsumond Runs the scriptable monitors nslog.sh Controls Logging for newnslog nsconmsg Controls writing of newnslog nssync HA sync nscollect Statistics gathering for historical reporting nsreadfile Used to read SSL Cert Files imi/ripd/ ospfd/bgpd Routing processes nscrlrefresh SSL CRL list update
    • 36. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute System Entities Protocols
    • 37. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 38. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 39. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 40. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute > stat interface 1/1 Interface [1/1]: Interface StateUP Link uptime 00:40:21 Link downtime00:00:00 Throughput Statistics Rate (/s) Total Bytes received42393 54497294 Bytes transmitted 2584 20222135 Packets received 629 710246 Packets transmitted 47 69066 Packet Statistics Rate (/s) Total Multicast packets 18 41219 NetScaler packets 85 98954 LACP Statistics Rate (/s) Total LACPDUs received 0 0 LACPDUs transmitted 0 0 Error Statistics Rate (/s) Total Error packets received (hw) 0 0 Error packets transmitted (hw) 0 0 Inbound packets discarded (hw) 0 0 Outbound packets discarded (hw)0 0 Packets dropped in Rx (sw) 539 599904 Packets dropped in Tx (sw) 0 0 NIC hangs -- 0 Status stalls -- 0 Transmit stalls -- 0 Receive stalls -- 0 Error-disables -- 0 Duplex mismatches -- 0 Link re-initializations -- 0 MAC moves registered 0 0 Times NIC became muted -- 0
    • 41. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute Non-authoritative entries 0 Authoritative entries 98 Error Statistics Nonexistent domain 102359 Response class unsupported 0 Invalid query format 0 Stray answers 0 Incorrect RD length 0 Requests refused 0 Response type unsupported 0 Query class unsupported 0 Invalid response format 0 No answer responses 102334 Multi queries disabled 0 Other errors 0
    • 42. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 43. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute Vserver hits 0 0 Requests 0 0 Responses 0 0 Request bytes 0 65 Response bytes 0 188 Total Packets rcvd 0 5 Total Packets sent 0 4 Current client connections -- 0 Current Client Est connections -- 0 Current server connections -- 0 Requests in surge queue -- 0 Requests in vserver's surgeQ -- 0 Requests in service's surgeQs -- 0 Spill Over Threshold -- 0
    • 44. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 45. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute NetScaler File System
    • 46. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 47. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 48. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 49. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 50. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute View events View console messages View statistics Debug system counters Debug load balancing issues Debug CPU/Memory utilization Use cases
    • 51. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute nsconmsg Common Syntax Make sure to use –K, NOT -k
    • 52. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute Live CPU related stats Archived events HA Failover cause LB stats CS related counters Real-time policy hits SSL related counters Compression related counters
    • 53. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 54. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute 1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013 1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013 1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)' DOWN Wed Aug 14 12:27:25 2013 1913 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.51:53(SVC_CTXANG_DNS2)' DOWN Wed Aug 14 12:27:25 2013 1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping- default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013 1911 0 PPE-0 MonServiceBinding_192.168.204.50:53_(ping- default)(SVC_CTXANG_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013
    • 55. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 56. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 57. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024 Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE) override(0) newlyUP(0) Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0) slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0-
    • 58. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute ipConflictAddr = 192.168.47.1 last message repeated 169 times
    • 59. © 2014 Citrix | Confidential – Do Not Distribute Troubleshooting Techniques Common NetScaler issues/resolutions HDD Issues Logging fails /var missing Flash Issues Config fails to save Config saves partially Sync fails Device fails to boot Memory starvation Dropped sessions CPU starvation All services failing All VIPS down Degraded performance
    • 60. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 61. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 62. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 63. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 64. © 2014 Citrix | Confidential – Do Not Distribute nstrace & nstcpdump nstrace Common syntax: Nstrace.sh –sz 0 Nstrace.sh –sz 0 –filter “SOURCEIP = 10.198.4.10” –link enabled Filter qualifiers and operators: SOURCEIP, SOURCEPORT, DESTIP, DESTPORT, SVCNAME, VSVRNAME, STATE ==, eq, !=, neq, >, gt, <, lt, >=, ge, <=, le, BETWEEN Compound filters using || and && nstcpdump Common syntax: Nstcpdump.sh –X tcp port 80 Nstcpdump.sh –w testcapture.cap –X src host 10.198.4.10 tcp port 80 Filter qualifiers and operators: tcpdump standard
    • 65. © 2014 Citrix | Confidential – Do Not Distribute Packet Tracing Differences between nstrace.sh & nstcpdump.sh Nstcpdump.sh Nstrace.sh Useful if traces are to be viewed on standard output nstcpdump.sh –w <filename> option helps writing output to file Useful for offline collection nstrace.sh –sz 0 Saves traces in /var/nstrace in cap format can be used with expressions so that you get to see filtered traffic nstcpdump.sh host <IP> nstcpdump.sh port 21 useful for collection of traces in separate log files based on NICs nstrace.sh –tcpdump1 –nic 1
    • 66. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute -h - prints this message - exclusive option -nf - number of files to be generated in cycle (def. 24) -time - seconds per file (def. 3600) (could be an expression) -sz - size of the captured data (bytes from 60 to 1514) -tcpdump - 0=nstrace-format (default) or 1=tcpdump-format -nic - use separate trace files for each interface (only works if -tcpdump option is set) -name - name of the trace file -filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of following format: <expression> [<relop> <expression>] -link - Log filtered connection's peer's (linked connection's) traffic. Works only with -filter option
    • 67. © 2014 Citrix | Confidential – Do Not Distribute Packet Tracing nstrace.sh nstcpdump.sh Proprietary capture format TCPdump PCAP capture format Native format captures more information Useful for live capture from CLI Files are stored in /var/nstrace Option to write to a file Needs custom dissector in Wireshark (1.6+) Most TCPdump options supported Works in standard Wireshark
    • 68. © 2014 Citrix | Confidential – Do Not Distribute Trace analysis nstrace.sh By Default if nstrace.sh is executed from shell prompt Trace files are stored in NetScaler proprietary (.cap) format The trace capture runs for 1 hr. (3600 sec) if not interrupted Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB and RX packets are captured) The size of the captured data for each packet is 164
    • 69. © 2014 Citrix | Confidential – Do Not Distribute Trace Analysis nstrace syntax examples Command Purpose # nstrace.sh Stores the traces in default (proprietary) format # nstrace.sh -tcpdump 1 Begins to save the traces in the TCPDUMP format for a default 3600 seconds # nstrace.sh -tcpdump 1 -nic 1 Logs the traces (in TCPDUMP format) into separate log files based on the NIC IDs # nstrace.sh –nf <value> No of files to be generated in cycle by default is 24 # nstrace.sh –time <value> Seconds per file by default 3600 seconds # nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire packet length # nstrace.sh –m Capturing mode: sum of the values (def. 6): 1-Transmitted packets (TX) 2 - Packets buffered for transmission (TXB) 4 - Received packets (RX) # nstrace.sh -stop Can be used to disable tracing (when 'nstrace.sh' is run in the background
    • 70. © 2014 Citrix | Confidential – Do Not Distribute Trace Analysis nstcpdump.sh example syntax Command Purpose # nstcpdump.sh <type> Possible types are host, net and port .If there is no type qualifier host assumed # nstcpdump.sh <dir> Qualifiers specify a particular transfer direction to and/or from id. Possible directions are src, dst, src or dst and src and dst. If there is no dir qualifier, src or dst is assumed. src foo', `dst net 128.3', `src or dst port ftp- data' # nstcpdump.sh <proto> ether, fddi, ip, arp, rarp, decnet, tcp and udp. # nstcpdump.sh –c <value> Exit after receiving ‘value’ number of packets # nstcpdump.sh –F –I –r Not supported and not to give them as options to the script # nstcpdump.sh –w <file name> Write the raw packets to file rather than parsing and printing them out. Read by typing tcpdump –r test
    • 71. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 72. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 73. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 74. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 75. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 76. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 77. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute
    • 78. © 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute CTX109304 Data Collection Procedure to Troubleshoot NetScaler Related Issues http://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for- the-first-time http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Ty pe,Technotes&refinement=Product+Family,NetScaler CTX114999 How to Troubleshoot Authentication with Aaad.debug https://taas.citrix.com/
    • 79. © 2014 Citrix | Confidential – Do Not Distribute Maximize your knowledge. Continue your journey with Citrix Education. Recommended next step for hands-on technical training: CNS-205 Citrix NetScaler 10 Essentials and Networking Identify the capabilities and functionality of the NetScaler Explain basic NetScaler network architecture Obtain, install, and manage NetScaler licenses Explain how SSL is used to secure the NetScaler Implement NetScaler TriScale Technology, including Clustering Visit bit.ly/NSCOURSE to save 10% now through April 30.* *Not valid with any other promotions, packages or discounts. Applies only to new purchases. Regional limitations may apply.
    • 80. © 2014 Citrix | Confidential – Do Not Distribute About Citrix Services Citrix Services make sure you succeed with your virtualization programs. How we can help Citrix Education – The fastest, most efficient way to get your team the virtualization skills they need. Online, on-site or in class. citrix.com/training Citrix Consulting – Intensive engagements for complex, critical or just plain massive projects. citrix.com/consulting Citrix Support – Always-on support services that leverage everything we know about best-practice deployment and maintenance. citrix.com/support Educate | Guide | Support | Succeed
    • 81. © 2014 Citrix | Confidential – Do Not Distribute • 40 insider troubleshooting tips • Covering XenDesktop, XenServer, XenApp and NetScaler • Citrix Support top engineers • FREE eBook • Citrix Auto Support • Now available! Secrets of the Citrix Support Ninjas
    • 82. © 2014 Citrix | Confidential – Do Not Distribute Premier Support Calculator Check it out
    • 83. Work better. Live better. Use NetScaler

    ×