Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

12,435 views

Published on

This session will cover advanced techniques in troubleshooting the Citrix NetScaler Appliance using tools such as Citrix TaaS, IPMI, nsconmsg, wireshark and log analysis. We will review usages of these tools along with case studies showing how to best troubleshoot common issues seen in operating Citrix NetScaler Appliances.

What you will learn
- Various tools available to troubleshoot issues and how to use them to isolate NetScaler Issues
- Common deployment problems and how to isolate the causes

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
12,435
On SlideShare
0
From Embeds
0
Number of Embeds
347
Actions
Shares
0
Downloads
1,133
Comments
0
Likes
17
Embeds 0
No embeds

No notes for slide
  • CNS-205: Citrix Netscaler 10 Essentials and Networking
    The objective of the Citrix NetScaler 10 Essentials and Networking course is to provide the foundational concepts and advanced skills necessary to implement, configure, secure, monitor, optimize, and troubleshoot a Citrix NetScaler system from within a networking framework.  This course is designed specifically for learners who have limited or no previous NetScaler experience.  In order to successfully complete this course, learners will have access to hands-on exercises within a virtual lab environment. An optional module on NetScaler SDX appliances is included with reinforcement simulation exercises.

    CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam

    CNS-301: Citrix NetScaler 10 Advanced Implementation
    his course provides the foundation to manage, configure and monitor advanced features and components of Citrix NetScaler 10. Interactive discussion and hands-on labs guide learners through advanced administration tasks such as troubleshooting, configuring application security with Citrix Application Firewall, tuning the NetScaler for high-traffic loads, configuring AAA for system management, and configuring advanced policies using service callouts. Advanced monitoring and management tasks such as configuring and implementing NetScaler Insight Center, Command Center, and NetScaler Web Logging are also covered. Prior NetScaler knowledge is strongly recommended. In order to successfully complete this course, learners will have access to hands-on exercises within a virtual lab environment.

     
  • Advanced Tools and Techniques for Troubleshooting NetScaler Appliances

    1. 1. Advanced Tools and Techniques for Troubleshooting NetScaler Appliances Andrew Redman | Lead Escalation Engineer Citrix Support Webinar August 26 & 27, 2014
    2. 2. 2 © 2014 Citrix. Confidential. NetScaler System Overview Troubleshooting Tools & Techniques Case Studies Resources Conclusion Q&A Agenda
    3. 3. NetScaler System Overview
    4. 4. Key NetScaler Processes Process Description ns_master/NSPPE 4 © 2014 Citrix. Confidential. nsvpnd nsaaad nsconf nsauthd nslog.sh nssync nsreadfile nslcd nsfsyncd nsnetsvc nsconmsg nscollect Runs Citrix NetScaler OS SSL VPN File Transfer RBA and SSL VPN external authorization Writes the ns.conf file CLI authentication Controls logging for the newnslog HA synchronization Used to read SSL certificate files Runs the front panel LCD Synchronizes bookmarks and SSL certificates Used by the GUI for configuration changes Controls writing of the newnslog Statistics gathering for historical purposes
    5. 5. NetScaler File System /var (hard drive) Logs - /var/log & /var/nslog 5 © 2014 Citrix. Confidential. Install - /var/nsinstall Trace - /var/nstrace Core Dumps - /var/crash & /var/core /flash (flash drive) Config - /flash/nsconfig SSL Certificates - /flash/nsconfig/ssl
    6. 6. NetScaler File System (cont.) /flash (cont.) User Monitors - /flash/nsconfig/monitors 6 © 2014 Citrix. Confidential. Custom Options - /flash/nsconfig / (ram drive) OS - (operating system)
    7. 7. Troubleshooting Tools & Techniques
    8. 8. NetScaler Tech Support Bundle > show techsupport Critical System Data In-Depth Performance Monitoring Stats Detailed Log Files USER Command Logging /var/tmp/support/collector_P_10.10.10.10_21Apr2014_21_42_tar.gz
    9. 9. The NetScaler Tech Support Bundle 9 © 2014 Citrix. Confidential.
    10. 10. Citrix Insight Services 10 © 2014 Citrix. Confidential.
    11. 11. 11 © 2014 Citrix. Confidential. Data Collection Analyze Data ‘Single Mission … Data Collection’ FAQ: http://support.citrix.com/article/CTX131233 Tailored Recommendations
    12. 12. https://taas.citrix.com 12 © 2014 Citrix. Confidential.
    13. 13. Technical Issues Flagged 13 © 2014 Citrix. Confidential.
    14. 14. Investigate Issues Further The ‘BELL’ 14 © 2014 Citrix. Confidential.
    15. 15. Crash File(s) Location 15 © 2014 Citrix. Confidential. nscollect-542.gz
    16. 16. Intuitive Navigation 16 © 2014 Citrix. Confidential. Select a different newnslog file to view
    17. 17. Detailed Graphs 17 © 2014 Citrix. Confidential. Mouse over graph to see more detail Informative reference legend Download the data as an excel sheet
    18. 18. IPMI – Intelligent Platform Management Interface Change NetScaler 18 © 2014 Citrix. Confidential. Default LOM IP Address: http://192.168.1.3 IP Address Obtain Health Monitoring Detail Harvest Serial Number Determine MAC Address
    19. 19. Common CLI Show Commands Common show commands for system information: show node, show info, show license Common show commands for vserver and service: show lb vserver, show cs vserver, show service, show persistencesession show connectiontable Other common show commands: show route, show ip 19 © 2014 Citrix. Confidential.
    20. 20. Common CLI Stat Commands Common stat commands for system information: stat ns, stat cpu, stat interface Common stat commands for vserver and service: stat lb vserver, stat cs vserver, stat service Other common stat commands: stat dns, stat ssl, stat http 20 © 2014 Citrix. Confidential.
    21. 21. Leveraging ‘nsconmsg’ Nsconmsg common use cases: View events View console messages View statistics Debug system counters Debug load balancing issues Debug CPU/Memory utilization 21 © 2014 Citrix. Confidential. Make absolutely sure that you use a capital -K and NOT a lower-case -k
    22. 22. Example ‘nsconmsg’ Usage # cd /var/nslog # nsconmsg -K newnslog -j fqdn-ssl-vip -s ConLb=1 -d oldconmsg 22 © 2014 Citrix. Confidential. newnslog current log file fqdn-ssl-vip name of vserver ConLb=1 LB stats
    23. 23. Displaying debug performance information NetScaler current V20 time Performance is Sun Mar Data 23 18:33:43 2014 NetScaler NS10.1: Build 123.11.nc, Date: Feb 24 2014, 17:30:43 current time is Sun Mar 23 18:33:43 2014 ------------------------------------------------------- NATSession : Free(6553)A(6553)InUse(0) NATSession: Cur(Tcp[0] Udp[0] Icmp[0] Other[0]) NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0]) Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0 SSF: Conn (Srvr 0 Clnt 0) U:0 Mon: Probes: 434562009, Failed: 15 CM: VIP(Conn 10.54.169.75:(Srvr 0 Clnt 0) Sessions 443:UP:PCB LEASTCONNS): 0 NATPCB 0 Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Z(SIP[S(10.54.148.201:0], C[0], SSL[0] Server[80:0] UP) SIPDIP[Hits(0] 7317, DIP[0] SO[0/sec, 0]) P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 Mon: Probes: 434562009, Failed: 15 VIP(10.54.169.75:443:UP:LEASTCONNS): Hits(7317, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 0 S(10.54.148.201:80:UP) Hits(7317, 0/sec, P[0, 0/sec]) ATr(0:0) Mbps(0.00) BWlmt(0 kbits) RspTime(0.00 ms) Load(0) LConn_Idx: (C:0; V:0,I:1) ------------------------------------------------------- CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014 23 © 2014 Citrix. Confidential. kbits) RspTime(0.00 ms) CPU:0.2% MEM:182472560 UP:10.00:00:38 since:Thu Mar 13 18:33:05 2014
    24. 24. # nsconmsg -K newnslog -j <name of VIP> -s ConLb=1(2 or 3) -d oldconmsg | more # nsconmsg -K newnslog -s ConMon=1 -d oldconmsg # nsconmsg -K newnslog -s ConMEM=1 -d oldconmsg # nsconmsg -K newnslog -s ConSSL=1 -d oldconmsg ConDebug - Debugging ConLb - Load Balancing ConMon - Monitoring Probes ConMEM - Memory Management ConCSW - Content Switching ConSSL - SSL Offload ConCMP - Compression ConIC - Integrated Caching 24 © 2014 Citrix. Confidential.
    25. 25. Log File Analysis # cd /var/log # zgrep -i cmd_executed ns.log* | more (the -i means ignore CASE) ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2947 0 : User nsroot - Remote_ip 10.13.73.65 - Command "login nsroot "********"" - Status "Success" ns.log:Mar 20 16:45:06 <local0.info> 10.54.169.73 03/20/2014:20:45:06 GMT atlvpx 0-PPE-0 : UI CMD_EXECUTED 2948 0 : User nsroot - Remote_ip 10.13.73.65 - Command "show ns license" - Status "Success" 25 © 2014 Citrix. Confidential. interface down vServer down panic signaled
    26. 26. NetScaler + Wireshark = ‘thumbs up’ http://www.wireshark.org 26 © 2014 Citrix. Confidential.
    27. 27. RED HOT Wireshark Tip 27 © 2014 Citrix. Confidential. Custom Columns Custom Menu Options Pre-build Custom Filters Much Faster Analysis Get the red hot details on how to empower your default Wireshark configuration in the Reference Section at the end of this presentation.
    28. 28. Troubleshooting Techniques & Case Studies
    29. 29. Top Tips 29 © 2014 Citrix. Confidential. Use Citrix Insight Services Pay attention to the issues! Note the highlighted counter(s) Use nsconmsg to see even more detail Correlate time-frames in other log files Achieve root cause analysis faster!
    30. 30. Case #1 - High Availability Synchronization
    31. 31. Same type of appliance 31 © 2014 Citrix. Confidential. HA Pre-requisites Same firmware version Recommend same nsroot password Same RPC Node password Open requisite TCP ports Primary NetScaler Secondary NetScaler
    32. 32. Insight Services Flags The Issues 32 © 2014 Citrix. Confidential.
    33. 33. The HA Pair Struggled To Synchronize # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved (nic_err_bdg_muted) 57520 0 71837018 nic_tot_bdg_mac_moved interface(0/1) 57521 0 71837018 nic_tot_bdg_mac_moved interface(0/2) 9861 0 65 nic_err_bdg_muted interface(0/1) 9862 0 65 nic_err_bdg_muted interface(0/2) 33 © 2014 Citrix. Confidential. 71,837,018 MAC Moves 65 Interface Mutes
    34. 34. The ‘newnslog’ Time-Frame # nsconmsg -K newnslog -d setime Displaying start and end time information NetScaler V20 Performance Data NetScaler NS9.3: Build 54.4.nc, Date: Dec 20 2011, 22:44:41 start time Fri Feb 28 21:49:58 2014 total duration 00.00:03:30 end time Fri Feb 28 21:53:28 2014 total duration 00.00:03:30 data size 1,718,949 bytes 34 © 2014 Citrix. Confidential.
    35. 35. Case #2 - XA/XD Slow Performance
    36. 36. Smartphones XenDesktop Tablets XenApp 36 © 2014 Citrix. Confidential. Insight Services Critical Insight Gleaned Preventative Approach Don’t Underestimate XA/XD Slow Performance
    37. 37. Insight Services Again Flags The Issues 37 © 2014 Citrix. Confidential. http://support.citrix.com/article/CTX136926
    38. 38. Performance Was Extremely Latent # nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg_mac_moved 4263 0 23 nic_tot_bdg_mac_moved interface(0/1) 4264 0 51 nic_tot_bdg_mac_moved interface(1/1) 4265 0 28 nic_tot_bdg_mac_moved interface(1/2) 38 © 2014 Citrix. Confidential. 23, 51 & 28 MAC Moves
    39. 39. Networking Issues Again? # nsconmsg -K newnslog -d statswt0 | grep nic_err 4274 0 1995 nic_err_rl_pkt_drops interface(1/1) 4275 0 40736 nic_err_rl_pkt_drops interface(1/2) 4276 0 1995 nic_err_rl_rate_pkt_drops interface(1/1) 4277 0 40736 nic_err_rl_rate_pkt_drops interface(1/2) 4678 0 42731 allnic_err_rl_rate_pkt_drops 39 © 2014 Citrix. Confidential. System Limits Exceeded Rate-limited Packets!
    40. 40. 40 © 2014 Citrix. Confidential. The Moral of the Story Leverage Citrix Insight Services Leverage Insight Services Pay Attention Gain Quick Insight Dig Into ‘nsconmsg’ On Target for Success!
    41. 41. Resources
    42. 42. Helpful Resources Comprehensive NetScaler Counters Wireshark Developer Editions Customizing Wireshark Tutorial Citrix Insight Services Forum NSTRACE Options How To Manage VLAN’s, Interfaces and Subnets 42 © 2014 Citrix. Confidential.
    43. 43. Conclusion
    44. 44. What We’ve Actually Covered An Overview of the NetScaler System to give you a high-level understanding of the core system. I shared with you some excellent Troubleshooting Tools that are available at your disposal. I also discussed a few key Troubleshooting Techniques that you can use to diagnose issues. I then highlighted two different Case Studies leveraging the tools & techniques that I shared with you in the presentation. In addition I provided you with a few Resources for your future reference and edification. 44 © 2014 Citrix. Confidential.
    45. 45. Fuel your talent with continuous learning. 93% of Citrix Education students became more effective in their role after attending a course. TVID: CFB-61B-A26 Citrix Education offers the following technical training for Networking professionals: CNS-205: Citrix Netscaler 10 Essentials and Networking CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam CNS-301: Citrix NetScaler 10 Advanced Implementation 45 © 2014 Citrix. Confidential. Visit (bit.ly/05Webinar) to save 10% off through September 30* *Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.
    46. 46. Simplify your journey, let us guide you. Accelerate your implementation and minimize risk by taking advantage of Citrix Consulting. You’ll get the expertise of certified Citrix Consulting Architects to successfully deploy Citrix solutions in any phase of your project. 93% of Citrix Education students became more effective in their role after attending a course. TVID: CFB-61B-A26 Visit bit.ly/CTXConsulting to learn more about our proven methodology. 46 © 2014 Citrix. Confidential.
    47. 47. 47 © 2014 Citrix. Confidential. WORK BETTER. LIVE BETTER.

    ×