Citrix Troubleshooting 101

© eG Innovations, Inc. | www.eginnovations.com
Citrix Troubleshooting 101
How to Resolve and Prevent Business-Impacting Citrix Problems

Know Your Speaker
Natalie Tomko
Digital Marketing Manager
eG Innovations
Moderator
George Spiers
Citrix Technology Professional (CTP)
and EUC Architect
Speakers
Barry Schiffer
CTP and Pre-sales Consultant
eG Innovations
Over 20 years of experience working with Citrix
technologies.
barry.schiffer@eginnovations.com | www.eginnovations.com
Citrix Certified Expert and Microsoft Solutions
Expert. One of the 50 CTPs in the world.
george@jgspiers.com | www.jgspiers.com

Agenda
• Understanding the Impact of Poor Citrix Performance
• 3 Key Steps to Troubleshoot Citrix Problems
─ Determining the Scope, Magnitude, and Source of the Problem
• Troubleshooting Tips from the Trenches
─ Citrix tiers, supporting infrastructure, client-side problems
• Supporting remote workers in the new normal
• Tips to Optimize your Citrix Deployment
• How Automated Monitoring Aids in Troubleshooting

Citrix is Performance Sensitive
48%
45%
7%
The most performance sensitive service
Similar in performance demands as other services
Not that performance sensitive
93% believe that Citrix is one of the
performance sensitive services in
enterprise networks.
Source: 2016 Citrix Performance Survey
https://www.eginnovations.com/solutions/view-resources?whitepaper=36
• Many components are involved
in Citrix delivery
• Too many interdependencies
between tiers
• Any fault or failure cascades into
a bigger problem that affects
performance
• Performance relies heavily on
supporting infrastructure
• Configuration can make or break
performance

What Happens When Users Complain that “Citrix is Slow”?
Users start to complain that Citrix has suddenly become very slow
Helpdesk starts getting user complaints
Helpdesk alerts the Citrix administration team to begin investigations
Citrix admins begin investigating the cause for slowness
• Loss of productivity to users • Pressure on IT to fix the problem soon• Customer satisfaction is affected

Slowness is Expensive!
Loss of Productivity
Application slowness for 3 hours; User
productivity loss: $200/hr X 3 hours X
1000 users = $600,000 per incident
Operations Cost
Expert IT staff involved in troubleshooting
costing $45/hr X 10 staff X 3hrs = $1,350 per
incident

Slowness is Expensive!
Unforeseen Capital Expense
Projection
50 users/
server
Actual
25 users/
server
Double the number of servers
needed; $50,000 per server,
5 servers = $250,000 additional cost
Failure of Transformation Project
Performance
slower than
physical
desktops
Users unhappy; want physical
desktops back; Cost increase: $600
per user * 1000 users = $600,000

Performance Must Be Considered at Every Stage of
the Citrix Deployment Lifecycle
Design
Test &
Deploy
Operate
&
Maintain
Trouble-
shoot
Optimize
Rectify Monitor
Trouble-
shoot

Troubleshooting Tips
from the Trenches
Solve Top Citrix Problems

Common Citrix Problems
Logon is
slow
Application
is slow to
launch
Keystrokes
don’t show
up on time
Multimedia
playback is
choppy
User
ComplaintsPrinting
doesn’t
work
Session gets
disconnected
Where to begin troubleshooting?

3 Key Steps to Troubleshooting
#1 Determine SCOPE of the problem
#2 Determine MAGNITUDE of the problem
#3 Determine SOURCE of the problem

Determining The SCOPE of The Problem
Troubleshooting must be problem specific
• For e.g.: Citrix logon slowness will have different reasons and
troubleshooting fixes from slow application access
Is it just one task that the user is facing an issue with,
or are all tasks slow?
Logon
Session
Setup
App
Launch
App
Usage
Remote
Access
Printing

Determining The MAGNITUDE of the Problem
One user is impacted Several users are impacted
Less impact Moderate impact
All users are impacted
Critical impact
• Which user? Where? When?
• Is there a problem with the
user's network?
• What version of Citrix
Workspace app does the
user have?
• Problem with a specific
application or all
applications?
• Is there any commonality between
these users?
─ Same server?
─ Same apps?
─ Same domain?
─ Same location?
─ Same Workspace app
version?
─ Same client Operating
System?
• Is there DC network latency?
• Is there WAN network latency?
• Is the Citrix Provisioning server or
license server having an issue?
• Are StoreFront and Citrix ADC
working fine?
• Is the virtual environment
experiencing a bottleneck?
• Is there a problem in the storage
environment?

Determining The SOURCE of the Problem
StoreFront Delivery Controller
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Active DirectorySQL Server
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Client Side
Citrix Delivery
Infrastructure
Supporting
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network
Corporate
Network

Best Practices for Troubleshooting Common Citrix Problems
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Citrix Delivery
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network
 Slow Logon
 Slow App Launch
 Session Disconnects
 Frozen Sessions
 Session Latency

Slow Logon is the #1 Problem for Citrix Admins
Source: 2018 Citrix Migration Survey
www.eginnovations.com/Citrix-Migration-Trends
• In an environment that averages 500 user
logons per business day, an average 10-
second increase in logon times over a 5-day
week period results in over 6 hours of lost
productivity!
• Slow logons are ranked the #1 problem for
Citrix admins
#1 Slow Logon

Investigating Citrix Logon Slowness
This will help isolate which part of the infrastructure to go investigate for issues
Identify which step of the logon process is taking more time to execute
#1 Slow Logon

Monitoring Citrix Logons: No Longer a Science
Logon Simulation
#1 Slow Logon
Simulated logon via
Citrix Gateway
Applications
enumerated on the
StoreFront console
Launching application
on StoreFront
Citrix Director
Real User Monitoring

Tips to Avoid Logon Slowness
• If using Citrix Profile Management, make sure your storage location is configured with high speed
access to your desktops and applications
• Make sure that you have enough compute available to cope with demand. Always size for peak load +
10-20%.
• Use Citrix Workspace Environment Management (WEM) to map printers, drives and other actions after
logon.
• Reduce the Group Policy settings that need to apply to your end-users.
• Reduce the amount of applications installed on your gold images. Plan your application and image
strategy.
• Optimise your images – Disable unneeded services, scheduled tasks, and tweak the OS for better
performance
#1 Slow Logon
─ https://support.citrix.com/article/CTX224676 - Citrix Optimizer

Citrix Workspace Environment Management (WEM)
With WEM
Without WEM
All the processing is
done after logon
Logon speed is
improved
All the processing steps happen before logon
#1 Slow Logon

Investigating Slow Application Launch
• Is the launch for only one application slow, or for all applications?
• Is slowness affecting application launch on one VDA or all VDAs?
• Compare workloads of VDAs in use
• Published apps may involve user session establishment. So brokering, GPOs etc.
can cause slowness.
• Look for add-ins/plugins (e.g., Outlook, Browser, etc.) that may be causing app
launch slowness
• Look for proxies and other intermediary hops between the end user and VDA
• When you console directly on to the VDA, does the application launch slowly?
#2 Slow App/Desktop Launch

Monitoring Slow Application Launch
eG Enterprise
• Application availability and launch with real
simulation
─ App Probing using Director
─ Logon simulation tools
• Monitor actual application launch
times (using Windows APIs)

Tips to Avoid Slow Application/Desktop Launch
• Optimise images
• Make use of session prelaunch for published applications when makes sense
• Make sure VDAs are spec’d appropriately and not overloaded, especially during
peak times. Remember, size for peak + 10-20%
• Make sure you have enough VDAs powered on and available to handle requests
• Make sure you have enough DDCs to handle brokering requests
• Configure Group Policy as per best practices
• Ensure your Citrix profile/3rd party profile solution is optimized & not a
bottleneck
• Keep your Virtual Apps and Desktops deployment up-to-date, including
Workspace app
• Make use of Zones to broker users to desktops/applications closest to them,
if you have multiple-datacentres

Implement Brokering Improvements (7.11+)
Improvements to Latency and SQL Blocking Query
Before XA/XD 7.11
(90ms latency)
XA/XD 7.11+
(90ms latency)
Brokering requests per sec 3.7 12.6
Time to launch 10k users 44m 55s 13m 10s
Quicker brokering = quicker launch times!
Keep your environments up to date

Use Session Prelaunch (7.6+)
• Session is pre-prepared when user logs on to StoreFront
• When user clicks to launch application, the majority―if not entire―logon process
would have already completed in background
• Not always will a user launch a Virtual Apps
application straight away, they may check
emails. In the meantime, let Virtual Apps
prepare the session in the background.
• Keep in mind that licenses are checked out,
so you can set a time limit to end pre-
launched sessions if they are not used after a
period of time.

Workspace app 2006.1 – Long Launch Times
Problem: Workspace app 2006.1 takes a long time to launch an ICA session
Cause: This can occur if the endpoint running Workspace app cannot connect to
locus.analytics.cloud.com:443
Resolution: Create DWORD SendPublicIP with a value of 0x1 under
HKLMSOFTWAREWOW6432NodeCitrixDazzle

Investigating User Session Disconnects
• Follow a process of elimination to determine if:
─ This happens to particular users, or all users
─ Happens when using specific desktops or published applications
─ Happens to users from particular offices and so on
#3 User Session Disconnects
• Check Event Logs on VDAs around the time when disconnect happened
• Check Event Logs on endpoint around the time when disconnect happened
• Run network connectivity tests for affected users
• Run tests on VDAs to make sure they are not losing network connection briefly
• Check that proxies, firewalls and antivirus products are not interfering with ICA
traffic

Tips to Avoid User Session Disconnects
• Make sure your environment is on the latest updates/versions:
─ Workspace app should be up to date
─ VDAs are always patched with the latest releases from Microsoft
─ Update Hypervisor tools such as VMware Tools
─ Ensure that Citrix Virtual Apps and Desktops is kept up to date
• Ensure network monitoring is in place and the networking fabric is kept up to
date to ensure optimal performance and stability
• Ensure your desktops are kept up to date with the latest Microsoft patches,
and drives for network cards etc are running from latest versions
• Keep a record of what has changed in the environment so you can roll-back
easily if disconnects happen suddenly after a change e.g. Workspace app
upgrade

Enable Workspace app Logging for further evidence
https://www.jgspiers.com/citrix-receiver-windows-logging/
General Workspace app logging:
Set ReceiverVerboseTracingEnabled to 1 under
HKLMSOFTWARECitrix for 32-bit machines or
HKLMSOFTWAREWOW6432NodeCitrix for 64-bit machines

Investigating Frozen Sessions
• Are all users on the same VDA affected or just one?
• Does exiting Receiver, re-launching and connecting back to the
session work?
• Can you force the user off via Director, or by killing processes via
Director or Task Manager on the VDA?
• Is a process thrashing CPU, memory, or disk on the VDA(s)?
• Does the VDA need to be drained and rebooted to clear the frozen
session?
#4 Frozen Sessions

Tips to Avoid Frozen Sessions
• Make sure your environment is on the latest updates/versions:
─ Citrix Receiver/Workspace app should be up to date
─ VDAs are always patched with the latest releases from Microsoft
─ Update Hypervisor tools such as VMware Tools
─ Hypervisors should have latest patches
─ Ensure that Citrix Virtual Apps and Desktops is kept up to date
• Make sure that VDAs are spec’d appropriately if a session froze due to CPU
thrashing or lack of RAM. WEM can help here.
• If using Citrix Provisioning (PVS), make sure your Write Cache set to “Cache on
device RAM with overflow to HDD”
─ Ensure the Write Cache for PVS and MCS is large enough to cope with writes.
• Follow Citrix recommendations for Antivirus exclusions:
─ https://support.citrix.com/article/CTX127030
#4 Frozen Sessions

Hide Citrix Sessions
https://www.jgspiers.com/user-stuck-citrix-desktop-force-log-off/
Session frozen on VDA, but you
cannot reboot immediately
(Virtual Apps)?
#4 Frozen Sessions

Best Practices for Troubleshooting Problems in the
Infrastructure Supporting Citrix Deployments (on-premises)
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual
Apps Server
Hypervisor
Internet/
WAN
Storage
Supporting
Infrastructure
Citrix End Users
Firewall
Corporate
Network
Client
Network

Best Practices for Troubleshooting Problems in the
Infrastructure Supporting Citrix Deployments (Citrix Cloud)
Workspace
Cloud Connector
Gateway Service
Citrix End Users
License
Server
Citrix Provisioning
Active Directory
SQL Server
Citrix Virtual
Apps Server
Hypervisor
CorporateLAN
Internet/
WAN
Storage
Supporting
Infrastructure
Citrix End Users
Firewall
Public
Internet
Client
Network
Delivery Controller
Cloud Connector
Citrix
_Cloud

Investigating Issues with the Supporting Infra
• Network
─ Don’t assume, monitor!
─ May not just be your routers or switches. Network card issues can cause performance problems
─ Speed negotiation problems, driver issues can also affect performance
• Active Directory
─ Sites and Services may not be correctly configured if you have multiple sites across your organisation
─ Logon scripts may be configured on user accounts through ADUC which can be bad for logon times
─ DNS if incorrectly configured could cause problems with applications and services
• Hypervisor
─ Monitor resource usage – look for bottlenecks; Follow vendor-specified best practices
─ VM ready time, memory ballooning and swapping, disk queueing and aborts
─ Monitor resources used by other VMs
• Storage
─ Detect storage failures (failures remain hidden in RAID environments)
─ Monitor LUN-level I/O activity
• Cloud
─ Ensure you are not reaching capacity limits
─ Look for bandwidth / connection problems to the cloud
Infrastructure Issues

Tips to Avoid Issues in the Supporting Infra
Infrastructure Issues
• Make sure Group Policy is configured to apply only relevant policies. You can move a lot
of Group Policy settings off to WEM if you are entitled
• Size your hypervisors to cope with peak load. That is, the maximum number of sessions
you need to support concurrently with a 10-20% buffer.
─ Also factor in host failure or datacentre outage and how that will affect remaining healthy
infrastructure. Can you support all users in one datacenter?
• If using file servers for Profile Management or Folder Redirection, make sure enough disk
capacity, RAM and CPU is assigned to these servers, and that high availability is in place
• Keep the number of Organizational Units and Security Groups at a minimum. This
will make management easier and speed up troubleshooting
• Monitor and always regularly patch storage, networking and hypervisor infrastructure
• Maintain tight change-control and always keep track of changes to the environment
• Keep track of known issues through software releases. Release notes and social channels
recommended.

Supporting remote workers
In the new normal

Investigating Internal or External Session Latency
• If only external users are facing issues, but not internal users:
─ Are the Citrix ADCs under load?
─ Is latency caused by routing or issues in the DMZ?
─ Could user's WAN connection be a problem?
#5 Session Latency
• If both internal and external users are affected, check StoreFront,
other Citrix components, other supporting infrastructure
components, and the datacentre network
StoreFront
Citrix ADC
Internet/
WAN
Firewall
WAN Latency DC Latency
Internal
User
External User

Monitoring Session Latency
• Use Citrix ADM to view DC latency and WAN latency metrics obtained by
analysing the HDX traffic flowing through Citrix Gateway
#5 Session Latency
Customers not on Citrix
Gateway Premium licensing
can rely on AppFlow
monitoring tools from vendors
(eG Innovations, others)

Tips to Avoid Session Latency
• Monitor resource usage on your VDAs
─ Detect users/applications taking up too much resources proactively
• Monitor browser activity
─ Many monitoring tools can show URLs accessed by users
─ Limit browser access to web sites required for business activities
• For remote access, make sure Citrix ADC/Gateway is configured with
appropriate licenses that support throughput requirements
• Make sure the internal network is performing as should
─ Unforeseen problems can occur – e.g., with network teaming
#5 Session Latency

Bind the best TCP profile to your Gateway
1. Browse to your Gateway Virtual Server and bind the TCP Profile
nstcp_default_XA_XD_profile to it.
#5 Session Latency
2. Edit the nstcp_default_XA_XD_profile profile and uncheck Use Nagle’s algorithm.
- Should be done automatically in later builds.

Enable Citrix HDX Adaptive Transport (EDT Protocol)
to Improve Latency and Throughput
A copy of a 45MB file over a 200ms line is still an incredible
36 seconds faster than the 100ms TCP copy.
#5 Session Latency

Enable Audio over RTP to Improve Audio Recording and
Playback
#5 Session Latency
• Set audio to ‘Medium’
• Edit Default.ica on StoreFront to enable RTP
• Open firewall ports
• Enable DTLS on Gateway Virtual Server

Optimized for WAN policy template
Unsure of which policy settings are friendly with WAN
connections?
#5 Session Latency
Check out the Optimized for WAN policy template.
Use that as your foundation, then tweak as required.

Best Practices for Troubleshooting Problems Originating
from the Client Side
Citrix ADC
Citrix End Users
License
Server
Citrix Provisioning
Citrix Virtual Apps
Server
Hypervisor
Internet/
WAN
Storage
Client Side
Citrix End Users
Firewall
Corporate
Network
Client
Network

Investigating Issues on The Client Side
• Is the client endpoint running the latest version of Citrix Workspace?
• Is there anti-virus running on the client workstation that may be interfering with
ICA traffic?
• Is the browser that the user is using to access Citrix session slow?
─ Are there browser add-ons such as proxies or content filtering software that is blocking
or preventing ICA from optimally performing?
• Is the client workstation having any problems?
─ A background application/process taking too much resources
• Old hardware running an outdated, unpatched Operating System?
Client-side Issues

Investigating Issues on The Client Side
• How is the network connectivity on the client side?
─ Are there any firewall settings on the workstation that’s affecting connectivity?
─ What is the network latency? Bandwidth?
─ Citrix APIs/counters provide latency and bandwidth information for virtual channels
Client-side Issues

Tips to Avoid Issues on the Client Side
• Make sure to have the most recent version of Citrix Workspace app installed on the
endpoint
• Test network connection strength from the user terminal to Citrix Site
─ Citrix’s Connection Quality Indicator tool can be useful
(https://support.citrix.com/article/CTX220774)
• Check the user’s profile and see if there’s any issue, such as a large profile size
• Check Active Directory to see if the user account does not have any non-essential
Group Policy settings applied or logon scripts
• Check for memory, disk and storage issues on the client workstation
• Make sure the browser used to access StoreFront or Citrix Gateway is of the latest
version and is clean e.g. not overloaded with plugins, toolbars etc.
Client-side Issues

Optimizing the Citrix
Environment
For Maximum Performance

Achieve Persistency Even Across Non-Persistent Desktops
• Citrix Profile Management and Citrix WEM allow profiles, mapped drives, printers,
and other actions to roam with users regardless of which desktop they connect to,
providing a persistent feel
• Citrix App Layering (Elastic Layers) can solve niche cases where users need access to
specific applications, providing them with a unique desktop experience even on non-
persistent desktops

Achieve a Roaming OST for a Great Email Experience
Office 365 User Layers captures the
OST file in a VHDX container, which
can roam between desktops.
FSLogix Office 365 Containers
provide the same solution, plus
many additional advanced features.
Cache Exchange Mode with Office 365 is desired, but how
can that work effectively in a non-persistent environment?

Achieve Successful Patching of Citrix Images
• Review Citrix validated patching reports
─ https://support.citrix.com/pages/interop
• Perform your own tests on patched images before rolling out to production
• Carry out full change control procedures
• Roll out patches to 10% of estate initially, before rolling out to entire Site
• Use disk versioning to ensure successful rollback (Citrix Provisioning/PVS)
• Keep previous MCS images or image versions to ensure rollback of Machine Catalogs
• Use Citrix App Layering to reduce patching efforts

How to Keep on Top of Your Images & VDA Housekeeping Tips
• Consult with HR/Active Directory teams to be notified when staff have left
the company
─ Remove persistent desktops or reassign to other staff
• Review desktop count regularly and reduce pool if required, or increase it
• Retire VDAs hosting applications that have since been migrated to newer
VDAs, or moved to the cloud/SaaS
• Use monitoring software that can alert you when VDAs have not been
logged on to for a period of time
• Take backups of images and then delete the live copies as part of
decommission process Script to find unused VDAs:
https://www.jgspiers.com/find-unused-vdas-script/

Successfully Deploy New Desktops or Applications to Citrix
• Consult with application vendors to understand h/w requirements to run their software,
especially in a multi-user environment
• Consult with application vendors to understand s/w requirements to run or work alongside the
applications, such as Outlook/Excel/Word/Java/.NET/Internet Explorer
• Deploy OS and s/w that both are in support by Microsoft, Citrix, and the application vendor(s)
• Profile staff, learning which peripherals they will use with the application, what network drives
they need access to, what printers they need access to and so on
• Design desktops and application servers according to requirements
• Design Citrix and Group Policies appropriately according to requirements. Lifting restrictions on
clipboard access, client drive mapping and so on may be required
• Run initial pilots to determine user acceptability, and gather resource consumption metrics to
determine impact through scale. Move pilot into staged production

Effectively Plan for Disaster Recovery
• Document steps required to recover a site in the event of a disaster, i.e. datacentre
outage. This allows staff to understand what they need to do both technical and
non-technical
• Ensure that the load of a datacentre can be hosted elsewhere, i.e. cloud or a
recovery datacentre
• Use N+1 method to deploy StoreFront and Delivery Controllers
• SQL, Domain Controllers, hypervisors, and so on need to be redundant and cope
under partial failure
• Citrix ADC can help route traffic away from StoreFront etc. servers that are down
• Test Local Host Cache and configure your DDCs to cope with extra registration
requests, etc.
• Regularly review disaster plan and annually or bi-annually perform disaster testing
https://www.jgspiers.com/citrix-local-host-cache/

How Automated Monitoring
Aids in Citrix
Troubleshooting

Lifecycle of a Citrix Deployment
Design
Test &
Deploy
Operate
&
Maintain
Trouble-
shoot
Optimize
Rectify Monitor

Synthetic and Real User
Experience
Citrix and Supporting
Infrastructure
End-to-End
Performance Correlation
Reports, Analytics, Machine
Learning
Monitor Every Layer,
Every Tier
Pinpoint Root Cause of
Problems
Right-size & Optimize for
Maximum Performance
Measure All Aspects of Citrix
User Experience
1
2
3
4
End-to-End Monitoring with eG Enterprise

eG Enterprise Demo

Dramatically simplify and
accelerate application
performance management.
Deliver peak performance,
user satisfaction and ROI.
User Satisfaction
IT Productivity
Higher service
uptime
Simplicity
Cost control & ROI
Total Citrix
Performance
Management
“
”
Benefits of Using eG Enterprise

Thank You!
For more information on Citrix
monitoring, contact eG Innovations:
www.eginnovations.com
info@eginnovations.com
For more information on Citrix
troubleshooting, contact George Spiers:
www.jgspiers.com
george@jgspiers.com

Citrix Troubleshooting 101

More Related Content

What's hot

Similar to Citrix Troubleshooting 101

More from eG Innovations

Recently uploaded

Citrix Troubleshooting 101