SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop

© 2014 Citrix. Confidential.1
• Good morning and welcome to “Citrix Group Policy Troubleshooting for XenApp and XenDesktop”,
my name is Rick Berry and I’ll be your guide to help clear up the mysterious world of Citrix Group
Policies.
• A little bit about myself:
• I’ve been on our Global Escalation team for 8 years and I work on both XenApp and XenDesktop
related issues.
• I specialize in the Citrix Group Policy components and work closely with our Group Policy architects
in addition to providing troubleshooting assistance to our global support teams.
• When you’re having Group Policy issues, they can create a significant business impact in your
environment, plus to complicate matters, you may not even know you’re having Citrix policy issues.
So my key objectives for this session are to: Guide you through the Citrix Group Policy components,
discuss troubleshooting tools and methods to help resolve your Group Policy issues or even better,
prevent them altogether. Also, since the Citrix GP components are shared amongst our XenApp and
XenDesktop platforms, the concepts I’ll be discussing today apply to both platforms.
• Lastly, we should have time for one or two questions. I’ll be covering a lot of material today, so if
you have a question please hold it until this time. I’ll be close by after this session and later in the
day over at the Ask the Experts bar in the Citrix booth if you think of something later. Plus this slide
deck including my narrative will be available for download after Synergy in addition to the video
recording being available on Citrix TV.
• So with that in mind, let’s get started

You can tweet about this session using #SYN406

There are multiple components involved with Citrix Group Policy and understanding
how they work together is key to better troubleshooting Group Policy issues.

First, I want to establish a baseline for some terminology we’ll be using today on the different
locations Citrix policies can be applied. This is key since troubleshooting methods can differ a bit
depending on where you apply your Citrix policy settings.
 First, you can set Citrix policies via the local group policies
‐Each server and workstation has a single group policy object and Citrix policies can be set here
 Next, we have Citrix Farm policies
‐These are also known as IMA policies (usually in XenApp environments)
‐These are policies that are set via AppCenter or the DSC (for XenApp 6.x) or Studio (for
XenDesktopXenApp 7.5)
‐These policy settings are stored in the Farm datastoredatabase
 Lastly, we can set Citrix policies via Active Directory Policies
‐Set via Site, Domain or OU GPO’s
‐Stored in Active Directory
‐This allows for a single GPO object with both Citrix and Microsoft policy settings
 Here are the methods for applying Citrix Policy settings
 However, for todays presentation, I’ll be focusing on the FarmIMA and Active Directory method
since these are the most common

Next, it’s important to understand the processing and precedence order for policies. Especially if you
are applying policies using different locations. So lets take a look at this. order for policies. This is the
order that the GPO’s are reviewed for applicable policy settings.
 Here are the available GPO types where we can place Citrix policy Settings starting at the local
policies
 This is the processing order, in other words, the order in which analyze each GPO type for
applicable Citrix policy settings
 Now the precedence order is opposite of this, meaning that policy settings in the higher GPO will
win when there’s a conflict in settings
 For example: say you have Client Drive Mapping enabled via a Citrix Policy in AD whereas in the
local GPO you have it Prohibited, while processing policy settings, we’ll see both settings, but due
to the precedence order, the resultant set of policies will contain the winning setting from the AD
GPO

policies
GPO

The first major component I’ll discuss is the Citrix Group Policy Management Console.
 I’ll refer to this component as the Citrix GPMC from here on out, and provides Citrix policy visibility
within the Microsoft GPMC
 It allows you to manage Citrix policies via AppCenterStudio (for Farm policies) or the Microsoft
GPMC (for AD GPO’s)
 This component also allows you to perform modeling or comparison of policy settings
 It can be installed on a device for managing policies, requires that GPMC is installed
 In case you ever need to review version details for these binaries during troubleshooting, the core
binaries are located here

The second major component I’ll go over is the Citrix Group Policy Client Side Extension
 I’ll refer to this from here on out as the Citrix CSE, this is separate from the MS CSE, and loaded via
this DLL
 This component is initialized via the Winlogon process (for both Computer and User)
 It generates the policy requests
 It will also retrieve certain aspects of the Computer or User connection, this is needed to see if any
policy filters will be leveraged
 This component also forwards policy request to the caching service
 The core binaries for this component are located here

Next lets look at our Caching service
 Handled via the CitrixCseEngine process which runs as a service, and it’s a part of the Citrix CSE
 It handles the policy calculation in addition to writing the resultant settings to the registry
 It also caches the FarmAD GPO objects to optimize the policy application process
 For ADFarm GPO’s, the cache folder is located here
 In addition, it caches per‐computer and per‐user policy files which we review in detail later in the
sessions

Now that we’ve gone over the core components and services involved, let’s review what data files
Citrix Group Policy uses.
 First, we have the resultant set of policies file, known as RSOP.gpf, and exists per‐computer and per‐
user
 This binary file location is based on whether it’s a per‐computer or per‐user file
 The RSOP.gpf file is parsed into registry settings and are applied to the proper location,
 NOTE: the <SESSIONID> you see here is going to be used quite a bit for user policies

Let’s talk about removing or rolling back Citrix policy settings
 We need a way to remove policy settings that aren’t explicitly configured, you may ask why ?? If a
user logs in and gets say a printer virtual channel bandwidth setting via Citrix policy, if the next user
to log in to that same session they may not have that setting configured, so we need to be able to
“remove” or “undo” this settings
 The data file for this mechanism is the Rollback.gpf file. It exists for both per‐computer and per‐
user settings
 The rollback.gpf file contains the instructions needed to remove pre‐existing RSOP settings in the
registry
 The rollback file is cached to the same location as the RSOP.gpf file

Next, I want to review the Citrix Policy Filters
 The Citrix policy filters allow granular control of policy settings
 Allows filtering based on certain criteria, for instance workstations coming from a certain IP range,
or connections coming thru your external Netscaler
 The available filters vary based on whether you are dealing with a Computer or User policy
 The filters can be used with any policy set with the exception of the “Unfiltered” default policy

Lets take a look at the Computer filters for XenApp 6.5 from AppCenter.
 You can see the filter types listed here
 We’ll zoom in a bit on the filter types, and we see that for a Computer policy we have these
available filters
 When selecting a filter type, you’ll get additional details on the filter in this pane
 Zooming in again, we can see the products and versions the filter applies to plus details of the filter

available filters

Now lets take a look at that same details for User Policies.
 We see the available filter types here
 We’ll zoom in a bit to show you the additional filters that are present for User policies.

Let’s review our default policy information along with the built‐in templates
 There’s a default unfiltered policy (per‐computer and per‐user)
 The policy settings you add to the Unfiltered policy apply to all related objects (Computer or User)
 The unfiltered policy can be disabled if needed (if you disable the Unfiltered policy move it to the
lowest priority)
 By default there are pre‐configured policy templates in place
 Templates are group based on user connectivity methods (WAN, etc.)
 In addition, once you create your own polices, they can be saved as a template, this is a good
backup method, but this shouldn’t be your only backup method

Now, as you move from XenApp 6.x and XenDesktop 5.x to newer versions, you going to see some
difference when managing Citrix policies.
 This is the policy node from a XenApp 6.5 environment
 In this case, you’ll see separate Computer and User policy nodes

Now for newer versions of XenDesktop 7.x and XenApp 7.5, this view is different. Here we see a Studio
screen shot of the policy node from XenApp 7.5
 We see that there's only a single node for the policy objects. There’s no differentiation between
computer and user policies within Studio for these newer versions.
We take care of that on the backend when determining the resultant set of policies.
The exception to this is when you manage Citrix policies via AD GPO’s, in that case, because you’re
using the GPMC to manage policy settings, there will still be separate Computer and User sections.

Here are some details surrounding policy refresh intervals, there are some differences to be aware of
 For policies setup in StudioAppCenter
 The Computer and logged in User policies will refresh every 90 minutes
 For AD GPO’s
 The refresh interval is typically every 90 minutes plus or minus a random offset
 This update interval can also be set via AD GPO at the domain level
 For either method
 Computer policies will update at machine startup
 User policies will refresh during login or reconnection to an active or disconnected session
 You can also manually update the policies by issuing gpupdate /force (this is helpful when you
modify policies and want them to refresh quicker)

Now that we’ve reviewed the components and files involved in Citrix Group Policy, I want to tie this
altogether for you by showing you how Citrix Group Policies are applied when a user logs in. NOTE,
this process is similar for when a computer starts up.
 Here we have an end user ready to log in <PAUSE><PAUSE> What, this isn’t your end user, lucky
 Ok, here’s your happy end user logging in
 Once the user logs in, the Microsoft Winlogon process starts up
 Then it loads the available Client Side Extensions
 This includes not only the Microsoft policy extension, the Citrix one as well
 From this point I’ll focus on the Citrix CSE
 Citrix CSE starts to process the policies and the local GPO’s are processed first
 Then CSE process the Farm policies
 Then lastly CSE processes the Active Directory policies
 Now the precedence order is just the opposite of our processing order (so Active Directory policies
l take precedence over Farm policies and so on), using this precedence order and any policy filters
involved, a resultant set of policies file (RSOP.GPF) is created
 This file is then used to make the actual policy settings in the registry (the registry location is based
on whether we dealing with Computer or User policies)
 Let’s take a closer look at this final stage of Citrix Group Policy processing, since there’s a lot more
going on here and understanding this critical stage can help you significantly when troubleshooting

 The Rollback.GPF file is loaded from the cache (keep in mind this is the file containing instructions
to remove or undo the existing RSOP settings)
 We then apply the new RSOP policy settings, this results in the registry being updated
 We then remove the previous cached GPF files
 Then we copy the new RSOP.gpf file to the cache and create a new Rollback.gpf file for the existing
session which is cached as well
 The newer Rollback.gpf file will then be used the next time the User logs in (or Computer starts up)
 Then the LastUpdate value is updated in the registry for the Computer or Session
 At this point, we’re done

Now, I want to review recommended practices based on issues encountered during Citrix Support
cases.

First, we’ll discuss policy architecture
 While it’s fully supported to use both Farm and AD policies together, it can lead to confusion during
troubleshooting or when something is not working properly
 If possible try to use one method or the other based on your requirements
 We’ve seen some issues where the use of WMI filters on AD GPO’s (for example, you want to apply
the AD GPO to a certain OS), can cause issues during session reconnects. This is due to some issues
with the MS API’s used by Terminal Services during a reconnect and AD timing out for WMI calls.
 Try to use WMI filters sparingly and if you have to, it maybe possible to prevent issues by using a
setting we have, I’ll talk a bit about this more later

<????> I have a question for you, no need to raise your hand, think about if you’ve got your Citrix
policies documented or even better backed up in a manner you could easily restore them
<PAUSE> Ok, that’s a good amount of you. Documenting policies is important especially when things
go wrong. No worries for those of you that don’t have them documented yet as I’m going to go over
steps you can take to easy accomplish this.
We have several methods to document these, based on how you apply your policies
 For Farm policies applied via AppCenter or Studio
 It’s best to get them written out say in Excel, etc. Or you can also leverage our Citrix Scout tool to
get the settings and filters exported.
 For policies applied via Active Directory
 Use the GPMC and select the Save Report option on your GPOs, this will create a HTML report
(NOTE: this needs to be done a device with both the MS GPMC and Citrix GPMC installed)
 For either method, there are a few other tools that will help with this as well and I’ll go into more
details on these in a bit
 The first is CtxCseUtil, which will generate resultant set of policy reports
 We also have a PowerShell module that contains cmdlets to export and import Citrix Group Policies
(the export cmdlet is great since this gives you a snapshot of your policies that can be imported
later)

There are certain actions, that if taken, can cause group policy consistency issues. These are some of
the more difficult issues to troubleshoot and you may not even know you’re having an issue until it’s
too late (you’ll get the dreaded call from security or your bosses, bosses boss). So here are some
things you shouldn’t do.
 Unless instructed by someone from Citrix Technical Support, don’t manually manipulate any of the
Citrix Group Policy filesfoldersregistry areas
 This includes these areas which relate to the policy cache and resultant registry areas. These areas
are tightly linked together, so altering any one area can have negative effects.
 There is a certain use case where we might ask you to clear these areas out, for instance a specific
fix as referenced here for LA5051, in this case, we clearly document the steps needed.

Ok, now that I’ve gone over the architecture and components of Citrix group policy, let’s talk about
some recommended practices based on issues we’ve seen while working on support cases.

When troubleshooting Citrix group policy issues, we have a recommended troubleshooting path.
These steps will help you isolate the issue or in case you need to contact Citrix Technical Support for
assistance, you’re going to have the information ready to go and help expedite the resolution process.
 I’ll be covering these steps in greater detail in the upcoming slides

The best starting point when troubleshooting Citrix group policy issues is to know what the desired
policy settings are for the given scenario, so knowing you’re baseline configuration is key.
 Then to isolate the issue further, make sure you can answer what I call the four W’s
 Who is seeing the issue, are there specific users in a separate office, or all users?
 What issue is being seen, are they getting client drives mapped when they shouldn’t be or vice
versa

versa

 When are they seeing the issue. For new sessions, reconnecting, smooth roaming or maybe all of
those?
 Lastly, where are they seeing the issue? Are they working from home and coming in via your
Netscaler?
 Or are they working in the corporate office?

those?
Netscaler?

The next step in the process is to be sure you know what version of components you are dealing with.
 Now looking at this screen shot, we only see the core XenApp 7.5 install the related VDA, but no
group Policy components
 So what version am I really at?? In this case, the GP components are at the baseline versions that
come with the core product. The best method to confirm the versions is to check the individual
components.
 Note that our Citrix Scout tool does collect versions of the Citrix Group Policy components as I’ve
highlighted in this Scout data file for a XenApp 6.5 server

components.

So let’s take a look at getting the CSE version.
 You can look in the component directory
 As I mentioned earlier, here’s the path to the CSE, you can right click on the CitrixCseEngine
executable, which is the primary CSE binary
 And going to the Details tab will show you this is version 2.1

Now to determine the GPMC version, we can do the same thing
 We go to the components directory as I’ve referenced here and we can right click on the main
binary
 And we get the version here, in this v2.2

Now that we’ve reviewed how to get the component version here’s a reference chart to show the
versions for our core XenApp products. This will help you when you have the baseline versions of the
components installed.
 I’ve provided the XenApp version, GPMC and CSE versions
 For 6.0
 For 6.5, When there’s a higher version in parenthesis, this means this component has a higher
available version. You can review these on our support site and you’d likely be asked to upgrade if
you’re having issues
 And lastly for 7.5

 For 6.0

versions for our core XenDesktop products.
 Again showing the version of XenDesktop, GPMC and CSE
 For 5.5, 5.6
 For 7.1

 For 5.5, 5.6
 For 7.1

Next we’d want to check our the locally cached policy objects, first we’ll look at cached objects from
AD GPO’s
 On the serverVDA in question, here’s a look at the GPO cache
 As I noted earlier, it’s under PROGRAMDATA folder
 Let’s look closer at the GPF files
 In this case, we see a GUID in the file name
 When we see GUID in the file name, this means it’s a cached AD GPO
 You can there are two files with the GUID, the one with a 0 is the cached User policy
 The 1 in the name is the cached Computer policy

AD GPO’s

Now that we have the GUID, let’s take a closer look in AD
 We look at the local cache again and note the GUID
 We can then search in the GPMC (or use PowerShell cmdlets) to find a match
 Here we’ve found a GUID match, so this is our AD GPO in the GPMC
 We then also know what the GPO display name is
 So now we want to compare the time details for the AD GPO, here we see the creation date
 Here is the modified date (meaning the last time a change was made to it), this is the date we’re
really interested in
 In this case, the dates are a match between the AD GPO and the locally cached version, so our
cached version is up to date. If we didn’t match, usually issuing a gpupdate /force will resolve that,
if not, there’s a likely an issue with the GPO in AD (look in event viewer for errors)
NOTE: There could be a difference in the modified timestamps of the cache files since there is a
separate one for User and one for Computer. The modified timestamp in the GPMC is the last update
and should reflect in one of the files.

Now for farm policies created with AppCenter or Studio, the cache files look a little different.
 We’re looking in the same cache folder
 Looking closer, we see these files lack the GUID
 Which means they are Farm policies
 But the User and Computer notations are similar

Now we should check the Computer and User cache files
 Here we see the location of the cached RSOP and Rollback GPF files
 The per‐computer cache files are located in the root of this folder
 Now we can look into the sub‐folders, the folder is named for the SessionID, if you know of a
specific user seeing the issue, you can issue a QUSER to get their session and check from there
 These sub‐folders are for the Per‐User files (for the given SessionID)
 These are the files for Session 2, note for either computer or users files, you should always have
both rollback and rsop gpf files, if not, this needs to be resolved

Now, let’s review what’s in the registry for Citrix Group Policies
 Here we see the core location for the per‐computer settings
 The various sets of Computer policies usually are contained in sub keys, here we see the Graphics
and ICA policy settings
 Looking at the values, we can see the final resultant policy settings, NOTE: we have references on
our e‐docs site to match up a registry value to the policy in question

Now we should review the per‐user RSOP settings in the registry, especially if there’s some issue with
a policy applying (or not applying)
 For the per‐user registry area, we need to so into the sub‐key noted with the SessionID which we
get from QUSER, in this case we go into Session 2
 Here I’ve highlighted two sub‐keys of interest, the Events key contains a last Update time of the
policy settings whereas the Evidence key contains details of the connection itself (things like Client
IP, Client Name, etc.)
 Under the User sub‐key we get the various groups of GPO’s and their settings

Now let’s look at connection details and how that relates to Citrix Group Policy.
<<QUESTION>>Can anyone tell me why we’d be interested in client connection details in relation to
Citrix group policy? <PAUSE>
Correct, we’re interested in connections details because we can filter User policies based on certain
connection filters
 Things like Client IP address or Client name, so if connection info isn’t right, there’s no way we’ll be
able to apply the right policies

Lets look at the connection details closer.
 On the XenAppXenDesktop device the end user is connecting to, we track the connections details
under HKLMSoftwareCitrixICASession, each session has a sub‐key noted with the Session ID
which we can get from running Quser on the device, then we can look in the Connection sub‐key
for details
 Here we see the connections Client Name and Client Address, which is good to know in case we’re
filtering on this. Issues with policies not applying might be related to sub‐net changes or a rebuild
of a client device
 On a side note, you can also look up the full client version for connection here as well

for details
of a client device

CTXCseUtil is a Citrix tool that can translate the RSOP.gpf for a user or computer into a html report.
This can be beneficial when troubleshooting policy issues.

Once we have CTXCseUtil on the ServerVDA, we can get to a cmd prompt on that device and run it
(or we can run it against a remote device)
 Usually, when you run CtxCseUtil the first time on a device, you’ll likely get a general WinRM error.
CtxCseUtil needs WinRm configured so it can run.
 The solution to this is to run WinRM quickConfig on the device

Here’s another common issue when running CtxCseUtil.
 This “A specified Logon session” message error is another common issue
 This error usually means that you are trying to get RSOP report when using AD GPO’s and you’ve
logged in with a local Administrators account that doesn’t have access to AD, The solution is to log
in or run the tool with an account that has AD privileges
 NOTE: There are more details on using the tool in the Help Message document located in the same
folder as the tool

 Once you run CtxCseUtil, the resultant report will be created in the same folder. The file name is
CitrixRsopResult.html.
 Here’s a screen clip of the report detailing the User policy section
 Here’s a shot of the Computer policy section, showing what filters were applied or not
 Then lastly the report will show the Setting name, What Value was set and the GPO name the
setting comes from

setting comes from

The next tool we’ll look at is our PowerShell Module for Citrix group policies.
 This powerful tool contains cmdlets for managing all types of Citrix policies
 The module is not built‐in and needs to be imported, I have a link in the resources section to the
module
 It contains cmdlets to: setget policy settings, exportimport policy objects
 The module will importexport policy settings, configuration details and filters used, so a this is the
preferred method to backup your polices for easy restoral

So let’s walk thru using the PowerShell module to export our Farm policies
 Here we have our PowerShell cmd prompt up and we’re in the directory where we’ve copied the
PowerShell module to
 Then we issue the Import‐Module command
 Once Loaded we can use the Get‐Command on the module to see the available functions, we can
also use the Get‐Help command on any of these functions for more details, not only can you import
or export policy settings but you can modify or create them as well. Be sure to have a backup of
your existing policies before modifying your policies

 Then we run the command to export the farm policies to a specific folder
 Once that completes, you should have these files in the folder specified

When exporting Citrix policies from Active Directory GPO’s, there’s a few more steps involved.
• We use the same Citrix PowerShell module and cmdlets
• We create a PowerShell drive to the Active Directory GPO
• We have all of the steps in this Citrix Support article
• In the article we detail how to export settings
And how to import settings from AD GPO’s

Next, at some point you might need or be asked to capture a CDF trace of your policy issue and our
latest CDFControl tool is a great way to expedite this process. <CLICK> Here we see that CDFControl
already has a built in trace category for capturing Citrix Group Policy issues
<CLICK> Zooming in, we can capture all modules, just the CSE or GPMC, if there’s a need to contact
Citrix Technical Support, having a clear snapshot of the issue and details involved will expedite the
resolution process.

Now, I’m going to move on to two specific issues we’ve seen from time to time related to Group Policy
issues. The first one relates a farmIMA policy issue.
 Has anyone ever seen this error when accessing your Computer or User policies within AppCenter
or Studio?
 With farm policies (not AD), the policy object is stored within the datastoredatabase as a single
entity or blob
 This error can occur when the policy object gets corrupted, one possible use case is when multiple
people are access Citrix policies and making changes at the same time, this use case has been
resolved in newer versions of AppCenterStudio
 This error will be seen when you attempt to access either the Computer or User policies node
within AppCenter or Studio
 Try not to just jump in and restore the datastoredatabase for this issue, since you may not know
when the corruption occurred.
 If you do see this error however, please contact Citrix Technical Support for assistance, we have a
set of focused steps to help you resolve this issue
 When using Farm type policies, it’s important to maintain a policy export, this can be used for easy
restoral in cases like this

The second issue I want to bring up relates to the use of WMI filters on your AD GPO’s that contain
Citrix policy settings.
 If you are using WMI filters on your AD GPO’s, it’s possible you may see some issues such as
‐Citrix policies not correct for reconnected sessions
‐Or reconnectslogins taking a extremely long time to complete (especially, if the issue seems to
resolve itself after a while)
 In order to debug this further, it’s best to enable Microsoft Group Policy logging
‐ This can be accomplished using this registry value (I have resource link to a TechNet article on this)
 This will create a log file here, look for this entry related to a FilterCheck API call, if it’s returning this
error you are seeing AD time out on the WMI call
 You should also scan the event logs for WMI related errors, if any of these are seen, you should
contact Microsoft for further assistance, there are several MS hotfixes out there to help with this
issue, but the particular events surrounding this need to be reviewed for your specific environment.

Here are the key items we reviewed during todays presentation.

This section contains references, articles and links to tools that I’ve discussed during the presentation.

Here are links for Group Policy Documentation

Here are links to Citrix Technical Support articles on Group Policy issues and a planning guide

And lastly here are links to tools related to Citrix Group Policy and Microsoft Group Policy.

SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop

SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop

Similar to SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop (20)

More from Citrix

More from Citrix (20)

Recently uploaded

Recently uploaded (20)

SYN406: Citrix Group Policy architecture and troubleshooting for XenApp and XenDesktop