VMworld 2013 Matt Coppinger, VMware Jack McMichaels, VMware Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare

1. Horizon View Troubleshooting:
Looking under the Hood
Matt Coppinger, VMware
Jack McMichaels, VMware
EUC5291
#EUC5291

2. 2
Disclaimer
 This session may contain product features that are
currently under development.
 This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
 Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
 Technical feasibility and market demand will affect final delivery.
 Pricing and packaging for any new technologies or features
discussed or presented have not been determined.

3. 3
Agenda
 Common Issues
 What can go wrong?
 Domain 1: View Client Connectivity issues
 Domain 2: Desktop Not Available!
 Domain 3: Broken Broker
 Domain 4: Why Is My Desktop So Slow!?
 Resources

4. 4
Top 6 GSS tickets
1. SSL, SSL, SSL – View 5.1/5.2 introduces security improvements
• "Obtaining SSL Certificates for VMware View Servers” PDF on vmware.com
2. Persona – Make sure permissions are set correctly, and avoid
this common misconfiguration error.
http://kb.vmware.com/kb/2019502
3. Parent VM issues – Do NOT P2V! Clean VM please…
4. Connectivity – Understand the View network requirements!
5. PCoIP sizing – PCoIP Calculator -
http://myvirtualcloud.net/?page_id=1562
6. PCoIP Log Viewer - http://mindfluxinc.net

5. 5
Generating a Certificate Signing Request – the Right Way
Read the
Manual!
Create a
config file
Generate cert
signing
request (CSR)
Validate CSR
and Private
Key
Send CSR to
CA
Receive
Signed Cert
from CA
Import Cert
Configure
View Server
to use Cert
Test!

6. 6
The Ultimate Resource
• http://pubs.vmware.com/view-
52/topic/com.vmware.ICbase/PD
F/horizon-view-52-obtaining-
certificates.pdf
Obtaining
SSL
Certificates
for Horizon
View

7. 7
Understanding View Connectivity Requirements

8. 8
Broker Specifications
 Common Issues resulting from undersizing a broker:
• Memory Heap Issues
• Threading issues
• Latency Issues
 Recommendations:
• Always size your broker right from the beginning, avoid downtime or
issues later.
• Minimum recommended specifications:
• 2 vCPU / 10 GB RAM for normal sizing (50-500 VMs)
• 4 vCPU / 16+ GB RAM for large sizing (500+ VMs)

9. 9
Troubleshooting Keys
 Check View
Administrator Dashboard
• Shows system health and any issues
 Understand the client connection
process (where most problems lie)
 Set the Logging Level on CS
 Check Connection Broker Logs
• Match and Filter SessionID, User, FSP
 Check View Agent Logs
 Check View Desktop PCoIP Logs
 Use kb.vmware.com!

10. 10
Tracking Sessions
Use BareTail/BareGrep or similar 3rd party text parsers

11. 11
You’ve Deployed View. What Could Go Wrong?
You’ve got a problem:
• I can’t even connect to View
• I get disconnected randomly!
• Why is the display so bad?
• Why is my desktop not available?
• I’m seeing an error in View, what does it mean?
• vCenter is reporting an error
• My desktop is slow…

12. 12
Identifying the Problem Domain
View Client
Network
View Manager
View Composer
vCenter Server
Compute
Storage

13. 13
Domain 1: View Client Connectivity Issues
 Common challenges
• View Client can’t connect
• Logon failure
• Black screen
• Poor quality display
• Randomly disconnected session

14. 14
Domain 1: Failure to Communicate…
 Where to look
• Connection Broker logs
• Windows 2008 - <DriveLetter>:ProgramDataApplication DataVMwareVDMlogs
• Event Database
 What to look for –
• (Client connects) [SimpleAJPService] (ajp:broker:Request9) Request from
/192.168.2.1: POST /broker/xml
• (Broker authentication) [WinAuthFilter] (SESSION:7072-***-a79c mattc) Attempting
to authenticate user 'mattc' in domain 'FUTUREOFFICE’
• (User has authenticated to Broker) [AuthorizationFilter] (SESSION:7072-***-a79c)
User FUTUREOFFICEmattc has successfully authenticated to VDM
• (Audit Entry) [Audit] (SESSION:7072-***-a79c)
BROKER_LOGON:USER:FUTUREOFFICEmattc;USERSID:S-1-5-21-326850759-
2560684469-1780228732-1113;USERDN:CN=S-1-5-21-326850759-2560684469-
1780228732-1113,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;
• Event Database: BROKER_USERLOGGEDIN

15. 15
User Experience Issues
 Black screen of death – instead of desktop!
• PCoIP port blocked (TCP and UDP 4172) or SVGA Driver issue
• pcoip_server/client logs - C:UsersAll UsersVMwareVDMlogs
• Error attaching to SVGADevTap, error 4000: EscapeFailed
• MGMT_SCHAN :scnet_client_open: tera_sock_connect returned error 10060 -
Connection timed out!
• Incorrect PCoIP External URL configured for Security/Connection Servers

16. 16
User Experience Issues
 Poor quality display
• Bandwidth, latency, or QoS
• Pcoip_server logs report
• VGMAC :Stat frms: Loss=0.45%/0.21% (R/T)
• MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438
 Randomly disconnected session?
• 15 min after established - wssm process hasn't started on desktop
• View Agent logs (<DriveLetter>:ProgramDataVMwareVDMlogs)
• PENDING_EXPIRED
• Sometimes caused by daisy-chaining the GINA (WinXP)

17. 17
Domain 2: Desktop Not Available
 Common Issues
• No Desktop Available
• Pool provisioning issues –
customization
• Agent not communicating
with broker
• Stuck at desktop login
screen (SSO)
 Where to look
• Connection Broker/
View Agent logs
• Event Database
 What to look for
• Broker returns list of desktops available to client
• [DesktopsHandler] (SESSION:7072-***-a79c) For user [S-1-5-21-326850759-
2560684469-1780228732-1113] and pool [cn=gold-np,ou=server
groups,dc=vdi,dc=vmware,dc=int] DesktopTracker returned 2 guest DNs

18. 18
Desktop Not Available
What to look for… (walk through successful connection)
 Client requests desktop
• Event Database: BROKER_DESKTOP_REQUEST
 Broker allocates session to user
• [FarmImp] (SESSION:7072-***-a79c) cn=3f974017-409f-4912-83bc-
2ee794f22fab,ou=servers,dc=vdi,dc=vmware,dc=int, total session count: 0
• [FarmImp] (SESSION:7072-***-a79c) allocateNewSession - identified server
for application CN=GOLD-NP,OU=Applications,DC=vdi,DC=vmware,DC=int
• Event Database: BROKER_MACHINE_ALLOCATED
 Broker attempts SSO
• [FarmImp] (SESSION:7072-***-a79c) Using domain for SSO:
FUTUREOFFICE**
• User won’t be logged on to the VM without this!

19. 19
ADSI Edit – Check Desktop Allocated

20. 20
ADSI Edit – Common Key Values to Inspect
 pae-DisplayName
• VM name as displayed in View Admin
 pae-DirtyForNewSessions
• Indicates whether the VM is “Dirty” and can be re-used in a non-persistent pool
 pae-SVIVMSnapshot
• Indicates the current Snapshot that is in use
 pae-VmPath
• Indicates the full Path to the VM in vCenter
 pae-VmState
• Indicates the current state of the Desktop – some states are a combination of
this value and other values

21. 21
ADSI Edit – Searching for a Desktop
 Find VMs with a Snapshot:
• (&(objectClass=pae-VM)(pae-SVIVmSnapshot=/Baseline/Snapshot1/Snapshot2))
 Find VMs with a Name:
• (&(objectClass=pae-VM)(pae-DisplayName=Desktop-234))

22. 22
Desktop Not Available
What to look for…
 Broker starts session on VM
• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession –
sending StartSession message
 Agent responds…
• "DesktopManager got a StartSession message”
• Client Info should be in Agent Log along with PCoIP launch
 Event Database: AGENT_PENDING
 [DesktopSessionImp] (SESSION:7072-***-a79c) startSession completed:
 [DesktopTracker] User FUTUREOFFICEmattc connected to machine gold-
np-2 for desktop gold-np
 Client connects to VM (Agent)
• “PCoIPCnx::OnConnectionComplete Begin (PCOIP)”
• “WTS_SESSION_LOGON”
• Event Database: AGENT_CONNECTED

23. 23
Event Database

24. 24
Desktop Not Available
What to look for… Pool Provisioning
 Desktops not available due to provisioning error?
• Check View Administrator for Pool status, check datastore capacity
• Check Event Database - BROKER_PROVISIONING_ERROR_*
• Check View Composer has network access to ESX hosts
 Desktop not available due to customization?
• Check Desktop status – AGENT UNAVAILABLE
• Check View Dashboard
• Desktop Status > Preparing Desktops OR Problem Desktops
• Check Desktop connectivity to DNS/AD/Connection Server

25. 25
Desktop Not Available
 Desktop not available due to VM reset/crash?
• Check Desktop status – ALREADY USED
• Typical on refresh-on-logoff or delete-on-use desktops
• Broker never received an explicit logout message from the agent
• Missing AGENT_ENDED event in DB for VM
 View Composer Issues associated with incorrect domain
credentials
• C:ProgramDataVMwareView ComposerLogs
• FATAL CSvmGaService - [svmGaService.cpp, 116] Domain join failed Error 5 (0x5):
Access is denied.

26. 26
Domain 3: Broken Broker
 Common Issues
• Cannot connect to vCenter
• View Composer errors/issues
• JMS connectivity
• ADAM replication failure
 Where to look
• View Administrator
• Event Database
• Windows Event Logs
• View Composer Logs
• Connection Server Logs

27. 27
Broken Broker
What to look for
 ADAM Replication
• Check the Connection Broker Window Event Logs
• ADAM (VMwareVDMDS)-log
• Error: ADAM Replication
• Check ADAM replication status on Connection Server
• C:WINDOWSadamrepadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int

28. 28
Broken Broker
 vCenter Server Connectivity
• Admin UI will show RED status
• Check Event Database
• VC_DOWN events
• Impacts provisioning and power operations ONLY
• Check Connectivity from Connection Server to vCenter Server
• Check credentials used to connect to vCenter Server
• Attempt to login in directly to vCenter using vSphere Client

29. 29
Broken Broker
What to look for…
 View Composer
• VMs stuck in DELETING status
• VMs have been manually deleted – then pool/desktop deleted
• Causes Composer DB and VC DB to get out of sync
• Composer thinks VM already exists
• Orphaned VMs – KB-2015112 (kb.vmware.com)
• Desktop Composer Fault: 'Virtual Machine with Input Specification already exists‘
 JMS Connectivity
• Split site architecture / firewall causes “split brain”
• View Dashboard shows RED status
• Connection Server Logs
• tracker REJOIN messages – JMS connectivity
• tracker RESYNC messages – messages being delayed

30. 30
Cleaning Up Broken / Orphaned Pools
 BACKUP
 Disable Provisioning on Broker
 Stop View Composer
• Remove Composer Database Objects
• Remove ADLDS Servers, Server Groups, and Applications
• Remove AD Computer Entries
• Remove vCenter objects
• Unprotect replicas with sviconfig
• SviConfig -operation=RemoveSviClone -VmName=replica-<guid> -AdminUser=administrator
-AdminPassword=passowrd -ServerUrl=https://localhost:18443/SviService/v2_0
Active Directory View Composer
AD LDS

31. 31
One Query To Rule Them All (Proceed with Extreme Caution!)
 Delete One VM
DELETE FROM dbo.SVI_VM_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_COMPUTER_NAME WHERE NAME='replaceMe'
DELETE FROM dbo.SVI_SC_PDISK_INFO WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')
DELETE FROM dbo.SVI_REQUEST WHERE ID=(SELECT REQUEST_ID FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM
dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'))
DELETE FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'
 Delete ALL VMs and Pools
DELETE FROM dbo.SVI_VM_NAME
DELETE FROM dbo.SVI_COMPUTER_NAME
DELETE FROM dbo.SVI_SC_PDISK_INFO
DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS
DELETE FROM dbo.SVI_TASK_STATE
DELETE FROM dbo.SVI_REQUEST
DELETE FROM dbo.SVI_SIM_CLONE
DELETE FROM dbo.SVI_REPLICA
DELETE FROM dbo.SVI_DG_CUST_PROP
DELETE FROM dbo.SVI_DEPLOYMENT_GROUP
Note: Composer will auto clean replicas when no dependent SIM_CLONE objects are available.

32. 32
Domain 4: Why is My Desktop So Slow?
 Common Issues
• Storage IO bottleneck
• Memory contention
• CPU contention
• Network issues
 Where to look
• vCenter Server
• ESXTOP
• vCops for View
• 3rd Party Tools?

33. 33
Why is My Desktop So Slow?
What to look for
 CPU
• Cluster/Host utilization < 90%
• VM utilization - %USED (ESXTOP)
• VM %RDY Time (ESXTOP) < 10
 Memory
• Host utilization < 85%
• VM utilization
• Swapping / Ballooning SWCUR > 1 / MCTLSZ > 1 (ESXTOP)
 Storage
• Disk Read Latency < 25ms
• ESXTOP DAVG or KAVG < 25ms (ESXTOP)

34. 34
ESX Performance Snapshots
 Use the following command to collect performance metrics for
8 hours on a host:
• for i in `seq 8`;do esxtop -a -b -d 5 -n 720 > $i.<hostname>.csv;done
 The above command will create eight 100mb files consisting of
1 hour's worth of ESXTOP snapshots.
• -d = delay in seconds
• -n = iterations
• (-d 5 x -n 720 = 3600 seconds or 1 hour).
 Useful for replaying performance data over wide time periods
for support to analyze!

35. 35
CPU

36. 36
Memory

37. 37
Storage

38. 38
vSphere Performance Management Tools – vCenter Ops Mgr
vCenter Operations
Manager
• Aggregates metrics into
workload, capacity and
health scores
• Relies on
dynamic thresholds

39. 39
VCOPs – My Favorite Tool
DatastoreLatencyDatastoreThroughput

40. 40
Getting Help
 Read the product documentation
 Double check your configuration!
 Check kb.vmware.com for your issue
 http://communities.vmware.com
 Run the Support.bat to extract the logs
• Notice the diagnostic tests that run…
• http://kb.vmware.com/kb/1017939
• Check the product documentation for using VDMADMIN command for creating
various Data Collection Tool bundles
 Submit a Support Request

41. 41
Summary
 Understand where the issue may lie
• Client? Agent? Server? Composer? vCenter? ESX?
 Know the problem domains
 Check the View Dashboard and Event Database
 Identify the issue – know what a successful connections looks like
 Check the logs
 Use Performance Tools – ESXTOP, vCenter, vCenter Operations
for View
 Get Help

42. 42
Questions

43. 43
Other VMware Activities Related to This Session
 HOL:
HOL-MBL-1301
Horizon View from A to Z
 Group Discussions:
EUC1006-GD
View with Andre Leibovici

44. THANK YOU

46. Horizon View Troubleshooting:
Looking under the Hood
Matt Coppinger, VMware
Jack McMichaels, VMware
EUC5291
#EUC5291