Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Top Troubleshooting Tips and Techniques for Citrix XenServer Deployments

48,620 views

Published on

This session will provide an expert insight into the most common issues encountered by Customers, Partners and Support engineers.
It’s a feature packed agenda which gets to the point quickly and concentrates on the issues we encounter continuously with XenServer deployments.

Published in: Technology
  • My life is back!!! After 1 years of Broken marriage, my husband left me with two kids . I felt like my life was about to end i almost committed suicide, i was emotionally down for a very long time. Thanks to a spell caster called Dr Mohammed, which i met online. On one faithful day, as I was browsing through the internet,I came across allot of testimonies about this particular spell caster. Some people testified that he brought their Ex lover back, some testified that he restores womb,cure cancer,and other sickness, some testified that he can cast a spell to stop divorce and so on. i also come across one particular testimony,it was about a woman called Sonia,she testified about how he brought back her Ex lover in less than 2 days, and at the end of her testimony she dropped Dr Mohammed's e-mail address. After reading all these,I decided to give it a try. I contacted him via email and explained my problem to him. In just 48hours, my husband came back to me. We solved our issues, and we are even happier than before Dr Mohammed, is really a gifted man and i will not stop publishing him because he is a wonderful man... If you have a problem and you are looking for a real and genuine spell caster to solve all your problems for you. Try High monicaspiritualtemple@gmail.com anytime, he might be the answer to your problems. Here's his contact: monicaspiritualtemple@gmail.com or call him +2348134493948
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • The recording of this webinar can be found here: https://www1.gotomeeting.com/register/177877672
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Top Troubleshooting Tips and Techniques for Citrix XenServer Deployments

  1. 1. June 27, 2013 Citrix Support Secrets Webinar Series Top Troubleshooting Tips and Techniques for Citrix XenServer Deployments Mark Butterly, Senior Readiness Specialist Giovanni Di Tizio, Technical Relationship Manager Mark Butterly Giovanni Di Tizio
  2. 2. Introduction • Storage Management and Snapshot Secrets ᵒHow it works - How space is utilized • The „VDI not available‟ problem • Hotfixes and Driver installation • XenTools / XenTools installation • Q+A Tips, Tricks + Techniques
  3. 3. Storage Management and Snapshot Secrets
  4. 4. Control Domain (Dom 0) XenServer Architecture Xen Hypervisor Hardware Virtual Machines Local Storage Network Card Remote or SAN based Storage SSH Linux XAPI
  5. 5. Thin vs. Thick Provisioning
  6. 6. Snapshots • Disk and Memory ! • Enables Backup vendors to interface directly with XenServer • Snapshots available on all storage platforms • Thin vs Thick Provisioning
  7. 7. 400 Snapshot (NFS and EXT Local Storage) • Resulting VDI tree • Disk utilization ᵒVHD files thin provisioned ᵒVDI A contains writes up to point of snapshot ᵒVDI B and C are empty* ᵒTotal: • VDI A: 20 • VDI B: 0* • VDI C: 0* ᵒSnapshot requires no space* A B 20 40 40 C 0 (1)(2) (1) Size of VDI (2) Data written in VDI Key Snapshot CloneParent Active* Plus VHD headers
  8. 8. Snapshot (Local LVHD, iSCSI or FC SR) • Resulting VDI tree • Disk utilization ᵒVolumes are thick provisioned ᵒDeflated where possible ᵒTotal: • VDI A: 20 • VDI B: 40* • VDI C: 0* ᵒSnapshot requires 40 + 20GB A 4020 400 B 40 C 0 (3) (1)(2) (1) Size of VDI (2) Data written in VDI (3) Inflated / deflated state Key Snapshot CloneParent Active * Plus VHD headers
  9. 9. Automated Coalescing Example 1) VM with two snapshots, C and E A CB D E A + B 3) Parent B is no longer required and will be coalesced into A D E Key Snapshot CloneParent Active 2) When snapshot C is deleted… A B D E http://support.citrix.com/article/CTX122978
  10. 10. Snapshot - LVHD based SR example • Space cost = existing data on disk + VDI size • Empty VDI size on disk = 8.00M • Example: vhd-util scan -m 'V*' -l <SR_VG> -p vhd-util check -n <LV_PATH> LV needs to be: available (LV active). Error 2, LV not active. LV read/write or read only - check gives different output
  11. 11. VHD Repair Example: vhd-util repair -n <LV_PATH> Repair command can fix VHD cookies and other VHD header elements only if backup header is present and valid. Repair is unable to fix VHD content, data.
  12. 12. VHD Chains • VDI chain growth • VDI chain hits length limit = 30 ᵒError code: SR_BACKEND_FAILURE_109 Error parameters: , The snapshot chain is too long • Trigger coalescing • VHD format introduces overhead on IO ᵒwrite overhead added by VHD format • Read overhead of VHD format multiplied by chain length
  13. 13. Key tips for handling Snapshots • For "golden image“: ᵒCreate a new VM and convert to template ᵒDo not create it based on a snapshot • Full vdi-copy is the only way to reduce VHD chain to 0 • NEVER EVER MANUALLY REMOVE ANY VDI(s) FROM A CHAIN!! • Manually reclaim disk space: http://support.citrix.com/article/CTX123400 • Don’t use Snapshots as a backup. ᵒBad use cases – Databases, MS Exchange,
  14. 14. VDI not Available
  15. 15. VDI not available… It may happen for different reasons, this are the 2 more common: • The VDI is actually missing (maybe deleted by mistake) • The VDI is locked to a host where the VM was running.
  16. 16. VDI is actually missing… In this example we can see that our VM “Linux03” has 2 VDIs assigned:
  17. 17. VDI is actually missing… Here we can see further details about the VDIs:
  18. 18. VDI is actually missing… If might happen that someone (by mistake of course) deletes one of the VDIs (in this example Test03_0): Note: Renaming the LV will have the same effect.
  19. 19. VDI is actually missing… When you try to start the VM you get on XenCenter: But if you try from the CLI we can see further information:
  20. 20. VDI is actually missing… If you check on SMlog you can see:
  21. 21. VDI is actually missing… Summary • In this case the only way to recover would be to restore the missing VDI. • Is not uncommon on NFS scenarios that files might be removed, very unlikely in a LVS scenario. • Starting the VM from the CLI provides a quick way of knowing what is the problem.
  22. 22. The VDI is locked to a host where the VM was running… To prevent data corruption XAPI keeps a lock on the VDI of the VMs in use. The lock is indicated by: host_OpaqueRef, under the parameter: sm-config of the VDI. In this example we are going to use a VM called Linux01 which is already running:
  23. 23. The VDI is locked to a host where the VM was running… In this example we are going to use a VM called Linux01 which is already running:
  24. 24. The VDI is locked to a host where the VM was running… When we see the details of one the VDIs the “host_OpaqueRef” setting is indeed there:
  25. 25. The VDI is locked to a host where the VM was running… If we dig into XAPI we can see that it refers to a host: In this case “xs-lab02-giovad”
  26. 26. The VDI is locked to a host where the VM was running… When the VM is not running that parameter is cleared from XAPI
  27. 27. The VDI is locked to a host where the VM was running… So what happens if the parameter is stale in XAPI and the VM is no longer running: This could happen if the host were the VM is running dies or becomes unresponsive.
  28. 28. The VDI is locked to a host where the VM was running… When you try to start the VM in XenCenter you will see: While in the CLI you will see instead:
  29. 29. The VDI is locked to a host where the VM was running… In /var/log/SMlog you will see:
  30. 30. The VDI is locked to a host where the VM was running… Troubleshooting: • Try to start the VM from the CLI, this will provide further details about the error, in this case “The VDI is not available… Already attached RW” • The error showed on the CLI or SMlog will also indicate which is the problematic VDI. • Further confirmation can be gathered by checking for the “host_OpaqueRef” in the sm_config parameter of the VDI. • Make sure the VM is not running and the power status is actually wrongly set in XAPI. Run list_domains in all the hosts of the pool and grep for the uuid of the VM.
  31. 31. The VDI is locked to a host where the VM was running… How to Recover: • We need to clear host_OpaqueRef but the parameter is RO • Basically there are 2 options: 1. Manually modify XAPI‟s database (VERY DANGEROUS and Completely Unsupported) 2. Forget the VDI and add it again will clear the host_OpaqueRef parameter.
  32. 32. The VDI is locked to a host where the VM was running… How to Recover: • Once the VDI has been forgotten we do a SR scan to add it back • It will show with no name and no description, we should rename it accordingly using the properties option in XenCenter
  33. 33. The VDI is locked to a host where the VM was running… How to Recover: • We then re-attach the disk to the VM
  34. 34. The VDI is locked to a host where the VM was running… How to Recover: • We then re-attach the disk to the VM • And if we check for host_OpaqueRef we‟ll that is gone…
  35. 35. The VDI is locked to a host where the VM was running… How to Recover: • At this stage we can start the VM again
  36. 36. The VDI is locked to a host where the VM was running… Tips: • If you have many VDIs which are the same size it will be hard to know which one is the correct one if you forget many at once, so better do one at the time. • The order in which you attach the VDI is important so this also needs to be done in order. • In XenServer 6.1 we provide a script that does this automatically for all the VMs in the missing host as part of the procedure to recover from missing members of a pool, see: http://docs.vmd.citrix.com/XenServer/6.1.0/1.0/en_gb/reference.html#pool_failures
  37. 37. Hotfixes and driver installation
  38. 38. Hotfixes and driver installation • XenServer uses industry-standard open source device drivers. • Is not possible for Citrix to test every piece of hardware available so we rely on the vendors for the testing and certification. • Driver are updated on a regular basis to fix bugs or improve performance, however a driver must always match the firmware of the device. Installing a newer driver on a very old firmware will cause more problems than solution the same happens the other way around.
  39. 39. Hotfixes and driver installation My device doesn‟t work anymore after applying a HotFix!! • Some hotfixes provide a new kernel, therefore we need to install the driver that matches the new kernel • For example Hotfix XS602E021 - For XenServer 6.0.2 provides a new kernel:
  40. 40. Hotfixes and driver installation My device doesn‟t work anymore after applying a HotFix!! • Therefore we need to install the drivers compiled for that kernel version:
  41. 41. Hotfixes and driver installation How do I know which driver/firmware am I running? • In the case of network interfaces “ethtool -i” will help: • If you know the name of the driver you could also use modinfo, for example: # ethtool -i eth0 driver: bnx2 version: 2.2.1j firmware-version: 7.2.20 bc 5.2.3 NCSI 2.0.11 bus-info: 0000:01:00.0
  42. 42. Hotfixes and driver installation How do I know which driver/firmware am I running? • For HBAs drivers you will need to check on kern.log during boot time, for example: • Or use modinfo if you know the driver involved:
  43. 43. Hotfixes and driver installation How do I know which driver/firmware am I running? • For HBAs firmware you will need to check on the HBA BIOS during boot time, the way to access it changes from vendor to vendor so check with them if in doubt. • Qlogic HBA load the firmware when they load the driver so unless is running a very old firmware (previous to version 4.x) there‟s no need to verify the version.
  44. 44. Xenserver Tools (aka. XenTools)
  45. 45. Why XenServer Tools (aka XenTools)? • Optimize your VMs !!! • Provides high performance drivers and a management agent • Always install after a new VM to make it optimized
  46. 46. Legacy support for drivers (XenServer 6.1) • For XP & Windows Server 2003:
  47. 47. To make things simpler …. • For All Versions of Windows: • Install Wizard provided • .Net 3.5 or above is required • 2 parts/phases to install • MetaInstaller needs to be installed first • Drivers are then installed
  48. 48. Troubleshooting … • If the install wizard installer goes wrong, run it again with the argument • /log logfile.txt • If it still goes wrong, look in • C:programdatacitrixInstallWizard* •Logs the driver and agent msi install logs •Install.txt has other (verbose) install message • The old installer has install.log in c:program filesxentools as before • Watch the Antivirius
  49. 49. Issues with XenTools (XenServer 6.1) • VMs with out of date XenServer tools may not be shown in XenCenter • Slow boots for a PVS target device using a Boot Device Manager • PVS licensing issues • Intermittent blue screen errors when shutting down MS Vista VMs • Adding more than eight NICs to MS Vista causing blue screens • Copying data to a MS Windows 2003 VM causes the VMs to hang / grey screen • When Dynamic Memory Control (DMC) is enabled, using XenMotion causes VMs to hang and blue screen.
  50. 50. Issues with XenTools (XenServer 6.1) • Cut and paste issues between a XenDesktop VDA and an endpoint (when running Citrix Xen Guest agent is running) • Windows XP and later VMs intermittently hanging during the boot process • Installation hangs when installing tools on Vista / later VMs without access to PV / emulated network device • Manually installing legacy tools without changing a device id to 0001 can result in a blue screen • VSS (required for third party backup solutions) is now available for use with XenServer 6.1
  51. 51. • Apply the following Hotfixes …… Resolving issues with XenTools XS61E009 XS61E010
  52. 52. XenTools Troubleshooting http://support.citrix.com/article/CTX135099 • Consult the following …… particularly if you have applied workarounds to fix the issues discussed.
  53. 53. • Drivers model restructuring allows independent upgrades • Deployment of bug fixes and new features out of band • No need for ISO injection • Updates can be installed automatically at domain administrator‟s discretion • WSUS deployments not yet available but will be made available Future Windows Update Support
  54. 54. About Citrix Services Citrix Services make sure you succeed with your virtualization programs. How we can help Citrix Education – The fastest, most efficient way to get your team the virtualization skills they need. Online, on-site or in class. citrix.com/training Citrix Consulting – Intensive engagements for complex, critical or just plain massive projects. citrix.com/consulting Citrix Support – Always-on support services that leverage everything we know about best-practice deployment and maintenance. citrix.com/support Educate | Guide | Support | Succeed
  55. 55. • 40 insider troubleshooting tips • Covering XenDesktop, XenServer, XenApp and NetScaler • Citrix Support top engineers • FREE eBook • Citrix Auto Support • Now available! Secrets of the Citrix Support Ninjas
  56. 56. Premier Support Calculator Check it out
  57. 57. Next Webinar: July • Title: Troubleshooting XenApp with the Citrix Diagnostic Toolkit • Description: When problems occur, support engineers need data points, debug tracing and context information to help determine root causes. Preparation and organization of commonly used tools has always been a time- consuming challenge, especially during outages. The Citrix diagnostics toolkit (CDT) addresses these challenges by rapidly deploying a suite of tools and options in an easy-to-use structured format. • When: July 25th • Registration Now!
  58. 58. Work better. Live better.

×