Citrix XenServer 5.5 Troubleshooting


Published on

Citrix XenServer 5.5 Troubleshooting Deep Dive.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The XS installation CD runs a minimal Linux environment, so most standard Linux commands can be used to diagnose install problems. There are three “virtual terminals” available during installation, which display the install program, an interactive console and a detail event log. Use ALT + Function Keys to switch between the virtual terminals. Login to the interactive console with userid ‘root’. Technical Support may request that the command “” be run to create a diagnostic tarball to be submitted when filing a new case about an installation problem.
  • After logging into the interactive virtual terminal (Alt-F2), here are some example of standard Linux commands that provide information about devices. Fdisk will list all disks that can be seen as a result of the loaded storage device drivers. If a particular device driver did not load (e.g. for a RAID card), then the disks attached to that card will not be visible to “fdisk”. Ifconfig will show the network configuration of physical NICs, e.g. IP address and gateway. By default, the XE Install CD assumes the presence of a DHCP server on the local network. If no DHCP server is present, a static IP address can be configured with the command “ifconfig eth0” (replacing with a valid static IP address on your local network). Ping can be used to verify network connectivity from the XE Server to a remote IP address.
  • Xen-bugtool: This is the most important command when submitting an XE Server problem report to Technical Support. It captures static and runtime state on a XenEnterprise Server, for use in diagnosing a reported problem. The compressed file can be unpacked on a Linux system and manually inspected. The support file may contain sensitive information about the customer’s operational environment and should not be posted to public mailing lists. We have also included a command line option that will run xen-bugtool remotely package up the results and upload them to an incoming support server at XenSource. This removes the need for a user to transfer the xen-bugtool output to another computer and then e-mail in. This command can be run on the XenSource server or run remotely from an administrator system.
  • While xen-bugtool is first and foremost a technical support diagnostic tool, the state that it records can be useful in manual emergency recovery of a critical system failure. Such a manual procedure is beyond the scope of this document or field technical staff, but it is recommended that copies of the xen-bugtool tarball be backed up to a safe location off-host from the XE Server.
  • XE proprietary agent uses syslog for all log messages. Can edit /etc/syslog.conf to send messages to central log server. For diagnosis of specific failures that are not easy to reproduce, Technical Support Representative may request an increase in log level, which is performed using the XE CLI (local or remote). xend.log and xend-debug.log contain messages from the Xen OSS tool stack, which does not use syslog (local logs only). All server log files are captured by the xen-bugtool inventory command (see later slide), for filing a case with Technical Support.
  • Administrator Console log files are used to diagnose: connection problem to XE Servers crash of the admin console Java application functional failure of admin console Java application File locations (logs, log property file, client property file) are different in Windows and Linux. For diagnosis of specific failures that are not easy to reproduce, Technical Support Representative may request an increase in log level, which is performed by editing a text file. Windows XE 3.1 clients have a per-user log file. Windows XE 3.0 clients have a central log file. Linux XE 3.0 and 3.1 clients have a per-user log file. When there is a per-user log file and that user needs to run multiple XE console instances (e.g. 3.0 and 3.1 in parallel), the startup script (Linux) or shortcut (Windows) must be manually edited to specify a per-instance log file, instead of the default per-user log file.
  • Citrix XenServer 5.5 Troubleshooting

    1. 1. Citrix® XenServer <ul><li>Webcast 26. März 2010 </li></ul>Advanced Troubleshooting Thomas Krampe Citrix Technology Professional
    2. 2. Agenda 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>Troubleshooting </li></ul><ul><li>Birth and Death of a VM </li></ul><ul><li>Object modell & CLI </li></ul><ul><li>Unattended Setup & Configuration </li></ul><ul><li>Question & Answers </li></ul>
    3. 3. Architecture 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>XenServer Architecture </li></ul><ul><li>Xen Domain0 </li></ul><ul><li>Guest OS Internals </li></ul><ul><li>XAPI </li></ul><ul><li>Storage Manager </li></ul><ul><li>CLI </li></ul><ul><li>GUI (XenCenter) </li></ul>
    4. 4. XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI NFS iSCSI FC
    5. 5. Xen Domain0 04/19/10 (c) 2009 Thomas Krampe <ul><li>Dom0 kernel and initrd passed on start up </li></ul><ul><li>Dom0 has direct access to HW </li></ul><ul><li>Bootloader </li></ul><ul><li>/boot/grub/grub.conf </li></ul><ul><li>/boot/extlinux.conf </li></ul><ul><li>[Ctrl-A] 3 times will trap into the Xen debug console </li></ul>
    6. 6. Guest OS internals 04/19/10 (c) 2009 Thomas Krampe PV Guest <ul><li>Modified Kernel (“Xenified”) </li></ul><ul><li>Kernel name will have “xs” somewhere in the filename </li></ul><ul><li>Guest agent reports VM stats </li></ul><ul><li>Use pygrub bootloader </li></ul>HVM Guest (Windows) <ul><li>Unmodified Kernel </li></ul><ul><li>Need PV drivers </li></ul><ul><li>Event Channel, Network and Block </li></ul><ul><li>Guest agent uses WMI </li></ul>
    7. 7. XAPI 04/19/10 (c) 2009 Thomas Krampe <ul><li>/opt/xensource/bin/xapi </li></ul><ul><li>Service, autostarted on boot </li></ul><ul><li>Implements the Xen API spec </li></ul><ul><li>Talks to Storage Manager (sm) for SR control operations </li></ul><ul><li>Stores configuration in sqlite DB </li></ul><ul><li>/var/xapi/state.db </li></ul><ul><li>sqlite3 /var/xapi/state.db </li></ul><ul><li>DB is replicated on all hosts in a pool </li></ul><ul><li>Sets up SR defaults on firstboot </li></ul><ul><li>Failing firstboot scripts might cause issues </li></ul><ul><li>/var/xapi/generate-* and /var/xapi/firstboot-* </li></ul><ul><li>Defaults from /etc/xensource-inventory </li></ul>
    8. 8. Storage Manager 04/19/10 (c) 2009 Thomas Krampe <ul><li>Manages SR control operations to various backends </li></ul><ul><li>/opt/xensource/sm/ </li></ul><ul><li>Plugin model </li></ul><ul><li>One plugin for each SR type </li></ul><ul><li>/opt/xensource/sm/<Type> </li></ul><ul><li>Error codes listed in /opt/xensource/sm/XE_SR_ERRORCODES.xml </li></ul><ul><li>xe sm-list lists all supported drivers </li></ul>
    9. 9. CLI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Local or Remote </li></ul><ul><li>Talks to XAPI </li></ul><ul><li>Bash auto completion </li></ul><ul><li>xe.exe (for Windows) </li></ul><ul><li>xe help or xe help --all </li></ul>
    10. 10. GUI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Uses Microsoft .NET framework </li></ul><ul><li>Uses https/xml-rpc to communicate with XAPI </li></ul><ul><li>VNC Client for guest frame buffer </li></ul><ul><li>VNC traffic is tunneled over https </li></ul><ul><li>RDP client for Windows guests </li></ul>
    11. 11. Troubleshooting 04/19/10 (c) 2009 Thomas Krampe <ul><li>Troubleshooting </li></ul><ul><li>Server Install </li></ul><ul><li>Useful Linux Commands </li></ul><ul><li>Server Diagnostics & Inventory </li></ul><ul><li>Saving System State </li></ul><ul><li>XS Server Log Files </li></ul><ul><li>XenSource Agent </li></ul><ul><li>XenCenter Logging </li></ul><ul><li>Summery </li></ul>
    12. 12. Server Install 04/19/10 (c) 2009 Thomas Krampe <ul><li>After Product CD Boots </li></ul><ul><li>Alt-F2 displays Linux login prompt </li></ul><ul><ul><li>Login: root (no password required) </li></ul></ul><ul><li>Alt-F3 displays install event log </li></ul><ul><li>Alt-F1 displays original install menu </li></ul><ul><li>Support Tarball </li></ul><ul><li> (creates /tmp/support.tar.bz2) </li></ul><ul><li>Support tarball is created in /tmp on install error </li></ul><ul><li>Support tarball is saved in /root after successful install </li></ul>
    13. 13. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Storage </li></ul><ul><li>List Disks detected </li></ul><ul><ul><li>fdisk –l </li></ul></ul><ul><li>Network </li></ul><ul><li>Show NIC configuration </li></ul><ul><ul><li>ifconfig eth0 [eth1 ..] </li></ul></ul><ul><ul><li>ifup eth0 </li></ul></ul><ul><li>TCP/IP Tools </li></ul><ul><ul><li>ping, route -n </li></ul></ul><ul><ul><li>dhclient eth0 </li></ul></ul>
    14. 14. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Process Management </li></ul><ul><li>ps –Af </li></ul><ul><li>top </li></ul><ul><li>xentop </li></ul>
    15. 15. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>xen-bugtool </li></ul><ul><li>Run on the XS Console </li></ul><ul><li>Records all log files </li></ul><ul><li>Writes compressed file for submission to L2 support </li></ul><ul><li>/var/opt/xen/bug-report/bug-report-xxx.tar.bz2 </li></ul>
    16. 16. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>Automatic Upload for Support </li></ul><ul><li>xe host-bugreport-upload </li></ul><ul><li>Can be run from admin system </li></ul><ul><li>Runs xen-bugtool and uploads to XS Support </li></ul><ul><li>Note unique ID of report for support </li></ul>
    17. 17. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Inventory </li></ul><ul><li> | less </li></ul><ul><li>Interactive script that captures all non-log state </li></ul><ul><li>Hardware devices & Device drivers </li></ul><ul><li>Storage repository and VM configurations </li></ul>
    18. 18. Saving System State 04/19/10 (c) 2009 Thomas Krampe <ul><li>xen-bugtool archives state and configuration files </li></ul><ul><li>Network config </li></ul><ul><li>Install options </li></ul><ul><li>Install logs </li></ul><ul><li>Storage Repository metadata </li></ul><ul><li>VM configs </li></ul><ul><li>Xenstore </li></ul>
    19. 19. XS Server Log Files 04/19/10 (c) 2009 Thomas Krampe <ul><li>XenSource XAPI Agent </li></ul><ul><li>/var/log/messages </li></ul><ul><li>Contains other system logging </li></ul><ul><li>/var/log/xensource.log </li></ul><ul><li>Xen/Domain 0 </li></ul><ul><li>Xen Boot Messages </li></ul><ul><li>xe host-dmesg </li></ul><ul><li>Domain 0 Boot Messages </li></ul><ul><li>dmesg </li></ul>
    20. 20. Debug Logging for XAPI Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>Enable debug logging for all subsystems </li></ul><ul><li>xe log-set-output level=debug output=file:/tmp/debug-all </li></ul><ul><li>Debug for only a certain subsystem </li></ul><ul><li>xe log-set-output level=debug key=storage output=file:/tmp/storage-debug </li></ul><ul><li>List Available Subsystems </li></ul><ul><li>xe log-get-keys </li></ul><ul><li>Restore normal logging </li></ul><ul><li>xe log-set-output level=debug output=nil </li></ul><ul><li>xe log-set-output level=debug key=storage output=nil </li></ul>
    21. 21. XenSource Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>XS Agent runs on top Xen Hypervisor API and XenStore </li></ul><ul><li>Can be restarted without harming VMs or other members of the pool </li></ul><ul><li>service xapi restart </li></ul><ul><li>If the UI seems to get stuck or goes into a weird state, restart the agent and reconnect the UI </li></ul><ul><li>Another simular command is xe-toolstack-restart </li></ul>
    22. 22. XenCenter Logging 04/19/10 (c) 2009 Thomas Krampe <ul><li>Location </li></ul><ul><li>{user home}Application DataXenSourceXenCenterlogsXenCenter.log </li></ul><ul><li>Rolling </li></ul><ul><li>Log files will get rolled </li></ul><ul><li>XenCenter.log.# </li></ul><ul><li>Increase Log Level </li></ul><ul><li>C:Program FilesXenSourceXenCenter directory </li></ul><ul><li><level value=“DEBUG” /> </li></ul>
    23. 23. Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Support Representative. xe host-bugreport-upload runs xen-bugtool and uploads the result to XenSource support. xe host-bugtool can be used to save Domain 0 state, to assist during emergency recovery Use Linux commands (network, storage, modules) to troubleshoot server install.
    24. 24. Birth and Death of a VM 04/19/10 (c) 2009 Thomas Krampe <ul><li>Birth and Death of a VM </li></ul><ul><li>VM Installation Internals </li></ul><ul><li>Behind the Scenes </li></ul><ul><li>Debugging VMs </li></ul>
    25. 25. VM Installation Internals 04/19/10 (c) 2009 Thomas Krampe <ul><li>Debian Sarge/Etch </li></ul><ul><li>Root file system is extracted into the VDI </li></ul><ul><li>Debian Xen kernel </li></ul><ul><li>RHEL (3.x,4.x,5) / SLES (9.x, 10.x) </li></ul><ul><li>Uses regular repositories </li></ul><ul><li>RHEL / SLES Xen kernel </li></ul><ul><li>Windows (all) </li></ul><ul><li>Installs from CD/ISO bits </li></ul><ul><li>Unmodified Windows kernel </li></ul>
    26. 26. Behind the Scenes 04/19/10 (c) 2009 Thomas Krampe <ul><li>Paravirtualized VMs use pygrub </li></ul><ul><li>Simple python based bootloader for Xen </li></ul><ul><li>Finds kernel and initrd from VM’s boot device </li></ul><ul><li>CPU then jumps to this kernel to start this VM </li></ul><ul><li>HVM VMs use hvmloader </li></ul><ul><li>hvmloader creates a virtual BIOS and jumps to it. This is like a BIOS in a physical machine </li></ul><ul><li>qemu-dm does device emulation, provides disks, NICs, serial ports, USB devices etc to this VM </li></ul><ul><li>Follows standard boot process to boot from disk </li></ul>
    27. 27. Debugging Windows VMs 04/19/10 (c) 2009 Thomas Krampe <ul><li>Have client Windows machine ready, install WinDbg </li></ul><ul><li>Add /DEBUGPORT=com1 to boot.ini in VM, shut down VM </li></ul><ul><li>xe vm-param-add uuid=<VM UUID> param-name=other-config:hvm_serial=tcp:<client IP>:7001 </li></ul><ul><li>sockpipe my_pipe 7001 on client machine </li></ul><ul><li>windbg -k com:pipe,port=pipemy_pipe,resets=0 </li></ul><ul><li>Start VM </li></ul><ul><li>Refer to WinDbg manual for commands </li></ul><ul><li> </li></ul>
    28. 28. Object Model and CLI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Object Model and CLI </li></ul><ul><li>Object Model </li></ul><ul><li>CLI Command Format </li></ul><ul><li>Maps </li></ul><ul><li>Constarining Information </li></ul><ul><li>Interesting Commands </li></ul><ul><li>Looking inside a log file </li></ul>
    29. 29. Object Model 04/19/10 (c) 2009 Thomas Krampe
    30. 30. CLI Command Format 04/19/10 (c) 2009 Thomas Krampe <ul><li>Object commands </li></ul><ul><li>xe <class>-<TAB> </li></ul><ul><li>Object Param commands </li></ul><ul><li>xe <class>-param-<TAB> </li></ul><ul><li>Some fields are marked <expensive </li></ul><ul><li>Data Types </li></ul><ul><li>Single Valued/Multi-Valued </li></ul><ul><li>Read Only/Read-Write </li></ul><ul><li>Set/Map </li></ul>
    31. 31. Maps 04/19/10 (c) 2009 Thomas Krampe Setting Map values [root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa Getting Map values [root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config mac_seed: 6fjafhahffhadfbadfad [root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config param-key=mac_seed 6fjafhahffhadfbadfad
    32. 32. Constraining Information 04/19/10 (c) 2009 Thomas Krampe <ul><li>Filtering </li></ul><ul><li>vm-list HVM-boot-policy=&quot;BIOS order&quot; power-state=halted </li></ul><ul><li>Parameterization </li></ul><ul><li>xe vm-list params=name-label </li></ul><ul><li>Minimal Output </li></ul><ul><li>xe vm-list --minimal </li></ul><ul><li>Parameterization and filtering can be combined </li></ul>
    33. 33. Interesting Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Wait for a VM to run </li></ul><ul><li>xe event-wait class=vm name-label=myvm power-state=running </li></ul><ul><li>List of CPUs on a host </li></ul><ul><li>xe host-cpu-list </li></ul><ul><li>CPU Flags must match across all hosts in pool </li></ul><ul><li>List of subsystems </li></ul><ul><li>xe log-get-keys </li></ul><ul><li>Can be used to isolate subsystem logs </li></ul>
    34. 34. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe Each Entry looks like [timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3…. [timestamp] Timestamp of log entry [hostname] Host name (in a pool) [level] Debug level [ref-1] ID of the task on whose behalf this is created num-1 Thread ID, local to the current process unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP) text-1 Subsystem text2/text3 more information on action performed
    35. 35. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe xensource.log [20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done [20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed [root@localhost log]# Xenstored-access.log [20100209 16:49:05.381] D35 write data/meminfo_free 551040 [20100209 16:49:05.381] D35 write data/meminfo_total 1035760 [20100209 16:49:05.381] D35 write data/updated 1 messages Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current task Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid= Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d
    36. 36. Unattend Installation 04/19/10 (c) 2009 Thomas Krampe <ul><li>Possible over TFTP - PXE </li></ul><ul><li>use a single xml configuration file </li></ul>
    37. 37. Questions & Answers 04/19/10 (c) 2009 Thomas Krampe