Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Citrix® XenServer <ul><li>Webcast 26. März 2010 </li></ul>Advanced Troubleshooting Thomas Krampe Citrix Technology Profess...
Agenda 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>Troubleshooting </li></ul><ul><li>Birth and ...
Architecture 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>XenServer Architecture </li></ul><ul><...
XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI ...
Xen Domain0 04/19/10 (c) 2009 Thomas Krampe <ul><li>Dom0 kernel and initrd passed on start up </li></ul><ul><li>Dom0 has d...
Guest OS internals 04/19/10 (c) 2009 Thomas Krampe PV Guest  <ul><li>Modified Kernel (“Xenified”) </li></ul><ul><li>Kernel...
XAPI 04/19/10 (c) 2009 Thomas Krampe <ul><li>/opt/xensource/bin/xapi </li></ul><ul><li>Service, autostarted on boot </li><...
Storage Manager 04/19/10 (c) 2009 Thomas Krampe <ul><li>Manages SR control operations to various backends </li></ul><ul><l...
CLI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Local or Remote </li></ul><ul><li>Talks to XAPI </li></ul><ul><li>Bash auto co...
GUI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Uses Microsoft .NET framework </li></ul><ul><li>Uses https/xml-rpc to communic...
Troubleshooting 04/19/10 (c) 2009 Thomas Krampe <ul><li>Troubleshooting </li></ul><ul><li>Server Install </li></ul><ul><li...
Server Install 04/19/10 (c) 2009 Thomas Krampe <ul><li>After Product CD Boots </li></ul><ul><li>Alt-F2  displays Linux log...
Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Storage </li></ul><ul><li>List Disks detected </li></ul><ul>...
Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Process Management </li></ul><ul><li>ps –Af </li></ul><ul><l...
Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>xen-bugtool ...
Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>Automatic Up...
Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Inventory </li></ul><ul><li>system-info.sh...
Saving System State 04/19/10 (c) 2009 Thomas Krampe <ul><li>xen-bugtool  archives state and configuration files </li></ul>...
XS Server Log Files 04/19/10 (c) 2009 Thomas Krampe <ul><li>XenSource XAPI Agent </li></ul><ul><li>/var/log/messages  </li...
Debug Logging for XAPI Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>Enable debug logging for all subsystems </li></ul><ul...
XenSource Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>XS Agent runs on top Xen Hypervisor API and XenStore </li></ul><ul...
XenCenter Logging 04/19/10 (c) 2009 Thomas Krampe <ul><li>Location </li></ul><ul><li>{user home}Application DataXenSourceX...
Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Suppor...
Birth and Death of a VM 04/19/10 (c) 2009 Thomas Krampe <ul><li>Birth and Death of a VM </li></ul><ul><li>VM Installation ...
VM Installation Internals 04/19/10 (c) 2009 Thomas Krampe <ul><li>Debian Sarge/Etch </li></ul><ul><li>Root file system is ...
Behind the Scenes 04/19/10 (c) 2009 Thomas Krampe <ul><li>Paravirtualized VMs use pygrub </li></ul><ul><li>Simple python b...
Debugging Windows VMs 04/19/10 (c) 2009 Thomas Krampe <ul><li>Have client Windows machine ready, install WinDbg </li></ul>...
Object Model and CLI  04/19/10 (c) 2009 Thomas Krampe <ul><li>Object Model and CLI  </li></ul><ul><li>Object Model </li></...
Object Model 04/19/10 (c) 2009 Thomas Krampe
CLI Command Format 04/19/10 (c) 2009 Thomas Krampe <ul><li>Object commands </li></ul><ul><li>xe <class>-<TAB> </li></ul><u...
Maps 04/19/10 (c) 2009 Thomas Krampe Setting Map values [root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa...
Constraining Information 04/19/10 (c) 2009 Thomas Krampe <ul><li>Filtering </li></ul><ul><li>vm-list HVM-boot-policy=&quot...
Interesting Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Wait for a VM to run </li></ul><ul><li>xe event-wait class=vm...
Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe Each Entry looks like [timestamp][hostname][level] [ref-1] [(num...
Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe xensource.log [20100209 16:58:28.688|debug|localhost|3078703|SR ...
Unattend Installation 04/19/10 (c) 2009 Thomas Krampe <ul><li>Possible over TFTP - PXE </li></ul><ul><li>use a single xml ...
Questions & Answers 04/19/10 (c) 2009 Thomas Krampe
Upcoming SlideShare
Loading in …5
×

Citrix XenServer 5.5 Troubleshooting

25,811 views

Published on

Citrix XenServer 5.5 Troubleshooting Deep Dive.

Published in: Technology
  • Login to see the comments

Citrix XenServer 5.5 Troubleshooting

  1. 1. Citrix® XenServer <ul><li>Webcast 26. März 2010 </li></ul>Advanced Troubleshooting Thomas Krampe Citrix Technology Professional
  2. 2. Agenda 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>Troubleshooting </li></ul><ul><li>Birth and Death of a VM </li></ul><ul><li>Object modell & CLI </li></ul><ul><li>Unattended Setup & Configuration </li></ul><ul><li>Question & Answers </li></ul>
  3. 3. Architecture 04/19/10 (c) 2009 Thomas Krampe <ul><li>Architecture </li></ul><ul><li>XenServer Architecture </li></ul><ul><li>Xen Domain0 </li></ul><ul><li>Guest OS Internals </li></ul><ul><li>XAPI </li></ul><ul><li>Storage Manager </li></ul><ul><li>CLI </li></ul><ul><li>GUI (XenCenter) </li></ul>
  4. 4. XenServer Architecture 04/19/10 (c) 2009 Thomas Krampe SM Virtualized Hardware Hardware VT/AMD-V XAPI Drivers Storage CLI NFS iSCSI FC
  5. 5. Xen Domain0 04/19/10 (c) 2009 Thomas Krampe <ul><li>Dom0 kernel and initrd passed on start up </li></ul><ul><li>Dom0 has direct access to HW </li></ul><ul><li>Bootloader </li></ul><ul><li>/boot/grub/grub.conf </li></ul><ul><li>/boot/extlinux.conf </li></ul><ul><li>[Ctrl-A] 3 times will trap into the Xen debug console </li></ul>
  6. 6. Guest OS internals 04/19/10 (c) 2009 Thomas Krampe PV Guest <ul><li>Modified Kernel (“Xenified”) </li></ul><ul><li>Kernel name will have “xs” somewhere in the filename </li></ul><ul><li>Guest agent reports VM stats </li></ul><ul><li>Use pygrub bootloader </li></ul>HVM Guest (Windows) <ul><li>Unmodified Kernel </li></ul><ul><li>Need PV drivers </li></ul><ul><li>Event Channel, Network and Block </li></ul><ul><li>Guest agent uses WMI </li></ul>
  7. 7. XAPI 04/19/10 (c) 2009 Thomas Krampe <ul><li>/opt/xensource/bin/xapi </li></ul><ul><li>Service, autostarted on boot </li></ul><ul><li>Implements the Xen API spec </li></ul><ul><li>Talks to Storage Manager (sm) for SR control operations </li></ul><ul><li>Stores configuration in sqlite DB </li></ul><ul><li>/var/xapi/state.db </li></ul><ul><li>sqlite3 /var/xapi/state.db </li></ul><ul><li>DB is replicated on all hosts in a pool </li></ul><ul><li>Sets up SR defaults on firstboot </li></ul><ul><li>Failing firstboot scripts might cause issues </li></ul><ul><li>/var/xapi/generate-* and /var/xapi/firstboot-* </li></ul><ul><li>Defaults from /etc/xensource-inventory </li></ul>
  8. 8. Storage Manager 04/19/10 (c) 2009 Thomas Krampe <ul><li>Manages SR control operations to various backends </li></ul><ul><li>/opt/xensource/sm/sm.py </li></ul><ul><li>Plugin model </li></ul><ul><li>One plugin for each SR type </li></ul><ul><li>/opt/xensource/sm/<Type>SR.py </li></ul><ul><li>Error codes listed in /opt/xensource/sm/XE_SR_ERRORCODES.xml </li></ul><ul><li>xe sm-list lists all supported drivers </li></ul>
  9. 9. CLI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Local or Remote </li></ul><ul><li>Talks to XAPI </li></ul><ul><li>Bash auto completion </li></ul><ul><li>xe.exe (for Windows) </li></ul><ul><li>xe help or xe help --all </li></ul>
  10. 10. GUI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Uses Microsoft .NET framework </li></ul><ul><li>Uses https/xml-rpc to communicate with XAPI </li></ul><ul><li>VNC Client for guest frame buffer </li></ul><ul><li>VNC traffic is tunneled over https </li></ul><ul><li>RDP client for Windows guests </li></ul>
  11. 11. Troubleshooting 04/19/10 (c) 2009 Thomas Krampe <ul><li>Troubleshooting </li></ul><ul><li>Server Install </li></ul><ul><li>Useful Linux Commands </li></ul><ul><li>Server Diagnostics & Inventory </li></ul><ul><li>Saving System State </li></ul><ul><li>XS Server Log Files </li></ul><ul><li>XenSource Agent </li></ul><ul><li>XenCenter Logging </li></ul><ul><li>Summery </li></ul>
  12. 12. Server Install 04/19/10 (c) 2009 Thomas Krampe <ul><li>After Product CD Boots </li></ul><ul><li>Alt-F2 displays Linux login prompt </li></ul><ul><ul><li>Login: root (no password required) </li></ul></ul><ul><li>Alt-F3 displays install event log </li></ul><ul><li>Alt-F1 displays original install menu </li></ul><ul><li>Support Tarball </li></ul><ul><li>support.sh (creates /tmp/support.tar.bz2) </li></ul><ul><li>Support tarball is created in /tmp on install error </li></ul><ul><li>Support tarball is saved in /root after successful install </li></ul>
  13. 13. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Storage </li></ul><ul><li>List Disks detected </li></ul><ul><ul><li>fdisk –l </li></ul></ul><ul><li>Network </li></ul><ul><li>Show NIC configuration </li></ul><ul><ul><li>ifconfig eth0 [eth1 ..] </li></ul></ul><ul><ul><li>ifup eth0 </li></ul></ul><ul><li>TCP/IP Tools </li></ul><ul><ul><li>ping, route -n </li></ul></ul><ul><ul><li>dhclient eth0 </li></ul></ul>
  14. 14. Useful Linux Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Process Management </li></ul><ul><li>ps –Af </li></ul><ul><li>top </li></ul><ul><li>xentop </li></ul>
  15. 15. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>xen-bugtool </li></ul><ul><li>Run on the XS Console </li></ul><ul><li>Records all log files </li></ul><ul><li>Writes compressed file for submission to L2 support </li></ul><ul><li>/var/opt/xen/bug-report/bug-report-xxx.tar.bz2 </li></ul>
  16. 16. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Diagnostics </li></ul><ul><li>Automatic Upload for Support </li></ul><ul><li>xe host-bugreport-upload </li></ul><ul><li>Can be run from admin system </li></ul><ul><li>Runs xen-bugtool and uploads to XS Support </li></ul><ul><li>Note unique ID of report for support </li></ul>
  17. 17. Server Diagnostics and Inventory 04/19/10 (c) 2009 Thomas Krampe <ul><li>Server Inventory </li></ul><ul><li>system-info.sh | less </li></ul><ul><li>Interactive script that captures all non-log state </li></ul><ul><li>Hardware devices & Device drivers </li></ul><ul><li>Storage repository and VM configurations </li></ul>
  18. 18. Saving System State 04/19/10 (c) 2009 Thomas Krampe <ul><li>xen-bugtool archives state and configuration files </li></ul><ul><li>Network config </li></ul><ul><li>Install options </li></ul><ul><li>Install logs </li></ul><ul><li>Storage Repository metadata </li></ul><ul><li>VM configs </li></ul><ul><li>Xenstore </li></ul>
  19. 19. XS Server Log Files 04/19/10 (c) 2009 Thomas Krampe <ul><li>XenSource XAPI Agent </li></ul><ul><li>/var/log/messages </li></ul><ul><li>Contains other system logging </li></ul><ul><li>/var/log/xensource.log </li></ul><ul><li>Xen/Domain 0 </li></ul><ul><li>Xen Boot Messages </li></ul><ul><li>xe host-dmesg </li></ul><ul><li>Domain 0 Boot Messages </li></ul><ul><li>dmesg </li></ul>
  20. 20. Debug Logging for XAPI Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>Enable debug logging for all subsystems </li></ul><ul><li>xe log-set-output level=debug output=file:/tmp/debug-all </li></ul><ul><li>Debug for only a certain subsystem </li></ul><ul><li>xe log-set-output level=debug key=storage output=file:/tmp/storage-debug </li></ul><ul><li>List Available Subsystems </li></ul><ul><li>xe log-get-keys </li></ul><ul><li>Restore normal logging </li></ul><ul><li>xe log-set-output level=debug output=nil </li></ul><ul><li>xe log-set-output level=debug key=storage output=nil </li></ul>
  21. 21. XenSource Agent 04/19/10 (c) 2009 Thomas Krampe <ul><li>XS Agent runs on top Xen Hypervisor API and XenStore </li></ul><ul><li>Can be restarted without harming VMs or other members of the pool </li></ul><ul><li>service xapi restart </li></ul><ul><li>If the UI seems to get stuck or goes into a weird state, restart the agent and reconnect the UI </li></ul><ul><li>Another simular command is xe-toolstack-restart </li></ul>
  22. 22. XenCenter Logging 04/19/10 (c) 2009 Thomas Krampe <ul><li>Location </li></ul><ul><li>{user home}Application DataXenSourceXenCenterlogsXenCenter.log </li></ul><ul><li>Rolling </li></ul><ul><li>Log files will get rolled </li></ul><ul><li>XenCenter.log.# </li></ul><ul><li>Increase Log Level </li></ul><ul><li>C:Program FilesXenSourceXenCenter directory </li></ul><ul><li><level value=“DEBUG” /> </li></ul>
  23. 23. Summary 04/19/10 (c) 2009 Thomas Krampe xen-bugtool generates XE Server diagnostic file for submission to Technical Support Representative. xe host-bugreport-upload runs xen-bugtool and uploads the result to XenSource support. xe host-bugtool can be used to save Domain 0 state, to assist during emergency recovery Use Linux commands (network, storage, modules) to troubleshoot server install.
  24. 24. Birth and Death of a VM 04/19/10 (c) 2009 Thomas Krampe <ul><li>Birth and Death of a VM </li></ul><ul><li>VM Installation Internals </li></ul><ul><li>Behind the Scenes </li></ul><ul><li>Debugging VMs </li></ul>
  25. 25. VM Installation Internals 04/19/10 (c) 2009 Thomas Krampe <ul><li>Debian Sarge/Etch </li></ul><ul><li>Root file system is extracted into the VDI </li></ul><ul><li>Debian Xen kernel </li></ul><ul><li>RHEL (3.x,4.x,5) / SLES (9.x, 10.x) </li></ul><ul><li>Uses regular repositories </li></ul><ul><li>RHEL / SLES Xen kernel </li></ul><ul><li>Windows (all) </li></ul><ul><li>Installs from CD/ISO bits </li></ul><ul><li>Unmodified Windows kernel </li></ul>
  26. 26. Behind the Scenes 04/19/10 (c) 2009 Thomas Krampe <ul><li>Paravirtualized VMs use pygrub </li></ul><ul><li>Simple python based bootloader for Xen </li></ul><ul><li>Finds kernel and initrd from VM’s boot device </li></ul><ul><li>CPU then jumps to this kernel to start this VM </li></ul><ul><li>HVM VMs use hvmloader </li></ul><ul><li>hvmloader creates a virtual BIOS and jumps to it. This is like a BIOS in a physical machine </li></ul><ul><li>qemu-dm does device emulation, provides disks, NICs, serial ports, USB devices etc to this VM </li></ul><ul><li>Follows standard boot process to boot from disk </li></ul>
  27. 27. Debugging Windows VMs 04/19/10 (c) 2009 Thomas Krampe <ul><li>Have client Windows machine ready, install WinDbg </li></ul><ul><li>Add /DEBUGPORT=com1 to boot.ini in VM, shut down VM </li></ul><ul><li>xe vm-param-add uuid=<VM UUID> param-name=other-config:hvm_serial=tcp:<client IP>:7001 </li></ul><ul><li>sockpipe my_pipe 7001 on client machine </li></ul><ul><li>windbg -k com:pipe,port=pipemy_pipe,resets=0 </li></ul><ul><li>Start VM </li></ul><ul><li>Refer to WinDbg manual for commands </li></ul><ul><li>http://www.microsoft.com/whdc/DevTools/Debugging/default.mspx </li></ul>
  28. 28. Object Model and CLI 04/19/10 (c) 2009 Thomas Krampe <ul><li>Object Model and CLI </li></ul><ul><li>Object Model </li></ul><ul><li>CLI Command Format </li></ul><ul><li>Maps </li></ul><ul><li>Constarining Information </li></ul><ul><li>Interesting Commands </li></ul><ul><li>Looking inside a log file </li></ul>
  29. 29. Object Model 04/19/10 (c) 2009 Thomas Krampe
  30. 30. CLI Command Format 04/19/10 (c) 2009 Thomas Krampe <ul><li>Object commands </li></ul><ul><li>xe <class>-<TAB> </li></ul><ul><li>Object Param commands </li></ul><ul><li>xe <class>-param-<TAB> </li></ul><ul><li>Some fields are marked <expensive </li></ul><ul><li>Data Types </li></ul><ul><li>Single Valued/Multi-Valued </li></ul><ul><li>Read Only/Read-Write </li></ul><ul><li>Set/Map </li></ul>
  31. 31. Maps 04/19/10 (c) 2009 Thomas Krampe Setting Map values [root@localhost] xe vm-param-set uuid=VM uuid other-config:foo=baa Getting Map values [root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config mac_seed: 6fjafhahffhadfbadfad [root@localhost] xe vm-param-get uuid=VM-uuid param-name=other-config param-key=mac_seed 6fjafhahffhadfbadfad
  32. 32. Constraining Information 04/19/10 (c) 2009 Thomas Krampe <ul><li>Filtering </li></ul><ul><li>vm-list HVM-boot-policy=&quot;BIOS order&quot; power-state=halted </li></ul><ul><li>Parameterization </li></ul><ul><li>xe vm-list params=name-label </li></ul><ul><li>Minimal Output </li></ul><ul><li>xe vm-list --minimal </li></ul><ul><li>Parameterization and filtering can be combined </li></ul>
  33. 33. Interesting Commands 04/19/10 (c) 2009 Thomas Krampe <ul><li>Wait for a VM to run </li></ul><ul><li>xe event-wait class=vm name-label=myvm power-state=running </li></ul><ul><li>List of CPUs on a host </li></ul><ul><li>xe host-cpu-list </li></ul><ul><li>CPU Flags must match across all hosts in pool </li></ul><ul><li>List of subsystems </li></ul><ul><li>xe log-get-keys </li></ul><ul><li>Can be used to isolate subsystem logs </li></ul>
  34. 34. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe Each Entry looks like [timestamp][hostname][level] [ref-1] [(num-1) unix_rpc] text-1 text2 text3…. [timestamp] Timestamp of log entry [hostname] Host name (in a pool) [level] Debug level [ref-1] ID of the task on whose behalf this is created num-1 Thread ID, local to the current process unix_rpc Unix Domain Socket (as opposed to inet_rpc=TCP) text-1 Subsystem text2/text3 more information on action performed
  35. 35. Looking inside a log file 04/19/10 (c) 2009 Thomas Krampe xensource.log [20100209 16:58:28.688|debug|localhost|3078703|SR scanner D:dfd9ef0b708d|helpers] login done [20100209 16:58:28.921|debug|localhost|3078706 unix-RPC|SR.scan R:1ac783270499|taskhelper] task destroyed [root@localhost log]# Xenstored-access.log [20100209 16:49:05.381] D35 write data/meminfo_free 551040 [20100209 16:49:05.381] D35 write data/meminfo_total 1035760 [20100209 16:49:05.381] D35 write data/updated 1 messages Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078916|SR scanner D:dfd9ef0b708d|dispatcher] spawning a new thread to handle the current task Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078917 unix-RPC|session.slave_login D:58b671575b1e|xapi] Session.create trackid=ea252f3e59cc6f0d2d3de1345967d54b pool=true uname= is_local_superuser=true auth_user_sid= Feb 9 17:04:58 fragmgrid2001 xapi: [ info|localhost|3078919 unix-RPC|dispatch:SR.scan D:64e76e160ebc|taskhelper] task SR.scan R:98637b78c34f (uuid:6e8469dc-724d-0260-a584-44d36bca2953) created (trackid=ea252f3e59cc6f0d2d3de1345967d54b) by task D:dfd9ef0b708d
  36. 36. Unattend Installation 04/19/10 (c) 2009 Thomas Krampe <ul><li>Possible over TFTP - PXE </li></ul><ul><li>use a single xml configuration file </li></ul>http://wiki.xenmaster.de/doku.php/xenserver/xspxeunattend
  37. 37. Questions & Answers 04/19/10 (c) 2009 Thomas Krampe

×