SlideShare a Scribd company logo
SAFE HARBOR STATEMENT



  <Insert Picture Here>   Xen Debugging




                               Zhigang Wang

                          zhigang.x.wang@oracle.com

                               November 2009
Agenda
• Debugging requirements
• gdbsx - gdbserver for xen
• kdb for xen
• Xen debug keys
• Guest core dumps
• Xen crash dumps
• Xentrace
• Windows debug
• Xend debug
• Resources
• Q&A


            Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                  2
Debugging requirements

• Debugging various xen components:
   • Hypervisor
   • Dom0 kernel
   • Dom0 user space (xend)
   • DomU (pvm/hvm)
• Debugging methods:
   • Dynamic debugging
   • Static debugging (analyze debug core dumps)




             Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                   3
gdbsx - gdbserver for xen

• A simple gdbserver stub that runs on dom0 and
  connects to an unprivileged guest, PV or HVM.
• Makes hypercalls directly without going thru libxc.
• Full standard debugger support via gdb:
  register/memory read and modifications, single step,
  etc.
• Support for multiple VCPUs, and single step will
  execute only one VCPU.




            Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                  4
gdbsx - examples
# gdbsx -a 5 32 9999 <-- on domain0
$ gdb <-- on debugging host
(gdb) file vmlinux
(gdb) dir linux-src/
(gdb) target remote <domain0 ip>:9999
(gdb) info thread
[New Thread 1]
 2 Thread 1 0xc04013a7 in hypercall_page ()
 * 1 Thread 0 0xc04013a7 in hypercall_page ()
(gdb) bt
#0 0xc04013a7 in hypercall_page ()
#1 0xc0408664 in raw_safe_halt () at include/asm/mach-
  xen/asm/hypercall.h:197
#2 0xc040321a in xen_idle () at arch/i386/kernel/process-xen.c:109
#3 0xc0403339 in cpu_idle () at arch/i386/kernel/process-xen.c:161
#4 0xc06f49f5 in start_kernel () at init/main.c:618
#5 0xc040006f in startup_32 ()


               Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                     5
gdbsx - useful gdb macros
(gdb) source gdbmacros
(gdb) ps
Pointer     PID   Command
0xc0d5daa0 1        init
0xc0d5d000 2        migration/0
...
(gdb) lsmod
Pointer     Address      Name
0xe1234c00 0xe1231000 xenblk
0xe130b200 0xe12fc000 dm_raid45
0xe120d900 0xe120d000 dm_message
...
(gdb) log
<5>Linux version 2.6.18-128.0.0.0.2.el5xen
(mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat
4.1.2-44)) #1 SMP Wed Jan 21 05:49:36 EST 2009
<6>BIOS-provided physical RAM map:
<4> Xen: 0000000000000000 - 0000000020800000 (usable)
...

                 Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                            6
kdb for xen - features

• Fully implemented in the hypervisor.
• Need serial access to activate and use it.
• earlykdb: to break into kdb early during boot.
• All CPUs are paused, essentially freezing the system.
• Single step will only step that CPU while others stay in
  kdb.
• Examine domain and vcpu details in pretty print.
• Set breakpoints in the hypervisor.
• Display special registers, irq table, etc.



            Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                  7
kdb for xen - entering kdb

• Breaking into kdb: once the serial line is setup, and the
  system booted with kdb-built hypervisor, ctrl+ will
  break into it (requires make kdb=y when compile xen).
• IPI sent to all cpus to pause.
• All cpus enter kdb. One becomes main kdb cpu, while
  others enter kdb pause mode.
• Switching cpu via the cpu command causes target cpu
  to become kdb main cpu, while the leaving cpu goes
  into kdb pause mode.



             Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                   8
kdb for xen - commands overview

• h: for list of commands
• dr/mr : display modify registers
• dw/dd/mw/md : display modify words/dwords
• dwm/ddm: display words/dwords at machine addr
• dis : disassemble instructions
• f : display stack frame
• ni/ss: single step, over or into functions
• cpu all : show cpu status for all cpus
• dom/vcpu: display domain/vcpu details
• gdt/dirq: display gdt, irq table, etc...

             Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                   9
kdb for xen - examples
Type CTRL- to break into kdb
Enter kdb (cpu:0 reason:1 vcpu=0 domid:32767 eflg:0x246 irqs:1)
ffff828c8017a922: acpi_safe_halt+2          ret
[0]xkdb> cpu all
[0]ffff828c8017a922: acpi_safe_halt+2          ret
[1]ffff828c8017a922: acpi_safe_halt+2          ret
[0]xkdb> dr
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff828c8017a922>] acpi_safe_halt+0x2/0x10
(XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
(XEN) rax: 0000000000000003 rbx: 00000000008e026c rcx: 0000000000000001
(XEN) rdx: 0000000000000808 rsi: 0000000004d2b4e8 rdi: ffff83007f2c2460
(XEN) rbp: ffff83007f2c2400 rsp: ffff828c802d7ec0 r8: 00000000000002b9
(XEN) r9: 0000000000000002 r10: 0000000000000000 r11: ffff828c8031a3e0
(XEN) r12: ffff83007f2c2460 r13: 00000b2a9fba595a r14: 0000000000000000
(XEN) r15: ffff828c8024d100 cr0: 000000008005003b cr4: 00000000000026f0
(XEN) cr3: 0000000020fc0000 cr2: 00000000b7f44000
…


               Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                          10
kdb for xen - examples continued
[0]xkdb> bp schedule
BP set for domid:32767 at: 0xffff828c801178b0 schedule+0
[0]xkdb> go
Breakpoint on cpu 0 at 0xffff828c801178b0
ffff828c801178b0: schedule+0               subq $0x78, %rsp
[0]xkdb> ss
ffff828c801178b4: schedule+4               mov %rbx, 0x48(%rsp)
[0]xkdb> bc all
Deleted breakpoint [0] addr:0xffff828c801178b0 domid:32767
[0]xkdb> cpu 1
Switching to cpu:1
[1]xkdb>
[1]xkdb> f
(XEN) Xen call trace:
(XEN) [<ffff828c80117a13>] schedule+0x163/0x3a0
(XEN) [<ffff828c80118968>] do_softirq+0x58/0x80
(XEN) [<ffff828c8013fedc>] idle_loop+0x4c/0xa0
[1]xkdb> go



                 Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                       11
kdb and gdbsx - status update

• Written and maintained by Mukesh Rathor
  <mukesh.rathor@oracle.com>
• gdbsx merged to xen-unstable by changeset:
  20319:de04fe4e472c on Oct. 15 2009.
• kdb still on http://xenbits.xensource.com/ext/debuggers.hg
• Future work:
    • Hardware watchpoints.
    • Improve timer handling.
    • Some sort of exception handling for very occasional protection
      faults upon bad data.
• Tests and contributions are welcome.




               Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                       12
Xen debug keys - installed handlers
# xm debug-key h
# xm dmesg
...
(XEN) 'h' pressed -> showing installed handlers
(XEN) key '%' (ascii '25') => Trap to xendbg
(XEN) key '0' (ascii '30') => dump Dom0 registers
(XEN) key 'C' (ascii '43') => trigger a crashdump
(XEN) key 'H' (ascii '48') => dump heap info
(XEN) key 'N' (ascii '4e') => NMI statistics
(XEN) key 'Q' (ascii '51') => dump PCI devices
(XEN) key 'R' (ascii '52') => reboot machine
(XEN) key 'a' (ascii '61') => dump timer queues
(XEN) key 'c' (ascii '63') => dump cx structures
(XEN) key 'd' (ascii '64') => dump registers
(XEN) key 'e' (ascii '65') => dump evtchn info
(XEN) key 'h' (ascii '68') => show this message
(XEN) key 'i' (ascii '69') => dump interrupt bindings
(XEN) key 'm' (ascii '6d') => memory info
…

                   Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                         13
Xen debug keys - examples
# xm debug-key d
# xm dmesg
...
(XEN) 'd' pressed -> dumping registers
(XEN)
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]----
(XEN) CPU: 1
(XEN) RIP: e008:[<ffff828c8010d291>] __dump_execstate+0x1/0x60
(XEN) RFLAGS: 0000000000210246 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: 0000000000000064 rcx: 000000000200046
(XEN) rdx: 000000000000000a rsi: 000000000000000a rdi: 0000000000000000
(XEN) rbp: ffff83007f0f7f28 rsp: ffff83007f0f7d98 r8: 0000000000000001
XEN) r9: 0000000000000001 r10: 00000000fffffffc r11: ffff828c8012d020
(XEN) r12: ffff828c802e48a0 r13: ffff83007f0f7f28 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000026f0
…




               Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                          14
Guest core dumps

• Get manually: xm dump-core
• Xen support automatic core dump. On vm.cfg:
   • on_crash = ‘coredump_restart’
   • on_crash = ‘coredump_destroy’
• Guest crash dump generated cores.
• Dom0 and hypervisor crash dump generated cores.




           Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                 15
Xen crash dumps
• Using kexec and kdump ported from Linux.
• Provide a crash-dump facility for both domain 0 and the
  hypervisor.
• Kexec can reboot into xen or into a Linux kernel.
• Kdump under xen is similar to the standard Linux
  implementation.
   • Both dom0 kernel panic and hypervisor panic are
       supported.
   •   dom0 loads the secondary “crash kernel”.
   •   The secondary kernel starts if dom0 panics or xen panics.
   •   The physical memory range is reserved in the hypervisor.
   •   Range is reserved using xen command line options:
       crashkernel=Y@X

              Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                    16
Guest core dumps analyze
$ crash vmlinux core-OVM_EL5U3_X86_PVM_4GB
crash> ps
   PID    PPID CPU   TASK    ST %MEM     VSZ   RSS COMM
>     0      0  0 c06762c0 RU    0.0       0     0 [swapper]
>     0      1  1 c0d5d550 RU    0.0       0     0 [swapper]
      1      0  0 c0d5daa0 IN    0.1    2080   604 init
      2      1  0 c0d5d000 IN    0.0       0     0 [migration/0]
      3      1  0 c0d54aa0 IN    0.0       0     0 [ksoftirqd/0]
...
crash> mod
 MODULE   NAME             SIZE OBJECT FILE
e1206b80 scsi_dh          11713 (not loaded) [CONFIG_KALLSYMS]
e120a500 dm_mem_cache     10049 (not loaded) [CONFIG_KALLSYMS]
e120d900 dm_message        6977 (not loaded) [CONFIG_KALLSYMS]
...




              Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                    17
Xen crash dumps analyze
$ crash xen-syms-2.6.18-128.el5 xen-syms-2.6.18-128.el5.debug core-EL5U3
crash> doms
   DID DOMAIN ST T MAXPAGE TOTPAGE VCPU SHARED_I P2M_MFN
  32753 ff21c080 RU O 0          0  0   0   ----
  32754 ff1c8080 RU X 0          0  0   0   ----
  32767 ff214080 RU I 0         0  2   0   ----
>* 0 ffbf0080 RU 0 ffffffff 67100 2 ffbec000 3e308
crash> pcpus
   PCID PCPU CUR-VCPU TSS
 * 0 ff1d3fb4 ffbe7080 ff1fa180
    1 ff21bfb4 ffbe6080 ff1fa200
crash> vcpus
   VCID PCID VCPU ST T DOMID DOMAIN
    0 0 ffbfd080 RU I 32767 ff214080
    1 1 ff1cc080 RU I 32767 ff214080
>* 0 0 ffbe7080 RU 0 0 ffbf0080
> 1 1 ffbe6080 RU 0 0 ffbf0080



                Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                           18
Xentrace - capture trace buffer data
•   Get trace data:
     # xentrace -D -S 256 -T 1 trace.raw
•   Analyze using xentrace_format:
     # cat trace.raw | xentrace_format formats >trace.txt
     …
     CPU0 16939048793256 (+       0) do_block        [ domid = 0x00000000, edomid =
       0x00000000 ]
     CPU0 16939048795152 (+     1896) switch_infprev    [ old_domid = 0x00000000, runtime
       = 610046 ]
     CPU0 16939048795536 (+ 384) switch_infnext        [ new_domid = 0x00007fff, time =
       610046, r_time = 4294967295 ]
     CPU0 16939048795912 (+      376) running_to_blocked [ dom:vcpu = 0x00000000 ]
     CPU0 16939048796272 (+      360) runnable_to_running [ dom:vcpu = 0x7fff0000 ]
     CPU0 16939048797048 (+ 776) __enter_scheduler [ prev<domid:edomid> =
       0x00000000 : 0x00000000, next<domid:edomid> = 0x00007fff : 0x00000000 ]
     …




                  Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                                            19
Xentrace - analyze using xenalyze
$ xenalyze --cpu-hz=1.8G --dump-all /tmp/test.trace | less -B -b 4096
...
] 1.000744383 -x vmexit exit_reason EXCEPTION_NMI eip fffff80001029795
] 1.000744383 -x mmio_assist w gpa fee000b0 data 0
] 1.000745561 x- vmexit exit_reason EXCEPTION_NMI eip fffff8000102ff73
] 1.000745561 x- mmio_assist w gpa fee000b0 data 0 vlapic eoi
] 1.000745561 x- fast mmio va fffffffffffe00b0
] 1.000748415 -x vmentry
] 1.000748692 x- vmentry
] 1.000749948 -x vmexit exit_reason HLT eip fffffadffab2fb41
] 1.000749948 -x hlt [ 0 0 a0028006 10dced2d ]
] 1.000750155 x- vmexit exit_reason HLT eip fffffadffab2fb41
] 1.000750155 x- hlt [ 0 0 a0028006 10dcef1c ]
…

Xenalyze by George Dunlap: http://xenbits.xensource.com/ext/xenalyze.hg




                 Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                          20
Xentrace - guest context
# xenctx --symbol-table=System.map-2.6.18-128.0.0.0.2.el5xen --all 1
cs:eip: 0061:c04013a7 hypercall_page+0x3a7
flags: 00001246 i z p
ss:esp: 0069:c06effc0
eax: 00000000 ebx: 00000001 ecx: 00000000 edx: 00000000
...
Stack:
 c0408664 c141c6c8 ffffffff c040321a c0403339 c141c6c8 c071fc64 c0627831
 c06f49f5 00004b64 c0765800 01020800 c0dcb000 00000000 00000000 c040006f

Call Trace:
 [<c04013a7>] hypercall_page+0x3a7 <--
 [<c0408664>] raw_safe_halt+0x8c
 [<c040321a>] xen_idle+0x22
 [<c0403339>] cpu_idle+0x91
 [<c06f49f5>] start_kernel+0x37a




                Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                           21
Xentrace - hvm context
# xen-hvmctx 2
HVM save record for domain 2
Entry 0: type 1 instance 0, length 24
   Header: magic 0x54381286, version 1
         Xen changeset ffffffffffffffff
         CPUID[0][%eax] 0x000006f2
Entry 1: type 2 instance 0, length 1016
   CPU: rax 0x0000000000000000 rbx 0x0000000000000000
        rcx 0x00000000c0403bb0 rdx 0x00000000c06f0000
        rbp 0x00000000c0627743 rsi 0x00000000c072dee4
        rdi 0x00000000c0627765 rsp 0x00000000c06f0fd8
         r8 0x0000000000000000          r9 0x0000000000000000
        r10 0x0000000000000000 r11 0x0000000000000000
        r12 0x0000000000000000 r13 0x0000000000000000
        r14 0x0000000000000000 r15 0x0000000000000000
        rip 0x00000000c0403be1 rflags 0x0000000000000246
…



                Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                      22
Windows debug

• Dynamic debug with WinDbg
   • Enable debug mode on Windows guest.
   • In vm.cfg: serial = 'tcp:<Debug Host IP>:<Port>‘
   • On debug host: start HW VSP and transfer IP package to
     virtual COM port. Then start WinDbg to open virtual COM
     port to control.
• Analyze windows core dumps
   • Tool to convert ELF core dump to WinDbg recognized
     Windows kernel memory dump.


HW VSP: http://www.hw-group.com/products/hw_vsp/index_en.html



              Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                    23
Xend debug

• Xend is written in Python and can be debugged by
  pdb.
• All user space debugging techniques are applied.
• Xend support tracing.
• Refer to:
  http://wiki.xensource.com/xenwiki/XenDebugging.




            Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                  24
Conclusion

• Xen has full-featured debug support for every
  component.
• Leverage existing techniques as much as possible.
• Documentations need improvement.




            Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                  25
Resources
•   Xen project: http://xen.org/
•   Xen mailing list: http://lists.xensource.com/
•   Xen Wiki: http://wiki.xensource.com/xenwiki/
•   Xen unstable:: http://xenbits.xensource.com/xen-unstable.hg
•   Xen Debuggers: http://xenbits.xensource.com/ext/debuggers.hg
•   Xen Serial Console: http://os-drive.com/wiki/XenSerialConsole
•   Xen Debugging: http://os-drive.com/wiki/XenDebugging
•   Debugging Xen and Xen guests:
    http://www.xen.org/files/xensummitboston08/Mukesh%20xendbg-present.pdf
•   Xen port of kexec/kdump: http://www.xen.org/files/summit_3/kexec_kdump.pdf
•   Crash: http://people.redhat.com/anderson/
•   GDB: http://www.gnu.org/software/gdb/
•   Oracle VM: http://oracle.com/virtualization
•   Oracle Enterprise Linux: http://www.oracle.com/linux


                     Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                                                 26
Q
 &A
Q U E S T I O N S
 A N S W E R S



 Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang
                                                       27

More Related Content

What's hot

Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
Stefano Stabellini
 
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) MechanismRedesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
The Linux Foundation
 
QEMU - Binary Translation
QEMU - Binary Translation QEMU - Binary Translation
QEMU - Binary Translation
Jiann-Fuh Liaw
 
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, CitrixXPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
The Linux Foundation
 
Linux dma engine
Linux dma engineLinux dma engine
Linux dma engine
pradeep_tewani
 
Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
Stefano Stabellini
 
HKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
HKG15-505: Power Management interactions with OP-TEE and Trusted FirmwareHKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
HKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
Linaro
 
Securing your cloud with Xen's advanced security features
Securing your cloud with Xen's advanced security featuresSecuring your cloud with Xen's advanced security features
Securing your cloud with Xen's advanced security features
The Linux Foundation
 
Reconnaissance of Virtio: What’s new and how it’s all connected?
Reconnaissance of Virtio: What’s new and how it’s all connected?Reconnaissance of Virtio: What’s new and how it’s all connected?
Reconnaissance of Virtio: What’s new and how it’s all connected?
Samsung Open Source Group
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
Emertxe Information Technologies Pvt Ltd
 
Read-only rootfs: theory and practice
Read-only rootfs: theory and practiceRead-only rootfs: theory and practice
Read-only rootfs: theory and practice
Chris Simmonds
 
Improve your storage with bcachefs
Improve your storage with bcachefsImprove your storage with bcachefs
Improve your storage with bcachefs
Marian Marinov
 
Linux DMA Engine
Linux DMA EngineLinux DMA Engine
Static partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-VStatic partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-V
RISC-V International
 
Hardware accelerated Virtualization in the ARM Cortex™ Processors
Hardware accelerated Virtualization in the ARM Cortex™ ProcessorsHardware accelerated Virtualization in the ARM Cortex™ Processors
Hardware accelerated Virtualization in the ARM Cortex™ Processors
The Linux Foundation
 
Booting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot imagesBooting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot images
Chris Simmonds
 
Design and Concepts of Android Graphics
Design and Concepts of Android GraphicsDesign and Concepts of Android Graphics
Design and Concepts of Android Graphics
National Cheng Kung University
 
from Binary to Binary: How Qemu Works
from Binary to Binary: How Qemu Worksfrom Binary to Binary: How Qemu Works
from Binary to Binary: How Qemu Works
Zhen Wei
 
Linux kernel
Linux kernelLinux kernel
Quick and Easy Device Drivers for Embedded Linux Using UIO
Quick and Easy Device Drivers for Embedded Linux Using UIOQuick and Easy Device Drivers for Embedded Linux Using UIO
Quick and Easy Device Drivers for Embedded Linux Using UIO
Chris Simmonds
 

What's hot (20)

Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
Static Partitioning with Xen, LinuxRT, and Zephyr: A Concrete End-to-end Exam...
 
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) MechanismRedesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
 
QEMU - Binary Translation
QEMU - Binary Translation QEMU - Binary Translation
QEMU - Binary Translation
 
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, CitrixXPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
XPDDS17: PVH Dom0: The Road so Far - Roger Pau Monné, Citrix
 
Linux dma engine
Linux dma engineLinux dma engine
Linux dma engine
 
Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)Xen and the art of embedded virtualization (ELC 2017)
Xen and the art of embedded virtualization (ELC 2017)
 
HKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
HKG15-505: Power Management interactions with OP-TEE and Trusted FirmwareHKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
HKG15-505: Power Management interactions with OP-TEE and Trusted Firmware
 
Securing your cloud with Xen's advanced security features
Securing your cloud with Xen's advanced security featuresSecuring your cloud with Xen's advanced security features
Securing your cloud with Xen's advanced security features
 
Reconnaissance of Virtio: What’s new and how it’s all connected?
Reconnaissance of Virtio: What’s new and how it’s all connected?Reconnaissance of Virtio: What’s new and how it’s all connected?
Reconnaissance of Virtio: What’s new and how it’s all connected?
 
U-Boot - An universal bootloader
U-Boot - An universal bootloader U-Boot - An universal bootloader
U-Boot - An universal bootloader
 
Read-only rootfs: theory and practice
Read-only rootfs: theory and practiceRead-only rootfs: theory and practice
Read-only rootfs: theory and practice
 
Improve your storage with bcachefs
Improve your storage with bcachefsImprove your storage with bcachefs
Improve your storage with bcachefs
 
Linux DMA Engine
Linux DMA EngineLinux DMA Engine
Linux DMA Engine
 
Static partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-VStatic partitioning virtualization on RISC-V
Static partitioning virtualization on RISC-V
 
Hardware accelerated Virtualization in the ARM Cortex™ Processors
Hardware accelerated Virtualization in the ARM Cortex™ ProcessorsHardware accelerated Virtualization in the ARM Cortex™ Processors
Hardware accelerated Virtualization in the ARM Cortex™ Processors
 
Booting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot imagesBooting Android: bootloaders, fastboot and boot images
Booting Android: bootloaders, fastboot and boot images
 
Design and Concepts of Android Graphics
Design and Concepts of Android GraphicsDesign and Concepts of Android Graphics
Design and Concepts of Android Graphics
 
from Binary to Binary: How Qemu Works
from Binary to Binary: How Qemu Worksfrom Binary to Binary: How Qemu Works
from Binary to Binary: How Qemu Works
 
Linux kernel
Linux kernelLinux kernel
Linux kernel
 
Quick and Easy Device Drivers for Embedded Linux Using UIO
Quick and Easy Device Drivers for Embedded Linux Using UIOQuick and Easy Device Drivers for Embedded Linux Using UIO
Quick and Easy Device Drivers for Embedded Linux Using UIO
 

Similar to Xen Debugging

XS Boston 2008 Debugging Xen
XS Boston 2008 Debugging XenXS Boston 2008 Debugging Xen
XS Boston 2008 Debugging Xen
The Linux Foundation
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
Linaro
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness
Peter Griffin
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
The Linux Foundation
 
Qemu device prototyping
Qemu device prototypingQemu device prototyping
Qemu device prototyping
Yan Vugenfirer
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
libfetion
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2
Aero Plane
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
Anne Nicolas
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
Adrian Huang
 
Dx11 performancereloaded
Dx11 performancereloadedDx11 performancereloaded
Dx11 performancereloaded
mistercteam
 
Genode Compositions
Genode CompositionsGenode Compositions
Genode Compositions
Vasily Sartakov
 
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
Akihiro Hayashi
 
Cinder Live Migration and Replication - OpenStack Summit Austin
Cinder Live Migration and Replication - OpenStack Summit AustinCinder Live Migration and Replication - OpenStack Summit Austin
Cinder Live Migration and Replication - OpenStack Summit Austin
Ed Balduf
 
LCA13: Xen on ARM
LCA13: Xen on ARMLCA13: Xen on ARM
LCA13: Xen on ARM
Linaro
 
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixXPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
The Linux Foundation
 
ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
Peter Griffin
 
DockerCon EU '17 - Dockerizing Aurea
DockerCon EU '17 - Dockerizing AureaDockerCon EU '17 - Dockerizing Aurea
DockerCon EU '17 - Dockerizing Aurea
Łukasz Piątkowski
 
Live deployment, ci, drupal
Live deployment, ci, drupalLive deployment, ci, drupal
Live deployment, ci, drupal
Andrii Podanenko
 
Lecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports DevelopmentLecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports Development
Mohammed Farrag
 
Haproxy - zastosowania
Haproxy - zastosowaniaHaproxy - zastosowania
Haproxy - zastosowania
Łukasz Jagiełło
 

Similar to Xen Debugging (20)

XS Boston 2008 Debugging Xen
XS Boston 2008 Debugging XenXS Boston 2008 Debugging Xen
XS Boston 2008 Debugging Xen
 
LAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel AwarenessLAS16-403: GDB Linux Kernel Awareness
LAS16-403: GDB Linux Kernel Awareness
 
LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness LAS16-403 - GDB Linux Kernel Awareness
LAS16-403 - GDB Linux Kernel Awareness
 
Linaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARMLinaro connect : Introduction to Xen on ARM
Linaro connect : Introduction to Xen on ARM
 
Qemu device prototyping
Qemu device prototypingQemu device prototyping
Qemu device prototyping
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
Advanced Diagnostics 2
Advanced Diagnostics 2Advanced Diagnostics 2
Advanced Diagnostics 2
 
Kernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysisKernel Recipes 2015 - Kernel dump analysis
Kernel Recipes 2015 - Kernel dump analysis
 
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
qemu + gdb: The efficient way to understand/debug Linux kernel code/data stru...
 
Dx11 performancereloaded
Dx11 performancereloadedDx11 performancereloaded
Dx11 performancereloaded
 
Genode Compositions
Genode CompositionsGenode Compositions
Genode Compositions
 
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
Exploring Compiler Optimization Opportunities for the OpenMP 4.x Accelerator...
 
Cinder Live Migration and Replication - OpenStack Summit Austin
Cinder Live Migration and Replication - OpenStack Summit AustinCinder Live Migration and Replication - OpenStack Summit Austin
Cinder Live Migration and Replication - OpenStack Summit Austin
 
LCA13: Xen on ARM
LCA13: Xen on ARMLCA13: Xen on ARM
LCA13: Xen on ARM
 
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, CitrixXPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
XPDS14 - Scaling Xen's Aggregate Storage Performance - Felipe Franciosi, Citrix
 
ELC-E Linux Awareness
ELC-E Linux AwarenessELC-E Linux Awareness
ELC-E Linux Awareness
 
DockerCon EU '17 - Dockerizing Aurea
DockerCon EU '17 - Dockerizing AureaDockerCon EU '17 - Dockerizing Aurea
DockerCon EU '17 - Dockerizing Aurea
 
Live deployment, ci, drupal
Live deployment, ci, drupalLive deployment, ci, drupal
Live deployment, ci, drupal
 
Lecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports DevelopmentLecture 6 Kernel Debugging + Ports Development
Lecture 6 Kernel Debugging + Ports Development
 
Haproxy - zastosowania
Haproxy - zastosowaniaHaproxy - zastosowania
Haproxy - zastosowania
 

More from The Linux Foundation

ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
The Linux Foundation
 
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
The Linux Foundation
 
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
The Linux Foundation
 
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
The Linux Foundation
 
XPDDS19 Keynote: Unikraft Weather Report
XPDDS19 Keynote:  Unikraft Weather ReportXPDDS19 Keynote:  Unikraft Weather Report
XPDDS19 Keynote: Unikraft Weather Report
The Linux Foundation
 
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
The Linux Foundation
 
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, XilinxXPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
The Linux Foundation
 
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
The Linux Foundation
 
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, BitdefenderXPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
The Linux Foundation
 
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
The Linux Foundation
 
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making... OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
The Linux Foundation
 
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, CitrixXPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
The Linux Foundation
 
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltdXPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
The Linux Foundation
 
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
The Linux Foundation
 
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&DXPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
The Linux Foundation
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
The Linux Foundation
 
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
The Linux Foundation
 
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
The Linux Foundation
 
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
The Linux Foundation
 
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSEXPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
The Linux Foundation
 

More from The Linux Foundation (20)

ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
 
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
 
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
 
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
 
XPDDS19 Keynote: Unikraft Weather Report
XPDDS19 Keynote:  Unikraft Weather ReportXPDDS19 Keynote:  Unikraft Weather Report
XPDDS19 Keynote: Unikraft Weather Report
 
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
 
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, XilinxXPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
 
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
 
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, BitdefenderXPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
 
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
 
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making... OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, CitrixXPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
 
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltdXPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
 
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
 
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&DXPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
 
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
 
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
 
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
 
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSEXPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
 

Recently uploaded

[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
digitalxplive
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
Axel Rennoch
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 

Recently uploaded (20)

[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
The Rise of AI in Cybersecurity How Machine Learning Will Shape Threat Detect...
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
The importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT StandardizationThe importance of Quality Assurance for ICT Standardization
The importance of Quality Assurance for ICT Standardization
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 

Xen Debugging

  • 1. SAFE HARBOR STATEMENT <Insert Picture Here> Xen Debugging Zhigang Wang zhigang.x.wang@oracle.com November 2009
  • 2. Agenda • Debugging requirements • gdbsx - gdbserver for xen • kdb for xen • Xen debug keys • Guest core dumps • Xen crash dumps • Xentrace • Windows debug • Xend debug • Resources • Q&A Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 2
  • 3. Debugging requirements • Debugging various xen components: • Hypervisor • Dom0 kernel • Dom0 user space (xend) • DomU (pvm/hvm) • Debugging methods: • Dynamic debugging • Static debugging (analyze debug core dumps) Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 3
  • 4. gdbsx - gdbserver for xen • A simple gdbserver stub that runs on dom0 and connects to an unprivileged guest, PV or HVM. • Makes hypercalls directly without going thru libxc. • Full standard debugger support via gdb: register/memory read and modifications, single step, etc. • Support for multiple VCPUs, and single step will execute only one VCPU. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 4
  • 5. gdbsx - examples # gdbsx -a 5 32 9999 <-- on domain0 $ gdb <-- on debugging host (gdb) file vmlinux (gdb) dir linux-src/ (gdb) target remote <domain0 ip>:9999 (gdb) info thread [New Thread 1] 2 Thread 1 0xc04013a7 in hypercall_page () * 1 Thread 0 0xc04013a7 in hypercall_page () (gdb) bt #0 0xc04013a7 in hypercall_page () #1 0xc0408664 in raw_safe_halt () at include/asm/mach- xen/asm/hypercall.h:197 #2 0xc040321a in xen_idle () at arch/i386/kernel/process-xen.c:109 #3 0xc0403339 in cpu_idle () at arch/i386/kernel/process-xen.c:161 #4 0xc06f49f5 in start_kernel () at init/main.c:618 #5 0xc040006f in startup_32 () Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 5
  • 6. gdbsx - useful gdb macros (gdb) source gdbmacros (gdb) ps Pointer PID Command 0xc0d5daa0 1 init 0xc0d5d000 2 migration/0 ... (gdb) lsmod Pointer Address Name 0xe1234c00 0xe1231000 xenblk 0xe130b200 0xe12fc000 dm_raid45 0xe120d900 0xe120d000 dm_message ... (gdb) log <5>Linux version 2.6.18-128.0.0.0.2.el5xen (mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Jan 21 05:49:36 EST 2009 <6>BIOS-provided physical RAM map: <4> Xen: 0000000000000000 - 0000000020800000 (usable) ... Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 6
  • 7. kdb for xen - features • Fully implemented in the hypervisor. • Need serial access to activate and use it. • earlykdb: to break into kdb early during boot. • All CPUs are paused, essentially freezing the system. • Single step will only step that CPU while others stay in kdb. • Examine domain and vcpu details in pretty print. • Set breakpoints in the hypervisor. • Display special registers, irq table, etc. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 7
  • 8. kdb for xen - entering kdb • Breaking into kdb: once the serial line is setup, and the system booted with kdb-built hypervisor, ctrl+ will break into it (requires make kdb=y when compile xen). • IPI sent to all cpus to pause. • All cpus enter kdb. One becomes main kdb cpu, while others enter kdb pause mode. • Switching cpu via the cpu command causes target cpu to become kdb main cpu, while the leaving cpu goes into kdb pause mode. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 8
  • 9. kdb for xen - commands overview • h: for list of commands • dr/mr : display modify registers • dw/dd/mw/md : display modify words/dwords • dwm/ddm: display words/dwords at machine addr • dis : disassemble instructions • f : display stack frame • ni/ss: single step, over or into functions • cpu all : show cpu status for all cpus • dom/vcpu: display domain/vcpu details • gdt/dirq: display gdt, irq table, etc... Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 9
  • 10. kdb for xen - examples Type CTRL- to break into kdb Enter kdb (cpu:0 reason:1 vcpu=0 domid:32767 eflg:0x246 irqs:1) ffff828c8017a922: acpi_safe_halt+2 ret [0]xkdb> cpu all [0]ffff828c8017a922: acpi_safe_halt+2 ret [1]ffff828c8017a922: acpi_safe_halt+2 ret [0]xkdb> dr (XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e008:[<ffff828c8017a922>] acpi_safe_halt+0x2/0x10 (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor (XEN) rax: 0000000000000003 rbx: 00000000008e026c rcx: 0000000000000001 (XEN) rdx: 0000000000000808 rsi: 0000000004d2b4e8 rdi: ffff83007f2c2460 (XEN) rbp: ffff83007f2c2400 rsp: ffff828c802d7ec0 r8: 00000000000002b9 (XEN) r9: 0000000000000002 r10: 0000000000000000 r11: ffff828c8031a3e0 (XEN) r12: ffff83007f2c2460 r13: 00000b2a9fba595a r14: 0000000000000000 (XEN) r15: ffff828c8024d100 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000020fc0000 cr2: 00000000b7f44000 … Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 10
  • 11. kdb for xen - examples continued [0]xkdb> bp schedule BP set for domid:32767 at: 0xffff828c801178b0 schedule+0 [0]xkdb> go Breakpoint on cpu 0 at 0xffff828c801178b0 ffff828c801178b0: schedule+0 subq $0x78, %rsp [0]xkdb> ss ffff828c801178b4: schedule+4 mov %rbx, 0x48(%rsp) [0]xkdb> bc all Deleted breakpoint [0] addr:0xffff828c801178b0 domid:32767 [0]xkdb> cpu 1 Switching to cpu:1 [1]xkdb> [1]xkdb> f (XEN) Xen call trace: (XEN) [<ffff828c80117a13>] schedule+0x163/0x3a0 (XEN) [<ffff828c80118968>] do_softirq+0x58/0x80 (XEN) [<ffff828c8013fedc>] idle_loop+0x4c/0xa0 [1]xkdb> go Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 11
  • 12. kdb and gdbsx - status update • Written and maintained by Mukesh Rathor <mukesh.rathor@oracle.com> • gdbsx merged to xen-unstable by changeset: 20319:de04fe4e472c on Oct. 15 2009. • kdb still on http://xenbits.xensource.com/ext/debuggers.hg • Future work: • Hardware watchpoints. • Improve timer handling. • Some sort of exception handling for very occasional protection faults upon bad data. • Tests and contributions are welcome. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 12
  • 13. Xen debug keys - installed handlers # xm debug-key h # xm dmesg ... (XEN) 'h' pressed -> showing installed handlers (XEN) key '%' (ascii '25') => Trap to xendbg (XEN) key '0' (ascii '30') => dump Dom0 registers (XEN) key 'C' (ascii '43') => trigger a crashdump (XEN) key 'H' (ascii '48') => dump heap info (XEN) key 'N' (ascii '4e') => NMI statistics (XEN) key 'Q' (ascii '51') => dump PCI devices (XEN) key 'R' (ascii '52') => reboot machine (XEN) key 'a' (ascii '61') => dump timer queues (XEN) key 'c' (ascii '63') => dump cx structures (XEN) key 'd' (ascii '64') => dump registers (XEN) key 'e' (ascii '65') => dump evtchn info (XEN) key 'h' (ascii '68') => show this message (XEN) key 'i' (ascii '69') => dump interrupt bindings (XEN) key 'm' (ascii '6d') => memory info … Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 13
  • 14. Xen debug keys - examples # xm debug-key d # xm dmesg ... (XEN) 'd' pressed -> dumping registers (XEN) (XEN) *** Dumping CPU1 host state: *** (XEN) ----[ Xen-3.4.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e008:[<ffff828c8010d291>] __dump_execstate+0x1/0x60 (XEN) RFLAGS: 0000000000210246 CONTEXT: hypervisor (XEN) rax: 0000000000000000 rbx: 0000000000000064 rcx: 000000000200046 (XEN) rdx: 000000000000000a rsi: 000000000000000a rdi: 0000000000000000 (XEN) rbp: ffff83007f0f7f28 rsp: ffff83007f0f7d98 r8: 0000000000000001 XEN) r9: 0000000000000001 r10: 00000000fffffffc r11: ffff828c8012d020 (XEN) r12: ffff828c802e48a0 r13: ffff83007f0f7f28 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000026f0 … Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 14
  • 15. Guest core dumps • Get manually: xm dump-core • Xen support automatic core dump. On vm.cfg: • on_crash = ‘coredump_restart’ • on_crash = ‘coredump_destroy’ • Guest crash dump generated cores. • Dom0 and hypervisor crash dump generated cores. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 15
  • 16. Xen crash dumps • Using kexec and kdump ported from Linux. • Provide a crash-dump facility for both domain 0 and the hypervisor. • Kexec can reboot into xen or into a Linux kernel. • Kdump under xen is similar to the standard Linux implementation. • Both dom0 kernel panic and hypervisor panic are supported. • dom0 loads the secondary “crash kernel”. • The secondary kernel starts if dom0 panics or xen panics. • The physical memory range is reserved in the hypervisor. • Range is reserved using xen command line options: crashkernel=Y@X Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 16
  • 17. Guest core dumps analyze $ crash vmlinux core-OVM_EL5U3_X86_PVM_4GB crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 c06762c0 RU 0.0 0 0 [swapper] > 0 1 1 c0d5d550 RU 0.0 0 0 [swapper] 1 0 0 c0d5daa0 IN 0.1 2080 604 init 2 1 0 c0d5d000 IN 0.0 0 0 [migration/0] 3 1 0 c0d54aa0 IN 0.0 0 0 [ksoftirqd/0] ... crash> mod MODULE NAME SIZE OBJECT FILE e1206b80 scsi_dh 11713 (not loaded) [CONFIG_KALLSYMS] e120a500 dm_mem_cache 10049 (not loaded) [CONFIG_KALLSYMS] e120d900 dm_message 6977 (not loaded) [CONFIG_KALLSYMS] ... Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 17
  • 18. Xen crash dumps analyze $ crash xen-syms-2.6.18-128.el5 xen-syms-2.6.18-128.el5.debug core-EL5U3 crash> doms DID DOMAIN ST T MAXPAGE TOTPAGE VCPU SHARED_I P2M_MFN 32753 ff21c080 RU O 0 0 0 0 ---- 32754 ff1c8080 RU X 0 0 0 0 ---- 32767 ff214080 RU I 0 0 2 0 ---- >* 0 ffbf0080 RU 0 ffffffff 67100 2 ffbec000 3e308 crash> pcpus PCID PCPU CUR-VCPU TSS * 0 ff1d3fb4 ffbe7080 ff1fa180 1 ff21bfb4 ffbe6080 ff1fa200 crash> vcpus VCID PCID VCPU ST T DOMID DOMAIN 0 0 ffbfd080 RU I 32767 ff214080 1 1 ff1cc080 RU I 32767 ff214080 >* 0 0 ffbe7080 RU 0 0 ffbf0080 > 1 1 ffbe6080 RU 0 0 ffbf0080 Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 18
  • 19. Xentrace - capture trace buffer data • Get trace data: # xentrace -D -S 256 -T 1 trace.raw • Analyze using xentrace_format: # cat trace.raw | xentrace_format formats >trace.txt … CPU0 16939048793256 (+ 0) do_block [ domid = 0x00000000, edomid = 0x00000000 ] CPU0 16939048795152 (+ 1896) switch_infprev [ old_domid = 0x00000000, runtime = 610046 ] CPU0 16939048795536 (+ 384) switch_infnext [ new_domid = 0x00007fff, time = 610046, r_time = 4294967295 ] CPU0 16939048795912 (+ 376) running_to_blocked [ dom:vcpu = 0x00000000 ] CPU0 16939048796272 (+ 360) runnable_to_running [ dom:vcpu = 0x7fff0000 ] CPU0 16939048797048 (+ 776) __enter_scheduler [ prev<domid:edomid> = 0x00000000 : 0x00000000, next<domid:edomid> = 0x00007fff : 0x00000000 ] … Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 19
  • 20. Xentrace - analyze using xenalyze $ xenalyze --cpu-hz=1.8G --dump-all /tmp/test.trace | less -B -b 4096 ... ] 1.000744383 -x vmexit exit_reason EXCEPTION_NMI eip fffff80001029795 ] 1.000744383 -x mmio_assist w gpa fee000b0 data 0 ] 1.000745561 x- vmexit exit_reason EXCEPTION_NMI eip fffff8000102ff73 ] 1.000745561 x- mmio_assist w gpa fee000b0 data 0 vlapic eoi ] 1.000745561 x- fast mmio va fffffffffffe00b0 ] 1.000748415 -x vmentry ] 1.000748692 x- vmentry ] 1.000749948 -x vmexit exit_reason HLT eip fffffadffab2fb41 ] 1.000749948 -x hlt [ 0 0 a0028006 10dced2d ] ] 1.000750155 x- vmexit exit_reason HLT eip fffffadffab2fb41 ] 1.000750155 x- hlt [ 0 0 a0028006 10dcef1c ] … Xenalyze by George Dunlap: http://xenbits.xensource.com/ext/xenalyze.hg Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 20
  • 21. Xentrace - guest context # xenctx --symbol-table=System.map-2.6.18-128.0.0.0.2.el5xen --all 1 cs:eip: 0061:c04013a7 hypercall_page+0x3a7 flags: 00001246 i z p ss:esp: 0069:c06effc0 eax: 00000000 ebx: 00000001 ecx: 00000000 edx: 00000000 ... Stack: c0408664 c141c6c8 ffffffff c040321a c0403339 c141c6c8 c071fc64 c0627831 c06f49f5 00004b64 c0765800 01020800 c0dcb000 00000000 00000000 c040006f Call Trace: [<c04013a7>] hypercall_page+0x3a7 <-- [<c0408664>] raw_safe_halt+0x8c [<c040321a>] xen_idle+0x22 [<c0403339>] cpu_idle+0x91 [<c06f49f5>] start_kernel+0x37a Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 21
  • 22. Xentrace - hvm context # xen-hvmctx 2 HVM save record for domain 2 Entry 0: type 1 instance 0, length 24 Header: magic 0x54381286, version 1 Xen changeset ffffffffffffffff CPUID[0][%eax] 0x000006f2 Entry 1: type 2 instance 0, length 1016 CPU: rax 0x0000000000000000 rbx 0x0000000000000000 rcx 0x00000000c0403bb0 rdx 0x00000000c06f0000 rbp 0x00000000c0627743 rsi 0x00000000c072dee4 rdi 0x00000000c0627765 rsp 0x00000000c06f0fd8 r8 0x0000000000000000 r9 0x0000000000000000 r10 0x0000000000000000 r11 0x0000000000000000 r12 0x0000000000000000 r13 0x0000000000000000 r14 0x0000000000000000 r15 0x0000000000000000 rip 0x00000000c0403be1 rflags 0x0000000000000246 … Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 22
  • 23. Windows debug • Dynamic debug with WinDbg • Enable debug mode on Windows guest. • In vm.cfg: serial = 'tcp:<Debug Host IP>:<Port>‘ • On debug host: start HW VSP and transfer IP package to virtual COM port. Then start WinDbg to open virtual COM port to control. • Analyze windows core dumps • Tool to convert ELF core dump to WinDbg recognized Windows kernel memory dump. HW VSP: http://www.hw-group.com/products/hw_vsp/index_en.html Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 23
  • 24. Xend debug • Xend is written in Python and can be debugged by pdb. • All user space debugging techniques are applied. • Xend support tracing. • Refer to: http://wiki.xensource.com/xenwiki/XenDebugging. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 24
  • 25. Conclusion • Xen has full-featured debug support for every component. • Leverage existing techniques as much as possible. • Documentations need improvement. Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 25
  • 26. Resources • Xen project: http://xen.org/ • Xen mailing list: http://lists.xensource.com/ • Xen Wiki: http://wiki.xensource.com/xenwiki/ • Xen unstable:: http://xenbits.xensource.com/xen-unstable.hg • Xen Debuggers: http://xenbits.xensource.com/ext/debuggers.hg • Xen Serial Console: http://os-drive.com/wiki/XenSerialConsole • Xen Debugging: http://os-drive.com/wiki/XenDebugging • Debugging Xen and Xen guests: http://www.xen.org/files/xensummitboston08/Mukesh%20xendbg-present.pdf • Xen port of kexec/kdump: http://www.xen.org/files/summit_3/kexec_kdump.pdf • Crash: http://people.redhat.com/anderson/ • GDB: http://www.gnu.org/software/gdb/ • Oracle VM: http://oracle.com/virtualization • Oracle Enterprise Linux: http://www.oracle.com/linux Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 26
  • 27. Q &A Q U E S T I O N S A N S W E R S Xen Summit 2009 Shanghai Xen Debugging Zhigang Wang 27