Xenalyze
Finding meaning in the
        chaos
      George Dunlap
george.dunlap@eu.citrix.com
  Citrix Systems, UK Ltd
Introduction
 Modern   operating systems are
  complex
 Xentrace for gathering in-depth
  information
 Too much informa...
Talk goals
 Those for whom xenalyze is
  useful will use it
 Basic understanding of what
  xenalyze does, and what it’s
...
Outline
 Overview of Xen tracing
 When xentrace is useful
 Core functionality of xenalyze
 Xenalyze as a platform
 Ca...
Xen tracing
   Trace records
       Single 4-byte event number
       Optional TSC timestamp
       Optional trace-spe...
Xen tracing: What it’s good
for
   Key attributes
       Lots of detailed data
       Moderate cpu, disk overhead
    ...
Key trace events
 Runstate    change
   Figure out who’s running where
   Analyze how much time is spent

    blocked, ...
Xenalyze: Core functionality
 Problem:    xentrace file not in
 order
     Attempt to process records in
      order
 M...
Example output, dump
mode




         Xen Summit AMD 2010   9
Example output, Summary
mode




         Xen Summit AMD 2010   10
Example output, Interval

350


300
                                                         other
250                    ...
Example output, Scatterplot




          Xen Summit AMD 2010   12
Advanced features
 “Enumeration”  of MMIO, IO,
  addresses, and so on
 Symbol file translation
 Linear pagetable back-
...
Platform for new analysis
 Xenalyze may not be able to
  answer the questions you have
 But it’s a great platform to
  m...
Case study: WinXP and TPR




          Xen Summit AMD 2010   15
Case study: WinXP and TPR,
cont




          Xen Summit AMD 2010   16
Case study: Shadow
Performance




          Xen Summit AMD 2010   17
Case study: Shadow perf,
con’t
OS action    Sync                   Out-of-sync
Page fault   Propagate              Propaga...
Case study: Shadow perf,
con’t
OS action   Sync                   Out-of-sync
Map PTE     Emulate
Access      (TLB miss)  ...
Outline
 Overview of Xen tracing
 When xentrace is useful
 Core functionality of xenalyze
 Xenalyze as a platform
 Ca...
Talk goals
 Those for whom xenalyze is
  useful will use it
 Basic understanding of what
  xenalyze does, and what it’s
...
Questions
 Download   now:
http://xenbits.xensource.com/ext/xenalyze.hg
Xenalyze
       Finding meaning in the
               chaos
                  George Dunlap
           george.dunlap@eu.ci...
Introduction
 Modern   operating systems are
  complex
 Xentrace for gathering in-depth
  information
 Too much informa...
Talk goals
 Those for whom xenalyze is
  useful will use it
 Basic understanding of what
  xenalyze does, and what it’s
...
Outline
 Overview of Xen tracing
 When xentrace is useful
 Core functionality of xenalyze
 Xenalyze as a platform
 Ca...
Xen tracing
   Trace records
          Single 4-byte event number
          Optional TSC timestamp
          Optional ...
Xen tracing: What it’s good
for
   Key attributes
       Lots of detailed data
       Moderate cpu, disk overhead
    ...
Key trace events
 Runstate         change
      Figure out who’s running where
      Analyze how much time is spent

  ...
Xenalyze: Core functionality
 Problem:         xentrace file not in
   order
          Attempt to process records in
   ...
Example output, dump
mode




03/05/10   Xen Summit AMD 2010   9
Example output, Summary
mode




03/05/10   Xen Summit AMD 2010   10
Example output, Interval

350


300
                                                          other
250                   ...
Example output, Scatterplot




03/05/10   Xen Summit AMD 2010   12
Advanced features
 “Enumeration” of MMIO, IO,
  addresses, and so on
 Symbol file translation
 Linear pagetable back-
 ...
Platform for new analysis
 Xenalyze may not be able to
  answer the questions you have
 But it’s a great platform to
  m...
Case study: WinXP and TPR




03/05/10   Xen Summit AMD 2010   15
Case study: WinXP and TPR,
cont




03/05/10   Xen Summit AMD 2010   16
Case study: Shadow
Performance




03/05/10   Xen Summit AMD 2010   17
Case study: Shadow perf,
con’t
OS action    Sync                   Out-of-sync
Page fault   Propagate              Propaga...
Case study: Shadow perf,
con’t
OS action   Sync                   Out-of-sync
Map PTE     Emulate
Access      (TLB miss)  ...
Outline
 Overview of Xen tracing
 When xentrace is useful
 Core functionality of xenalyze
 Xenalyze as a platform
 Ca...
Talk goals
 Those for whom xenalyze is
  useful will use it
 Basic understanding of what
  xenalyze does, and what it’s
...
Questions
 Download   now:
http://xenbits.xensource.com/ext/xenalyze.hg




03/05/10         Xen Summit AMD 2010       22
Upcoming SlideShare
Loading in …5
×

Xenalyze

2,696 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Xenalyze

  1. 1. Xenalyze Finding meaning in the chaos George Dunlap george.dunlap@eu.citrix.com Citrix Systems, UK Ltd
  2. 2. Introduction  Modern operating systems are complex  Xentrace for gathering in-depth information  Too much information  Xenalyze
  3. 3. Talk goals  Those for whom xenalyze is useful will use it  Basic understanding of what xenalyze does, and what it’s useful for
  4. 4. Outline  Overview of Xen tracing  When xentrace is useful  Core functionality of xenalyze  Xenalyze as a platform  Case studies
  5. 5. Xen tracing  Trace records  Single 4-byte event number  Optional TSC timestamp  Optional trace-specific data, up to 28 bytes  Event mask to control which events are logged  Per-cpu trace buffers  Buffers read by process in dom0, copied to disk
  6. 6. Xen tracing: What it’s good for  Key attributes  Lots of detailed data  Moderate cpu, disk overhead  Not persistent on host crash  Understand both macro and micro effects  Performance analysis  Debugging  Understanding guest behavior  Comparing to other techniques  printk  Xenoprof  Xen performance counters
  7. 7. Key trace events  Runstate change  Figure out who’s running where  Analyze how much time is spent blocked, preempted, waiting after wake, &c  VMEXIT / VMENTER  How much time, and for what reason, we’re spending time in Xen
  8. 8. Xenalyze: Core functionality  Problem: xentrace file not in order  Attempt to process records in order  Mapping small to large  Aggregate information to see larger trends  Datais per-cpu, but we want per-vcpu  Track vcpus across physical cpus
  9. 9. Example output, dump mode Xen Summit AMD 2010 9
  10. 10. Example output, Summary mode Xen Summit AMD 2010 10
  11. 11. Example output, Interval 350 300 other 250 hlt intr vmcall 200 cr3 propagate 150 mmio fixup unsync 100 emulate running 50 0 1 27 53 79 105 131 157 183 209 235 261 287 Xen Summit AMD 2010 11
  12. 12. Example output, Scatterplot Xen Summit AMD 2010 12
  13. 13. Advanced features  “Enumeration” of MMIO, IO, addresses, and so on  Symbol file translation  Linear pagetable back- calculation  Wake-to-halt, by interrupt  …and many more
  14. 14. Platform for new analysis  Xenalyze may not be able to answer the questions you have  But it’s a great platform to modify, because it’s already done a lot of the hard work for you
  15. 15. Case study: WinXP and TPR Xen Summit AMD 2010 15
  16. 16. Case study: WinXP and TPR, cont Xen Summit AMD 2010 16
  17. 17. Case study: Shadow Performance Xen Summit AMD 2010 17
  18. 18. Case study: Shadow perf, con’t OS action Sync Out-of-sync Page fault Propagate Propagate Transition Emulate PTE Real PTE Emulate Access (TLB miss) Fix-up fault Xen Summit AMD 2010 18
  19. 19. Case study: Shadow perf, con’t OS action Sync Out-of-sync Map PTE Emulate Access (TLB miss) Fix-up fault Unmap PTE Emulate Xen Summit AMD 2010 19
  20. 20. Outline  Overview of Xen tracing  When xentrace is useful  Core functionality of xenalyze  Xenalyze as a platform  Case studies
  21. 21. Talk goals  Those for whom xenalyze is useful will use it  Basic understanding of what xenalyze does, and what it’s useful for
  22. 22. Questions  Download now: http://xenbits.xensource.com/ext/xenalyze.hg
  23. 23. Xenalyze Finding meaning in the chaos George Dunlap george.dunlap@eu.citrix.com Citrix Systems, UK Ltd 03/05/10 Xen Summit AMD 2010 1
  24. 24. Introduction  Modern operating systems are complex  Xentrace for gathering in-depth information  Too much information  Xenalyze 03/05/10 Xen Summit AMD 2010 2
  25. 25. Talk goals  Those for whom xenalyze is useful will use it  Basic understanding of what xenalyze does, and what it’s useful for 03/05/10 Xen Summit AMD 2010 3
  26. 26. Outline  Overview of Xen tracing  When xentrace is useful  Core functionality of xenalyze  Xenalyze as a platform  Case studies 03/05/10 Xen Summit AMD 2010 4
  27. 27. Xen tracing  Trace records  Single 4-byte event number  Optional TSC timestamp  Optional trace-specific data, up to 28 bytes  Event mask to control which events are logged  Per-cpu trace buffers  Buffers read by process in dom0, copied to disk 03/05/10 Xen Summit AMD 2010 5
  28. 28. Xen tracing: What it’s good for  Key attributes  Lots of detailed data  Moderate cpu, disk overhead  Not persistent on host crash  Understand both macro and micro effects  Performance analysis  Debugging  Understanding guest behavior  Comparing to other techniques  printk  Xenoprof 03/05/10Xen performance Summit AMD 2010  Xen counters 6
  29. 29. Key trace events  Runstate change  Figure out who’s running where  Analyze how much time is spent blocked, preempted, waiting after wake, &c  VMEXIT / VMENTER  How much time, and for what reason, we’re spending time in Xen 03/05/10 Xen Summit AMD 2010 7
  30. 30. Xenalyze: Core functionality  Problem: xentrace file not in order  Attempt to process records in order  Mapping small to large  Aggregate information to see larger trends  Data is per-cpu, but we want per-vcpu  Track vcpus across physical cpus 03/05/10 Xen Summit AMD 2010 8
  31. 31. Example output, dump mode 03/05/10 Xen Summit AMD 2010 9
  32. 32. Example output, Summary mode 03/05/10 Xen Summit AMD 2010 10
  33. 33. Example output, Interval 350 300 other 250 hlt intr vmcall 200 cr3 propagate 150 mmio fixup unsync 100 emulate running 50 0 1 27 53 79 105 131 157 183 209 235 261 287 03/05/10 Xen Summit AMD 2010 11
  34. 34. Example output, Scatterplot 03/05/10 Xen Summit AMD 2010 12
  35. 35. Advanced features  “Enumeration” of MMIO, IO, addresses, and so on  Symbol file translation  Linear pagetable back- calculation  Wake-to-halt, by interrupt  …and many more 03/05/10 Xen Summit AMD 2010 13
  36. 36. Platform for new analysis  Xenalyze may not be able to answer the questions you have  But it’s a great platform to modify, because it’s already done a lot of the hard work for you 03/05/10 Xen Summit AMD 2010 14
  37. 37. Case study: WinXP and TPR 03/05/10 Xen Summit AMD 2010 15
  38. 38. Case study: WinXP and TPR, cont 03/05/10 Xen Summit AMD 2010 16
  39. 39. Case study: Shadow Performance 03/05/10 Xen Summit AMD 2010 17
  40. 40. Case study: Shadow perf, con’t OS action Sync Out-of-sync Page fault Propagate Propagate Transition Emulate PTE Real PTE Emulate Access (TLB miss) Fix-up fault 03/05/10 Xen Summit AMD 2010 18
  41. 41. Case study: Shadow perf, con’t OS action Sync Out-of-sync Map PTE Emulate Access (TLB miss) Fix-up fault Unmap PTE Emulate 03/05/10 Xen Summit AMD 2010 19
  42. 42. Outline  Overview of Xen tracing  When xentrace is useful  Core functionality of xenalyze  Xenalyze as a platform  Case studies 03/05/10 Xen Summit AMD 2010 20
  43. 43. Talk goals  Those for whom xenalyze is useful will use it  Basic understanding of what xenalyze does, and what it’s useful for 03/05/10 Xen Summit AMD 2010 21
  44. 44. Questions  Download now: http://xenbits.xensource.com/ext/xenalyze.hg 03/05/10 Xen Summit AMD 2010 22

×