Stub Domains
A Step Towards Dom0 Disaggregation

      Samuel Thibault, Citrix/XenSource
The Big Domain 0
    Runs a lot of Xen components

        Domain manager
    ◦
        Domain Builder
    ◦
        Devi...
What Are Stub Domains?
    Helper domains which run Xen components




    Based on Mini-OS




    Domain Builder (Dere...
What Are Stub Domains?
    Helper domains which run Xen components




    Based on Mini-OS




    Domain Builder (Dere...
POSIX Environment
on Top of Mini-OS
                                                       A p p l i c a ti o n


        ...
New Mini-OS Features
    Disk frontend


    FrameBuffer frontend


    FileSystem frontend

    ◦ Imported from JavaGu...
stubdom/
   Makefile
    ◦ Download and compile a cross-compilation
      environment
      binutils, gcc, newlib, lwip
...
Current HVM device model

         qem u


         L in u x
                                  H V M d o m a in
      dom ...
Current HVM dm
    Not always responsive

    ◦ Have to wait for dom0 Linux to schedule qemu
    Eats dom0 CPU time


  ...
HVM dm domain

                            qem u


                PV
     L in u x             M in i-O S
               ...
HVM dm domain
     45
     40
     35
     30
     25
                                      Dom0
     20                  ...
HVM dm domain Disk Perfs
    80
    70
    60
    50
                                      Dom0
    40                    ...
HVM dm domain Disk CPU%
          stubdom

                           DomU
                           Dom0
  Read         ...
HVM dm domain Net Perfs
e1000
    80
    70
    60
    50
                                     Dom0
    40
               ...
HVM dm domain Net CPU%
e1000 stubdom
                            DomU
    Recv                    Dom0
                   ...
HVM dm domain Net Perfs
bicore
    120

    100

     80

                                      Dom0
     60
             ...
HVM dm domain Net CPU%
bicore stubdom
                            DomU
    Recv                    Dom0
                  ...
HVM dm domain
    Almost unmodified qemu

    ◦ Disable e.g. sound support, plug Mini-OS PV drivers
    Relieves dom0


...
PyGRUB

           PyG R U B

              xend


            L in u x
                                     P V d o m a i...
PyGRUB
    Needs to be root to access guest disk

    ◦ Security issues
    Does not currently provide network boot


  ...
PV-GRUB start

        xend

                                    GRUB                lib x c
      L in u x
              ...
PV-GRUB loading

        xend                    P V k e rn e l          in itrd

                                    GRUB...
PV-GRUB loaded

        xend                    P V k e rn e l      in itrd

                                    GRUB     ...
PV-GRUB

       xend                    P V k e rn e l   in itrd


     L in u x
                              P V d o m a...
PV-kexec
              in i t r d

          P V k ern el



              boot
            kexec
             li b x c

 ...
PV-kexec
                                          s ta c k
              in it r d

          P V k ern el               ...
PV-kexec
                                          s ta c k
              in it r d
                                      ...
PV-kexec
                                          s ta c k
              in it r d
              boot                    ...
PV-kexec
                                          s ta c k
              in it r d
              boot                    ...
PV-kexec
                                          s ta c k
              in it r d
              boot                    ...
PV-kexec
                                          s ta c k
              in it r d
              boot                    ...
PV-GRUB
    Executes upstream GRUB

    ◦ Replace native drivers with Mini-OS drivers
    ◦ Add PV kexec implementation
 ...
Conclusion
    Dm domain

        Improves   security
    ◦
        Improves   accounting
    ◦
        Improves   scalab...
Future Work
    Dm domain

    ◦ Live migration, PCI PT
    ◦ IA-64 support
    ◦ Group scheduling with HVM domain
    PV...
XS Boston 2008 Stub Domains
Upcoming SlideShare
Loading in...5
×

XS Boston 2008 Stub Domains

569

Published on

Samuel Thibault: Stub Domains

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
569
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

XS Boston 2008 Stub Domains

  1. 1. Stub Domains A Step Towards Dom0 Disaggregation Samuel Thibault, Citrix/XenSource
  2. 2. The Big Domain 0 Runs a lot of Xen components  Domain manager ◦ Domain Builder ◦ Device Models ◦ PyGRUB ◦ These are currently running as root  ◦ e.g. PyGRUB to access guest's disk Security issues  Scalability issues 
  3. 3. What Are Stub Domains? Helper domains which run Xen components  Based on Mini-OS  Domain Builder (Derek Murray)  Device Model  PV-GRUB  ... 
  4. 4. What Are Stub Domains? Helper domains which run Xen components  Based on Mini-OS  Domain Builder (Derek Murray)  Device Model  PV-GRUB  ... 
  5. 5. POSIX Environment on Top of Mini-OS A p p l i c a ti o n getpid, sleep, n e w lib lw IP U n ix read, lines 1200 select, .. C o n s o le B lo c k FS FB N e tw o rk Sched MM fro n te n d fro n te n d fro n te n d fro n te n d fro n te n d M in i- O S X e n H y p e r v is o r
  6. 6. New Mini-OS Features Disk frontend  FrameBuffer frontend  FileSystem frontend  ◦ Imported from JavaGuest ◦ Remote access to some /export (e.g. of dom0) More advanced MM  ◦ Read-Only memory ◦ CoW for zeroed pages But still keep it simple  ◦ Single address space, mono-VCPU, no preemption Bugfixes! 
  7. 7. stubdom/  Makefile ◦ Download and compile a cross-compilation environment  binutils, gcc, newlib, lwip  c/ ◦ 'Hello World!' C application  caml/ ◦ 'Hello World!' Caml application  README ◦ Of course :)
  8. 8. Current HVM device model qem u L in u x H V M d o m a in dom 0 IN /O U T X e n H y p e rv is o r
  9. 9. Current HVM dm Not always responsive  ◦ Have to wait for dom0 Linux to schedule qemu Eats dom0 CPU time  Uses dom0 resources from userland  ◦ Disk, tap network ◦ Hence runs as root
  10. 10. HVM dm domain qem u PV L in u x M in i-O S H V M d o m a in dom 0 s tu b d o m IN /O U T X e n H y p e rv is o r
  11. 11. HVM dm domain 45 40 35 30 25 Dom0 20 Stubdom 15 10 5 0 Inb (Kcy) Boot time (s)
  12. 12. HVM dm domain Disk Perfs 80 70 60 50 Dom0 40 Stubdom 30 Native 20 10 0 Read (MB/s) Write (MB/s)
  13. 13. HVM dm domain Disk CPU% stubdom DomU Dom0 Read dom0 Stubdom Free stubdom DomU Dom0 Write dom0 Stubdom Free
  14. 14. HVM dm domain Net Perfs e1000 80 70 60 50 Dom0 40 Stubdom 30 20 10 0 Recv (MB/s) Send (MB/s)
  15. 15. HVM dm domain Net CPU% e1000 stubdom DomU Recv Dom0 dom0 Stubdom Free stubdom DomU Send Dom0 dom0 Stubdom Free
  16. 16. HVM dm domain Net Perfs bicore 120 100 80 Dom0 60 Stubdom 40 20 0 Recv (MB/s) Send (MB/s)
  17. 17. HVM dm domain Net CPU% bicore stubdom DomU Recv Dom0 dom0 Stubdom Free stubdom DomU Send Dom0 dom0 Stubdom Free
  18. 18. HVM dm domain Almost unmodified qemu  ◦ Disable e.g. sound support, plug Mini-OS PV drivers Relieves dom0  Provides better CPU usage accounting  ◦ Can charge HVM domain with dm domain time A lot safer  ◦ Only privilege is having the HVM dom as target ◦ Uses same resource access as PV guests More efficient  ◦ Let the hypervisor schedule it directly ◦ More lightweight OS
  19. 19. PyGRUB PyG R U B xend L in u x P V d o m a in dom 0 X e n H y p e rv is o r menu.lst vmlinuz initrd
  20. 20. PyGRUB Needs to be root to access guest disk  ◦ Security issues Does not currently provide network boot  Reimplements GRUB 
  21. 21. PV-GRUB start xend GRUB lib x c L in u x M in i-O S dom 0 X e n H y p e rv is o r menu.lst vmlinuz initrd
  22. 22. PV-GRUB loading xend P V k e rn e l in itrd GRUB lib x c L in u x M in i-O S dom 0 b lk f r o n t n e tfr o n t X e n H y p e rv is o r menu.lst vmlinuz initrd
  23. 23. PV-GRUB loaded xend P V k e rn e l in itrd GRUB lib x c L in u x Kexec! M in i-O S dom 0 X e n H y p e rv is o r
  24. 24. PV-GRUB xend P V k e rn e l in itrd L in u x P V d o m a in dom 0 X e n H y p e rv is o r
  25. 25. PV-kexec in i t r d P V k ern el boot kexec li b x c GRUB M in i- O S M in i- O S v ir t u a l m e m o r y
  26. 26. PV-kexec s ta c k in it r d P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  27. 27. PV-kexec s ta c k in it r d boot P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  28. 28. PV-kexec s ta c k in it r d boot boot P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  29. 29. PV-kexec s ta c k in it r d boot boot P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  30. 30. PV-kexec s ta c k in it r d boot boot P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  31. 31. PV-kexec s ta c k in it r d boot boot P V k ern el p g t a b le in it r d boot P V k ern el kexec 0xc0000000 lib x c GRUB M in i- O S M in i- O S T arg et P V g u est v ir t u a l m e m o r y v ir t u a l m e m o r y
  32. 32. PV-GRUB Executes upstream GRUB  ◦ Replace native drivers with Mini-OS drivers ◦ Add PV kexec implementation Just uses the target PV guest resources  Supports network  Supports graphical menu 
  33. 33. Conclusion Dm domain  Improves security ◦ Improves accounting ◦ Improves scalability ◦ Improves performances ◦ PV-GRUB  ◦ Improves security ◦ Provides network boot Mini-OS also being tested at Cisco for IOS  Available in the unstable tree 
  34. 34. Future Work Dm domain  ◦ Live migration, PCI PT ◦ IA-64 support ◦ Group scheduling with HVM domain PV-GRUB  ◦ Kexec 64bit guest from 32bit PV-GRUB ◦ PVFB shutdown/restart OCaml support  ◦ 'Hello World!' works ◦ Needs runtime rebuild to properly hook into POSIX layer
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×