/VM

                Tsuyoshi Ozawa
                Twitter oza_x86




2009   11   9                     1
2009   11   9   2
2009   11   9   3
2009   11   9   4
2009   11   9   5
KVM


2009   11   9         6
for x86

2009   11   9             7
Linux on Real Machine   Linux on Virtual Machine




                          (=OS)
2009   11   9                                                      8
2009   11   9   9
x86

                      (Shadow Page Table)




2009   11   9                               10
CPU

                            0x1000   0x11000
                  Virtual                      Physical
                  Memory                       Memory
                            0x2000   0x12000

                              ...      ...

2009   11   9                                             11
•

                •


2009   11   9       12
•
                •


2009   11   9       13
• OS
                •      0x1000   0x11000

                •      0x2000   0x12000

                         ...      ...



2009   11   9                             14
•
                   0x1000   0x11000


            • OS   0x2000   0x12000

                     ...      ...



2009   11   9                         15
•
                                          ?

                •
                                swap in
                    Physical   swap out
                    Memory



2009   11   9                                 16
1.
                             CR3

                2.               MMU

                3.   TLB               CPU
                                             CR3
                           x86
                       CPU                   MMU
                                             TLB
2009   11   9                                      17
CR3
                •

                • Control Register   3


2009   11   9                                  18
•                      MMU


                • Memory Management Unit
2009   11   9                                19
• Translation Look-aside Buffer   TLB
                •


                • CPU      SRAM


2009   11   9                                           20
Virtual Address 0xc0003120        3



                       20bit        12bit     4.

                          CR3
                    2.
                                               MMU Physical
                                                   Memory

                          TLB        3.

                1.TLB
2009   11   9                                                 21
2009   11   9   22
VM


2009   11   9        23
2009   11   9   24
VM


2009   11   9        25
VM




                • OS
2009   11   9           26
VM




                Virtual
                Machine
                Monitor
                           VMM
                           Hardware
2009   11   9                         27
VMM

                •                 3         VMM

                • Robert P. Goldberg. ,Architectural
                  Principles for Virtual Computer Systems.




2009   11   9                                                28
Type I VMM

                •

                • Xen,VMware ESX
                                   VMM
                                   Hardware
2009   11   9                                 29
Type II VMM
                • OS


                • Qemu      Proc       VMM
                 LilyVM
                 Bochs               OS
                                   Hardware
2009   11   9                                 30
Hybrid    VMM

                • Type I + Type II
                •
                • VMware Proc VMM
                  Workstation
                 Linux KVM
                             OS          VMM
                              Hardware
2009   11   9                                  31
Hybrid    VMM


            •
                         VMM
                         OS     VMM
                          Hardware
2009   11   9                         32
VM


2009   11   9        33
2

                • Shadow Page Table ( SPT )
                • Nested Page Table (Extended Page Table)


2009   11   9                                               34
Shadow Page Table

                •     OS
                               Shadow
                      OS

                •     OS   Shadow



2009   11   9                           35
SPT
                Hybrid   VMM




                         OS      VMM
                           Hardware

2009   11   9                          36
SPT
                          OS
                          VM
       Guest side        vTLB   vCR3

            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                          37
SPT
                                create page table
                          OS                      Guest
                                                page table
                          VM
       Guest side        vTLB      vCR3

            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                                38
SPT
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3

            Host side   VMM
                                                    ...
                        Hardware
                          TLB    CR3

2009   11   9                                             39
SPT
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                       40
SPT
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM              PT
                        Hardware
                          TLB    CR3

2009   11   9                                       41
SPT
                          OS                   Guest
                                             page table
                          VM
       Guest side        vTLB   vCR3
                                create SPT
            Host side   VMM                   Shadow
                                             page table
                        Hardware
                          TLB    CR3

2009   11   9                                             42
SPT
                  OS             Guest
                               page table
                                                  Guest
                                                 Physical
                  VM                             Memory
                 vTLB   vCR3

                 VMM            Shadow
                               page table
                                            OS   vCR3
                                                  Host
                                                 Physical
                Hardware                         Memory
                  TLB    CR3

2009   11   9                                               43
SPT
                  OS             Guest
                               page table
                                                    Guest
                                                   Physical
                  VM                               Memory
                 vTLB   vCR3
                                            Hook
                 VMM            Shadow
                                   VMM
                               page table           Host
                                                   Physical
                Hardware                           Memory
                  TLB    CR3

2009   11   9                                                 44
SPT
                  OS             Guest
                               page table
                                                   Guest
                                                  SPT
                                                  Physical
                  VM                              Memory
                 vTLB   vCR3                CR3


                 VMM            Shadow
                               page table          Host
                                                  Physical
                Hardware                          Memory
                  TLB    CR3

2009   11   9                                                45
2009   11   9   46
SPT
                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                 VMM            Shadow
                               page table    Host
                                            Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          47
SPT
                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                 VMM            Shadow
                               page table    Host
                                            Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          48
SPT
                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                 VMM            Shadow
                               page table    Host
                                            Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          49
SPT

                •       OS




                • VMM        SPT




2009   11   9                      50
SPT

                • Pre-Validation
                • Virtual TLB
                • Virtual TLB + Memory Trace


2009   11   9                                  51
Pre-validation

                • CR3
                •            PT

                • SPT


2009   11   9                            52
Pre-validation
                                create page table
                          OS                      Guest
                                                page table
                          VM
       Guest side        vTLB      vCR3

            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                                53
Pre-validation
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                           OS
            Host side   VMM
                                        PT
                        Hardware
                          TLB    CR3

2009   11   9                                       54
Pre-validation
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       CR3
            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                       55
Pre-validation
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                       56
Pre-validation
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM              PT
                        Hardware
                          TLB    CR3

2009   11   9                                       57
Pre-validation
                          OS                   Guest
                                             page table
                          VM
       Guest side        vTLB   vCR3
                                create SPT
            Host side   VMM                   Shadow
                                             page table
                        Hardware
                          TLB    CR3

2009   11   9                                             58
Pre-validation
                  OS              Guest
                                page table
                                                    Guest
                                                   SPT
                                                   Physical
                  VM                               Memory
                 vTLB    vCR3                CR3


                VMM              Shadow
                                page table          Host
                                                   Physical
                Hardware                           Memory
                  TLB    CR3

2009   11   9                                                 59
SPT
                  OS             Guest
                               page table
                                                   Guest
                                                  Physical
                  VM                              Memory
                 vTLB   vCR3

                 VMM            Shadow
                               page table          Host
                                            SPT   Physical
                Hardware                          Memory
                  TLB    CR3

2009   11   9                                                60
Pre-validation
                •
                    ( CR3    )   SPT




                •

2009   11   9                          61
Virtual TLB
                • CR3
                    SPT 1

                •       TLB
                            SPT

                • TLB
                                  VMM

2009   11   9                             62
Virtual TLB
                                create page table
                          OS                      Guest
                                                page table
                          VM
       Guest side        vTLB      vCR3

            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                                63
Virtual TLB
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                           OS
            Host side   VMM
                                        PT
                        Hardware
                          TLB    CR3

2009   11   9                                       64
Virtual TLB
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       CR3
            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                       65
Virtual TLB
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM
                        Hardware
                          TLB    CR3

2009   11   9                                       66
Virtual TLB
                          OS             Guest
                                       page table
                          VM
       Guest side        vTLB   vCR3
                                       VMM
            Host side   VMM              PT
                        Hardware
                          TLB    CR3

2009   11   9                                       67
Virtual TLB
                          OS                   Guest
                                             page table
                          VM
       Guest side        vTLB   vCR3
                                create SPT
            Host side   VMM                   Shadow
                                             page table
                        Hardware              1st stage
                          TLB    CR3

2009   11   9                                             68
Virtual TLB
                          VMM
                          OS                   Guest
                                             page table
                          VM
       Guest side 1      vTLB   vCR3
                                create SPT
            Host side   VMM                   Shadow
                                             page table
                        Hardware              1st stage
                          TLB    CR3

2009   11   9                                             69
Virtual TLB
                  OS             Guest
                               page table
                                                   Guest
                                                  SPT
                                                  Physical
                  VM                              Memory
                 vTLB   vCR3                CR3


                VMM             Shadow
                               page table          Host
                                1st stage         Physical
                Hardware                          Memory
                  TLB    CR3

2009   11   9                                                70
Virtual TLB


                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                VMM             Shadow
                               page table    Host
                                1st stage   Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          71
Virtual TLB


                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                VMM             Shadow
                               page table    Host
                                1st stage   Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          72
Virtual TLB


                  OS             Guest
                               page table
                                             Guest
                                            Physical
                  VM                        Memory
                 vTLB   vCR3

                VMM             Shadow
                               page table    Host
                                1st stage   Physical
                Hardware                    Memory
                  TLB    CR3

2009   11   9                                          73
Virtual TLB


                  OS             Guest
                               page table
                                                   Guest
                                                  Physical
                  VM                              Memory
                 vTLB   vCR3
                                            VMM
                VMM             Shadow
                               page table    PT    Host
                                1st stage         Physical
                Hardware                          Memory
                  TLB    CR3

2009   11   9                                                74
Virtual TLB


                  OS             Guest
                               page table   2
                                                 Guest
                                                Physical
                  VM                            Memory
                 vTLB   vCR3

                                Shadow
                VMM            page table
                                                 Host
                               2nd stage        Physical
                Hardware                        Memory
                  TLB    CR3

2009   11   9                                              75
Virtual TLB


                         OS            Guest
                                     page table
                                                     Guest
                                                    Physical
                      VM                            Memory
                     vTLB     vCR3

                                      Shadow
                    VMM              page table
                                                     Host
                                     2nd stage      Physical
                    Hardware                        Memory
                      TLB    CR3
                (    )                        CPU
2009   11   9                                                  76
Virtual TLB

                                                    OS
                         OS            Guest
                                     page table
                                                          Guest
                                                         Physical
                      VM                                 Memory
                     vTLB     vCR3

                                      Shadow
                    VMM              page table
                                                          Host
                                     2nd stage           Physical
                    Hardware                             Memory
                      TLB    CR3
                (    )                        CPU
2009   11   9                                                       77
Virtual TLB


                         OS            Guest
                                     page table
                                                     Guest
                                                    Physical
                      VM                            Memory
                     vTLB     vCR3

                                      Shadow
                    VMM              page table
                                                     Host
                                     2nd stage      Physical
                    Hardware                        Memory
                      TLB    CR3
                (    )                        CPU
2009   11   9                                                  78
Virtual TLB

                •
                    •
                    •
                •          SPT



2009   11   9                         79
Virtual TLB

                •   TLB

                    •   CR3

                    •   invlpg

                •             OS



                •
                        PTE

2009   11   9                           80
SPT

                • TLB
                    SPT

                •         OS   PT SPT




2009   11   9                           81
Memory Trace

                •             OS   PT



                •             OS
                                   SPT
                    (Trace)



2009   11   9                            82
• x86
                 •
                   CPU

                 • Linux KVM


2009   11   9                  83
kvm-88/kernel/x86/paging_tmpl.h



2009   11   9                                     84
Nested Page Table
                (Extended Page Table)
                •

                    0x1000   0x11000   0x21000

                    0x2000   0x12000   0x42000

                      ...      ...

2009   11   9                                    85
NPT/EPT
                • AMD    CPU
                              CPU

                • Intel CPU
                              CPU(i7   )

                • VMM
                • SPT
2009   11   9                              86
• VM
                 • SPT
                   • Pre-validation
                   • Virtual TLB
                 • NPT/EPT
2009   11   9                         87

Memory Virtualization

  • 1.
    /VM Tsuyoshi Ozawa Twitter oza_x86 2009 11 9 1
  • 2.
    2009 11 9 2
  • 3.
    2009 11 9 3
  • 4.
    2009 11 9 4
  • 5.
    2009 11 9 5
  • 6.
    KVM 2009 11 9 6
  • 7.
  • 8.
    Linux on RealMachine Linux on Virtual Machine (=OS) 2009 11 9 8
  • 9.
    2009 11 9 9
  • 10.
    x86 (Shadow Page Table) 2009 11 9 10
  • 11.
    CPU 0x1000 0x11000 Virtual Physical Memory Memory 0x2000 0x12000 ... ... 2009 11 9 11
  • 12.
    • 2009 11 9 12
  • 13.
    • 2009 11 9 13
  • 14.
    • OS • 0x1000 0x11000 • 0x2000 0x12000 ... ... 2009 11 9 14
  • 15.
    0x1000 0x11000 • OS 0x2000 0x12000 ... ... 2009 11 9 15
  • 16.
    ? • swap in Physical swap out Memory 2009 11 9 16
  • 17.
    1. CR3 2. MMU 3. TLB CPU CR3 x86 CPU MMU TLB 2009 11 9 17
  • 18.
    CR3 • • Control Register 3 2009 11 9 18
  • 19.
    MMU • Memory Management Unit 2009 11 9 19
  • 20.
    • Translation Look-asideBuffer TLB • • CPU SRAM 2009 11 9 20
  • 21.
    Virtual Address 0xc0003120 3 20bit 12bit 4. CR3 2. MMU Physical Memory TLB 3. 1.TLB 2009 11 9 21
  • 22.
    2009 11 9 22
  • 23.
    VM 2009 11 9 23
  • 24.
    2009 11 9 24
  • 25.
    VM 2009 11 9 25
  • 26.
    VM • OS 2009 11 9 26
  • 27.
    VM Virtual Machine Monitor VMM Hardware 2009 11 9 27
  • 28.
    VMM • 3 VMM • Robert P. Goldberg. ,Architectural Principles for Virtual Computer Systems. 2009 11 9 28
  • 29.
    Type I VMM • • Xen,VMware ESX VMM Hardware 2009 11 9 29
  • 30.
    Type II VMM • OS • Qemu Proc VMM LilyVM Bochs OS Hardware 2009 11 9 30
  • 31.
    Hybrid VMM • Type I + Type II • • VMware Proc VMM Workstation Linux KVM OS VMM Hardware 2009 11 9 31
  • 32.
    Hybrid VMM • VMM OS VMM Hardware 2009 11 9 32
  • 33.
    VM 2009 11 9 33
  • 34.
    2 • Shadow Page Table ( SPT ) • Nested Page Table (Extended Page Table) 2009 11 9 34
  • 35.
    Shadow Page Table • OS Shadow OS • OS Shadow 2009 11 9 35
  • 36.
    SPT Hybrid VMM OS VMM Hardware 2009 11 9 36
  • 37.
    SPT OS VM Guest side vTLB vCR3 Host side VMM Hardware TLB CR3 2009 11 9 37
  • 38.
    SPT create page table OS Guest page table VM Guest side vTLB vCR3 Host side VMM Hardware TLB CR3 2009 11 9 38
  • 39.
    SPT OS Guest page table VM Guest side vTLB vCR3 Host side VMM ... Hardware TLB CR3 2009 11 9 39
  • 40.
    SPT OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM Hardware TLB CR3 2009 11 9 40
  • 41.
    SPT OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM PT Hardware TLB CR3 2009 11 9 41
  • 42.
    SPT OS Guest page table VM Guest side vTLB vCR3 create SPT Host side VMM Shadow page table Hardware TLB CR3 2009 11 9 42
  • 43.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table OS vCR3 Host Physical Hardware Memory TLB CR3 2009 11 9 43
  • 44.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 Hook VMM Shadow VMM page table Host Physical Hardware Memory TLB CR3 2009 11 9 44
  • 45.
    SPT OS Guest page table Guest SPT Physical VM Memory vTLB vCR3 CR3 VMM Shadow page table Host Physical Hardware Memory TLB CR3 2009 11 9 45
  • 46.
    2009 11 9 46
  • 47.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host Physical Hardware Memory TLB CR3 2009 11 9 47
  • 48.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host Physical Hardware Memory TLB CR3 2009 11 9 48
  • 49.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host Physical Hardware Memory TLB CR3 2009 11 9 49
  • 50.
    SPT • OS • VMM SPT 2009 11 9 50
  • 51.
    SPT • Pre-Validation • Virtual TLB • Virtual TLB + Memory Trace 2009 11 9 51
  • 52.
    Pre-validation • CR3 • PT • SPT 2009 11 9 52
  • 53.
    Pre-validation create page table OS Guest page table VM Guest side vTLB vCR3 Host side VMM Hardware TLB CR3 2009 11 9 53
  • 54.
    Pre-validation OS Guest page table VM Guest side vTLB vCR3 OS Host side VMM PT Hardware TLB CR3 2009 11 9 54
  • 55.
    Pre-validation OS Guest page table VM Guest side vTLB vCR3 CR3 Host side VMM Hardware TLB CR3 2009 11 9 55
  • 56.
    Pre-validation OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM Hardware TLB CR3 2009 11 9 56
  • 57.
    Pre-validation OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM PT Hardware TLB CR3 2009 11 9 57
  • 58.
    Pre-validation OS Guest page table VM Guest side vTLB vCR3 create SPT Host side VMM Shadow page table Hardware TLB CR3 2009 11 9 58
  • 59.
    Pre-validation OS Guest page table Guest SPT Physical VM Memory vTLB vCR3 CR3 VMM Shadow page table Host Physical Hardware Memory TLB CR3 2009 11 9 59
  • 60.
    SPT OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host SPT Physical Hardware Memory TLB CR3 2009 11 9 60
  • 61.
    Pre-validation • ( CR3 ) SPT • 2009 11 9 61
  • 62.
    Virtual TLB • CR3 SPT 1 • TLB SPT • TLB VMM 2009 11 9 62
  • 63.
    Virtual TLB create page table OS Guest page table VM Guest side vTLB vCR3 Host side VMM Hardware TLB CR3 2009 11 9 63
  • 64.
    Virtual TLB OS Guest page table VM Guest side vTLB vCR3 OS Host side VMM PT Hardware TLB CR3 2009 11 9 64
  • 65.
    Virtual TLB OS Guest page table VM Guest side vTLB vCR3 CR3 Host side VMM Hardware TLB CR3 2009 11 9 65
  • 66.
    Virtual TLB OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM Hardware TLB CR3 2009 11 9 66
  • 67.
    Virtual TLB OS Guest page table VM Guest side vTLB vCR3 VMM Host side VMM PT Hardware TLB CR3 2009 11 9 67
  • 68.
    Virtual TLB OS Guest page table VM Guest side vTLB vCR3 create SPT Host side VMM Shadow page table Hardware 1st stage TLB CR3 2009 11 9 68
  • 69.
    Virtual TLB VMM OS Guest page table VM Guest side 1 vTLB vCR3 create SPT Host side VMM Shadow page table Hardware 1st stage TLB CR3 2009 11 9 69
  • 70.
    Virtual TLB OS Guest page table Guest SPT Physical VM Memory vTLB vCR3 CR3 VMM Shadow page table Host 1st stage Physical Hardware Memory TLB CR3 2009 11 9 70
  • 71.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host 1st stage Physical Hardware Memory TLB CR3 2009 11 9 71
  • 72.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host 1st stage Physical Hardware Memory TLB CR3 2009 11 9 72
  • 73.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM Shadow page table Host 1st stage Physical Hardware Memory TLB CR3 2009 11 9 73
  • 74.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 VMM VMM Shadow page table PT Host 1st stage Physical Hardware Memory TLB CR3 2009 11 9 74
  • 75.
    Virtual TLB OS Guest page table 2 Guest Physical VM Memory vTLB vCR3 Shadow VMM page table Host 2nd stage Physical Hardware Memory TLB CR3 2009 11 9 75
  • 76.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 Shadow VMM page table Host 2nd stage Physical Hardware Memory TLB CR3 ( ) CPU 2009 11 9 76
  • 77.
    Virtual TLB OS OS Guest page table Guest Physical VM Memory vTLB vCR3 Shadow VMM page table Host 2nd stage Physical Hardware Memory TLB CR3 ( ) CPU 2009 11 9 77
  • 78.
    Virtual TLB OS Guest page table Guest Physical VM Memory vTLB vCR3 Shadow VMM page table Host 2nd stage Physical Hardware Memory TLB CR3 ( ) CPU 2009 11 9 78
  • 79.
    Virtual TLB • • • • SPT 2009 11 9 79
  • 80.
    Virtual TLB • TLB • CR3 • invlpg • OS • PTE 2009 11 9 80
  • 81.
    SPT • TLB SPT • OS PT SPT 2009 11 9 81
  • 82.
    Memory Trace • OS PT • OS SPT (Trace) 2009 11 9 82
  • 83.
    • x86 • CPU • Linux KVM 2009 11 9 83
  • 84.
  • 85.
    Nested Page Table (Extended Page Table) • 0x1000 0x11000 0x21000 0x2000 0x12000 0x42000 ... ... 2009 11 9 85
  • 86.
    NPT/EPT • AMD CPU CPU • Intel CPU CPU(i7 ) • VMM • SPT 2009 11 9 86
  • 87.
    • VM • SPT • Pre-validation • Virtual TLB • NPT/EPT 2009 11 9 87