Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
OSb: OSv on BitVisor 
@ysmoo 
Saturday, October 18, 14
BitVisor 
App App App 
BitVisor 
Hardware 
OS 
http://www.justis.as-1.co.jp 
Saturday, October 18, 14
OSv 
VMM 
Hardware 
App 
OSv 
http://medical-care.feed.jp 
Saturday, October 18, 14
BitVisor 
Saturday, October 18, 14
BitVisor OSv 
Saturday, October 18, 14
BitVisor OSv 
http://www.root.ne.jp/nishide/shs/ 
Saturday, October 18, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
Saturday, October 18, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
OSb 
Saturday, October 18, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
What the fuck 
no Virtio ! 
OSb 
Physical interface 
sucks ! 
What the hell...
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
What the fuck 
no Virtio ! 
OSb 
Physical interface 
sucks ! 
What the hell...
~ The road to OSb ~ 
Saturday, October 18, 14
OSv code reading 
• Boot process 
• Some drivers 
Saturday, October 18, 14
OSv code reading 
• Boot process 
• Some drivers 
- MBR 
- Read local disk with INT13/42. 
- Load command line, boot loade...
OSv code reading 
• Boot process 
• Some drivers 
- MBR 
- Read local disk with INT13/42. 
- Load command line, boot loade...
OSv code reading 
• Boot process 
• Some drivers 
- MBR 
- Read local disk with INT13/42. 
- Load command line, boot loade...
Booting OSb 
./scripts/run.py 
Saturday, October 18, 14
Booting OSb 
In the case of normal OSv... 
./scripts/run.py 
OSv 
VMM 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
IPMI 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
Saturday, October 18, 14
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
OSv 
Network Boot 
Saturday, October 18,...
Booting OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
OSv 
Network Boot 
App 
Saturday, Octobe...
Booting OSb 
BitVisor 
Hardware 
App 
OSv 
Network Boot 
IPMI 
./scripts/run.py 
TTY 
Image 
Update 
PXE Boot 
Saturday, O...
ACPI workaround 
drivers/acpi.cc: 
// Copy the root table list to dynamic memory 
if (!is_bitvisor()) { 
status = AcpiReal...
ACPI workaround 
drivers/acpi.cc: 
// Copy the root table list to dynamic memory 
if (!is_bitvisor()) { 
status = AcpiReal...
ACPI workaround 
drivers/acpi.cc: 
// Copy the root table list to dynamic memory 
if (!is_bitvisor()) { 
status = AcpiReal...
ACPI workaround 
drivers/acpi.cc: 
// Copy the root table list to dynamic memory 
if (!is_bitvisor()) { 
status = AcpiReal...
Hello world ! 
Saturday, October 18, 14
Performance? 
20.75& 21.20& 
25.00& 
20.00& 
15.00& 
10.00& 
5.00& 
0.00& 
OSb& OSv& 
Elapsed(Time((sec) 
for((12(billion(...
Virtio NIC 
BitVisor 
Hardware 
App Actually PRO/1000... 
OSv 
Saturday, October 18, 14
Virtio NIC 
BitVisor 
Hardware 
App 
OSv 
Virtio NIC 
Actually PRO/1000... 
Fake Virtio NIC... 
Saturday, October 18, 14
Virtio NIC 
BitVisor 
Hardware 
App 
OSv 
Virtio NIC 
Actually PRO/1000... 
Fake Virtio NIC... 
Hook PCI config access, MM...
Faking 
PCI configuration space 
• Faking IDs 
• Faking BARs 
• Faking capabilities 
Saturday, October 18, 14
Faking IDs 0x1AF4 
(Virtio Device) 
Saturday, October 18, 14
Faking IDs 0x1AF4 
(Virtio Device) 
0x1000 
(Virtio NIC) 
Saturday, October 18, 14
Faking IDs 0x1AF4 
(Virtio Device) 
0x1000 
(Virtio NIC) 
0 (Legacy Virtio) 
Saturday, October 18, 14
Faking IDs 0x1AF4 
(Virtio Device) 
0x1000 
(Virtio NIC) 
0 (Legacy Virtio) 
1 (Virtio NIC) 
Saturday, October 18, 14
Faking BARs 
BAR0: IO space 
for Virtio 
BAR1: MMIO space 
for MSI-X 
Saturday, October 18, 14
Faking BARs 
BAR0: IO space 
for Virtio 
BAR1: MMIO space 
for MSI-X 
= PRO/1000 IO space 
Saturday, October 18, 14
Faking BARs 
BAR0: IO space 
for Virtio 
BAR1: MMIO space 
for MSI-X 
= PRO/1000 IO space 
= PRO/1000 MMIO space 
Saturday...
Faking capabilities 
Add MSI-X 
capability 
Saturday, October 18, 14
Faking capabilities 
Add MSI-X 
capability Support 
3 vectors 
Saturday, October 18, 14
Faking capabilities 
Add MSI-X 
capability Support 
3 vectors 
Vector 0: 
Reception 
Saturday, October 18, 14
Faking capabilities 
Add MSI-X 
capability Support 
3 vectors 
Vector 0: 
Reception 
Vector 1: 
Transmission 
Saturday, Oc...
Faking capabilities 
Add MSI-X 
capability Support 
3 vectors 
Vector 0: 
Reception 
Vector 1: 
Transmission 
Vector 2: 
C...
Virtio NIC Operations 
• Virtio ring 
• Packet transmission 
• Packet reception 
Saturday, October 18, 14
Virtio ring 
Controlled by 
these registers 
BAR0 
(IO) 
Saturday, October 18, 14
Virtio ring 
Controlled by 
these registers 
3 rings 
available 
BAR0 
(IO) 
Saturday, October 18, 14
Virtio ring 
BAR0 Ring 0: Reception 
(IO) 
Controlled by 
these registers 
3 rings 
available 
Saturday, October 18, 14
Virtio ring 
Controlled by 
these registers 
3 rings 
available 
Ring 0: Reception 
Ring 1: Transmission 
BAR0 
(IO) 
Satu...
Virtio ring 
Controlled by 
these registers 
3 rings 
available 
Ring 0: Reception 
Ring 1: Transmission 
Ring 2: Control ...
Virtio ring 
Controlled by 
these registers 
3 rings 
available 
Ring 0: Reception 
Ring 1: Transmission 
Ring 2: Control ...
Virtio ring 
Controlled by 
these registers 
3 rings 
available 
Ring 0: Reception 
Ring 1: Transmission 
Ring 2: Control ...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
PRO/1000 
Saturday, October 18, 14
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. Req. 
Saturday, October 18, 14
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. Req. 
Req. Req. Req. 
Saturday, October 18, 1...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet transmission 
OSv 
(Driver) 
BitVisor 
(Device) 
Req. 
PRO/1000 
Req. 
TX Ring 
Req. 
Req. Req. Req. 
Req. Req. Req...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Packet reception 
OSv 
(Driver) 
BitVisor 
(Device) 
Buf. 
PRO/1000 
Buf. 
RX Ring 
Buf. 
Buf. Buf. Buf. 
Buf. Buf. Buf. 
...
Interrupt BAR1 
(MMIO) 
MSI-X Message Data: 
Int. vector number (e.g. 0x26) 
VMX event injection 
for virtual interrupts 
...
DHCP succeeded ! 
Saturday, October 18, 14
Summary  
future work 
• Summary 
• Booting OSv on BitVisor 
• Virtio NIC 
• Future work 
• Further testing 
• Virtio BLK/...
Thank you ! 
http://www.root.ne.jp/nishide/shs/ 
Saturday, October 18, 14
Upcoming SlideShare
Loading in …5
×

OSb: OSv on BitVisor

1,941 views

Published on

An attempt to run OSv on BitVisor.

Published in: Engineering
  • Be the first to comment

OSb: OSv on BitVisor

  1. 1. OSb: OSv on BitVisor @ysmoo Saturday, October 18, 14
  2. 2. BitVisor App App App BitVisor Hardware OS http://www.justis.as-1.co.jp Saturday, October 18, 14
  3. 3. OSv VMM Hardware App OSv http://medical-care.feed.jp Saturday, October 18, 14
  4. 4. BitVisor Saturday, October 18, 14
  5. 5. BitVisor OSv Saturday, October 18, 14
  6. 6. BitVisor OSv http://www.root.ne.jp/nishide/shs/ Saturday, October 18, 14
  7. 7. OSv on BitVisor BitVisor Hardware App OSv Saturday, October 18, 14
  8. 8. OSv on BitVisor BitVisor Hardware App OSv OSb Saturday, October 18, 14
  9. 9. OSv on BitVisor BitVisor Hardware App OSv What the fuck no Virtio ! OSb Physical interface sucks ! What the hell is PRO/1000 ! Saturday, October 18, 14
  10. 10. OSv on BitVisor BitVisor Hardware App OSv What the fuck no Virtio ! OSb Physical interface sucks ! What the hell is PRO/1000 ! You have no choice. Virtio Saturday, October 18, 14
  11. 11. ~ The road to OSb ~ Saturday, October 18, 14
  12. 12. OSv code reading • Boot process • Some drivers Saturday, October 18, 14
  13. 13. OSv code reading • Boot process • Some drivers - MBR - Read local disk with INT13/42. - Load command line, boot loader, OSv kernel. - Get memory map with INT15/E820. - Setup segment descriptors/page tables. - Switch to 64-bit mode. - premain() - main() ... Saturday, October 18, 14
  14. 14. OSv code reading • Boot process • Some drivers - MBR - Read local disk with INT13/42. - Load command line, boot loader, OSv kernel. - Get memory map with INT15/E820. - Setup segment descriptors/page tables. - Switch to 64-bit mode. - premain() - main() ... - Serial / VGA output. - SATA. - Virtio NIC/BLK/RNG/SCSI. - ACPI. - APIC. ... Saturday, October 18, 14
  15. 15. OSv code reading • Boot process • Some drivers - MBR - Read local disk with INT13/42. - Load command line, boot loader, OSv kernel. - Get memory map with INT15/E820. - Setup segment descriptors/page tables. - Switch to 64-bit mode. - premain() - main() ... - Serial / VGA output. - SATA. - Virtio NIC/BLK/RNG/SCSI. - ACPI. - APIC. ... Straightforward... Saturday, October 18, 14
  16. 16. Booting OSb ./scripts/run.py Saturday, October 18, 14
  17. 17. Booting OSb In the case of normal OSv... ./scripts/run.py OSv VMM Saturday, October 18, 14
  18. 18. Booting OSb Hardware ./scripts/run.py Saturday, October 18, 14
  19. 19. Booting OSb Hardware ./scripts/run.py Image Update Saturday, October 18, 14
  20. 20. Booting OSb Hardware ./scripts/run.py Image Update IPMI Saturday, October 18, 14
  21. 21. Booting OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI Saturday, October 18, 14
  22. 22. Booting OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor Saturday, October 18, 14
  23. 23. Booting OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor OSv Network Boot Saturday, October 18, 14
  24. 24. Booting OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor OSv Network Boot App Saturday, October 18, 14
  25. 25. Booting OSb BitVisor Hardware App OSv Network Boot IPMI ./scripts/run.py TTY Image Update PXE Boot Saturday, October 18, 14
  26. 26. ACPI workaround drivers/acpi.cc: // Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %sn", AcpiFormatException(status)); return; } } Saturday, October 18, 14
  27. 27. ACPI workaround drivers/acpi.cc: // Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %sn", AcpiFormatException(status)); return; } } Triple fault Saturday, October 18, 14
  28. 28. ACPI workaround drivers/acpi.cc: // Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %sn", AcpiFormatException(status)); return; } } Triple fault Skip in case of OSb Saturday, October 18, 14
  29. 29. ACPI workaround drivers/acpi.cc: // Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %sn", AcpiFormatException(status)); return; } } Triple fault Skip in case of OSb Machine-specific...? Saturday, October 18, 14
  30. 30. Hello world ! Saturday, October 18, 14
  31. 31. Performance? 20.75& 21.20& 25.00& 20.00& 15.00& 10.00& 5.00& 0.00& OSb& OSv& Elapsed(Time((sec) for((12(billion(6mes)(val++; Saturday, October 18, 14
  32. 32. Virtio NIC BitVisor Hardware App Actually PRO/1000... OSv Saturday, October 18, 14
  33. 33. Virtio NIC BitVisor Hardware App OSv Virtio NIC Actually PRO/1000... Fake Virtio NIC... Saturday, October 18, 14
  34. 34. Virtio NIC BitVisor Hardware App OSv Virtio NIC Actually PRO/1000... Fake Virtio NIC... Hook PCI config access, MMIO, PIO and fake! Saturday, October 18, 14
  35. 35. Faking PCI configuration space • Faking IDs • Faking BARs • Faking capabilities Saturday, October 18, 14
  36. 36. Faking IDs 0x1AF4 (Virtio Device) Saturday, October 18, 14
  37. 37. Faking IDs 0x1AF4 (Virtio Device) 0x1000 (Virtio NIC) Saturday, October 18, 14
  38. 38. Faking IDs 0x1AF4 (Virtio Device) 0x1000 (Virtio NIC) 0 (Legacy Virtio) Saturday, October 18, 14
  39. 39. Faking IDs 0x1AF4 (Virtio Device) 0x1000 (Virtio NIC) 0 (Legacy Virtio) 1 (Virtio NIC) Saturday, October 18, 14
  40. 40. Faking BARs BAR0: IO space for Virtio BAR1: MMIO space for MSI-X Saturday, October 18, 14
  41. 41. Faking BARs BAR0: IO space for Virtio BAR1: MMIO space for MSI-X = PRO/1000 IO space Saturday, October 18, 14
  42. 42. Faking BARs BAR0: IO space for Virtio BAR1: MMIO space for MSI-X = PRO/1000 IO space = PRO/1000 MMIO space Saturday, October 18, 14
  43. 43. Faking capabilities Add MSI-X capability Saturday, October 18, 14
  44. 44. Faking capabilities Add MSI-X capability Support 3 vectors Saturday, October 18, 14
  45. 45. Faking capabilities Add MSI-X capability Support 3 vectors Vector 0: Reception Saturday, October 18, 14
  46. 46. Faking capabilities Add MSI-X capability Support 3 vectors Vector 0: Reception Vector 1: Transmission Saturday, October 18, 14
  47. 47. Faking capabilities Add MSI-X capability Support 3 vectors Vector 0: Reception Vector 1: Transmission Vector 2: Control Saturday, October 18, 14
  48. 48. Virtio NIC Operations • Virtio ring • Packet transmission • Packet reception Saturday, October 18, 14
  49. 49. Virtio ring Controlled by these registers BAR0 (IO) Saturday, October 18, 14
  50. 50. Virtio ring Controlled by these registers 3 rings available BAR0 (IO) Saturday, October 18, 14
  51. 51. Virtio ring BAR0 Ring 0: Reception (IO) Controlled by these registers 3 rings available Saturday, October 18, 14
  52. 52. Virtio ring Controlled by these registers 3 rings available Ring 0: Reception Ring 1: Transmission BAR0 (IO) Saturday, October 18, 14
  53. 53. Virtio ring Controlled by these registers 3 rings available Ring 0: Reception Ring 1: Transmission Ring 2: Control BAR0 (IO) Saturday, October 18, 14
  54. 54. Virtio ring Controlled by these registers 3 rings available Ring 0: Reception Ring 1: Transmission Ring 2: Control BAR0 (IO) (Memory) Saturday, October 18, 14
  55. 55. Virtio ring Controlled by these registers 3 rings available Ring 0: Reception Ring 1: Transmission Ring 2: Control BAR0 (IO) (Memory) Driver 2 Device Device 2 Driver Saturday, October 18, 14
  56. 56. Packet transmission OSv (Driver) BitVisor (Device) PRO/1000 Saturday, October 18, 14
  57. 57. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. Req. Saturday, October 18, 14
  58. 58. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. Req. Req. Req. Req. Saturday, October 18, 14
  59. 59. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Saturday, October 18, 14
  60. 60. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Req. Saturday, October 18, 14
  61. 61. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Req. Notify! (PIO) Saturday, October 18, 14
  62. 62. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Req. Notify! (PIO) Req. Req. Saturday, October 18, 14
  63. 63. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Req. Notify! (PIO) Req. Req. Saturday, October 18, 14
  64. 64. Packet transmission OSv (Driver) BitVisor (Device) Req. PRO/1000 Req. TX Ring Req. Req. Req. Req. Req. Req. Req. Req. Notify! (PIO) Req. Req. Reuse Saturday, October 18, 14
  65. 65. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Saturday, October 18, 14
  66. 66. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Saturday, October 18, 14
  67. 67. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Saturday, October 18, 14
  68. 68. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Saturday, October 18, 14
  69. 69. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Saturday, October 18, 14
  70. 70. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Int! Saturday, October 18, 14
  71. 71. Packet reception OSv (Driver) BitVisor (Device) Buf. PRO/1000 Buf. RX Ring Buf. Buf. Buf. Buf. Buf. Buf. Buf. Buf. Int! Buf. Buf. Receive Saturday, October 18, 14
  72. 72. Interrupt BAR1 (MMIO) MSI-X Message Data: Int. vector number (e.g. 0x26) VMX event injection for virtual interrupts Saturday, October 18, 14
  73. 73. DHCP succeeded ! Saturday, October 18, 14
  74. 74. Summary future work • Summary • Booting OSv on BitVisor • Virtio NIC • Future work • Further testing • Virtio BLK/RNG/... Saturday, October 18, 14
  75. 75. Thank you ! http://www.root.ne.jp/nishide/shs/ Saturday, October 18, 14

×