Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
OSb: OSv on BitVisor 
(2014/11/21 BitVisor Summit 3) 
Yushi Omote 
University of Tsukuba 
Takahiro Shinagawa 
The Universi...
BitVisor 
App App App 
BitVisor 
Hardware 
OS 
http://www.justis.as-1.co.jp 
Monday, November 24, 14
OSv 
VMM 
Hardware 
App 
OSv 
http://medical-care.feed.jp 
Monday, November 24, 14
BitVisor 
Monday, November 24, 14
BitVisor OSv 
Monday, November 24, 14
BitVisor OSv 
http://www.root.ne.jp/nishide/shs/ 
Monday, November 24, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
Monday, November 24, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
OSb 
Monday, November 24, 14
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
What the fuck 
no Virtio ! 
OSb 
Physical interface 
sucks ! 
What the hell...
OSv on BitVisor 
BitVisor 
Hardware 
App 
OSv 
What the fuck 
no Virtio ! 
OSb 
Physical interface 
sucks ! 
What the hell...
~ The Road to OSb ~ 
Monday, November 24, 14
OSv Code Reading 
• Boot Process 
• Some Drivers 
- MBR 
- Read local disk with INT13/42. 
- Load command line, boot loade...
Only one thing to fix 
drivers/acpi.cc: 
// Copy the root table list to dynamic memory 
if (!is_bitvisor()) { 
status = Ac...
Hello world ! 
Monday, November 24, 14
OSb 
On BitVisor, 
Whenever updating OSv images, 
you update the entire Physical Disk. 
http://a.fsdn.com/sd/articles/14/1...
Network-boot of OSb 
./scripts/run.py 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
IPMI 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
Monday, November 24, 14
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
OSv 
Network Boot 
Monday, Novem...
Network-boot of OSb 
Hardware 
./scripts/run.py 
Image 
Update 
PXE Boot 
IPMI 
BitVisor 
OSv 
Network Boot 
App 
Monday, ...
Network-boot of OSb 
BitVisor 
Hardware 
App 
OSv 
Network Boot 
IPMI 
./scripts/run.py 
TTY 
Image 
Update 
PXE Boot 
Mon...
Performance? 
26.00 
19.50 
13.00 
6.50 
0 
OSb Linux (Host, Bare-metal) 
Elapsed Time (sec) 
for (12-billion times) val++...
Virtio NIC 
BitVisor 
Hardware 
App 
OSv 
No PRO/1000... 
Virtio NIC please... 
PRO/1000... 
Monday, November 24, 14
Virtio NIC 
BitVisor 
Hardware 
App 
OSv 
Virtio NIC 
No PRO/1000... 
Virtio NIC please... 
Implementing Virtio 
PRO/1000....
How to implement? 
BitVisor 
Hardware 
App 
OSv 
Virtio NIC 
Finally, Our 
Virtualization? 
Monday, November 24, 14
How to implement? 
BitVisor 
Hardware 
App 
OSv 
Virtio NIC 
Finally, Our 
Virtualization? 
No, 
BitVisor’s Way! 
Monday, ...
Para Pass-through 
Virtio 
BitVisor 
Hardware 
Intercept only interesting I/Os! 
Virtio Ring 
PRO/1000 Ring 
App 
OSv 
MSI...
Faking PCI IDs 0x1AF4 
(Virtio Device) 
0x1000 
(Virtio NIC) 
0 (Legacy Virtio) 
1 (Virtio NIC) 
Monday, November 24, 14
Faking PCI BARs 
BAR0: IO space 
for Virtio = PRO/1000 IO space 
Everything else 
Pass-through 
(e.g. MSI-X) 
Monday, Nove...
Virtio NIC Operations 
• Virtio Ring 
• Packet Transmission 
• Packet Reception 
Monday, November 24, 14
Virtio Ring 
Guest selects rings: 
* Transmission Ring 
* Reception Ring 
BAR0 
(IO) 
BitVisor emulates 
these registers 
...
Packet Transmission 
OSv 
(Driver) 
BitVisor 
(Virtio NIC) 
Transmission Ring 
Req. 
PRO/1000 
Req. 
Req. 
Req. Req. Req. ...
Packet Reception 
OSv 
(Driver) 
BitVisor 
(Virtio NIC) 
Reception Ring 
Req. 
PRO/1000 
Req. 
Req. 
Req. Req. Req. 
Recep...
Netperf TCP_STREAM 
1000.00 
750.00 
500.00 
250.00 
0 
941.39 
816.78 
OSb Linux (Host, Bare-metal) 
Throughput (Mbps) 
M...
VMExits are Costly... 
OSv 
(Driver) 
Notify! 
(PIO) VMExit 
BitVisor 
(Virtio NIC) 
PRO/1000 
Interrupt 
VMExit 
Guest 
m...
Exitless Virtio with 
Dedicated Core 
Req. Req. Req. Transmission 
Ring 
Req. Req. Req. Reception 
Ring 
ELVIS [Har’El et ...
Implementation 
Summary 
• Core Concealing 
• Modifying ACPI MADT Table 
• Notification PIO Pass-through 
• Pass-through t...
Netperf TCP_STREAM 
(Exitless Virtio) 
950.00 
712.50 
475.00 
237.50 
0 
921.83 941.39 
OSb Linux (Host, Bare-metal) 
Thr...
Ping 
(Exitless Virtio) 
0.16 
0.12 
0.08 
0.04 
0 
0.16 
0.12 
OSb Linux (Host, Bare-metal) 
Latency (msec) 
Monday, Nove...
Still Under 
Optimization 
• TODOs 
• Interrupt Moderation 
• Zero Copy/VMDq 
• Advanced PRO/1000 Descriptor 
• TCP Offloa...
Summary & 
Future Work 
• Summary 
• Running OSv on BitVisor 
• Virtio NIC + Optimization 
• Future work 
• Further Optimi...
Thank you ! 
http://www.root.ne.jp/nishide/shs/ 
Monday, November 24, 14
Upcoming SlideShare
Loading in …5
×

OSb: OSv on BitVisor (2)

1,129 views

Published on

A progress report of OSb at BitVisor Summit 3 (2014/11/21)

Published in: Engineering
  • Be the first to comment

OSb: OSv on BitVisor (2)

  1. 1. OSb: OSv on BitVisor (2014/11/21 BitVisor Summit 3) Yushi Omote University of Tsukuba Takahiro Shinagawa The University of Tokyo Monday, November 24, 14
  2. 2. BitVisor App App App BitVisor Hardware OS http://www.justis.as-1.co.jp Monday, November 24, 14
  3. 3. OSv VMM Hardware App OSv http://medical-care.feed.jp Monday, November 24, 14
  4. 4. BitVisor Monday, November 24, 14
  5. 5. BitVisor OSv Monday, November 24, 14
  6. 6. BitVisor OSv http://www.root.ne.jp/nishide/shs/ Monday, November 24, 14
  7. 7. OSv on BitVisor BitVisor Hardware App OSv Monday, November 24, 14
  8. 8. OSv on BitVisor BitVisor Hardware App OSv OSb Monday, November 24, 14
  9. 9. OSv on BitVisor BitVisor Hardware App OSv What the fuck no Virtio ! OSb Physical interface sucks ! What the hell is PRO/1000 ! Monday, November 24, 14
  10. 10. OSv on BitVisor BitVisor Hardware App OSv What the fuck no Virtio ! OSb Physical interface sucks ! What the hell is PRO/1000 ! You have no choice. Virtio Monday, November 24, 14
  11. 11. ~ The Road to OSb ~ Monday, November 24, 14
  12. 12. OSv Code Reading • Boot Process • Some Drivers - MBR - Read local disk with INT13/42. - Load command line, boot loader, OSv kernel. - Get memory map with INT15/E820. - Setup segment descriptors/page tables. - Switch to 64-bit mode. - premain() - main() ... - Serial / VGA output. - SATA. - Virtio NIC/BLK/RNG/SCSI. - ACPI. - APIC. ... OSv may run on physical machine. (=BitVisor) Monday, November 24, 14
  13. 13. Only one thing to fix drivers/acpi.cc: // Copy the root table list to dynamic memory if (!is_bitvisor()) { status = AcpiReallocateRootTable(); if (ACPI_FAILURE(status)) { acpi_e("AcpiReallocateRootTable failed: %sn", AcpiFormatException(status)); return; } } Triple fault Skip on BitVisor Monday, November 24, 14
  14. 14. Hello world ! Monday, November 24, 14
  15. 15. OSb On BitVisor, Whenever updating OSv images, you update the entire Physical Disk. http://a.fsdn.com/sd/articles/14/11/12/1946208-1.jpg http://www.dreamstime.com Troublesome... (especially for development) Monday, November 24, 14
  16. 16. Network-boot of OSb ./scripts/run.py Monday, November 24, 14
  17. 17. Network-boot of OSb Hardware ./scripts/run.py Monday, November 24, 14
  18. 18. Network-boot of OSb Hardware ./scripts/run.py Image Update Monday, November 24, 14
  19. 19. Network-boot of OSb Hardware ./scripts/run.py Image Update IPMI Monday, November 24, 14
  20. 20. Network-boot of OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI Monday, November 24, 14
  21. 21. Network-boot of OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor Monday, November 24, 14
  22. 22. Network-boot of OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor OSv Network Boot Monday, November 24, 14
  23. 23. Network-boot of OSb Hardware ./scripts/run.py Image Update PXE Boot IPMI BitVisor OSv Network Boot App Monday, November 24, 14
  24. 24. Network-boot of OSb BitVisor Hardware App OSv Network Boot IPMI ./scripts/run.py TTY Image Update PXE Boot Monday, November 24, 14
  25. 25. Performance? 26.00 19.50 13.00 6.50 0 OSb Linux (Host, Bare-metal) Elapsed Time (sec) for (12-billion times) val++; Monday, November 24, 14
  26. 26. Virtio NIC BitVisor Hardware App OSv No PRO/1000... Virtio NIC please... PRO/1000... Monday, November 24, 14
  27. 27. Virtio NIC BitVisor Hardware App OSv Virtio NIC No PRO/1000... Virtio NIC please... Implementing Virtio PRO/1000... Monday, November 24, 14
  28. 28. How to implement? BitVisor Hardware App OSv Virtio NIC Finally, Our Virtualization? Monday, November 24, 14
  29. 29. How to implement? BitVisor Hardware App OSv Virtio NIC Finally, Our Virtualization? No, BitVisor’s Way! Monday, November 24, 14
  30. 30. Para Pass-through Virtio BitVisor Hardware Intercept only interesting I/Os! Virtio Ring PRO/1000 Ring App OSv MSI-X etc. PCI Functions Pass-through! Monday, November 24, 14
  31. 31. Faking PCI IDs 0x1AF4 (Virtio Device) 0x1000 (Virtio NIC) 0 (Legacy Virtio) 1 (Virtio NIC) Monday, November 24, 14
  32. 32. Faking PCI BARs BAR0: IO space for Virtio = PRO/1000 IO space Everything else Pass-through (e.g. MSI-X) Monday, November 24, 14
  33. 33. Virtio NIC Operations • Virtio Ring • Packet Transmission • Packet Reception Monday, November 24, 14
  34. 34. Virtio Ring Guest selects rings: * Transmission Ring * Reception Ring BAR0 (IO) BitVisor emulates these registers Memory space Virtio Ring (Available + Used) Req. Req. Req. Monday, November 24, 14
  35. 35. Packet Transmission OSv (Driver) BitVisor (Virtio NIC) Transmission Ring Req. PRO/1000 Req. Req. Req. Req. Req. Transmission Ring Notify! (PIO) Req. Req. Reuse VMExit Monday, November 24, 14
  36. 36. Packet Reception OSv (Driver) BitVisor (Virtio NIC) Reception Ring Req. PRO/1000 Req. Req. Req. Req. Req. Reception Ring Interrupt Req. Req. Reuse VMExit Monday, November 24, 14
  37. 37. Netperf TCP_STREAM 1000.00 750.00 500.00 250.00 0 941.39 816.78 OSb Linux (Host, Bare-metal) Throughput (Mbps) Monday, November 24, 14
  38. 38. VMExits are Costly... OSv (Driver) Notify! (PIO) VMExit BitVisor (Virtio NIC) PRO/1000 Interrupt VMExit Guest mode Host mode Switching Overhead Monday, November 24, 14
  39. 39. Exitless Virtio with Dedicated Core Req. Req. Req. Transmission Ring Req. Req. Req. Reception Ring ELVIS [Har’El et al. ATC’13] Req. Req. Req. Req. Req. Req. Interrupt (IPI) Always Guest Mode Always Host Mode Polling Monday, November 24, 14
  40. 40. Implementation Summary • Core Concealing • Modifying ACPI MADT Table • Notification PIO Pass-through • Pass-through to ineffective PIO • Interrupt • Get vector # from MSI-X table and Send IPI • (Or ask PRO/1000 to trigger) Monday, November 24, 14
  41. 41. Netperf TCP_STREAM (Exitless Virtio) 950.00 712.50 475.00 237.50 0 921.83 941.39 OSb Linux (Host, Bare-metal) Throughput (Mbps) Monday, November 24, 14
  42. 42. Ping (Exitless Virtio) 0.16 0.12 0.08 0.04 0 0.16 0.12 OSb Linux (Host, Bare-metal) Latency (msec) Monday, November 24, 14
  43. 43. Still Under Optimization • TODOs • Interrupt Moderation • Zero Copy/VMDq • Advanced PRO/1000 Descriptor • TCP Offloading Monday, November 24, 14
  44. 44. Summary & Future Work • Summary • Running OSv on BitVisor • Virtio NIC + Optimization • Future work • Further Optimization & Evaluation • Other Virtio Devices (BLK, RNG...) Monday, November 24, 14
  45. 45. Thank you ! http://www.root.ne.jp/nishide/shs/ Monday, November 24, 14

×