The document discusses running OSv virtual machines on the BitVisor hypervisor. It describes implementing virtio network device support for OSv on BitVisor by para-virtualizing the physical network interface and emulating virtio rings and operations. This avoids expensive VM exits by using a dedicated core to handle virtio operations in host mode while the guest runs on another core. Performance tests show the exitless virtio approach achieves network throughput and latency comparable to running on bare metal. Future work includes further optimizing exitless virtio and adding support for other virtio devices like block and random number generators.

1. OSb: OSv on BitVisor
(2014/11/21 BitVisor Summit 3)
Yushi Omote
University of Tsukuba
Takahiro Shinagawa
The University of Tokyo
Monday, November 24, 14

2. BitVisor
App App App
BitVisor
Hardware
OS
http://www.justis.as-1.co.jp
Monday, November 24, 14

3. OSv
VMM
Hardware
App
OSv
http://medical-care.feed.jp
Monday, November 24, 14

4. BitVisor
Monday, November 24, 14

5. BitVisor OSv
Monday, November 24, 14

6. BitVisor OSv
http://www.root.ne.jp/nishide/shs/
Monday, November 24, 14

7. OSv on BitVisor
BitVisor
Hardware
App
OSv
Monday, November 24, 14

8. OSv on BitVisor
BitVisor
Hardware
App
OSv
OSb
Monday, November 24, 14

9. OSv on BitVisor
BitVisor
Hardware
App
OSv
What the fuck
no Virtio !
OSb
Physical interface
sucks !
What the hell is
PRO/1000 !
Monday, November 24, 14

10. OSv on BitVisor
BitVisor
Hardware
App
OSv
What the fuck
no Virtio !
OSb
Physical interface
sucks !
What the hell is
PRO/1000 !
You have no
choice.
Virtio
Monday, November 24, 14

11. ~ The Road to OSb ~
Monday, November 24, 14

12. OSv Code Reading
• Boot Process
• Some Drivers
- MBR
- Read local disk with INT13/42.
- Load command line, boot loader, OSv kernel.
- Get memory map with INT15/E820.
- Setup segment descriptors/page tables.
- Switch to 64-bit mode.
- premain()
- main()
...
- Serial / VGA output.
- SATA.
- Virtio NIC/BLK/RNG/SCSI.
- ACPI.
- APIC.
...
OSv may run on
physical machine.
(=BitVisor)
Monday, November 24, 14

13. Only one thing to fix
drivers/acpi.cc:
// Copy the root table list to dynamic memory
if (!is_bitvisor()) {
status = AcpiReallocateRootTable();
if (ACPI_FAILURE(status)) {
acpi_e("AcpiReallocateRootTable failed: %sn",
AcpiFormatException(status));
return;
}
}
Triple fault
Skip on BitVisor
Monday, November 24, 14

14. Hello world !
Monday, November 24, 14

15. OSb
On BitVisor,
Whenever updating OSv images,
you update the entire Physical Disk.
http://a.fsdn.com/sd/articles/14/11/12/1946208-1.jpg http://www.dreamstime.com
Troublesome...
(especially for development)
Monday, November 24, 14

16. Network-boot of OSb
./scripts/run.py
Monday, November 24, 14

17. Network-boot of OSb
Hardware
./scripts/run.py
Monday, November 24, 14

18. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
Monday, November 24, 14

19. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
IPMI
Monday, November 24, 14

20. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
Monday, November 24, 14

21. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
Monday, November 24, 14

22. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
OSv
Network Boot
Monday, November 24, 14

23. Network-boot of OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
OSv
Network Boot
App
Monday, November 24, 14

24. Network-boot of OSb
BitVisor
Hardware
App
OSv
Network Boot
IPMI
./scripts/run.py
TTY
Image
Update
PXE Boot
Monday, November 24, 14

25. Performance?
26.00
19.50
13.00
6.50
0
OSb Linux (Host, Bare-metal)
Elapsed Time (sec)
for (12-billion times) val++;
Monday, November 24, 14

26. Virtio NIC
BitVisor
Hardware
App
OSv
No PRO/1000...
Virtio NIC please...
PRO/1000...
Monday, November 24, 14

27. Virtio NIC
BitVisor
Hardware
App
OSv
Virtio NIC
No PRO/1000...
Virtio NIC please...
Implementing Virtio
PRO/1000...
Monday, November 24, 14

28. How to implement?
BitVisor
Hardware
App
OSv
Virtio NIC
Finally, Our
Virtualization?
Monday, November 24, 14

29. How to implement?
BitVisor
Hardware
App
OSv
Virtio NIC
Finally, Our
Virtualization?
No,
BitVisor’s Way!
Monday, November 24, 14

30. Para Pass-through
Virtio
BitVisor
Hardware
Intercept only interesting I/Os!
Virtio Ring
PRO/1000 Ring
App
OSv
MSI-X etc.
PCI Functions
Pass-through!
Monday, November 24, 14

31. Faking PCI IDs 0x1AF4
(Virtio Device)
0x1000
(Virtio NIC)
0 (Legacy Virtio)
1 (Virtio NIC)
Monday, November 24, 14

32. Faking PCI BARs
BAR0: IO space
for Virtio = PRO/1000 IO space
Everything else
Pass-through
(e.g. MSI-X)
Monday, November 24, 14

33. Virtio NIC Operations
• Virtio Ring
• Packet Transmission
• Packet Reception
Monday, November 24, 14

34. Virtio Ring
Guest selects rings:
* Transmission Ring
* Reception Ring
BAR0
(IO)
BitVisor emulates
these registers
Memory space
Virtio Ring (Available + Used)
Req. Req. Req.
Monday, November 24, 14

35. Packet Transmission
OSv
(Driver)
BitVisor
(Virtio NIC)
Transmission Ring
Req.
PRO/1000
Req.
Req.
Req. Req. Req.
Transmission Ring
Notify!
(PIO)
Req.
Req.
Reuse
VMExit
Monday, November 24, 14

36. Packet Reception
OSv
(Driver)
BitVisor
(Virtio NIC)
Reception Ring
Req.
PRO/1000
Req.
Req.
Req. Req. Req.
Reception Ring
Interrupt
Req.
Req.
Reuse
VMExit
Monday, November 24, 14

37. Netperf TCP_STREAM
1000.00
750.00
500.00
250.00
0
941.39
816.78
OSb Linux (Host, Bare-metal)
Throughput (Mbps)
Monday, November 24, 14

38. VMExits are Costly...
OSv
(Driver)
Notify!
(PIO) VMExit
BitVisor
(Virtio NIC)
PRO/1000
Interrupt
VMExit
Guest
mode
Host
mode
Switching
Overhead
Monday, November 24, 14

39. Exitless Virtio with
Dedicated Core
Req. Req. Req. Transmission
Ring
Req. Req. Req. Reception
Ring
ELVIS [Har’El et al. ATC’13]
Req. Req. Req.
Req. Req. Req.
Interrupt (IPI)
Always
Guest
Mode
Always
Host
Mode
Polling
Monday, November 24, 14

40. Implementation
Summary
• Core Concealing
• Modifying ACPI MADT Table
• Notification PIO Pass-through
• Pass-through to ineffective PIO
• Interrupt
• Get vector # from MSI-X table and Send IPI
• (Or ask PRO/1000 to trigger)
Monday, November 24, 14

41. Netperf TCP_STREAM
(Exitless Virtio)
950.00
712.50
475.00
237.50
0
921.83 941.39
OSb Linux (Host, Bare-metal)
Throughput (Mbps)
Monday, November 24, 14

42. Ping
(Exitless Virtio)
0.16
0.12
0.08
0.04
0
0.16
0.12
OSb Linux (Host, Bare-metal)
Latency (msec)
Monday, November 24, 14

43. Still Under
Optimization
• TODOs
• Interrupt Moderation
• Zero Copy/VMDq
• Advanced PRO/1000 Descriptor
• TCP Offloading
Monday, November 24, 14

44. Summary &
Future Work
• Summary
• Running OSv on BitVisor
• Virtio NIC + Optimization
• Future work
• Further Optimization & Evaluation
• Other Virtio Devices (BLK, RNG...)
Monday, November 24, 14

45. Thank you !
http://www.root.ne.jp/nishide/shs/
Monday, November 24, 14