OSb: OSv on BitVisor
@ysmoo
Saturday, October 18, 14

BitVisor
App App App
BitVisor
Hardware
OS
http://www.justis.as-1.co.jp
Saturday, October 18, 14

OSv
VMM
Hardware
App
OSv
http://medical-care.feed.jp
Saturday, October 18, 14

BitVisor
Saturday, October 18, 14

BitVisor OSv
Saturday, October 18, 14

BitVisor OSv
http://www.root.ne.jp/nishide/shs/
Saturday, October 18, 14

OSv on BitVisor
BitVisor
Hardware
App
OSv
Saturday, October 18, 14

OSv on BitVisor
BitVisor
Hardware
App
OSv
OSb
Saturday, October 18, 14

OSv on BitVisor
BitVisor
Hardware
App
OSv
What the fuck
no Virtio !
OSb
Physical interface
sucks !
What the hell is
PRO/1000 !
Saturday, October 18, 14

OSv on BitVisor
BitVisor
Hardware
App
OSv
What the fuck
no Virtio !
OSb
Physical interface
sucks !
What the hell is
PRO/1000 !
You have no choice.
Virtio
Saturday, October 18, 14

~ The road to OSb ~
Saturday, October 18, 14

OSv code reading
• Boot process
• Some drivers
Saturday, October 18, 14

OSv code reading
• Boot process
• Some drivers
- MBR
- Read local disk with INT13/42.
- Load command line, boot loader, OSv kernel.
- Get memory map with INT15/E820.
- Setup segment descriptors/page tables.
- Switch to 64-bit mode.
- premain()
- main()
...
Saturday, October 18, 14

OSv code reading
• Boot process
• Some drivers
- MBR
- Read local disk with INT13/42.
- Load command line, boot loader, OSv kernel.
- Get memory map with INT15/E820.
- Setup segment descriptors/page tables.
- Switch to 64-bit mode.
- premain()
- main()
...
- Serial / VGA output.
- SATA.
- Virtio NIC/BLK/RNG/SCSI.
- ACPI.
- APIC.
...
Saturday, October 18, 14

OSv code reading
• Boot process
• Some drivers
- MBR
- Read local disk with INT13/42.
- Load command line, boot loader, OSv kernel.
- Get memory map with INT15/E820.
- Setup segment descriptors/page tables.
- Switch to 64-bit mode.
- premain()
- main()
...
- Serial / VGA output.
- SATA.
- Virtio NIC/BLK/RNG/SCSI.
- ACPI.
- APIC.
...
Straightforward...
Saturday, October 18, 14

Booting OSb
./scripts/run.py
Saturday, October 18, 14

Booting OSb
In the case of normal OSv...
./scripts/run.py
OSv
VMM
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
IPMI
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
OSv
Network Boot
Saturday, October 18, 14

Booting OSb
Hardware
./scripts/run.py
Image
Update
PXE Boot
IPMI
BitVisor
OSv
Network Boot
App
Saturday, October 18, 14

Booting OSb
BitVisor
Hardware
App
OSv
Network Boot
IPMI
./scripts/run.py
TTY
Image
Update
PXE Boot
Saturday, October 18, 14

ACPI workaround
drivers/acpi.cc:
// Copy the root table list to dynamic memory
if (!is_bitvisor()) {
status = AcpiReallocateRootTable();
if (ACPI_FAILURE(status)) {
acpi_e("AcpiReallocateRootTable failed: %sn",
AcpiFormatException(status));
return;
}
}
Saturday, October 18, 14

ACPI workaround
drivers/acpi.cc:
// Copy the root table list to dynamic memory
if (!is_bitvisor()) {
status = AcpiReallocateRootTable();
if (ACPI_FAILURE(status)) {
acpi_e("AcpiReallocateRootTable failed: %sn",
AcpiFormatException(status));
return;
}
}
Triple fault
Saturday, October 18, 14

ACPI workaround
drivers/acpi.cc:
// Copy the root table list to dynamic memory
if (!is_bitvisor()) {
status = AcpiReallocateRootTable();
if (ACPI_FAILURE(status)) {
acpi_e("AcpiReallocateRootTable failed: %sn",
AcpiFormatException(status));
return;
}
}
Triple fault
Skip in case of OSb
Saturday, October 18, 14

ACPI workaround
drivers/acpi.cc:
// Copy the root table list to dynamic memory
if (!is_bitvisor()) {
status = AcpiReallocateRootTable();
if (ACPI_FAILURE(status)) {
acpi_e("AcpiReallocateRootTable failed: %sn",
AcpiFormatException(status));
return;
}
}
Triple fault
Skip in case of OSb
Machine-specific...?
Saturday, October 18, 14

Hello world !
Saturday, October 18, 14

Performance?
20.75& 21.20&
25.00&
20.00&
15.00&
10.00&
5.00&
0.00&
OSb& OSv&
Elapsed(Time((sec)
for((12(billion(6mes)(val++;
Saturday, October 18, 14

Virtio NIC
BitVisor
Hardware
App Actually PRO/1000...
OSv
Saturday, October 18, 14

Virtio NIC
BitVisor
Hardware
App
OSv
Virtio NIC
Actually PRO/1000...
Fake Virtio NIC...
Saturday, October 18, 14

Virtio NIC
BitVisor
Hardware
App
OSv
Virtio NIC
Actually PRO/1000...
Fake Virtio NIC...
Hook PCI config access, MMIO, PIO and fake!
Saturday, October 18, 14

Faking
PCI configuration space
• Faking IDs
• Faking BARs
• Faking capabilities
Saturday, October 18, 14

Faking IDs 0x1AF4
(Virtio Device)
Saturday, October 18, 14

Faking IDs 0x1AF4
(Virtio Device)
0x1000
(Virtio NIC)
Saturday, October 18, 14

Faking IDs 0x1AF4
(Virtio Device)
0x1000
(Virtio NIC)
0 (Legacy Virtio)
Saturday, October 18, 14

Faking IDs 0x1AF4
(Virtio Device)
0x1000
(Virtio NIC)
0 (Legacy Virtio)
1 (Virtio NIC)
Saturday, October 18, 14

Faking BARs
BAR0: IO space
for Virtio
BAR1: MMIO space
for MSI-X
Saturday, October 18, 14

Faking BARs
BAR0: IO space
for Virtio
BAR1: MMIO space
for MSI-X
= PRO/1000 IO space
Saturday, October 18, 14

Faking BARs
BAR0: IO space
for Virtio
BAR1: MMIO space
for MSI-X
= PRO/1000 IO space
= PRO/1000 MMIO space
Saturday, October 18, 14

Faking capabilities
Add MSI-X
capability
Saturday, October 18, 14

Faking capabilities
Add MSI-X
capability Support
3 vectors
Saturday, October 18, 14

Faking capabilities
Add MSI-X
capability Support
3 vectors
Vector 0:
Reception
Saturday, October 18, 14

Faking capabilities
Add MSI-X
capability Support
3 vectors
Vector 0:
Reception
Vector 1:
Transmission
Saturday, October 18, 14

Faking capabilities
Add MSI-X
capability Support
3 vectors
Vector 0:
Reception
Vector 1:
Transmission
Vector 2:
Control
Saturday, October 18, 14

Virtio NIC Operations
• Virtio ring
• Packet transmission
• Packet reception
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
BAR0
(IO)
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
3 rings
available
BAR0
(IO)
Saturday, October 18, 14

Virtio ring
BAR0 Ring 0: Reception
(IO)
Controlled by
these registers
3 rings
available
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
3 rings
available
Ring 0: Reception
Ring 1: Transmission
BAR0
(IO)
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
3 rings
available
Ring 0: Reception
Ring 1: Transmission
Ring 2: Control
BAR0
(IO)
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
3 rings
available
Ring 0: Reception
Ring 1: Transmission
Ring 2: Control
BAR0
(IO)
(Memory)
Saturday, October 18, 14

Virtio ring
Controlled by
these registers
3 rings
available
Ring 0: Reception
Ring 1: Transmission
Ring 2: Control
BAR0
(IO)
(Memory)
Driver 2 Device
Device 2 Driver
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
PRO/1000
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req. Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req. Req.
Req. Req. Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Req.
Notify!
(PIO)
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Req.
Notify!
(PIO)
Req.
Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Req.
Notify!
(PIO)
Req.
Req.
Saturday, October 18, 14

Packet transmission
OSv
(Driver)
BitVisor
(Device)
Req.
PRO/1000
Req.
TX Ring
Req.
Req. Req. Req.
Req. Req. Req.
Req.
Notify!
(PIO)
Req.
Req.
Reuse
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Buf.
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Buf.
Buf.
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Buf.
Buf.
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Buf.
Buf.
Int!
Saturday, October 18, 14

Packet reception
OSv
(Driver)
BitVisor
(Device)
Buf.
PRO/1000
Buf.
RX Ring
Buf.
Buf. Buf. Buf.
Buf. Buf. Buf.
Buf.
Int!
Buf.
Buf.
Receive
Saturday, October 18, 14

Interrupt BAR1
(MMIO)
MSI-X Message Data:
Int. vector number (e.g. 0x26)
VMX event injection
for virtual interrupts
Saturday, October 18, 14

DHCP succeeded !
Saturday, October 18, 14

Summary
future work
• Summary
• Booting OSv on BitVisor
• Virtio NIC
• Future work
• Further testing
• Virtio BLK/RNG/...
Saturday, October 18, 14

Thank you !
http://www.root.ne.jp/nishide/shs/
Saturday, October 18, 14