Implements BIOS emulation support for BHyVe: A BSD Hypervisor
Implements BIOS emulation support for BHyVe: A BSD HypervisorAbstractCurrent BHyVe only supports FreeBSD/amd64 as a GuestOS.One of the reason why BHyVe cannot support other OSes is lack of BIOSsupport.My project is implementing BIOS emulator on BHyVe, to remove theselimitations.What is BHyVe?BHyVe is new project to implement a hypervisor witch will integrate in FreeBSD.The concept is similar to Linux KVM, it provides hypervisor driver tounmodiﬁed BSD kernel running on baremetal machine.With the driver, the kernel become a hypervisor, able to run GuestOS just likenormal process on the kernel.Both hypervisors are designed for VT-x and AMD-V, hardware assistedvirtualization, unlike Xen s paravirtualization and VMware s binary translation1.The kernel module only provides a feature to switch CPU modes between Hostmode and Guest mode2, almost all device emulation is performed in userlandprocess(Figure 1).Linux KVM uses modiﬁed QEMU3 as the userland part.It s good way to support large coverage of Guest OSes, because QEMU is highlydeveloped emulator, many people already conﬁrmed to run variety of OSes on it.KVM could support almost same features what QEMU has, and it just workedﬁne.BHyVe s approach is diﬀerent.BHyVe implements minimum set of device support which required to runFreeBSD guest, from scratch.In the result, we could have completely GPL-free, BSD licensed, well codedhypervisor, but it only supports FreeBSD/amd64 as a Guest OS.One of the reason why BHyVe cannot support other OSes is lack of BIOSsupport.1 Xen and VMware had provided hypervisors which use software technique tovirtualize non VT-x x86 CPUs. KVM and BHyVe doesn t take care such old CPUs,designed for using the feature at ﬁrst.2 In Intel VT-x, it called VMX root mode and VMX non-root mode .3 Original QEMU has full emulation of x86 CPU, but on KVM we want to use VT-xhardware assisted virtualization instead of CPU emulation.So they replace CPU emulation code to KVM driver call.
BHyVe loads and executes FreeBSD kernel directly using custom OS loader runson Host OS, instead of boot up from disk image.With this method, we need to implement OS loader for each OSes, and currentlywe don t have any loader other than FreeBSD.Also, it doesn t support some OSes which calls BIOS function while running.So I started the project to implementing BIOS emulator on BHyVe, to removethese limitations./usr/sbin/bhyveBSDkernelvmm.koioctl(VM_RUN)guestkernelVMExitVMEntryuserprogramFigure 1. BHyVe overviewBIOS on real hardwareBIOS call is implemented as software interrupt handler on real mode(Figure 2).CPU executes initialization code on BIOS ROM at the beginning of startupmachine, the code initialize real mode interrupt vector, to handle number ofsoftware interrupts which are reserved for BIOS call(Figure 3).And BIOS call is not only for legacy OSes like MS-DOS, almost all boot loadersfor mordan OSes are using BIOS call to access disks, display and keyboard.Software interrupt(INTx)CPU reads interrupt vectorExecute BIOS call handlerIOHardwareint 13hFigure 2. BIOS call mechanism on real hardware
Interrupt vectorlowmemVideo RAM, etcROM BIOShighmem0000:00000000:0400A000:0000F000:0000FFFF:0000FFFF:000F①Fetch interrupt handler address②Jump to the handler address③Handler accesses HW by IO instructionFigure 3. Memory map on real hardwareBIOS on Linux KVMOn Linux KVM, QEMU loads Real BIOS(called SeaBIOS) on guest memory area.Real BIOS means it also can install on some real motherboard, has fullfunctionality just like other proprietary BIOSes which installed on real hardware.As I mentioned SeaBIOS support real motherboard, but usually used onhypervisors and emulators.KVM version of SeaBIOS s BIOS call handler accesses hardware by IOinstruction, and that behavior is basically same as BIOS for real hardware.The diﬀerence is how the hardware access handled.On KVM, the hardware access will trapped by KVM hypervisor driver, and QEMUemulates hardware device, then KVM hypervisor driver resume a guestenvironment(Figure 4).In this implementation, KVM and QEMU does nothing about BIOS call, justimplements hardware device emulation and load real BIOS on guest memoryspace(Figure 5).
Software interrupt(INTx)CPU reads interrupt vectorExecute BIOS call handlerQEMU HWEmulationIO TrapSeaBIOS preforms IOto virtual HWQEMU emulates HW IOHyperVisorGuestint 13hFigure 4. BIOS call mechanism on KVMInterrupt vectorlowmemVideo RAM, etcSeaBIOShighmem0000:00000000:0400A000:0000F000:0000FFFF:0000FFFF:000F①Fetch interrupt handler address②Jump to the handler address③Handler accesses HW by IO instrQEMU emulates the IOFigure 5. Memory map on KVMHow can we implement BIOS on BHyVe?Port SeaBIOS on BHyVe and implement hardware emulation is an option, and it sbest way to improve compatibility of legacy code, but SeaBIOS is GPL dsoftware, it s not comfortable to bring in FreeBSD code tree.And there s no implementation non-GPL real BIOS.Instead, there s BSD licensed DOS Emulator called doscmd .It s the software to run old DOS application on FreeBSD, the mechanism isdescribed as follows:• Map page to 0x0• Load DOS program• run DOS program by entering v8086 mode
• DOS program calls BIOS call or DOS API by invoke INTx instruction• DOS Emulator traps software interrupt, emulate BIOS call or DOS API• Resuming v8086 modeSo it has BIOS emulation code which runs on FreeBSD protected modeprogram.Then, we decided to port BIOS emulation code to BHyVe and trap BIOS callwhich executed on guest environment, instead of bring in real BIOS.Trapping BIOS callVT-x has functionality to trap various event on guest mode, it can be done bychanging VT-x conﬁguration structure called VMCS.And BHyVe kernel module can notify these events by IOCTL return.So we need to trap BIOS call by changing conﬁguration on VMCS, but whatevent can be used for trap BIOS call?Looks like trapping software interrupt is the easiest way, but let s think carefullyabout protected mode.On protected mode, it uses diﬀerent interrupt vector called IDT, even userlandprogram calls INT 13H, BIOS INT 13H service won t called.Maybe we can detect mode change between real mode/protected mode, andenable/disable software interrupt trapping, but it s bit complicated.Pseudo BIOSInstead of implement complicated mode change detection, we decided to useVMCALL instruction.VMCALL instruction is the instruction which causes unconditional VMExit4.And we made really simple pseudo BIOS for it.Every software interrupt handler for BIOS call is like this:VMCALLIRETBHyVe make interrupt vector when initialize guest memory area for BIOS callhandler, set pointer of the BIOS call handler described above.In this way, it doesn t take care about mode changes anymore.ConclusionWe decided to choose diﬀerent approach than Linux KVM, to getting BIOSsupport on BHyVe.Linux KVM uses real BIOS and perform hardware emulation, BHyVe emulatesBIOS call it self(Figure 6).4 Trap by hypervisor