This document provides an overview of device emulation in bhyve, including virtio devices. It discusses how devices are emulated in userspace using bhyve and implemented using virtio frameworks. Virtio provides a common way to virtualize I/O using devices like network, block storage, and entropy via transports like PCI and MMIO. An example virtio random number generator device is provided, showing how it is implemented as a basic PCI device and how it generates random numbers by requesting them from the host /dev/random driver.

1. device emulation
an introduction
leon dang
nahanni systems

2. overview
● Most emulated in userspace usr.sbin/bhyve
– kernel ones in vmm/io/ (PICs and timers)
● ISA-LPC
– uart, rtc
● PCI
– virtio
● block – storage
● net – tap networking
● rng – random entropy from /dev/random
– ahci
– pass-through

3. architecture
bhyve kernel
libvmmapi
ioctl() vm_run
guest
driver
mmio
pci
lpc
device
virtio
vm_loop
inout
VM_EXIT_INOUT VM_EXIT_INST_EMUL
io/mem r/w
VM_EXIT

4. virtio
● Virtual I/O (virtio) device spec
– common framework for IO virtualization
– Linux and FreeBSD built-in support; Windows requires
custom drivers
– transports: PCI, MMIO, channel (e.g. S/390)
– devices types: network, block, entropy, console, SCSI
– devices implement virtqueues for data transport
– http://docs.oasis-open.org/virtio/virtio/v1.0/virtio-v1.0.html

5. example virtio device
● basic pci device
– virtio vendor ID 0x1AF4, dev-id 0x1000-0x103F
● virtio random number generator
– usr.sbin/bhyve/pci_virtio_rnd.c
– guest rng driver requests 32-bit number to replenish
its random pool
– FreeBSD /dev/random non-blocking
● Yarrow PRNG
● in-progress: Fortuna

6. device (virtio) registration
● declare virtio constants and handlers
static struct virtio_consts vtrnd_vi_consts = {
"vtrnd", /* our name */
1, /* we support 1 virtqueue */
0, /* config reg size */
pci_vtrnd_reset, /* reset */
pci_vtrnd_notify, /* device-wide qnotify */
NULL, /* read virtio config */
NULL, /* write virtio config */
0, /* our capabilities */
};
● virtio:
– qnotify handler called on new message from guest
to read/write

7. device (pci) registration
struct pci_devemu pci_de_vrnd = {
.pe_emu = "virtio-rnd", /* bhyve cmd line */
.pe_init = pci_vtrnd_init, /* device init */
.pe_barwrite = vi_pci_write, /* virtio framework */
.pe_barread = vi_pci_read /* virtio framework */
};
PCI_EMUL_SET(pci_de_vrnd);
● register handlers
– use vi_pci_{read|write} for virtio

8. device initialization
● .pe_init
– init device
– virtio: link virtio constants to softc
– register PCI config space, intr, io
vi_softc_linkup(&sc->vrsc_vs, &vtrnd_vi_consts, sc, pi, &sc->vrsc_vq);
pci_set_cfgdata16(pi, PCIR_DEVICE, VIRTIO_DEV_RANDOM);
pci_set_cfgdata16(pi, PCIR_VENDOR, VIRTIO_VENDOR);
pci_set_cfgdata8(pi, PCIR_CLASS, PCIC_CRYPTO);
pci_set_cfgdata16(pi, PCIR_SUBDEV_0, VIRTIO_TYPE_ENTROPY);
if (vi_intr_init(&sc->vrsc_vs, 1, fbsdrun_virtio_msix()))
return (1);
vi_set_io_bar(&sc->vrsc_vs, 0);

9. launching bhyve
$ bhyve … -s <slot>,emulation,{conf} … vmname
● specify free slot for pci device
– bus:slot:function
– slot:function
– slot
● lpc uses device name and options, e.g.
– -l com1,stdio
● refer to bhyve(8)

10. device i/o
guest
driver
host
device
?

11. device i/o
guest
driver
device model
bhyve
vmx_run() vmx_enter_guest
VM EXIT
condition
io/mem
read/write
return from vm_run ioctl
io: handler[VM_EXITCODE_INOUT]
mem: handler[VM_EXITCODE_INST_EMUL]
vm_run ioctl
pci / virtio / lpc

12. device interrupts
device
bhyve
pci
libvmmapi
ioctl()
lpc
vm_isa_pulse_irq
vm_lapic_msi
pci_lintr_assert /
pci_generate_msi[x]
kernel
vmm
vioapic
vlapicmsi
irq
vm_run:
● calls vmx_inject_interrupts
to push pending intr's
posted interrupts:
● calls ipi_cpu to send to target
vtapic
irq
vm_ioapic_assert_irq

13. accessing guest memory
● paddr_guest2host() helper function
– access to memory mapped regions
– device can directly access returned address

14. in-progress & future
● in progress
– at-keyboard
– vga
– watchdog
– e1000
● future?
– other virtio: scsi, serial, gpu
– audio, usb, ethernets