This document presents an overview of QEMU-assisted jails for FreeBSD ports and packages, detailing both kernel and userland components essential for building and managing jails. It discusses the use of binmiscctl for configuring execution settings with QEMU, along with guidance on utilizing poudriere for creating and managing jails. The document also outlines future development plans, including support for additional architectures and enhancements to documentation and debugging capabilities.

1. QEMU Assisted Jails
for Ports and
Packages
Sean Bruno, sbruno@freebsd.org

2. Overview and Presentation Goals
● Kernel Components (11-Current)
○ Image Activators
● Userland Components (11-Current)
○ binmiscctl(8)
○ qemu-static-user
○ poudriere-devel
● Nullfs and Chroot
○ Treat the environments as normal jails.

3. Contributors and Generally Awesome
People
Stacey Son - binmiscctl(8) and imgact_binmisc(4) author
Juergen Lock - QEMU maintainer for BSD, many, many patches
Ed Maste - QEMU patches and cat herder
Peter Wemm - Sigtramp patch, because wow.
Alexander Kabaev - QEMU patches
Adrian Chadd - Ignored my pleading for help with kern_imgact.c
Baptiste Daroussin - Inflicting me with a ports commit bit

4. Contributors and Generally Awesome
People
Dimitry Andric - CLANG Help and Updates
Andrew Turner - GCC and Ports Patches
Mikael Urankar - Mysql Patches
Bryan Drewery - Taking me on a journey through poudriere
Warner Losh - Created the native-xtools target (doubled speed)
Ian Lapore - ARMv6 Assembly Help
Brooks Davis - Inspiration and initial guidance

5. Kernel Components (11-Current)
imgact_binmisc.ko
● Kernel Hook to redirect execution
● Examines ELF Header
● If match on preconfigured setting, modifes
ARGV[0] with emulator helper

6. Kernel Components - Continued
binmiscctl(8) will load and configure settings, e.
g.
binmiscctl add armv6 --interpreter "/usr/local/bin/qemu-arm" --magic
"x7fx45x4cx46x01x01x01x00x00x00x00x00x00x00x00x00x02x00x28x00"
--mask "xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxfexffxffxff"
--size 20 --set-enabled
binmiscctl add mips --interpreter "/usr/local/bin/qemu-mips" --magic
"x7fx45x4cx46x01x02x01x00x00x00x00x00x00x00x00x00x00x02x00x08"
--mask "xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxffxfexffxff"
--size 20 --set-enabled

7. Kernel Components - Continued
binmisctl list
name: mips
interpreter: /usr/local/bin/qemu-mips
flags: ENABLED USE_MASK
magic size: 20
magic offset: 0
magic: 0x7f 0x45 0x4c 0x46 0x01 0x02 0x01 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x02 0x00 0x08
mask: 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0xff 0xff 0xff 0xff
0xff 0xff 0xff 0xff 0xff 0xfe 0xff 0xff

8. Userland Components (11-Current)
QEMU
● Use emulators/qemu-static-user
● Installs statically linked binaries
● Emulation is a journey not a destination
● IOCTL emulation is still in progress

9. Userland Components - Continued
Poudriere-devel
● Easiest way to get started building jails
● Knows how to to understand binmiscctl(8)
● Knows to copy QEMU into jails
● Creates clean backup, in case of accident
● Use ZFS, save yourself some pain

10. Build the jail -- poudriere-devel
● poudriere jail -c -j 11armv632 -m svn -v head -a arm.
armv6 -x
● poudriere jail -c -j 11mips32 -m svn -v head -a mips.
mips -x
● poudirere jail -c -j 11mips64 -m svn -v head -a mips.
mips64 -x
● poudriere ports -c -m svn (so you have something to
build)

11. Configure the jail and mount
poudriere will create:
zroot/poudriere/jails/11armv632 112176079 1096098 111079980 1%
/usr/local/poudriere/jails/11armv632
zroot/poudriere/ports/default 118616467 7536487 111079980 6% /usr/local/poudriere/ports/default
Manually mount devfs/ports:
mount -t devfs devfs <path to jail/dev
mount -t nullfs /usr/local/poudriere/ports/default <path to jail>/usr/ports

12. Chroot and enjoy
root@dirty.ysv:/home/sbruno/fbsd_head # uname -a
FreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11
02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY amd64
root@dirty.ysv:~ # chroot /usr/local/poudriere/jails/11armv632
root@dirty:/ # uname -a
FreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11
02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY arm

13. Poudriere Demo
● poudriere bulk -a -j 11armv632
● nginx rendering of status
● sharp edges remain, signal handling

14. Building and testing - DEMO
● chroot and build, its magic
● needs better documentation
● man page updates
● binmiscctl(8) syntax is precise and you will
always copy/paste it from somewhere

15. Future Development
● CLANG for MIPS will make more ports build
● AARCH64 (ARM64) support will start soon,
or send me patches for QEMU now.
● IOCTL debug handling, human readable
format

16. Questions?
There’s plenty of things that I’ve probably
skipped, what do you want to do with this
feature?