Abstract Explanation and use of QEMU user mode on FreeBSD in tandem with binmiscctl tools to create and manipulate arbitrary hardware architecture jails on AMD64/i386. Detailed setup of tools and use for creating and maintaining ports packages, initial prototype of disk images and testing of concepts on architectures without having real hardware in play. Examples of MIPS and ARM execution for the crowd and demonstration of setup and configuration on AMD64 hardware. Display use of poudriere to build PKG style repos for these architectures. Simple how with regards to rapid prototyping of compressed flash images to assist in the propagation of FreeBSD on other platforms. Speaker bio Sean Bruno is FreeBSD src committer, ports maintainer and member of the cluster administration team. He is the lead mirror manager interacting with external organizations in the installation of new style PKG and SVN mirrors around the world. Living in the San Francisco, CA, USA area, with his family, Sean is a FreeBSD developer working on several projects for various employers. Sean enjoys spinning records and has a shoutcast show on Radio KoL twice a week.

1. QEMU Assisted Jails
for Ports and
Packages
Sean Bruno, sbruno@freebsd.org

2. Overview and Presentation Goals
● Kernel Components (11-Current)
○ Image Activators
● Userland Components (11-Current)
○ binmiscctl(8)
○ qemu-static-user
○ poudriere-devel
● Nullfs and Chroot
○ Treat the environments as normal jails.

3. Contributors and Generally Awesome
People
Stacey Son - binmiscctl(8) and imgact_binmisc(4) author
Juergen Lock - QEMU maintainer for BSD, many, many patches
Ed Maste - QEMU patches and cat herder
Peter Wemm - Sigtramp patch, because wow.
Alexander Kabaev - QEMU patches
Adrian Chadd - Ignored my pleading for help with kern_imgact.c
Baptiste Daroussin - Inflicting me with a ports commit bit

4. Contributors and Generally Awesome
People
Dimitry Andric - CLANG Help and Updates
Andrew Turner - GCC and Ports Patches
Mikael Urankar - Mysql Patches
Bryan Drewery - Taking me on a journey through poudriere
Warner Losh - Created the native-xtools target (doubled speed)
Ian Lapore - ARMv6 Assembly Help
Brooks Davis - Inspiration and initial guidance

5. Kernel Components (11-Current)
imgact_binmisc.ko
● Kernel Hook to redirect execution
● Examines ELF Header
● If match on preconfigured setting, modifes
ARGV[0] with emulator helper

6. Kernel Components - Continued
binmiscctl(8) will load and configure settings, e.
g.
binmiscctl add armv6 --interpreter "/usr/local/bin/qemu-arm" --magic
"x7fx45x4cx46x01x01x01x00x00x00x00x00x00x00x00x00x02x00x28x00"
--mask "xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxfexffxffxff"
--size 20 --set-enabled
binmiscctl add mips --interpreter "/usr/local/bin/qemu-mips" --magic
"x7fx45x4cx46x01x02x01x00x00x00x00x00x00x00x00x00x00x02x00x08"
--mask "xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxffxfexffxff"
--size 20 --set-enabled

7. Kernel Components - Continued
binmisctl list
name: mips
interpreter: /usr/local/bin/qemu-mips
flags: ENABLED USE_MASK
magic size: 20
magic offset: 0
magic: 0x7f 0x45 0x4c 0x46 0x01 0x02 0x01 0x00 0x00 0x00 0x00 0x00
0x00 0x00 0x00 0x00 0x00 0x02 0x00 0x08
mask: 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0x00 0xff 0xff 0xff 0xff
0xff 0xff 0xff 0xff 0xff 0xfe 0xff 0xff

8. Userland Components (11-Current)
QEMU
● Use emulators/qemu-static-user
● Installs statically linked binaries
● Emulation is a journey not a destination
● IOCTL emulation is still in progress

9. Userland Components - Continued
Poudriere-devel
● Easiest way to get started building jails
● Knows how to to understand binmiscctl(8)
● Knows to copy QEMU into jails
● Creates clean backup, in case of accident
● Use ZFS, save yourself some pain

10. Build the jail -- poudriere-devel
● poudriere jail -c -j 11armv632 -m svn -v head -a arm.
armv6 -x
● poudriere jail -c -j 11mips32 -m svn -v head -a mips.
mips -x
● poudirere jail -c -j 11mips64 -m svn -v head -a mips.
mips64 -x
● poudriere ports -c -m svn (so you have something to
build)

11. Configure the jail and mount
poudriere will create:
zroot/poudriere/jails/11armv632 112176079 1096098 111079980 1%
/usr/local/poudriere/jails/11armv632
zroot/poudriere/ports/default 118616467 7536487 111079980 6% /usr/local/poudriere/ports/default
Manually mount devfs/ports:
mount -t devfs devfs <path to jail/dev
mount -t nullfs /usr/local/poudriere/ports/default <path to jail>/usr/ports

12. Chroot and enjoy
root@dirty.ysv:/home/sbruno/fbsd_head # uname -a
FreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11
02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY amd64
root@dirty.ysv:~ # chroot /usr/local/poudriere/jails/11armv632
root@dirty:/ # uname -a
FreeBSD dirty.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r271414: Thu Sep 11
02:48:22 UTC 2014 sbruno@dirty.ysv.freebsd.org:/usr/obj/usr/src/sys/DIRTY arm

13. Poudriere Demo
● poudriere bulk -a -j 11armv632
● nginx rendering of status
● sharp edges remain, signal handling

14. Building and testing - DEMO
● chroot and build, its magic
● needs better documentation
● man page updates
● binmiscctl(8) syntax is precise and you will
always copy/paste it from somewhere

15. Future Development
● CLANG for MIPS will make more ports build
● AARCH64 (ARM64) support will start soon,
or send me patches for QEMU now.
● IOCTL debug handling, human readable
format

16. Questions?
There’s plenty of things that I’ve probably
skipped, what do you want to do with this
feature?