Dieser Vortrag zeigt, welchen Herausforderungen im Hosting punkt.de in der jüngeren Vergangenheit gegenüber stand und welche Änderungen gegenüber unserer 2010 vorgestellten "NanoBSD"-Architektur wir seitdem umgesetzt haben. Der proServer hat auf der Seite des Hosting-Anbieters viele der erwünschten Eigenschaften eines "Private Cloud"-Produkts, stellt sich dem Kunden aber wie ein klassischer Root-Server dar. Für Anwendungen, bei denen eine solche Plattform gefordert ist, ein unschätzbarer Vorteil gegenüber reinen Container-Lösungen, die praktisch immer eine speziell angepasste Anwendungs-Architektur benötigen.

1. FreeBSD as a Hosting Platform – Revisited
Patrick M. Hausen
EuroBSDCon 2017, Paris

2. Agenda
• Introduction
• Challenges in Hosting
• Our Old NanoBSD Setup
• Why a Jail Based Architecture
• How We Do it Today
• What We Would Like to Do in the Future

3. About Me
• Working in IT since 1986
• Minix 1.1 since 1989
• FreeBSD since 1993
• In charge of network and data centre operations at punkt.de

4. About Our Team
• mOps – the Magnificent Operators
• 3 (originally) operators
• 1 (originally) developer

5. About punkt.de
• Founded in 1996
• Started as an ISP
• Today:Hosting and development of web applications
• Roughly 100 Servers
• RIPE Member
• DENIC Member
• 2 development, 1 operations team

6. Challenges
• Availability
• Performance
• Cost
• Manageability
• … Scalability – of the entire data centre!

7. „Updates?
Never change a
running system!“

8. „Nobody wants
backup – everyone
wants restore.“

9. Our First Attempt: nanobsd
root
root
(alternate slice)
/cfg
/var
(customer data)
m0s1a m0s2a m0s3a m0s3d
mirror/m0
ada0 ada1

10. Advantages
• OS and packages are read–only
• Atomic updates
• Rollback (with exceptions)
• Identical software across all servers

11. Drawbacks
• We did not go all the way – image creation remained manual
• Reboot of the entire machine required
• Installation of additional ports afterwards is difficult
• Too little flexibility – PHP, MySQL versions …
We did address some of these:
• Image creation is now vagrant up
• Packages come from our own poudriere

12. Goals for the New Architecture
• Better isolation of customers on the same machine
• Individual configurations per customer (PHP, MySQL, …)
• N instances per physical machine
• Faster more reliable updates
• Fully automated

13. All the Rage Now …

14. So, Why Not a Hypervisor?
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Machine MachineMachineMachine

15. So, Why Not a Hypervisor?
• Each VM is managed like a separate system
• Memory and/or storage overprovisioning at least difficult
• Which one to pick?
• Storage is either fast or cheap

17. What Do You Want to Virtualize Today?
Machine Hypervisor
/sbin/init Jail
Single Server
Process
Docker

18. So …
• Our customers want
• The semantics of a VM
• We as operators want
• Fast provisioning
• Easy updates
• Our CEO wants
• Low cost (within reason)

19. Advantage Jails
• Look like a VM to the customer
• Low overhead
• Don’t require a running server process they depend on
• Look just like regular processes from the outside host
• Local filesystem semantics!
• Now come with virtualized network stack!
• Networking is straightforward and simple (if you know your basics)

20. VIMAGE
• Introduces the epair(4) virtual interface
• Essentially a virtual patch cable
• One end inside the jail, other end on the host system
• Bridge, route, NAT to your heart’s content

21. System Architecture
LAN
Jail Jail Jail Jail
Bridge Interface
vnet0 vnet0 vnet0 vnet0
igb0, ix0, …
IPv6 and IPv4 (subject to change)

22. Shameless Marketing Plug

23. What the Customer Gets
• Virtual proServer:
One jail instance on a large host.
• Dedicated proServer:
Own host with as many jails as he desires or the host can
bear.
• But it’s all the same technology!
Which makes it way easier for us.

24. Virtual proServer Host
• All SSD based
• All ZFS
• 256 GB of RAM
• 2x 10 Cores / 20 Threads
• 50 customer jails and twiddling it’s thumbs

25. Jail Management Tools
• Ezjail
• Warden („old“ FreeNAS jails)
• py-iocage („new“ FreeNAS jails)
We picked iocage and actively contribute to the current rewrite.
https://github.com/iocage/iocage

26. Peculiarities …
Template
Jail
Instance
Instance
Instance
copy-on-write
clones
One does not simply update …
Snapshot!

27. Blueprint Jails
• Not iocage templates!
• Regular jails with FreeBSD-X.Y-RELEASE (11.1 as of now)
• Contain all the software we think is relevant for the customer
PHP-FPM, MySQL/MariaDB, Elastic, NginX and Apache, …
• Use our own poudriere as the repo for pkg
• Created and configured with Ansible
• Not running! (after initial creation)

28. Instance Jail
• Empty jail in iocage
• Blueprint jail mounted on top – read-only, nullfs
• All the read-write directories are separate ZFS datasets
• Mountpoints are set to legacy
• Mounted at jail startup by iocage’s fstab feature

29. Filesystem Layout
/etc /usr/local/etc /tmp /var …
/iocage/jails/2017Q3-php70-es1/root (blueprint)
/iocage/jails/vpro0042/root (empty)
nullfs
RO
RW
zfs

30. A Jail’s fstab
/iocage/jails/2017Q3-php70-es1/root /iocage/jails/vpro0042/root nullfs ro …
zdata/iocage/jails/vpro0042/root/etc /iocage/jails/vpro0042/root/etc zfs rw …
zdata/iocage/jails/vpro0042/root/home /iocage/jails/vpro0042/root/home zfs rw …
zdata/iocage/jails/vpro0042/root/tmp /iocage/jails/vpro0042/root/tmp zfs rw …
zdata/iocage/jails/vpro0042/root/var /iocage/jails/vpro0042/root/var zfs rw …
[…]
/iocage/jails/2017Q3-php70-es1/root/usr/local/etc/rc.d
/iocage/jails/vpro0042/root/usr/local/etc/rc.d nullfs ro …
/iocage/jails/2017Q3-php70-es1/root/var/db/pkg
/iocage/jails/vpro0042/root/var/db/pkg nullfs ro …
zroot/vpro0042/mysql /iocage/jails/vpro0042/root/var/db/mysql zfs rw …
zroot/vpro0042/mysql/logs /iocage/jails/vpro0042/root/var/db/mysql/logs zfs rw …

31. Provisioning
proServer Host PXE (unattended)
Blueprint Jail Ansible, Poudriere
Instance Jail Ansible

32. Making Updating Great Again
• chroot <blueprint> pkg upgrade
• Actually we don’t do that, although we could …
• Immutable infrastructure!
• We create a new blueprint jail
• Then update all the dependent instances to use the new one

33. Backups
• Easy – ZFS snapshots
• Hourly, Daily, …
• sysutils/zfstools
• Differential clones to central (per rack) backup server
https://github.com/adaugherity/zfs-backup
• We have a port – will need some polishing to be included in the tree

34. What Now?
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Jail
Storage?

35. Self–Provisioning
• Would be nice ;-)
• Essentially a complete private cloud solution
• But first: an API (REST possibly)
• Then a UI can be done by a frontend developer
Possibly this already exists …

36. Questions?
?

37. Thanks!