My talk on LX branded zones at Snow.nl March 26 and 27, 2015.

1. LX Branded Zones
1
Friday, March 27, 15

2. Topic Outline
2
• Brief description of container mechanisms
• SmartOS Zones
• Branded Zones
• LX Branded Zones
• Demos
• References
• slides at http://www.slideshare.net/mrbruning/lxbrand
Friday, March 27, 15

3. 3
OS OSOS
Hypervisor
Hardware
OS OSOS
Hypervisor
Hardware
OS
AppsApps Apps
OS
Hardware
Apps Apps Apps Apps Apps Apps
Container Architectures
• Protection via OS per
Container and Hypervisor
• Visibility within
hypervisor is limited
• Duplication of effort
• Example: xen
• Can be done without
OS support
• Protection via OS per
container and host OS
• Duplication of effort
• Example: qemu/kvm
• Fast (no duplication of
work)
• High visibility of all
components
• Protection via OS
• Requires OS changes
• Example: SmartOS
Friday, March 27, 15

4. OS Virtualization
4
•Each container sees their own (virtualized) OS
•In reality, only one OS
•From within a container you can only
“see” (depending on permissions) things within that
container
•Minimizes code and data path between application
and hardware
•Problem
•Apps must be built for the OS
Friday, March 27, 15

5. SmartOS LX-Branded Zones
5
•Execute Linux binaries directly on SmartOS kernel
•Uses zfs datasets for file system
•Uses zones for protection
•Processes running in a zone can only see what is in
the zone
•vxlan networking
•Allows use of SmartOS tools within the zone
•DTrace, mdb, proc tools, zlogin, etc.
Friday, March 27, 15

6. LX-branded Zones -
Implementation
•Processes within LX-branded zone use library
interpositioning (done transparently) to handle
system calls
•Some system calls are passed directly to the
SmartOS system call
• read(2), write(2), getpid(2), etc.
•System calls not in SmartOS are implemented using
library or in SmartOS
•Each thread within an lx process has 2 stacks
•One for the native stack, and one for the “brand”
stack. Also used for signal handling. 6
Friday, March 27, 15

7. Why LX Branded Zones
•Allows you to run docker containers on SmartOS
•Gets rid of problems with running Docker on a
hardware virtualized environment
•Security
•Observability
•Bare Metal Performance
•Start at http://www.joyent.com to read about Triton,
our Elastic Container-Native Infrastructure
7
Friday, March 27, 15

8. Images
•The data and metadata required to create a new
container.
•Existing images can be “imported”.
•New images can be created and provisioned.
8
Friday, March 27, 15

9. Creating an LX-Branded Zone -
Import an Image
9
# imgadm sources -a https://updates.joyent.com
Added "imgapi" image source "https://updates.joyent.com"
# imgadm avail | grep -i lx
f7c19252-c998-11e4-be95-3315493f3741 lx-centos-6 20150313 other
2015-03-13T15:52:35Z
818cc79e-ceb3-11e4-99ee-7bc8c674e754 lx-ubuntu-14.04 20150320 other
2015-03-20T03:45:09Z
...
# imgadm import 818cc79e-ceb3-11e4-99ee-7bc8c674e754
Importing 818cc79e-ceb3-11e4-99ee-7bc8c674e754 (lx-ubuntu-14.04@20150320) from
"https://images.joyent.com”
# imgadm list
UUID NAME VERSION OS
PUBLISHED
818cc79e-ceb3-11e4-99ee-7bc8c674e754 lx-ubuntu-14.04 20150320 other
2015-03-20T03:45:09Z
#
Friday, March 27, 15

10. Creating an LX Branded Zone -
JSON config file
•File used when creating a new virtual system
•Contains properties of the new system
•image uuid, alias brand, cpu cap and shares,
metadata, file system info, network info, etc.
•See vmadm(1M)for list of properties
10
Friday, March 27, 15

11. Creating an LX Branded Zone -
Example JSON config file
11
# cat /var/tmp/lx.json
{
"alias": "lxtest03242015",
"brand": "lx",
"kernel_version": "3.13.0",
"max_physical_memory": 2048,
"image_uuid": "818cc79e-ceb3-11e4-99ee-7bc8c674e754",
"resolvers": ["8.8.8.8","8.8.4.4"],
"nics": [
{
"nic_tag": "admin",
"ip": "10.88.88.71",
"netmask": "255.255.255.0",
"gateway": "10.88.88.2"
}
],
"customer_metadata": {
"root_authorized_keys":
"ssh-rsa AAAA...”
}
}
#
Friday, March 27, 15

12. vmadm(1M) - Create the new
machine
12
# vmadm create -f lx.json
Successfully created VM 203e8515-a1fa-4150-ad93-6e3ce1ee3b21
#
# zlogin 203e8515-a1fa-4150-ad93-6e3ce1ee3b21
[Connected to zone '203e8515-a1fa-4150-ad93-6e3ce1ee3b21' pts/4]
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0 x86_64)
* Documentation: https://help.ubuntu.com/
...
#ifconfig eth0
eth0 Link encap:Ethernet HWaddr 72:af:f4:3c:a0:23
inet addr:10.88.88.71 Mask:255.255.255.0
...
#
# ssh root@10.88.88.71 <-- from another host
[Connected to zone '203e8515-a1fa-4150-ad93-6e3ce1ee3b21' pts/4]
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0 i686)
...
#
Friday, March 27, 15

13. SmartOS Commands within LX
Branded Zones
•SmartOS binaries are available within the zone
•lofi mounted from global zone
•/native/usr
•/native/sbin
•/native/lib
•Others can be added
•Apps in LX zone should look and act like apps on
linux
•Subset of Linux /proc
•/sys exists but is empty
13
Friday, March 27, 15

14. Some Example Commands
14
# /native/usr/bin/prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
30597 root 25M 3796K sleep 59 0 0:00:00 0.0% top/1
13593 root 18M 2408K sleep 20 0 0:00:00 0.0% upstart-
socket-/1
30389 root 108M 7372K sleep 59 0 0:00:00 0.0% sshd/1
...
Total: 17 processes, 23 lwps, load averages: 0.10, 0.03, 0.01
# top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
30597 root 59 0 26076 3796 0 R 0.2 0.1 0:00.06 top
30564 root 59 0 4220 3448 0 S 0.1 0.1 0:00.02
prstat
30389 root 59 0 110912 7372 0 S 0.0 0.2 0:00.06 sshd
...
top - 23:11:33 up 2 days, 1:08, 2 users, load average: 0.11, 0.02,
0.00
Tasks: 16 total, 1 running, 15 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.6 us, 7.9 sy, 0.0 ni, 89.3 id, 0.0 wa, 0.2 hi, 0.0 si,
0.0 st
KiB Mem: 4194304 total, 45480 used, 4148824 free, 0 buffers
KiB Swap: 8388608 total, 24632 used, 8363976 free. 0 cached
Mem
Friday, March 27, 15

15. Using DTrace in LX Branded
Zone
•Login via zlogin or ssh is slow.
•Internet search for slow login on Ubuntu yields
(among many others)
•http://askubuntu.com/questions/11538/long-wait-
time-on-login
•Suggested solution is to comment out pam_motd
lines in /etc/pam.d/sshd and /etc/pam.d/
login
•We’ll use DTrace within the LX zone to come up with
a solution
•Note that we shall not “root cause” the problem (but
we could) 15
Friday, March 27, 15

16. DTrace Example
16
# export PATH=/native/usr/sbin
# dtrace -qn 'BEGIN{printf("PID PPID COMMANDn");}
proc:::exec-success/progenyof($target)/
{printf("%d %d %sn", pid, ppid, curpsinfo->pr_psargs);}'
-p `pgrep sshd`
PID PPID COMMAND
4483 2824 /usr/sbin/sshd -D -R
...
4488 4487 run-parts --lsbsysinit /etc/update-motd.d
4489 4488 /bin/sh /etc/update-motd.d/00-header
...
4493 4488 /bin/sh /etc/update-motd.d/10-help-text
4496 4488 /bin/sh /etc/update-motd.d/50-landscape-sysinfo
4497 4496 grep -c ^processor /proc/cpuinfo
...
4773 4753 /bin/sh -e /usr/lib/update-notifier/update-motd-updates-
available
...
4516 4488 /bin/sh /etc/update-motd.d/98-fsck-at-reboot
4516 4488 /bin/sh /usr/lib/update-notifier/update-motd-fsck-at-reboot
4517 4516 stat -c %Y /var/lib/update-notifier/fsck-at-reboot
...
4535 4523 mesg n
^C
Friday, March 27, 15

17. DTrace Example - Continued
17
# run-parts --lsbsysinit /etc/update-motd.d
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0 i686)
* Documentation: https://help.ubuntu.com/
System information as of Fri Mar 20 03:08:11 MDT 2015
System load: 0.0 Memory usage: 0% Users logged in: 1
Usage of /home: unknown Swap usage: 1%
=> There were exceptions while processing one or more plugins. See
/var/log/landscape/sysinfo.log for more information.
Graph this data and manage this system at:
https://landscape.canonical.com/
pause occurs here...
154 packages can be updated.
78 updates are security updates.
Friday, March 27, 15

18. DTrace Example - Continued
18
# dtrace -q -n 'proc:::exec-success/progenyof($target)/{printf("%d %d
%sn", pid, ppid, curpsinfo->pr_psargs);}' -c "run-parts --lsbsysinit /
etc/update-motd.d"
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0 i686)
...
Graph this data and manage this system at:
https://landscape.canonical.com/
4923 4921 /bin/sh4924 4923 uname -o
...
4942 4921 /bin/sh /etc/update-motd.d/90-updates-available
4942 4921 /bin/sh -e /usr/lib/update-notifier/update-motd-updates-
available
...
154 packages can be updated.
78 updates are security updates.
Friday, March 27, 15

19. DTrace Example - Continued
•In a more recent ubuntu image, /etc/update-
motd.d had several files removed, including /90-
updates-available. Problem is gone.
19
# /bin/sh -e /usr/lib/update-notifier/update-motd-updates-available
long pause here...
154 packages can be updated.
78 updates are security updates.
#
Friday, March 27, 15

20. DEMO
20
Friday, March 27, 15

21. References
• Creating an LX image - http://us-east.manta.joyent.com/jperkin/public/lximg/README
• General wiki page on lx branded zones - https://wiki.smartos.org/display/DOC/LX+Branded+Zones
• The source. A mid-level description of how it all works - /smartos-live/projects/illumos/usr/src/lib/brand/lx/
lx_brand/common/lx_brand.c has a large comment explaining a lot of the internals
• http://www.slideshare.net/bcantrill/illumos-lx - Some history and other details
• https://www.joyent.com/blog/container-service-preview - instructions for docker setup
• Thanks to Snow.nl for inviting me to speak.
21
Friday, March 27, 15