Understanding and building Your Own Docker

1/22
Building blocks of Linux Containers
Motiejus Jakˇstys
motiejus@uber.com
@mo kelione
2016-11-18
c 2016. Uber Technologies Inc. All rights reserved.

2/22
Table of Contents
Introduction
Why me
A container in Linux is...
Namespaces
Isolation in Linux
What did we just do
File systems and COW
What did we forget?
Leftover elephants in the room
The End

3/22
Conclusion!

3/22
Conclusion!
Devil Hides in The Details.

3/22
Conclusion!
Devil Hides in The Details?

3/22
Conclusion!
Many use Docker.

3/22
Conclusion!
Many use Docker.
We lack time to understand.

3/22
Conclusion!
Many use Docker.
You need to understand infra to successfully
troubleshoot infra.

3/22
Conclusion!
Many use Docker.
troubleshoot infra.
There are trade-oﬀs in the conﬁguration.

3/22
Conclusion!
Many use Docker.
troubleshoot infra.
Make container engine in 30 minutes.

3/22
Conclusion!
Many use Docker.
troubleshoot infra.
Details!

3/22
Conclusion!
Many use Docker.
troubleshoot infra.
Details! → You will still pick existing tools.

4/22
Why me
My resume: oncall experience.
2009 − 2012 Telecom (Dev + Ops).
2012 − 2014 Online Gaming (Dev + Ops).
2014 − 2016 Amazon (Dev + Ops).
2016 − now Uber (Dev + Ops):
From 2016.02: Dev.
From 2016.11: SRE.

4/22
Why me
My resume: oncall experience.
2009 − 2012 Telecom (Dev + Ops).
2012 − 2014 Online Gaming (Dev + Ops).
2014 − 2016 Amazon (Dev + Ops).
2016 − now Uber (Dev + Ops):
From 2016.02: Dev.
From 2016.11: SRE.
I had to understand how exactly infrastructure
works.

5/22
A container in Linux is ...

5/22
Fork/exec with bells & whistles:

5/22
Fancy tarball for distribution.

5/22
COW ﬁlesystem to make it start fast.

5/22
Cgroups for fairness.

5/22
Cgroups for fairness.
Namespaces for isolation.

6/22
Table of Contents
Introduction
Why me
Namespaces
Isolation in Linux
What did we just do
What did we forget?
The End

7/22
We will cover

7/22
We will cover
User namespaces.

7/22
We will cover
User namespaces.
Pid namespaces.

7/22
We will cover
User namespaces.
Pid namespaces.
Mount namespaces.

7/22
We will cover
User namespaces.
Pid namespaces.
Mount namespaces.
Network namespaces.

7/22
We will cover
User namespaces.
Pid namespaces.
Mount namespaces.
Network namespaces.
There are more, but not today.

8/22
User namespace
Become container-local root.
unshare --map-root-user

9/22
Mount namespace
Hide container mounts.
unshare --mount

10/22
Pid namespace
Hide other pids.
unshare --pid --mount-proc --fork

11/22
Network namespace
Demonstrate this:

11/22
Network namespace
Demonstrate this:
Create namespace.

11/22
Network namespace
Demonstrate this:
Create namespace.
Activate loopback (lo).

11/22
Network namespace
Demonstrate this:
Create namespace.
Create pair of devices veth1a and veth1b:

11/22
Network namespace
Demonstrate this:
Create namespace.
veth1b will go to the namespace.

11/22
Network namespace
Demonstrate this:
Create namespace.
veth1a will stay in default.

11/22
Network namespace
Demonstrate this:
Create namespace.
Add ip addresses.

11/22
Network namespace
Demonstrate this:
Create namespace.
Add ip addresses.
curl and ping.

11/22
Network namespace
Demonstrate this:
Create namespace.
Add ip addresses.
curl and ping.
lsof, bind on ports separately.

11/22
Network namespace
Demonstrate this:
Create namespace.
Add ip addresses.
curl and ping.
lsof, bind on ports separately.
Ever wanted to run tcpdump on an application?

12/22
Network namespace
default
lo
127.0.0.1
eth0
192.0.2.1

12/22
Network namespace
default
lo
127.0.0.1
eth0
192.0.2.1
t1
lo

12/22
Network namespace
default
lo
127.0.0.1
eth0
192.0.2.1
t1
lolo
127.0.0.1

12/22
Network namespace
default
lo
127.0.0.1
eth0
192.0.2.1
t1
lolo
127.0.0.1
veth1a veth1b

12/22
Network namespace
default
lo
127.0.0.1
eth0
192.0.2.1
t1
lolo
127.0.0.1
veth1a veth1bveth1a
10.0.0.1
veth1b
10.0.0.2

13/22
What did we just do
Created a container:

13/22
What did we just do
User namespace apt-get, iptables, mount, etc.

13/22
What did we just do
Isolated pids no nobody, isolate from each other.

13/22
What did we just do
Isolated mounts e.g. for /tmp.

13/22
What did we just do
Isolated network safely bind to :80.

13/22
What did we just do
Isolated network safely bind to :80.
An improvement over ”run and hope it doesn’t
aﬀect anything else”.

14/22
Table of Contents
Introduction
Why me
Namespaces
Isolation in Linux
What did we just do
What did we forget?
The End

15/22
A container:
Needs a ﬁle system.

15/22
A container:
Starts quickly regardless of size.

15/22
A container:
Do not want to copy 1GB with every startup.

15/22
A container:
Copy On Write!

15/22
A container:
Copy On Write!
lvm? zfs? btrfs?

16/22
A quick demo
Create tank/images/debian@latest
Create tank/containers/t1 from @latest
unshare --mount --pid --fork chroot . bash

17/22
Table of Contents
Introduction
Why me
Namespaces
Isolation in Linux
What did we just do
What did we forget?
The End

18/22

18/22
Trivial to escape this ”container”.

18/22
Sec: no leftover ﬁle descriptors.

18/22
Resource fairness.

18/22
Resource fairness.
Sec/DoS: shared kernel resources.

18/22
Resource fairness.
Supervision, daemonization and cleanup.

18/22
Resource fairness.
Logging.

18/22
Resource fairness.
Logging.
Collect zombie processes.

18/22
Resource fairness.
Logging.
Image management.

18/22
Resource fairness.
Logging.
Image management.
Should someone else do it?

19/22
We almost have a container engine

19/22
But look at my conclusions again.

19/22
Devil hides in the details.

19/22
Tooling companies (Docker, CoreOS, etc)
raised > $108
.

20/22
To recap
Easy to understand kernel facilities.

20/22
To recap

20/22
To recap
Either spend a lot of time and headache, or
re-use existing tools.

21/22
Table of Contents
Introduction
Why me
Namespaces
Isolation in Linux
What did we just do
What did we forget?
The End

22/22
We’re hiring!
Uber SRE locations: SF, NYC, Seattle, Vilnius.
Check out join.uber.com
Also, contact me at motiejus@uber.com

Understanding and building Your Own Docker

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to Understanding and building Your Own Docker

Similar to Understanding and building Your Own Docker (20)

Recently uploaded

Recently uploaded (20)

Understanding and building Your Own Docker