Extending docker using plugin example. Example using go language

1. Extending Docker: Image
Whitelist Plugin
mohammad fairus khalid
fairus.khalid@gmail.com
fairus.khalid@mimos.my

2. 2 hours quite long to talk
• Background (3 min)
• Motivation (2 min)
• Docker (25 min)
• Image whitelist concept (5)
• Setup (15 min)
• Demo (15 min)
• going through the code (15 min)
• Troubleshoot (10 min)
• Build & Package (10 min)
• Improvement (5 min)
• Q & A (15 min)

3. Background
generation processing

4. Edge Computing
DCThing
Own by Telco
Base station
Own by entities
Malls, schools, clinics
Voluntary Computing
Laptops, desktops, smartphones
ETSI – European Telecommunication Standards Institute
Control
QoS
Control
QoS

5. Motivation
• Put control at uncontrolled environment
• Only allow image/workload that is sanction to
run on the remote resources

6. Docker
• Docker Architecture
• Docker Plugin
• Plugin examples

7. Docker’s architecture
https://docs.docker.com/engine/understanding-docker/

8. Docker Plugin
• Docker plugins are out-of-process extensions
which add capabilities to the Docker Engine.
• They come in specific types. For example, a volume
plugin might enable Docker volumes to persist across
multiple Docker hosts and a network plugin might
provide network plumbing.
• A plugin is a process running on the same or a different
host as the docker daemon, which registers itself by
placing a file on the same docker host in one of the
plugin directories described in Plugin discovery.
https://docs.docker.com/engine/extend/plugins/

9. Plugin Discovery
• Docker discovers plugins by looking for them in the plugin
directory whenever a user or container tries to use one by
name.
• There are three types of files which can be put in the plugin
directory.
– .sock files are UNIX domain sockets.
– .spec files are text files containing a URL, such
as unix:///other.sock or tcp://localhost:8080.
– .json files are text files containing a full json specification for the
plugin.
• Plugins with UNIX domain socket files must run on the
same docker host, whereas plugins with spec or json files
can run on a different host if a remote URL is specified.
https://docs.docker.com/engine/extend/plugin_api/

10. Plugin API design
• The Plugin API is RPC-style JSON over HTTP, much like
webhooks.
• Requests flow from the Docker daemon to the plugin. So
the plugin needs to implement an HTTP server and bind
this to the UNIX socket mentioned in the “plugin discovery”
section.
• All requests are HTTP POST requests.
• The API is versioned via an Accept header, which currently
is always set to application/vnd.docker.plugins.v1+json.

11. 2 ways of Plugin Deployment
Host 1
dockerd
plugin
Host 2
plugin
unix
socket
tcp
socket
1
2

12. Plugin Type
Plugin Type Description Documentation
Authorization Extend API authorization
mechanism
https://docs.docker.com/engine/ext
end/authorization/
Network Extend network management https://docs.docker.com/engine/ext
end/plugins_network/
Volume Extend persistent storage https://docs.docker.com/engine/ext
end/plugins_volume/
IPAM Extend IP address management https://github.com/docker/libnetwo
rk/blob/master/docs/ipam.md

13. Volume Plugin
• Docker Engine volume plugins enable Engine
deployments to be integrated with external storage
systems, such as Amazon EBS, and enable data
volumes to persist beyond the lifetime of a single
Engine host.
• A volume plugin makes use of the -v and --volume-
driver flag on the docker run command. The -v flag
accepts a volume name and the--volume-driver flag a
driver type, for example:
– $ docker run -ti -v volumename:/data --volume-
driver=flocker busybox sh
https://docs.docker.com/engine/extend/plugins_volume/

14. Example Plugin Volume: Flocker
https://clusterhq.com/flocker/introduction/

15. Network Plugin
• Docker Engine network plugins enable Engine
deployments to be extended to support a
wide range of networking technologies, such
as VXLAN, IPVLAN, MACVLAN or something
completely different.

16. Docker Bridge Network
https://docs.docker.com/engine/userguide/networking/dockernetworks/
A bridge network is useful in cases where you want to run a relatively
small network on a single host. You can, however, create significantly
larger networks by creating an overlay network

17. Example Network plugin: Weave Net
https://www.weave.works/products/weave-net/
Weave Net creates a virtual network that connects Docker containers deployed
across multiple hosts. To application containers, the network established by
Weave resembles a giant Ethernet switch, where all containers are connected
and can easily access services from one another.

18. Weave Net Demo
• Install weave on 2 hosts
– sudo curl -L git.io/weave -o /usr/local/bin/weave
– sudo chmod a+x /usr/local/bin/weave
• Launch weave on both hosts
– weave launch <ip peer host>
fairus@ubuntu:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21e88b0b71e7 weaveworks/plugin:1.5.0 "/home/weave/plugin" 21 minutes ago Up 21 minutes
weaveplugin
4ff36b0361b0 weaveworks/weaveexec:1.5.0 "/home/weave/weavepro" 21 minutes ago Up 21 minutes weaveproxy
b9944c11dadf weaveworks/weave:1.5.0 "/home/weave/weaver -" 21 minutes ago Up 21 minutes weave
– check connection status
fairus@ubuntu:~$ weave status connections
-> 192.168.56.101:6783 established fastdp 56:d7:c9:8e:d3:c3(ubuntu)
– If fail reset and launch again
• weave reset; weave launch <ip peer host>

19. Weave Net Demo
• Test the weave network
– List weave network
fairus@ubuntu:~$ docker network ls
NETWORK ID NAME DRIVER
88033f39d3bf bridge bridge
02ae59ef8859 docker_gwbridge bridge
f610c728464c host host
41b5fa0b4ff7 none null
ff85a0ceb3cd weave weavemesh
– Run cntrn on both hosts
• docker run --net weave -it alpine sh
– Ping each other using ipaddress
– Run cntrn with hostname on both machine
• docker run --net weave -h c1.weave.local $(weave dns-args) -it alpine sh
• docker run --net weave -h c2.weave.local $(weave dns-args) -it alpine sh
– From the cntr ping each other using hostname (i.e. c1 & c2)
– From host check the dns entry
fairus@ubuntu:~$ weave status dns
c1 10.32.0.1 47097e28f5ee 56:d7:c9:8e:d3:c3
c2 10.40.0.1 acd772f5b19e 3e:40:22:84:b7:e7

20. Authorization Plugin
• An authorization plugin approves or denies
requests to the Docker daemon based on both the
current authentication context and the
command context.
• The authentication context contains all user details and the
authentication method.
• The command context contains all the relevant request
data.

21. Authorization Scenarios
https://docs.docker.com/engine/extend/plugins_authorization/

22. hands on

23. What is Image Whitelist
• https://github.com/fairuskhalid/whitelist
• Image Whitelist Docker plugin implementation
is based on Docker Authorization plugin
model. The plugin will look for the allowed
images from the whitelist before an image can
be run. With this the owner or administrator
of the host machine can control what can be
run on the machine.

24. Image Whitelist
Docker
daemon
whitelist
plugin
Docker
client
whitelist
server
getlist:
get updated list
Every specified
interval
run:
Try to run a
container
start:
Try to start
container
Check either image in the
whitelist or not. If yes
allow otherwise disallow.
Admin update
the whitelist.dat
allow/disallow
Command
success or
failed
1
2
3
4
5

25. Setup
• To use image whitelist plugin
– Install docker
– Pull image whitelist plugin from docker hub
– Try out the image whitelist
• To update and build the code
– Install go
– Setup environment
– Install make
– Build

26. Install docker
• https://docs.docker.com/engine/installation/
• Make sure it works:
– sudo docker run hello-world
• Allow user to run docker
– sudo groupadd docker
– sudo usermod –aG docker username
– Logout and login again

27. Try out image whitelist
• Pull the image from docker hub
– docker pull fairus/wlserver:v1
– docker pull fairus/wlplugin:v1
• Run the container
– docker run -d --restart=always -p 8080:8080
fairus/wlserver:v1
– docker run -d --restart=always -v /var/run:/var/run -v
/run/docker/plugins/:/run/docker/plugins -v
/etc/group:/etc/group fairus/wlplugin:v1 /wlplugin -
wlhost http://localhost:8080/getlist

28. Try out image whitelist
# pull image from docker hub
fairus@ubuntu:~$ docker pull fairus/wlserver:v1
fairus@ubuntu:~$ docker pull fairus/wlplugin:v1
# run the plugin and server
fairus@ubuntu:~$ docker run -d --restart=always -p 8080:8080 fairus/wlserver:v1
fairus@ubuntu:~$ docker run -d --restart=always -v /var/run:/var/run -v
/run/docker/plugins/:/run/docker/plugins -v /etc/group:/etc/group fairus/wlplugin:v1
/wlplugin -wlhost http://192.168.56.101/getlist
# update docker service in systemd
root@ubuntu:/run/docker/plugins# systemctl edit --full docker.service
.. .
ExecStart=/usr/bin/docker daemon -H fd:// --authorization-plugin=whitelist-plugin
.. .
# restart docker service
root@ubuntu:/run/docker/plugins# service docker restart
Terminal 1
Terminal 2
root user
# try run alpine container
fairus@ubuntu:~$ docker run -it alpine sh
docker: Error response from daemon: authorization denied by plugin whitelist-plugin:
Unauthorized Image.
Terminal 1

29. Try out image whitelist
# check alpine image id
fairus@ubuntu:~$ docker inspect alpine | grep Id
"Id": "sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac",
# copy whitelist.dat from wlserver container to the host
fairus@ubuntu:~$ docker ps | grep wlserver
5c561b1cd9b2 fairus/wlserver:v1 "/wlserver" 24 minutes ago
Up 21 minutes 0.0.0.0:8080->8080/tcp prickly_jang
fairus@ubuntu:~$ docker cp 5c561b1cd9b2:whitelist.dat whitelist.dat
fairus@ubuntu:~$ ls
goenv.sh src whitelist.dat work
# update whitelist.dat with alpine image id and copy back into wlserver container
fairus@ubuntu:~$ docker inspect alpine | grep Id
"Id": "sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac",
fairus@ubuntu:~$ echo sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac
> whitelist.dat
fairus@ubuntu:~$ docker cp whitelist.dat 5c561b1cd9b2:whitelist.dat
# wait for couple of minutes before run the alpine container again
fairus@ubuntu:~$ docker cp whitelist.dat 5c561b1cd9b2:whitelist.dat
fairus@ubuntu:~$ docker run -it alpine sh
/ #
Terminal 1

30. Setup Build Environment
• Install go
– follow guide in: https://golang.org/doc/install
– wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz
– sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz
– export PATH=$PATH:/usr/local/go/bin
• Test go
– go version
• Create and export working dir (e.g. work)
– mkdir work
– export GOPATH=$HOME/work
• Install godep
– go get github.com/tools/godep
• Install make
– sudo apt-get install make

31. Going Through The Code
• Go helpers package:
– https://github.com/docker/go-plugins-helpers
– https://godoc.org/github.com/docker/go-plugins-helpers/authorization

32. Bare Minimum
Include Package
Plugin socket you bind to
Plugin class
Plugin instantiation
Interface Implementation

33. build and run plugin
# build the code
fairus@ubuntu:wshop$ go build -o wplugin main.go
fairus@ubuntu:wshop$ ls -l
total 8200
-rw-rw-r-- 1 fairus fairus 624 May 10 08:15 main.go
-rwxrwxr-x 1 fairus fairus 8382368 May 10 08:46 wplugin
# run the plugin
fairus@ubuntu:wshop$ sudo ./wplugin
# check the plugin is running
root@ubuntu:/run/docker/plugins# ls -l
total 0
srw-rw---- 1 root root 0 May 10 08:47 plugin.sock
srwxr-xr-x 1 root root 0 May 9 15:39 weavemesh.sock
srwxr-xr-x 1 root root 0 May 9 15:39 weave.sock
Terminal 1
Terminal 2
root user

34. setup dockerd & run test
# update docker service in systemd
root@ubuntu:/run/docker/plugins# systemctl edit --full docker.service
.. .
ExecStart=/usr/bin/docker daemon -H fd:// --authorization-plugin=plugin
.. .
# restart docker service
root@ubuntu:/run/docker/plugins# service docker restart
# try it out
root@ubuntu:/run/docker/plugins# docker ps
Terminal 2
Root user
# kill the plugin
fairus@ubuntu:wshop$ sudo ./wplugin
^C
# try it out again
root@ubuntu:/run/docker/plugins# docker ps
Error response from daemon: plugin plugin failed with error: Post
http://%2Frun%2Fdocker%2Fplugins%2Fplugin.sock/AuthZPlugin.AuthZReq: dial
unix /run/docker/plugins/plugin.sock: connect: connection refused
Terminal 2
Root user
Terminal 1

35. Looking at the message
https://github.com/docker/docker/blob/master/pkg/authorization/api.go

36. build and run again
This how the message looks like. Now we can add in our logic.

37. Image whitelist implementation
• Package engineapi provides libraries to implement client and server
components compatible with the Docker engine. The client package in
github.com/docker/engine-api/client implements all necessary requests to
implement the official Docker engine cli.
– Create a new client, then use it to send and receive messages to the Docker
engine API:
defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"}
cli, err := client.NewClient("unix:///var/run/docker.sock", "v1.22", nil, defaultHeaders)
– https://godoc.org/github.com/docker/engine-api
• This library implements a cron spec parser and runner.
– https://godoc.org/github.com/robfig/cron
• Package logrus is a structured logger for Go, completely API compatible
with the standard library logger.
– https://godoc.org/github.com/Sirupsen/logrus

38. Build and Package
• Get the code from github using go
– go get github.com/fairuskhalid/whitelist
• Restore the dependencies (this will take a while)
– cd $HOME/work/src/github.com/fairuskhalid/whitelist
– godep restore
• Build
– make
• Create a docker image
– cp wlplugin executable and Dockerfile.plugin into empty
dir
– Rename Dockerfile.plugin into Dockerfile
– docker build –t plugin .

39. Troubleshoot
Docker
daemon
whitelist
plugin
Docker
client
/run/docker/plugins/whitelist-plugin.sock
Create fake socket
/run/docker/plugins/fake.sock
/var/run/docker.sock
1. Install socat
• Sudo apt-get install socat
2. Run Plugin
3. Run socat
• sudo socat -v UNIX-LISTEN:/run/docker/plugins/fake.sock UNIX-CONNECT:/run/docker/plugins/whitelist-plugin.sock
4. Run docker daemon with plugin option point to fake socat
• --authorization-plugin=fake
5. Run docker command and view message output at terminal

40. Improvement
• Any suggestion?

41. Q & A

42. Terima Kasih