Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

moscmy2016: Extending Docker

2,427 views

Published on

Extending docker using plugin example. Example using go language

Published in: Software
  • Be the first to comment

moscmy2016: Extending Docker

  1. 1. Extending Docker: Image Whitelist Plugin mohammad fairus khalid fairus.khalid@gmail.com fairus.khalid@mimos.my
  2. 2. 2 hours quite long to talk • Background (3 min) • Motivation (2 min) • Docker (25 min) • Image whitelist concept (5) • Setup (15 min) • Demo (15 min) • going through the code (15 min) • Troubleshoot (10 min) • Build & Package (10 min) • Improvement (5 min) • Q & A (15 min)
  3. 3. Background generation processing
  4. 4. Edge Computing DCThing Own by Telco Base station Own by entities Malls, schools, clinics Voluntary Computing Laptops, desktops, smartphones ETSI – European Telecommunication Standards Institute Control QoS Control QoS
  5. 5. Motivation • Put control at uncontrolled environment • Only allow image/workload that is sanction to run on the remote resources
  6. 6. Docker • Docker Architecture • Docker Plugin • Plugin examples
  7. 7. Docker’s architecture https://docs.docker.com/engine/understanding-docker/
  8. 8. Docker Plugin • Docker plugins are out-of-process extensions which add capabilities to the Docker Engine. • They come in specific types. For example, a volume plugin might enable Docker volumes to persist across multiple Docker hosts and a network plugin might provide network plumbing. • A plugin is a process running on the same or a different host as the docker daemon, which registers itself by placing a file on the same docker host in one of the plugin directories described in Plugin discovery. https://docs.docker.com/engine/extend/plugins/
  9. 9. Plugin Discovery • Docker discovers plugins by looking for them in the plugin directory whenever a user or container tries to use one by name. • There are three types of files which can be put in the plugin directory. – .sock files are UNIX domain sockets. – .spec files are text files containing a URL, such as unix:///other.sock or tcp://localhost:8080. – .json files are text files containing a full json specification for the plugin. • Plugins with UNIX domain socket files must run on the same docker host, whereas plugins with spec or json files can run on a different host if a remote URL is specified. https://docs.docker.com/engine/extend/plugin_api/
  10. 10. Plugin API design • The Plugin API is RPC-style JSON over HTTP, much like webhooks. • Requests flow from the Docker daemon to the plugin. So the plugin needs to implement an HTTP server and bind this to the UNIX socket mentioned in the “plugin discovery” section. • All requests are HTTP POST requests. • The API is versioned via an Accept header, which currently is always set to application/vnd.docker.plugins.v1+json.
  11. 11. 2 ways of Plugin Deployment Host 1 dockerd plugin Host 2 plugin unix socket tcp socket 1 2
  12. 12. Plugin Type Plugin Type Description Documentation Authorization Extend API authorization mechanism https://docs.docker.com/engine/ext end/authorization/ Network Extend network management https://docs.docker.com/engine/ext end/plugins_network/ Volume Extend persistent storage https://docs.docker.com/engine/ext end/plugins_volume/ IPAM Extend IP address management https://github.com/docker/libnetwo rk/blob/master/docs/ipam.md
  13. 13. Volume Plugin • Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Engine host. • A volume plugin makes use of the -v and --volume- driver flag on the docker run command. The -v flag accepts a volume name and the--volume-driver flag a driver type, for example: – $ docker run -ti -v volumename:/data --volume- driver=flocker busybox sh https://docs.docker.com/engine/extend/plugins_volume/
  14. 14. Example Plugin Volume: Flocker https://clusterhq.com/flocker/introduction/
  15. 15. Network Plugin • Docker Engine network plugins enable Engine deployments to be extended to support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different.
  16. 16. Docker Bridge Network https://docs.docker.com/engine/userguide/networking/dockernetworks/ A bridge network is useful in cases where you want to run a relatively small network on a single host. You can, however, create significantly larger networks by creating an overlay network
  17. 17. Example Network plugin: Weave Net https://www.weave.works/products/weave-net/ Weave Net creates a virtual network that connects Docker containers deployed across multiple hosts. To application containers, the network established by Weave resembles a giant Ethernet switch, where all containers are connected and can easily access services from one another.
  18. 18. Weave Net Demo • Install weave on 2 hosts – sudo curl -L git.io/weave -o /usr/local/bin/weave – sudo chmod a+x /usr/local/bin/weave • Launch weave on both hosts – weave launch <ip peer host> fairus@ubuntu:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 21e88b0b71e7 weaveworks/plugin:1.5.0 "/home/weave/plugin" 21 minutes ago Up 21 minutes weaveplugin 4ff36b0361b0 weaveworks/weaveexec:1.5.0 "/home/weave/weavepro" 21 minutes ago Up 21 minutes weaveproxy b9944c11dadf weaveworks/weave:1.5.0 "/home/weave/weaver -" 21 minutes ago Up 21 minutes weave – check connection status fairus@ubuntu:~$ weave status connections -> 192.168.56.101:6783 established fastdp 56:d7:c9:8e:d3:c3(ubuntu) – If fail reset and launch again • weave reset; weave launch <ip peer host>
  19. 19. Weave Net Demo • Test the weave network – List weave network fairus@ubuntu:~$ docker network ls NETWORK ID NAME DRIVER 88033f39d3bf bridge bridge 02ae59ef8859 docker_gwbridge bridge f610c728464c host host 41b5fa0b4ff7 none null ff85a0ceb3cd weave weavemesh – Run cntrn on both hosts • docker run --net weave -it alpine sh – Ping each other using ipaddress – Run cntrn with hostname on both machine • docker run --net weave -h c1.weave.local $(weave dns-args) -it alpine sh • docker run --net weave -h c2.weave.local $(weave dns-args) -it alpine sh – From the cntr ping each other using hostname (i.e. c1 & c2) – From host check the dns entry fairus@ubuntu:~$ weave status dns c1 10.32.0.1 47097e28f5ee 56:d7:c9:8e:d3:c3 c2 10.40.0.1 acd772f5b19e 3e:40:22:84:b7:e7
  20. 20. Authorization Plugin • An authorization plugin approves or denies requests to the Docker daemon based on both the current authentication context and the command context. • The authentication context contains all user details and the authentication method. • The command context contains all the relevant request data.
  21. 21. Authorization Scenarios https://docs.docker.com/engine/extend/plugins_authorization/
  22. 22. hands on
  23. 23. What is Image Whitelist • https://github.com/fairuskhalid/whitelist • Image Whitelist Docker plugin implementation is based on Docker Authorization plugin model. The plugin will look for the allowed images from the whitelist before an image can be run. With this the owner or administrator of the host machine can control what can be run on the machine.
  24. 24. Image Whitelist Docker daemon whitelist plugin Docker client whitelist server getlist: get updated list Every specified interval run: Try to run a container start: Try to start container Check either image in the whitelist or not. If yes allow otherwise disallow. Admin update the whitelist.dat allow/disallow Command success or failed 1 2 3 4 5
  25. 25. Setup • To use image whitelist plugin – Install docker – Pull image whitelist plugin from docker hub – Try out the image whitelist • To update and build the code – Install go – Setup environment – Install make – Build
  26. 26. Install docker • https://docs.docker.com/engine/installation/ • Make sure it works: – sudo docker run hello-world • Allow user to run docker – sudo groupadd docker – sudo usermod –aG docker username – Logout and login again
  27. 27. Try out image whitelist • Pull the image from docker hub – docker pull fairus/wlserver:v1 – docker pull fairus/wlplugin:v1 • Run the container – docker run -d --restart=always -p 8080:8080 fairus/wlserver:v1 – docker run -d --restart=always -v /var/run:/var/run -v /run/docker/plugins/:/run/docker/plugins -v /etc/group:/etc/group fairus/wlplugin:v1 /wlplugin - wlhost http://localhost:8080/getlist
  28. 28. Try out image whitelist # pull image from docker hub fairus@ubuntu:~$ docker pull fairus/wlserver:v1 fairus@ubuntu:~$ docker pull fairus/wlplugin:v1 # run the plugin and server fairus@ubuntu:~$ docker run -d --restart=always -p 8080:8080 fairus/wlserver:v1 fairus@ubuntu:~$ docker run -d --restart=always -v /var/run:/var/run -v /run/docker/plugins/:/run/docker/plugins -v /etc/group:/etc/group fairus/wlplugin:v1 /wlplugin -wlhost http://192.168.56.101/getlist # update docker service in systemd root@ubuntu:/run/docker/plugins# systemctl edit --full docker.service .. . ExecStart=/usr/bin/docker daemon -H fd:// --authorization-plugin=whitelist-plugin .. . # restart docker service root@ubuntu:/run/docker/plugins# service docker restart Terminal 1 Terminal 2 root user # try run alpine container fairus@ubuntu:~$ docker run -it alpine sh docker: Error response from daemon: authorization denied by plugin whitelist-plugin: Unauthorized Image. Terminal 1
  29. 29. Try out image whitelist # check alpine image id fairus@ubuntu:~$ docker inspect alpine | grep Id "Id": "sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac", # copy whitelist.dat from wlserver container to the host fairus@ubuntu:~$ docker ps | grep wlserver 5c561b1cd9b2 fairus/wlserver:v1 "/wlserver" 24 minutes ago Up 21 minutes 0.0.0.0:8080->8080/tcp prickly_jang fairus@ubuntu:~$ docker cp 5c561b1cd9b2:whitelist.dat whitelist.dat fairus@ubuntu:~$ ls goenv.sh src whitelist.dat work # update whitelist.dat with alpine image id and copy back into wlserver container fairus@ubuntu:~$ docker inspect alpine | grep Id "Id": "sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac", fairus@ubuntu:~$ echo sha256:d7a513a663c1a6dcdba9ed832ca53c02ac2af0c333322cd6ca92936d1d9917ac > whitelist.dat fairus@ubuntu:~$ docker cp whitelist.dat 5c561b1cd9b2:whitelist.dat # wait for couple of minutes before run the alpine container again fairus@ubuntu:~$ docker cp whitelist.dat 5c561b1cd9b2:whitelist.dat fairus@ubuntu:~$ docker run -it alpine sh / # Terminal 1
  30. 30. Setup Build Environment • Install go – follow guide in: https://golang.org/doc/install – wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz – sudo tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz – export PATH=$PATH:/usr/local/go/bin • Test go – go version • Create and export working dir (e.g. work) – mkdir work – export GOPATH=$HOME/work • Install godep – go get github.com/tools/godep • Install make – sudo apt-get install make
  31. 31. Going Through The Code • Go helpers package: – https://github.com/docker/go-plugins-helpers – https://godoc.org/github.com/docker/go-plugins-helpers/authorization
  32. 32. Bare Minimum Include Package Plugin socket you bind to Plugin class Plugin instantiation Interface Implementation
  33. 33. build and run plugin # build the code fairus@ubuntu:wshop$ go build -o wplugin main.go fairus@ubuntu:wshop$ ls -l total 8200 -rw-rw-r-- 1 fairus fairus 624 May 10 08:15 main.go -rwxrwxr-x 1 fairus fairus 8382368 May 10 08:46 wplugin # run the plugin fairus@ubuntu:wshop$ sudo ./wplugin # check the plugin is running root@ubuntu:/run/docker/plugins# ls -l total 0 srw-rw---- 1 root root 0 May 10 08:47 plugin.sock srwxr-xr-x 1 root root 0 May 9 15:39 weavemesh.sock srwxr-xr-x 1 root root 0 May 9 15:39 weave.sock Terminal 1 Terminal 2 root user
  34. 34. setup dockerd & run test # update docker service in systemd root@ubuntu:/run/docker/plugins# systemctl edit --full docker.service .. . ExecStart=/usr/bin/docker daemon -H fd:// --authorization-plugin=plugin .. . # restart docker service root@ubuntu:/run/docker/plugins# service docker restart # try it out root@ubuntu:/run/docker/plugins# docker ps Terminal 2 Root user # kill the plugin fairus@ubuntu:wshop$ sudo ./wplugin ^C # try it out again root@ubuntu:/run/docker/plugins# docker ps Error response from daemon: plugin plugin failed with error: Post http://%2Frun%2Fdocker%2Fplugins%2Fplugin.sock/AuthZPlugin.AuthZReq: dial unix /run/docker/plugins/plugin.sock: connect: connection refused Terminal 2 Root user Terminal 1
  35. 35. Looking at the message https://github.com/docker/docker/blob/master/pkg/authorization/api.go
  36. 36. build and run again This how the message looks like. Now we can add in our logic.
  37. 37. Image whitelist implementation • Package engineapi provides libraries to implement client and server components compatible with the Docker engine. The client package in github.com/docker/engine-api/client implements all necessary requests to implement the official Docker engine cli. – Create a new client, then use it to send and receive messages to the Docker engine API: defaultHeaders := map[string]string{"User-Agent": "engine-api-cli-1.0"} cli, err := client.NewClient("unix:///var/run/docker.sock", "v1.22", nil, defaultHeaders) – https://godoc.org/github.com/docker/engine-api • This library implements a cron spec parser and runner. – https://godoc.org/github.com/robfig/cron • Package logrus is a structured logger for Go, completely API compatible with the standard library logger. – https://godoc.org/github.com/Sirupsen/logrus
  38. 38. Build and Package • Get the code from github using go – go get github.com/fairuskhalid/whitelist • Restore the dependencies (this will take a while) – cd $HOME/work/src/github.com/fairuskhalid/whitelist – godep restore • Build – make • Create a docker image – cp wlplugin executable and Dockerfile.plugin into empty dir – Rename Dockerfile.plugin into Dockerfile – docker build –t plugin .
  39. 39. Troubleshoot Docker daemon whitelist plugin Docker client /run/docker/plugins/whitelist-plugin.sock Create fake socket /run/docker/plugins/fake.sock /var/run/docker.sock 1. Install socat • Sudo apt-get install socat 2. Run Plugin 3. Run socat • sudo socat -v UNIX-LISTEN:/run/docker/plugins/fake.sock UNIX-CONNECT:/run/docker/plugins/whitelist-plugin.sock 4. Run docker daemon with plugin option point to fake socat • --authorization-plugin=fake 5. Run docker command and view message output at terminal
  40. 40. Improvement • Any suggestion?
  41. 41. Q & A
  42. 42. Terima Kasih

×