We deployed a clustered real-time data processing platform on a group of Mac Minis for a proof of concept. Our experience was presented to the Melbourne Docker Meetup in October 2015.
3. Docker on OSX3 | 21 October 2015
Docker on OSX
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Container App Container App
4. Docker on OSX
Network
4 | 21 October 2015
Future Grid Platform
Agent
Cassandra
Application
Agent
Cassandra
Application
Agent
Cassandra
Application
Agent
Cassandra
Application
Agent
Cassandra
Application
• Built on Mac Minis to allow for destructive testing.
(e.g. Unplug network cables, unplug machines, etc)
5. Docker on OSX5 | 21 October 2015
Docker on OSX
Target Configuration
Cassandra, Agent and
Application put into Docker
containers.
Deployed and managed from a
single server.
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
6. Docker on OSX6 | 21 October 2015
Docker on OSX
Three Challenges
- Allow containers to
communicate via public IP
- Manage all hosts from a
single location.
- Store Cassandra data to
OSX file system.
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
7. Docker on OSX
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
7 | 21 October 2015
Networking
192.168.99.100
192.168.15.14
172.17.0.66 172.17.0.62
172.17.42.1
First Challenge
Docker uses two network
interfaces
o Adapter 1: NAT
(192.168.99.1)
o Adapter 2: Host-Only
(172.17.42.1)
Neither accessible from public
interface.
192.168.99.1
8. Docker on OSX
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
8 | 21 October 2015
Networking
192.168.99.100
192.168.15.14
172.17.0.66 172.17.0.62
172.17.42.1
Option 1 – Change NAT
interface to be bridged. ✗
Option 2 – Add bridged mode
Adapter 3 and share Ethernet
adapter. ✗
Option 3 – Add a USB Gigabit
adapter and add Adapter 3 in
Bridged mode. ✓
192.168.99.1
192.168.15.24
9. Docker on OSX9 | 21 October 2015
Networking
Docker uses Desktop adapter
type. Using anything else will
cause issues with
misnumbered adapters.
(i.e. eth0, eth1, eth2)
10. Docker on OSX10 | 21 October 2015
Networking
> docker-machine ssh default
> cd /var/lib/boot2docker
> vi bootlocal.sh
#!/bin/sh
ifconfig eth2 192.168.15.24 netmask 255.255.255.0
11. Docker on OSX11 | 21 October 2015
Remote Docker
Second Challenge
docker is exposed to a public
IP address.
docker-machine is designed to
be locally managed.
docker-machine env command
is very opinionated. Builds its
own certificate chain.
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
192.168.15.24
12. Docker on OSX12 | 21 October 2015
Remote Docker
Ideally, create a single
Certificate Authority (CA) and
have it sign each machine and
client certificate.
Follow directions provided by
Docker.
https://docs.docker.com/articles/https/
CA
Machine-1Machine-1Machine-1Machine-1Machine-1
client
13. Docker on OSX13 | 21 October 2015
Remote Docker
> docker-machine stop default
> docker-machine rm default
> docker-machine --tls-ca-cert=/data/certs/ca.pem
--tls-ca-key=/data/certs/ca-key.pem
--tls-client-cert=/data/certs/user-cert.pem
--tls-cert-key=/data/certs/user-key.pem create -d virtualbox
--engine-env HTTP_PROXY=http://user:AxYtVzz@proxy:8080
--engine-env HTTPS_PROXY=http://user:AxYtVzz@proxy:8080
--engine-insecure-registry 192.168.15.100 default
> docker-machine start default
Create the docker machine with specified server cert and key
Reconfigure eth2 IP Address!
14. Docker on OSX14 | 21 October 2015
Remote Docker
> docker –tlsverify=false –H tcp://m01:2376 ps –a
Finally, we can run commands remotely.
15. Docker on OSX
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
15 | 21 October 2015
Storage
Challenge Three
docker-machine configures
/Users to map to the local
machine.
You can add other mappings
in VirtualBox
Cassandra uses hardlinks
which both VirtualBox and
VMWare do not support.
/Users
/data
16. Docker on OSX16 | 21 October 2015
Storage
> sudo vi /etc/exports
/data -mapall=user:staff -network 192.168.99 –mask 255.255.255.0
> sudo nfsd update
Configure OSX host to allow VirtualBox to connect via NFS
17. Docker on OSX17 | 21 October 2015
Storage
> sudo vi /etc/exports
/data -mapall=user:staff -network 192.168.99 –mask 255.255.255.0
> sudo nfsd update
Configure docker-machine to map /data to server.
> docker-machine ssh default
> cd /var/lib/boot2docker
> vi bootlocal.sh
#!/bin/sh
sleep 1
sudo mkdir -p /data
sudo /usr/local/etc/init.d/nfs-client start
sudo mount 192.168.99.1:/data /data –o
rw,async,noatime,rsize=32768,wsize=32768,proto=tcp
ifconfig eth2 192.168.15.24 netmask 255.255.255.0
18. Docker on OSX
VirtualBox
Boot2docker (Tiny Core Linux)
Docker daemon
Agent ApplicationCassandra
18 | 21 October 2015
Storage
Success
Cassandra, and applications
can communicate over
network.
Docker accessible from
single host.
Data stored via NFS
One last thing!
/data
192.168.15.24
19. Docker on OSX19 | 21 October 2015
Storage
Memory and CPU
Increase processors
Increase memory
Finally…
Deploy and run!
20. Docker on OSX20 | 21 October 2015
Questions
Questions?
www.future-grid.com.au
@futuregridaus
David Ryan
@oobles
Editor's Notes
Docker requires certain linux features which are not available directly on OSX. Docker on OSX uses VirtualBox and a Tiny Core Linux distribution boot2docker to run docker daemon.
Our test/poc environment uses mac minis to allow for destructive testing. Purchased before our move to docker, we decided to try move our installation to docker on OSX for testing.
Docker-machine is used to create and manage the VirtualBox configuration. As I learned through this.. It is very opinionated about how the environment should be configured.
Make sure the Adapter type is the same as Adapter 1 and 2. Docker-machine will reset Adapter 1 and 2 (during docker-machine env) if you attempt to change them.
Need to login to the virtual machine and configure the the new adapter.
WARNING: VirtualBox bridged networking is a bit flaky. Seems to work best if no other applications are running on the host when started.
For instance, when Cassandra was running on OSX VirtualBox failed to configure the birdged mode adapter.
Docker creates the server cert and key from the ca. I would prefer to pass in the server cert and key, but this doesn’t seem possible in the current version (1.8) of docker-machine.
Tlsverify is required because the host name doesn’t match what is in the certificate. The server cert generated will generally have the ip address 192.168.99.100. As you can’t pass in the server cert and key you’re stuck with this restriction.
For 95% of applications the VirtualBox mappings is probably enough. However, any application using hardlinks will not work correctly due to VirtualBox/VMWare fusion virtual file system.
We used NFS to map the directory from VirtualBox to OSX.