Building and deploying a distributed application with Docker, Mesos and Marathon

Building and
deploying a
distributed
application with
Docker, Mesos and
Marathon
Julia MateoMADRID · NOV 27-28 · 2015
Friday, November 27, 15

MADRID · NOV 27-28 · 2015
WHO AM I
- Java developer consultant and tech lead at Hortis GRC
(Geneva, Switzerland)
- Team jDuchess Swiss
@juliamateodc
@duchessswiss
http://jduchess.ch/

MADRID · NOV 27-28 · 2015
GOALS (1st part of the workshop)
• Present cm-voting, a distributed web application
• Run cm-voting with docker links
• Run cm-voting with docker compose

MADRID · NOV 27-28 · 2015
GOALS (2nd part of the workshop)
• Create a Mesos cluster
• Mesos DNS
• Deploy cm-voting on mesos cluster

MADRID · NOV 27-28 · 2015
IN THE MEANTIME....
- 2 ways of doing this
workshop :
Mode A :
- RECOMMENDED : install
virtualbox (available USB) and
copy workshop virtual box image
Mode B :
- You will need a browser and a Client HTTP
(like Postman Rest client for Chrome)
- Install docker 1.9.1 (Linux) or Docker
toolbox (Windows and Mac). Copy and load
the docker images we will use from the USB
keys
- Then, load them on your host :
docker load -i <path_image_tar_file>

MADRID · NOV 27-28 · 2015
CODE MOTION VOTING
- Go webapp build on Revel
- Voting app for Code Motion
conference

MADRID · NOV 27-28 · 2015
CODE MOTION VOTING

MADRID · NOV 27-28 · 2015
DEPLOY APP ON DOCKER

From https://www.docker.com/whatisdocker/
• Docker is an open platform for developers and
sysadmins to build, ship, and run distributed
applications.
DOCKER

DOCKER
Server
Host OS
Hypervisor
Guest
OS
Guest
OS
Guest
OS
Guest
OS
Libs/ Libs/ Libs/
App 1 App 2 App 3 App 4
Libs/
Server
Host OS
Docker Engine
Libs/Bins Libs/
App 1 App 2 App 3 App 4
c1 c2 c3 c4
VM1 VM2 VM3 VM4

Docker
Images
Docker
Images
libcontainer, Union Filesystem
Centos Ubuntu
Jetty
add App.war
Container 1
OracleDB
Container 2
Read
only
Writable
Writable
Read
only
CONTAINERS AND IMAGES

MADRID · NOV 27-28 · 2015
DEPLOY CODE MOTION VOTING WITH DOCKER
- Let’s start !
- See webapp doc https://github.com/karesti/cm-voting
- Git clone https://github.com/karesti/cm-voting
- If using virtual box image, go to :
/home/codemotion/cm-voting

MADRID · NOV 27-28 · 2015
DEPLOY CODE MOTION VOTING WITH DOCKER
- Follow the instructions in https://github.com/karesti/cm-voting :
- Open your terminal and run the mongo container :
> sudo docker run -i -t -d --name mongo_cmvoting -p 27017:27017 mongo
> cd cm-voting
- Build and run cm-voting:
> sudo docker build -t cm-voting .
> sudo docker run -i -t -p 9000:9000 --link mongo_cmvoting:mongo cm-voting

MADRID · NOV 27-28 · 2015
DOCKER COMPOSE
- Instead of using docker links :
> sudo docker run -i -t -p 9000:9000 --link mongo_cmvoting:mongo cm-voting
- You can use docker compose
> sudo docker-compose up

MADRID · NOV 27-28 · 2015
DEPLOY WEB APP ON MESOS

• Abstraction of cluster resources
• Share resources across multiple frameworks (versions of the same fwk)
• Resource fair sharing : alternative to static partitioning
• Data locality
MESOS

ZK
Master Master
Executor Executor Executor
Executor ExecutorExecutor
Slaves
MESOS

http://mesos.apache.org/documentation/latest/mesos-architecture/
MESOS

• Marathon is a Mesos framework written in Scala
• Provides easy deployment of Docker containers
• Manages of long running apps
• Rest API for developers
MESOS

ZK
Master Master
Executor ExecutorExecutor
Slaves
Marathon
MESOS

MADRID · NOV 27-28 · 2015
LET’S CREATE A MESOS CLUSTER !
- Create a mesos cluster in localhost
- Docker binding
- Using different ports

MADRID · NOV 27-28 · 2015
DOCKER VOLUMES AND DOCKER BINDING
Host
/home/codemotion/mongo/data
Docker container
/data/db
Host
Docker container
/var/run/docker.sock
/var/run/docker.sock
Docker volume : directory Docker volume : file

MADRID · NOV 27-28 · 2015
CREATE MESOS CLUSTER
- If you are using Docker toolkit : set HOST_IP env variable with your Docker host (in
my case, 10.0.2.15)
export HOST_IP=10.0.2.15
- Launch zookeeper container
sudo docker run -d -e SERVER_ID=1 -p 2181:2181 zookeeper
- Launch mesos master container
sudo docker run -d -p 5050:5050 -e "MESOS_HOSTNAME=${HOST_IP}" -e "MESOS_IP=${HOST_IP}" -e
"MESOS_QUORUM=1" -e "MESOS_ZK=zk://${HOST_IP}:2181/mesos" --name mesos-master -e
"MESOS_LOG_DIR=/var/log/mesos" --net host --restart always mesoscloud/mesos-master:0.23.0-centos-7

MADRID · NOV 27-28 · 2015
- Launch marathon
sudo docker run -d -e "MARATHON_HOSTNAME=${HOST_IP}" -e "MARATHON_HTTPS_ADDRESS=$
{HOST_IP}" -e "MARATHON_HTTP_ADDRESS=${HOST_IP}" -e "MARATHON_MASTER=zk://${HOST_IP}:
2181/mesos" -e "MARATHON_ZK=zk://${HOST_IP}:2181/marathon" --name marathon --net host --restart
always mesoscloud/marathon:0.10.0-centos-7

MADRID · NOV 27-28 · 2015
- Launch mesos slave 1
sudo docker run -p 5051:5051 -d -e "MESOS_HOSTNAME=${HOST_IP}" -e "MESOS_IP=${HOST_IP}" -e
"MESOS_MASTER=zk://${HOST_IP}:2181/mesos" -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/docker.sock:/var/run/
docker.sock --name slave1 --net host --privileged --restart always mesoscloud/mesos-slave:0.23.0-centos-7
sudo docker run -p 5052:5052 -d -e "MESOS_HOSTNAME=${HOST_IP}" -e MESOS_PORT=5052 -e "MESOS_IP=$
{HOST_IP}" -e "MESOS_MASTER=zk://${HOST_IP}:2181/mesos" -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/
docker.sock:/var/run/docker.sock --name slave2 --net host --privileged --restart always mesoscloud/mesos-slave:
0.23.0-centos-7
sudo docker run -p 5053:5053 -d -e "MESOS_HOSTNAME=${HOST_IP}" -e MESOS_PORT=5053 -e "MESOS_IP=$
{HOST_IP}" -e "MESOS_MASTER=zk://${HOST_IP}:2181/mesos" -v /sys/fs/cgroup:/sys/fs/cgroup -v /var/run/
docker.sock:/var/run/docker.sock --name slave3 --net host --privileged --restart always mesoscloud/mesos-slave:
0.23.0-centos-7

MADRID · NOV 27-28 · 2015
TEST THAT EVERYTHING WORKS OK
- Portal Mesos : http://<HOST_IP>:5050

MADRID · NOV 27-28 · 2015
TEST THAT EVERYTHING WORKS OK
- Portal Marathon : http://<HOST_IP>:8080

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY
cm-voting
Node 1 Node 2

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY
cm-voting
Node 1 Node 2
IP ?
PORT ?
CREDENTIALS
??

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY
cm-voting
Mesos
slave 1
Mesos
slave 2
REPLICATION,
LOAD BALANCING...

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY
cm-voting
Node 1 Node 2
... AND FAILOVER

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY : MESOS DNS
https://github.com/mesosphere/mesos-dns

MADRID · NOV 27-28 · 2015
Build Mesos DNS Image
- Git clone : https://github.com/jmateo/mesosdns
- cd mesosdns
- Edit config.json :
sed -i -e “s/HOST_IP/$HOST_IP/g” conﬁg.json
- Build the image :
docker build -t mesosdns .

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY WITH MESOS DNS
1. Launch Mesos DNS container with Marathon
2. Launch Mongo container
3. Connect to one of the Mesos slaves and look for the
mongo service (you can use dig command for doing this)
4. Modify the class connection.go to connect dynamically
cm-voting to mongo using service discovery (see
MesosDNS REST API)

MADRID · NOV 27-28 · 2015
LAUNCH MESOS DNS WITH MARATHON
HTTP POST : http://<HOST_IP>:8080/v2/apps?force=true

MADRID · NOV 27-28 · 2015
LAUNCH MESOS DNS WITH MARATHON
{
"id": "mesos-dns",
"cmd": "/mesos-dns",
"cpus": 0.5,
"mem": 500,
"container": {
"type": "DOCKER",
"docker": {
"image": "mesosdns",
"network": "HOST"
}
}
}

MADRID · NOV 27-28 · 2015
LAUNCH MONGO CONTAINER WITH MARATHON

MADRID · NOV 27-28 · 2015
LAUNCH MONGO CONTAINER WITH MARATHON
{
"id": "mongo",
"cmd": "cat /etc/resolv.conf > /tmp/temp && echo "nameserver 10.0.2.15" > /etc/resolv.conf && cat /tmp/temp >> /etc/resolv.conf && mongod",
"cpus": 0.5,
"mem": 500.0,
"container": {
"type": "DOCKER",
"docker": {
"image": "mongo",
"privileged": true,
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 27017, "hostPort": 0 }
]
}
}
}

MADRID · NOV 27-28 · 2015
LAUNCH CMVOTING WITH MARATHON
{
"id": "cmvoting",
"cmd": "cat /etc/resolv.conf > /tmp/temp && echo "nameserver 10.0.2.15" > /etc/resolv.conf && cat /tmp/temp
>> /etc/resolv.conf && revel run github.com/karesti/cm-voting prod 9000",
"cpus": 0.5,
"mem": 500.0,
"container": {
"type": "DOCKER",
"docker": {
"image": "cm-voting",
"network": "BRIDGE",
"portMappings": [
{ "containerPort": 9000, "hostPort": 0 }
]
}
},
"env": {
! "SERVICE_NAME": "mongo"
}
}

MADRID · NOV 27-28 · 2015
SERVICE DISCOVERY WITH MESOS DNS
4. Test it :

Building and deploying a distributed application with Docker, Mesos and Marathon

More Related Content

What's hot

Viewers also liked

Similar to Building and deploying a distributed application with Docker, Mesos and Marathon

Recently uploaded

Building and deploying a distributed application with Docker, Mesos and Marathon