Slides from a presentation at Ansiblefest in London. An intro to provisioning and managing a mesos cluster in a cloud agnostic way.

2. Who are we?
Jim Vanns
Aaron Carey
Production Engineers at ILM London

3. What we’re covering
● Deploying a Mesos cluster in the cloud using
Ansible
● Running services on Mesos using Ansible
● Differences between AWS/GCE modules
● Tips we’ve learnt along the way

4. Step 1:

5. Ansible Dockerfile (example follows)
Fix the ansible version
Install a custom dynamic inventory file (or two)
Consider your credentials and security
Install (bake-in a ‘git archive’) your playbooks

6. Concise Ansible Dockerfile
FROM ubuntu:14.04
RUN apt-get update && apt-get -y install python python-pip python-dev
RUN pip install -U ansible==1.9.4 boto apache-libcloud httplib2
ADD ansible.cfg /etc/ansible/ansible.cfg
ADD ec2.py /etc/ansible/ec2.py
ADD ec2.ini /etc/ansible/ec2.ini
ADD gce.py /etc/ansible/gce.py
ADD gce.ini /etc/ansible/gce.ini
WORKDIR /srv/ansible
ADD ansible .

7. Step 2: Deploy cloud hosts

8. Requirements
Cloud Agnostic (Work with both AWS and GCE)
Split provisioning and bootstrapping into roles
Need a way to determine which are new hosts
Need a way to group hosts during and after provisioning
Tagging!

9. Cloud Start Role
# Launch the primary (master/leader) nameserver
- name: Launch a bootstrap consul server
hosts: localhost
connection: local
gather_facts: False
vars:
cloud_provider: ec2
count: 3
service_name: Consul
launch_group: tag_consul_server_{{ consul_domain }}
tags:
consul: server_{{ consul_domain }}
roles:
- cloud_start

10. Cloud Start Role
---
- include: "{{ cloud_provider }}.yml"

11. Cloud Start Role
- name: Launch instance
ec2:
key_name: "{{ key_name }}"
instance_type: "{{ instance_type }}"
image: "{{ image }}"
wait: yes
vpc_subnet_id: "{{ vpc_subnet_id }}"
group_id: "{{ security_group_id }}"
region: "{{ region }}"
instance_tags:
ansible: base
count: "{{ count }}"
assign_public_ip: "{{ public_ip }}"
register: ec2
- name: Add instances to launched group
add_host: hostname={{ item.private_ip }} groupname=launched_cloud_default
with_items: ec2.instances
- name: Add instances to parameter group
add_host: hostname={{ item.private_ip }} groupname={{ launch_group }}
with_items: ec2.instances

12. Cloud Start Role
- name: Tag instances
ec2_tag: resource={{item.id}} state=present region={{region}}
with_items: ec2.instances
args:
tags:
"{{ tags }}"
- name: Wait for sshd to come up
wait_for: host={{ item.private_ip }}
port=22
state=started
delay=60
timeout=300
connect_timeout=2
with_items: ec2.instances

13. Taking it further...
Parallelise the provisioning
Take advantage of host groups where you can
Use ansible environment variables to target hosts and groups

15. AWS vs GCE
AWS - Tags are key-value pairs
GCE - Tags are string labels, Metadata are key-value pairs
We used a custom (slightly modified) GCE inventorytags = node.extra['tags']
for t in tags:
tag = 'tag_%s' % t
if groups.has_key(tag): groups[tag].append(name)
else: groups[tag] = [name]
if 'items' in node.extra['metadata']:
for item in node.extra['metadata']['items']:
tag = 'tag_%s_%s' % (item['key'], item['value'])
if groups.has_key(tag): groups[tag].append(name)
else: groups[tag] = [name]

16. Step 3: Mesos

17. Our Mesos Architecture
Zookeeper*
Mesos Master*
Mesos Agents*
Consul
Marathon (The scheduler’s scheduler)
Mesos-consul

18. Deploying Zookeeper
- name: Set zookeeper ID facts
gather_facts: True
hosts: tag_zookeeper_server_{{ consul_domain }}
user: ilm-user
tasks:
- set_fact: zkid={{ item.0 | int + 1 }}
when: hostvars[item.1]['ansible_hostname'] == ansible_hostname
with_indexed_items: groups['tag_zookeeper_server_{{ consul_domain }}']
- name: Apply zookeeper role
gather_facts: True
hosts: tag_zookeeper_server_{{ consul_domain }}
sudo: True
user: ilm-user
roles:
- zookeeper

19. Zookeeper Role
- name: Register zookeeper name with consul
uri: >
url=http://127.0.0.1:8500/v1/agent/service/register
HEADER_Content-Type=application/json
method=PUT
body_format=json
body='{
"Name": "zookeeper",
"Tags": [
"zookeeper",
"{{ zkid }}"
],
"Port": 2181
}'
- name: Register individual zookeeper node with consul
uri: >
url=http://127.0.0.1:8500/v1/agent/service/register
method=PUT
body_format=json
HEADER_Content-Type=application/json
body='{
"Name": "zookeeper-{{ zkid }}",
"Tags": [
"zookeeper",
"{{ zkid }}"
],
"Port": 2181
}'

20. Zookeeper Role
- name: Run zookeeper container
docker:
name: zookeeper
image: "mesoscloud/zookeeper:3.4.6-ubuntu-14.04"
state: started
net: host
restart_policy: always
volumes:
- /mnt/data/log:/tmp/zookeeper
env:
MYID: "{{ zkid }}"
SERVERS: "zookeeper-1,zookeeper-2,zookeeper-3"

21. Mesos Containers
- name: Run mesos-master container
docker:
name: mesos-master
image: "mesosphere/mesos-master:{{ img_version }}"
state: started
net: host
restart_policy: always
volumes:
- /mnt/data/log:/var/log
env:
MESOS_ZK: "zk://zookeeper:2181/mesos"
MESOS_CLUSTER: "{{ cluster_name }}"
MESOS_QUORUM: "1"
MESOS_LOG_DIR: "/var/log/mesos"
MESOS_WORK_DIR: "/var/lib/mesos"
MESOS_HOSTNAME: "mesos-master"

22. Submit tasks to Marathon
- name: Launch docker-registry
hosts: "tag_build_docker_{{ consul_domain }}"
gather_facts: False
tasks:
- name: Submit docker-registry job to marathon
uri: >
url=http://marathon:8080/v2/apps
HEADER_Content-Type=application/json
method=POST
status_code=200,201,409
body_format=json
body='{
"args": [ ],
"container": {
"type": "DOCKER",
"docker": {
"network": "HOST",
"image": "registry:2.2",
"forcePullImage": true,
"parameters":
[
{ "key": "env", "value": "REGISTRY_STORAGE=s3" }
]
}
},
"id": "docker-registry",
"instances": 1,
"cpus": 2,
"mem": 4096
}'

23. Done