this series of slides showing a configuration of an Elasticsearch cluster on AWS following the principles of mass production system getting fully reproducible and scalable environments without any handy configuration.

1. TOWARD MASS
PRODUCTION SYSTEM
DEMO: ELASTICSEARCH CLUSTER ON
AWS
USING ANSIBLE

2. our demo will be on AWS multi AZ……

3. Why Elasticsearch is fit for CM management
tools like Ansible?
Lot of sys adm configuration tips for a cluster
environment
• java settings( jmx, mlockall….)
• sysctl settings( swappiness, max_map,count..)
• ulimit settings
Do I have to change these settings by hand
repeated for n° instance times?
NOTHANKS!

4. As a mention before Ansible has a plenty of
sysadm modules:
- name: firewalld applying conf
firewalld: service=elasticsearch
permanent=true zone=public state=enabled
tags:
- firewall
- name: sysctl configs
sysctl: name=fs.file-max value=64000 state=present
tags:
- sysctl

5. Here’ s come AWS
AWS provides a special plugin for discovery your ES
instances inside your cluster just by
their security group!
discovery.type: ec2
discovery.zen.ping.multicast.enabled: false
discovery.ec2.groups: my_security_group
I don’t have to update the other node -1 configurations
if i need to replace or add a new node!!

7. Create our instances--
- hosts: localhost
connection: local
vars_files:
- host_vars/el.yml
vars:
security_group: elsg
instance_type: t2.medium
image: ami-7cc4f661
region: eu-central-1
keypair: fkft.pem
n_instances: "1"
tasks:
- name: Launch Instance to Frankfurt av 1
ec2:
group: elsg
instance_type: "{{ instance_type }}"
image: ami-7cc4f661
wait: true
region: eu-central-1
keypair: "{{ keypair }}"
vpc_subnet_id: subnet-dfd42aa4
count: "1"
register: ec2
with_items: ec2_instances_fav1

8. …
- name: Launch Instance to Frankfurt av 2
ec2:
group: elsg
instance_type: "{{ instance_type}}"
image: ami-7cc4f661
wait: true
region: eu-central-1
keypair: "{{ keypair }}"
vpc_subnet_id: subnet-5a7ca333
count: "2"
register: ec2
with_items: ec2_instances_fav2
remote_user: centos
gather_facts: True
sudo: false

9. Ansible provides a special plugin to find the running
instances inside your EC2 account…
it’s called dynamic inventory
ansible-playbook -i inventories/dynamic/ec2.py
el-aws_create-instance.yml
./ec2.py —list
"eu-central-1b": [
"52.28.81.48",
"52.28.33.99",
"52.28.49.30"
],

10. Insert some data
let’s try to insert a small dataset(i.e):
{
"account_number": 0,
"balance": 16623,
"firstname": "Bradshaw",
"lastname": "Mckenzie",
"age": 29,
"gender": "F",
"address": "244 Columbus Place",
"employer": "Euron",
"email": "bradshawmckenzie@euron.com",
"city": "Hobucken",
"state": “CO"
}
curl -XPOST 'localhost:9200/bank/account/
_bulk?pretty' --data-binary @accounts.json

11. Let’s see the output

12. What we have achieved?
• a mass production system without handy configuration
• a fully reproducible environment
• scalability
• availability
• exit staff proof
• fully documentated by the code
• reduced stress
……………………………………………………

13. Resources
http://www.ansible.com/home
https://docs.ansible.com/playbooks_vault.html
https://puppetlabs.com/
http://docs.puppetlabs.com/hiera/1/
https://www.elastic.co/
https://github.com/elastic/elasticsearch-cloud-aws
https://github.com/ansible/ansible/blob/devel/plugins/inventory/ec2.py
http://pavelpolyakov.com/2014/08/14/elasticsearch-cluster-on-aws-
part-2-configuring-the-elasticsearch/
http://getidephix.com/

14. Questions???