Successfully reported this slideshow.

CloudOpen 2014 - Extending Cloud Automation, When OpenStack Meets Ansible

1,010 views

Published on

This presentation provides an introduction to Ansible OpenSource IT automation and Cloud orchestration engine. It comes with an overview of what Ansible is and how it can compete (and outcome) legacy automation systems such as Chef or Puppet. Based on real-life experience, Ansible tips & tricks will be provided as how to automate your servers' deployment ever more easily. Additional software will be presented, allowing people to describe their full Cloud application topology, getting the best out of OpenStack APIs and Ansible Playbooks to fully automatize one's application's deployment and continuous integration.

Published in: Technology
  • Be the first to comment

CloudOpen 2014 - Extending Cloud Automation, When OpenStack Meets Ansible

  1. 1. Extending Cloud Automation: When OpenStack Meets Ansible Benjamin Zores, Alcatel-Lucent Enterprise CloudOpen 2014 – 14th October 2014 – Dusseldorf, Germany 1 COPYRIGHT © 2012 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  2. 2. Extending Cloud Automation: When OpenStack Meets Ansible A Few Things About Me … 2 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. ALCATEL LUCENT ENTERPRISE TECHNICAL DIRECTOR, PERSONAL CLOUD SOLUTIONS • Cloud Architecture and Application/Infrastructure Design • R&D Development, Automation, Operations … OPEN SOURCE PROJECT FOUNDER, LEADER AND/OR CONTRIBUTOR FOR: • OpenBricks & GeeXboX : Embedded Linux cross-build tool and distribution. • uShare UPnP A/V & DLNA Media Server, FFMpeg, MPlayer … LINUX FOUNDATION CONFERENCES REGULAR LINUX FOUNDATION’S EVENTS SPEAKER • Various talks on: • Linux Embedded Systems at Embedded Linux Conference (Europe) • Android Architecture and Device Porting at Android Builder Summit GNU/LINUX MAGAZINE FRANCE RECURRENT TECHNICAL WRITER • Various publications on: • Android Architecture Internals • Cloud (OpenStack, Ansible …)
  3. 3. Extending Cloud Automation: When OpenStack Meets Ansible Self-Promotion Time ! Android 4: Fondements Internes Benjamin Zores, Ed. Diamond – Sept. 2014 3 Series of articles published in GNU/Linux Magazine France COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  4. 4. 4 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  5. 5. Extending Cloud Automation: When OpenStack Meets Ansible Why we’re here ? 5 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. A Good Developer Is A Lazy Developer ! Don’t do over and over again things that someone else can do for you. (especially if that someone is a bot)
  6. 6. Extending Cloud Automation: When OpenStack Meets Ansible How one feels after deploying its servers through shell scripts … #!/bin/sh echo “net.core.rmem_default=16384" | sudo tee -a /etc/sysctl.conf echo “deb http://nwps.ws/pub/mariadb/repo/5.5/debian wheezy main” | sudo tee –a /etc/apt/conf.d/mariadb.conf sudo apt-get -y install mariadb-server […] 6 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. AWESOME !!
  7. 7. Extending Cloud Automation: When OpenStack Meets Ansible Shell Scripts + Money = ? ( © John Lynch, http://goo.gl/gkmKGN ) “Model-driven orchestration frameworks for complex infrastructure management and automation” 7 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  8. 8. 8 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  9. 9. Extending Cloud Automation: When OpenStack Meets Ansible State of the Art 9 Original Rockstars ! - Great Tools - Field Pioneers But just incredibly complex to start with, even for simple cases. COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Not everyone is managing 500 servers in the cloud after all …
  10. 10. 10 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  11. 11. Extending Cloud Automation: When OpenStack Meets Ansible Introducing Ansible … - SW to manage and configure computers. - Python + Jinja2 + YAML + SSH (that’s it !). - Manages nodes over SSH. - Does not require additional remote dependencies. - First Release: February 20th 2012. 11 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. A fictional machine capable of instantaneous or superluminal communication (© Wikipedia). Design Goals: 1. Minimal in nature: Python based with no dependencies on the environment. 2. Consistent. 3. Secure: relies on OpenSSH only, with no vulnerable remote agents. 4. Highly-Reliable: N re-deployments provide the same result. 5. Low Learning Curve.
  12. 12. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Architecture 12 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  13. 13. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Default Model: Push to Nodes 13 © Julien Ponge, http://goo.gl/CB5f8a COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  14. 14. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Alternative Model: Pull From Server 14 © Julien Ponge, http://goo.gl/CB5f8a COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  15. 15. Extending Cloud Automation: When OpenStack Meets Ansible Introduction to Ansible Ansible Inventory File 15 Module Arguments Name COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. [lan1] 192.168.0.1 192.168.0.2 Module # ansible -i inventory.txt lan1 -m shell -a "/bin/echo Hello World" 192.168.0.1 | success | rc=0 >> Hello World 192.168.0.2 | success | rc=0 >> Hello World # ansible -i inventory.txt lan1 -u ben --sudo -m shell -a "/bin/echo Hello World"
  16. 16. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Inventory 16 With support for wildcards And per-host tuning variables. COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. [europe] foo.domain.com [usa] 192.168.0.1 [world:children] europe usa [webservers] www[01:50].domain.com [databases] db-[a:f].domain.com Hosts can be described by FQDN or IP With support for infinite depth inheritance [targets] localhost ansible_connection=local web1.domain.com ansible_connection=ssh ansible_ssh_user=user1 web2.domain.com ansible_connection=ssh ansible_ssh_user=user2
  17. 17. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Variables 17 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. [lan1] 192.168.0.1 msg="Hey !" 192.168.0.2 msg= "What’s up ?" # ansible -i inventory.txt lan1 -m shell -a "/bin/echo {{msg}}" 192.168.0.1 | success | rc=0 >> Hey ! 192.168.0.2 | success | rc=0 >> What’s up ? Inventory.txt: [europe:vars] domain=my.domain.eu [usa:vars] domain=my.domain.com ./group_vars/europe : -- domain: my.domain.eu ./group_vars/usa : -- domain: my.domain.com YAML File Format
  18. 18. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Modules (235 in Ansible v1.7.1) # ansible-doc -l acl Sets and retrieves file ACL information. add_host Add a host (and alternatively a group) to the ansible-playbook alternatives Manages alternative programs for common commands apache2_module Enables/disables a module of the Apache2 webserver apt Manages apt-packages apt_key Add or remove an apt key apt_repository Add and remove APT repositories [...] xattr Set/retrieve extended attributes yum Manages packages with the `yum' package manager zfs Manage zfs zypper Manage packages on SuSE and openSuSE 18 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  19. 19. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Module How-To # ansible-doc shell > SHELL The [shell] module takes the command name followed by a list of space-delimited arguments. It is almost exactly like the [command] module but runs the command through a shell (`/bin/sh') on the remote node. Options (= is mandatory): - chdir cd into this directory before running the command - creates a filename, when it already exists, this step will *not* be run. - executable change the shell used to execute the command. Should be an absolute path to the executable. = free_form The shell module takes a free form command to run, as a string. There's not an actual option named "free form". See the examples! - removes a filename, when it does not exist, this step will *not* be run. Notes: If you want to execute a command securely and predictably, it may be better to use the [command] module instead. Best practices when writing playbooks will follow the trend of using [command] unless [shell] is explicitly required. When running ad-hoc commands, use your best judgement. # Execute the command in remote shell; stdout goes to the specified # file on the remote - shell: somescript.sh >> somelog.txt 19 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  20. 20. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Playbooks 20 As appealing as they are, they’ve got nothing to do COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. with this !
  21. 21. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Playbooks 21 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. --- - hosts: lan1 remote_user: ben sudo: no vars: name: ”My First Playbook” tasks: - name: verify servers activity ping: - name: say hello shell: /bin/echo "{{name}} {{msg}}” notify: - we are done handlers: - name: we are done shell: /bin/echo ”That’s it !" YAML File Format
  22. 22. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Playbooks # ansible-playbook -i inventory.txt playbook.yml -v PLAY [lan1] *********************************************************** GATHERING FACTS ***************************************************** ok: [192.168.0.1] ok: [192.168.0.2] TASK: [verify servers activity] ********************************************* ok: [192.168.0.1] => {"changed": false, "ping": "pong”} ok: [192.168.0.2] => {"changed": false, "ping": "pong"} TASK: [say hello] ******************************************************* changed: [192.168.0.1] => {"changed": true, "cmd": "/bin/echo ”My First Playbook Hey !" ", "delta": "0:00:00.005264", "end": "2014-07-06 16:42:54.115860", "rc": 0, "start": "2014-07-06 16:42:54.110596", "stderr": "", "stdout": ”My First Playbook Hey !"} changed: [192.168.0.2] => {"changed": true, "cmd": "/bin/echo ”My First Playbook What’s up ?" ", "delta": "0:00:00.002732", "end": "2014-07-06 16:42:54.078013", "rc": 0, "start": "2014-07-06 16:42:54.075281", "stderr": "", "stdout": ”My First Playbook What’s up ?"} […] 22 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  23. 23. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Playbooks NOTIFIED: [we are done] **************************************** changed: [192.168.0.1] => {"changed": true, "cmd": "/bin/echo "That’s it !" ", "delta": "0:00:00.005559", "end": "2014-07-06 16:42:54.312184", "rc": 0, "start": "2014-07-06 16:42:54.306625", "stderr": "", "stdout": "That’s it !"} changed: [192.168.0.2] => {"changed": true, "cmd": "/bin/echo "That’s it !" ", "delta": "0:00:00.002824", "end": "2014-07-06 16:42:54.306878", "rc": 0, "start": "2014-07-06 16:42:54.304054", "stderr": "", "stdout": "That’s it !"} PLAY RECAP ************************************************** 192.168.0.1 : ok=4 changed=2 unreachable=0 failed=0 192.168.0.2 : ok=4 changed=2 unreachable=0 failed=0 23 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  24. 24. Extending Cloud Automation: When OpenStack Meets Ansible Ansible Playbook 24 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. vars_file: - “vars/common.yml” - “vars/production.yml” tasks: - include: tasks/nginx.yml - include: tasks/php-fpm.yml - include: tasks/mariadb.yml
  25. 25. Extending Cloud Automation: When OpenStack Meets Ansible Example: (Parts of) MariaDB Cluster Automation - hosts: mariadb vars: domain : domain.com hosts_list: mariadb tasks: - include: tasks/hosts.yml 25 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. [mariadb] 192.168.0.1 name=my-db-maria1 192.168.0.2 name=my-db-maria2 192.168.0.3 name=my-db-maria3 tasks/hosts.yml: - name: declare hosts lineinfile: dest=/etc/hosts regexp='.*{{ item }} .*$' line="{{item}} {{ hostvars[item]['name'] }} {{ hostvars[item]['name'] }}.{{domain}}" state=present with_items: groups['{{hosts_list}}'] Targets /etc/hosts: 192.168.0.1 my-db-maria1 my-db-maria1.domain.com 192.168.0.2 my-db-maria2 my-db-maria2.domain.com 192.168.0.3 my-db-maria3 my-db-maria3.domain.com
  26. 26. Extending Cloud Automation: When OpenStack Meets Ansible Example: (Parts of) MariaDB Cluster Automation vars: mariadb_debian_password: my_debian_password mariadb_root_password: my_root_password mariadb_cluster_name: my_cluster mariadb_cluster_list: "{{ groups['mariadb'] }}” mariadb_gcache_size: 4G mariadb_extra_cfg: skip-external-locking: ~ skip-name-resolve: ~ # Force no DNS resolution tasks: - include: tasks/mariadb-galera.yml tasks/mariadb-galera.yml: - name: Add MariaDB APT key apt_key: url=http://keyserver.ubuntu.com/pks/lookup? op=get&fingerprint=on&search=0xcbcb082a1bb943db - name: Add MariaDB APT repository apt_repository: repo='deb http://ftp.igh.cnrs.fr/pub/mariadb/repo/5.5/debian wheezy main' - name: APT pinning for MariaDB action: copy src=files/mariadb.pref dest=/etc/apt/preferences.d/mariadb.pref 26 owner=root group=root mode=0644 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  27. 27. Extending Cloud Automation: When OpenStack Meets Ansible Example: (Parts of) MariaDB Cluster Automation templates/mariadb_secure_installation.j2: tasks/mariadb-galera.yml: - name: install mariadb-galera-server action: apt name=mariadb-galera-server update_cache=yes - name: copy mysql_secure_installation credentials when: mariadb_root_password is defined action: template src=templates/mariadb_secure_installation.j2 dest=/tmp/mariadb_secure_installation owner=root group=root mode=0600 27 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. {{ mariadb_root_password }} {{ mariadb_root_password }}
  28. 28. Extending Cloud Automation: When OpenStack Meets Ansible Example: (Parts of) MariaDB Cluster Automation templates/mariadb.cnf.j2: [mysqld] wsrep_provider=/usr/lib/galera/libgalera_smm.so {% if mariadb_cluster_name is defined %} wsrep_cluster_name='{{ mariadb_cluster_name }}’ {% endif %} {% if mariadb_cluster_list is defined %} wsrep_cluster_address=gcomm:// {{ mariadb_cluster_list[0] }}{% for node in mariadb_cluster_list[1:] %},{{ node }}{% endfor %}{% endif %} tasks/mariadb-galera.yml: - name: write conf.d/mariadb.cnf action: template src=templates/mariadb.cnf.j2 dest=/etc/mysql/conf.d/mariadb.cnf Targets /etc/mysql/mariadb.cnf: wsrep_cluster_name=‘my_cluster’ wsrep_cluster_address=gcomm://192.168.0.1,192.168.0.2,192.168.0.3 28 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Jinja2 Syntax
  29. 29. Extending Cloud Automation: When OpenStack Meets Ansible Example: (Parts of) MariaDB Cluster Automation templates/mariadb.cnf.j2: [mysqld] wsrep_node_address={{ ansible_eth0.ipv4.address }} wsrep_node_name='{{ ansible_hostname }}’ {% if mariadb_extra_cfg is defined %} {% for key, value in mariadb_extra_cfg.iteritems() %} {{ key }}{% if value is not none %}={{ value }}{% endif %} {% endfor %} {% endif %} 29 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. Target /etc/mysql/mariadb.cnf: wsrep_node_address=192.168.0.1 wsrep_node_name=my-db-maria1 skip-external-locking skip-name-resolve
  30. 30. 30 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  31. 31. Extending Cloud Automation: When OpenStack Meets Ansible OpenStack in a Nutshell - #1 (most widely adopted) Open-Source IaaS project. - Awesome REST Management API. - Perfect for instant spawning of new Virtual Machines (VMs) - But VMs yet to be configured ... 31 What if I could connect OpenStack VMs with Ansible for nightly CI ? COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  32. 32. Extending Cloud Automation: When OpenStack Meets Ansible (One of my) Cloud application: OpenTouch TeamShare - Online collaboration tool for SMBs. - Provides multi-projects file storage and sharing for enterprises. - With project management, chat and collaboration capabilities. 32 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  33. 33. Extending Cloud Automation: When OpenStack Meets Ansible (One of my) Cloud application network topology - 2x HAProxy Load-Balancers - 2x NGINX frontal Web Servers - 2x NGINX frontal Web File Servers - 2x PHP Backends - 1x SMTP Server - 3x MariaDB Master-Master Galera Cluster + 2x Galera Arbiters - 2x MongoDB Master-Slave Cluster + 1x MongoDB Arbiter - 3x RabbitMQ Master-Master Clusters - 1x LibreOffice Server - 1x NFS Server Now let’s say that I want to test both my application and my infrastructure every single night for non-regression ! 33 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  34. 34. 34 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  35. 35. Extending Cloud Automation: When OpenStack Meets Ansible AvOID: Ansible OpenStack Instances Deployer - Open Source software by Alcatel-Lucent Enterprise. - Written in Python with dependencies to OpenStack Nova/Neutron APIs. - Relies on Ansible with IaaS bindings: - Currently OpenStack only - But wide open to support many much more … who knows … - Comes as a library with both CLI and Web clients. How it works: 1. Describe your infrastructure topology in a YAML file once and for all. 2. Run avoid-cli. Grab a coffee, that’s it. How it (internally) works: 1. Parses your topology file. 2. Optionally terminates (all) OpenStack VMs and spawn new ones and build Ansible inventory file. 3. Creates VMs dependency graph for parallelized post-configuration by Ansible. 4. Post-configure VMs through Ansible playbooks (continuous-integration style). 35 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  36. 36. Extending Cloud Automation: When OpenStack Meets Ansible AvOID: Ansible OpenStack Instances Deployer Check it out on GitHub: https://github.com/OpenTouch/AvOID 36 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  37. 37. Extending Cloud Automation: When OpenStack Meets Ansible AvOID Topology File - globals: env: dev ssh_key: /path/to/ssh_private_key.pem ssh_user: remote_user os_user: openstack_user os_passwd: openstack_password os_tenant: openstack_tenant os_auth_url: http://my.private.cloud.com:5000/v2.0 os_image: Debian – Wheezy os_network: My OpenStack Tenant Network os_ssh_key: My OpenStack Tenant SSH Key Name ansible_inventory_template: /path/to/ansible/inventory_template.txt ansible_playbooks_directory: /path/to/ansible/playbooks 37 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  38. 38. Extending Cloud Automation: When OpenStack Meets Ansible AvOID Topology File 38 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. nodes: - node: name: web-server-1 flavor: m1.small ansible_config_keys: webserver playbook: webserver additional_network: LAN Network 2, LAN Network 3 security: http, https depends: file-server-1 floating_ips: 1.2.3.4, 5.6.7.8, 10.20.30.40 vips: 10.0.1.2, 10.0.2.2, 10.0.3.2 - node: name: file-server-1 flavor: m3.medium ansible_config_keys: fileserver playbook: fileserver volumes: - { name: web-volume1, size: 1 } - { name: web-volume2, size: 10 }
  39. 39. Extending Cloud Automation: When OpenStack Meets Ansible AvOID command-line # avoid-cli Usage: avoid-cli <topology_file.yml> <command> [opts] where <command> is: - status: list topology status - redeploy <list of playbooks or VM names>: terminate VM, spawn a new one and restart playbook - redeployall: redeploy all nodes - runplaybook <list of playbooks>: restart playbook as it - runallplaybooks: restart all playbooks - geninventory: generate Ansible inventory file based on topology.yml 39 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  40. 40. Extending Cloud Automation: When OpenStack Meets Ansible AvOID Web UI # avoid-web my-topo-dev.yml Read of my-dev.yml done: 32 VMs and 22 playbooks Now go to http://localhost:8888/ ! 40 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  41. 41. 32 VMs fully deployed in 19mn40s (5mn40s for OpenStack and 14mn and for Ansible post-configuration) 41 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  42. 42. 42 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
  43. 43. Extending Cloud Automation: When OpenStack Meets Ansible Let’s keep in touch … 43 COPYRIGHT © 2014 ALCATEL-LUCENT. ALL RIGHTS RESERVED. benjaminzores @gxben #Benjamin Zores

×