Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Ansible benelux meetup - Amsterdam 27-5-2015

705 views

Published on

I'm talking about how Ansible helps Backbase establish testing pipeline to ensure the quality of Customer Experience Platform - the leading horizontal portal software. This is done by utilizing the concept of immutable infrastructure to provision on-demand infrastructure use it and the dispose.

Published in: Software
  • Be the first to comment

Ansible benelux meetup - Amsterdam 27-5-2015

  1. 1. How Ansible helps Backbase Ansible Benelux meetup Pavel Chunyayev Amsterdam, 27-5-2015
  2. 2. Who am I • Come from Ukraine • 11 years in IT • Worked in Ukraine, Estonia and the Netherlands • Continuous Delivery architect at Levi9 IT Services • Last 6 months - Automation architect at Backbase
  3. 3. Backbase CXP
  4. 4. Backbase Customer Experience Platform • Core services • Content services • Publication services • 3 environments – Editorial, Staging, Live
  5. 5. Different configuration options • Java version • Application Server • RDBMS • HTTP/HTTPS • Internal configuration options • Optional application features
  6. 6. A lot of things are already automated • There are servers for released CXP version • With different configurations • They can be started/stopped when needed • Newest version of the application needs to be deployed. • In most cases manually. • For some configurations deployment required repackaging of the application. • Automated through maven • There is a sandbox environment with the nightly build • Deployed automatically • Far from production setup
  7. 7. Handcrafted servers • Hard to maintain • Very time/cost sensitive • Setup is not easily reproducible • May be buggy • It should take less time to rebuild a server from the scratch than to log in and fix/update it.
  8. 8. Solution
  9. 9. Solution diagram
  10. 10. Why Ansible • Python powered • No master, agentless • Free, open source • Plenty of modules (batteries included) • Great EC2 support • Windows support (kind of) • Parallel, but controllable execution • Quite simple for developer to understand
  11. 11. Why REST service • Create infrastructure easily • Just send JSON formatted configuration • Service will analyze it and trigger Ansible run • Service is the single point of contact for any infrastructure requests • Can be integrated into any CI, script, application or other service
  12. 12. Why UI • Everyone needs to create an environment from time to time • Opening a ticket and then waiting is not an option • In most situations environments are required for a short period of time. • Self-service
  13. 13. Demo • Directory structure • Flow of work • Decision tree
  14. 14. Ansible features we are using • Handlers • Variables • Jinja2 templates • Facts • Conditions • Playbook includes • Inventory (fake) – hostgroups! • Roles :(
  15. 15. ec2 ec2: key_name: '{{ keypair }}' group_id: '{{ security_group }}' instance_type: '{{ instance }}' image: '{{ image }}' region: '{{ region }}' vpc_subnet_id: '{{ subnet }}' user_data: "{{ item }}" instance_profile_name: 'access-to-s3' instance_tags: origin: "{{ origin }}" environment_name: "{{ environment_name }}" stack_id: "{{ stack_id }}" owner_id: "{{ owner_id }} " role: "{{ item }}" timestamp: "{{ timestamp }}" with_items: server_roles register: ec2
  16. 16. Facts - name: Set the facts and hostnames hosts: all_hosts connection: ssh gather_facts: True max_fail_percentage: 0 tasks: - name: Gather EC2 facts ec2_facts: - name: Set environment fact set_fact: this_environment="{{ ansible_ec2_user_data }}" - name: Set hostnames hostname: name="{{ environment_name }}-{{ this_environment }}"
  17. 17. route53 route53: command: create zone: backbase.dev private_zone: yes overwrite: yes record: "{{ environment_name }}-{{ item.0 }}.backbase.dev" type: A ttl: 10 value: "{{ item.1.instances[0].private_ip }}" with_together: - server_roles - ec2.results register: r53_result until: r53_result|success retries: 20 delay: "{{ 10 |random }}"
  18. 18. Jinja2 templates {% block portal_db %}{% endblock %} {% if http_or_https == 'http' %} {% set port = http_port %} {% else %} {% set port = https_port %} {% endif %} {% if this_environment == "editorial" %} foundation.environment.editorial=true {% else %} foundation.environment.editorial=false {% endif %} foundation.content.proxy.destination={{ http_or_https }}://{{ environment_name }}-{{ this_environment }}.backbase.dev:{{ port }}/contentservices
  19. 19. wait_for - name: Start WSLC shell: /opt/IBM/Websphere/INIT.websphere start {{ this_environment }} - name: Wait for WSLC to start wait_for: path=“/opt/IBM/Websphere/usr/servers/{{ this_environment }}/logs/console.log” search_regex=“The server {{ this_environment }} is ready to run a smarter planet.” timeout=30" - name: Run the trigger shell: /opt/install/app_start_trigger.sh &> /opt/install/app_start_trigger.log; sleep 2 - name: Wait for all apps to start wait_for: path="/opt/IBM/Websphere/usr/servers/{{ this_environment }}/logs/messages.log" search_regex="SRVE0242I: [portalserver] [/portalserver] [/WEB-INF/index.jsp]: Initialization successful." timeout=600
  20. 20. Recovering from failure - name: Download CXP shell: s3cmd get s3://s3_bucket_here/Backbase_Portal_5.6.0-{{ version }}.zip /opt/install/portal-package-5.6.0-{{ version }}.zip --force 2>&1 | tee /opt/install/direct_loader.log register: cxp_download_sleeper until: cxp_download_sleeper.stdout.find("saved as") != -1 retries: 10 delay: "{{ 10 | random }}"
  21. 21. API • /api/stacks - GET - List stacks available for provisioning • /api/stacks/stack_name - GET - List the stack configuration • /api/environments - GET - List all currently provisioned environments • /api/stacks/stack_name - POST - Provision specified stack • /api/environments/environment_id - DELETE - Destroy environment with specified id • /api/environments/all - DELETE - Destroy all environments
  22. 22. Infrastructure life cycle • Create • Check if the user is valid • Parse the requested configurarion • Generate unique environment name • Trigger Ansible run • Return environment name • Destroy • Check if requested environment exists • Check if the user can destroy this environment • Delete environment and clean everything up (DNS, ELB, etc.)
  23. 23. REST Service demo • Create a set of instances • Destroy them
  24. 24. Current UI :)
  25. 25. Ansible testing • No way to test playbook without applying it • Currently there’s a quick sanity test suite • We do testing every commit for a selected number of stacks
  26. 26. Demo • Stash • Feature branches • Ansible testing pipeline
  27. 27. Results • 14-40 minutes to provision and fully configure environments • From 1 stack to 10 stacks automated testing (~25 soon) • We continuously improve to make a robust process
  28. 28. Continuous Delivery without Production
  29. 29. Goals for Continuous Delivery • Create a repeatable and robust process • Treat all configurations identically • Some are more important of course • Provide feedback as soon as possible • For now – in the morning • Provide feedback for feature branches • Release tested artifacts more frequently • For now – every iteration
  30. 30. Stages for Continuous Delivery • Components are built • Unit and integration tests • Main application is build and packaged • Published to Artifactory and s3 • Testing pipeline is triggered • Environments are created • Sanity tests, API tests, Functional tests, etc. are run • Notification is sent in case of any test failure • Environments are disposed
  31. 31. Continuous Delivery diagram Build components Package Provision Stack1 API Tests E2E Tests … Dispose Provision Stack 2 API Tests E2E Tests … Dispose Provision Stack 3 API Tests E2E Tests … Dispose … … … … … Provision Stack 10 API Tests E2E Tests … Dispose
  32. 32. More pipelines… • Performance tests pipeline • Feature branches • Security tests • Earlier versions of the applications • Bugfix releases • More applications
  33. 33. Achievements • Huge quality improvements • Numerous bugs were found in the ‘rare’ stacks • Regressions are found during the night • Minimum cycle time is 1h 30m, maximum – 2h 30m • Dozens environments are created every day • Repeatable process allows to identify instabilities in tests and configurations
  34. 34. Closing thoughts
  35. 35. Zero downtime deployments
  36. 36. Future • Asynchronous provisioning • Ansible roles? • Optimization (time) • Pre-baked images • Docker containers • Plugins to help our specific needs
  37. 37. Ansible v2 • Blocks • begin • rescue • always • Execution strategy – linear vs free (or anything else) • Execution time include evaluation (with*) • Better variable management
  38. 38. Key takeaways • Use Ansible – it’s a great tool :) • Think about immutable infrastructure • Create repeatable and reliable process for releasing software • Build quality in • Improve continuously pavel@levi9.com @PavelChunyayev Any questions?

×