SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack

949 views

Published on

This talk will demonstrate how to use Salt Mine leveraging Salt grains to create several environments (parallel universes) that decide how to run the same Salt formulas with different outcomes. "Roles” will be defined in an OpenStack Marconi (queuing as a service) deployment and a few formulas will be shared to demonstrate the concept.

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
949
On SlideShare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
19
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

SaltConf14 - Oz Akan, Rackspace - Deploying OpenStack Marconi with SaltStack

  1. 1. Deploying OpenStack Marconi Creating Parallel Universes with SaltStack Oz Akan, Cloud Engineering Manager, Rackspace
  2. 2. Outline • Marconi • Why SaltStack? • Universe | Environment • Salt Concepts • Framework • Summary
  3. 3. Marconi
  4. 4. Marconi Marconi Message Queue
  5. 5. Marconi 6 data centers
  6. 6. Marconi 360servers
  7. 7. Marconi 5 Billion transactions per day
  8. 8. Marconi …but something more remarkable
  9. 9. Marconi load balancers web servers catalog databases queues databases zenoss master zenoss collectors graylog servers elastic search servers bastions usage tracking workers usage tracking databases
  10. 10. Marconi from nothing
  11. 11. Marconi
  12. 12. Marconi 45 minutes from nothing to web scale
  13. 13. Why Salt?
  14. 14. Challenges human mitsakes
  15. 15. Challenges scaleof web
  16. 16. Challenges sc li g dynamism a n
  17. 17. Challenges environments multiple environments
  18. 18. Universe | Environment
  19. 19. Universe | Environment laws | rules defined by
  20. 20. Salt Concepts
  21. 21. Salt Concepts grainon minions
  22. 22. Salt Concepts pillaron master
  23. 23. Salt Concepts environmentmaps to a folder
  24. 24. Salt Concepts directory overlay for states and pillar
  25. 25. Directory Overlay Example file_roots: prod: - /srv/salt/prod - /srv/salt/base
  26. 26. Salt Concepts minequery minions
  27. 27. Salt Concepts map for salt-cloud
  28. 28. Salt Concepts overstate more orchestration
  29. 29. Salt Concepts / overstate example set-mongodb_server: match: 'G@environment_id:marconi-prod-ord and G@roles:mongodb_server' sls: - mongodb_server require: - set-firewall set-mongodb_replica: match: 'G@environment_id:marconi-prod-ord and G@roles:mongodb_server and G@mongodb_role:primary' sls: - mongodb_server.replica require: - set-mongodb_server
  30. 30. Framework
  31. 31. Framework role is many things
  32. 32. Framework / role grain role: web_server
  33. 33. Framework / role formulas if..else in db_server mongodb
  34. 34. Framework / role / formulas example # queues_server/init.sls {% if 'roles' in grains and 'queues_server' in grains['roles'] %} include: - marconi - memcached - queues_server.kernel - queues_server.install {% endif %}
  35. 35. Framework / role minions pillar
  36. 36. Framework / role / minions example # pillar/minions.sls minions: cdb1a-cqp-ord: roles: - mongodb_server attributes: mongodb_replica_set: catalog-rs1 mongodb_role: primary db_type: catalog … web4a-cqp-ord: roles: - queues_server attributes: mongodb_replica_set: catalog-rs1 queues_api: queue
  37. 37. Framework / role devices pillar
  38. 38. Framework / role / devices example # pillar/devices.sls devices: load_balancers: text: 'cloud load balancers' addresses: - 10.183.250.0/23 marconi-endpoint: text: 'marconi ORD endpoint' fqdn: ord.queues.api.rackspacecloud.com protocol: https address: 192.237.142.76 … graylog_lb: text: 'graylog load balancer' fqdn: log.marconi-graylog.com
  39. 39. Framework / role networks pillar
  40. 40. Framework / role / networks example # pillar/networks.sls networks: vpn-all: text: ’vpn networks' addresses: - '10.1.2.3/22' - '10.2.3.4/24’ … salt-master: text: 'salt master servers' addresses: - '10.178.129.47/32' - '162.200.150.120/32'
  41. 41. Framework / role roles pillar sections per formula
  42. 42. Framework / role / pillar example # pillar/roles.sls roles: role: text: attributes: flags: clients: minions: networks: devices
  43. 43. Framework / role / pillar example # pillar/roles.sls roles: mongodb_server: text: 'marconi mongodb database server’ attributes: - mongodb_replica_set - mongodb_role - db_type flags: - mongodb_replica_set_configured
  44. 44. Framework / role / pillar example # pillar/roles.sls roles: mongodb_server: clients: minions: - roles: ['bastion_server'] protocols: - name: tcp ports: ['22'] states: ['NEW','ESTABLISHED'] text: 'ssh access'
  45. 45. Framework / role / pillar example # pillar/roles.sls roles: mongodb_server: clients: minions: - roles: ['queues_server','mongodb_server','memcached_server','bastion_server'] protocols: - name: icmp types: ['0','8'] text: 'ping access'
  46. 46. Framework / role / pillar example # pillar/roles.sls roles: mongodb_server: clients: networks: - name: vpn-all protocols: - name: icmp types: ['0','8'] text: 'ping access from zenoss server'
  47. 47. Framework / role / pillar example # pillar/roles.sls roles: web_server: clients: devices - name: load_balancers protocols: - name: tcp ports: ['443'] text: 'http access from lb to server' states: ['NEW','ESTABLISHED','RELATED']
  48. 48. Framework environment is many things
  49. 49. Framework / environment • project • purpose • location set of grains
  50. 50. Framework / environment environment_id project-purpose-location
  51. 51. Framework / environment /etc/salt/master file_roots, pillar_roots
  52. 52. Framework / environment example file_roots: base: - /srv/salt/marconi/base marconi-prod-lon: - /srv/salt/marconi/prod-lon - /srv/salt/marconi/base marconi-test-lon: - /srv/salt/marconi/test-lon - /srv/salt/marconi/base pillar_roots: base: - /srv/salt/marconi/base/pillar marconi-prod-lon: - /srv/salt/marconi/prod-lon/pillar - /srv/salt/marconi/base/pillar marconi-test-lon: - /srv/salt/marconi/test-lon/pillar - /srv/salt/marconi/base/pillar
  53. 53. Framework / environment example # folder layout root@salt1a:/srv/salt/marconi# ls -1 base prev-ord prod-dfw prod-hkg prod-iad prod-lon prod-ord prod-syd test-ord …
  54. 54. Framework mine in formulas
  55. 55. Framework / mine firewall jinja template
  56. 56. Framework / mine / firewall {%- if 'scope' in minion %} {%- if minion.scope == 'project' %} {%- for key, value in salt['mine.get']('project:' + grains['project'], 'grains.items', expr_form='grain').items() %} {%- if role in value['roles'] %} -A INPUT -s {{ key }} -j {{ role|upper }} … {%- elif minion.scope == 'environment_id' %} {%- for key, value in salt['mine.get']('environment_id:' + grains['environment_id'], 'grains.items', expr_form='grain').items() %} {%- if role in value['roles'] %} -A INPUT -s {{ key }} -j {{ role|upper }} …
  57. 57. Framework / mine hosts jinja template
  58. 58. Framework / mine / hosts {%- for key, value in salt['mine.get']('environment_id:' + grains['environment_id'], 'grains.items', expr_form='grain').items() %} {{ value['id'] }}: host: - present - ip: {{ salt['mine.get'](value['id'], 'network.ip_addrs').values()[0][0] }} {%- endfor %}
  59. 59. Framework / mine / zenoss hosts {%- if 'roles' in grains and 'zenoss_server' in grains['roles'] %} {%- for key, value in salt['mine.get']('roles:zenoss_server', 'grains.items', expr_form='grain').items() %} {%- if value['project'] == pillar['project'] %} host_{{ value['id'] }}: host: - present - name: {{ value['id'] }} - ip: {{ salt['mine.get'](value['id'], 'network.ip_addrs').values()[0][1] }} {%- endif %} {%- endfor %} {%- endif %}
  60. 60. Summary • grains • mark minions (project, purpose, location, role) • pillar • to define global rules per role • salt-mine • to be able to query minions in the environment • environments and directory overlay
  61. 61. Multiple Environments Q&A

×