Cloud party 2014 - Deploy your infrastructure with Saltstack - Salt Cloud with OpenStack

2,222 views
1,995 views

Published on

Deploy in minutes your web infrastructure on OpenStack with Saltstack and Salt Cloud extension!

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,222
On SlideShare
0
From Embeds
0
Number of Embeds
358
Actions
Shares
0
Downloads
20
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Cloud party 2014 - Deploy your infrastructure with Saltstack - Salt Cloud with OpenStack

  1. 1. CloudParty 2014 Application and Infrastructure deployment on Cloud Providers by / - www.corley.it Walter Dal Mut @walterdalmut Corley S.r.l.
  2. 2. The Problem Your web application never scale
  3. 3. It is really scalable?
  4. 4. Our daily goal?
  5. 5. A simple scalable web app
  6. 6. Why so many layers? Manage every layer by it-self Optimize every layer by it-self Scale every layer by it-self Monitor every layer by it-self Securize every layer by it-self ...
  7. 7. The Cloud Add more resources when we need in seconds Remove resources when we don't need them anymore Reduce time to market Turn a fixed cost into a variable costs pay only for what you use
  8. 8. Deploy & Orchestation Automate your infrastructure and deployment
  9. 9. Distributed Application
  10. 10. Split out your Infrastructure Networks Subnetworks
  11. 11. Security Groups We will assign a different security-group for every group of VMs. In that way we can apply our security policy in a simple and powerful way.
  12. 12. Security Map Database layer (MySQL) PORT 3306 <- from web layer Cache layer (Memcached) PORT 11211 <- from web layer Web layer (Apache2) PORT 80 <- from proxy layer Proxy layer (Nginx) PORT 80 <- from everywhere
  13. 13. Security Groups
  14. 14. We will use Salt-Cloud It means that we also need to allow SSH connections from the "master" Every "minion" has also another security-group "salt-minion" that allows SSH connections from the "salt-master" instance
  15. 15. Salt TOP.SLS base:     '*':         ‐ base dev:     ... prod:     'proxy.milan.enter.*.prod':         ‐ nginx     'web.milan.enter.*.prod':         ‐ webserver         ‐ webapp     'cache.milan.enter.*.prod':         ‐ memcached     'rdb.milan.enter.*':         ‐ mysql         ‐ mysql.master     'srdb.milan.enter.*':         ‐ mysql
  16. 16. What about "salt-cloud" Salt cloud is made to integrate Salt into cloud providers in a clean way so that minions on public cloud systems can be quickly and easily modeled and provisioned. http://salt-cloud.readthedocs.org/en/latest/
  17. 17. salt-cloud for OpenStack We need a Provider definition enter‐openstack‐config:   minion:     master: 111.111.111.111   identity_url: https://api‐legacy.entercloudsuite.com:5000/v2.0/tokens   compute_name: nova   protocol: ipv4   compute_region: ItalyMilano1   user: name@user.tld   password: YourPassword   tenant: name@user.tld   provider: openstack
  18. 18. salt-cloud for OpenStack We need VMs profiles rdb:     provider: enter‐openstack‐config     size: e1standard.x4     image: GNU/Linux Ubuntu Server 12.04 LTS Precise Pangolin x64     ssh_username: ubuntu     ssh_key_file: /root/private‐key.pem     ssh_key_name: 'private‐key‐name'     ssh_interface: public_ips     security_groups: salt‐minion,mysql     networks:         ‐ fixed:             ‐ xxxxxxxx‐xxxx‐xxxx‐xxxx‐xxxxxxxxxxxx web:     ...
  19. 19. Automatic IP management Proxies need Web instaces IPs and so on...
  20. 20. Enable Peer communication allow Salt minions to pass commands to each other peer:   .*:     ‐ .* We will use this features to share IP addresses You can use "grains" or "mines" instead
  21. 21. Let's go
  22. 22. Database layer Create a new resource with salt-cloud salt‐cloud ‐p PROFILE VM‐NAME salt‐cloud ‐p rdb rdb.milan.enter.1.prod
  23. 23. Deploy the Master RDB salt 'rdb.*' state.highstate
  24. 24. Create a group of slaves Can we paralelize all VM creation? salt‐cloud ‐Pp PROFILE VM‐NAME VM‐NAME ... "-P" option means "parallel" salt‐cloud ‐Pp srdb      srdb.milan.enter.1.prod      srdb.milan.enter.2.prod      srdb.milan.enter.3.prod
  25. 25. Deploy Slave RDB salt 'srdb.*' state.highstate
  26. 26. Prepare all databases Now we have a Master instance and 3 slaves We have to prepare Read-Replicas CHANGE MASTER TO     MASTER_HOST='xxx.xxx.xxx.xxx',     MASTER_USER='repl‐user',     MASTER_PASSWORD='repl‐pass',     MASTER_LOG_FILE='mysql‐bin‐xxxxx',     MASTER_LOG_POS=xxx
  27. 27. Execute MySQL commands salt 'rdb.milan.enter.1.prod' mysql.query mysql 'show master status' salt 'srdb.*' mysql.query mysql '     CHANGE MASTER TO     MASTER_HOST="xxx.xxx.xxx.xxx",     MASTER_USER="repl‐user",     MASTER_PASSWORD="repl‐pass",     MASTER_LOG_FILE="mysql‐bin‐xxxxx",     MASTER_LOG_POS=xxx     '
  28. 28. Now we have our databases
  29. 29. Now the cache layer
  30. 30. Add cache resources salt‐cloud ‐Pp cache    cache.milan.enter.1.prod    cache.milan.enter.2.prod    cache.milan.enter.3.prod    cache.milan.enter.4.prod salt 'cache.*' state.highstate
  31. 31. Memcached will help us with caching and session management
  32. 32. When we distribute the load across a group of VMs all information should be available to the group otherwise we have connectivity problems Cache warm up, disconnected users, and more...
  33. 33. Distribute the load
  34. 34. The Web Tier All Web VMs need to know DB and Cache nodes addresses Master DB address Slaves DB addresses Session VMs addresses Cache VMs addresses
  35. 35. Distribute the load
  36. 36. Memcached Session handler # php.ini session.save_handler = memcached session.save_path = "192.168.0.5, 192.168.0.6, 192.168.0.7, 192.168.0.8" {% set memcached_servers = [] %} {% for server,ip in salt['publish.publish']('cache.*', 'network.interfaces').items() %} {% set m = memcached_servers.append(ip.eth0.inet[0].address) %} {% endfor %} session.save_handler = memcached session.save_path = "{{ memcached_servers|join(", ") }}
  37. 37. Database The problem: no default multiple connections MySQLi($host, $username, $password); //PDO, ... How to handle multiple connections? Write-Read and Read only? Master/Slave Async Replication
  38. 38. MySQL_ND Replace libmysql driver Default connector in PHP 5.4 MySQL_ND Master/Slave (plugin)
  39. 39. MySQLND_MS Configuration {     "myapp": {         "master": {             "master_0": {                 "host": "localhost",             }         },         "slave": {             "slave_0": {                 "host": "192.168.2.27",             }         }     } }
  40. 40. Configuration in Salt {     "cwitter.db": {         "master": {             {% for server,ip in salt['publish.publish']('rdb.*', 'network.interfaces').items() %}             "master_{{ server[0] }}": {                 "host": "{{ ip.eth0.inet[0].address }}"             }             {% endfor %}         },         "slave": {             {% for server,ip in salt['publish.publish']('srdb.*', 'network.interfaces').items() %}                 "slave_{{ server[0] }}": {                 "host": "{{ ip.eth0.inet[0].address }}"             }{% if not loop.last %},{% endif %}             {% endfor %}         },         "trx_stickiness": "master"     } }
  41. 41. Transaction Aware By default MySQLND_MS is not transaction aware trx_stickiness: master BEGIN TRANSACTION INSERT INTO ... DELETE FROM SELECT u1, u2, ... FROM ... UPDATE FROM COMMIT
  42. 42. Now the application distribute user sessions and DB queries
  43. 43. Distribute also HTTP requests!
  44. 44. Proxy HTTP/s requests
  45. 45. Proxy configuration needs Pubic IPs proxy:     provider: enter‐openstack‐config     size: e1standard.x1     image: GNU/Linux Ubuntu Server 12.04 LTS Precise Pangolin x64     ssh_username: ubuntu     ssh_key_file: /root/test.pem     ssh_key_name: 'my key name'     ssh_interface: public_ips     security_groups: salt‐minion,proxy     networks:         ‐ fixed:             ‐ xxxxxxxx‐xxxx‐xxxx‐xxxx‐xxxxxxxxxxxx         ‐ floating:             ‐ yyyyyyyy‐yyyy‐yyyy‐yyyy‐yyyyyyyyyyyy
  46. 46. NGINX as a proxy upstream app {     server 192.168.0.10:80;     server 192.168.0.11:80;     server 192.168.0.12:80;     # web server list } server {     listen 80;     location / {         proxy_pass http://app;     } }
  47. 47. NGINX proxy with Salt upstream app {     {% for server,ip in salt['publish.publish']('web.*', 'network.interfaces').items() %}     server {{ ip.eth0.inet[0].address }}:80;     {% endfor %} } server {     listen 80;     location / {          proxy_pass http://app;     } }
  48. 48. Use DNS round-robin feature in order to resolve proxies's IP
  49. 49. The app is ready!
  50. 50. Thanks for listening Walter Dal Mut Github: Twitter: Linkedin: wdalmut @walterdalmut Walter Dal Mut

×