Managing scalable infrastructure based on monitoring


Published on

A presentation from #CCCEU13. It is based on Forthscale`s method of running scalable infrastructure with the help of monitoring systems. Now updated as for OpsTalk TLV

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Managing scalable infrastructure based on monitoring

  1. 1. Managing scalable infrastructure with monitoring
  2. 2. What is ForthScale? Scalable infrastructure experts company kick ass team of skilled ninjas ● ● ● ● ● Consulting Design Integration Implementation Management
  3. 3. Groundlaying ● ● ● ● ● Semantics (intent or a desired state) Dynamics (actual behavior) Dev think mostly about semantics Ops think mostly about dynamics DevOps = complete picture ● Thank you Mark Burgess
  4. 4. Groundlaying ● Complexity is THE source of randomness. ● Complexity is the birthplace of chaos ● We all know the KISS principle but fail it. The maintenance theorem says: you can't really control anything over time. Best you can do is to keep it roughly in balance.
  5. 5. Groundlaying "Equilibrium" replaces determinism as the most important idea in science. It is the definition of dynamical stability.
  6. 6. What is scaling Scaling is switching multiple Equilibrium points in accordance with current resources demand.
  7. 7. Groundlaying ● People tend to use modern IaaS same way they used to operate hardware systems. ● If you do not take true advantage of IaaS why pay it`s premia? ● If you are paying for it, maybe it could be a good idea to actually use it?
  8. 8. Basics - Operator goals Use IaaS to improve infrastructure lifecycle management. ● ● ● ● Provide availability Keep user experience Manage costs Account for usage
  9. 9. Cases Mainly running an operation with fluctuating resource consumption. ● Web app (externally or internally hosted) ● Mobile app (externally hosted) ● Intranet app (internally hosted) ● Data analysis (externally or internally hosted)
  10. 10. Basics - a perfectly good stack ● ● ● ● Clear policy on resource consumption Stack parts horizontal scalability Non sticky nodes (for easy replacements) Simply orchestrated
  11. 11. The Forthscale way What is the best way to allocate resources? Monitoring stack and learning what resources are needed and when
  12. 12. Controller building blocks ● CloudStack allows to manage allocation of computation resources (IaaS). ● Munin monitoring collects analytical data about infrastructure performance. ● Ansible for infrastructure orchestration.
  13. 13. CloudStack ● ● ● ● Is a cloud orchestration platform Manages computing resources allocation Has an extensive API Yes we can use other platforms
  14. 14. Munin Munin is an open source client /server monitoring application that presents output in graphs through a web interface yes, any monitoring will do.
  15. 15. Munin ● ● ● ● ● ● ● highly configurable stores to rrd makes graphs shows trends can alert very very very simple to deploy text based configuration
  16. 16. Cloudstack + Munin = 日本のスタイル ● ● ● ● ● Set of plugins done with CloudMonkey Can store all metrics provided by api Correlate IaaS, OS and AppStack issues Can use Nagios or similar for alerting But why just alert then you can handle?
  17. 17. Handling ● ● ● ● ● Alert issue Re-allocate resources (increase / decrease) Switch resources between components Replace components / nodes Report issue
  18. 18. Handling - Orchestration Issues can arise on any level ● ● ● ● IaaS resources OS Middleware Continuous Integration Better use orchestration tool that supports any dynamic, any part of the stack.
  19. 19. Ansible Why ? “why not just use Ansible instead?” case. ● Simple ● Agentless ● Powerfull ● Python
  20. 20. CloudStack+Ansible=No limits Generate playbooks to: ● ● ● ● Add or remove nodes Change nodes sizes Upgrade components or application Add / remove nodes to / from arrays
  21. 21. CloudStack+Munin+Ansible Scalable infrastructure what: ● Operates on true metrix. ● Efficient and accountable. ● Available, Scalable and Redundant.
  22. 22. Use scenarios - Vertica Vertica is a SQL compliant grid database. 1. Adding or removing nodes based on amount of resources needed. 2. Changing nodes sizes based on amount of resources needed. 3. Proactive policy for resource allocation.
  23. 23. Use scenarios - version release New version can also lead to changes in computing power requests. 1. 2. 3. 4. Building a stage environment. Executing a load test. Comparing monitoring metrics. Adjusting resources together with deployment
  24. 24. Forthscale contacts Naor Weissmann IL: +972 (0)505 980005 US: +1 303 5867379 UK: +44 (0)203 1292157 Email: Web site:
  25. 25. Feel free to contact me any time Thank you for your time. I hope that at least I’ve raised some questions.