Spit, Gather, Churn - Mining Infrastructure Data for Ops Intelligence

5,350 views

Published on

Presentation on how to design infrastructure services for meaningful ops intelligence, and how to integrate ops intelligence as feedback for software development

Published in: Technology, Education

Spit, Gather, Churn - Mining Infrastructure Data for Ops Intelligence

  1. 1. Spit , Gather, ChurnMining Infrastructure Data for Ops Intelligence Ranjib Dey Twitter: @RanjibDey IRC/Github :@ranjibd
  2. 2. About Me• Senior software engineer in the CD practice group @ThoughtWorks India• Was system administrator before @ThoughtWorks India• Worked on life science related algorithms @Persistent Systems before that.• Masters in Bio-Informatics (thesis on HPC, Machine Learning)• Life Science graduate
  3. 3. Agenda• What is Ops intelligence?• Why its needed? Implications of Ops Intelligence.• Why it is important now?• Designing intelligent infrastructure services• How the future looks like?• Q&A
  4. 4. What is Ops Intelligence?• Suitable for fast , meaningful ops feedback to business• Abstracts infrastructure details• Tech-Stack neutral• Allows forecasting• Pre-emptive in nature
  5. 5. What is intelligence? Data Mining Data Information Knowledge
  6. 6. Why its needed? Implications• Self serving• Lean• Elasticity• Adaptive
  7. 7. Why its important now?• Market volatility increased• Its not the development, but the deployment , release and maintenance that’s introducing delay.• Cloud is here• Infrastructure tooling is matured• Continuous Delivery and DevOps movement is on
  8. 8. Designing intelligent infrastructure services• End user driven services• Adhere to core unix philosophies• Remember the ‘|’ , don’t create dead ends• Feedback driven , iterative improvement• Think of horizontal scalability• Infrastructure as a code
  9. 9. Spitting out ops information• State and Metrics• Logs
  10. 10. Metrics• An unit test for a method and a monitoring service for each infrastructure service• A single monitoring service can have multiple metrics• Metrics can have relationships• These features should be configurable
  11. 11. Metrics driven infrastructure developmentService Metric
  12. 12. Logging• Decouple logging framework from the core services• Have configurable logging levels• Enforce appropriate logging and levels• Enforce logging patterns• Logs and logging patterns can be modeled as metric too.
  13. 13. Metrics on LogLog Metric on log pattern
  14. 14. Gathering Ops Information• Information aggregation• Consider how you will use it• Metrics and Logs• Centralized logging
  15. 15. Gathering Ops information• Two main patterns: – Time series data – OLAP Cubes• Storage engine considerations – Flat files – RRDs – NoSQLs and other distributed storage systems
  16. 16. Churning Ops Information• Visualizations – Charting – Trending – Customized Visualizations• Dashboards – Customized views for stake holders – Information Radiators
  17. 17. Churning Ops Information• Logs – Search – Index – Alerts and notification on top of aggregated logs
  18. 18. Validation 1: Continuous Delivery
  19. 19. Validation 1: Continuous Delivery
  20. 20. Validation 2: Performance Enhancements
  21. 21. Validation 3: Holistic information
  22. 22. Validation 4: Meaningful information• Meaningful alerts: – Nodable http://www.nodeable.com/• Log analytics: – Loggly http://loggly.com/ – SplunkStorm https://www.splunkstorm.com/ – Graylog2/Logstash• Dashboards for Metrics – Graphite (+graphiti)
  23. 23. How the future looks like?• IaaS• Ops is not the bottleneck• Context aware infrastructure• Test driven infrastructure• SSH is not a must• “ The machines are alive” – Jon Crosby …… and they are emerging
  24. 24. Thank You

×