Monitor your Atlassian stack like the NSA

6,124 views

Published on

Monitor your Atlassian stack like the NSA by Tom Moors, Atlassian Expert at ACA IT-Solutions.

Published in: Business, Technology, Design

Monitor your Atlassian stack like the NSA

  1. 1. Monitor your Atlassian stack like the NSA Tom Moors January 16th 2014 Atlassian User Group Belgium aca-it.be - 2014 http://www.flickr.com/photos/zigazou76/7670889680
  2. 2. @tommoors Who is Tom Moors? ✓ Atlassian Expert @ ACA IT-Solutions ✓ Poor drawer ✓ @tommoors ✓ t.moors@aca-it.be aca-it.be - 2014
  3. 3. @tommoors Why monitoring? aca-it.be - 2014
  4. 4. @tommoors Costs money Why monitoring? Needs extra time aca-it.be - 2014
  5. 5. @tommoors Detect fraud Reduce costs Improve service Why monitoring? Save time Offer insights Opportunities aca-it.be - 2014 …
  6. 6. “If you can not measure it, you can not improve it.” –Lord Kelvin aca-it.be - 2014
  7. 7. @tommoors …but #monitoringsucks! Grea t mo veme aca-it.be - 2014 nt!
  8. 8. @tommoors Agenda • Introduction • Building blocks • Monitoring levels • Conclusion aca-it.be - 2014
  9. 9. @tommoors What is monitoring? Observe and check the progress or quality of (something) over a period of time; keep under systematic review http://www.oxforddictionaries.com/definition/english/monitor aca-it.be - 2014
  10. 10. @tommoors What is monitoring? Super Monitoring aca-it.be - 2014
  11. 11. @tommoors Approach to monitoring? • Ride on the DevOps/Infrastructure as Code wave • Build a toolchain aca-it.be - 2014
  12. 12. @tommoors Goal Alerts ... when things go wrong ... before disaster strikes ... no spamming / false metrics User Interface • Context for post-mortem • Trends • Current state aca-it.be - 2014
  13. 13. @tommoors Monitoring basics • Gather data • Transport data • Process and store data • Present data • Alerts • Trends • History / Future aca-it.be - 2014
  14. 14. @tommoors Monitoring basics Metric Event ✓ ✓ Time Fields ✓ ✓ ✓ Time Name Value can be converted to aca-it.be - 2014 http://www.slideshare.net/roidelapluie/devops-andmonitoringowf13
  15. 15. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  16. 16. @tommoors Availability • Built-in checks: ping • Service Delivery? aca-it.be - 2014
  17. 17. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  18. 18. @tommoors Infrastructure System Services • CPU usage/min/max/average • Network • Memory usage/min/max/ average • Database • • Disk space • • Query timing LDAP / Active Directory IO Wait Other • Process load • SSL/TLS Certificates aca-it.be - 2014
  19. 19. @tommoors Infrastructure • Do-It-Yourself (Not Invented Here Syndrome) • CollectD • StatsD • Cacti • Ganglia • Hyperic • Nagios, Icinga • Zabbix aca-it.be - 2014
  20. 20. @tommoors Infrastructure google.com results Cacti Nagios 1,690,000 StatsD 1,470,000 Ganglia 1,150,000 Zabbix 922,000 Icinga 730,000 Hyperic 524,000 CollectD aca-it.be - 2014 1,740,000 225,000
  21. 21. @tommoors Infrastructure: Icinga aca-it.be - 2014
  22. 22. @tommoors Infrastructure: Icinga aca-it.be - 2014
  23. 23. @tommoors Icinga: Alerts Extensible with plugins • Mail • JIRA issue • Jabber/XMPP • HipChat • SMS • PagerDuty • … aca-it.be - 2014
  24. 24. @tommoors StatsD • UDP Network daemon • Aggregates data • Buckets with values • Type • • Timer • • Counter Gauge Backend: eg Graphite aca-it.be - 2014
  25. 25. @tommoors Graphite • Stores numeric time-series • Render graph of this data on demand ! Architecture • carbon: daemon • whisper: data storage • graphite webapp: frontend aca-it.be - 2014
  26. 26. @tommoors Graphite aca-it.be - 2014
  27. 27. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  28. 28. @tommoors Process • Is the service/process still running? • Watchdog to (repair and) restart eg upstart, Monit, jsvc ! Potential problem ➞ send event aca-it.be - 2014
  29. 29. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  30. 30. @tommoors JavaMelody • Plugin for JIRA, Confluence, Bamboo • Lightweight • Statistics and charts aca-it.be - 2014
  31. 31. @tommoors Java Applications: JMX Java Management eXtensions • MBeans representing device, application, resource • Set of readable (writable) attributes • Set of operations aca-it.be - 2014
  32. 32. @tommoors Java Applications: JMX aca-it.be - 2014
  33. 33. @tommoors Java Applications: JMX aca-it.be - 2014
  34. 34. @tommoors Working with JMX • Well-supported in tools (plugins) • Jolokia • jmxtrans aca-it.be - 2014
  35. 35. @tommoors Commercial solutions • New Relic • Keynote aca-it.be - 2014
  36. 36. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  37. 37. @tommoors Business Value How many… • new spaces? • issues created? • users active? • popular pages? • … Very specific metrics aca-it.be - 2014
  38. 38. @tommoors Business Value • Google Analytics • Piwik • (Hummingbird) • JIRA: reporting • Custom development aca-it.be - 2014
  39. 39. @tommoors Monitoring levels • Availability • Infrastructure • Process • Application • Business BONUS: Logs aca-it.be - 2014
  40. 40. @tommoors Logs • Most up-to-date information • Best insights, eg why a HTTP 500? • Unstructured data • Massive amount from multiple sources aca-it.be - 2014
  41. 41. @tommoors Working with Logs Scribe Logstash free and open source commercial aca-it.be - 2014
  42. 42. @tommoors Logstash http://logstash.net aca-it.be - 2014
  43. 43. @tommoors LogStash: design Input aca-it.be - 2014 Filter Output
  44. 44. @tommoors LogStash: design Input Filter Output 2013-10-29 12:48:43,663 Thread-67 ERROR [jira.issue.managers.DefaultCustomFieldManager] Could not load custom field type plugin with key 'com.pyxis.greenhopper.jira:gh-epic-label'. Is the plugin present and enabled? aca-it.be - 2014
  45. 45. @tommoors LogStash: design Input Filter Output 2013-10-29 12:48:43,663 Thread-67 ERROR [jira.issue.managers.DefaultCustomFieldManager] Could not load custom field type plugin with key 'com.pyxis.greenhopper.jira:gh-epic-label'. Is the plugin present and enabled? aca-it.be - 2014
  46. 46. @tommoors LogStash: design Input Filter Output Input • tcp • file • udp • imap • rabbitmq • stdin • collectd • log4j • redis • syslog • … aca-it.be - 2014
  47. 47. @tommoors LogStash: design Input Filter Output Filter • • • • mutate • • extractnumbers • • dns • • split … grep grok geo ip anonymize drop aca-it.be - 2014
  48. 48. @tommoors LogStash: design Input Output Filter Output • http • elasticsearch • StatsD • email • Nagios • graphite • PagerDuty • HipChat • Cloudwatch • JIRA • … aca-it.be - 2014
  49. 49. @tommoors LogStash: architecture Shipper Shipper Shipper aca-it.be - 2014 Broker Indexer Search & Storage Web Interface
  50. 50. @tommoors LogStash: Kibana aca-it.be - 2014
  51. 51. @tommoors One more thing… aca-it.be - 2014
  52. 52. @tommoors Dashboards • Buy a big TV for the office • Combine the most important metrics • SLA • Builds • Work done • Deployments • … aca-it.be - 2014
  53. 53. @tommoors Dashboards JIRA Wallboard Plugin aca-it.be - 2014
  54. 54. @tommoors Dashboards AtlasBoard aca-it.be - 2014
  55. 55. @tommoors Dashboards GDash aca-it.be - 2014
  56. 56. @tommoors Dashboards Graphene aca-it.be - 2014
  57. 57. @tommoors Dashboards Giraffe aca-it.be - 2014
  58. 58. @tommoors Conclusion Possible toolchain LogStash StatsD Graphite System Icinga aca-it.be - 2014
  59. 59. @tommoors Conclusion • Monitoring is a must • Great tools support great responsibility • There is no one-fits-all • Iterate over your configuration • Get close(r) with logs • Dashboards are a great reason to buy a big TV aca-it.be - 2014
  60. 60. @tommoors Questions? To oors mM -it.be @aca oors t.m aca-it.be - 2014 @to oors mm
  61. 61. @tommoors Further Reading • https://github.com/monitoringsucks • http://jedi.be/blog/2012/01/03/monitoring-wonderland-survey-introduction • http://jedi.be/blog/2012/01/03/monitoring-wonderland-metrics-api-gateways • http://jedi.be/blog/2012/01/04/monitoring-wonderland-moving-up-the-stack-application-user-metrics • http://jedi.be/blog/2012/01/03/monitoring-wonderland-nagios-the-mighty-beast • http://www.paperplanes.de/2011/1/5/the_virtues_of_monitoring.html • http://assets.en.oreilly.com/1/event/65/Metrics-driven%20Engineering%20at%20Etsy%20Presentation.pdf • http://www.devco.net/archives/2011/03/19/thinking_about_monitoring_frameworks.php • http://www.slideshare.net/roidelapluie/devops-andmonitoringowf13 • http://serialized.net/2011/02/getting-more-signal-from-your-noise • http://obfuscurity.com/static/TrendingWithPurpose_20110318.pdf aca-it.be - 2014

×