Monitoring with Stackdriver

2,148 views
1,856 views

Published on

A presentation on how we came to find Stackdriver for system and service monitoring, how we implemented it and how we use it on a day to day basis.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,148
On SlideShare
0
From Embeds
0
Number of Embeds
160
Actions
Shares
0
Downloads
31
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Monitoring with Stackdriver

  1. 1. Monitoring with Stackdriver Or: How I Learned How to Stop Worrying and Dig Hosted Monitoring. denise stockman - denise@stockmans.org
  2. 2. Mad Props Thanks to Coupa for hosting and to Stackdriver for sponsoring and to Advanced AWS for organizing.
  3. 3. Hi, I’m denise ● ● ● ● ● System Architect @ Maverix Biomics 18 years button pushing Musical Chair Champion Data center to cloud Early Stackdriver user & puppet module maintainer
  4. 4. What is Stackdriver? “Intelligent Monitoring for your AWS Infrastructure, Systems and App”
  5. 5. Agenda ● ● ● ● ● ● The Environment Selection Process Implementation Day to Day TODO Questions
  6. 6. A Diverse Stack
  7. 7. Further Context ● ● ● ● ● Green field! We treat instances like cattle* Modern and crusty backend services “Quaint” team Monitor & report system, service, event and application statistics in one place. * As much as possible
  8. 8. What to do for monitoring? Evaluate & Select ● In-house or hosted? ● Modern vs proven? ● Specialized vs general? ● Paid vs free?
  9. 9. Let’s Be Honest ● ● ● ● Not enough time Small environment More important work to do Everyone else is doing it
  10. 10. Winner, Winner, Chicken Dinner ● ● ● ● ● ● ● System & service metrics Alerting Custom metrics & annotations collectd agent Anomaly detection AWS integration & cloud friendly Cost per resource
  11. 11. Implementation
  12. 12. Consistent Resource Naming Resource name format env-function-id.location.domain.com Tags for resources env, owner, purpose, version
  13. 13. ● ● ● ● Installs base agent on all hosts Configures API key Includes plugins for supported services Auto-upgrades agent node /^dev-redis.*.stockmans.org$/ { class { "stackdriver::redis": stackdriver_redis_port => 6379, stackdriver_redis_host => localhost, stackdriver_redis_timeout => 2000, } } Agent Deployment
  14. 14. ● ● ● ● Create groups based on filtering criteria Dashboards auto-created for each group Groups used in custom dashboarding and alerting policies Set it once and fuggetaboutit* Groups
  15. 15. Alerting
  16. 16. Day to Day Use
  17. 17. Custom Dashboards
  18. 18. Anomaly Detection
  19. 19. Event Correlation
  20. 20. Event Logging
  21. 21. curl -X POST -H "x-stackdriver-apikey: DOOOOWEEEEOOOO" -H "Content-Type: application/json" -d '{"message":"Mind taking!", "annotated_by":"Mentok"}' https://event-gateway.stackdriver.com/v1/annotationevent Annotations
  22. 22. Ruby example customer_id = 31337 api_key = "DOOOOWEEEEOOOO" StackDriver.init customer_id, api_key redis = Redis.new mindsTaken = redis.zcard("minds:taken") StackDriver.send_metric "mentok.minds.taken", mindsTaken, Time.now.to_i Custom Metrics
  23. 23. TODO.txt ● ● ● ● ● Expand app metrics with statsd Automate group creation & management Annotate orchestrated events Cloudwatch custom metrics Stackdriver @ our colo
  24. 24. Thanks! Questions? denise stockman denise@stockmans.org github.com/maverixbio/puppet-stackdriver-agent forge.puppetlabs.com/dstockman/stackdriver

×