Monitoring with Stackdriver

  • 825 views
Uploaded on

A presentation on how we came to find Stackdriver for system and service monitoring, how we implemented it and how we use it on a day to day basis.

A presentation on how we came to find Stackdriver for system and service monitoring, how we implemented it and how we use it on a day to day basis.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
825
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
19
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Monitoring with Stackdriver Or: How I Learned How to Stop Worrying and Dig Hosted Monitoring. denise stockman - denise@stockmans.org
  • 2. Mad Props Thanks to Coupa for hosting and to Stackdriver for sponsoring and to Advanced AWS for organizing.
  • 3. Hi, I’m denise ● ● ● ● ● System Architect @ Maverix Biomics 18 years button pushing Musical Chair Champion Data center to cloud Early Stackdriver user & puppet module maintainer
  • 4. What is Stackdriver? “Intelligent Monitoring for your AWS Infrastructure, Systems and App”
  • 5. Agenda ● ● ● ● ● ● The Environment Selection Process Implementation Day to Day TODO Questions
  • 6. A Diverse Stack
  • 7. Further Context ● ● ● ● ● Green field! We treat instances like cattle* Modern and crusty backend services “Quaint” team Monitor & report system, service, event and application statistics in one place. * As much as possible
  • 8. What to do for monitoring? Evaluate & Select ● In-house or hosted? ● Modern vs proven? ● Specialized vs general? ● Paid vs free?
  • 9. Let’s Be Honest ● ● ● ● Not enough time Small environment More important work to do Everyone else is doing it
  • 10. Winner, Winner, Chicken Dinner ● ● ● ● ● ● ● System & service metrics Alerting Custom metrics & annotations collectd agent Anomaly detection AWS integration & cloud friendly Cost per resource
  • 11. Implementation
  • 12. Consistent Resource Naming Resource name format env-function-id.location.domain.com Tags for resources env, owner, purpose, version
  • 13. ● ● ● ● Installs base agent on all hosts Configures API key Includes plugins for supported services Auto-upgrades agent node /^dev-redis.*.stockmans.org$/ { class { "stackdriver::redis": stackdriver_redis_port => 6379, stackdriver_redis_host => localhost, stackdriver_redis_timeout => 2000, } } Agent Deployment
  • 14. ● ● ● ● Create groups based on filtering criteria Dashboards auto-created for each group Groups used in custom dashboarding and alerting policies Set it once and fuggetaboutit* Groups
  • 15. Alerting
  • 16. Day to Day Use
  • 17. Custom Dashboards
  • 18. Anomaly Detection
  • 19. Event Correlation
  • 20. Event Logging
  • 21. curl -X POST -H "x-stackdriver-apikey: DOOOOWEEEEOOOO" -H "Content-Type: application/json" -d '{"message":"Mind taking!", "annotated_by":"Mentok"}' https://event-gateway.stackdriver.com/v1/annotationevent Annotations
  • 22. Ruby example customer_id = 31337 api_key = "DOOOOWEEEEOOOO" StackDriver.init customer_id, api_key redis = Redis.new mindsTaken = redis.zcard("minds:taken") StackDriver.send_metric "mentok.minds.taken", mindsTaken, Time.now.to_i Custom Metrics
  • 23. TODO.txt ● ● ● ● ● Expand app metrics with statsd Automate group creation & management Annotate orchestrated events Cloudwatch custom metrics Stackdriver @ our colo
  • 24. Thanks! Questions? denise stockman denise@stockmans.org github.com/maverixbio/puppet-stackdriver-agent forge.puppetlabs.com/dstockman/stackdriver