Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Clientless Monitoring with Sensu


Published on

From the talk Chris Chandler, Principal Engineer at T-Mobile, gave at Sensu Summit 2018:

We all know and love sensu-client, but there are some times where deploying the client is either sub-optimal (e.g.: a box you don’t control) or impossible (e.g.: appliances, 3rd party services).

I opted to tackle these kind of challenges Serverless-style, running my own code, then shipping the check outcome to the /results API.

Some reasons for going this route include:

- Sometimes you simply need to do more than the community plugins offer, which means you’re already writing custom code anyway. All you need to do is do a POST to the /results API, and boom.

- I didn’t want to manage “bastion” clients that do proxy client style monitors, because I don’t live in a world where I can run Configuration Management tools against boxes at-will. Running in a PaaS/Serverless mode gave me more autonomy to iterate.

- Checks can be more dynamic vs having to re-run Configuration Management to change sensu-client configs. For example, I can pull a list of service endpoints from Service Registry in real-time, then iterate that list in your code. Having to re-trigger Ansible/Chef/Puppet when endpoints light up/die would cause a lot of churn in highly dynamic environments.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Clientless Monitoring with Sensu

  1. 1. T-Mobile Public Clientless Monitoring with Sensu Chris Chandler Principal Engineer @cjchand
  2. 2. AKA: Look, ma! No Clients!
  3. 3. Part 2
  4. 4. So, wait… NO clients? How does that work?
  5. 5. Simple: Proxy clients
  6. 6. Yes, technically still a “client”, but there’s no client process/daemon Come on, work with me here...
  7. 7. Proxy clients allow check results to be shipped on behalf of anything where you can’t or don’t want to run a client
  8. 8. Appliances, Network gear, SaaS / PaaS / FaaS, etc.
  9. 9. Super High-Level Architecture
  10. 10. Clientless monitor is 100% technology agnostic Just needs to ship a result to Sensu’s API
  11. 11. Example Clientless API call curl -s POST '' -H 'Content-Type: application/json' -d '{ "source": "really-important-database", "name": "check-my-db", "output": "ZOMG! The DB is dead!", "status": 2, "occurrences": 1, "refresh": 3600, "handlers": [ "pagerduty", "email" ] }' Required Optional
  12. 12. Just because we can do it, why should we?
  13. 13. Unable/Unwilling to deploy a client (You did see prior slides, right?)
  14. 14. Reuse tests as monitors (e.g.: Postman Collections, unit/functional tests)
  15. 15. Leverage Service Registry to discover and monitor endpoints
  16. 16. Getting the “twofer” of monitoring + service registry
  17. 17. Demo Overview • docker-compose based • 3x web servers • Consul • Sensu and Co. • Python-based monitor
  18. 18. Containers and Communications
  19. 19. High-Level Monitoring Logic 1. Get all services in Consul via REST API 2. For each service, inspect nodes for IP and ServicePort 3. If URI in metadata, add that to IP + Port 4. Poll HTTP endpoint 5. Send results to Sensu’s /results API 6. Delete any Sensu proxy clients that are no longer registered in Consul
  20. 20. Consul Metadata { "service": { "name": "web-hello-world", <... content omitted …> "meta": { "environment": "production", "handler": "pagerduty", "pdkey": "1234567890", "uri": "/health" } <...>
  21. 21. Self-Service via Metadata • Devs can use Consul’s “ServiceMeta” to specify: • Dependencies • Subdue • Handlers • TTL • Routing/contact info (e.g.: PagerDuty API key) • URI to check • $your_idea_here
  22. 22. Processing Consul Metadata try: for k,v in node['ServiceMeta'].items(): payload[k] = v except: # ServiceMeta section is empty, ignore pass
  23. 23. Some caveats... • No keepalives with proxy clients • Use TTL in check payload • Best if run in PaaS/FaaS/Container platform • Reachability of servers
  24. 24. Other Possibilities • By passing a “type” k/v pair in the metadata, you can specify what type of service this is: • SNMP • DBs • $your_service_here
  25. 25. And now, on to the shameless self- promotion!
  26. 26. Twitter: @cjchand (Rarely updated) Blog: Clone the Demo Code: