Chef Actions: Delightful near real-time activity tracking!

1,325 views
1,131 views

Published on

Actions is a new analytics add-on for Enterprise Chef, letting you see who did what on your Chef Server.

Published in: Data & Analytics, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,325
On SlideShare
0
From Embeds
0
Number of Embeds
45
Actions
Shares
0
Downloads
10
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Chef Actions: Delightful near real-time activity tracking!

  1. 1. Chef Actions Actions: Delightful near real-time activity tracking! Allen Goodman (@goodmanio), Software Engineer, Chef James Casey (@jamesc_000), Engineering Lead, Chef
  2. 2. A first look at Chef Actions Chef Actions answers questions about what is happening on your Chef Server • What changed on your Chef Server ? • Clients, Cookbooks, Data Bags, Environments, Nodes, Roles • Who changed it ? • What did they do ? • Create, Update, Delete • When did they do it ?
  3. 3. Principles • Provide a read-only view of what happened • Road to audit • Allow to react to events as they happen • Also, enable after the fact investigation • “What happened just before nodes started failing runs?” • “When did our systems gets patched for Heartbleed?”
  4. 4. Architecture
  5. 5. Chef Analytics’ Web Application
  6. 6. Event Processing
  7. 7. Message Format from Chef { "entity_name": "2.8.1", "entity_type": "version", "parent_name": "apache2", "parent_type": "cookbook”, "task": "update", "organization_name": ”my_corp", "recorded_at": "20014-04-05T18:26:13Z", "remote_hostname": ”33.33.33.10", "remote_request_id": "D5974D8A-E961-4845-8D4F-5BB92DAD1BD5", "service_hostname": ”private-chef-fe.example.com”, "request_id": "hY2UqBZTuqAr65M8t4FmYAD5RUjbnhpBhubxBZqVZJZgNX7x", "requestor_name": ”bob", "requestor_type": ”user”, "user_agent": "Chef Client/11.12.0" }
  8. 8. Event Processing Architecture
  9. 9. Classification • Static and Dynamic Analysis • Conformance • Are conformant cookbooks less susceptible to failure? • Coverage • Is coverage correlated with success? • Style • Does my preferred style require more maintenance? • Syntactic and Semantic Mistakes • Is this resource broken? • Supervised Learning • Predicting Imminent and Long-term Problems • Why does my infrastructure break?
  10. 10. Notifications IRC notifications: irc: - "chat.freenode.net#chef" template: - "%{resource} %{action}d!” skip_join: true use_notice: true E-mail notifications: email: - a@getchef.com - james@getchef.com on_update: never Campfire, HipChat, Slack, et al.
  11. 11. Webhooks • POST a full copy of a message to an external service • Optionally contains copy of object • E.g. full node object or role • Ships with a stub service for you start from • Usage: • Monitoring – notified on create/delete • CMDB – monitor software/OS version changes for compliance Webhooks hooks: endpoint: - https://my-service.example.com/ingest with_object: true
  12. 12. Creating Metrics metrics: - id: james - "What’s up with James’ weird hours?” - actions: - create - delete - update - actors: - "james" - resources: - cookbook - statistics: - frequency - mean - window - hourly
  13. 13. Persistence
  14. 14. Query Parameters /actions?tasks=create, delete, update &types=environment &name=prod &before=2014-04-17 &since=2014-04-10
  15. 15. Response{ "id": "e1e2c057-b18f-46db-8c84-7d87efc07260", "tags": [ "Knife” ], "recorded_at": "2014-04-13T04:38:03.000Z", "remote_hostname": "76.115.172.22", "remote_request_id": "d57845ce-3b48-4070-9ed4-4b4333d112cb", "request_id": "g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEDAAKEjwAAAAkAAAAA", "service_hostname": "action-rspreprod-i-c63b3e97.opscode.us", "task": "delete", "url": "/activities/e1e2c057-b18f-46db-8c84-7d87efc07260", "user_agent": "Chef Knife/11.12.2 (ruby-1.9.3-p194; ohai-7.0.2; x86_64-darwin11.4.2; +http://opscode.com)", "requestable": { "name": "bob", "type": "user”, "url": "/users/bob”, "properties": {"email": "james+1@getchef.com", "gravatar_id": "55bf19f1b9fa6ac9ab755488e29a5779”, "avatar_url": “https://gravatar.com/avatar/55bf19f1b9fa6ac9ab755488e29a5779.png?default=mm” }, "entity": { "name": "prod", "parent": null, "type": "environment”, "url": "/organizations/chef/environments/prod”, "manage_url": https://manage-chefconf-demo.opscode.com/organizations/chef/environments/prod } }
  16. 16. Ruby Example client = Actions::Client.new(ACTIONS_KEY) actions = client.actions.where action: %i(create delete update), actor: 'james' resource: :cookbook, window: { recorded_at: :hourly } { frequency: actions.frequency, mean: actions.mean }
  17. 17. R Example install.packages('chef-actions') actions = chef.actions.where(action %in% ["create", "delete", "update"]) .where(actor %in% ["james"]) .where(resource %in% ["cookbook"]) .window(recorded_at, "hourly") summary(actions)
  18. 18. Integration points • RabbitMQ • Consume raw messages in real-time out of Chef Server • Notifications • Send formatted messages to external services in real-time • Webhooks • Send whole message with annotations to your external service in real-time • REST API • Query-only view of historical data
  19. 19. Deployment
  20. 20. Deployment topologies • Supports same HA architecture as Enterprise Chef • Backend • PostgreSQL • Frontend • Analytics pipeline, Query API, Ingest service • Recommended to deploy on separate hardware than Enterprise Chef
  21. 21. Omnibus • New add-on chef-analytics • Delivered as a single omnibus package • Hosted on separate domain • E.g. analytics.getchef.com • Only interactions with Private Chef • RabbitMQ configuration details • Manage root URL for generation of links
  22. 22. Futures
  23. 23. What’s Next? • Permissions – ACL changes, group join/leave • Object Diffs • Live Feed • Aggregation (all actions from a chef-client run, berkshelf, knife run) • Knife instrumentation • Cloud plugins
  24. 24. Q1 Q2 Q3 Q4 Account Improvements Large Customer Features Service Provider Features HA Restructuring Chef Actions and Run History Compliance Reporting Compliance Content Windows Container Improvements Red Hat The right premium features • Make existing differentiation more accessible, address concerns of our largest customers • Extend our analytics capability to provide best of breed compliance reporting • Ensure we retain a lead on Windows support • Embrace containers as a first class part of our ecosystem
  25. 25. Event Processing Architecture
  26. 26. Chef Actions • We know there is valuable data locked up inside your Chef Server • Chef Actions exposes that data to you via a variety of APIs • We have built a flexible architecture that will allow us to expose more and more data • Available today – Limited availability • No release schedule – continuous updates • We’ll rapidly open to more and more customers • Defined roadmap for 2014 for compliance and audit

×