The IBM dashboard for operational metrics

4,931 views
4,635 views

Published on

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
4,931
On SlideShare
0
From Embeds
0
Number of Embeds
1,672
Actions
Shares
0
Downloads
51
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

The IBM dashboard for operational metrics

  1. 1. 1
  2. 2. 2 Daniel Krook Senior Certified IT Specialist, IBM The IBM dashboard for operational metrics
  3. 3. 3 We run Cloud Foundry on dozens of OpenStack VMs Two intranet clusters In the past year, we’ve learned how to Classic: 38 huge VMs deployed with Chef: 1,302 users, 1,710 apps NG: 41 medium VMs deployed with BOSH: 123 users, 247 apps Not counting Dev deployments All on 50+ Nova Compute nodes • Keep Cloud Foundry running smoothly • Discover and prevent impending problems • Resolve unexpected issues quickly
  4. 4. 4 1. Show the key data points we track 2. Show how our metrics dashboard helps us monitor that data 3. Share ideas on how to find better data in NG and beyond 4. Spark discussion on improved visibility for CF admins and customers. Goals of this lightning talk We are looking to get better at this, and help the community get better as well.
  5. 5. 5 1. The key data
  6. 6. 6 What are the important metrics? Data that can be tracked over time to see trends and behaviors Data that can help us predict problems before they happen DEAs and apps health  Memory reserved as a proportion of the memory available General health of all components  Health of the virtual machines  Status of the processes running on them Database nodes and services  Number of provisioned services against capacity available At the PaaS layer, that means:
  7. 7. 7  Deliver continuous availability in the cloud  Proactively solve problems rather than react to them  Understand the behavior of the system to automate it Why do we need metrics?
  8. 8. 8  NATS message bus • Discover the components to interrogate • Best for dynamically changing data Where can we find them?  Cloud Controller database (CCDB) • Longer lived data that isn’t in the varz endpoints
  9. 9. 9 2. Monitoring that data
  10. 10. 10 1. Views of component health 2. Resource usage details 3. Ongoing growth trends 4. Access to logs and raw varz 5. Email notifications Our metrics dashboard provides…
  11. 11. 11  Components nearing capacity or failure  Already failed components  Out of control apps and noisy users  Active/inactive users and apps  Growth trends and runtime/service adoption It helps us find (and fix) problems It helps us see patterns
  12. 12. 12 User and app trends There is also one unauthenticated page for high level stats
  13. 13. 13 DEA list
  14. 14. 14 DEA details
  15. 15. 15 Service node list
  16. 16. 16 Service node details
  17. 17. 17 User list
  18. 18. 18 User details
  19. 19. 19 App list
  20. 20. 20 App details
  21. 21. 21 Log list
  22. 22. 22 Log details
  23. 23. 23 Email notifications
  24. 24. 24 3. Finding and acting on better data
  25. 25. 25  NG provides granular user/org/space views… • This enables better BSS potential in terms of QoS and departmental billing  …But we lost user and app data linkages from the health manager • Can’t see what DEA my app resides on (not currently enabled in our NG version) • Can’t see how many apps a user has (replaced by orgs and spaces, but still valuable to trace) • See https://github.com/cloudfoundry/cloud_controller_ng/issues/81  We’d like to restore that data, either surface it • in varz endpoints (dynamic data, preferred) or • CC_DB (static data, could be a security concern) Let’s resolve gaps in data captured from NG
  26. 26. 26  Detect errors in applications that are traceable to users/orgs • Preemptively reach out to them to see if they need help • Think customer service and proactive support! • Can we hook into to BOSH or Jenkins for automation?  Automate (and expand links to the IaaS and SaaS stacks) • Self healing systems (out of disk, move apps) • Self scaling systems (detect when nearing thresholds) • Evolving topologies (replace unused service nodes with popular ones) Let’s begin to link metrics to automation
  27. 27. 27  Admins are the primary beneficiary right now • But data is almost completely read only • Should we provide UAA based tiers of access to admins?  Others can and should benefit • Customers • End users • Developers • Management • Executives, line of business owners • Finance Let’s expand the broadcast of metrics to more users
  28. 28. 28 Thanks!
  29. 29. 29 The metrics dashboard innovators Chris Peters Russell Boykin Doug Davis Wei Feng
  30. 30. 30 We’re hiring! Search Jobs at IBM by: SmartCloud Application Services
  31. 31. 31

×