1

2
Daniel Krook
Senior Certified IT Specialist, IBM
The IBM dashboard for operational metrics

3
We run Cloud Foundry on dozens of OpenStack VMs
Two intranet clusters
In the past year, we’ve learned how to
Classic: 38 huge VMs deployed with Chef: 1,302 users, 1,710 apps
NG: 41 medium VMs deployed with BOSH: 123 users, 247 apps
Not counting Dev deployments
All on 50+ Nova Compute nodes
• Keep Cloud Foundry running smoothly
• Discover and prevent impending problems
• Resolve unexpected issues quickly

4
1. Show the key data points we track
2. Show how our metrics dashboard helps us monitor that data
3. Share ideas on how to find better data in NG and beyond
4. Spark discussion on improved visibility for CF admins and customers.
Goals of this lightning talk
We are looking to get better at this, and help the community get better as well.

5
1. The key data

6
What are the important metrics?
Data that can be
tracked over time to see
trends and behaviors
Data that can help
us predict problems
before they happen
DEAs and apps health
 Memory reserved as a proportion of the
memory available
General health of all components
 Health of the virtual machines
 Status of the processes running on them
Database nodes and services
 Number of provisioned services against
capacity available
At the PaaS layer, that means:

7
 Deliver continuous
availability in the cloud
 Proactively solve
problems rather than
react to them
 Understand the behavior
of the system to
automate it
Why do we need metrics?

8
 NATS message bus
• Discover the components to interrogate
• Best for dynamically changing data
Where can we find them?
 Cloud Controller database (CCDB)
• Longer lived data that isn’t in the varz endpoints

9
2. Monitoring that data

10
1. Views of component health
2. Resource usage details
3. Ongoing growth trends
4. Access to logs and raw varz
5. Email notifications
Our metrics dashboard provides…

11
 Components nearing capacity or failure
 Already failed components
 Out of control apps and noisy users
 Active/inactive users and apps
 Growth trends and runtime/service adoption
It helps us find (and fix) problems
It helps us see patterns

12
User and app trends
There is also one unauthenticated page for high level stats

13
DEA list

14
DEA details

15
Service node list

16
Service node details

17
User list

18
User details

19
App list

20
App details

21
Log list

22
Log details

23
Email notifications

24
3. Finding and acting on better data

25
 NG provides granular user/org/space views…
• This enables better BSS potential in terms of QoS and departmental billing
 …But we lost user and app data linkages from the health manager
• Can’t see what DEA my app resides on (not currently enabled in our NG version)
• Can’t see how many apps a user has (replaced by orgs and spaces, but still
valuable to trace)
• See https://github.com/cloudfoundry/cloud_controller_ng/issues/81
 We’d like to restore that data, either surface it
• in varz endpoints (dynamic data, preferred) or
• CC_DB (static data, could be a security concern)
Let’s resolve gaps in data captured from NG

26
 Detect errors in applications that are traceable to users/orgs
• Preemptively reach out to them to see if they need help
• Think customer service and proactive support!
• Can we hook into to BOSH or Jenkins for automation?
 Automate (and expand links to the IaaS and SaaS stacks)
• Self healing systems (out of disk, move apps)
• Self scaling systems (detect when nearing thresholds)
• Evolving topologies (replace unused service nodes with popular ones)
Let’s begin to link metrics to automation

27
 Admins are the primary beneficiary right now
• But data is almost completely read only
• Should we provide UAA based tiers of access to admins?
 Others can and should benefit
• Customers
• End users
• Developers
• Management
• Executives, line of business owners
• Finance
Let’s expand the broadcast of metrics to more users

28
Thanks!

29
The metrics dashboard innovators
Chris Peters Russell Boykin
Doug Davis Wei Feng

30
We’re hiring!
Search Jobs at IBM by:
SmartCloud Application Services

31