Key takeaways: How Bright Cluster Manager allows you to monitor HPC and Hadoop clusters How Bright Cluster Manager allows you to monitor public and private clouds How Bright Cluster Manager enables monitoring with alerts and health checks How Bright Cluster Manager enables customized monitoring - including how to incorporate your own monitors This recording (http://hubs.ly/y0JtjX0) includes live-product demonstrations of Bright Cluster Manager.

1. Modernized Monitoring for Clusters
and Clouds of All Types
Ian Lumb
Product Marketing Manager
Bright Topics Webinar
April 15, 2015
RECORDING

2. 2
 Modernized monitoring
 Monitoring HPC and Hadoop clusters
 Monitoring public and private clouds
 Monitoring with alerts and health checks
 Customized monitoring - including how to incorporate
your own monitors
Key Takeaways
RECORDING

3. The Five Essential Strategies
1. Plan to manage the impact of software complexity
2. Plan for scalable growth
3. Plan to manage heterogeneous hardware/software
solutions
4. Be ready for the Cloud
5. Have an answer for the Hadoop question
http://insidehpc.com/2014/05/five-essential-strategies-successful-hpc-clusters/

4. http://insidehpc.com/2014/11/monitoring-hpc-clusters-modernized/

5. 5
The problem with “Toolkits”
 Toolkits — “A patchwork of disparate tools”
• Tools typically used: Ganglia, Nagios, Cfengine, System Imager,
Puppet, Chef, Cobbler, Hobbit, Big Brother, Zabbix, etc.
• Scripts
 Issues with the “toolkit” approach:
• Scripts poorly documented and hard to maintain
• Tools not designed to work together
• Each tool has its own user interface (CLI/GUI)
• Each tool has its own agent and database
Hidden assumptions and biases re: sampling and more
• Tools rarely designed for scale & high performance
• Accelerators and coprocessors often not supported
 Making a collection of unrelated tools work together
• Requires a lot of expertise and scripting
• Rarely leads to a really easy-to-use and scalable solution

6. 6
The problem with “Meta-Toolkits”
 Meta-Toolkits likely to obfuscate
• Assumptions and biases involved in sampling and processing
Was interpolation or extrapolation required?
• Scalability limitations
• Existing capabilities within a specific toolkit
User beware the LCD effect!
• The ongoing burden of management and maintenance
http://insidehpc.com/2014/11/monitoring-hpc-clusters-modernized/

7. 7
Pressing concerns, real implications
 Significant toolkit legacy in HPC
• Use of meta-toolkits escalating
 Hadoop deployments rediscovering
toolkit legacy
• Hadoop monitoring +
{ NAGIOS || Ganglia || ??? }
• Apache Ambari an evolving meta-toolkit
‘Modernized’ monitoring with meta-toolkits?
http://www.hpcwire.com/2014/09/18/modernizing-hpc-cluster-monitoring/
"Those who cannot
remember the past
are condemned to
repeat it“
George Santayana
The Life of Reason, Vol. 1
1905
Hadoop Users: Stop Settling for the Santayana Effect TODAY!
https://www.linkedin.com/pulse/hadoop-users-stop-settling-santayana-effect-today-ian-lumb

8. https://lnkd.in/eymE82J

11. 11
Pressing concerns, real implications
 Significant toolkit legacy in HPC
• Use of meta-toolkits escalating
 Hadoop deployments rediscovering
toolkit legacy
• Hadoop monitoring +
{ NAGIOS || Ganglia || ??? }
• Apache Ambari an evolving meta-toolkit
 OpenStack on track to also
rediscover the toolkit legacy
‘Modernized’ monitoring with meta-toolkits?
http://www.hpcwire.com/2014/09/18/modernizing-hpc-cluster-monitoring/
"Those who cannot
remember the past
are condemned to
repeat it“
George Santayana
The Life of Reason, Vol. 1
1905
Hadoop Users: Stop Settling for the Santayana Effect TODAY!
https://www.linkedin.com/pulse/hadoop-users-stop-settling-santayana-effect-today-ian-lumb

12. http://docs.openstack.org/admin-guide-cloud/content/figures/2/figures/openstack-arch-havana-logical-v1.jpg
OpenStack Architecture (Havana)

13. 13
Pressing concerns, real implications
 Significant toolkit legacy in HPC
• Use of meta-toolkits escalating
 Hadoop deployments rediscovering
toolkit legacy
• Hadoop monitoring +
{ NAGIOS || Ganglia || ??? }
• Apache Ambari an evolving meta-toolkit
 OpenStack on track to also
rediscover the toolkit legacy
‘Modernized’ monitoring with meta-toolkits?
http://www.hpcwire.com/2014/09/18/modernizing-hpc-cluster-monitoring/
"Those who cannot
remember the past
are condemned to
repeat it“
George Santayana
The Life of Reason, Vol. 1
1905
Hadoop Users: Stop Settling for the Santayana Effect TODAY!
https://www.linkedin.com/pulse/hadoop-users-stop-settling-santayana-effect-today-ian-lumb

14. 14
A Better Solution
 Bright Cluster Manager takes a much more fundamental
& integrated approach
• Designed and written from the ground up
• Single cluster management agent provides all functionality
• Single, central database for configuration and monitoring data
• Single UI for ALL cluster management functionality
 Which makes Bright Cluster Manager …
• Extremely easy to use
• Extremely scalable
• Secure & reliable
• Complete
• Flexible
• Maintainable

15. Bright Cluster
Architecture — Monitoring
CMDaemon
head node
node001
node003
node002
data
Cluster
Management
GUI
Cluster
Management
Shell
Web-Based
User Portal
Third-Party
Applications
BMC
BMC
BMCraw data consolidated
data
metrics
metrics
metrics
metrics
metrics

16. 16
Native Metrics for Clusters & Clouds
 Over 160 relating to HPC
• From bare metal to workload managers to apps
Includes accelerators and coprocessors
• On-the-ground and in-the-public-cloud
 Over 400 relating to Hadoop
• From distros, HDFS & YARN to data-platform apps
 Almost 90 relating to OpenStack
• Tenant-specific plus private cloud as-a-whole
 Over 60 relating to Ceph
http://www.brightcomputing.com/Linux-Cluster-Monitoring

19. 19
Monitoring++
 Proactive alert-based monitoring
• Define thresholds for any metric
• Associate actions with thresholds
Actions execute when thresholds exceeded
 Health checks
• Invasive plus dynamic diagnostics
Cluster monitoring vs. health checking: What’s the difference?
http://info.brightcomputing.com/blog/cluster-monitoring-vs.-health-checking-whats-the-difference
http://www.brightcomputing.com/Linux-Cluster-Health

20. 20

23. 23
 Modernized monitoring
 Monitoring HPC and Hadoop clusters
 Monitoring public and private clouds
 Monitoring with alerts and health checks
 Customized monitoring - including how to incorporate
your own monitors
Key Takeaways
RECORDING

24. Q & A
Ian Lumb, ian.lumb@brightcomputing.com
http://www.brightcomputing.com/