This document summarizes Ganglia, an open-source scalable distributed monitoring system. It introduces Ganglia's architecture which uses gmond daemons to gather metrics from servers and gmetad daemons to aggregate metrics. Metrics are stored and visualized using RRDTool and presented through a web frontend. Potential problems like a central node bottleneck are discussed along with solutions like distributing monitoring across multiple gmetad instances and storing databases in RAM.

1. Summary Report
Yupeng Chen
2012.8.15

2. Introduce to Ganglia
Problem & Solution
My Harvest

3. Introduction and overview
• Scalable distributed monitoring system for high-performance
computing systems
• XML - data representation
• XDR(XML-Data Reduced) - compact, portable data transport
• RRDTool - data storage and visualization
• PHP - web frontend interface

4. Ganglia Architecture
• Gmond - Ganglia Monitoring Daemon
Metric gathering agent installed on individual servers
• Gmetad - Ganglia Meta Daemon
Metric aggregation agent installed on specificservers
• Apache(Nginx + php5-fpm) web frontend
Metric presentation and analysis server
• Model - Multicast or Unicast

5. Multicast – All gmond nodes are capable of listening to and
reporting on the status of the entire cluster

6. Unicast - Send the localhost monitoring data to specific
machines, cross-network segment is allowed.

7. VMHTest Clustermaster
gmond
gmond
slave1
gmond
slave2
gmond
slave3
gmond
slave4
gmond
zookeeper
XDR / UDP
Omnilab Cluster
dev gmond
ganglia
gmond
omnilab gmond
XDR / UDP
gmetad
pollXML / TCP
XML / TCP
poll
rrdtool
web-frontend
push
push

8. Gmond – Metric Gathering Agent
• Built-in metrics
– Various CPU, Network I/O, Disk and Memory
• Extensible
– Gmetric – Out-of-process utility capable of invoking
command line based metric gathering scripts
– Loadable modules capable of gathering multiple
metrics or using advanced metric gathering APIs
• Work with Hadoop & HBase
– NameNode, DataNode, JobTracker, TaskTracker, etc.
– JVM, rpc, etc.

9. • Based on open standard
• Low per-node overheads and high concurrency
• High reliability and independence : failover
• Data storage and presentation : RRDTool
• Ported to various different platforms
(Linux, FreeBSD, Solaris, others)
Feature & Advantage

10. Problems & Bottlenecks
• Overhead evaluation of central node
• CPU ( XDR XML )
• network I/O
• disk I/O
• Gmetad RRD write bottleneck
• Every metric has a corresponding a data file (*.rrd )
• Write a large number of small files at the same time
20 nodes,
for each has 500+ metrics
10000+ times read/write
requests in a few seconds

11. Solutions
• Distributed monitoring system
• Separated clusters into small pieces
• Multiple Gmetad

12. • Datebase should be placed in RAM
• tmpfs
• RAID 0
• Reduce the sampling frequency
Solutions

13. My Harvest
• Dev - Ops
• Linux
• git
• wiki
• Cloud computing
• OpenStack
• Virtualization
• BigData
• Hadoop
• HBase

14. Thank youThank youThank youThank you
WPS Office
Make Presentation much more fun