This document describes the design and implementation of a service monitoring console within a service oriented architecture framework. It discusses using Cassandra for data collection and storage, implementing message queues for data collection, and using OpenTSDB as a time series database. It also provides an implementation schedule and references several technologies including MapReduce, Cassandra, Turmeric, and Google Web Toolkit.

1. Design and Implementation of a Service
Monitoring Console within a Service Oriented
Architecture Framework
Arvind Krishnaa .J
31508104017
Guided By
Dr. Chitra Babu
HOD/CSE
SSN College of Engineering
First Review - 23rd February, 2012

2. OLAP2Db Interface
Data from each data center
CAL log accumulated in
PHX colo.
Data in PHX colo stored in
centralized data cube.
Data from cube pushed onto
relational DB, by OLAP2Db
scripts.
Figure: OLAP2Db Interface with Data loss occurs here.
cube delay

3. Minimizing the data loss
Run CAL cube data availability scripts.
CALDb records were found to be consistent with
static CAL data.
Data loss was isolated to partioned relational
database.
2/3 cron jobs pushing data to the database were
idle.
Once the cron jobs were restarted, the relational
database synced the data accurately.

4. Minimizing the data loss
Figure: Before cron jobs were Figure: After cron jobs were
activated activated

5. SMC Vs TMC : Feature Comparison

6. Turmeric Monitoring Console
Figure: Java Class Hierarchy

7. Data Collection Possibilities
A real-time Cassandra
cluster where metrics are
stored.
Metrics reader class reads
data from each cluster.
Offline aggregator
accumulates the data.
Metrics writer pushes back
the data to each node.
Previously stored data in
Figure: Using Cassandra Storage each node is deleted
No single point of failure

8. Data Collection Possibilities
Implement a message queue
on each server.
This message queue will
push the metric data onto a
central database.
This database acts as an
aggregated storage provider.
Single point of failure, at
the centralized database.
Figure: Using Message Queue

9. Minimizing the data loss
Using OpenTSDB
Distributed, scalable high performance time
series database, implemented over HBase.
Allows simple storage and retrieval of metrics.
Easy to graph the trends.
Difficult to set up hardware components.
Complicated installation procedure.

10. Data Collection Workflow
Figure: Activity diagram representing the data collection workflow

11. Implementation Schedule

12. References
[1]Jeffrey Dean, Sanjay Ghemawat, Google Inc., MapReduce:
Simplified Data Processing on Large Clusters, In Sixth Symposium
on Operating System Design and Implementation(OSDI’04), San
Francisco, CA, December, 2004
[2] Eben Hewitt, Cassandra: The Definitive Guide, OReilly
Publications, November 2010.
[3] eBay Open Source Project, Turmeric SOA platform,
http:
//www.ebayopensource.org/index.php/Turmeric/HomePage

13. References
[4] eBay Open Source Project, Documentation of Turmeric SOA
platform,
https://www.ebayopensource.org/wiki/display/
TURMERICDOC110GA/Turmeric+Documentation+Overview
[5] eBay Open Source Project, Turmeric Source Code,
http://www.github.com/ebayopensource
[6] Internal eBay documentation
[7] Google Web Toolkit,
http://code.google.com/webtoolkit
[8] Apache Cassandra,
http://cassandra.apache.org/