Design and Implementation of a ServiceMonitoring Console within a Service Oriented Architecture Framework Arvind Krishnaa .J 31508104017 Guided By Dr. Chitra Babu HOD/CSE SSN College of Engineering First Review - 23rd February, 2012
OLAP2Db Interface Data from each data center CAL log accumulated in PHX colo. Data in PHX colo stored in centralized data cube. Data from cube pushed onto relational DB, by OLAP2Db scripts. Figure: OLAP2Db Interface with Data loss occurs here. cube delay
Minimizing the data loss Run CAL cube data availability scripts. CALDb records were found to be consistent with static CAL data. Data loss was isolated to partioned relational database. 2/3 cron jobs pushing data to the database were idle. Once the cron jobs were restarted, the relational database synced the data accurately.
Minimizing the data loss Figure: Before cron jobs were Figure: After cron jobs were activated activated
Turmeric Monitoring Console Figure: Java Class Hierarchy
Data Collection Possibilities A real-time Cassandra cluster where metrics are stored. Metrics reader class reads data from each cluster. Oﬄine aggregator accumulates the data. Metrics writer pushes back the data to each node. Previously stored data in Figure: Using Cassandra Storage each node is deleted No single point of failure
Data Collection Possibilities Implement a message queue on each server. This message queue will push the metric data onto a central database. This database acts as an aggregated storage provider. Single point of failure, at the centralized database. Figure: Using Message Queue
Minimizing the data loss Using OpenTSDB Distributed, scalable high performance time series database, implemented over HBase. Allows simple storage and retrieval of metrics. Easy to graph the trends. Diﬃcult to set up hardware components. Complicated installation procedure.
Data Collection Workﬂow Figure: Activity diagram representing the data collection workﬂow
References Jeﬀrey Dean, Sanjay Ghemawat, Google Inc., MapReduce: Simpliﬁed Data Processing on Large Clusters, In Sixth Symposium on Operating System Design and Implementation(OSDI’04), San Francisco, CA, December, 2004  Eben Hewitt, Cassandra: The Deﬁnitive Guide, OReilly Publications, November 2010.  eBay Open Source Project, Turmeric SOA platform, http: //www.ebayopensource.org/index.php/Turmeric/HomePage
References  eBay Open Source Project, Documentation of Turmeric SOA platform, https://www.ebayopensource.org/wiki/display/ TURMERICDOC110GA/Turmeric+Documentation+Overview  eBay Open Source Project, Turmeric Source Code, http://www.github.com/ebayopensource  Internal eBay documentation  Google Web Toolkit, http://code.google.com/webtoolkit  Apache Cassandra, http://cassandra.apache.org/