The Grid Observatory Initiative develops a scientific view of the dynamics and usage of globalized IT systems by monitoring and analyzing the EGI grid. The overall goal is to create a full-‐fledged digital curation process, with its four components: preservation, validation, indexation and knowledge building. As the largest non-‐profit globalized system worldwide and with demanding scientific users, the EGI infrastructure is one of the most exciting artificial complex systems. With extensive monitoring facilities already in place, it offers an unprecedented opportunity to observe and to understand the computing practices within the e-‐Science community. Grid and cloud share a common paradigm: they are globalized at a large scale. As such, the data collected and the knowledge built from analyzing EGI concern cloud modeling as well. Ongoing work integrates monitoring data from the StratusLab cloud. The Grid Observatory is an open collaboration, keen to foster dialog and partnerships with others in the relevant areas of computer science and engineering. The Laboratoire de Recherche en Informatique and Laboratoire de l’Accélérateur Linéaire, from CNRS and University Paris-‐Sud, along with the London Imperial College operate data production. The initiative is supported by France-‐Grilles, INRIA and CNRS. A trove of experimental data: www.grid-‐observatory.org The first role of the Grid Observatory is to preserve the monitoring data, normally discarded after operational usage, and to make them available to the wider scientific community. Through its web portal, the Grid Observatory offers public access to a repository of grid traces to observe e-‐Science practice and infrastructure. • EGI provides an accessible approximation of the current and future requirements of e-‐Science users. • Grid status and middleware activity are recorded. These can be explored for a wide range of motivations, from operational usage, e.g. improving performance, to scientific research, e.g. testing classification methods for fault detection. The Grid Observatory follows Tim Berners Lee’s recommendation for Raw Data Now. It exemplifies the Big Data challenges: semantic organization, provenance, interoperability, and next generation analytics. Emerging technologies such as Linked Open Data will be explored to further address those challenges. The Green Computing Observatory The Grid Observatory offers extensive traces of energy consumption. Because green IT is becoming an increasingly urgent need and also because there was no existing EGI monitoring tool, this action has its own name: the Green Computing Observatory. • The traces integrate motherboard-‐level monitoring with information on computing, networking, storage, and cooling. • Acquisition exploits the de facto standards IPMI and Ganglia. • Integration is based on an ontology of IT system measurements, including virtual machines, developed by University Picardie Jules Verne. From applied to fundamental research Research exploiting the monitoring data should demonstrate verifiable and positive impact on production systems. • Beyond-‐power-‐law and non-‐stationary behavior are pervasive. With sequential testing, segmentation and adaptive on-‐line clustering, we advanced fault detection and parsimonious model building. • Efficient autonomic policies must combine a priori knowledge and on-‐line adaptation, but reference interpretations are most often missing. Data-‐driven topic modeling in the spirit of text mining, and heterogeneous data integration with Statistical Relational Learning help to build intelligible representations.
Digital curation The overall goal of the Grid Observatory is to create a full-‐fledged digital curation process, with its four components. Establishing and developing a long-‐term repository of digital assets for current and future references. The Grid Observatory operates since October 2008. It continuously records and publishes various traces. An essential achievement is to cover the complete scope of the grid middleware and users activity, beyond particular aspects such as job lifecycle or failure events, and including for instance logging the Information System (BDII). Providing digital asset search and retrieval facilities to scientific communities through a gateway. The middleware traces are currently made available only in raw format, on a weekly basis. Much remains to be done in the direction of a more semantic organization. The Green Computing Observatory data are organized along an XML schema associated with the measurement ontology. All are available trough the Grid Observatory portal. Tackling the good data creation and management issues, and interoperability, through formal ontology building. The Grid Observatory most often builds on EGI and gLite monitoring, thus benefits from their collective effort of middleware development and EMI standardization. The Green Computing Observatory builds on IPMI and Ganglia. Calibration of IPMI measurements is made possible by PDU (Power Distribution Unit) measurements. The Green Computing Observatory participates in the COST action IC0804 -‐ Energy efficiency in large scale distributed systems. Adding value to data by generating new sources of information and knowledge through semantic, statistical and Machine Learning based inference. The general framework for the Grid Observatory is to turn it into a social intelligence system to pool scientific and engineering expertise, in order to build gradually more integrated models of the European e-‐infrastructures, and to define and validate autonomic-‐oriented policies addressing their operational challenges. More information: • The Green Computing Observatory: a data curation approach for green IT. 9th IEEE Int. Conf. on Dependable, Autonomic and Secure Computing. • The Grid Observatory. 11th IEEE/ACM Int. Symp. on Cluster, Cloud and Grid Computing. Towards Open Linked Data *** Data are accessible on the web through the portal; the only protection implemented is against malicious usage. All formats are machine readable and open: ASCII, XML, SQL, LDIF RDF and Linked RDF are the next step. Selected contributions from the Grid Observatory initiative and its users Fault detection and diagnosis, smart probing. Distributed Monitoring with Collaborative Prediction. 12th IEEE/ACM Int. Symp. on Cluster, Cloud and Grid Computing. Toward Autonomic Grids: Analyzing the Job Flow with Affinity Streaming. 15th ACM SIGKDD Conf. on Knowledge Discovery and Data Mining. Optimization of jobs submission on the EGEE production grid: modeling faults using workload. Journal of Grid Computing , 8(2). Grid models Characterizing e-‐science file access behavior via latent Dirichlet allocation . 4th IEEE/ACM Int. Conf. on Utility and Cloud Computing. Towards non-‐stationary Grid models. Journal of Grid Computing, 9(4). Autonomic Quality of Service and Green Computing Multiobjective reinforcement learning for responsive grids. Journal of Grid Computing 8:3.. Autonomic policy adaptation using decentralized online clustering. 7th IEEE/ACM int. conf. on Autonomic computing.