Embedding Existing Heterogeneous Monitoring Techniques into
a Lightweight, Distributed Integration Platform

ADVCOMP 2009 ...
DLR                           Research Institution
German Aerospace Center       Space Agency
                            ...
Motivation
Monitoring in HPC/CAE Environments

  In CAE environments, a lot of software components do monitoring!




    ...
Motivation
Different Purposes for Monitoring

  Resource Management
      Monitoring Resources like Load, Licenses, …
    ...
Motivation


  A typical setup has 3 Software Components that are monitoring
  All with different types of Sensors
  All w...
Monitoring Techniques
Ganglia (gmetric)

  Basically, a Sensor-Script for Ganglia just reports a simple value.
  May be wr...
Monitoring Techniques
Grid Engine (Load Sensors)

  We use a lot of scripts for the Grid Engine to make the scheduler awar...
Realizing the Monitoring Environment


Requirements
        Collect heterogeneous monitoring data from distributed hosts
 ...
Realizing the Monitoring Environment
Unify the Heterogeneous Monitoring Data

  Wrapping each sensor script with an adapte...
Realizing the Monitoring Environment
Integration Platform

   Unified environment with base software components
         D...
Realizing the Monitoring Environment
Remote Component Environment (RCE)

   RCE is such an integration platform


Characte...
Realizing the Monitoring Environment
Remote Component Environment (RCE)

  Distributed deployment scenario of RCE




    ...
Realizing the Monitoring Environment


  Adapter concept




  Distributed integration platform RCE




                  ...
Monitoring Environment
On Top of RCE




                                                                                 ...
Outlook


  Implement clients using the monitoring environment
       GUI
       Alert system
       Resource management s...
Questions?


                                                                                                             ...
Upcoming SlideShare
Loading in …5
×

Embedding Existing Heterogeneous Monitoring Techniques into a Lightweight, Distributed Integration Platform

3,593 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,593
On SlideShare
0
From Embeds
0
Number of Embeds
2,352
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Embedding Existing Heterogeneous Monitoring Techniques into a Lightweight, Distributed Integration Platform

  1. 1. Embedding Existing Heterogeneous Monitoring Techniques into a Lightweight, Distributed Integration Platform ADVCOMP 2009 (October 12th 2009, Sliema) Jan Niemann, GNS Systems Doreen Seider, German Aerospace Center (DLR) Folie 1 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into … > 12.10.2009
  2. 2. DLR Research Institution German Aerospace Center Space Agency Project Management Agency GNS Systems UNIX/Windows System Management High-Performance-Computing Engineering Data Management Software Development Folie 2 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  3. 3. Motivation Monitoring in HPC/CAE Environments In CAE environments, a lot of software components do monitoring! Folie 3 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  4. 4. Motivation Different Purposes for Monitoring Resource Management Monitoring Resources like Load, Licenses, … Examples: Grid Engine, PBS, Platform LSF Health Management Monitoring Resources like Load, Heat, SMART-Status Examples: Ganglia, Nagios Resource Planning Esp. License Monitoring Examples: We have created License Monitoring and Accounting Solutions for some of our customers. Folie 4 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  5. 5. Motivation A typical setup has 3 Software Components that are monitoring All with different types of Sensors All with different administrative and user interfaces Some Resources are monitored more than once (Load, Licenses) Let‘s do it once! But allow reusability of existing solutions! Folie 5 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  6. 6. Monitoring Techniques Ganglia (gmetric) Basically, a Sensor-Script for Ganglia just reports a simple value. May be written in sh, perl, python, … Whatever you/your Administrators like! #!/bin/sh echo 60.3 # gmetric --name temperature --value > $(cputemp) --type float --units Celsius There are a lot of these scripts available on the internet for free. Measure Infiniband performance Temperatures via IPMI … Folie 6 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  7. 7. Monitoring Techniques Grid Engine (Load Sensors) We use a lot of scripts for the Grid Engine to make the scheduler aware of license usage. A bit more complicated, but again: may be written in any language. #!/bin/sh while read x ; do echo begin echo “global:lic_abq_free:5“ echo “global:lic_abq_used:8“ echo “global:lic_abq_total:13“ echo end done Platform LSF and PBS use similar mechanics. Administrators know how to write these scripts. We think that acceptability is a key factor for success! Folie 7 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  8. 8. Realizing the Monitoring Environment Requirements Collect heterogeneous monitoring data from distributed hosts Provide them centralized via one unified interface Reuse existing sensor scripts Solution Unify accumulated data by wrapping the scripts with adapters Embed the scripts (i.e. the adaptors) in an integration platform Make use of the platform‘s infrastructure to collect the data Realize an embedded component providing the data Folie 8 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  9. 9. Realizing the Monitoring Environment Unify the Heterogeneous Monitoring Data Wrapping each sensor script with an adapter which Executes the script Transforms the data Provides them via one unified interface Folie 9 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  10. 10. Realizing the Monitoring Environment Integration Platform Unified environment with base software components Data management, distribution, privilege management, ... Integrating diverse and specific applications Benefit Integrated applications simply use already realized software components Enables basically independent applications to Interact Operate on same data Base software components are realized only once Folie 10 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  11. 11. Realizing the Monitoring Environment Remote Component Environment (RCE) RCE is such an integration platform Characteristics Component-based Distributed OSGi/Eclipse RCP Developed by German Aerospace Center (DLR) – http://www.dlr.de Fraunhofer SCAI – http://scai.fraunhofer.de http://sourceforge.net/projects/rcenvironment Folie 11 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  12. 12. Realizing the Monitoring Environment Remote Component Environment (RCE) Distributed deployment scenario of RCE Folie 12 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  13. 13. Realizing the Monitoring Environment Adapter concept Distributed integration platform RCE Folie 13 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  14. 14. Monitoring Environment On Top of RCE Folie 14 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  15. 15. Outlook Implement clients using the monitoring environment GUI Alert system Resource management system Implement more adapters as needed Folie 15 ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009
  16. 16. Questions? Folie 16 Jan.Niemann@gns-systems.de, Doreen.Seider@dlr.de ADVCOMP 2009 > Jan Niemann, Doreen Seider > Embedding Existing Heterogeneous Monitoring Techniques into ... > 12.10.2009

×