slides (PPT)


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Na tym rysunku mamy schemat architektury. WP2 i WP3 – to nasze nowe serwisy i uslugi. Trzeba będzie zdefiniować interfejsy między nimi.
  • Data gathering, processing and interpretation in health and environment is driven by modern technology. The geographical distribution of data generators (Medical scanners, Patient Databases, Environment-Actuators and Monitors) requires and integrated approach to access and process these data. With the advent of Grid-based technology many new possibilities for data presentation and fast decision making come within reach. It is the goal of the CrossGrid project to explore, incorporate adapt and validate this technology for those application areas that require interactive access to resources, be it Databases, Super Computers, Visualisation Engines, Medical Scanners, or environmental data input devices
  • slides (PPT)

    1. 1. The CrossGrid Project Marcel Kunze, FZK representing the X#-Collaboration
    2. 2. Main Objectives <ul><li>New category of Grid enabled applications </li></ul><ul><ul><li>Computing and data intensive </li></ul></ul><ul><ul><li>Distributed </li></ul></ul><ul><ul><li>Interactive, near real time response (a person in a loop) </li></ul></ul><ul><ul><li>Layered </li></ul></ul><ul><li>New programming tools </li></ul><ul><li>Grid more user friendly, secure and efficient </li></ul><ul><li>Interoperability with other Grids </li></ul><ul><li>Implementation of standards </li></ul>
    3. 3. CrossGrid Collaboration Poland: Cyfronet & INP Cracow PSNC Poznan ICM & IPJ Warsaw Portugal: LIP Lisbon Spain: CSIC Santander Valencia & RedIris UAB Barcelona USC Santiago & CESGA Ireland: TCD Dublin Italy: DATAMAT Netherlands: UvA Amsterdam Germany: FZK Karlsruhe TUM Munich USTU Stuttgart Slovakia: II SAS Bratislava Greece: Algosystems Demo Athens AuTh Thessaloniki Cyprus: UCY Nikosia Austria: U.Linz 21 institut e s 11 countries
    4. 4. IST Grid Project Space Applications Middleware & Tools Underlying Infrastructu re s - Links with European National efforts - Links with US projects (GriPhyN, PPDG, iVDGL,…) GRIDLAB GRIA EGSO DATATAG CROSSGRID DATAGRID GRIP EUROGRID DAMIEN Science Industry / business
    5. 5. Collaboration with other # Projects <ul><li>Objective – Exchange of </li></ul><ul><ul><li>Information </li></ul></ul><ul><ul><li>Software components </li></ul></ul><ul><li>Partners </li></ul><ul><ul><li>DATAGRID </li></ul></ul><ul><ul><li>DATATAG </li></ul></ul><ul><ul><li>GRIDLAB </li></ul></ul><ul><ul><li>EUROGRID and GRIP </li></ul></ul><ul><li>GRIDSTART </li></ul><ul><li>Participation in GGF </li></ul>
    6. 6. Project Phases M 1 - 3: requirements definition and merging M 4 - 12: first development phase: design, 1st prototypes, refinement of requirements M 13 -24: second development phase: integration of components, 2nd prototypes M 25 -32: third development phase: complete integration, final code versions M 33 -36: final phase: demonstration and documentation
    7. 7. Structure Overview Network infrastructure, archivers, HPC/HPV systems, Labour instruments, etc. Local domain services Protocols, Authentication, Authorization, Access policy, Resource management, etc. Remote Data Access Optimization Monitoring Schedulers Roaming Access Portals Interactive simulation and visualisation of a biomedical system Flooding crisis team support Distributed Data Analysis in High Energy Physics Weather forecast and air pollution modeling FABRIC INFRASTRUCTURE Grid SERVICES TOOLS APPLICATIONS GLOBUS TOOLKIT , Condor-G, ... DATAGRID SET OF TOOLS Grid Visualization Kernel Benchmarks
    8. 8. CrossGrid Architecture Applications And Supporting Tools Applications Development Support Grid Common Services Grid Visualisation Kernel Data Mining on Grid Interactive Distributed Data Access Globus Replica Manager Roaming Access Grid Resource Management Grid Monitoring MPICH-G Distributed Data Collection User Interaction Service DataGrid Replica Manager Datagrid Job Manager GRAM GSI Replica Catalog GASS MDS GridFTP Globus-IO Resource Manager CPU Resource Manager Resource Manager Secondary Storage Resource Manager Scientific Instruments (Medical Scaners, Satelites, Radars) Resource Manager Detector Local High Level Trigger Resource Manager VR systems (Caves, immerse desks) Resource Manager Visualization tools Optimization of Data Access Tertiary Storage Local Resources Biomedical Application Portal Performance Analysis MPI Verification Metrics and Benchmarks HEP High LevelTrigger Flood Application HEP Interactive Distributed Data Access Application HEP Data Mining on Grid Application Weather Forecast application
    9. 9. Layered Structure Interactive and Data Intensive Applications (WP1) <ul><li>I nteractive simulation and visualization of </li></ul><ul><li>a biomedical system </li></ul><ul><li>Flooding crisis team support </li></ul><ul><li>Distributed data analysis in HEP </li></ul><ul><li>Weather forecast and air pollution modeling </li></ul>Grid Application Programming Environment (WP2) <ul><li>MPI code debugging and </li></ul><ul><li>verification </li></ul><ul><li>Metrics and benchmarks </li></ul><ul><li>Interactive and semiautomatic </li></ul><ul><li>performance evaluation tools </li></ul>Grid Visualization Kernel Data Mining New CrossGrid Services (WP3) Globus Middleware Fabric Infrastructure (Testbed WP4) Data G rid GriPhyN ... Services HLA <ul><li>Portals and roaming access </li></ul><ul><li>Grid resource management </li></ul><ul><li>Grid monitoring </li></ul><ul><li>Optimization of data access </li></ul>
    10. 10. Scope of Applications <ul><li>Applications in health and environment </li></ul><ul><ul><li>Data federation, processing and interpretation in geographically distributed locations </li></ul></ul><ul><ul><li>Fast, interactive decision making </li></ul></ul><ul><li>Interactive access to distributed </li></ul><ul><ul><li>Databases </li></ul></ul><ul><ul><li>Super computers and High Performance Clusters </li></ul></ul><ul><ul><li>Visualisation engines </li></ul></ul><ul><ul><li>Medical scanners </li></ul></ul><ul><ul><li>Environmental data input devices </li></ul></ul>
    11. 11. Application Requirements <ul><li>High quality presentation </li></ul><ul><li>High frame rate </li></ul><ul><li>Intuitive interaction </li></ul><ul><li>Real-time response </li></ul><ul><li>Interactive algorithms </li></ul><ul><li>High performance computing and networking </li></ul><ul><li>Distributed resources and data </li></ul>
    12. 12. Role of Network Latency Communication delay and rendering delay are negligible
    13. 13. CrossGrid Application Development (WP1) <ul><li>Interactive simulation and visualisation of a biomedical system </li></ul><ul><ul><li>Grid-based system for pre-treatment planning in vascular interventional and surgical procedures through real-time interactive simulation of vascular structure and flow. </li></ul></ul><ul><li>Flooding crisis team support </li></ul><ul><li>Distributed interactive data analysis in HEP </li></ul><ul><ul><li>F ocus on LHC experiments (ALICE, ATLAS, CMS and LHCb) </li></ul></ul><ul><li>Weather forecast and air pollution modelling </li></ul><ul><ul><li>Porting distributed/parallel codes on Grid </li></ul></ul><ul><ul><li>Coupled Ocean/Atmosphere Mesoscale Prediction System </li></ul></ul><ul><ul><li>STEM-II Air Pollution Code </li></ul></ul>
    14. 14. Interactive Simulation and Visualisation of a Biomedical System <ul><li>Grid-based prototype system for treatment planning in vascular interventional and surgical procedures through near real-time interactive simulation of vascular structure and flow. </li></ul><ul><li>The system will consist of a distributed near real-time simulation environment, in which a user interacts in Virtual Reality (VR) and other interactive display environments. </li></ul><ul><li>A 3D model of the arteries, derived using medical imaging techniques, will serve as input to a simulation environment for blood flow calculations. </li></ul><ul><li>The user will be allowed to change the structure of the arteries, thus mimicking an interventional or surgical procedure. </li></ul>
    15. 15. Current Situation Observation Diagnosis & Planning Treatment
    16. 16. Experimental Setup
    17. 17. Simulation Based Planning and Treatment Angio w/ Fem-Fem & Fem-Pop AFB w/ E-S Prox. Anast. Angio w/ Fem-Fem AFB w/ E-E Prox. Anast. Preop Alternatives
    18. 18. VR-Interaction
    19. 19. Flood Crisis Prevention <ul><li>Support system for establishment and operation of Virtual Organization for Flood Forecasting associating a set of individuals and institutions involved in flood prevention and protection. </li></ul><ul><li>The system will employ a Grid technology to seamlessly connect together the experts, data and computing resources needed for quick and correct flood management decisions. </li></ul><ul><li>The main component of the system will be a highly automated early warning system based on hydro-meteorological (snowmelt) rainfall-runoff simulations. </li></ul><ul><li>System will integrate the advanced communication techniques allowing the crisis management teams to consult the decisions with various experts. The experts will be able to run the simulations with changed parameters and analyze the impact. </li></ul>
    20. 20. Virtual Organization for Flood Forecasting Storage systems databases surface automatic meteorological and hydrological stations systems for acquisition and processing of satellite information meteorological radars <ul><li>External sources of information </li></ul><ul><li>Global and regional centers GTS </li></ul><ul><li>EUMETSAT and NOAA </li></ul><ul><li>Hydrological services of other countries </li></ul>Data sources meteorological models hydrological models hydraulic models HPC, HTC Grid infrastructure Flood crisis teams <ul><li>meteorologists </li></ul><ul><li>hydrologists </li></ul><ul><li>hydraulic engineers </li></ul>Users <ul><li>river authorities </li></ul><ul><li>energy </li></ul><ul><li>insurance companies </li></ul><ul><li>navigation </li></ul><ul><li>media </li></ul><ul><li>public </li></ul>
    21. 21. Flood Crisis Prevention V á h River Pilot Site <ul><li>Water stages/discharges in the real time operating hydrological stations </li></ul><ul><li>Mapping of the flooded areas </li></ul>V á h River Catchment Area: 19700km 2 , 1/3 of Slovakia (Inflow point) Nosice Strečno (Outflow point ) Pilot Site Catchment Area: 2500km 2 (above Strečno : 5500km 2 )
    22. 22. F lood S imulations R esults Flow + water depths
    23. 23. Distributed Analysis in High Energy Physics <ul><li>Challenging points </li></ul><ul><ul><li>Access to large distributed databases in the Grid </li></ul></ul><ul><ul><li>Development of distributed data-mining techniques </li></ul></ul><ul><ul><li>Definition of a layered application structure </li></ul></ul><ul><ul><li>Integration of user-friendly interactive access (based on PROOF) </li></ul></ul><ul><li>F ocus on LHC experiments (ALICE, ATLAS, CMS and LHCb) </li></ul>
    24. 24. PROOF Local Remote Selection Parameters Procedure Proc.C Proc.C Proc.C Proc.C Proc.C PROOF CPU CPU CPU CPU CPU CPU TagDB RDB DB1 DB4 DB5 DB6 DB3 DB2
    25. 25. Weather Forecast and Air Pollution Modeling <ul><li>Integration of distributed databases into Grid </li></ul><ul><li>Migration of data mining algorithms to Grid </li></ul><ul><li>Porting distributed atmospheric & wave models to Grid </li></ul><ul><li>Porting parallel codes for air quality models to Grid </li></ul><ul><li>Integration, testing and demonstration of the application in the testbed environment </li></ul>
    26. 26. COAMPS Coupled Ocean/Atmosphere Mesoscale Prediction System: Atmospheric Components <ul><li>Complex Data Quality Control </li></ul><ul><li>Analysis: </li></ul><ul><ul><li>Multivariate Optimum Interpolation Analysis of Winds and Heights </li></ul></ul><ul><ul><li>Univariate Analyses of Temperature and Moisture </li></ul></ul><ul><ul><li>Optimum Interpolation Analysis of Sea Surface Temperature </li></ul></ul><ul><li>Initialization: </li></ul><ul><ul><li>Variational Hydrostatic Constraint on Analysis Increments </li></ul></ul><ul><ul><li>Digital Filter </li></ul></ul><ul><li>Atmospheric Model: </li></ul><ul><ul><li>Numerics: Nonhydrostatic, Scheme C, Nested Grids, Sigma-z </li></ul></ul><ul><ul><li>Physics: Convection, Explicit Moist Physics, Radiation, Surface Layer </li></ul></ul><ul><li>Features: </li></ul><ul><ul><li>Globally Relocatable (5 Map Projections) </li></ul></ul><ul><ul><li>User-Defined Grid Resolutions, Dimensions, and Number of Nested Grids </li></ul></ul><ul><ul><li>6 or 12 Hour Incremental Data Assimilation Cycle </li></ul></ul><ul><ul><li>Can be Used for Idealized or Real-Time Applications </li></ul></ul><ul><ul><li>Single Configuration Managed System for All Applications </li></ul></ul><ul><ul><li>Operational at: </li></ul></ul><ul><ul><ul><li>7 Areas, Twice Daily, using 81/27/9 km or 81/27 km grids </li></ul></ul></ul><ul><ul><ul><li>Forecasts to 72 hours </li></ul></ul></ul><ul><ul><li>Operational at all Navy Regional Centers (w/GUI Interface) </li></ul></ul>
    27. 27. Status Quo … Quo Vadis ? <ul><li>Current state (briefly) </li></ul><ul><li>S imulation done on a single system or local clusters </li></ul><ul><li>V isualisation on a single system, locally </li></ul><ul><li>What we are going to achieve </li></ul><ul><li>HPC, HTC, HPV in geographically distributed environment </li></ul><ul><li>I mproved interaction with the end user </li></ul><ul><li>N ear real time simulations </li></ul><ul><li>D ifferent visualisation equipments (adaptive according to the end-user needs), like </li></ul><ul><ul><li>PDA </li></ul></ul><ul><ul><li>W orkstations </li></ul></ul><ul><ul><li>VR studio (e.g. CAVE) </li></ul></ul>
    28. 28. Grid Application Programming Environment (WP2) <ul><li>MPI code debugging and verification </li></ul><ul><li>Metrics and benchmarks </li></ul><ul><li>Interactive and semiautomatic performance evaluation tools </li></ul><ul><li>Specify, develop, integrate, test tools for HPC and HTC applications on the Grid </li></ul>
    29. 29. New Grid Services and Tools (WP3) <ul><li>Portals and roaming access </li></ul><ul><li>Grid resource management </li></ul><ul><li>Grid monitoring </li></ul><ul><li>Optimisation of data access </li></ul><ul><li>Objectives </li></ul><ul><ul><li>To develop interactive compute- and data-intensive applications </li></ul></ul><ul><ul><li>To develop user-friendly Grid environments </li></ul></ul><ul><ul><li>To offer easy access to the applications and Grid </li></ul></ul><ul><ul><li>To have reasonable trade-off between resource usage efficiency and application speedup </li></ul></ul><ul><ul><li>To support management issues while accessing resources </li></ul></ul>
    30. 30. International Testbed Organisation (WP4) 15 si t e s <ul><li>Testbed setup & incremental evolution </li></ul><ul><li>Integration with DataGrid </li></ul><ul><li>Infrastructure support </li></ul><ul><li>Verification & quality control </li></ul>Auth Thessaloniki U v Amsterdam FZK Karlsruhe TCD Dublin U A Barcelona LIP Lisbon CSIC Valencia CSIC Madrid USC Santiago CSIC Santander DEMO Athens UCY Nikosia CYFRONET Cracow II SAS Bratislava PSNC Poznan ICM & IPJ Warsaw
    31. 31. Summary <ul><li>Layered structure of all X# applications </li></ul><ul><li>Reuse of SW from DataGrid and other # projects </li></ul><ul><li>Globus as the bottom layer of the middleware </li></ul><ul><li>Heterogeneous computer and storage systems </li></ul><ul><li>Distributed development and testing of SW </li></ul><ul><ul><li>12 partners in applications </li></ul></ul><ul><ul><li>14 partners in middleware </li></ul></ul><ul><ul><li>15 partners in testbeds </li></ul></ul>
    32. 32. 1980s: Internet 1990s: Web 2000s: Grid <ul><li>Where do we need to get to ? </li></ul><ul><ul><li>Applications to support an “e-society” (“Cyber-Infrastructure”) </li></ul></ul><ul><ul><li>An international Grid infrastructure which hides the complexities from the users (“Invisible Computing”) </li></ul></ul><ul><ul><li>A powerful and flexible network infrastructure </li></ul></ul><ul><li>Where do we need to invest ? </li></ul><ul><ul><li>Applications targeted at realistic problems in “e-science” </li></ul></ul><ul><ul><li>Prototypes of Grid infrastructures </li></ul></ul><ul><ul><li>Maintain and improve the GEANT network </li></ul></ul><ul><li>Expression of Interest for EU FP6 program: </li></ul><ul><ul><li>“ Enabling Grids and e-Science in Europe (EGEE)” </li></ul></ul>Grid-enabled Applications Prototype Grid Infrastructures G èant: World Class Networking
    33. 33. EGEE Project Space Applications Middleware & Tools Underlying Infrastructu re s GRIDLAB GRIA EGSO DATATAG CROSSGRID DATAGRID GRIP EUROGRID DAMIEN Science Industry / business Enabling Grids and E-Science in Europe (EGEE)
    34. 34. First results of EGEE Brainstorming C Jones, D O Williams, et al