Grid Projects in the US (an inevitably incomplete view) Ian Foster Computation Institute Argonne National Lab & University...
Grid Projects in the US Resources Resource Provider Resource Provider Resource Provider
Grid Projects in the US Service  Provider Service  Provider Service  Provider Services Resources Resource Provider
Grid Projects in the US Community Community Community Service  Provider Content Services Resources Resource Provider Softw...
Grid Projects in the US Community Service  Provider Content Services Resources Software Providers Resource Provider
Resource Providers <ul><li>Campus and regional grids </li></ul><ul><ul><li>Purdue, Wisc, UCLA, …, … </li></ul></ul><ul><ul...
Open Science Grid Sites  (5/4/08) +3 in Brazil; 2 in Mexico; 2 in Taiwan; 1 in the UK. Grows by 10-20 per year.
Use by Community CMS ATLAS CDF Local Usage & bugs (unmapped to VO) D0 2,000,000 a week 1,000,000 a week
TeraGrid Participants
Growing User Community Source: TeraGrid Central Database
Growing Usage Source: TeraGrid Central Database 3.95B NUs delivered in CY2007
CY2007 Usage by Discipline 3.95B NUs delivered in CY2007 Molecular  Biosciences 31% Chemistry 17% Physics 17% Astronomical...
Grid Projects in the US <ul><li>For example: </li></ul><ul><li>Build and test service (Wisc) </li></ul><ul><li>Certificate...
caBIG: sharing of infrastructure, applications, and data.  Data Integration! Services & Cancer Biology Globus
caBIG Under the Covers NCICB Research Center Grid-Enabled Client Research Center Tool 1 Tool 2 Tool 3 Tool 4 Grid Data Ser...
LIGO Data Grid Birmingham • Replicating >1 Terabyte/day to 8 sites 770 TB replicated to date: >120 million replicas MTBF =...
Grid Projects in the US <ul><li>For example: </li></ul><ul><li>Earth System Grid </li></ul><ul><li>Children’s Oncology Gri...
Earth System Grid Main ESG Portal CMIP3 (IPCC AR4) ESG Portal <ul><li>198 TB of data at four locations </li></ul><ul><li>1...
SCEC Community  Modeling Environment Pathway Instantiations Knowledge Base Ontologies Curated taxonomies, Relations & cons...
Seismic Hazard Analysis <ul><li>Defn:  Max. intensity of shaking expected at a site during a fixed time interval </li></ul...
SCEC Computations & Grid <ul><li>Prepare input to Pathway2    wave propagation code  </li></ul><ul><li>Pathway2PGV convert...
Children’s Oncology Grid and MEDICUS Globus
Grid Projects in the US Community Service  Provider Content Services Resources Resource Provider Software Providers
Software Providers <ul><li>Globus  [GT4.2 released July 2, 2008] </li></ul><ul><ul><li>GRAM, GridFTP, MDS, RLS, DRS, … </l...
Virtual Data Toolkit (VDT) Software Release Process VDT components over time: built for 15 Linux Versions Development & te...
Creating Services: Introduce and gRAVI  <ul><li>Introduce </li></ul><ul><ul><li>Define service </li></ul></ul><ul><ul><li>...
Composing  Services Globus
Service Discovery: Registries Globus
Challenges Community Community Community Service  Provider Content Services Resources Resource Provider Software Providers...
The Future <ul><li>NSF eXtreme Digital (XD) solicitation </li></ul><ul><ul><li>Aka “TeraGrid III” </li></ul></ul><ul><li>D...
Upcoming SlideShare
Loading in...5
×

Grid Projects In The US July 2008

1,400

Published on

A talk given at the HPC 2008 meeting in Cetraro, Italy

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,400
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
34
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Grid Projects In The US July 2008

    1. 1. Grid Projects in the US (an inevitably incomplete view) Ian Foster Computation Institute Argonne National Lab & University of Chicago
    2. 2. Grid Projects in the US Resources Resource Provider Resource Provider Resource Provider
    3. 3. Grid Projects in the US Service Provider Service Provider Service Provider Services Resources Resource Provider
    4. 4. Grid Projects in the US Community Community Community Service Provider Content Services Resources Resource Provider Software Providers
    5. 5. Grid Projects in the US Community Service Provider Content Services Resources Software Providers Resource Provider
    6. 6. Resource Providers <ul><li>Campus and regional grids </li></ul><ul><ul><li>Purdue, Wisc, UCLA, …, … </li></ul></ul><ul><ul><li>TIGRE, UC system, … </li></ul></ul><ul><li>Open Science Grid </li></ul><ul><ul><li>43,000 CPUs, 6 PB disk, 15,000 CPU days/day </li></ul></ul><ul><ul><li>Allocations on basis of MOUs </li></ul></ul><ul><li>TeraGrid </li></ul><ul><ul><li>~ 1.2 Pflop/s </li></ul></ul><ul><ul><li>National Allocation Committee </li></ul></ul><ul><li>Amazon, Microsoft, IBM, etc. </li></ul><ul><ul><li>?? CPUs, ?? storage </li></ul></ul><ul><ul><li>Fee for service </li></ul></ul>
    7. 7. Open Science Grid Sites (5/4/08) +3 in Brazil; 2 in Mexico; 2 in Taiwan; 1 in the UK. Grows by 10-20 per year.
    8. 8. Use by Community CMS ATLAS CDF Local Usage & bugs (unmapped to VO) D0 2,000,000 a week 1,000,000 a week
    9. 9. TeraGrid Participants
    10. 10. Growing User Community Source: TeraGrid Central Database
    11. 11. Growing Usage Source: TeraGrid Central Database 3.95B NUs delivered in CY2007
    12. 12. CY2007 Usage by Discipline 3.95B NUs delivered in CY2007 Molecular Biosciences 31% Chemistry 17% Physics 17% Astronomical Sciences 12% Materials Research 6% Earth Sciences 3% All 19 Others 4% Advanced Scientific Computing 2% Atmospheric Sciences 3% Chemical, Thermal Systems 5%
    13. 13. Grid Projects in the US <ul><li>For example: </li></ul><ul><li>Build and test service (Wisc) </li></ul><ul><li>Certificate Authorities </li></ul><ul><li>Cancer Biology Informatics Grid </li></ul><ul><li>LIGO Data Grid </li></ul>Community Service Provider Content Services Resources Software Providers Resource Provider Service Provider
    14. 14. caBIG: sharing of infrastructure, applications, and data. Data Integration! Services & Cancer Biology Globus
    15. 15. caBIG Under the Covers NCICB Research Center Grid-Enabled Client Research Center Tool 1 Tool 2 Tool 3 Tool 4 Grid Data Service Analytical Service Grid Portal Microarray Gene Database caArray Protein Database Image Tool 2 Tool 3 Grid Services Infrastructure (Metadata, Registry, Query, Invocation, Security, etc.) Globus
    16. 16. LIGO Data Grid Birmingham • Replicating >1 Terabyte/day to 8 sites 770 TB replicated to date: >120 million replicas MTBF = 1 month LIGO Gravitational Wave Observatory Ann Chervenak et al., ISI; Scott Koranda et al, LIGO <ul><li>Cardiff </li></ul>AEI/Golm Globus
    17. 17. Grid Projects in the US <ul><li>For example: </li></ul><ul><li>Earth System Grid </li></ul><ul><li>Children’s Oncology Grid </li></ul><ul><li>Southern California Earthquake Center (SCEC) </li></ul><ul><li>Science gateways </li></ul>Community Service Provider Content Services Resources Software Providers Resource Provider Community
    18. 18. Earth System Grid Main ESG Portal CMIP3 (IPCC AR4) ESG Portal <ul><li>198 TB of data at four locations </li></ul><ul><li>1,150 datasets </li></ul><ul><li>1,032,000 files </li></ul><ul><li>Includes the past 6 years of joint DOE/NSF climate modeling experiments </li></ul><ul><li>35 TB of data at one location </li></ul><ul><li>74,700 files </li></ul><ul><li>Generated by a modeling campaign coordinated by the Intergovernmental Panel on Climate Change </li></ul><ul><li>Data from 13 countries, representing 25 models </li></ul>8,000 registered users 1,900 registered projects <ul><li>Downloads to date </li></ul><ul><li>49 TB </li></ul><ul><li>176,000 files </li></ul><ul><li>Downloads to date </li></ul><ul><li>387 TB </li></ul><ul><li>1,300,000 files </li></ul><ul><li>500 GB/day (average) </li></ul>400 scientific papers published to date based on analysis of CMIP3 (IPCC AR4) data ESG usage: over 500 sites worldwide ESG monthly download volumes Globus
    19. 19. SCEC Community Modeling Environment Pathway Instantiations Knowledge Base Ontologies Curated taxonomies, Relations & constraints Pathway Models Pathway templates, Models of simulation codes Code Repositories Data & Simulation Products Data Collections FSM RDM AWM SRM Storage GRID Pathway Execution Policy, Data ingest, Repository access Grid Services Compute & storage management, Security DIGITAL LIBRARIES Navigation & Queries Versioning, Replication Mediated Collections Federated access KNOWLEDGE ACQUISITION Acquisition Interfaces Dialog planning, Pathway construction strategies Pathway Assembly Template instantiation, Resource selection, Constraint checking KNOWLEDGE REPRESENTATION & REASONING Knowledge Server Knowledge base access, Inference Translation Services Syntactic & semantic translation Computing Users A collaboratory for system-level earthquake science Globus
    20. 20. Seismic Hazard Analysis <ul><li>Defn: Max. intensity of shaking expected at a site during a fixed time interval </li></ul><ul><li>Example: National seismic hazard maps </li></ul><ul><li>Intensity measure: peak ground acceleration </li></ul><ul><li>Interval: 50 yrs </li></ul><ul><li>Probability of exceedance: 2% </li></ul>(http://geohazards.cr.usgs.gov/eq/) Globus
    21. 21. SCEC Computations & Grid <ul><li>Prepare input to Pathway2 wave propagation code </li></ul><ul><li>Pathway2PGV converts output into hazard map </li></ul><ul><li>Map is visualized </li></ul>SDSC USC SCEC PSC TeraGrid ISI 12 CPUs 1,700 CPUs 1,200 CPUs 1 CPU 4 CPUs Globus
    22. 22. Children’s Oncology Grid and MEDICUS Globus
    23. 23. Grid Projects in the US Community Service Provider Content Services Resources Resource Provider Software Providers
    24. 24. Software Providers <ul><li>Globus [GT4.2 released July 2, 2008] </li></ul><ul><ul><li>GRAM, GridFTP, MDS, RLS, DRS, … </li></ul></ul><ul><ul><li>GSI, GridShib, MyProxy, … </li></ul></ul><ul><ul><li>GridWay (Spain), OGSA-DAI (UK), Introduce, … </li></ul></ul><ul><li>Condor </li></ul><ul><li>MPI-G, Swift, Pegasus, Taverna (UK), Kepler </li></ul><ul><li>caBIG: e.g., Introduce </li></ul><ul><li>Virtual Data Toolkit (includes VOMS [Italy], …) </li></ul><ul><li>SRB, iRODS, MyCluster, … </li></ul><ul><li>… </li></ul>Globus
    25. 25. Virtual Data Toolkit (VDT) Software Release Process VDT components over time: built for 15 Linux Versions Development & testing Globus
    26. 26. Creating Services: Introduce and gRAVI <ul><li>Introduce </li></ul><ul><ul><li>Define service </li></ul></ul><ul><ul><li>Create skeleton </li></ul></ul><ul><ul><li>Discover types </li></ul></ul><ul><ul><li>Add operations </li></ul></ul><ul><ul><li>Configure security </li></ul></ul><ul><li>Grid R emote A pplication V irtualization Infrastructure </li></ul><ul><ul><li>Wrap executables </li></ul></ul>Index service Repository Service Introduce Container Ohio State University and Argonne/U.Chicago Appln Service Create Store Advertize Discover Invoke; get results Transfer GAR Deploy Globus
    27. 27. Composing Services Globus
    28. 28. Service Discovery: Registries Globus
    29. 29. Challenges Community Community Community Service Provider Content Services Resources Resource Provider Software Providers Conflicting Missions Sustainability Discipline science pull
    30. 30. The Future <ul><li>NSF eXtreme Digital (XD) solicitation </li></ul><ul><ul><li>Aka “TeraGrid III” </li></ul></ul><ul><li>DOE, NIH, etc.—what do they want? </li></ul><ul><li>International cooperation </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×