Aaas Data Intensive Science And Grid

New computing platforms for data-intensive science Ian Foster Computation Institute Argonne National Lab & University of Chicago

Abstract
New laboratory devices, sensor networks, high-throughput instruments, and numerical simulation systems are producing data at rates that are both without precedent and rapidly growing. The resulting increases in the size, number, and variety of data are revolutionizing scientific practice. These changes demand new computing infrastructures and tools. Until recently, most laboratories and collaborations managed their own data, operated their own computers, and used remote high-performance computers only when required. We are moving to a paradigm in which data will primarily be located and managed on remote clusters, grids, and data centers. In this symposium, we will examine the computing infrastructure designed to serve this emerging era of data-intensive computing from three perspectives: (1) that of grid computing, which enables the creation of virtual organizations that can share remote and distributed resources over the Internet; (2) that of data centers, which are transitioning to providers of integrated storage, data, compute, and collaboration services (the offering of one or more of these integrated services over the Internet is beginning to be called cloud computing); and (3) that of e-science, in which grids, Web 2.0 technologies, and new collaboration and analysis services are merging and changing the way science is conducted. Each speaker will focus on one perspective but also compare and contrast with the others.

Growth of Genbank (1982-2005) Broad Institute

Proteomics
Genomics
Transcriptomics
Protein sequence prediction
Phenotypic studies
Phylogeny
Sequence analysis
Protein structure prediction
Protein-protein interaction
Metabolomics
Model organism collections
Systems biology
Health epidemiology
Organisms
Disease ….
1070 molecular bio databases Nucleic Acids Research Jan 2008 (96 in Jan 2001) Slide: Carole Goble

New problem solving methodologies <0 1700 1950 1990 “ Applied computer science is now playing the role that mathematics did from the 17th through the 20th centuries: providing an orderly, formal framework and exploratory apparatus for other sciences” – G. Djorgovski Empirical Data Theory Simulation

More data does not always mean more knowledge Folker Meyer, Genome Sequencing vs. Moore’s Law: Cyber Challenges for the Next Decade, CTWatch , August 2006.

enormous
Data is Infrastructure Storage & computing Economics of scale Aggregation Data & software People & disciplines Algorithms Scalable, probabilistic Errors & ambiguity distributed noisy Cloud Grid

An incomplete list of process steps
Discover
Access
Integrate
Analyze
Mine
Publish
Annotate
Validate
Curate Share
Data Artisanal Industrial Data Analyses Models Experiments Literature

SOA as an integrating framework?
We expose data and software as services …
which others discover , decide to use, …
and compose to create new functions ...
which they publish as new services.
Technical …
Complexity
Semantics
Distribution
Scale
socio-technical challenges
Incentives
Policy, trust
Reproducibility
Life cycle
“ Service-oriented science”, Science , 2005 and

Grid technology

NAE Grand Challenges

The future of multi-site data integration: An example fMRI Are positive symptom schizophrenics associated with more severe superior temporal gyrus dysfunction? Receptor Density ERP Web PubMed, Expasy, Brain Map, Etc. Structure Clinical Portal

caBIG: sharing of infrastructure, applications, and data. Aggregation in cancer biology Globus

As of Feb 16 , 2009 123 participants 104 services 65 data 39 analytical

Microarray clustering in caBIG
Query and retrieve microarray data from a caArray data service: cagridnode.c2b2.columbia.edu:8080/wsrf/services/cagrid/CaArrayScrub
Normalize microarray data using GenePattern analytical service node255.broad.mit.edu:6060/wsrf/services/cagrid/PreprocessDatasetMAGEService
Hierarchical clustering using geWorkbench analytical service: cagridnode.c2b2.columbia.edu:8080/wsrf/services/cagrid/HierarchicalClusteringMage
Workflow in/output caGrid services “ Shim” services others Wei Tan (Taverna workflow)

Children’s Oncology Group clinical imaging irials (Erberich)

Wide-area medical interface service
Converts local medical workflow actions into wide area operations
Image workflow, EHR, …
Transparently manages federation of
Security
Data replication and recovery
Data discovery
Enterprise/Grid Interface Service DICOM Protocols Grid Protocols (Web services) DICOM XDS HL7 Vendor Specific Wide Area Service Actor Plug-in Adapters

Earth System Grid www.earthsystemgrid.org Main ESG Portal CMIP3 (IPCC AR4) ESG Portal
198 TB of data at four locations
1,150 datasets
1,032,000 files
Includes the past 6 years of joint DOE/NSF climate modeling experiments
35 TB of data at one location
74,700 files
Generated by a modeling campaign coordinated by the Intergovernmental Panel on Climate Change
Data from 13 countries, representing 25 models
8,000 registered users 1,900 registered projects
Downloads to date
49 TB
176,000 files
Downloads to date
387 TB
1,300,000 files
500 GB/day (average)
400 scientific papers published to date based on analysis of CMIP3 (IPCC AR4) data ESG usage: over 500 sites worldwide ESG monthly download volumes Globus

Understanding interactions between human and natural systems IPCC Emissions scenarios Numerical Simulations IPCC 4 th Assessment 2007 IPCC process: Bill Collins, LBNL Mitigation Adaptation

A C ommunity I ntegrated M odel for E conomic a nd R esource T rajectories for H umankind ( CIM-EARTH ) Dynamics, foresight, uncertainty, resolution, … Agriculture, transport, taxation, … Data (global, local, …) (Super) computers CIM-EARTH Framework Community process Open code, data www.cim-earth.org

Alleviating Poverty in Thailand: Modeling Entrepreneurship Consider only wealth, access to capital Consider also distance to 6 major cities Rob Townsend, Tibi Stef-Praun, Victor Zhorin Match High Low

enormous
Data is Infrastructure Storage & computing Economics of scale Aggregation Data & software People & disciplines Algorithms Scalable, probabilistic Errors & ambiguity distributed noisy Cloud Grid

Thank you! Computation Institute www.ci.uchicago.edu

Aaas Data Intensive Science And Grid

by Ian Foster on Feb 19, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 5

Statistics

Aaas Data Intensive Science And Grid — Presentation Transcript