0
Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. H...
Policy Guidance for HTAP IT <ul><li>Terry Keating, HTAP Task Force Co-Chair:  </li></ul><ul><li>Transparency  of the HTAP ...
Current Air Quality Information ‘Ecosystem’ <ul><li>AQ info is distributed over many ‘dimensions’: Geography, Content, Age...
GEOSS Info Flow Architecture A General Framework Accepted by Members Model Model Data Data
Federated Network for Air Quality Data and Processing Services Project Team: Software Architecture:  R. Husar Software Imp...
Federated Data System for Air Quality <ul><li>The challenge is to design a general supportive infrastructure </li></ul><ul...
Data Handling Approach: DataFed <ul><li>  Approach: Mediation Between Users and Data Providers </li></ul><ul><li>DataFed a...
Typical DataFed AQ Analysis Tools Consoles :  Data from diverse sources are displayed to create a rich context for explora...
Quebec Smoke July, 7, 2002 SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder Web Service Composition
Software Tools: Demo: Networked Data Access, Processing and Fusion AeroCOM – VIEWS SO4 <ul><li>http://webapps.datafed.net/...
Visualization Goal: 4D, User-selectable layers Image below completed in 1998  Satellite Data Layers: Land Reflectance (Sea...
4D Dynamic Visualization  Demonstrate interactions, allow exploration ‘Google Earth’ for Earth Science is now possible
Earth Science Information Partners Air Quality Cluster 1. Serve as  facilitator  to Earth Science  information community ....
Data Dissemination & Use – Service Based <ul><li>Provide Catalog Services (Publish, Find data) </li></ul><ul><li>Allow Dat...
WCS - Interoperable Data Access Service netCDF – Machine independent encoding  ncML – XML data description CF – Naming, st...
GALEON Interoperability Experiment Geo-interface for Atmosphere, Land, Earth, and Ocean netCDF Unify Earth Science & GIS D...
OGC WCS Demonstration:  THREDDS_GFS  4Dim Dataset <ul><li>Lat/Lon Box  Elev Range  Time Range </li></ul><ul><li>Map:  BBOX...
Summary <ul><li>Current systems data & model analysis are heterogeneous </li></ul><ul><li>Standardization is a key need fo...
OGC WCS Demonstration:  Grid, Image, Station Data Types <ul><li>Coverage=THEEDDS.T&  BBOX=-126,24,-65,52,0,0 &TIME=2002-07...
Single Data Model for All AQ Data <ul><li>Most  Views  are slices through a cube of data organized by lat, lon, altitude, ...
OGC Web Coverage Service (WCS) Specification <ul><li>HTTP GET/POST  based interfaces </li></ul><ul><li>Services have XML  ...
AQ Monitoring Network Data Storage and Delivery through OGC Protocols Relational Data Model Star Schema WMS WCS SOS Sensor...
Technology Support for ‘Integrated Solutions’ Air Quality Information System
Data Access through Adapters through DataFed  SOAP,HTTP Get OGC WCS   HTTP Get, Post OGC WMS   HTTP Get Station-Point  SQL...
Upcoming SlideShare
Loading in...5
×

2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives

798

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
798
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Transcript of "2006-03-14 WG on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives"

    1. 1. Work Group Meeting on HTAP-Relevant IT Techniques, Tools and Philosophies: DataFed Experience and Perspectives Rudolf B. Husar CAPITA, Washington University, St. Louis, MO Ispra, JRC, March 14. 2004 <ul><li>Data handling approach </li></ul><ul><li>Software tools </li></ul><ul><li>Participant involvement </li></ul><ul><li>Data dissemination </li></ul><ul><li>Integration problems </li></ul><ul><li>Summary </li></ul>
    2. 2. Policy Guidance for HTAP IT <ul><li>Terry Keating, HTAP Task Force Co-Chair: </li></ul><ul><li>Transparency of the HTAP Process </li></ul><ul><ul><li>Acceptance of the Tech Report will depend on openness </li></ul></ul><ul><li>Inclusiveness and Ease of Participation </li></ul><ul><ul><li>Facilitate participation by smaller contributors </li></ul></ul>
    3. 3. Current Air Quality Information ‘Ecosystem’ <ul><li>AQ info is distributed over many ‘dimensions’: Geography, Content, Agency, Program… </li></ul><ul><li>AQ info content includes: emissions , ambient & satellite data and model outputs </li></ul><ul><li>Info is provided and consumed by different agencies , (NASA, NOAA, EPA…) </li></ul><ul><li>Providers have different access protocols , formats, and information usage conditions </li></ul>Lack of Interoperability Poor data & model utilization Less informed decision making
    4. 4. GEOSS Info Flow Architecture A General Framework Accepted by Members Model Model Data Data
    5. 5. Federated Network for Air Quality Data and Processing Services Project Team: Software Architecture: R. Husar Software Implementation: K. Höijärvi Data and Applications: S. Falke, R. Husar Data Handling Approach
    6. 6. Federated Data System for Air Quality <ul><li>The challenge is to design a general supportive infrastructure </li></ul><ul><li>Simply connecting the relevant provides and users for each info product is messy </li></ul><ul><li>The info system infrastructure needs to facilitate the creation of info products </li></ul>AQ Compliance Nowcast/Forecast Status & Trends Find Data Gaps ID New Problems ……… Info Needs Reports <ul><li>Providers supply the ‘raw material’ (data and models) for ‘refined’ info products </li></ul>Emission Surface Satellite Model Single Datasets Providers Wrappers Where? What? When? Federate Data Structuring <ul><li>Structuring the heterogeneous data into where-when-what ‘cubes’ simplifies the mess </li></ul>Slice & Dice Explore Data Viewers <ul><li>The ‘cubed’ data can be accessed and explored by slicing-dicing tools </li></ul>Programs Integrate Understand <ul><li>More elaborate data integration and fusion can be done by web service chaining </li></ul>Non-intrusive Linking & Mediation Data Users Data Providers
    7. 7. Data Handling Approach: DataFed <ul><li>  Approach: Mediation Between Users and Data Providers </li></ul><ul><li>DataFed assumes autonomous data management ( a la Internet) </li></ul><ul><li>Non-intrusive third-party data wrapping for unified web service (WS) access </li></ul><ul><li>End-user programming of applications through chaining of WS components </li></ul><ul><li>  </li></ul><ul><li>Applications </li></ul><ul><li>Building browsers and analysis tools for distributed monitoring data    </li></ul><ul><li>Serve as data and service resources for user programs (science, GIS tools) </li></ul><ul><li>Support application projects , e.g. FASTNET, Exceptional Event Rule </li></ul>
    8. 8. Typical DataFed AQ Analysis Tools Consoles : Data from diverse sources are displayed to create a rich context for exploration and analysis CATT : Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets
    9. 9. Quebec Smoke July, 7, 2002 SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder Web Service Composition
    10. 10. Software Tools: Demo: Networked Data Access, Processing and Fusion AeroCOM – VIEWS SO4 <ul><li>http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_VIEWS </li></ul><ul><li>http://webapps.datafed.net/dvoy_services/datafed.aspx?page=KYU_V </li></ul><ul><li>http:// webapps.datafed.net/dvoy_services/datafed.aspx?page =VIEWS_KYU </li></ul>Networking
    11. 11. Visualization Goal: 4D, User-selectable layers Image below completed in 1998 Satellite Data Layers: Land Reflectance (SeaWiFS) Fire Pixels (ASTR Night) High Cloud (GOES, Meteosat, GMS2) Aerosol AOT (AVHRR)
    12. 12. 4D Dynamic Visualization Demonstrate interactions, allow exploration ‘Google Earth’ for Earth Science is now possible
    13. 13. Earth Science Information Partners Air Quality Cluster 1. Serve as facilitator to Earth Science information community . 2. Promote efficient flow of ES data from collection to end-use . 3. Improve quality and usability of ES data and info systems 4. Expand the use of Earth science information Participant Involvement Partners <ul><li>N ASA </li></ul><ul><li>NOAA </li></ul><ul><li>EPA </li></ul><ul><li>USGS </li></ul><ul><li>DOE </li></ul><ul><li>NSF </li></ul><ul><li>Industry… </li></ul>
    14. 14. Data Dissemination & Use – Service Based <ul><li>Provide Catalog Services (Publish, Find data) </li></ul><ul><li>Allow Data Access (‘Bind’) – Use International Standards </li></ul><ul><li>Add tools for exploration and analysis </li></ul>Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sites METAR Visibility, 1200 sites VIEWS_OL 40+ Aerosol Parameters Satellite MODIS_AOT AOT, Idea Project GASP Reflectance, AOT TOMS Absorption Indx, Refl. SEAW_US Reflectance, AOT Model Output NAAPS Dust, Smoke, Sulfate, AOT WRF Sulfate Fire Data HMS_Fire Fire Pixels MODIS_Fire Fire Pixels Surface Meteorology RADAR NEXTRAD SURF_MET Temp, Dewp, Humidity… SURF_WIND Wind vectors ATAD Trajectory, VIEWS locs.
    15. 15. WCS - Interoperable Data Access Service netCDF – Machine independent encoding ncML – XML data description CF – Naming, structure convention OGC Web Coverage Service - Interoperable Data Access Query Data Syntax + Semantics Coverage (parameter) BBOX Time Range netCDF Example Profile <ul><li>SERVICE=wcs ‘service </li></ul><ul><li>REQUEST=GetCoverage,VER=1.0 ‘service method </li></ul><ul><li>COVERAGE=AIRNOW.pmfine ‘what </li></ul><ul><li>CRS=EPSG:4326 ‘projection </li></ul><ul><li>BBOX=-125,22,-61,51,0,0 ‘where </li></ul><ul><li>TIME=2005-06-6T15:00:00Z ‘when </li></ul><ul><li>FORMAT=netCDF ‘return format </li></ul>
    16. 16. GALEON Interoperability Experiment Geo-interface for Atmosphere, Land, Earth, and Ocean netCDF Unify Earth Science & GIS Data Flows Strong European Participation IT – S. Nativi, L. Bigagli UK – Andrew Wolf DE – Peter Bauman) B. Domenico B. Domenico GALEON UNIDATA U Florence/CNR-IMMA WCS Server
    17. 17. OGC WCS Demonstration: THREDDS_GFS 4Dim Dataset <ul><li>Lat/Lon Box Elev Range Time Range </li></ul><ul><li>Map: BBOX=-180,-90,180,90, 1350,1350& TIME=2005-12-06/2005-12-06/PT3H </li></ul><ul><li>Time: BBOX=-34,49.05,-34,49.05, 1350,1350& TIME=2005-12-05/2005-12-08/PT3H </li></ul><ul><li>Elev: BBOX=-34,49.05,-34,49.05, 0,18000 & TIME=2005-12-06/2005-12-06/PT3H </li></ul>The form of the WCS query is the same for all slices through the data cube (views) The only difference in the views is the thickness of the slices in each dimension Return grid is in multiple formats (NetCDF, CSV, GML, PNG, … ) Map View Services WCS Query Time View Services WCS Query Elevation View Services WCS Query
    18. 18. Summary <ul><li>Current systems data & model analysis are heterogeneous </li></ul><ul><li>Standardization is a key need for agile IT systems </li></ul><ul><li>Non-intrusive mediators can achieve virtual standardization </li></ul><ul><li>Technologies are currently available for dynamic NETWORKING </li></ul><ul><li>We eager to share our networked data, tools and methods </li></ul>
    19. 19. OGC WCS Demonstration: Grid, Image, Station Data Types <ul><li>Coverage=THEEDDS.T& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF </li></ul><ul><li>Coverage=SURF.Bext& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=NetCDF-table </li></ul><ul><li>Coverage=SEAW.Refl& BBOX=-126,24,-65,52,0,0 &TIME=2002-07-07/2002-07-07&FORMAT=GeoTIFF </li></ul><ul><li>COVERAGE=sst& BBOX=-126,24,-65,52,0,0 &TIME=2001-01-01,2001-01-01&FORMAT=NetCDF </li></ul>UNIDATA – THREDDS/GALEON WCS DataFed GALEON WCS U Florence, It GALEON WCS DataFed GALEON WCS Grid Grid Image Station Services WCS Query Services WCS Query Services WCS Query Services WCS Query
    20. 20. Single Data Model for All AQ Data <ul><li>Most Views are slices through a cube of data organized by lat, lon, altitude, and time (X,Y,Z,T) </li></ul>Multidimensional Data Cube
    21. 21. OGC Web Coverage Service (WCS) Specification <ul><li>HTTP GET/POST based interfaces </li></ul><ul><li>Services have XML service descriptions (“Capabilities”, “Description”) </li></ul><ul><li>Filter parameters allow selection of subsets of source data </li></ul><ul><li>Output formats advertised by each service instance </li></ul>OGC WCS getCoverage Schema Suitable for wrapping with SOAP envelope, WSDL access, loose coupling WCS is for &quot;coverages&quot; – information representing space -time- varying phenomena WCS describes, requests and delivers coverages in spatio-temporal domain WCS version 1.1 is limited to grids/&quot;simple” coverages with homogeneous range sets
    22. 22. AQ Monitoring Network Data Storage and Delivery through OGC Protocols Relational Data Model Star Schema WMS WCS SOS SensorML WFS Observations Station Info. Param/Sensor/Method Data View Services WMS Stations Par-Meth Observations SOS WCS Data Access
    23. 23. Technology Support for ‘Integrated Solutions’ Air Quality Information System
    24. 24. Data Access through Adapters through DataFed SOAP,HTTP Get OGC WCS HTTP Get, Post OGC WMS HTTP Get Station-Point SQL Server, Files… Sequence Image, file nDim Grid OpenDAP NetCDF, … Other Traject., Event, Pic Sources Diverse formats Many data models Data Wrapper Data into geo-cubes Queries to views Virtual Data Cube Global geo-cube data model Makes queries data-neutral Others? e.g. OpenDAP Output Protocol dependent User specified GeoTable CSV,XLS,GML GeoGrid GML,NetCDF.. GeoImage GeoTIFF, PNG.. Other MS Dataset.. Query Adapter Maps query to protocol User selects protocols
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×