Web Services-Based Mediator of Distributed Data Flow and Processing Project Coordinators: Software Architecture:  R. Husar...
DataFed Description <ul><li>DataFed Vision </li></ul><ul><li>Better air quality management and science through by effectiv...
DataFed Multidimensional Data Model 4 D Geo-Environmental  Data Cube (X, Y, Z, T) <ul><li>Environmental  data represent me...
Data Flow & Processing in Air Quality Management AQ DATA EPA Networks IMPROVE Visibility  Satellite-PM Pattern  METEOROLOG...
Mediator-Based Integration Architecture   (Wiederhold, 1992)   <ul><li>The job of the mediator is to provide an answer to ...
Generic Data Flow and Processing in DataFed DataView 1 Data Processed Data Portrayed Data Process Data Portrayal/ Render A...
Anatomy of a Wrapper Service: TOMS Satellite Image Data <ul><li>Given the URL template and the image description, the wrap...
An Application Program: Voyager Data Browser <ul><li>The web-program consists of a stable  core  and adoptive  input/outpu...
<ul><li>SeaWiFS Satellite </li></ul>SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder VIEW by Web Service Comp...
Air Quality  Datasets <ul><li>Data are accessed from autonomous, distributed providers </li></ul><ul><li>DataFed ‘wrappers...
Some of the Tools of DataFed Consoles:  Data from diverse sources are displayed to create a rich context for exploration a...
Sulfate in the Northeast Sahara Dust in the Gulf Fires in the Southeast Time Series Console: Southeast Analyst Console App...
Upcoming SlideShare
Loading in …5
×

051207 Agu Sna Frncisco Data Fed Web Services Based Mediator Of Distributed Data Flow And Processing Files Data Fed 051207 Agu

1,652 views

Published on

http://capitawiki.wustl.edu/index.php/20051205_Web_Services-Based_Mediator_of_Distributed_Data_Flow_and_Processing

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,652
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 051207 Agu Sna Frncisco Data Fed Web Services Based Mediator Of Distributed Data Flow And Processing Files Data Fed 051207 Agu

    1. 1. Web Services-Based Mediator of Distributed Data Flow and Processing Project Coordinators: Software Architecture: R. Husar Software Implementation: K. Höijärvi Data and Applications: S. Falke, R. Husar Center for Air Pollution Impact and Trend Analysis (CAPITA) Washington University, St. Louis, MO 63130
    2. 2. DataFed Description <ul><li>DataFed Vision </li></ul><ul><li>Better air quality management and science through by effective use of relevant data </li></ul><ul><li>  </li></ul><ul><li>DataFed Goals </li></ul><ul><li>Facilitate the access and flow of atmospheric data from provider to users </li></ul><ul><li>Support the development of user-driven data processing value chains </li></ul><ul><li>Participate in specific application projects </li></ul><ul><li>  </li></ul><ul><li>Approach: Mediation Between Users and Data Providers </li></ul><ul><li>DataFed assumes spontaneous, autonomous emergence of AQ data ( a la Internet) </li></ul><ul><li>Non-intrusively wraps datasets for access by web services </li></ul><ul><li>WS-based mediators provide homogeneous data views e.g. geo-spatial, time... </li></ul><ul><li>End-user programming of data access and processing through WS composition (limited) </li></ul><ul><li>  </li></ul><ul><li>Applications </li></ul><ul><li>Building browsers and analysis tools for distributed monitoring data    </li></ul><ul><li>Serve as data gateway for user programs; web pages, GIS, science tools </li></ul><ul><li>DataFed is currently focused on the mediation of air quality data </li></ul>
    3. 3. DataFed Multidimensional Data Model 4 D Geo-Environmental Data Cube (X, Y, Z, T) <ul><li>Environmental data represent measurements in the physical world which has space (X, Y, Z) and time (T) as its dimensions. </li></ul><ul><li>The specific inherent dimensions for geo-environmental data are: Longitude X, Latitude Y, Elevation Z and DateTime T . </li></ul><ul><li>The needs for finding, sharing and integration of geo-environmental data requires that data are ‘coded’ in this 4D data space – at the minimum. </li></ul>
    4. 4. Data Flow & Processing in Air Quality Management AQ DATA EPA Networks IMPROVE Visibility Satellite-PM Pattern METEOROLOGY Met. Data Satellite-Transport Forecast model EMISSIONS National Emissions Local Inventory Satellite Fire Locs Status and Trends AQ Compliance Exposure Assess. Network Assess. Tracking Progress AQ Management Reports ‘ Knowledge’ Derived from Data Primary Data Diverse Providers Data ‘Refining’ Processes Filtering, Aggregation, Fusion
    5. 5. Mediator-Based Integration Architecture (Wiederhold, 1992) <ul><li>The job of the mediator is to provide an answer to a user query ( Ullman , 1997 ) </li></ul><ul><li>In database theory sense, a mediator is a view of the data found in one or more sources </li></ul><ul><li>Heterogeneous sources are wrapped by translation software local to global language </li></ul><ul><li>Mediators (web services) obtain data from wrappers or other mediators and process it … </li></ul>Wrapper Wrapper Service Service User Query Views Heterogeneous Data
    6. 6. Generic Data Flow and Processing in DataFed DataView 1 Data Processed Data Portrayed Data Process Data Portrayal/ Render Abstract Data Access View Wrapper Physical Data Abstract Data Physical Data Resides in autonomous servers; accessed by view-specific wrappers which yield abstract data ‘slices’ Abstract Data Abstract data slices are requested by viewers; uniform data are delivered by wrapper services DataView 2 DataView 3 View Data Processed data are delivered to the user as multi-layer views by portrayal and overlay web services Processed Data Data passed through filtering, aggregation, fusion and other web services
    7. 7. Anatomy of a Wrapper Service: TOMS Satellite Image Data <ul><li>Given the URL template and the image description, the wrapper service can access the image for any day, any spatial subset using a HTTP URL or SOAP protocol: </li></ul><ul><li>Wrapper classes are available for geo-spatial (incl. satellite) images, SQL servers, text files,etc. The mediator classes are implemented as web services for uniform data access, transformation and portrayal. </li></ul>src_img_width src_img_height src_margin_right src_margin_left src_margin_top src_margin_bottom src_lon_min src_lat_max src_lat_min src_lon_max Image Description for Data Access: src_image_width=502 src_image_height=329 src_margin_bottom=105 src_margin_left=69 src_margin_right=69 src_margin_top=46 src_lat_min=-70 src_lat_max=70 src_lon_min=-180 src_lon_max=180 The daily TOMS images reside on the FTP archive, e.g. ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y2000/ea000820.gif URL template: ftp://toms.gsfc.nasa.gov/pub/eptoms/images/aerosol/y[yyyy]/ea[yy][mm][dd].gif Transparent colors for overlays RGB(89,140,255) RGB(41,117,41) RGB(23,23,23) RGB(0,0,0)
    8. 8. An Application Program: Voyager Data Browser <ul><li>The web-program consists of a stable core and adoptive input/output layers </li></ul><ul><li>The core maintains the state and executes the data selection, access and render services </li></ul><ul><li>The adoptive, abstract I/O layers connects the core to evolving web data, flexible displays and to the a configurable user interface: </li></ul><ul><ul><li>Wrappers encapsulate the heterogeneous external data sources and homogenize the access </li></ul></ul><ul><ul><li>Device Drivers translate generic, abstract graphic objects to specific devices and formats </li></ul></ul><ul><ul><li>Ports connect the internal parameters of the program to external controls </li></ul></ul><ul><ul><li>WDSL web service description documents </li></ul></ul>Data Sources Controls Displays I/O Layer Device Drivers Wrappers App State Data Flow Interpreter Core Web Services WSDL Ports
    9. 9. <ul><li>SeaWiFS Satellite </li></ul>SeaWiFS Satellite Aerosol Chemical Air Trajectory Map Boarder VIEW by Web Service Composition
    10. 10. Air Quality Datasets <ul><li>Data are accessed from autonomous, distributed providers </li></ul><ul><li>DataFed ‘wrappers’ provide uniform geo-time referencing </li></ul><ul><li>Tools allow space/time overlay, comparisons and fusion </li></ul>Near Real Time Data Integration Delayed Data Integration Surface Air Quality AIRNOW O3, PM25 ASOS_STI Visibility, 300 sites METAR Visibility, 1200 sites VIEWS_OL 40+ Aerosol Parameters Satellite MODIS_AOT AOT, Idea Project GASP Reflectance, AOT TOMS Absorption Indx, Refl. SEAW_US Reflectance, AOT Model Output NAAPS Dust, Smoke, Sulfate, AOT WRF Sulfate Fire Data HMS_Fire Fire Pixels MODIS_Fire Fire Pixels Surface Meteorology RADAR NEXTRAD SURF_MET Temp, Dewp, Humidity… SURF_WIND Wind vectors ATAD Trajectory, VIEWS locs.
    11. 11. Some of the Tools of DataFed Consoles: Data from diverse sources are displayed to create a rich context for exploration and analysis CATT: Combined Aerosol Trajectory Tool for the browsing backtrajectories for specified chemical conditions Viewer: General purpose spatio-temporal data browser and view editor applicable for all DataFed datasets
    12. 12. Sulfate in the Northeast Sahara Dust in the Gulf Fires in the Southeast Time Series Console: Southeast Analyst Console Applications: Sulfate Episode: 8/27/04

    ×