Your SlideShare is downloading. ×
0
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
OGSA-DAI-3-Introduct..
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

OGSA-DAI-3-Introduct..

441

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
441
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Project team members drawn from Commercial organisations and Non-commercial organisations
  • Strictly speaking data location is not hidden by the factory. Can gather this from the factory’s SDEs!!!
  • The breakdown areas are somewhat arbitrary for now. It's only to give an indication of how the space partitions. Feedback from users has been largely positive. We could do with a user contribution framework/policy.
  • Source: presentation by Mike Brady at the 2004 AHM. Statistics from NHS Cancer Screening web site ( http://www.cancerscreening.nhs.uk/) . Regarding views: two views of the breast taken at every screen instead of just at the first screen as at present-one from above (craniocaudal) and one into the armpit diagonally across the breast (mediolateral)
  • CHU - Churchill Hospital and Oxford University UCL - St George's Hospital at University College London KCL - Guy's & St Thomas' Hospital & King College London UED - The Ardmillan and Edinburgh University
  • Bank queue runs on an HPC resource EPCC and data stored at EPCC while the Telco queue runs on an HPC resource at Curtin data stored at Curtin.
  • Time zone date problem - a date was converted to an absolute time which goes over as UTC for midnight local time on that date which converts to the previous day locally.
  • Transcript

    • 1. Introduction to OGSA-DAI The OGSA-DAI Team [email_address]
    • 2. The OGSA-DAI Project <ul><li>A generic framework for integrating data access and computation </li></ul><ul><ul><li>Uniform interface to relational, XML, flat file data resources </li></ul></ul><ul><li>Using the grid to take specific classes of computation nearer to the data </li></ul><ul><li>Kit of parts for building tailored access and integration applications </li></ul><ul><li>Investigations to inform DAIS-WG </li></ul><ul><li>One reference implementation for DAIS </li></ul><ul><li>Releases publicly available NOW </li></ul>
    • 3. Project Partners Funded by the Grid Core Programme Powered by ….
    • 4. Project Membership IBM Dissemination Team Charaka Charaka Mike Ally Amy Mario Malcolm Kostas Norman Paul Neil Andy Simon Brian Dave Patrick Neil IBM Development Team Principal Investigators Project Manager Programme Management Board Chair Technical Review Board Chair Research Team EPCC Team
    • 5. Infrastructure Architecture OGSA Grid or Web Service Infrastructure Data Intensive Applications for Science X Compute, Data & Storage Resources Distributed Simulation, Analysis & Integration Technology for Science X Data Intensive X Scientists Virtual Integration Architecture Generic Virtual Data Access and Integration Layer OGSA-DAI Structured Data Integration Structured Data Access Structured Data Relational XML Semi-structured - Transformation Registry Job Submission Data Transport Resource Usage Banking Brokering Workflow Authorisation
    • 6. Project Status <ul><li>Current release 4.0 </li></ul><ul><ul><li>Globus Toolkit 3.2 compliant </li></ul></ul><ul><ul><li>Platform and language independent </li></ul></ul><ul><ul><ul><li>Java 1.4 </li></ul></ul></ul><ul><ul><ul><li>Document model </li></ul></ul></ul><ul><li>Work concentrated on data access </li></ul><ul><ul><li>Wraps data resources without hiding underlying data model </li></ul></ul><ul><ul><li>Provide base for higher-level services </li></ul></ul><ul><ul><ul><li>Distributed Query Processing (DQP) </li></ul></ul></ul><ul><ul><ul><li>Data federation services </li></ul></ul></ul>
    • 7. Supported Data Resources ?        SQLServer PostgreSQL Oracle eXist DB2 Files Xindice MySQL Other XML Relational
    • 8. Web Service Architecture Service Registry Service Consumer Service Provider Publish Bind Discover
    • 9. OGSA-DAI Service Architecture DAISGR Service Consumer GDSF GDS Publish Bind Discover
    • 10. OGSA-DAI Services <ul><li>OGSA-DAI uses three main service types </li></ul><ul><ul><li>DAISGR (registry) for discovery </li></ul></ul><ul><ul><li>GDSF (factory) to represent a data resource </li></ul></ul><ul><ul><li>GDS (data service) to access a data resource </li></ul></ul><ul><li>This will change </li></ul>accesses represents DAISGR GDSF GDS Data Resource locates creates
    • 11. GDSF and GDS <ul><li>Grid Data Service Factory (GDSF) </li></ul><ul><ul><li>Represents a data resource </li></ul></ul><ul><ul><li>Persistent service </li></ul></ul><ul><ul><ul><li>Currently static (no dynamic GDSFs) </li></ul></ul></ul><ul><ul><ul><ul><li>Cannot instantiate new services to represent other/new databases </li></ul></ul></ul></ul><ul><ul><li>Exposes capabilities and metadata </li></ul></ul><ul><ul><li>May register with a DAISGR </li></ul></ul><ul><li>Grid Data Service (GDS) </li></ul><ul><ul><li>Created by a GDSF </li></ul></ul><ul><ul><li>Generally transient service </li></ul></ul><ul><ul><li>Required to access data resource </li></ul></ul><ul><ul><li>Holds the client session </li></ul></ul>
    • 12. Heterogeneity Grid Data Service Xindice MySql Oracle DB2 <ul><li>Data source abstraction behind GDS instance </li></ul><ul><ul><li>plug in “data resource implementations” for different data source technologies </li></ul></ul><ul><ul><li>does not mandate any particular query language or data format </li></ul></ul>
    • 13. DAISGR <ul><li>DAI Service Group Registry (DAISGR) </li></ul><ul><ul><li>Persistent service </li></ul></ul><ul><ul><li>Based on OGSI ServiceGroups </li></ul></ul><ul><ul><li>GDSFs may register with DAISGR </li></ul></ul><ul><ul><li>Clients access DAISGR to discover </li></ul></ul><ul><ul><ul><li>Resources </li></ul></ul></ul><ul><ul><ul><li>Services (may need specific capabilities) </li></ul></ul></ul><ul><ul><ul><ul><li>Support a given portType or activity </li></ul></ul></ul></ul>
    • 14. Location Analyst Registry DAISGR Factory GDSF registerService findServiceData findServiceData <ul><li>Data resource publication through registry </li></ul><ul><li>Data location hidden by factory </li></ul><ul><li>Data resource meta data available through Service Data Elements </li></ul>
    • 15. Interaction Model: Start up Frogs Database 1. Start OGSI containers with persistent services. 2. Here GDSF represents Frog database. OGSI Container OGSI Container GDSF DAISGR
    • 16. Interaction Model: Registration Frogs Database 3. GDSF registers with DAISGR. Frogs: GSH OGSI Container OGSI Container GDSF DAISGR
    • 17. Interaction Model: Discovery Frogs Database 4. Client wants to know about frogs. Can: (i) Query the GDSF directly if known or (ii) Identify suitable GDSF through DAISGR. Frogs: GSH Mmmmm… Frogs? FindService: Frogs GSH: GDSF OGSI Container OGSI Container GDSF DAISGR
    • 18. Interaction Model: Service Creation Frogs Database 5. Having identified a suitable GDSF client asks a GDS to be created. Frogs: GSH GDS CreateService GSH: GDS OGSI Container OGSI Container GDSF DAISGR
    • 19. Interaction Model: Perform Frogs Database 6. Client interacts with GDS by sending Perform documents. 7. GDS responds with a Response document. 8. Client may terminate GDS when finished or let it die naturally . Frogs: GSH GDS Perform Document Response Document OGSI Container OGSI Container GDSF DAISGR
    • 20. Interaction Model: Summary <ul><li>Only described an access use case </li></ul><ul><ul><li>Client not concerned with connection mechanism </li></ul></ul><ul><ul><li>Similar framework could accommodate service-service interactions </li></ul></ul><ul><li>Discovery aspect is important </li></ul><ul><ul><li>Probably requires a human </li></ul></ul><ul><ul><li>Needs adequate definition of metadata </li></ul></ul><ul><ul><ul><li>Definitions of ontologies and vocabularies - not something that OGSA-DAI is doing … </li></ul></ul></ul>
    • 21. More Complex Behaviour GDS And there's a lot more that you can do … Data Resource Container Client GDT Data Resource Container GDS GDT Deliver data back to the client. Data Resource Deliver data to a third party. Deliver data another GDS.
    • 22. Usage Patterns Data Q - Query D - Delivery S - Status R - Result U - Update I - Data id Retrieve Update/Insert Pipeline G A Q S+R Q+D A C G S R G C A Q S D R A G Q+U S G2=C G1=P A I Q1 S2 S1 U/R Q2+D Q1+D G2=C A G1=P S2 S1 Q2 U/R Actors - OGSI process - Non-OGSI process A - Analyst C - Consumer G - GDS P - Producer Call Response Data Flow A P G U I Q S A P G U I S Q+D
    • 23. Project Using OGSA-DAI
    • 24. Projects Using OGSA-DAI OGSA-DAI (http://www.ogsadai.org.uk) AstroGrid (http://www.astrogrid.org/) BioSimGrid (http://www.biosimgrid.org/) BioGrid (http://www.biogrid.jp/) Bridges (http://www.brc.dcs.gla.ac.uk/projects/bridges/) eDiaMoND (http://www.ediamond.ox.ac.uk/) FirstDig (http://www.epcc.ed.ac.uk/~firstdig/) GeneGrid (http://www.qub.ac.uk/escience/projects.php#genegrid) GEON (http://www.geongrid.org/) IU RGRBench (http://www.cs.indiana.edu/~plale/projects/RGR/OGSA-DAI.html) myGrid (http://www.mygrid.org.uk/) N2Grid (http://www.cs.univie.ac.at/institute/index.html?project-80=80) ODD-Genes (http://www.epcc.ed.ac.uk/oddgenes/) OGSA-WebDB (http://www.gtrc.aist.go.jp/dbgrid/) INWA (http://www.epcc.ed.ac.uk/)
    • 25. Project classification OGSA-DAI Biological Sciences Physical Sciences Commercial Applications Computer Sciences <ul><li>FirstDig </li></ul><ul><li>INWA </li></ul><ul><li>Bridges </li></ul><ul><li>AstroGrid </li></ul><ul><li>BioSimGrid </li></ul><ul><li>BioGrid </li></ul><ul><li>eDiamond </li></ul><ul><li>myGrid </li></ul><ul><li>ODD-Genes </li></ul><ul><li>N2Grid </li></ul><ul><li>GEON </li></ul><ul><li>MCS </li></ul><ul><li>IU RGBench </li></ul><ul><li>OGSA Web-DB </li></ul><ul><li>GeneGrid </li></ul><ul><li>GridMiner </li></ul>
    • 26. Points to Note <ul><li>Feedback from users largely positive </li></ul><ul><ul><li>Good suggestions </li></ul></ul><ul><ul><li>Fair criticisms </li></ul></ul><ul><ul><li>How OGSA-DAI is being used </li></ul></ul><ul><ul><li>Where it succeeds and where it fails </li></ul></ul><ul><ul><li>Helping us to capture requirements </li></ul></ul><ul><li>Hope to allow user contributions </li></ul><ul><ul><li>Plan to establish a policy/framework for this </li></ul></ul><ul><li>Engage more with User Community </li></ul><ul><ul><li>Meetings scheduled for this year </li></ul></ul><ul><ul><ul><li>OGSA-DAI mini-workshop at AHM 2004 </li></ul></ul></ul><ul><ul><ul><li>OGSA-DAI tutorials at various meetings/locations </li></ul></ul></ul>
    • 27. <ul><li>e-D igital M amm O graphy N ational D atabase </li></ul><ul><ul><li>Mammogram - X-ray of the breast </li></ul></ul><ul><li>Built prototype of a national database of mammographic images </li></ul><ul><ul><li>In support of the UK Breast screening programme </li></ul></ul><ul><li>Employed Grid technologies to facilitate process </li></ul>Thanks to eDiaMonND project and the Digital Database for Screening Mammography for this image.
    • 28. <ul><li>Breast screening in the UK began in 1988 </li></ul><ul><ul><li>Women aged 50-64 screened every 3 Years </li></ul></ul><ul><ul><li>Women aged 50-70 from 2004 </li></ul></ul><ul><ul><li>1 View/Breast -> 2 views by 2003 </li></ul></ul><ul><li>UK has </li></ul><ul><ul><li>Over 90 Breast screening units throughout the UK </li></ul></ul><ul><ul><li>Each one deals with about 45000 women on average p.a. </li></ul></ul><ul><li>Each centre sees 5000-20000 images/year </li></ul><ul><li>In 2001-02 -> 2002-03 </li></ul><ul><ul><li>Screened: 1.4M -> 1.5M </li></ul></ul><ul><ul><li>Recalled for Assessment : 77911 -> 79441 </li></ul></ul><ul><ul><li>Cancers detected : 10003 -> 10467 </li></ul></ul><ul><ul><li>Lives per year Saved: 300 -> 1250 (by 2010) </li></ul></ul><ul><li>Distributed team of doctors perform the analysis </li></ul>
    • 29. DB2 Content Manager DB2 Content Manager DB2 Content Manager DB2 Content Manager DB2 Federation OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI Database Files OGSA-DAI Core Services Core Services Core Services Core Services Data Load Training App Training Services UCL KCL UED CHU Core API Training API Training Application Core & Training API OGSA-DAI Data Load Training App Core & Training API Data Load Training App Core & Training API Data Load Training App Core & Training API
    • 30. <ul><li>eDiaMoND Findings: </li></ul><ul><ul><li>OGSA-DAI provides a flexible framework </li></ul></ul><ul><ul><li>Dynamically configure the system through discovery </li></ul></ul><ul><ul><li>Activities can operate with different levels of granularity </li></ul></ul><ul><ul><li>Federation can be introduced at various levels </li></ul></ul><ul><ul><li>Good documentation on how to extend the framework </li></ul></ul><ul><ul><ul><li>Extended Activities to access IBM DB2 Content Manager </li></ul></ul></ul><ul><ul><li>Changes between versions broke some things </li></ul></ul><ul><ul><ul><li>Low level XML issues </li></ul></ul></ul>
    • 31. FirstDIG <ul><li>Data mining with the First Transport Group, UK </li></ul><ul><ul><li>Example: “When buses are more than 10 minutes late there is an 82% chance that revenue drops by at least 10%” </li></ul></ul><ul><ul><li>&quot;The results of this exercise will revolutionise the way we do things in the bus industry.“, Darren Unwin, Divisional Manager, First South Yorkshire. </li></ul></ul>OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI Client Application Data Mining Application
    • 32. INWA <ul><li>Innovation Node: Western Australia </li></ul><ul><ul><li>Informing Business & Regional Policy: Grid-enabled fusion of global data and local knowledge </li></ul></ul><ul><li>Project </li></ul><ul><ul><li>Run from Nov 2003 - Aug 2004 </li></ul></ul><ul><ul><li>Involved 10 partners (6 UK + 4 Australia) </li></ul></ul><ul><li>Aim </li></ul><ul><ul><li>Data mine commercially sensitive data </li></ul></ul><ul><ul><li>Security an absolute MUST </li></ul></ul><ul><ul><li>Employ Grid technologies </li></ul></ul><ul><ul><li>Need access to data and computational resources </li></ul></ul><ul><li>Demonstrator using: </li></ul><ul><ul><li>OGSA-DAI </li></ul></ul><ul><ul><ul><li>Incorporate data resources </li></ul></ul></ul><ul><ul><li>Sun DCG's TOG (Transfer-queue Over Globus) </li></ul></ul><ul><ul><ul><li>Handle job submission to analyse micro array data </li></ul></ul></ul>
    • 33. INWA [email_address] Curtin,Australia EPCC,UK Grid Engine Bank Telco Grid Engine Bank Telco OGSA-DAI OGSA-DAI OGSA-DAI OGSA-DAI TOG TOG Data Browser Data Browser [email_address] Telco data Bank data Australian property UK Property
    • 34. INWA: Lessons Learned <ul><li>Performing Data Integration: </li></ul><ul><ul><li>TimeZone date problems </li></ul></ul><ul><li>Security issues: </li></ul><ul><ul><li>Bugs in </li></ul></ul><ul><ul><ul><li>JavaCoG in GT3 </li></ul></ul></ul><ul><ul><ul><li>OGSA-DAI could not switch security for Grid data transfers </li></ul></ul></ul><ul><ul><ul><li>TOG had no security option </li></ul></ul></ul><ul><ul><li>All of these have been fixed </li></ul></ul><ul><li>Middleware not mature enough for commercial deployment </li></ul>
    • 35. Why OGSA-DAI? <ul><li>Why use OGSA-DAI over JDBC? </li></ul><ul><ul><li>Can embed additional functionality at the service end </li></ul></ul><ul><ul><ul><li>Transformations, compressions </li></ul></ul></ul><ul><ul><ul><li>Third party delivery </li></ul></ul></ul><ul><ul><ul><li>The extensible activity framework </li></ul></ul></ul><ul><ul><li>Avoiding unnecessary data movement </li></ul></ul><ul><ul><li>Common interface to heterogeneous data resources </li></ul></ul><ul><ul><ul><li>Relational, XML databases, and files </li></ul></ul></ul><ul><ul><li>Usefulness of the Registry for service discovery </li></ul></ul><ul><ul><ul><li>Dynamic service binding process </li></ul></ul></ul><ul><ul><ul><li>Provision of good meta-data is necessary </li></ul></ul></ul><ul><ul><li>Language independence at the client end </li></ul></ul><ul><ul><ul><li>Do not need to use Java </li></ul></ul></ul><ul><ul><li>Platform independence </li></ul></ul><ul><ul><ul><li>Do not have to worry about connection technology, drivers, etc </li></ul></ul></ul>

    ×