SCENZ-Grid   The implementation of a Science Collaboration and Computation Environment Niels Hoffmann Landcare Research
<ul><li>Manaaki Whenua / Landcare Research is </li></ul><ul><li>New Zealand's foremost environmental research  organisatio...
Sustaining biodiversity & restoration Sustaining land environments Sustaining business & living Climate change Maori susta...
<ul><li>Data and Computational pressure </li></ul><ul><li>NOW – 25m national data density  </li></ul><ul><li>NEAR FUTURE –...
science  :  collaboration  :  environment SCE NZ-Grid proposes that we can Do science research on-line together   Share ea...
Dublin Core / RDF Repository Web Services Doc Img Geo Name Search Consume Query Tag Comment Create Workflow OAI-ORE SPARQL
A personal home page with links to relevant work for organising and retrieving a large variety of digital resources
Detailed view of a resource – in this case a modelling workflow, that can be edited by maybe changing the inputs or logic ...
Databases Content (WMS, WFS, WCS) Functionality (WPS) Desktop Web
PostGIS / SqlServer2008 GeoServer GeoWebCache 52North  Geoprocessing ArcGIS / Udig OpenLayers GeoExt
<ul><li>Why WPS ? </li></ul><ul><li>Distributed Architecture </li></ul><ul><li>Interoperability </li></ul><ul><li>Modeling...
<ul><li>Why 52North ? </li></ul><ul><li>Build upon robust OS Libraries (JTS, Geotools,  xmlBeans, Servlet API) </li></ul><...
<ul><li>Currently using Unicore Middleware </li></ul><ul><li>Planning a migration to Globus Middleware to integrate with B...
WPS Geoserver 104 Intel Xeon cores 2.8GHz each 386GB RAM 2.6TB storage ~1.16 TFLOPS Air cooled Gb Ethernet nterconnects 4....
<ul><li>Spatially query across datasets from multiple organisations. </li></ul><ul><li>Qmap: Geology from GNS </li></ul><u...
<ul><li>Regolith: </li></ul><ul><li>Layer between Bedrock and Soil </li></ul><ul><li>Erosion modelling </li></ul><ul><li>G...
Establish the ‘similarity’ of the datasets Different origin of attributes Different classification Similarity Matrix quant...
QMAP WMS NZFSL WMS Lookup WS Portal User WPS User Interface View Portlet WebService WorkFlow Engine WPS Web Services Data:...
Lookup WS WPS User
 
<ul><li>Thank You, </li></ul><ul><li>Any questions? </li></ul><ul><li>[email_address] </li></ul><ul><li>Slide Credits: </l...
Upcoming SlideShare
Loading in …5
×

Foss4G 2009 Scenz Grid

505 views

Published on

Landcare Research SCENZ-Grid architecture presentation as presented at FOSS4G

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
505
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Manaaki Whenua / Landcare Research is New Zealand&apos;s foremost environmental research organisation. Manaaki Whenua = Care for the land We do Environmental Landscape modelling
  • Trends Catchment modeling nutrient loss =&gt; hydrology model needs 1m dem 3d soil characterization =&gt; micron scale data
  • To respond to the challenges posed by our Scientists, Informatics came up with the SCENZ-Grid vision Data and Computational pressure Modelling environment Managed data Two Pillars: Collaboration and Computation
  • Collaboration Open architecture with a content repository at it’s heart, a cross cutting ‘catalog’ wider than just geospatial metadata or a content management system Spatial is ‘just a datatype’ in the database, likewise a dataset is just an object in the repository Standards: Open Archives Initiative - Object Reuse and Exchange SPARQL - S PARQL P rotocol a nd R DF Q uery L anguage Web Services in general
  • MyExperiment Evernote Social Networking DSpace
  • MyExperiment Taverna Kepler
  • Where does it fit in the Open Geo Stack?
  • Which components in the Geo Stack are we using?
  • Extensions to core WPS: WPS-T WPS-G
  • 1 x SGI Altix XE250 6 x SGI Altix XE320 Each XE320 is a pair of servers 1 + 6 x 2 = 13 nodes 2 x Intel quad core Xeons per node = 8 cores per node 13 x 8 = 104 cores
  • Example Focused on Collaboration as much as compute power, the analysis in this example does not necessarily require a cluster
  • What we’re talking about is the ‘regolith’ layer Solution has 2 components: Computation = WPS Algorithm Collaboration = Working on Similarity matrix
  • Q-Map is Geological mapping done by Geologists with their background LRIS / NZFSL is Soil map, has an attribute for bedrock which is mapped by Soil Scientists What we want to know is how the mappings from those 2 communities match LRIS is a pure hierarchical Classification: Order&gt;Group&gt;Subgroup&gt;Class QMAP is a multi-attribute Classification Sedimentary&gt;clastic Sediment&gt;mud&gt;mud but also mudstone&gt;mud We assigned weights to the attributes in the classification &gt; the more exact the description the higher the weight. If you add the weights you get the ‘similarity value’ a number between 0 and 20
  • Foss4G 2009 Scenz Grid

    1. 1. SCENZ-Grid The implementation of a Science Collaboration and Computation Environment Niels Hoffmann Landcare Research
    2. 2. <ul><li>Manaaki Whenua / Landcare Research is </li></ul><ul><li>New Zealand's foremost environmental research organisation. </li></ul><ul><li>Our research focuses on three key areas: </li></ul><ul><li>Sustaining and restoring biodiversity; </li></ul><ul><li>Sustaining land environments; </li></ul><ul><li>Sustaining business and living. </li></ul><ul><li>Three themes relate to these three areas: </li></ul><ul><li>Climate change mitigation and adaptation </li></ul><ul><li>Maori sustainability </li></ul><ul><li>Invasive species and disease impacts </li></ul>
    3. 3. Sustaining biodiversity & restoration Sustaining land environments Sustaining business & living Climate change Maori sustainable futures Weeds, pests and diseases Capability and collaboration Landcare Research Manaaki Whenua Key outcomes Cross-cutting outcomes Underpinning strengths
    4. 4. <ul><li>Data and Computational pressure </li></ul><ul><li>NOW – 25m national data density </li></ul><ul><li>NEAR FUTURE – sub 5m national data density / </li></ul><ul><li>(peri-)urban sub 1m LIDAR data density </li></ul><ul><li>Modelling environment </li></ul><ul><li>NOW – essentially batch oriented & 2.5D </li></ul><ul><li>DESIRED – interactive 4D, with real-time visualisation feedback </li></ul><ul><li>Managed data </li></ul><ul><li>NOW – preserve the data, memorise the model </li></ul><ul><li>DESIRED – keep the model for on-demand re-use </li></ul>
    5. 5. science : collaboration : environment SCE NZ-Grid proposes that we can Do science research on-line together Share each other’s data – not duplicate it Collaboratively develop & use shared models / workflows Use shared compute resources Connect researchers directly to consumers : policy / managers / educators / public spatial : computation : engine
    6. 6. Dublin Core / RDF Repository Web Services Doc Img Geo Name Search Consume Query Tag Comment Create Workflow OAI-ORE SPARQL
    7. 7. A personal home page with links to relevant work for organising and retrieving a large variety of digital resources
    8. 8. Detailed view of a resource – in this case a modelling workflow, that can be edited by maybe changing the inputs or logic and then re-used
    9. 9. Databases Content (WMS, WFS, WCS) Functionality (WPS) Desktop Web
    10. 10. PostGIS / SqlServer2008 GeoServer GeoWebCache 52North Geoprocessing ArcGIS / Udig OpenLayers GeoExt
    11. 11. <ul><li>Why WPS ? </li></ul><ul><li>Distributed Architecture </li></ul><ul><li>Interoperability </li></ul><ul><li>Modeling approach </li></ul><ul><li>(as opposed to data centric outcome) </li></ul><ul><li>(Grid-) Computing </li></ul>
    12. 12. <ul><li>Why 52North ? </li></ul><ul><li>Build upon robust OS Libraries (JTS, Geotools, xmlBeans, Servlet API) </li></ul><ul><li>Pluggable framework for algorithms </li></ul><ul><li>Support for raster processing </li></ul><ul><li>Support for Grid-Computing </li></ul>
    13. 13. <ul><li>Currently using Unicore Middleware </li></ul><ul><li>Planning a migration to Globus Middleware to integrate with BeSTGRID </li></ul>Landcare repository Sextante repository
    14. 14. WPS Geoserver 104 Intel Xeon cores 2.8GHz each 386GB RAM 2.6TB storage ~1.16 TFLOPS Air cooled Gb Ethernet nterconnects 4.2kW power
    15. 15. <ul><li>Spatially query across datasets from multiple organisations. </li></ul><ul><li>Qmap: Geology from GNS </li></ul><ul><li>NZFSL: Soil Data from Landcare Research </li></ul><ul><li>What kind of relationships exist between soil type and bedrock. </li></ul><ul><li>What is the association between groundwater quality and soil. </li></ul><ul><li>Investigate relationships between ecology, ground water and soil. </li></ul>
    16. 16. <ul><li>Regolith: </li></ul><ul><li>Layer between Bedrock and Soil </li></ul><ul><li>Erosion modelling </li></ul><ul><li>Groundwater flow modelling </li></ul><ul><li>WPS Algorithm to combine 2 datasets based on a similarity matrix </li></ul><ul><li>User interface to enable experts to adapt the similarity matrix </li></ul>
    17. 17. Establish the ‘similarity’ of the datasets Different origin of attributes Different classification Similarity Matrix quantify where the classifications match Expert to decide similarity based on documentation
    18. 18. QMAP WMS NZFSL WMS Lookup WS Portal User WPS User Interface View Portlet WebService WorkFlow Engine WPS Web Services Data: WMS Business Logic: SOAP
    19. 19. Lookup WS WPS User
    20. 21. <ul><li>Thank You, </li></ul><ul><li>Any questions? </li></ul><ul><li>[email_address] </li></ul><ul><li>Slide Credits: </li></ul><ul><li>Chris McDowall, Aaron Hicks </li></ul>science : collaboration : environment spatial : computation : engine

    ×