GeoServer, GeoTools and GeoBatch:
      supporting operational
  Meteorology and Oceanography

       Ing Simone Giannecch...
Outline
   GeoSolutions Facts

   Reference Scenario/Domain

   Enterprise SDI

   SDI Building Blocks explained
    ...
GeoSolutions - Facts
   Funded mid 2006 in Italy
   Staff of 7 engineers (2 consultants)
   Expertise
           Image...
Reference Scenario/Domain




Enterprise Boundaries




                                               REP/COP




       ...
Enterprise Spatial Data Infrastructure

Reporting             A&A&A                          GeoPortal


                 ...
Enterprise Spatial Data Infrastructure

  GeoBI               A&A&A                       geoSDI-ERA   OpenLayers



     ...
Ingestion/Preprocessing
   Real-time produced data
       Remote Sensing, In situ, Models, etc..

   Recognizable, isol...
GeoBatch: Basics
   Event-based geospatial-aware batch ingestion/processing
    system
   Basic tools for automatic coll...
GeoBatch:FlowManager

                    FileSystemMonitor


Directory




                                              ...
GeoBatch:Sample Flow
 - <FlowConfiguration>
      <workingDirectory>AISAnomalies</workingDirectory>
      <autorun>true</a...
GeoBatch:Sample Flow
-              <GeoServerActionConfiguration>
                    <crs>EPSG:4326</crs>
              ...
GeoBatch: New UI




   FOSS4G 2009, Sydney
    23rd September 2009
Management/Dissemination/Exploitation
   Raw Data Storage and Dissemination
   Metadata Storage and Dissemination
     ...
GeoServer
   GeoSpatial enterprise gateway
           Management raster and vector data
   Standards compliant
        ...
GeoServer: Limits/Shortfalls
   In-memory internal catalog
       Non transactional, Scale sub-optimally
       Replica...
GeoServer: Hibernate catalog

Web REST WMS WFS WCS           ...       Configuration stored in DB
       Catalog Interfac...
GeoServer: Hibernate catalog

Web REST WMS WFS WCS            ...        Web REST WMS WFS WCS           ...
       Catalog...
GeoServer: Transactional API

JVM 1
                                  In-JVM communication

   GeoServer
                ...
GeoServer: Advanced Raster




        FOSS4G 2009, Sydney
         23rd September 2009
GeoServer: RasterVault




     FOSS4G 2009, Sydney
      23rd September 2009
GeoServer: RasterVault

Web REST WMS WFS WCS           ...
                                                             Ge...
Index & Search Capabilities: ebRIM
   Registry versus Repository
                                                        ...
ERGO Buddata EbRR
   Open Source ebXML Registry/Repository (or ebRR in short)
                                           ...
EbRR: Architecture overview
Web Service Interface Layer
                                                                  ...
Buddata EbRR @ Work
   Metadata published with data contextually
                                                        ...
GeoPortal: geoSDI - ERA
   Modular/Extensible Rapid-mapping Web Application
   EXT-GWT + Openlayers + Hibernate Spatial ...
geoSDI – ERA Modular
   GeoSDI-ERA Architecture Schema
                        GeoSDI-ERA                          GeoSDI...
Use Case: AIS data management
   Automated Identification System (AIS)
       WHF emitter/receiver
       GPS
   Conti...
Use Case: AIS data management
     Fully automatic AIS coverage and anomaly detection ingestion and
      publishing proc...
Use case: SAS Mission Manager
   SAS: Synthetic Aperture Sonar
   Sea bed analysis and detection of features/artifacts

...
Use case: SAS Mission Manager
                               SAS Mission Manager
                                  Web int...
Use case: SAS Mission Manager




                        Sonar images are Curtesy of NURC

         FOSS4G 2009, Sydney
 ...
Use Case: Super Ensemble Modeling
   Optimal MetOc models mixing
         Same model, different boundary conditions
    ...
Use Case: Super Ensemble Modeling

        Discover   Buddata
                    EbRR
                                   ...
Use Case: Super Ensemble Modeling
      Current solution
          Split 3D hyper-cubes of data into a set of 2D slices
...
Use Case: Super Ensemble Modeling




           FOSS4G 2009, Sydney
            23rd September 2009
Use Case: TDA


  Satellite data




                                                Value
                               ...
Use Case: TDA

 Discover     Buddata
               EbRR
                                        Toolbox WPS
             ...
Use Case: TDA

                                         Search page




WPS invocation




                  FOSS4G 2009, ...
Use Case: LAMMA


                WRF                Postproc   Postproc
NCEP-GFS
                NMM




 ECMWF




     ...
Use Case: LAMMA


                 WRF                 Postproc
NCEP-GFS
                 NMM

                           ...
Use Case: FAO FIGIS
   Dynamic Statistical Query Engine
                                             - Compute GIS Layer ...
Use Case: FAO FIGIS
                           The Service Units interact
                           directly with FIGIS
 ...
Upcoming SlideShare
Loading in …5
×

Geosolutions FOSS4g 2009 Redux

1,904
-1

Published on

Standard version of the GeoSolutions presentation @ FOSS4g 2009.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,904
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Geosolutions FOSS4g 2009 Redux

  1. 1. GeoServer, GeoTools and GeoBatch: supporting operational Meteorology and Oceanography Ing Simone Giannecchini FOSS4G 2009, Sydney 23rd September 2009
  2. 2. Outline  GeoSolutions Facts  Reference Scenario/Domain  Enterprise SDI  SDI Building Blocks explained  GeoBatch  GeoServer  BUDDATA ebRIM  geoSDI-ERA  Relevant Use cases FOSS4G 2009, Sydney 23rd September 2009
  3. 3. GeoSolutions - Facts  Funded mid 2006 in Italy  Staff of 7 engineers (2 consultants)  Expertise  Image Processing, GeoSpatial Data Fusion  Java, Java Enterprise, C++, Python  JPEG2000, JPIP, Advanced 2D visualization  Supporting/Developing FOSS4G projects  GeoTools, GeoServer, GeoBatch, ImageIO-Ext, uDig  Focus on  Consultancy (agencies, large private companies, etc…)  Building stack of geospatial Open Source products FOSS4G 2009, Sydney 23rd September 2009
  4. 4. Reference Scenario/Domain Enterprise Boundaries REP/COP FOSS4G 2009, Sydney 23rd September 2009
  5. 5. Enterprise Spatial Data Infrastructure Reporting A&A&A GeoPortal Data Collection Data Management Layer Monitoring Agents Batch Processing ESB Map Accelerator Online Data Dissemination Index & Search Processing Server Data Management Layer Spatial Net DBMS Storage FOSS4G 2009, Sydney 23rd September 2009
  6. 6. Enterprise Spatial Data Infrastructure GeoBI A&A&A geoSDI-ERA OpenLayers Data Collection GeoTools Monitoring ESB Agents GeoWebCache GeoBatch GeoServer Buddata ebRIM GeoTools Net PostGIS Storage FOSS4G 2009, Sydney 23rd September 2009
  7. 7. Ingestion/Preprocessing  Real-time produced data  Remote Sensing, In situ, Models, etc..  Recognizable, isolate data streams  Flow  Incoming series of geospatial data  Specific format  Defined rules for preprocessing  Defined rules for ingestion and/or exploitation  Automatic Ingestion and preprocessing  No manual intervention  Example, flow of EO raster GeoWebCache Producer Tile Pyramid Publish GeoServer FOSS4G 2009, Sydney 23rd September 2009
  8. 8. GeoBatch: Basics  Event-based geospatial-aware batch ingestion/processing system  Basic tools for automatic collection, processing and publication of data.  Open Source (leverages on GeoTools, Apache FTP, Spring, Xstream, etc…)  Embedded FTP Server  Publish to GeoServer  Publish to GeoWebCache (ongoing)  Web based  XML configuration  http://docs.codehaus.org/display/GEOBATCH FOSS4G 2009, Sydney 23rd September 2009
  9. 9. GeoBatch:FlowManager FileSystemMonitor Directory Mailbox Event Monitoring Action Action Action EventDispatcher Event Event Consumer Consumer FlowManager FOSS4G 2009, Sydney 23rd September 2009
  10. 10. GeoBatch:Sample Flow - <FlowConfiguration> <workingDirectory>AISAnomalies</workingDirectory> <autorun>true</autorun> - <EventConsumerConfiguration> <workingDirectory>AISAnomalies</workingDirectory> <performBackup>true</performBackup> - <FileEventRule> <optional>false</optional> <originalOccurrencies>1</originalOccurrencies> <regex>.*.shp</regex> <id>r1</id> <description>description</description> <name>test</name> </FileEventRule> …. - <GeoServerActionConfiguration> <crs>EPSG:4326</crs> <envelope /> <dataTransferMethod>URL</dataTransferMethod> <geoserverPWD>geoserver</geoserverPWD> <geoserverUID>admin</geoserverUID> <geoserverURL>http://localhost:8080/geoserver</geoserverURL> <wmsPath>/</wmsPath> <defaultStyle>polygon</defaultStyle> <styles /> </GeoServerActionConfiguration> FOSS4G 2009, Sydney 23rd September 2009
  11. 11. GeoBatch:Sample Flow - <GeoServerActionConfiguration> <crs>EPSG:4326</crs> <envelope /> <dataTransferMethod>URL</dataTransferMethod> <geoserverPWD>geoserver</geoserverPWD> <geoserverUID>admin</geoserverUID> <geoserverURL>http://localhost:8080/geoserver</geoserverURL> <wmsPath>/</wmsPath> <defaultStyle>polygon</defaultStyle> <serviceID>aisAnomaliesGeoServerGeneratorService</serviceID> <styles /> </GeoServerActionConfiguration> </EventConsumerConfiguration> <eventGeneratorConfiguration class="FsEventGeneratorConfiguration"> <wildCard>*.*</wildCard> <watchDirectory>AISAnomalies/in</watchDirectory> <keepFiles>true</keepFiles> <osType>OS_UNDEFINED</osType> <eventType>FILE_ADDED</eventType> </eventGeneratorConfiguration> <id>flowAISAnomalies</id> <description>flowAISAnomalies: testing injection of AISAnomalies shapefiles </description> <name>flowAISAnomalies</name> </FlowConfiguration> FOSS4G 2009, Sydney 23rd September 2009
  12. 12. GeoBatch: New UI FOSS4G 2009, Sydney 23rd September 2009
  13. 13. Management/Dissemination/Exploitation  Raw Data Storage and Dissemination  Metadata Storage and Dissemination  Index & Search Capabilities for data  Service Discovery  Index & Search Capabilities for services  Raw Data Portrayal  Coverage  Maps  Feature  Maps  Processing FOSS4G 2009, Sydney 23rd September 2009
  14. 14. GeoServer  GeoSpatial enterprise gateway  Management raster and vector data  Standards compliant  OGC WCS 1.0 - 1.1.1 (RI)  OGC WFS 1.0 - 1.1 (RI)  OGC WMS 1.1.1  OGC WPS 1.0  Google support  KML, GeoSearch, etc..  Enterprise ready (almost!) FOSS4G 2009, Sydney 23rd September 2009
  15. 15. GeoServer: Limits/Shortfalls  In-memory internal catalog  Non transactional, Scale sub-optimally  Replication is problematic  File Based Configuration  Non transactional, Scale sub-optimally  Replications is problematic  File based Coverage support (can be overcome easily)  1 CoverageStore  1 Coverage (can be overcome, not easily)  Embedding/Controlling GeoServer can be difficult  REST uses HTTP always  REST is non transactional  No Time and Height/Depth support for raster  No CSW or ebRIM support (yet!)  WCS support needs to be improved FOSS4G 2009, Sydney 23rd September 2009
  16. 16. GeoServer: Hibernate catalog Web REST WMS WFS WCS ...  Configuration stored in DB Catalog Interface  Hibernate Hibernate Based Catalog  Spring based Dao  Transactions via Spring/EJB3 ServiceDAO CatalogDAO  Caching with EhCache Hibernate  Same catalog interface, different 2°L cache implementation  Catalog interface needs tweaking  GeoServer UI still needs tweaking  Transaction demarcation needs Net DB tweaking Storage FOSS4G 2009, Sydney 23rd September 2009
  17. 17. GeoServer: Hibernate catalog Web REST WMS WFS WCS ... Web REST WMS WFS WCS ... Catalog Interface Catalog Interface Hibernate Based Catalog Hibernate Based Catalog ServiceDAO CatalogDAO ServiceDAO CatalogDAO Hibernate Hibernate 2° L Distributed Cache Net DB DB DB Storage FOSS4G 2009, Sydney 23rd September 2009
  18. 18. GeoServer: Transactional API JVM 1  In-JVM communication GeoServer  Transactional  No HTTP, No Serialization Master Application  Spring, EJB3, JMX  Intra JVM  Transactional  Serialization is involved  Separate mechanism for JVM 2 data upload  HTTP, TCP, etc… Master  Spring Remoting, EJB3, Application JMX FOSS4G 2009, Sydney 23rd September 2009
  19. 19. GeoServer: Advanced Raster FOSS4G 2009, Sydney 23rd September 2009
  20. 20. GeoServer: RasterVault FOSS4G 2009, Sydney 23rd September 2009
  21. 21. GeoServer: RasterVault Web REST WMS WFS WCS ... GeoServer Catalog Interface CoverageStore Interface CoverageStore RasterLayerDAO RangeDAO Hibernate Spatial 2°L cache RasterVault Net DB Storage FOSS4G 2009, Sydney 23rd September 2009
  22. 22. Index & Search Capabilities: ebRIM  Registry versus Repository Trader Authority  Flexible Information Model  Object Lifecycle Management  Content Management Service Provider  Event Notification  Protocol Bindings  Publish-Find-Subscribe(Consume) Consumer  Federation Registry Broker Publish Find Subscribe Provider Consume Consumer FOSS4G 2009, Sydney 23rd September 2009
  23. 23. ERGO Buddata EbRR  Open Source ebXML Registry/Repository (or ebRR in short) e brr buddata  ESA founded ERGO project lead by Intecs  OGC specifications implemented  07-110r4: CSW-ebRIM Registry Service - Part 1: ebRIM profile of CSW (1.0.1)  07-144r4: CSW-ebRIM Registry Service – Part 2: Basic extension package (1.0.1)  07-038: OGC Cataloguing of ISO Metadata (CIM) using the ebRIM profile of CS-W (0.1.11)  06-131r6: EO Products Extension Package for ebRIM (ISO/TS 15000- 3) Profile of CSW 2.0  Code on Google Code (http://code.google.com/p/buddata- ebxml-registry/)  Wiki ERGO project (ESA HMA wiki) FOSS4G 2009, Sydney 23rd September 2009
  24. 24. EbRR: Architecture overview Web Service Interface Layer e brr buddata HTTP GetRepositoryItem Interface HTTP GetRepositoryItem SOAP Interfaces Interface HTTP GetRepositoryItem Interface API Layer CSW SOAP Client Registry APIs CSW Backend Client Deployer Lifecycle Manager Business Layer Translator Transaction Manager Core Registry Add-on Functions Functions Harvester Query Manager Validator Repository Manager Data Layer Postgresql Database Persistence PostGIS Functions FOSS4G 2009, Sydney 23rd September 2009
  25. 25. Buddata EbRR @ Work  Metadata published with data contextually e brr buddata  Use of custom profiles Profiles Registry Publish Metadata Find Metadata GeoBatch Consumer Publish Consume Data GeoServer Data --------- --------- ----- FOSS4G 2009, Sydney 23rd September 2009
  26. 26. GeoPortal: geoSDI - ERA  Modular/Extensible Rapid-mapping Web Application  EXT-GWT + Openlayers + Hibernate Spatial + Spring  Core components to perform basic actions  OpenLayers integration  Layers Wide search  Layer Tree  Security subsystem  Style editor (ongoing)  Plugin Engine to perform specific tasks  Routing, geocoding, etc..  GEOGWT?  GEOEXT + EXT-GWT FOSS4G 2009, Sydney 23rd September 2009
  27. 27. geoSDI – ERA Modular  GeoSDI-ERA Architecture Schema GeoSDI-ERA GeoSDI-ERA CORE WEB OGC- Map Map WMS Context Plugins Map Engine Widgets GeoSDI-ERA SLD Editor I-Mobility Plug-Ins Tools Security Subsystem PostGIS -Raster Symbolizer -Geo-Coding DB -Classifier -Routing -SLD Load/Save -Traffic Density Maps FOSS4G 2009, Sydney 23rd September 2009
  28. 28. Use Case: AIS data management  Automated Identification System (AIS)  WHF emitter/receiver  GPS  Continuously recording AIS traffic  Mediterranean Sea  Black Sea  Providing support for storage and publishing of the results of the AIS processing  AIS Coverage  AIS Forecasts  AIS Anomalies  More in the near future, like Radar contacts and Fuse d Tracks FOSS4G 2009, Sydney 23rd September 2009
  29. 29. Use Case: AIS data management  Fully automatic AIS coverage and anomaly detection ingestion and publishing process. AIS Data Stream GeoBatch Standard OGC WMS/WFS clients GeoServer PostGIS Google Earth/KML compliant clients FOSS4G 2009, Sydney 23rd September 2009
  30. 30. Use case: SAS Mission Manager  SAS: Synthetic Aperture Sonar  Sea bed analysis and detection of features/artifacts  Automatic Ingestion of SAS data and metadata  SAS Data preprocessing  SAS mission data management and access  SAS mission data visualization FOSS4G 2009, Sydney 23rd September 2009
  31. 31. Use case: SAS Mission Manager SAS Mission Manager Web interface Presentation Tier Data and Metadata GeoServer Management Registry ebRIM WFS WCS WMS Middleware Tier Network GeoBatch Area Storage Mission Data DataIngestion Package Tier FOSS4G 2009, Sydney 23rd September 2009
  32. 32. Use case: SAS Mission Manager Sonar images are Curtesy of NURC FOSS4G 2009, Sydney 23rd September 2009
  33. 33. Use Case: Super Ensemble Modeling  Optimal MetOc models mixing  Same model, different boundary conditions  Different model same boundary coinditions  System training (Kalman filter)  historycal data  Output  Series of weight for optimal MetOc model mixing  Result  optimal MetOc model as fusion of weighted models FOSS4G 2009, Sydney 23rd September 2009
  34. 34. Use Case: Super Ensemble Modeling Discover Buddata EbRR Toolbox WPS Invoke Harvest MATLAB Ensemble GUI Publish Results WCS::getCoverage WMS::getMap GeoServer Input Data Deploy GeoBatch FOSS4G 2009, Sydney 23rd September 2009
  35. 35. Use Case: Super Ensemble Modeling  Current solution  Split 3D hyper-cubes of data into a set of 2D slices  Use registry for registering T and Z . .. . . GeoBatch Registry . .. . . ... .. . Publish . . .. . . .  Vertical Metadata . . . .... . . .. Interpolation . ... . . . .  Geo-Rectification . . .  Resampling Publish Input 3D Data GeoServer Cubes • Next Step • Handle multidim coverages through GeoServer + RasterVault FOSS4G 2009, Sydney 23rd September 2009
  36. 36. Use Case: Super Ensemble Modeling FOSS4G 2009, Sydney 23rd September 2009
  37. 37. Use Case: TDA Satellite data Value added Oceanographic/Meteo Fusion products forecast On-site observations FOSS4G 2009, Sydney 23rd September 2009
  38. 38. Use Case: TDA Discover Buddata EbRR Toolbox WPS Harvest Invoke FUZZY TDA logic Explorer Publish WCS:getCoverage WMS:getMap GeoServer Input Data Publish GeoBatch FOSS4G 2009, Sydney 23rd September 2009
  39. 39. Use Case: TDA Search page WPS invocation FOSS4G 2009, Sydney 23rd September 2009
  40. 40. Use Case: LAMMA WRF Postproc Postproc NCEP-GFS NMM ECMWF FOSS4G 2009, Sydney 23rd September 2009
  41. 41. Use Case: LAMMA WRF Postproc NCEP-GFS NMM GeoBatch Publish Data Publish GeoServer Metadata ECMWF RasterVault Registry FOSS4G 2009, Sydney 23rd September 2009
  42. 42. Use Case: FAO FIGIS  Dynamic Statistical Query Engine - Compute GIS Layer Intersections - Download statistics outcomes UI - Monitoring the Engine status HTTP HTTP Provider Consumer WFS-T Enterprise Service BUS GeoServer (Apache ServiceMix) WFS Scheduler GeomCompare Service Service The Service Units interact The Dynamic Query directly with FIGIS Engine consists of a set GeoServer instances of Service Units capable through the OGC WFS-T of generating statistics protocol. on a periodic interval fashion. DB (ORACLE) FOSS4G 2009, Sydney 23rd September 2009
  43. 43. Use Case: FAO FIGIS The Service Units interact directly with FIGIS GeoServer instances through the OGC WFS-T protocol. FOSS4G 2009, Sydney 23rd September 2009

×