Managing DataInteroperability with FME         Tony Kent        Applications Engineer               IMGS
IMGSWe deliver innovative spatial solutionsFor the desktop, web and mobileBuilt on our partner’s technologyDesigned to mee...
Safe Software   Powering the flow of spatial data with   FMEEnabling people to use their spatial data where, when and how ...
Why Spatial ETL?Significant proliferation of different spatial dataformats and types  Hundreds of formats, with more added...
Why Spatial ETL?Traditional approaches to data translationand data model manipulation are not viable  Complex, inefficient...
Why Spatial ETL?Increasing pressure for access to spatial data  More users, beyond traditional GIS users  Expectations of ...
FME Capabilities The only complete spatial ETL solution                     Translate spatial data from one             ...
FME DesktopFlexible and powerful spatial ETL toolsetTranslate, transform and integrate data in hundreds offormatsGraphical...
FME WorkbenchUse simplepoint and clickto easily definespatial dataflows totranslate,transform andintegrate yourdata       ...
ExamplesAutomating Ordnance Survey data updates   Pushing NTF data to multiple GIS platforms   Stripping out unnecessary d...
Open Data ChallengeYou want to meet Open data requirements, butyour data is organized rather differently                  ...
What FME does …     ?     Build data bridges        to your SDI
SDI Harmonization        Core ConceptsHarmonization: implied requirement for buildingan SDIDisparate sources must be mappe...
Harmonization PrinciplesTypical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. PublicationBased on the Sp...
EvaluationAssess destination schema and datarequirementsAssess source datasets and schemaConsider fundamental differences ...
Data Assembly  Assess the diversity of source data types:      vector, raster, CAD, GIS, database, text, XML,      web, 3D...
Metadata – Data about dataDescribes data structures  tables  geometry types  data types  fieldsDescribes data content  coo...
Metadata - Purpose
Key FME Metadata Capabilities Reading Writing Updating Harvesting Validating Integration with web services
CSW: Catalog Service for the Web Deegree GeoNetwork OpenGIS TerraCatalog
Data Transformation - SchemaReshape source data to match required destination schemaSchema mapping   feature type   attrib...
Schema Mapping in FME Feature Type Mapping in FME Workbench Attribute Mapping in FME Workbench
FME Data Model Restructuring:  Attribute Names & Values  Value Mapping
FME SchemaMapper:INSPIRE geographic names                           FME Workspace                           Name mapping  ...
Transformation: GeometryNon-spatial to spatialGeometry extraction (spatial to GML)Representation transform: CAD drawing li...
Validation  Schema validation i.e. INSPIRE (xsds)  Data integrity     Unique IDs     Geometric integrity (closed polygons)...
PublicationProduce INSPIRE compliant GMLProvide discovery, view or downloadservices, for WxS, GML and other desiredformats...
Publication with FME ServerPublish workspace to FME Server   Store the workspace in a central repositoryMake your FME work...
FME Tools for INSPIREFormat translationSchema mappingString and list manipulationData validationDatabase load and extractX...
Other ETL Software Options Intergraph GeoMedia Fusion GeoKettle – Open Source Snowflake Software Spatial Data Integrator –...
SummaryFME can provide all the tools to help build support yourdata sharing needs:   Integrate your data sources   Manage ...
Thank YouFor more information:  Email: ckirk@imgs.ie or tkent@imgs.ie  Twitter: @Ciarankirk  Web:     www.imgs.ie
Upcoming SlideShare
Loading in …5
×

Managing data interoperability with FME

822
-1

Published on

Presentation by Tony Kent on Managing data interoperability with FME at GIS Ireland 11/10/12

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
822
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://en.wikipedia.org/wiki/Geospatial_metadatahttp://en.wikipedia.org/wiki/Metadata
  • Supports searchesAllows you to review content without having to read all the data firstDiscover what’s availableSupport data management and updatesprovides parameters for automating processingControl accessTrack ownership
  • Read metadata attributesXML reading, writing, updatingSchema readingReading from web sourcesIntegration with web servicesHarvest and validation capabilitiesCreation of your own web services
  • Managing data interoperability with FME

    1. 1. Managing DataInteroperability with FME Tony Kent Applications Engineer IMGS
    2. 2. IMGSWe deliver innovative spatial solutionsFor the desktop, web and mobileBuilt on our partner’s technologyDesigned to meet the challenges of Government,Mapping Agencies, and Utility & CommunicationsCustomers
    3. 3. Safe Software Powering the flow of spatial data with FMEEnabling people to use their spatial data where, when and how they want to Most Used Spatial Interoperable Solution in Ireland 3
    4. 4. Why Spatial ETL?Significant proliferation of different spatial dataformats and types Hundreds of formats, with more added each year Multiple types of data stored in multiple systems Unique data model requirements for each application 4
    5. 5. Why Spatial ETL?Traditional approaches to data translationand data model manipulation are not viable Complex, inefficient and time-consuming 5
    6. 6. Why Spatial ETL?Increasing pressure for access to spatial data More users, beyond traditional GIS users Expectations of real-time custom data views, 24x7 6
    7. 7. FME Capabilities The only complete spatial ETL solution  Translate spatial data from one format to another  Transform spatial data into the precise data model you need  Integrate different data types into a single data model  Distribute spatial data to users where, when and how they need it 7
    8. 8. FME DesktopFlexible and powerful spatial ETL toolsetTranslate, transform and integrate data in hundreds offormatsGraphical authoring environment Step 1 - Extract Select and add the source dataset(s) Step 2 - Transform Add transformers to manipulate the data as it moves from source to destination Step 3 – Load Load the transformed data into a destination format and source 8
    9. 9. FME WorkbenchUse simplepoint and clickto easily definespatial dataflows totranslate,transform andintegrate yourdata 9
    10. 10. ExamplesAutomating Ordnance Survey data updates Pushing NTF data to multiple GIS platforms Stripping out unnecessary data Adding custom styling and symbology – CAD E.g. Eircom, ESB, Fingal County CouncilPublishing data to internal public portals Bulk and transactional updates Fire wall Friendly – use selected port Completely automated E.g. Dublin City Council
    11. 11. Open Data ChallengeYou want to meet Open data requirements, butyour data is organized rather differently ?
    12. 12. What FME does … ? Build data bridges to your SDI
    13. 13. SDI Harmonization Core ConceptsHarmonization: implied requirement for buildingan SDIDisparate sources must be mapped to a commondestination data modelCore to the harmonization workflow is a processcalled schema mapping.Delivered by services based on open standards
    14. 14. Harmonization PrinciplesTypical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. PublicationBased on the Spatial ETL concept (Extract,Transform and Load), as applied to INSPIRE SDI’s
    15. 15. EvaluationAssess destination schema and datarequirementsAssess source datasets and schemaConsider fundamental differences inrepresentation, resolutionClosely inspect actual representativedatasets
    16. 16. Data Assembly Assess the diversity of source data types: vector, raster, CAD, GIS, database, text, XML, web, 3D, sensor and non-spatial Review format and semantic translation needs Decide how to perform necessary joins ID joins, spatial relates, nearest neighbor, one to many relationshipsGoal is to build a data structure that corresponds with yourpublished standard
    17. 17. Metadata – Data about dataDescribes data structures tables geometry types data types fieldsDescribes data content coordinate system extent modification date quality, ownership, etc.
    18. 18. Metadata - Purpose
    19. 19. Key FME Metadata Capabilities Reading Writing Updating Harvesting Validating Integration with web services
    20. 20. CSW: Catalog Service for the Web Deegree GeoNetwork OpenGIS TerraCatalog
    21. 21. Data Transformation - SchemaReshape source data to match required destination schemaSchema mapping feature type attribute name new attribute creation code lists conditional value mappings
    22. 22. Schema Mapping in FME Feature Type Mapping in FME Workbench Attribute Mapping in FME Workbench
    23. 23. FME Data Model Restructuring: Attribute Names & Values  Value Mapping
    24. 24. FME SchemaMapper:INSPIRE geographic names FME Workspace Name mapping Name & value mapping
    25. 25. Transformation: GeometryNon-spatial to spatialGeometry extraction (spatial to GML)Representation transform: CAD drawing lines with labels toGIS polygonal features with attributesCoordinate System Reprojection (ED50 to ETRF89)Simple to complex geometry Source point and polygon data to multiple geometric representations (city as point / area, river as line / area)Generalization and interpolation Highly granular national and regional datasets often require thinning to be usable on pan-European scales
    26. 26. Validation Schema validation i.e. INSPIRE (xsds) Data integrity Unique IDs Geometric integrity (closed polygons) Null values (nullable?) Valid values: ranges and domain codes Data gaps Bounds Network integrity Custom validity rules specific to domain Validation automation via FME Server uploadEnsure data quality throughout the data transformation process
    27. 27. PublicationProduce INSPIRE compliant GMLProvide discovery, view or downloadservices, for WxS, GML and other desiredformatsPublish with FME Server or integrate withyour geo web server of choice:Spatial Data Services
    28. 28. Publication with FME ServerPublish workspace to FME Server Store the workspace in a central repositoryMake your FME workspaces available to others –overthe webRegister the workspace with one or more services (DataStreaming, Data Download, etc.) 28
    29. 29. FME Tools for INSPIREFormat translationSchema mappingString and list manipulationData validationDatabase load and extractXML,GML,WFS: reading, validation,publicationWeb services: WFS, WMS,integration with othersMetadata supportEnterprise services with FME Server
    30. 30. Other ETL Software Options Intergraph GeoMedia Fusion GeoKettle – Open Source Snowflake Software Spatial Data Integrator – Open Source
    31. 31. SummaryFME can provide all the tools to help build support yourdata sharing needs: Integrate your data sources Manage your meta data catalogues Transform your data to standard schemas Publish the data in the required formats
    32. 32. Thank YouFor more information: Email: ckirk@imgs.ie or tkent@imgs.ie Twitter: @Ciarankirk Web: www.imgs.ie
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×