• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Managing data interoperability with FME

Managing data interoperability with FME



Presentation by Tony Kent on Managing data interoperability with FME at GIS Ireland 11/10/12

Presentation by Tony Kent on Managing data interoperability with FME at GIS Ireland 11/10/12



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • http://en.wikipedia.org/wiki/Geospatial_metadatahttp://en.wikipedia.org/wiki/Metadata
  • Supports searchesAllows you to review content without having to read all the data firstDiscover what’s availableSupport data management and updatesprovides parameters for automating processingControl accessTrack ownership
  • Read metadata attributesXML reading, writing, updatingSchema readingReading from web sourcesIntegration with web servicesHarvest and validation capabilitiesCreation of your own web services

Managing data interoperability with FME Managing data interoperability with FME Presentation Transcript

  • Managing DataInteroperability with FME Tony Kent Applications Engineer IMGS
  • IMGSWe deliver innovative spatial solutionsFor the desktop, web and mobileBuilt on our partner’s technologyDesigned to meet the challenges of Government,Mapping Agencies, and Utility & CommunicationsCustomers
  • Safe Software Powering the flow of spatial data with FMEEnabling people to use their spatial data where, when and how they want to Most Used Spatial Interoperable Solution in Ireland 3
  • Why Spatial ETL?Significant proliferation of different spatial dataformats and types Hundreds of formats, with more added each year Multiple types of data stored in multiple systems Unique data model requirements for each application 4
  • Why Spatial ETL?Traditional approaches to data translationand data model manipulation are not viable Complex, inefficient and time-consuming 5
  • Why Spatial ETL?Increasing pressure for access to spatial data More users, beyond traditional GIS users Expectations of real-time custom data views, 24x7 6
  • FME Capabilities The only complete spatial ETL solution  Translate spatial data from one format to another  Transform spatial data into the precise data model you need  Integrate different data types into a single data model  Distribute spatial data to users where, when and how they need it 7
  • FME DesktopFlexible and powerful spatial ETL toolsetTranslate, transform and integrate data in hundreds offormatsGraphical authoring environment Step 1 - Extract Select and add the source dataset(s) Step 2 - Transform Add transformers to manipulate the data as it moves from source to destination Step 3 – Load Load the transformed data into a destination format and source 8
  • FME WorkbenchUse simplepoint and clickto easily definespatial dataflows totranslate,transform andintegrate yourdata 9
  • ExamplesAutomating Ordnance Survey data updates Pushing NTF data to multiple GIS platforms Stripping out unnecessary data Adding custom styling and symbology – CAD E.g. Eircom, ESB, Fingal County CouncilPublishing data to internal public portals Bulk and transactional updates Fire wall Friendly – use selected port Completely automated E.g. Dublin City Council
  • Open Data ChallengeYou want to meet Open data requirements, butyour data is organized rather differently ?
  • What FME does … ? Build data bridges to your SDI
  • SDI Harmonization Core ConceptsHarmonization: implied requirement for buildingan SDIDisparate sources must be mapped to a commondestination data modelCore to the harmonization workflow is a processcalled schema mapping.Delivered by services based on open standards
  • Harmonization PrinciplesTypical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. PublicationBased on the Spatial ETL concept (Extract,Transform and Load), as applied to INSPIRE SDI’s
  • EvaluationAssess destination schema and datarequirementsAssess source datasets and schemaConsider fundamental differences inrepresentation, resolutionClosely inspect actual representativedatasets
  • Data Assembly Assess the diversity of source data types: vector, raster, CAD, GIS, database, text, XML, web, 3D, sensor and non-spatial Review format and semantic translation needs Decide how to perform necessary joins ID joins, spatial relates, nearest neighbor, one to many relationshipsGoal is to build a data structure that corresponds with yourpublished standard
  • Metadata – Data about dataDescribes data structures tables geometry types data types fieldsDescribes data content coordinate system extent modification date quality, ownership, etc.
  • Metadata - Purpose
  • Key FME Metadata Capabilities Reading Writing Updating Harvesting Validating Integration with web services
  • CSW: Catalog Service for the Web Deegree GeoNetwork OpenGIS TerraCatalog
  • Data Transformation - SchemaReshape source data to match required destination schemaSchema mapping feature type attribute name new attribute creation code lists conditional value mappings
  • Schema Mapping in FME Feature Type Mapping in FME Workbench Attribute Mapping in FME Workbench
  • FME Data Model Restructuring: Attribute Names & Values  Value Mapping
  • FME SchemaMapper:INSPIRE geographic names FME Workspace Name mapping Name & value mapping
  • Transformation: GeometryNon-spatial to spatialGeometry extraction (spatial to GML)Representation transform: CAD drawing lines with labels toGIS polygonal features with attributesCoordinate System Reprojection (ED50 to ETRF89)Simple to complex geometry Source point and polygon data to multiple geometric representations (city as point / area, river as line / area)Generalization and interpolation Highly granular national and regional datasets often require thinning to be usable on pan-European scales
  • Validation Schema validation i.e. INSPIRE (xsds) Data integrity Unique IDs Geometric integrity (closed polygons) Null values (nullable?) Valid values: ranges and domain codes Data gaps Bounds Network integrity Custom validity rules specific to domain Validation automation via FME Server uploadEnsure data quality throughout the data transformation process
  • PublicationProduce INSPIRE compliant GMLProvide discovery, view or downloadservices, for WxS, GML and other desiredformatsPublish with FME Server or integrate withyour geo web server of choice:Spatial Data Services
  • Publication with FME ServerPublish workspace to FME Server Store the workspace in a central repositoryMake your FME workspaces available to others –overthe webRegister the workspace with one or more services (DataStreaming, Data Download, etc.) 28
  • FME Tools for INSPIREFormat translationSchema mappingString and list manipulationData validationDatabase load and extractXML,GML,WFS: reading, validation,publicationWeb services: WFS, WMS,integration with othersMetadata supportEnterprise services with FME Server
  • Other ETL Software Options Intergraph GeoMedia Fusion GeoKettle – Open Source Snowflake Software Spatial Data Integrator – Open Source
  • SummaryFME can provide all the tools to help build support yourdata sharing needs: Integrate your data sources Manage your meta data catalogues Transform your data to standard schemas Publish the data in the required formats
  • Thank YouFor more information: Email: ckirk@imgs.ie or tkent@imgs.ie Twitter: @Ciarankirk Web: www.imgs.ie