Implementing the Change Detector Transformer to Process Data in a Software Defined Environment (SDE)


Published on

This session demonstrates how FME's ChangeDetector was used to update changes to a spatial data compilation on a weekly basis without a large expenditure of time.

See more presentations from the FME User Conference 2014 at:

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Implementing the Change Detector Transformer to Process Data in a Software Defined Environment (SDE)

  1. 1. Implementing the Change Detector transformer to process data in an SDE June 12, 2014 Brooks Wilson – Northrop Grumman Jeff Safran – Bureau of Land Management Steve Gregonis – Bureau of Land Management
  2. 2. Abstract This project involved collecting spatial data from 10 different sources and creating a single compilation dataset. To avoid having to rebuild the compilation every week to keep it current, FME and the change detector were employed. An SDE for each source is updated through a replica. A version in the database is used to keep the status of the data before replication. The change detector transformer finds the differences between the default and the version. Features that are unchanged are ignored, added features are passed to the compilation SDE and deleted features are removed from the compilation SDE. Added features insert a new attribute to maintain the original Global ID so it can be identified for deletion in the future. This workflow greatly reduces processing time of large changing datasets.
  3. 3. Our Goal  Create a compiled dataset from many office submissions.  Track the changes from each office. To achieve these goals, we created a custom transformer that processed only the changes from each office and ignored the unchanged features.
  4. 4. How it works  Each office provides their data through a replica to a local SDE for that office.  The replica synchronizes with a version that is a child of the Default version.  When the replica is synchronized, Default contains the original features and the Replica Version contains the updated features.  The FeatureChangeDetector looks at the changes between Default and the Replica Version.  Added features are added to the compilation.  Deleted features are removed from the compilation.  SDE treats updated features as an Add and a Delete.
  5. 5. Inputs and Outputs INPUTS OUTPUTS Original Feature Class Unchanged Features Updated Feature Class Added Features Compilation Features Deleted Features
  6. 6. The Feature Change Detector
  7. 7. Change Detector  The first step is comparing the original features (Default) to the changed features (Replica Version) in the Change Detector transformer.  Unchanged features are passed out of the transformer with no further processing.
  8. 8. Added Features  As features are written to the Compilation SDE they are given a new GlobalID.  An attribute,Original_GlobalID, is added.  The Original_GlobalID field allows us to find the office features in the compilation feature class.  The ability to differentiate between office features is important during deletion of features
  9. 9. Deleted Features  Deleted features are passed to FeatureMerger transformer.  The feature is compared by GlobalID to the compilation SDE using the Original GlobalID.  Features that match are passed out the referenced port and are marked for deletion.
  10. 10. Output  Added and Deleted outputs are directed to the compilation writer.  Updated features are treated as an Add and a Delete by SDE  Unchanged features are already a part of the compilation, they are ignored or written out to a inspector.
  11. 11. Initialization  The first time the transformer runs, there will not be any differences between Default and the Replica Version.  For the initialization of the compilation, all features from the unchanged are routed as changed features.
  12. 12. Problems Encountered  An office would break replication, which would make it difficult to remove their work from the compilation.  One office did not use replication.  If the version was accidentally posted, it was difficult to find the changes that were missed.  It was eventually determined that it was not necessary to track all changes. Currently, the compilation tables are dropped and recreated each time the data is compiled as a way to ensure our datasets are complete. However, it does take much longer to process in this manner.
  13. 13. Review  Process works and is useable.  There should be a version per replica you are creating.  The data needs to be clean.  The data structure needs to be strong and unchanging.  All data suppliers need to use SDE.  There should be a requirement of speed or edit tracking to justify the work necessary to set up this process.
  14. 14. Contact Information  Brooks Wilson National ESRI Software Support Engineer National Operations Center (NOC) Office - (775) 861-6545 Email - Contractor - Northrop Grumman