The Dark Art of Data Migration


Published on

Presented at the SPATIALinfo conference in Denver on 9th June 2011.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • SPATIALinfo conference, Denver, CO, 9th June 2011Planning for the migration of digital spatial data into SpatialNet
  • NW England: Lake District, Britain’s Energy Coast
  • GIS consulting for clients worldwideSpatial systems & data integrationData migrationInformation strategyApplication development
  • Why should it be difficult – the source data is in a system that has probably successfully supported the business for a number of years.
  • Relational tablesAttributes with data typesGeometry also stored in native OracleIntegrity rules to enforce consistencyStandards based – other programs can access it.Structure means that the data can be queried with confidence to support business decision making.
  • All data, including geometries, can be accessed by Oracle tools – and many 3rd party tools.Geometry can be queried and manipulated through SQL.
  • The SpatialNet data model is complex as it supports many purposes.
  • Data held across different systems in different formatsHopefully with some form of documentationProbably some duplication with differentiationAs successful business applications often get more complex over time (as they take on additional responsibilities) then it becomes important to fully understand the match (and gaps) between the source and target data.Drawings are only fully intelligible to the trained human eye.
  • Data associated by proximity on a drawingExtended entity data, blocks, etc.GIS systems can also be configured to store data as annotation (i.e. unstructured) ….
  • Fiber connectivity is not represented spatially – it must be provided by attribution.
  • Does the new system provide all the business functions of the existing system?Does all of the existing data map neatly into the model of the new system?Can the new system be accessed by all of the people that currently access the existing system?Existing system is likely to have evolved over time and may have assumed some additional responisbilities.An architect is required to have overall visibility of the entire system replacement project.The data migration is only one project within a larger programme.
  • Dream on….Activities are generally correct, they are all required – but must expect several iterations to refine.
  • Iterations generally get fasterFirst is the longest as there is more to put in place; later iterations are just refinements; final is very fast
  • Magicians are really illusionists – there is always an explanation – Preparation is key.Master data may be in other systems!Must understand the CRUD matrix for the data being migrated – in which applications will each attribute be maintained? And what integration points are required (inbound and outbound) to maintain the data in the new system.Correct data errors in the existing system – greater levels of expertise and efficiency with the familiar system.Prepare for acceptance testing – how will we know when the data has been migrated correctly?What goes in is what came out.How to exchange data between source system/s and migration environment?
  • Investigate Oracle loading options for spatial data.Standardize DictionariesSymbol Orientation, Annotation positioning, Feature placement (e.g. dist between cables)Pre-processing – minimum changes to prepare the data for loading into Oracle using the selected tools. E.g. drop constraints, change reserved words, protect Identity columns, etc.
  • Load source into Oracle environment with minimal processing – use Oracle-based tools to process data with consistency.Bulk of the work happens in migrating from the staging environment to the neutral model.
  • Geometry values – recorded Vs calculatedOracle may well provide a better environment for data profiling than source applications
  • Every migration is unique.Need to be flexible with use of the most appropriate toolsYet try to be consistent in tool use.Script the migration as far as possible to enable efficient iterations.
  • Neutral model is relatively simple, easy to read and review.
  • Tom Ward presentation on afternoon of June 9th.
  • Support for Just in Time training.Script as much of the migration as possible – there will be several iterations; there may be some time between them.Consider option of automated data cleanup (e.g. data formats for dates, phone numbers, etc.) if time allows.
  • The Dark Art of Data Migration

    1. 1. The Dark Art<br />of Data Migration<br />Presented by Graham Morgan<br />MD Spatial Consultants Ltd<br />
    2. 2. Tuning in …<br />
    3. 3. 3<br />Expert independent advice<br />
    4. 4. Is it that Difficult?<br />Drawings with layout<br />Features split<br />Implied data (color)<br />Multiple sources<br />Choreography<br />4<br />Missing data<br />Unstructured data<br />Spatial formats<br />Variability within<br />Different meanings<br />Inaccuracies<br />
    5. 5. Target - SpatialNet<br />All data is stored in an Oracle database<br />5<br />Data is:<br />Feature based<br />Highly organized<br />Classified / Categorized<br />Consistent<br />Data integrity rules<br />Standardized<br />
    6. 6. Target - SpatialNet<br />All data stored in Oracle database<br />6<br />
    7. 7. 7<br />SpatialInfo data model<br />
    8. 8. 8<br />Where do we start?<br />
    9. 9. Features or Drawings?<br />9<br />
    10. 10. Data Challenges<br />Where is the feature record?<br />Different data sources / formats<br />No single business identifier for features<br />Which attribute values are correct?<br />Multiple copies, alternatives<br />Do we have all the attributes?<br />Do we have connectivity information?<br />Infrastructure Level<br />Fiber level<br />10<br />
    11. 11. System Challenges<br />It isn’t just a data migration…<br />System footprints <br />Business functions<br />Data (CRUD)<br />Access security<br />11<br />Data migration is a component of a business improvement programme<br />
    12. 12. Data Migration Process<br />Requirements<br />Estimate / Quote<br />Design<br />Implement<br />Test<br />Deploy<br />Celebrate …<br />12<br />
    13. 13. Data Migration Process<br />13<br />
    14. 14. Data Migration Process - Prepare<br />Map existing business functions to systems<br />Existing / Planned<br />Inventory data<br />Data to migrate to SpatialNet / other systems?<br />Prepare data documentation<br />Develop baseline tests<br />Feature counts<br />Network trace tests<br />Make local corrections<br />14<br />
    15. 15. Data Migration Process - Develop<br />Exchange data in situ (lossless)<br />Load directly ‘As Is’ into Oracle<br />Pre-process for load option<br />Re-project<br />Inventory / study / profile / explore / specify<br />Migrate to intermediate model<br />Validate<br />Migrate to SpatialNet<br />Validate<br />Correct data at source<br />… Repeat …<br />15<br />
    16. 16. 16<br />Migration Route<br />
    17. 17. Profile / Explore<br />Defend against duplicates – ensure unique keys<br />Missing features<br />Attribute value patterns / consistency<br />Mandatory, Unique, Domain values<br />Logical consistency<br />Geometry types & consistency<br />Connectivity: inferred vs. recorded<br />17<br />
    18. 18. Tools<br />Source to Oracle Staging<br />FME<br />FDO/.NET<br />Manifold<br />Oracle Staging to Neutral<br />PL/SQL<br />FME<br />NHibernate/.NET<br />18<br /><ul><li>AutoCAD
    19. 19. ESRI ArcGIS
    20. 20. MapInfo
    21. 21. Talend / Open Source
    22. 22. Oracle Integration
    23. 23. 1Spatial Radius</li></li></ul><li>Neutral Model<br />19<br />
    24. 24. Promote to SpatialNet<br />Standard load procedure<br />Dictionary mapping<br />Apply symbol rotation, generate annotation<br />Validation reports<br />20<br />
    25. 25. Shortcuts<br />Pitfalls<br />Data analysis in Oracle- rich toolset- neutral model<br />Correct data at source- with familiar tools<br />Train users on SpatialNet during migration- with familiar data<br />Script everything<br />Automate some clean up<br />Square Peg / Round Hole- start migration before defining scope & context<br />Migrate in isolation<br />Let the consultants figure it out<br />Plan for single migration<br />21<br />
    26. 26. Thank You<br />Graham MorganSpatial Consultants Ltd<br /><br />