Your SlideShare is downloading. ×

The Dark Art of Data Migration

755
views

Published on

Presented at the SPATIALinfo conference in Denver on 9th June 2011. …

Presented at the SPATIALinfo conference in Denver on 9th June 2011.

Published in: Technology

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
755
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • SPATIALinfo conference, Denver, CO, 9th June 2011Planning for the migration of digital spatial data into SpatialNet
  • NW England: Lake District, Britain’s Energy Coast
  • GIS consulting for clients worldwideSpatial systems & data integrationData migrationInformation strategyApplication development
  • Why should it be difficult – the source data is in a system that has probably successfully supported the business for a number of years.
  • Relational tablesAttributes with data typesGeometry also stored in native OracleIntegrity rules to enforce consistencyStandards based – other programs can access it.Structure means that the data can be queried with confidence to support business decision making.
  • All data, including geometries, can be accessed by Oracle tools – and many 3rd party tools.Geometry can be queried and manipulated through SQL.
  • The SpatialNet data model is complex as it supports many purposes.
  • Data held across different systems in different formatsHopefully with some form of documentationProbably some duplication with differentiationAs successful business applications often get more complex over time (as they take on additional responsibilities) then it becomes important to fully understand the match (and gaps) between the source and target data.Drawings are only fully intelligible to the trained human eye.
  • Data associated by proximity on a drawingExtended entity data, blocks, etc.GIS systems can also be configured to store data as annotation (i.e. unstructured) ….
  • Fiber connectivity is not represented spatially – it must be provided by attribution.
  • Does the new system provide all the business functions of the existing system?Does all of the existing data map neatly into the model of the new system?Can the new system be accessed by all of the people that currently access the existing system?Existing system is likely to have evolved over time and may have assumed some additional responisbilities.An architect is required to have overall visibility of the entire system replacement project.The data migration is only one project within a larger programme.
  • Dream on….Activities are generally correct, they are all required – but must expect several iterations to refine.
  • Iterations generally get fasterFirst is the longest as there is more to put in place; later iterations are just refinements; final is very fast
  • Magicians are really illusionists – there is always an explanation – Preparation is key.Master data may be in other systems!Must understand the CRUD matrix for the data being migrated – in which applications will each attribute be maintained? And what integration points are required (inbound and outbound) to maintain the data in the new system.Correct data errors in the existing system – greater levels of expertise and efficiency with the familiar system.Prepare for acceptance testing – how will we know when the data has been migrated correctly?What goes in is what came out.How to exchange data between source system/s and migration environment?
  • Investigate Oracle loading options for spatial data.Standardize DictionariesSymbol Orientation, Annotation positioning, Feature placement (e.g. dist between cables)Pre-processing – minimum changes to prepare the data for loading into Oracle using the selected tools. E.g. drop constraints, change reserved words, protect Identity columns, etc.
  • Load source into Oracle environment with minimal processing – use Oracle-based tools to process data with consistency.Bulk of the work happens in migrating from the staging environment to the neutral model.
  • Geometry values – recorded Vs calculatedOracle may well provide a better environment for data profiling than source applications
  • Every migration is unique.Need to be flexible with use of the most appropriate toolsYet try to be consistent in tool use.Script the migration as far as possible to enable efficient iterations.
  • Neutral model is relatively simple, easy to read and review.
  • Tom Ward presentation on afternoon of June 9th.
  • Support for Just in Time training.Script as much of the migration as possible – there will be several iterations; there may be some time between them.Consider option of automated data cleanup (e.g. data formats for dates, phone numbers, etc.) if time allows.
  • Transcript

    • 1. The Dark Art
      of Data Migration
      Presented by Graham Morgan
      MD Spatial Consultants Ltd
    • 2. Tuning in …
    • 3. 3
      Expert independent advice
    • 4. Is it that Difficult?
      Drawings with layout
      Features split
      Implied data (color)
      Multiple sources
      Choreography
      4
      Missing data
      Unstructured data
      Spatial formats
      Variability within
      Different meanings
      Inaccuracies
    • 5. Target - SpatialNet
      All data is stored in an Oracle database
      5
      Data is:
      Feature based
      Highly organized
      Classified / Categorized
      Consistent
      Data integrity rules
      Standardized
    • 6. Target - SpatialNet
      All data stored in Oracle database
      6
    • 7. 7
      SpatialInfo data model
    • 8. 8
      Where do we start?
    • 9. Features or Drawings?
      9
    • 10. Data Challenges
      Where is the feature record?
      Different data sources / formats
      No single business identifier for features
      Which attribute values are correct?
      Multiple copies, alternatives
      Do we have all the attributes?
      Do we have connectivity information?
      Infrastructure Level
      Fiber level
      10
    • 11. System Challenges
      It isn’t just a data migration…
      System footprints
      Business functions
      Data (CRUD)
      Access security
      11
      Data migration is a component of a business improvement programme
    • 12. Data Migration Process
      Requirements
      Estimate / Quote
      Design
      Implement
      Test
      Deploy
      Celebrate …
      12
    • 13. Data Migration Process
      13
    • 14. Data Migration Process - Prepare
      Map existing business functions to systems
      Existing / Planned
      Inventory data
      Data to migrate to SpatialNet / other systems?
      Prepare data documentation
      Develop baseline tests
      Feature counts
      Network trace tests
      Make local corrections
      14
    • 15. Data Migration Process - Develop
      Exchange data in situ (lossless)
      Load directly ‘As Is’ into Oracle
      Pre-process for load option
      Re-project
      Inventory / study / profile / explore / specify
      Migrate to intermediate model
      Validate
      Migrate to SpatialNet
      Validate
      Correct data at source
      … Repeat …
      15
    • 16. 16
      Migration Route
    • 17. Profile / Explore
      Defend against duplicates – ensure unique keys
      Missing features
      Attribute value patterns / consistency
      Mandatory, Unique, Domain values
      Logical consistency
      Geometry types & consistency
      Connectivity: inferred vs. recorded
      17
    • 18. Tools
      Source to Oracle Staging
      FME
      FDO/.NET
      Manifold
      Oracle Staging to Neutral
      PL/SQL
      FME
      NHibernate/.NET
      18
    • Neutral Model
      19
    • 24. Promote to SpatialNet
      Standard load procedure
      Dictionary mapping
      Apply symbol rotation, generate annotation
      Validation reports
      20
    • 25. Shortcuts
      Pitfalls
      Data analysis in Oracle- rich toolset- neutral model
      Correct data at source- with familiar tools
      Train users on SpatialNet during migration- with familiar data
      Script everything
      Automate some clean up
      Square Peg / Round Hole- start migration before defining scope & context
      Migrate in isolation
      Let the consultants figure it out
      Plan for single migration
      21
    • 26. Thank You
      Graham MorganSpatial Consultants Ltd
      gmorgan@spatialconsultants.comwww.spatialconsultants.com