The Dark Art of Data Migration

  • 733 views
Uploaded on

Presented at the SPATIALinfo conference in Denver on 9th June 2011. …

Presented at the SPATIALinfo conference in Denver on 9th June 2011.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
733
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
0
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • SPATIALinfo conference, Denver, CO, 9th June 2011Planning for the migration of digital spatial data into SpatialNet
  • NW England: Lake District, Britain’s Energy Coast
  • GIS consulting for clients worldwideSpatial systems & data integrationData migrationInformation strategyApplication development
  • Why should it be difficult – the source data is in a system that has probably successfully supported the business for a number of years.
  • Relational tablesAttributes with data typesGeometry also stored in native OracleIntegrity rules to enforce consistencyStandards based – other programs can access it.Structure means that the data can be queried with confidence to support business decision making.
  • All data, including geometries, can be accessed by Oracle tools – and many 3rd party tools.Geometry can be queried and manipulated through SQL.
  • The SpatialNet data model is complex as it supports many purposes.
  • Data held across different systems in different formatsHopefully with some form of documentationProbably some duplication with differentiationAs successful business applications often get more complex over time (as they take on additional responsibilities) then it becomes important to fully understand the match (and gaps) between the source and target data.Drawings are only fully intelligible to the trained human eye.
  • Data associated by proximity on a drawingExtended entity data, blocks, etc.GIS systems can also be configured to store data as annotation (i.e. unstructured) ….
  • Fiber connectivity is not represented spatially – it must be provided by attribution.
  • Does the new system provide all the business functions of the existing system?Does all of the existing data map neatly into the model of the new system?Can the new system be accessed by all of the people that currently access the existing system?Existing system is likely to have evolved over time and may have assumed some additional responisbilities.An architect is required to have overall visibility of the entire system replacement project.The data migration is only one project within a larger programme.
  • Dream on….Activities are generally correct, they are all required – but must expect several iterations to refine.
  • Iterations generally get fasterFirst is the longest as there is more to put in place; later iterations are just refinements; final is very fast
  • Magicians are really illusionists – there is always an explanation – Preparation is key.Master data may be in other systems!Must understand the CRUD matrix for the data being migrated – in which applications will each attribute be maintained? And what integration points are required (inbound and outbound) to maintain the data in the new system.Correct data errors in the existing system – greater levels of expertise and efficiency with the familiar system.Prepare for acceptance testing – how will we know when the data has been migrated correctly?What goes in is what came out.How to exchange data between source system/s and migration environment?
  • Investigate Oracle loading options for spatial data.Standardize DictionariesSymbol Orientation, Annotation positioning, Feature placement (e.g. dist between cables)Pre-processing – minimum changes to prepare the data for loading into Oracle using the selected tools. E.g. drop constraints, change reserved words, protect Identity columns, etc.
  • Load source into Oracle environment with minimal processing – use Oracle-based tools to process data with consistency.Bulk of the work happens in migrating from the staging environment to the neutral model.
  • Geometry values – recorded Vs calculatedOracle may well provide a better environment for data profiling than source applications
  • Every migration is unique.Need to be flexible with use of the most appropriate toolsYet try to be consistent in tool use.Script the migration as far as possible to enable efficient iterations.
  • Neutral model is relatively simple, easy to read and review.
  • Tom Ward presentation on afternoon of June 9th.
  • Support for Just in Time training.Script as much of the migration as possible – there will be several iterations; there may be some time between them.Consider option of automated data cleanup (e.g. data formats for dates, phone numbers, etc.) if time allows.

Transcript

  • 1. The Dark Art
    of Data Migration
    Presented by Graham Morgan
    MD Spatial Consultants Ltd
  • 2. Tuning in …
  • 3. 3
    Expert independent advice
  • 4. Is it that Difficult?
    Drawings with layout
    Features split
    Implied data (color)
    Multiple sources
    Choreography
    4
    Missing data
    Unstructured data
    Spatial formats
    Variability within
    Different meanings
    Inaccuracies
  • 5. Target - SpatialNet
    All data is stored in an Oracle database
    5
    Data is:
    Feature based
    Highly organized
    Classified / Categorized
    Consistent
    Data integrity rules
    Standardized
  • 6. Target - SpatialNet
    All data stored in Oracle database
    6
  • 7. 7
    SpatialInfo data model
  • 8. 8
    Where do we start?
  • 9. Features or Drawings?
    9
  • 10. Data Challenges
    Where is the feature record?
    Different data sources / formats
    No single business identifier for features
    Which attribute values are correct?
    Multiple copies, alternatives
    Do we have all the attributes?
    Do we have connectivity information?
    Infrastructure Level
    Fiber level
    10
  • 11. System Challenges
    It isn’t just a data migration…
    System footprints
    Business functions
    Data (CRUD)
    Access security
    11
    Data migration is a component of a business improvement programme
  • 12. Data Migration Process
    Requirements
    Estimate / Quote
    Design
    Implement
    Test
    Deploy
    Celebrate …
    12
  • 13. Data Migration Process
    13
  • 14. Data Migration Process - Prepare
    Map existing business functions to systems
    Existing / Planned
    Inventory data
    Data to migrate to SpatialNet / other systems?
    Prepare data documentation
    Develop baseline tests
    Feature counts
    Network trace tests
    Make local corrections
    14
  • 15. Data Migration Process - Develop
    Exchange data in situ (lossless)
    Load directly ‘As Is’ into Oracle
    Pre-process for load option
    Re-project
    Inventory / study / profile / explore / specify
    Migrate to intermediate model
    Validate
    Migrate to SpatialNet
    Validate
    Correct data at source
    … Repeat …
    15
  • 16. 16
    Migration Route
  • 17. Profile / Explore
    Defend against duplicates – ensure unique keys
    Missing features
    Attribute value patterns / consistency
    Mandatory, Unique, Domain values
    Logical consistency
    Geometry types & consistency
    Connectivity: inferred vs. recorded
    17
  • 18. Tools
    Source to Oracle Staging
    FME
    FDO/.NET
    Manifold
    Oracle Staging to Neutral
    PL/SQL
    FME
    NHibernate/.NET
    18
  • Neutral Model
    19
  • 24. Promote to SpatialNet
    Standard load procedure
    Dictionary mapping
    Apply symbol rotation, generate annotation
    Validation reports
    20
  • 25. Shortcuts
    Pitfalls
    Data analysis in Oracle- rich toolset- neutral model
    Correct data at source- with familiar tools
    Train users on SpatialNet during migration- with familiar data
    Script everything
    Automate some clean up
    Square Peg / Round Hole- start migration before defining scope & context
    Migrate in isolation
    Let the consultants figure it out
    Plan for single migration
    21
  • 26. Thank You
    Graham MorganSpatial Consultants Ltd
    gmorgan@spatialconsultants.comwww.spatialconsultants.com