Migrating data: How to reduce risk


Published on

A recommended methodology for reducing the risks involved in data migration.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Migrating data: How to reduce risk

  1. 1. Migrating data: Howto reduce riskwww.etlsolutions.com
  2. 2. Data migration: A risk!
  3. 3. A real-life example of the risks involvedWe were called in to rescue a project in which tools hadnot been used:• The vendor hand-wrote the migration to target system• Rewritten three times due to: • Structural limitations • Performance (twice)• Bugs in code delayed the migration• Bugs in code caused incorrect data, making it difficult to test actual migration logic
  4. 4. Recommended best practice Business ownership: transparency, control, engagement Use of toolsets Use of a project management methodology A team with skills and experience in data migration Use standards wherever applicable Use of a data migration methodology
  5. 5. Recommended team leader objectives Identify and mitigate risks Ensure confidence in the project team Appropriate resourcing Keep the project on track:  Within budget  On time Reduce the overall cost and effort needed
  6. 6. Overview of our recommended data migration process Business engagement Scoping Core Legacy migration decommissioning Assessments
  7. 7. Embed best practice within the process• Migration methodology: an in-depth process for each stage of the data migration, bringing in specialists at key times and working to a structured plan and documents• Project management methodology: We use PRINCE2 to ensure that the migration is well managed to a common standard• ISO standards: We use 15288,12207,and 8000 to ensure the methodologies have a core standard structure• Migration software: The core migration steps should be carried out using a well established tool• Migration specialists: Specialists should be available for the duration of the project
  8. 8. Data migration process: Assessments Staff competency assessment Project assessment• Helps to identify gaps in the • A strategic review of the existing skill base proposed project, examining:• Shows where training may be • Plans required • Processes• Reduces reliance on external • Workflows experts • Data targets• Grows the confidence of the • Systems project team • Identifies and mitigates risks• Provides individuals with greater and issues before they occur clarity about their roles• Our proprietary Data Migration Competency Framework is designed specifically to support data migration teams
  9. 9. Data migration process: Scoping Project scoping Technical scoping• A detailed, tactical examination • A detailed examination of the of the proposed project, project’s technical structure: including: • Available models • Stakeholders and deliverables • Available software expected • Data volume and quality • Budgets • Identifies potential technical • Deadlines issues before they occur • Communication plans • Available experts: business domain, systems, migration experts• Helps with project planning to mitigate risk• Provides business leaders with a clear view of the project plan
  10. 10. Data migration methodology Iterative agile development is used throughout Project scoping Core migration Configuration Landscape analysis Data assurance Migration design Migration developmentRequirements analysis Data discovery Data review Testing design Testing development Data modelling Data cleansing Execution Review Legacy decommissioning
  11. 11. Landscape analysis in more detail• Landscape analysis encompasses the systematic process of identifying all source and target systems that may be involved in the data migration• Gain an overview of the source and target systems• Key tasks: • Understand how each system works • Understand how the data within each system is structured • Model the systems • Model links and interactions between systems • Model data structures
  12. 12. Data assurance in more detail• Data assurance puts measures in place to ensure that all Data assurance questions: information used within the • What data migration and data migration is handled profiling tools are available? accurately • Are there key areas of• Data quality is a key element, weakness in data? along with data cleansing • Are rules for the data quality where required (attribute and relational) within• Planning is required for the the source already documented? retirement of data, for deletion • Which governing rules have to or for storage due to industry be applied to the data? regulations • Will all data be migrated?• Key tasks: • Data review • Data cleansing
  13. 13. The key data assurance tasks Data review Data cleansing• Profiling is carried out to identify • Define the cleansing rules which areas of the data that may not will be carried out manually and be of sufficient quality to meet those which will be automated: business requirements • The manual cleansing will be• Data quality definitions define typically be carried out prior to the quality that must be attained migration for elements, attributes and • The automated cleansing will be relationships within the source carried out as a first step of the migration or indeed may also be system; these definitions will be able to be carried out prior to used during the profiling to migration verify if the data adheres to the • Data verification is focused on the rules defined checking of data being available,• The system retirement plan accessible, in the correct format defines which data will be and complete moved from the old system to • Data impact analysis is carried the new and what is no longer out to evaluate the effect on other required elements and systems
  14. 14. What happens when data assurance is omitted? • Multinational 3-way multi-company venture • US vendor migrating data • UK project management • No migration management • Lack of resource • No verification or quality rules • No profiling carried out • Ad-hoc testing • Security delays of 6 months • Testing time increased by 3 months • 1 month delay for invalid characters • Additional dry-run required due to issues encountered
  15. 15. Core migration in more detail• Use a structured approach with methodologies to guide the process• Establish clear structures and boundaries: a methodology will help with this• Ensure tool-based execution: we use our own commercially available software, Transformation Manager• Carry out tool-based testing• Create an organisation-focused go live: scoping the project at an early stage can assist in disseminating the information to stakeholders around timings and resource required to achieve the goals
  16. 16. A core migration design
  17. 17. Our offerings for data management Transformation Data Manager data migration movement packaged software services Support, training and mentoring services
  18. 18. Why Transformation Manager?For the user:  Everything under one roof  Greater control and transparency  Identify and test against errors iteratively  Greater understanding of the transformation requirement  Automatically document  Re-use and change management  Uses domain specific terminology in the mapping
  19. 19. Why Transformation Manager?For the business:  Reduces cost and effort  Reduces risk in the project  Delivers higher quality and reduces error  Increases control and transparency in the development  Single product  Reduces time to market
  20. 20. Contact us for more information: Karl Glenn, Business Development Director kg@etlsolutions.com +44 (0) 1912 894040 Read more on our website: www.etlsolutions.com Raising data management standardswww.etlsolutions.com www.etlsolutions.com Images from Free Digital Photos freedigitalphotos.net