Data Migrations: How to avoid the #1 project pitfall


Published on

Migrating data has become one of the most challenging initiatives for IT managers. Although these projects yield high business benefits (such as cost savings, increased productivity, and improved data manageability), they tend to involve a high level of risk due to the volume and criticality of the data being moved.

So What Is The #1 Reason Why Many Data Migration Projects Fail?

Failure to validate and test throughout the data migration process.

You can save your project from this common pitfall! Attend the webinar to learn:

- Why validation and testing is so important
- How to integrate data testing into your project schedule
- How to automate data validation and testing
- How to show project stakeholders that your migration is free of bad data
- What tool and process will help formalize your validation and testing

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • QuerySurge helps your team coordinate your data quality initiatives while speeding up your development and testing cycles and finding your bad data. Why risk having your team identify trends and develop strategic initiatives when the underlying data is incorrect? QuerySurge reduces this risk.
  • Your distributed team from around the world can use any of these web browsers: Internet Explorer, Chrome, Firefox and Safari.
    Installs on operating systems: Windows & Linux.

    QS connects to any JDBC-compliant data source. Even if it is not listed here.
  • QuerySurge can utilized by active practitioners such as testers & developers to create and launch tests, or by managers, analysts and operations to view data test results and the overall health of the data. QuerySurge facilitates this by providing 2 types of licenses: (1) full user & (2) participant user.

    (1) Full User – This type of user has unlimited access to create QueryPairs, Suites, and Scenarios. This user can also schedule and run tests, see results, run and export reports, and export data. Perfect for anyone creating and/or running data tests while performing analysis of results.

    (2) Participant User – This user cannot create or run tests, but has access to all other information - including viewing all query pairs, results, and reports, receiving email notifications, and exporting test results and reports. Perfect for managers, analysts, architects, DBAs, developers, and operations users who need to know the health of their data.

  • Data Migrations: How to avoid the #1 project pitfall

    1. 1. Bill Hayduk CEO/President RTTS Data Migrations: How to avoid the #1 project pitfall Christopher Thompson Senior Domain Expert QuerySurge Automate your Data Warehouse & Big Data Testing and Reap the Benefits built by
    2. 2. Today’s Agenda • Data Migration o Business Reasons o Risks o Costs o Scary Metrics o Common Defects • #1 Project Pitfall • QuerySurge Solution • Demo AGENDA Data Migration o Business Reasons o Risks o Costs o Scary Metrics o Common Defects #1 Project Pitfall QuerySurge Solution Demo built by
    3. 3. built by QuerySurge™ About FACTS Founded: 1996 Locations: New York (HQ), Atlanta, Philadelphia, Phoenix Strategic Partners: IBM, Microsoft, HP, Oracle, Teradata, HortonWorks, Cloudera, Amazon Software: QuerySurge RTTS is the leading provider of software & data quality for critical business systems
    4. 4. 80 percent of data migration projects fail to meet expectations, running over time and budget. - Gartner 83 percent of migrations experience problems - IBM survey 80% of organizations… will underestimate the costs related to the data acquisition tasks by an average of 50 percent - Gartner Data Migrations built by QuerySurge™
    5. 5. Data Migration – Top Reasons Top reasons for Data Migration • Decommissioning legacy systems • Adding new line of business • New deployment (i.e. ERP or CRM system) • Consolidation of systems • Relocation built by QuerySurge™
    6. 6. • the average cost for a Data Migration = $875,000 • 34% of migrations have missing or lost data • 38% have some form of data corruption • 64% of migration projects have unexpected outage/downtime • typical cost of downtime: o $6.5 million/hour - Brokerage industry o $2.8 million/hour - Energy Industry • 72% of organizations deferred because data migration is too risky Data Migration – Scary Metrics built by QuerySurge™
    7. 7. • Unexpected downtime • Budget overruns • Customer or Brand Impact • Data corruption • Application performance issues • Data loss Data Migration – Key Risks Key Risks built by QuerySurge™
    8. 8. • Remediation activities • Server outage • Application outage • Labor costs • Software Tools • System Maintenance Data Migration – Cost Factors Key Cost Factors built by QuerySurge™
    9. 9. Data Migration – Key Pitfalls • Failing to engage the lines of business at the outset • Absence of data governance policies and organizational structure • Poor data quality in a legacy system • Neglecting to validate and redefine business rules • Failure to validate and test the data-migration process Key Pitfalls of Migration Projects • Failure to validate and test the data-migration process built by QuerySurge™
    10. 10. "You really need to make sure that you're validating and testing throughout the process”. “Unfortunately companies often don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline”. - InformationWeek Question: How are you going to test the data? Failure to validate and test the process built by QuerySurge™
    11. 11. Data Migration – Popular Test Strategy Post-Migration Testing • Row Counts • Comparing migrated records to source using sampling Question: Is there a better way? built by QuerySurge™
    12. 12. 12 Our Solution: Automated Testing of the Migration Process built by QuerySurge™
    13. 13. QuerySurge is the premier test tool built to automate Data Testing What is QuerySurge? built by QuerySurge™
    14. 14. • Reduce your costs & risks • Improve your data quality • Accelerate your testing cycles • Share information with your team with QuerySurge™ you can: built by QuerySurge™ • Provides huge ROI (i.e. 1,300%)* *based on client’s calculation of Return on Investment
    15. 15. the QuerySurge advantage built by QuerySurge™ Automate the entire testing cycle  Automate kickoff, tests, comparison, auto-emailed results Create Tests easily with no SQL programming  ensures minimal time & effort to create tests / obtain results Test across different platforms  data warehouse, Hadoop, NoSQL, database, flat file, XML Collaborate with team  Data Health dashboard, shared tests & auto-emailed reports Verify more data & do it quickly  verifies up to 100% of all data up to 1,000 x faster Integrate for Continuous Delivery  Integrates with most Build, ETL & QA management software
    16. 16. QuerySurge™ Architecture Web-based… Installs on... Linux Connects to… …or any other JDBC compliant data source built by QuerySurge™ QuerySurge Controller QuerySurge Server QuerySurge Agents Flat Files
    17. 17. Target Database Data Migration: Data Flow Source Data Migration Process ETL & EL SQL SQL SQL SQL SQL SQL built by QuerySurge™
    18. 18. Collaboration Testers - functional testing - regression testing - result analysis Developers / DBAs - unit testing - result analysis Data Analysts - review, analyze data - verify mapping failures Operations teams - monitoring - result analysis Managers - oversight - result analysis Share information on the built by QuerySurge™
    19. 19. built by QuerySurge™ all QuerySurge™ Modules Design Library SchedulingDeep-Dive Reporting Run Dashboard Query Wizards Data Health Dashboard
    20. 20. Fast and Easy. No programming needed. built by QuerySurge™ QuerySurge™ Modules Compare by Table, Column & Row • Perform 80% of all data tests •Automatically generates SQL code • Opens up testing to novice & non- technical team members • Speeds up testing for skilled SQL coders • provides a huge Return-On-Investment
    21. 21. built by QuerySurge™ QuerySurge™ Modules 3 Types of Data Comparison Wizards: The also provide you with automated features for: o filtering (‘Where’ clause) and o sorting (‘Order By’ clause) Column-Level Comparison: This is great for Big Data stores and Data Warehouses where tables will have some columns containing transformations and some columns with no transformations. Many tables and columns can be compared simultaneously and quickly. Table-Level Comparison: This comparator is great for Data Migrations and Database Upgrades with no transformations at all. Many tables can be compared simultaneously and quickly. Row Count Comparison: Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades. Many tables and rows can be compared simultaneously and quickly.
    22. 22. Design Library • Create custom Query Pairs (source & target SQLs) • Great for team members skilled with SQL QuerySurge™ Modules Scheduling  Build groups of Query Pairs  Schedule Test Runs for: • immediately • at a specific date/time • automatically after build or ETL process built by QuerySurge™
    23. 23. Deep-Dive Reporting  Examine and automatically email test results Run Dashboard  View real-time execution  Analyze real-time results QuerySurge™ Modules built by QuerySurge™
    24. 24. QuerySurge Test Management Connectors built by QuerySurge™  Drive QuerySurge execution from your Test Management Solution  Outcome results (Pass/Fail/etc.) are returned from QuerySurge to your Test Management Solution  Results are linked in your Test Management Solution so that you can click directly into detailed QuerySurge results • HP ALM (Quality Center) • Microsoft Team Foundation Server • IBM Rational Quality Manager Integration with leading Test Management Solutions
    25. 25. Automated Data Migration Strategy • Automated Row Counts • Automated Column Compares (no transformations) • Automated Table Compares (no transformations) #1: Automated testing with Query Wizards (no programming) #2: Automated testing with custom QueryPairs (custom SQL) • All tables with transformations are compared • Upwards of 100% coverage of all data quickly SQL SQLSQL SQL SQL SQL SQL SQL 2 Steps: built by QuerySurge™
    26. 26. 26 Fast and Easy. No programming needed. • Perfect for data migrations & upgrades • Great for verifying that all rows have come over without data issues • Superb at verifying table-to-table transfers • Ability to graphically create SQL stubs for advanced scripting Step #1: built by QuerySurge™
    27. 27. Uses: • Verify that the amount of rows match the amount that were expected • Great for smoke testing Result: Quickly compares rows of all tables SQL Skill set / Value added: beginner: no coding needed experienced user: saves time built by QuerySurge™
    28. 28. Uses: Tests the columns that have no transformations, which means it tests approximately 80% of your data store without you writing any SQL code Tests: Big data, data warehouses SQL Skill set / Value added: novice or non-technical: no coding needed, productive immediately experienced user: saves time built by QuerySurge™
    29. 29. Uses: - Verifies tables from source and target match Result: Can perform 100s of tables in a few minutes. SQL Skill set / Value added: beginner: no coding needed experienced user: saves time built by QuerySurge™
    30. 30. Custom QueryPairs • Write Query against the source and against the target • QuerySurge quickly compares tables after transformation Step #2: built by QuerySurge™
    31. 31. 31 QuerySurge Value-Add QuerySurge provides value by either: in testing data coverage from < 1% to upwards of 100% in testing time by as much as 1,000 x combination of in test coverage while in testing time built by QuerySurge™
    32. 32. 32 Return on Investment (ROI) • redeployment of head count because of an increase in coverage and decrease in need for testers • an increase in better data due to shorter / more thorough testing cycle, possibly saving $ millions by preventing bad data. built by QuerySurge™
    33. 33. Ensuring Data Warehouse Quality Demonstration built by QuerySurge™ Christopher Thompson Senior Domain Expert QuerySurge To see the video of this demonstration please visit:
    34. 34. 34 • SAP Data Migration: Turning Risk into Opportunity, Jerome Delune, EIM Global Practice Director, SAP Consulting • Data Migration Best Practices and Nondisruptive Migration Service Capability for Enterprise Storage, Hitachi Data Systems, February 2012 • Risks and Challenges in Data Migrations and Conversions, Gartner, February 2009 • Data Migration – Key Considerations, Glen Bell,, May, 2011 • Data Migration in the Global 2000, Bloor Research, September 2007 • Zero-Downtime Upgrades and Migrations with Oracle GoldenGate 11g, Oracle, 2009 • 10 Big Data Migration Mistakes, InformationWeek, August 9, 2012 • How to Implement an Effective Data Migration Testing Strategy, DataMigration Pro magazine, David Katzoff, Managing Director of Product Development, Valiance Partners • The Hidden Costs of Data Migration, IBM, 2007 References built bybuilt by