Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Migrations: How to avoid the #1 project pitfall

41,844 views

Published on

Migrating data has become one of the most challenging initiatives for IT managers. Although these projects yield high business benefits (such as cost savings, increased productivity, and improved data manageability), they tend to involve a high level of risk due to the volume and criticality of the data being moved.

So What Is The #1 Reason Why Many Data Migration Projects Fail?

Failure to validate and test throughout the data migration process.

You can save your project from this common pitfall! Attend the webinar to learn:

- Why validation and testing is so important
- How to integrate data testing into your project schedule
- How to automate data validation and testing
- How to show project stakeholders that your migration is free of bad data
- What tool and process will help formalize your validation and testing

Published in: Technology
  • Be the first to comment

Data Migrations: How to avoid the #1 project pitfall

  1. 1. QuerySurge Bill Hayduk CEO Christopher Thompson Senior Domain Expert Data Migrations: How to avoid the #1 project pitfall
  2. 2. Today’s Agenda • Data Migration o Business Reasons o Risks o Costs o Scary Metrics o Common Defects • #1 Project Pitfall • QuerySurge Solution • Demo AGENDA Data Migration o Business Reasons o Risks o Costs o Scary Metrics o Common Defects #1 Project Pitfall QuerySurge Solution Demo built by
  3. 3. QuerySurge™ FACTS Founded: 1996 Locations: New York, NY (Headquarters) Customer profile: • Fortune 1000 • 700+ customers Software Offering QuerySurge (2011) Tech Partners: IBM, Microsoft, HPE, Oracle, Teradata, Cloudera, HortonWorks, MongoDB RTTS is the parent company of QuerySurge and is the premier pure-play QA & Testing organization that specializes in test automation a software division of
  4. 4. 80 percent of data migration projects fail to meet expectations, running over time and budget. - Gartner 83 percent of migrations experience problems - IBM survey 80% of organizations… will underestimate the costs related to the data acquisition tasks by an average of 50 percent - Gartner Data Migrations built by QuerySurge™
  5. 5. Data Migration – Top Reasons Top reasons for Data Migration • Decommissioning legacy systems • Adding new line of business • New deployment (i.e. ERP or CRM system) • Consolidation of systems • Relocation • Switching vendors built by QuerySurge™
  6. 6. • the average cost for a Data Migration = $875,000 • 34% of migrations have missing or lost data • 38% have some form of data corruption • 64% of migration projects have unexpected outage/downtime • typical cost of downtime: o $6.5 million/hour - Brokerage industry o $2.8 million/hour - Energy Industry • 72% of organizations deferred because data migration is too risky Data Migration – Scary Metrics built by QuerySurge™
  7. 7. • Unexpected downtime • Budget overruns • Customer or Brand Impact • Data corruption • Application performance issues • Data loss Data Migration – Key Risks Key Risks built by QuerySurge™
  8. 8. • Remediation activities • Server outage • Application outage • Labor costs • Software Tools • System Maintenance Data Migration – Cost Factors Key Cost Factors built by QuerySurge™
  9. 9. Data Migration – Key Pitfalls • Failing to engage the lines of business at the outset • Absence of data governance policies and organizational structure • Poor data quality in a legacy system • Neglecting to validate and redefine business rules • Failure to validate and test the data-migration process Key Pitfalls of Migration Projects • Failure to validate and test the data-migration process built by QuerySurge™
  10. 10. "You really need to make sure that you're validating and testing throughout the process”. “Unfortunately companies often don't spend enough time aligning the data testing, validation, and migration cycles to the project timeline”. - InformationWeek Question: How are you going to test the data? Failure to validate and test the process built by QuerySurge™
  11. 11. Data Migration – Popular Test Strategy Post-Migration Testing • Row Counts • Comparing migrated records to source using sampling Question: Is there a better way? built by QuerySurge™
  12. 12. 12 Our Solution: Automated Testing of the Migration Process built by QuerySurge™
  13. 13. a software division of
  14. 14. is the leading Data Testing solution for automated validation & testing of Big Data QuerySurge QuerySurge™ Use Cases a software division of
  15. 15. • Reduce your costs & risks • Improve your data quality • Accelerate your testing cycles • Share information with your team with QuerySurge™ you can: built by QuerySurge™ • Provides huge ROI (i.e. 1,300%)* *based on client’s calculation of Return on Investment
  16. 16. QuerySurge™ Automate the entire testing cycle  Automate the test execution, the comparison, auto-email results Query Wizards - no coding needed  Query Wizards ensure minimal time & effort to create tests Test across different platforms  Data Warehouse, Hadoop, NoSQL, DB, flat files, XML, mainframe Analyze & Collaborate  Data Health dashboard, shared tests & auto-emailed reports Verify more data & do it quickly  Verifies up to 100% of all data up to 1,000 x faster Integrate for Continuous Delivery (DevOps)  Integrates with most Build, Configuration, ETL & QA mgmt solutions a software division of
  17. 17. Web-based… Installs on... Connects through… …to any JDBC compliant data store QuerySurge™ QuerySurge Controller QuerySurge Server QuerySurge Agents a software division of Installs in... the Cloud a VM
  18. 18. QuerySurge™ a software division of QuerySurge supports the following JDBC drivers: • Amazon Redshift • Cassandra • Couchbase • Exasol • Flat Files (delimited, fixed-width) • JSON • Hadoop/Hive (Apache, Cloudera, Hortonworks, Pivotal) • HPE Vertica • IBM (DB2, Netezza, Informix) • Mainframe • MAPR • Microsoft (SQL Server, PDW, SSAS, Excel, Access, Sharepoint) • MongoDB • Oracle (Oracle, MySQL, Exadata) • Pivotal GreenPlum • PostgreSQL • Salesforce • SAP (HANA, IQ, ASE, SQL Anywhere) • Spark • Teradata, Aster • WorkDay • XML And any other JDBC-compliant data store
  19. 19. 6/8/2017 19 Developer: Codes data movement based on Mapping Requirements Target Data Warehouse ETL Tester: Tests data movement based on Mapping Requirements Data Mart ETL Source Data Target Big Data Mainframe and other ETL tools…
  20. 20. QuerySurge™ Design Library SchedulingRun Dashboard Deep-Dive Reporting Data Health Dashboard Query Wizards a software division of
  21. 21. Fast and Easy. No programming needed. QuerySurge™ • Perform 80% of all data tests with no SQL coding • Opens up testing to novices & non-technical members • Speeds up testing for skilled coders • provides a huge Return-On-Investment a software division of
  22. 22. Design Library • Create custom Query Pairs (source & target SQLs) • Great for team members skilled with SQL Scheduling  Build groups of Query Pairs  Schedule Test Runs for: • immediately • at a specific date/time • automatically after build or ETL process built by QuerySurge™
  23. 23. Design Library • Create custom Query Pairs (source & target SQLs for tests that have transformations) Scheduling  Build groups of Query Pairs  Schedule Test Runs • Run immediately • Run at set date/time • Have event kick it off QuerySurge™ a software division of
  24. 24. Deep-Dive Reporting  Examine and automatically email test results Run Dashboard  View real-time execution  Analyze real-time results QuerySurge™ a software division of
  25. 25. QuerySurge™ • view data reliability & pass rate • add, move, filter, zoom-in on any data widget & underlying data • verify build success or failure a software division of
  26. 26. QuerySurge™  Drive QuerySurge execution from your Test Management Solution  See QuerySurge Pass/Fail results in your Test Management solution  Click link to drill into detailed results in QuerySurge Integration with leading Test Management Solutions • HPE ALM (Quality Center) • Microsoft Team Foundation Server • IBM Rational Quality Manager a software division of
  27. 27. QuerySurge™ Automated Testing Automated Reporting Automated Launch Data Integration/ETL solutions QuerySurge™ and many others… email report and many others… QuerySurge™ Build/Configuration solutions email report Test Management solutions QuerySurge™ email report a software division of
  28. 28. QuerySurge™ a software division of A 2-Step Process
  29. 29. Target Database Data Migration: Data Flow Source Data Migration Process ETL & EL SQL SQL SQL SQL SQL SQL built by QuerySurge™
  30. 30. • Automated Row Counts • Automated Column Compares (no transformations) • Automated Table Compares (no transformations) #1: Automated testing with Query Wizards (no programming) #2: Automated testing with custom QueryPairs (custom SQL) • All tables with transformations are compared • Upwards of 100% coverage of all data quickly SQL SQLSQL SQL SQL SQL SQL SQL 2 Steps: built by QuerySurge™
  31. 31. built by QuerySurge™ 3 Types of Data Comparison Wizards: The also provide you with automated features for: o filtering (‘Where’ clause) and o sorting (‘Order By’ clause) Column-Level Comparison: This is great for Big Data stores and Data Warehouses where tables will have some columns containing transformations and some columns with no transformations. Many tables and columns can be compared simultaneously and quickly. Table-Level Comparison: This comparator is great for Data Migrations and Database Upgrades with no transformations at all. Many tables can be compared simultaneously and quickly. Row Count Comparison: Great for all - Big Data stores, Data Warehouses, Data Migrations and Database Upgrades. Many tables and rows can be compared simultaneously and quickly.
  32. 32. 32 Fast and Easy. No programming needed. • Perfect for data migrations & upgrades • Great for verifying that all rows have come over without data issues • Superb at verifying table-to-table transfers • Ability to graphically create SQL stubs for advanced scripting Step #1: built by QuerySurge™
  33. 33. QuerySurge™ a software division of
  34. 34. Uses: • Verify that the amount of rows match the amount that were expected • Great for smoke testing Result: Quickly compares rows of all tables Skill set / Value added: beginner: no coding needed experienced user: saves time built by QuerySurge™
  35. 35. Uses: Tests the columns that have no transformations, which means it tests approximately 80% of your data store without you writing any SQL code Tests: Big data, data warehouses Skill set / Value added: novice or non-technical: no coding needed, productive immediately experienced user: saves time built by QuerySurge™
  36. 36. Uses: - Verifies tables from source and target match Result: Can perform 100s of tables in a few minutes. Skill set / Value added: beginner: no coding needed experienced user: saves time built by QuerySurge™
  37. 37. Step #2: built by QuerySurge™ Uses: - Create custom tests to mimic your transformations Result: - Can test all of you transformed data in your columns Skill set/Value added: - Need SQL skills. Can test all of your data Design Library • Create custom Query Pairs (source & target SQLs for tests that have transformations)
  38. 38. • Improve your data governance & data quality • Accelerate your data delivery cycles • Reduce your costs & risks • Realize a huge ROI QuerySurge™ a software division of
  39. 39. Ensuring Data Warehouse Qualitybuilt by QuerySurge™ Christopher Thompson Senior Domain Expert QuerySurge To see the video of this demonstration please visit: http://bit.ly/2sHWyO7
  40. 40. QuerySurge built by QuerySurge™ a software division of Any questions before the demo?
  41. 41. 41 • SAP Data Migration: Turning Risk into Opportunity, Jerome Delune, EIM Global Practice Director, SAP Consulting • Data Migration Best Practices and Nondisruptive Migration Service Capability for Enterprise Storage, Hitachi Data Systems • Risks and Challenges in Data Migrations and Conversions, Gartner • Data Migration – Key Considerations, Glen Bell, SearchDatamanagement.com • Data Migration in the Global 2000, Bloor Research • Zero-Downtime Upgrades and Migrations with Oracle GoldenGate, Oracle • 10 Big Data Migration Mistakes, InformationWeek • How to Implement an Effective Data Migration Testing Strategy, DataMigration Pro magazine, David Katzoff, Managing Director of Product Development, Valiance Partners • The Hidden Costs of Data Migration, IBM References built bybuilt by

×