Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QA Challenges in Data Integration

2,799 views

Published on

33% of your Data Integration project timeline can be reduced with proper QA methodologies. Lower risks in your Data warehouse, Data Migration, ETL, Data Lake and MDM projects.

Published in: Technology
  • Be the first to comment

QA Challenges in Data Integration

  1. 1. QA Challenges in Data Integration Projects & iCEDQ Software Data Warehouse | ETL | Data Migration | Data Lake | MDM ETL Testing & Monitoring Platform
  2. 2. The QA Challenge Lack of Automation Lack of Agility Clutter No Test Driven Development Lack of Regression Testing Data Warehouse | ETL | Data Migration | Data Lake | MDM www.icedq.com
  3. 3. vs. iCEDQ has in-Memory Rules Engine  Tests ETL processes by Validating the output data Validation Rule Reconciliation Rule  Tests ETL Transformations by Reconciling Source Data Vs. Target Data ETL  Too many tools – Excel, SQL, Scripting  Not repeatable and consistent  Can’t test Millions of rows  Forced to do sampling  Time wasted in coding rather than testing  Disorganized  Time & resource consuming iCEDQ Shift www.icedq.com
  4. 4. Waterfall SDLC vs. Agile iCEDQ Shift www.icedq.com
  5. 5. Clutter vs. Organized & Documented START Business Requirements Test Results Workflow QA Dashboard Audit (Testing) Requirements Audit Rules & Scripts Test Cases iCEDQ Repository iCEDQ Shift www.icedq.com
  6. 6. Conventional Waterfall SDLC vs.  Testing is too late in the project. 8th Step!  QA team sitting idle for developers to finish  Last minute rush Step 1. Get Requirements Step 2. Design the Tables Step 3 Create Data Mapping Step 4. Develop ETL Code Step 5. Get Test Requirements Step 6. Write Test Cases Step 7. Write Physical Test Step 8. Test Step 9. Deploy Step 1. Get Requirements Step 2. Design the Tables Step 3 Create Data Mapping Step 4. Develop ETL Code Step 5. Test Step 6. Deploy 7 8 9 Refactor Reduce timeline by 33% Step 5. Get Audit Requirements Step 6. Write Test Cases Step 7. Create Physical Rules  iCEDQ enables Test First Approach  Test Driven Development - TDD  Lift n’ Shift the QA to the beginning of projectiCEDQ -TDD iCEDQShift www.icedq.com
  7. 7. Quality ? Star t QA QC Regression Testing & Release Management Conventional data integration testing lacks…  Tools for regression testing  Sign-Off & release management  Continues Build Management  Quality Control iCEDQ iCEDQ Shift Business Requirements Audit Requirements Test Cases Regression Testing Deployment Release Management Quality Control & Production Monitoring Source Sys. Biz. Users Management www.icedq.com
  8. 8. Why iCEDQ?  Automation  Test high volume of data  Regression Testing  Need Agile and/or Test Driven Development  Lower Project Risks & Cost  Improve Productivity  Compliance  Security  Organized & Collaborative  Reporting, Dashboard & Notification  Transparency & Feedback  Shrink Project Timeline Data Warehouse | ETL | Data Migration | Data Lake | MDM www.icedq.com
  9. 9. 1x 100x 10,000x Save $$$$$$ Early is Better! CostofDelay$ Time  www.icedq.com
  10. 10. iCEDQ SERVER iCEDQ Architecture Audit Rules Engine iALMiGOVERNiQAiAUDIT Scheduler Repository SQL Server |Oracle Configuration Files Parameter Files Log Files Data Exceptions Files GUI Web Service CLI/Shell RPC Report JDBC Text File Cobol* Other Files Mainframe* Data Adapters Oracle | SQL Server | Redshift | DB2 | My SQL | Netezza |… Hadoop HIVE | HP Vertica XML | EDI | JSON CSV | Fixed Width | Excel Cobol Files ISAM | KISM | KSDS | QSAM |VSAM BIG DATA Redshift, HADOOP, Hive, S3, HP VERTICA, Cassandra, MONGODB, HDFS…www.icedq.com
  11. 11. Who uses iCEDQ?  Stock Exchange  Banks  Insurance  Manufacturing  Healthcare  E-Commerce  Manufacturing  … www.iCEDQ.com
  12. 12. game changer: noun a newly introduced element or factor that changes an existing situation or activity in a significant way ETL Testing & Monitoring Platform - Merriam Webster www.iCEDQ.com

×