Your SlideShare is downloading. ×
Oracle to Netezza Migration Casestudy
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Oracle to Netezza Migration Casestudy

9,231
views

Published on

Oracle to Netezza Migration Casestudy

Oracle to Netezza Migration Casestudy

Published in: Technology

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,231
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
33
Comments
0
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Oracle to NetezzaMigrationWhitepaperThis white paper provides key challenges faces and resolvedand the approach followed in Oracle to Netezza migration forone of the fortune 100 Retailer.Asis Mohanty, asismohanty@gmail.com11/11/2011
  • 2. Table of Contents1. Introduction .......................................................................................................................................... 32. Migration Approach and Process .......................................................................................................... 33. Impact Analysis ..................................................................................................................................... 3 3.1 Database Impact Analysis: .................................................................................................................. 3 3.2 ETL/ELT Impact Analysis: ..................................................................................................................... 4 3.3 Applications and Reports Impact Analysis: ......................................................................................... 44. Design and Development ...................................................................................................................... 4 4.1 Database Objects Design and Development Considerations: ............................................................ 4 4.2 ETL/ELT Design and Development Considerations: ............................................................................ 5 4.3 Report Design and Development Considerations: .............................................................................. 55. History Load .......................................................................................................................................... 56. Testing ................................................................................................................................................... 67. Lesson Learnt and Recommendations .................................................................................................. 78. Appendix ............................................................................................................................................... 9 Automation Recommendations ................................................................................................................ 9 About the Author ...................................................................................................................................... 9 | Oracle to Netezza Migration 2
  • 3. 1. IntroductionEvery organization wants to have advance analytics, real time or near real time data which will help thenin Business Intelligence and competitive Intelligence. High volume growth, scalability, Maintainabilitycost is the critical factor for any organization. Netezza provides a platform which is highly Scalable withavailability, easy to integrate and maintain, cost effective (Pay per usage), Asymmetric Massively ParallelProcessing architecture for performance.This white paper contains the approach followed, key challenges faces and resolved during the Oracle toNetezza migration program for one of the Fortune 100 Retailer.2. Migration Approach and ProcessOracle to Netezza migration has various SDLC process and steps as given below (Figure-1). Figure-1: Various steps involved in Oracle to Netezza Migration3. Impact AnalysisThis is the most critical phase to identify all impacted Data Base Objects, ETLs and applications/Reportsrequired for downstream.3.1 Database Impact Analysis:Identify all impacted Data Base Objects such as Tables, Views, Stored procedures, Index, Partitions,Volumetric (Volume of Data to be migrated) to be migrated from existing Oracle platform to NetezzaPlatform. Create the impact inventory list of DB Objects. | Oracle to Netezza Migration 3
  • 4. Figure-2: Example of Existing DB & ETL EnvironmentFor example in the above figure (Figure-2), as part of Data Base Object impact identify all Tables, Views,Stored procedures, Index, Partitions, Volumetric being used in Stage, Warehouse and Martenvironment.3.2 ETL/ELT Impact Analysis:Identify all ETL mapping or Jobs and ELT scripts being used by all impacted DB Objects. Create theinventory list of all impacted ETL mappings or Jobs and ELT Scripts along with patterns.For example in the above figure (Figure-2), as part of ETL/ELT impact identify all ETL/ELT Scripts beingused from Source to Stage, Stage to Warehouse and Warehouse to MART.3.3 Applications and Reports Impact Analysis:Identify all Reports or applications which are using the Warehouse or Mart Data. Create the inventorylist of all impacted Reports and applications. Figure-3: Example of Existing Reporting and Application EnvironmentFor Example as shown in the above figure (Figure-2) identify all Reports and various application which isusing the Warehouse or Mart DB Objects (Tables, Views .etc).4. Design and DevelopmentThis section contains the design and development of Database Objects, ETL/ELT andReports/Applications.4.1 Database Objects Design and Development Considerations: • As-Is migration verses re-architecture of Data model is most critical factor. | Oracle to Netezza Migration 4
  • 5. • Identify all Oracle Vs Netezza DB Object data type, Size and compatibility. For example the Date time stamp format, CLOB, LONG Data type etc. • Identify all Oracle Vs Netezza DB functions. Data type and function will be helpful while converting and Oracle SQL to Netezza SQL. • Netezza does not support Index where as it supports Distribution Key. While creating the new Netezza Tables the Oracle Table Index must be considered.4.2 ETL/ELT Design and Development Considerations: • Identify different patterns of ETL Mapping/Jobs. • Identify all impacted ETL where Source connection to be modified from Oracle to Netezza. (Example: All Sources to Stage ETL will fall under this category. Refer Figure-1) • Identify all impacted ETL where Source and Target connection to be modified from Oracle to Netezza. (Example: All ETL Mapping/Jobs being used in Stage to Warehouse and Warehouse to Mart load) • Identify all embedded SQL queries used in ETL mappings/Jobs which need to be converted from Oracle to Netezza specific SQL Query. • Identify all ELT Scripts where embedded Oracle SQL queries to be converted to Netezza SQLs. • Make the Netezza specific changes in ETL Mappings/Jobs and do unit testing. Early Unit testing identifies various issues.4.3 Report Design and Development Considerations: • Identify different patterns of Reports. • Identify all embedded SQL queries used in reports by report. • Identify all matrices which used Oracle specific Functions and Data type which needs to be converted to Netezza specific functions and data type. • Make Netezza specific changes in Reports. There might be some extra space, rounding, Special Character issue which needs to be addressed during development phase.5. History LoadHistory load from Oracle to Netezza can be done in following methods. • Extract Oracle Table to flat file using Oracle Database utilities. Load the flat file to corresponding Netezza table using NZ Load. (Always use Bulk load/insert/update utility which is faster) • Use the ETL tool to create one time mapping which will load history data from Oracle to Netezza. • Use Unix/Perl script to extract data from Oracle and load it to Netezza. • Use Third Party tool which uses the Bulk Read and Write mechanism to migrate history data from Oracle to Netezza. Wisdomforce Fastreader (www.wisdomforce.com/products- FastReader.html) supports the bulk/fast load from Oracle to Netezza. | Oracle to Netezza Migration 5
  • 6. • Define the History data Delta if the history data volume is high, for example the existing Enterprise Data Warehouse has 6 Years worth of Data. Define two to three deltas each 2 Years worth of data to load.6. TestingTesting is the most important phase to identify any issue. Following are few of the recommendations: • Database Object Testing o Compare the DB Objects (Oracle Vs Netezza) along with distribution Key. o Create automated script which can compare the DB Objects by querying System table scans. • SQL Testing o Compare the Oracle SQL results with Netezza SQL results. The SQL used in ETL, Scripts, Reports must be compared during the development phase. o There are third party tolls available for Oracle to Netezza SQL conversion such as Sqissql (www.swissql.com). Even automated Perl or UNIX scripts can be created to convert the SQL. • ETL Testing/Data Testing: o Run the existing Oracle ETL Job and new Netezza ETL Job. Compare the target Oracle table Vs Netezza table data. o If the Table volume is small, do a byte to byte data comparison of the data (Except Data Time Stamp fields) o If the volume is high use mathematical function (Min, Max and Average of additive field) along with count match. Do a sampling using Min/Max functions. o Created automated script to compare the Data. This script and method will be used for History data comparison too. • Report/Application Testing: o Run the report/Application in Oracle and Netezza environment. Compare the report manually or use Reporting environment specific tool to compare the result. For Example Microstrategy has ‘Microstrategy integrity manager’ which compares the reports. o There are third party tools available which compared Byte-to-Byte of .csv, pdf, xls, html reports, for example Beyondcompare (www.scootersoftware.com) . • Performance Testing: o Capture the ETL run statistics while running in Oracle environment and Netezza Environment. Compare the ETL load run statistics to know the performance. This must be done for the load window too. o Capture the Report run statistics for Oracle and Netezza environment. Compare the run statistics for Report performance in new Netezza environment. | Oracle to Netezza Migration 6
  • 7. 7. Lesson Learnt and RecommendationsFollowing are few of the lessons learnt from the Oracle to Netezza Migration Project. POC o Do POC for early identification of issues and approach of Migration. Product Issue Resolution o There might be product related issue (Compatibility) which must be resolved quickly. o Netezza Special Characters (Ex: Trademark) might appear differently in reports. o Rounding of matrices in Reports. o Extra space in matrices, field of Reports. Resource o Cross skilled resource (resource having Oracle and Netezza Knowledge) Automation accelerators o Automation is the key to avoid any manual mistake during the Analysis, Development and Testing. Following Figure-4 shows the automations recommendations for Oracle to Netezza Migration. Figure-4: Automation Recommendations in various Phases Upgrade of Reporting and ETL Environment | Oracle to Netezza Migration 7
  • 8. o Upgrade of ETL environment or Reporting environment along with DB Platform migration is not recommended. Performanceo Perform Performance testing for any early indication of system performance. Usage of Advance Feature of Netezzao Use the advance feature such as NZ Load utilities for better performance for high volume of data load. | Oracle to Netezza Migration 8
  • 9. 8. AppendixAutomation RecommendationsTools to Convert Oracle to Netezza SQLs and DDLs: Create Unix/Perl script which converts the Oraclespecific DDL or SQL to Netezza specific DDL/SQL. This script will use the Oracle Vs Netezza Data Type,Functions and patterns to convert.Tools to Convert the Oracle ETL to Netezza ETL: Create a tool which takes ETL xml as input and changesthe Oracle tag to Netezza tag (connections, extra Tags) and converts Oracle embedded SQLs to NetezzaSQLs.Data Testing Tool: Create a Unix/Perl script/tool which compares the Oracle Table data Vs Netezza Tabledata. Following are the two patterns:Low volume Table: Compare the byte-to-byte data.High Volume Table: Compare the Count of records, use mathematical functions such as Min, Max, Sum,Average .etc for numeric field. Use sampling of data comparison (Top 10 records using Rank functions.etc)About the AuthorAsis Mohanty has more than 11 Years of Industry experience on Data Warehousing and BusinessIntelligence field. He is a Certified Business Intelligence Professional from www.tdwi.org and CertifiedData Management Professional from www.dama.org and Netezza Silver Certified Professional. For anyquery contact Asis Mohanty at asismohanty@gmail.com . | Oracle to Netezza Migration 9