Your SlideShare is downloading. ×

OWB to ODI converter white paper

362

Published on

OWB to ODI migration document by D&T

OWB to ODI migration document by D&T

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
362
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. White paper
  • 2. INTRODUCTION OWB v.10.2 premier support ended on July 2010. OWB v.11.1 premier support will end on August 2012. OWB v.11.2 (available from September 2009) premier support will end on January 2015. No major enhancements are planned for OWB beyond the OWB 11.2 release. Earlier versions (9.2 and 10.1) of OWB will no longer be supported. ODI is Oracle’s strategic product for high performance flexible heterogeneous data integration. If the client has to migrate or wants to migrate from OWB to ODI, it needs to plan a manual transfer of the OWB components to equivalent ODI components. Product overview OWB2ODI Converter acquires OWB repository’s metadata and it generates corresponding ODI repository metadata. Algorithms in OWB are transferred in ODI, and OWB operators are transformed in corresponding ODI operators when present or are substituted by a custom solution. Supplied service Conversion service of an Oracle Warehouse Builder project to a corresponding Oracle Data Integrator project. Additive services Performance tuning, ODI and DBA consulting, ODI and DBA education. Supported product versions OWB v. 9.2 or higher ODI v. 10,11 g, 12c Conversion process After the initial assessment phase with the client, a specific Converter’s component extracts metadata from the OWB .mdl file. D&T uses the OWB2ODI Converter in its laboratories to automatically generate a corresponding ODI project from the original OWB project. The generated ODI project will be sent to the client for acceptance test and rollout to production. The conversion process consists of several tasks, which are divided into 5 phases: 1. Assessment 2. Conversion 3. Test 4. Parallel 5. Production INTRODUCTION Page 2 of 18
  • 3. PHASE 1: ASSESSMENT The assessment phase has following objectives: - To define the project framework, - To evaluate OWB project consistency (component number and type), - To define the conversion criteria. Phase 1 - ASSESSMENT Page 3 of 18
  • 4. The Assessment phase is composed of the following tasks: - Task 1: conversion assessment and statistics report generation, - Task 2: handling exceptions, - Task 3: topology definition, - Task 4: conversion mode explanation, - Task 5: KMs definition, - Task 6: configuration management definition. Task 1 needs an OWB .mdl file export, which the client has to send to the D&T services centre. The .mdl file export must: - Contain all project Locations, - Contain the Configuration, if it is different from default, - Contain all mappings and all process flows in validated state, - Contain all dependences. After the generation of the report, the client and D&T organize a meeting to execute tasks 2 to 6. This meeting and relative decisions are a basilar rock upon which the entire conversion project will be built. Task 1 - Conversion assessment and statistics report generation The .mdl file supplied by the client has to be validated and must include location and configuration parameters. It represents the necessary input for converter analyzer. Converter Analyzer can perform a multitude of operations. It analyses whether any special operators exist, for which essential changes are required. Then it deals with the analysis of each mapping. It starts from the target operator and ends on the leftmost operator, ensuring the entire mapping structure and its related operators remain unchanged. Phase 1 - ASSESSMENT Page 4 of 18
  • 5. The Converter analyzer generates the following report on the OWB process flows and mappings: PROCESS FLOWS ACTIVITY_TYPE AMOUNT OR TRANSFORMATION MAPPING SET_STATUS SUBPROCESS AND ASSIGN FORK FILE_EXISTS USER_DEFINED WAIT END_LOOP FOR_LOOP ROUTE WHILE_LOOP 7.942 5.995 2.199 1.930 1.505 1.217 332 252 92 49 21 21 20 14 1 TOTAL NUM_PROCESS_FLOWS 21.590 1.000 MAX_NUM_ACTIVITY_IN_PF 200 NUM_PROCESS_FLOWS_LOW NUM_PROCESS_FLOWS_MEDIUM NUM_PROCESS_FLOWS_HIGH 720 180 100 FIRST _FOLDER_NAME Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A Folder_A etc. SECONDARY _FOLDER_NAME Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 Folder_1 etc. ACTIVITY _PER_PF PF_NAME Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process Process etc. PF_ STATE 10 6 35 24 13 42 14 46 18 21 19 12 15 52 13 16 16 7 9 75 80 12 9 7 47 LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW LOW MEDIUM MEDIUM LOW LOW LOW LOW flow A1.1 flow A1.2 flow A1.3 flow A1.4 flow A1.5 flow A1.6 flow A1.7 flow A1.8 flow A1.9 flow A1.10 flow A1.11 flow A1.12 flow A1.13 flow A1.14 flow A1.15 flow A1.16 flow A1.17 flow A1.18 flow A1.19 flow A1.20 flow A1.21 flow A1.22 flow A1.23 flow A1.24 flow A1.25 MAPPINGS OPERATOR_TYPE AMOUNT TABLE EXPRESSION FILTER JOINER CONSTANT INPUT_PARAMETER AGGREGATOR SET_OPERATION SPLITTER PREMAPPING_PROCESS POSTMAPPING_PROCESS SEQUENCE FLAT_FILE VIEW DEDUPLICATOR TRANSFORMATION SORTER 7.011 2.928 1.590 1.336 687 630 603 447 346 323 183 151 128 89 52 51 4 TOTAL NUM_MAPPINGS MAX_NUM_OPERATORS_IN_MAPPING NUM_MAPPINGS_LOW NUM_MAPPINGS_MEDIUM NUM_MAPPINGS_HIGH Phase 1 - ASSESSMENT 16.559 1.000 50 400 300 300 MODULE _NAME Module 1 Module 1 Module 1 Module 1 Module 1 Module 1 Module 2 Module 2 Module 2 Module 2 Module 2 Module 2 Module 2 Module 2 Module 2 Module 3 Module 3 Module 3 Module 3 Module 3 Module 3 Module 3 Module 3 Module 4 Module 4 Module 4 Module 4 etc. MAPPING _NAME Mapping 1.1 Mapping 1.2 Mapping 1.3 Mapping 1.4 Mapping 1.5 Mapping 1.6 Mapping 2.0 Mapping 2.1 Mapping 2.2 Mapping 2.3 Mapping 2.4 Mapping 2.5 Mapping 2.6 Mapping 2.7 Mapping 2.8 Mapping 3.1 Mapping 3.2 Mapping 3.3 Mapping 3.4 Mapping 3.5 Mapping 3.6 Mapping 3.7 Mapping 3.8 Mapping 4.1 Mapping 4.2 Mapping 4.3 Mapping 4.4 etc. OPERATOR _PER_MAPPING MAPPING _STATE 3 12 7 21 13 8 19 21 25 32 14 6 9 12 18 23 34 2 6 7 10 27 16 11 4 4 30 LOW LOW LOW LOW LOW LOW LOW LOW MEDIUM MEDIUM LOW LOW LOW LOW LOW MEDIUM MEDIUM LOW LOW LOW LOW MEDIUM LOW LOW LOW LOW MEDIUM Page 5 of 18
  • 6. D&T and the client discuss the conversion assessment & statistics report in a specific meeting organized to execute tasks 2 to 6. During the meeting, the client and D&T will compile the “Roles and responsibilities matrix”. This matrix assigns roles and responsibilities to the client and D&T for every task within the conversion process. Task 2 – Exceptions handling This task is dedicated to deciding how to manage (work-around or manual conversion) any eventual case that cannot be automatically converted to ODI. A case can be represented by: • A particular OWB operator in a mapping, • A particular OWB activity or a particular transition condition in a process flow, • An external workflow layer (not in OWB) used to execute the OWB project’s components. These are extremely rare contingencies: most used OWB components are automatically converted. Task 3 – Topology Definition In ODI, the definition of the topology, logical architecture, and physical architecture are necessary to indicate where the data are physically located. To correctly collect these informations, the OWB2ODI Converter analyzes the OWB Repository Metadata to obtain: • Technology • Machine where the data is located • Database schema and/or file metadata ODI was developed to operate with any database technologies available (Oracle, DB2, Teradata, SQL Server, and many others). ODI provides the ability to refer to a database schema/user by its logical form. The logical user or logical schema is an abstract reference to a physical schema, which is defined within ODI as a real user in a specific database technology. Logical schemas and physical schemas are related through contexts. For example, the BUDGET logical schema may be associated with the BUDGET physical schema in the ORCL database through the CTX_BUDGET_ORCL context, etc. Everything is configured in ODI’s topology section, which contains all the information needed to switch between the logical and physical side. Phase 1 - ASSESSMENT Page 6 of 18
  • 7. The OWB2ODI Converter only works on logical schemas because the pointers to physical schemas are configured in the topology, and the context is assigned either at runtime or when “conversions” are executed. The result of that process shows how easy it is to generate a technology-independent code. As long as the right context is set up, everything will work properly. Please note: the logical schema is always associated with its own technology, although it can be easily moved to another by deleting it from the old platform and building it in the new one. Task 4 – Conversion mode explanation This task is dedicated to give a detailed explanation of the conversion mode of every OWB component to the client‘s project team. The OWB2ODI Converter can convert: • OWB mappings into ODI interfaces (with ODI 11g)/ ODI Mapping (with ODI 12c) and ODI packages, • OWB process flows logic into ODI native tools, ODI packages, ODI procedures and ODI load plans (available since ODI v.11.1.1.5). Task 5 – KMs definition ODI’s capabilities to handle any RDBMSs are represented by its KMs (Knowledge modules). The OWB2ODI Converter has a specific console to set parameters and options related to the use of the KMs. The present task will affect ODI project’s performances. The following is an example of how it is possible to customize the OWB2ODI Converter to choose the KMs better suited to the client’s technology and needs. KMs customizable choosing grid SOURCE TECH TARGET TECH TYPE OF LOADING ORACLE ORACLE ORACLE ORACLE ORACLE ORACLE DSNTIAUL DB2 DB2 INSERT UPDATE INSERT/UPDATE TRUNCATE/INSER Client Custom KM DB2 DB2 KNOWLEDGE MODULE Oracle SQLLDR Oracle Incremental Update D&T Custom KM Phase 1 - ASSESSMENT T INSERT/UPDATE DEFAULT OPERATING MODE SET BASED SET BASED SET_BASED SET BASED SET BASED FAIL OVER ROW BASED Page 7 of 18
  • 8. … … … … … D&T customized KMs can also manage other aspects such as: • Management of ANALYZE on target table, • Management of the maximum number of allowable errors on target table, • Management of loading HINT, • Management of selecting HINT, • Etc… Task 6 – Configuration management definition The last, but not the least important task. To obtain best results, it would be opportune to freeze any maintenance activity about OWB project to convert. The ideal scenario would be: - To rollout every OWB project undergoing modification into the client’s production environment, - To align each of the client’s environment (development, test and production), - To freeze any maintenance activity. The client has to decide if freezing is possible or not. If not, how does the client intend to execute configuration management of the OWB project to be converted. In this case, the client and D&T have to define a detailed operative protocol to be applied during the conversion period. During this task, the “fixing protocol”, which is the management process (roles, responsibilities, actions, etc.) needed to fix any bugs identified during the acceptance test task, has to be defined. Phase 1 - ASSESSMENT Page 8 of 18
  • 9. PHASE 2: CONVERSION The Conversion phase has the following objectives: - To convert an original OWB project into a new ODI project, - To generate the new ODI project’s metadata for the future import into the client ODI repository. The Conversion phase is composed of the following tasks: - Task 7: conversion of OWB Mapping and OWB Process Flow, - Task 8: formal test, Phase 2 - CONVERSION Page 9 of 18
  • 10. - Task 9: ODI project metadata generation. Task 7 is the core of the entire conversion process and it is executed by the D&T services centre. Before starting the task, the following should be carried out: - To supply a .mdl file aligned with the latest OWB project version, - To supply a DB schema export aligned with the latest OWB project version, - To freeze any maintenance activity related to the OWB project to convert. Task 8 is executed by the D&T services centre and only concerns formal correctness of the new ODI project, because data are not available for a real test run. Task 9 is dedicated to the transmission of the new ODI project metadata generated by the Converter. Task 7 – Conversion of OWB Mapping and OWB Process Flow According to task 3 “Topology definition” results, the D&T services centre carries out the ODI topology setting for its internal conversion environment. Then, all mappings are analyzed and a first transformation is applied whenever specific operators are found. This requires an onerous process in order to guarantee the proper functioning of the new Oracle Data Integrator flow, while keeping the semantic flow unchanged. This task, performed automatically by the OWB2ODI Converter, shows how costly and time-consuming it would be to carry out an entire manual conversion, not to mention the risk of introducing new mistakes due to haste or technical misunderstandings. Once additional mappings are “normalized”, recursive techniques are used to generate the operators tree of each mapping. The tree is retraced and each operator involved is transformed according to the new Oracle Data Integrator semantic. The correct topological information is maintained, considering the possible overruling of the location using a database of links or different schemas. The following tables indicate which OWB components the OWB2ODI Converter automatically converts and which components have to be converted via manual activity. From a conceptual point of view, OWB and ODI are similar, but they do not have equivalent features and have deep and significant differences. Thus, the OWB2ODI Converter does not handle some of the OWB components because they are rarely used or do not have a corresponding ODI function or because conversion would be too complex or ineffective or inefficient. Phase 2 - CONVERSION Page 10 of 18
  • 11. OWB mapping: converted operators Aggregator Joiner Sorter Constant Splitter Deduplicator (Distinct) Mapping input/output parm Mapping Match-Merge Expression Materialized View Transformation External Table Pivot/Unpivot View Filter Pre/Post mapping process Anydata Cast (since 11.1) Flat File (File multirecord) Sequence Key Lookup Set Operation Table OWB mapping: not handled operators Dimension Expand object Pluggable mapping Cube Varray iterator Queue (11.2) Construct Name and address Subquery filter (11.2) Data generator Table function LCR cast/splitter (11.2) OWB process flow: converted activities Data auditor FTP And / Or Mapping Manual End Subprocess Notification End Loop Transform Set status For Loop Assign Sqlplus While Loop Email User Defined Fork File Exists Wait Route OWB process flow: converted transition conditions Success Error Complex OWB process flow: not handled activities Web services (11.2) EJB / Java class (11.2) OMB plus (11.2) OWB process flow: not handled transition conditions Warning* ∗ Extended Since ODI does not have a “warning status”, the OWB “warning transition condition” is handled depending on each client’s specific needs. This is a possible topic to be covered in the assessment meeting. The OWB “Fork activity” is converted by using the ODI “Load plan” feature that is only available from Phase 2 - CONVERSION Page 11 of 18
  • 12. ODI v.11.1.1.5 onwards. Therefore, in order to manage and convert all of the most used and useful activities in the Process Flow, we need an ODI version that is equal or greater than ODI v.11.1.1.5. Task 8 – Formal test This is the first formal non-regression test executed on empty data structures, without data. This is carried out by the D&T services centre. Task 9 – ODI project metadata generation The last task of the conversion process is dedicated to the ODI project metadata generation. These metadata are included in the .xml files that are delivered to the client. Phase 2 - CONVERSION Page 12 of 18
  • 13. PHASE 3: TEST The test phase has following objectives: - To compare new ODI project results with old OWB project results (acceptance test), - To tune new ODI project performances. Phase 3 - TEST Page 13 of 18
  • 14. The test phase is composed of the following tasks: - Task 10: test environment arrangement, - Task 11: acceptance test, - Task 12: performance tuning. Task 10 – Test environment arrangement During this task, each test environment’s element has to be arranged. In detail it is necessary: - To install OWB product, - To install ODI product, - To copy OWB project’s components, - To import the two .xml files supplied from D&T services centre into ODI repository, - To copy the production DB twice: one copy for OWB project and another copy for ODI project. Task 11 – Acceptance test The ODI project is tested to ensure that regression issues are not present. In detail, a complete parallel run of the two projects is executed and then respective results are compared to verify their matching. Each incorrect result is investigated to determine relative causes. The fixing protocol established during the client/D&T initial meeting (Phase 1 – Assessment) is applied for each bug detected. Task 12 – Performance tuning The last step of the Test phase is related to performances. During this task, the ODI Project’s performances are tuned with data base administrator support. Phase 3 - TEST Page 14 of 18
  • 15. PHASE 4: PARALLEL The parallel phase has following objectives: - To assure new ODI project correctness in Production environment such as the Test environment, - To compare and eventually tune new ODI project performances compared to the original OWB project in Production environment. Phase 4 - PARALLEL Page 15 of 18
  • 16. The test phase is composed of the following tasks: - Task 13: parallel environment arrangement, - Task 14: verification test, - Task 15: performance verification. Task 13 – Parallel environment arrangement During this task, each parallel environment’s element has to be prepared in order to: - Deploy ODI project, - Copy production DB. Obviously, the DB copy has to be executed just before the parallel run is started. Task 14 – Verification test Theoretically, this is a redundant and unnecessary task, but experience teaches us that a parallel production period must be carried out in order to locate any possible problems that may affect the production environment (configuration, privileges, missing patches, etc.). The comparison of results of two projects (just executed during the Test phase) is executed again. Task 15 – Performance verification Theoretically, this is also a redundant and unnecessary task, but experience teaches us that a parallel comparison must be carried out between the original OWB project and new ODI project performances in the production environment. Phase 4 - PARALLEL Page 16 of 18
  • 17. PHASE 5: PRODUCTION The production phase has following objectives: - To remove the original OWB project and its pertinences (OWB installation, DB schema, etc.) from production environment, - To switch the scheduling tool from the original OWB project to the new ODI project. The test phase is composed of the following tasks: - Task 16: production environment cleaning, - Task 17: switching from OWB to ODI. Task 16 – Production environment cleaning The cleaning consists of: - OWB uninstallation, - DB schema deletion, - Removal of any other software components that are not necessary to the new ODI project running. Task 17 – Switching from OWB to ODI This consists of physical switching from the old OWB project to the new ODI project. Phase 5 - PRODUCTION Page 17 of 18
  • 18. CONTACTS For more information about OWB2ODI Converter, please visit www.owb2odiconverter.com or visit www.databtech.com or send an email to dt.marketing@databtech.net or call +39 02 8950 0080 to speak to a D&T representative. Database & Technology s.r.l. Largo Promessi Sposi, 4 20142 – Milano, Italy CONTACTS Page 18 of 18

×