Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to solve complex business requirements with Oracle Data Integrator?

4,981 views

Published on

Business requirements are always hard to implement, develop, operate and always changeable. In this session attendees will have some fact examples of turning unstructured data into structural meaning, writing complex queries without typing anything, adding function based joins, implementing CTAS (Create Table As Select) and IAS (Insert As Select) methods and simplifying business rules, writing optimized queries to decrease operational and development costs as well as the faster loads.
In this presentation, see how you can solve some of complex business requirements with Oracle Data Integrator's flexibility and ease of usage features.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

How to solve complex business requirements with Oracle Data Integrator?

  1. 1. http://gurcanorhan.wordpress.com @gurcan_orhan http://tr.linkedin.com/in/gurcanorhan Gürcan ORHAN Enterprise Data Warehouse Architect Ekol Logistics Corp. 23rd May 2017 How To Solve Complex Business Requirements with Oracle Data Integrator
  2. 2. Who Am I ? +20 years of IT experience. +14 years of DWH experience. +10 years of Oracle Data Integrator experience. +8 years of Oracle Warehouse Builder experience. Sybase Power Designer, ERwin Data Modeler, SDDM OBIEE, Cognos, Microstrategy, Business Objects, Qlikview, Tableau IBM Data Stage, SAP Data Services, Informatica, etc… Oracle Excellence Awards - Technologist of the Year 2011 : Enterprise Architect DWH & BI Chair : TROUG (Turkish Oracle User Group) Published Customer Snapshot for NODI @Oracle.com Published videos about ODI @Oracle.com (Oracle Media Network) Published OTN Podcasts about “Data Warehousing and ODI” “ODI and the Evolution of Data Integration” 3 different “2MTT”s Articles in OTech Magazine, SearchSoftwareQuality.com Annual panelist for ODTUG “Ask the Experts Panel : ODI” Presenter in OOW since 2010 (7 times in a row ⭐ ) Presenter in many OUG conferences in globe Presenter in various universities in Turkey 23RD MAY 2017 / #OTNEMEATOUR 2
  3. 3. Ekol Germany Warehousing Solutions begin with the Kardelen Facility 1996 2003 2010 2012 2014 2016 201520132011200820021990 Acquire STS Int. Transport Ekol Bosnia Ekol France Ekol Greece Ekol Ukraine Ekol Spain Ekol Bulgaria Ekol Czech Rep. Ekol Iran Ekol PolandEkol Italy Ekol Romania Ekol HungaryAcquire Unok/Unatsan Rainbow Replaced by Quadro (software) Intermodal operations Ro-Ro operations Established Ekol Milestones
  4. 4. USA Ekol Locations Currentcountries 15 80 locations 2017 23 countries 92 locations 2018 31 countries 104 locations 2019 42 countries 122 locations CN
  5. 5. Overview
  6. 6. 23RD MAY 2017 / #OTNEMEATOUR My Definition of ODI 6
  7. 7. 23RD MAY 2017 / #OTNEMEATOUR Before Start… 7
  8. 8. AGENDA
  9. 9. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data Definition; Unstructured Data refers to information that either does not have a pre- defined data model or is not organized in a pre-defined manner. Unstructured information is typically text-heavy, but may contain data such as dates, numbers, and facts as well. (wikipedia) 10
  10. 10. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data What if in database ? 11
  11. 11. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data Column 1 VARCHAR2(400) ZCTREERT tcell_ns:.ana.rcoss5.rcss.ONRM_RtttMo SNW TSP TSP SXCXN03 Column 2 (VARCHAR2 (4000) Authentication Failure An authentication Failure trap signifies that the sending protocol entity is the addressee of a protocol message that is not always properly authenticated. -ProbableCause(ZSS)=Authentication Failure -EventType(ZSS)=Security service violation start_nss_tags @AlarmId=10156479912789340 @ManagedObject=SubNetwork=ONRM_RootMo,SubNetwork=TCP,ManagedElement=SXCXN04 @SpecificProblem=Authentication Failure @ProposedRepairAction=NONE @Class=ZCTREERTSX end_nss_tags Source:ZSSRC_FM Real World (finding the exact data - location); 13
  12. 12. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data What are the rules to find matching record? 14
  13. 13. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data (REG_EXP) VALUE COLUMN_NAME VALUE COLUMN_NAME ZCTREERT COLUMN1 always COLUMN2 tcell_ns COLUMN1 properly COLUMN2 ana COLUMN1 authenticated COLUMN2 rcoss5 COLUMN1 ProbableCause COLUMN2 rcss COLUMN1 ZSS COLUMN2 ONRM_RtttMo COLUMN1 Authentication COLUMN2 SNW COLUMN1 Failure COLUMN2 TSP COLUMN1 EventType COLUMN2 TSP COLUMN1 ZSS COLUMN2 SXCXN03 COLUMN1 Security COLUMN2 Authentication COLUMN2 service COLUMN2 Failure COLUMN2 violation COLUMN2 An COLUMN2 start_nss_tags COLUMN2 authentication COLUMN2 @AlarmId COLUMN2 Failure COLUMN2 10156479912789300 COLUMN2 trap COLUMN2 @ManagedObject COLUMN2 signifies COLUMN2 SubNetwork COLUMN2 that COLUMN2 ONRM_RootMo COLUMN2 the COLUMN2 SubNetwork COLUMN2 sending COLUMN2 TCP COLUMN2 protocol COLUMN2 ManagedElement COLUMN2 entity COLUMN2 SXCXN04 COLUMN2 is COLUMN2 @SpecificProblem COLUMN2 the COLUMN2 Authentication COLUMN2 addressee COLUMN2 Failure COLUMN2 of COLUMN2 @ProposedRepairAction COLUMN2 a COLUMN2 NONE COLUMN2 protocol COLUMN2 @Class COLUMN2 message COLUMN2 ZCTREERTSX COLUMN2 that COLUMN2 end_nss_tags COLUMN2 is COLUMN2 Source COLUMN2 not COLUMN2 ZSSRC_FM COLUMN2 VALUE COLUMN_NAME ZCTREERT COLUMN1 rcoss5 COLUMN1 rcss COLUMN1 SNW COLUMN1 TSP COLUMN1 TSP COLUMN1 SXCXN03 COLUMN1 of COLUMN2 ZSS COLUMN2 ZSS COLUMN2 TCP COLUMN2 SXCXN04 COLUMN2 ZCTREERTSX COLUMN2 ZSSRC_FM COLUMN2 * Average of 80 ~ 200 records produces per alarm * Average of 10 ~ 30 records produces per alarm Remove unnecessary characters Find matching records 15
  14. 14. 23RD MAY 2017 / #OTNEMEATOUR Unstructured Data (Correlation) VALUE COLUMN_NAME ZCTREERT COLUMN1 rcoss5 COLUMN1 rcss COLUMN1 SNW COLUMN1 TSP COLUMN1 TSP COLUMN1 SXCXN03 COLUMN1 of COLUMN2 ZSS COLUMN2 ZSS COLUMN2 TCP COLUMN2 SXCXN04 COLUMN2 ZCTREERTSX COLUMN2 ZSSRC_FM COLUMN2 * Average of 10 ~ 30 records produces per alarm Found matching records VALUE COLUMN_NAME ZCTREERT COLUMN1 SXCXN03 COLUMN1 SXCXN04 COLUMN2 ZCTREERTSX COLUMN2 ZSSRC_FM COLUMN2 Location Found ZCTREERTSX * Average of 3 ~ 10 records produces per alarm Rank records Apply the rules 16
  15. 15. AGENDA
  16. 16. 23RD MAY 2017 / #OTNEMEATOUR SNP_EXP_TXT SNP_SCEN_REPORT SNP_SESS_STEP SNP_SESS_TASK SNP_SESS_TASK_LOG SNP_SESS_TXT_LOG SNP_SESSION SNP_STEP_LOG SNP_STEP_REPORT SNP_TASK_TXT SNP_VAR_DATA SNP_VAR_SESS SNP_PARAM_SESS SNP_SEQ_SESS SNP_SESS_TASK_LS SNP_SESS_STEP_LV SNP_EXP_TXT_HEADER Repository table list for logging of ODI stored in work repository schema Create these tables (without referential integrity, constraints, indexes, etc.) with a suffix or prefix in a different schema. Archiving ODI Logs, Required Tables ARC_SNP_EXP_TXT ARC_SNP_SCEN_REPORT ARC_SNP_SESS_STEP ARC_SNP_SESS_TASK ARC_SNP_SESS_TASK_LOG ARC_SNP_SESS_TXT_LOG ARC_SNP_SESSION ARC_SNP_STEP_LOG ARC_SNP_STEP_REPORT ARC_SNP_TASK_TXT ARC_SNP_VAR_DATA ARC_SNP_VAR_SESS ARC_SNP_PARAM_SESS ARC_SNP_SEQ_SESS ARC_SNP_SESS_TASK_LS ARC_SNP_SESS_STEP_LV ARC_SNP_EXP_TXT_HEADER 18
  17. 17. 23RD MAY 2017 / #OTNEMEATOUR Reverse all required tables in ODI Archiving ODI Logs, Reverse 19
  18. 18. 23RD MAY 2017 / #OTNEMEATOUR SNP_EXP_TXT ARC_SNP_EXP_TXT TRUNC(SNP_EXP_TXT.FIRST_DATE) < TRUNC(SYSDATE) - #V_Purge_Log_Retention (none) I_SNP_EXP_TXT Archiving ODI Logs, Create Interfaces 20
  19. 19. 23RD MAY 2017 / #OTNEMEATOUR SNP_SCEN_REPORT ARC_SNP_SCEN_REPORT SNP_SCEN_REPORT.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SCEN_REPORT.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention I_SNP_SCEN_REPORT (none) Archiving ODI Logs, Create Interfaces 21
  20. 20. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_STEP SNP_SESSION ARC_SNP_SESS_STEP SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention I_SNP_SESS_STEP SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 22
  21. 21. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_TASK SNP_SESS_STEP SNP_SESSION ARC_SNP_SESS_TASK I_SNP_SESS_TASK SNP_SESS_TASK.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_SESS_TASK.NNO = SNP_SESS_STEP.NNO AND SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention Archiving ODI Logs, Create Interfaces 23
  22. 22. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_TASK_LOG SNP_STEP_LOG SNP_SESS_STEP SNP_SESSION ARC_SNP_SESS_TASK_LOG SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention I_SNP_SESS_TASK_LOG SNP_SESS_TASK_LOG.NNO = SNP_STEP_LOG.NNO AND SNP_SESS_TASK_LOG.SESS_NO = SNP_STEP_LOG.SESS_NO AND SNP_SESS_TASK_LOG.NB_RUN = SNP_STEP_LOG.NB_RUN AND SNP_STEP_LOG.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_STEP_LOG.NNO = SNP_SESS_STEP.NNO AND SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 24
  23. 23. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_TXT_LOG SNP_SESS_TASK_LOG SNP_STEP_LOG SNP_SESS_STEP SNP_SESSION ARC_SNP_SESS_TXT_LOG SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention I_SNP_SESS_TXT_LOG SNP_SESS_TXT_LOG.SESS_NO = SNP_SESS_TASK_LOG.SESS_NO AND SNP_SESS_TXT_LOG.NNO = SNP_SESS_TASK_LOG.NNO AND SNP_SESS_TXT_LOG.NB_RUN = SNP_SESS_TASK_LOG.NB_RUN AND SNP_SESS_TXT_LOG.SCEN_TASK_NO = SNP_SESS_TASK_LOG.SCEN_TASK_NO SNP_SESS_TASK_LOG.NNO = SNP_STEP_LOG.NNO AND SNP_SESS_TASK_LOG.SESS_NO = SNP_STEP_LOG.SESS_NO AND SNP_SESS_TASK_LOG.NB_RUN = SNP_STEP_LOG.NB_RUN SNP_STEP_LOG.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_STEP_LOG.NNO = SNP_SESS_STEP.NNO SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 25
  24. 24. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESSION ARC_SNP_SESSION I_SNP_SESSION SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention (none) Archiving ODI Logs, Create Interfaces 26
  25. 25. 23RD MAY 2017 / #OTNEMEATOUR SNP_STEP_LOG SNP_SESS_STEP SNP_SESSION ARC_SNP_STEP_LOG I_SNP_STEP_LOG SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_STEP_LOG.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_STEP_LOG.NNO = SNP_SESS_STEP.NNO SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 27
  26. 26. 23RD MAY 2017 / #OTNEMEATOUR SNP_STEP_REPORT SNP_SCEN_REPORT ARC_SNP_STEP_REPORT I_SNP_STEP_REPORT SNP_SCEN_REPORT.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SCEN_REPORT.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_STEP_REPORT.SCEN_NO = SNP_SCEN_REPORT.SCEN_NO AND SNP_STEP_REPORT.SCEN_RUN_NO = SNP_SCEN_REPORT.SCEN_RUN_NO Archiving ODI Logs, Create Interfaces 28
  27. 27. 23RD MAY 2017 / #OTNEMEATOUR SNP_TASK_TXT SNP_SESS_TASK SNP_SESS_STEP SNP_SESSION ARC_SNP_TASK_TXT I_SNP_TASK_TXT SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_TASK_TXT.SESS_NO = SNP_SESS_TASK.SESS_NO AND SNP_TASK_TXT.NNO = SNP_SESS_TASK.NNO AND SNP_TASK_TXT.SCEN_TASK_NO = SNP_SESS_TASK.SCEN_TASK_NO SNP_SESS_TASK.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_SESS_TASK.NNO = SNP_SESS_STEP.NNO SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 29
  28. 28. 23RD MAY 2017 / #OTNEMEATOUR SNP_VAR_DATA ARC_SNP_VAR_DATA I_SNP_VAR_DATA SNP_VAR_DATA.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_VAR_DATA.FIRST_DATE) < TRUNC(SYSDATE) - #V_Purge_Log_Retention (none) Archiving ODI Logs, Create Interfaces 30
  29. 29. 23RD MAY 2017 / #OTNEMEATOUR SNP_VAR_SESS ARC_SNP_VAR_SESS I_SNP_VAR_SESS SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention (none) Archiving ODI Logs, Create Interfaces 31
  30. 30. 23RD MAY 2017 / #OTNEMEATOUR SNP_PARAM_SESS SNP_SESSION ARC_SNP_PARAM_SESS I_SNP_PARAM_SESS (ODI 11G) SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_SESSION.SESS_NO = SNP_PARAM_SESS.SESS_NO Archiving ODI Logs, Create Interfaces 32
  31. 31. 23RD MAY 2017 / #OTNEMEATOUR SNP_SEQ_SESS SNP_SESSION ARC_SNP_SEQ_SESS I_SNP_SEQ_SESS (ODI 11G) SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_SESSION.SESS_NO = SNP_SEQ_SESS.SESS_NO Archiving ODI Logs, Create Interfaces 33
  32. 32. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_TASK_LS SNP_SEQ_SESS SNP_SESSION ARC_SNP_SESS_TASK_LS I_SNP_SESS_TASK_LS (ODI11G) SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_SEQ_SESS.SESS_NO = SNP_SESS_TASK_LS.SESS_NO AND SNP_SEQ_SESS.SEQ_NAME = SNP_SESS_TASK_LS.SEQ_NAME SNP_SESSION.SESS_NO = SNP_SEQ_SESS.SESS_NO Archiving ODI Logs, Create Interfaces 34
  33. 33. 23RD MAY 2017 / #OTNEMEATOUR SNP_SESS_STEP_LV SNP_STEP_LOG SNP_SESS_STEP SNP_SESSION ARC_SNP_SESS_STEP_LV I_SNP_SESS_STEP_LV (ODI11G) SNP_SESSION.CONTEXT_CODE = 'PRODUCTION' AND TRUNC(SNP_SESSION.SESS_BEG) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_STEP_LOG.NNO = SNP_SESS_STEP_LV.NNO AND SNP_STEP_LOG. SESS_NO = SNP_SESS_STEP_LV. SESS_NO AND SNP_STEP_LOG.NB_RUN = SNP_SESS_STEP_LV.NB_RUN SNP_STEP_LOG.SESS_NO = SNP_SESS_STEP.SESS_NO AND SNP_STEP_LOG.NNO = SNP_SESS_STEP.NNO SNP_SESS_STEP.SESS_NO = SNP_SESSION.SESS_NO Archiving ODI Logs, Create Interfaces 35
  34. 34. 23RD MAY 2017 / #OTNEMEATOUR SNP_EXP_TXT_HEADER SNP_EXP_TXT ARC_SNP_EXP_TXT_HEADER I_SNP_EXP_TXT_HEADER (ODI11G) TRUNC(SNP_EXP_TXT.FIRST_DATE) < TRUNC(SYSDATE) - #V_Purge_Log_Retention SNP_EXP_TXT.I_TXT = SNP_EXP_TXT_HEADER.I_TXT Archiving ODI Logs, Create Interfaces 36
  35. 35. 23RD MAY 2017 / #OTNEMEATOUR Create a delete procedure… Delete from child to parent ORDER STEP NAME COMMAND 0 DELETE ODIWD.SNP_PARAM_SESS DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_PARAM_SESS A WHERE A.SESS_NO IN (SELECT SESS.SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 10 DELETE ODIWD.SNP_SEQ_SESS DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SEQ_SESS A WHERE A.SESS_NO IN (SELECT SESS.SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 20 DELETE ODIWD.SNP_SESS_TASK_LS DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_TASK_LS A WHERE A.SESS_NO IN (SELECT SESS.SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 30 DELETE ODIWD.SNP_SESS_STEP_LV DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_STEP_LV A WHERE A.SESS_NO IN (SELECT SESS.SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 40 DELETE ODIWD.SNP_EXP_TXT_HEADER DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_EXP_TXT_HEADER A WHERE I_TXT IN (SELECT I_TXT FROM ODIWD.SNP_EXP_TXT WHERE TRUNC (A.FIRST_DATE) < TRUNC (SYSDATE) - #V_Purge_Log_Retention Archiving ODI Logs, Create Procedure 37
  36. 36. 23RD MAY 2017 / #OTNEMEATOUR Create a delete procedure… Delete from child to parent ORDER STEP NAME COMMAND 50 DELETE ODIWD.SNP_SESS_TXT_LOG DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_TXT_LOG A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 60 DELETE ODIWD.SNP_SESS_TASK_LOG DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_TASK_LOG A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 70 DELETE ODIWD.SNP_TASK_TXT DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_TASK_TXT A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 80 DELETE ODIWD.SNP_STEP_LOG DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_STEP_LOG A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) 90 DELETE ODIWD.SNP_SESS_TASK DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_TASK A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) – #V_Purge_Log_Retention) Archiving ODI Logs, Create Procedure 38
  37. 37. 23RD MAY 2017 / #OTNEMEATOUR ORDER STEP NAME COMMAND 100 DELETE ODIWD.SNP_SESS_STEP DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESS_STEP A WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION SESS WHERE TRUNC (SESS.SESS_BEG) < TRUNC (SYSDATE) - #V_Purge_Log_Retention) 110 DELETE ODIWD.SNP_VAR_DATA DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_VAR_DATA A WHERE TRUNC (A.FIRST_DATE) < TRUNC (SYSDATE) - #V_Purge_Log_Retention 120 DELETE ODIWD.SNP_VAR_SESS DELETE FROM ODIWD.SNP_VAR_SESS WHERE SESS_NO IN (SELECT SESS_NO FROM ODIWD.SNP_SESSION A WHERE TRUNC (SESS_BEG) < TRUNC (SYSDATE) - #V_Purge_Log_Retention 130 DELETE ODIWD.SNP_EXP_TXT DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_EXP_TXT A WHERE TRUNC (A.FIRST_DATE) < TRUNC (SYSDATE) - #V_Purge_Log_Retention 140 DELETE ODIWD.SNP_SESSION DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SESSION A WHERE TRUNC (SESS_BEG) < TRUNC (SYSDATE) - #V_Purge_Log_Retention 140 DELETE ODIWD.SNP_STEP_REPORT DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_STEP_REPORT A WHERE TRUNC (A.STEP_BEG) < TRUNC (SYSDATE) - #V_Purge_Log_Retention 150 DELETE ODIWD.SNP_SCEN_REPORT DELETE /*+ USE_HASH(A) PARALLEL(A) */ FROM ODIWD.SNP_SCEN_REPORT A WHERE TRUNC (SESS_BEG) < TRUNC (SYSDATE) - #V_Purge_Log_Retention Archiving ODI Logs, Create Procedure 39
  38. 38. 23RD MAY 2017 / #OTNEMEATOUR Running in «Asynchronous Mode» Running in «Asynchronous Mode» Archiving ODI Logs, Packaging 40
  39. 39. AGENDA
  40. 40. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, Model 42
  41. 41. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, Query ~ 1000 lines of code (insert into select) Word (10 Pages) A “long” TOAD script 43
  42. 42. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, ODI Interface ~ 100 lines of code (produces same insert into select) 44
  43. 43. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, How to Keep It Simple (Folders) Grouping models into folders Grouping objects into folders 45
  44. 44. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, How to Keep It Simple (Packaging) Don’t be afraid of long packages 46
  45. 45. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, How to Keep It Simple (Loops) Use loops instead of PL/SQL cursors Use variables in other variables (DB / OS independent) 47
  46. 46. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, How to Keep It Simple (Markers) Use markers to prepare for production Regenerate scenarios related to markers 48
  47. 47. 23RD MAY 2017 / #OTNEMEATOUR Complex Queries, How to Keep It Simple (Repository Selects) Use selects to repository, for information & exclamation. Assign outputs to variables and share with related peers. 49
  48. 48. AGENDA
  49. 49. 23RD MAY 2017 / #OTNEMEATOUR Oracle 2 Oracle Load (Control Append) SOURCE A LKM SQL to Oracle Staging TARGET + ODI Agent (C$%-src A) + (C$-srcB) (I$%TARGET) TARGET SOURCE B ODI Agent 51
  50. 50. 23RD MAY 2017 / #OTNEMEATOUR Oracle 2 Oracle Load (Control Append - DBLink) SOURCE B SOURCE A LKM Oracle to Oracle (DBLINK) TARGET + ODI Agent ODI Agent TARGET Create DBLink-B View-B Synonym-B Create DBLink-A Create View-A Create Synonym-A 52
  51. 51. 23RD MAY 2017 / #OTNEMEATOUR Oracle 2 Oracle Load (DataPump) SOURCE B SOURCE A LKM Oracle to Oracle (datapump) External Table TARGET + ODI Agent ODI Agent TARGET 53
  52. 52. 23RD MAY 2017 / #OTNEMEATOUR Oracle 2 Oracle Load (Incremental Update) SOURCE B SOURCE A IKM Oracle Incremental Update TARGET + ODI Agent TARGET Staging (C$%-src A) + (C$-srcB) (I$%TARGET) U P D A T E U P D A T E ODI Agent 54
  53. 53. AGENDA
  54. 54. 23RD MAY 2017 / #OTNEMEATOUR Direct DBLink KM… The Original SOURCE B SOURCE A LKM Oracle to Oracle (DBLINK) Create DBLink-B View-B Synonym-B Create DBLink-A Create View-A Create Synonym-A TARGET ODI Agent 56
  55. 55. 23RD MAY 2017 / #OTNEMEATOUR Direct DBLink KM… Reality Replacing space (“ ”) character in Control Append KM 57
  56. 56. 23RD MAY 2017 / #OTNEMEATOUR Oracle 2 Oracle Load (How to handle DBLink) INSERT /*+ APPEND PARALLEL(t3, 8) */ INTO t3 SELECT /*+ parallel(t1) parallel(t2) ordered use_hash(t2) index(t1 t1_abc) index(t2 t2_abc) */ t1.*, t2.* FROM t1@dblink1 t1_alias, t2@dblink2 t2_alias WHERE t1.col1 = t2.col1; 58
  57. 57. 23RD MAY 2017 / #OTNEMEATOUR Direct DBLink KM… Solution With a little bunch of code (can be used anywhere else) from SOURCE_SCHEMA.TABLE_A A SOURCE_SCHEMA.TABLE_B B from SOURCE_SCHEMA.TABLE_A@DBLINK_NAMEA A SOURCE_SCHEMA.TABLE_B@DBLINK_NAMEB B 59
  58. 58. AGENDA
  59. 59. 23RD MAY 2017 / #OTNEMEATOUR IKM Oracle Incremental Update KM SOURCE B SOURCE A IKM Oracle Incremental Update TARGET + ODI Agent ODI Agent TARGET Staging (C$%-src A) + (C$-srcB) (I$%TARGET) U P D A T E U P D A T E 61
  60. 60. 23RD MAY 2017 / #OTNEMEATOUR IKM Oracle Incremental Update KM - Restructured 1. Create target table 2. Drop flow table 3. Create flow table I$ 4. Delete target table 5. Truncate target table 6. Analyze target table 7. Insert flow into I$ table 8. Recycle previous errors 9. Create Index on flow table 10.Analyze integration table 11.Remove deleted rows from flow table 12.Flag rows for update 13.Update existing rows 14.Flag useless rows 15.Update existing rows 16.Insert new rows 17.Commit transaction 18.Analyze target table 19.Drop flow table 1. Drop flow table (I$) 2. Create flow table (I$) 3. Insert flow into I$ table 4. Flag rows for update 5. Create Unique Index on flow table (I$) 6. Update existing rows 7. Insert new rows 8. Commit transaction 9. Analyze target table 10.Drop flow table LKM Oracle Incremental Update LKM Oracle Incremental Update (Reorganized) 62
  61. 61. 23RD MAY 2017 / #OTNEMEATOUR IKM Oracle Updateless Incremental Update KM Usage of Incremental Update KM  Dimension tables  Fact tables with known primary (update) key  Dimension tables in order not to Truncate DATA WAREHOUSING OLTP Reporting  Fact tables in order not to Truncate What if you have 63
  62. 62. 23RD MAY 2017 / #OTNEMEATOUR IKM Oracle Updateless Incremental Update KM SOURCE B SOURCE A IKM Oracle Incremental Update (Updateless) TARGET + ODI Agent ODI Agent TARGET Staging (C$%-src A) + (C$-srcB) (I$%TARGET) I N S E R T • Missing Records (News) • Matching Records (Updates) • Unmatching Records (Deletes) 64
  63. 63. 23RD MAY 2017 / #OTNEMEATOUR IKM Oracle Updateless Incremental Update KM How it works? 1. Drop and Create I$ flow table 2. Insert missing records to I$ table (option INSERT_NEW_ROWS = True 3. Insert matching records to I$ table (option UPDATE_EXISTING_ROWS = True) 4. Insert non-matching records to I$ table (option NOT_DELETE_EXISTING_ROWS = True) 5. Truncate target table 6. Insert I$ into target table 7. Drop I$ (option DELETE_TEMPORARY_OBJECTS = True) 8. Analyse Target Table (option ANALYSE_TARGET_TABLE = True) 65
  64. 64. AGENDA
  65. 65. 23RD MAY 2017 / #OTNEMEATOUR Keep It Simple 67
  66. 66. 23RD MAY 2017 / #OTNEMEATOUR http://gurcanorhan.wordpress.com @gurcan_orhan http://tr.linkedin.com/in/gurcanorhan …Final Words… … THE END …

×