Best Practices with Oracle Data Integrator (ODI)-BGOUG 20th November 2015

11,090 views
12,121 views

Published on

Presentation of webinar that I have delivered for LAOUC (Latin America Oracle User Community) at 27th of February 2014.

Published in: Technology
1 Comment
16 Likes
Statistics
Notes
No Downloads
Views
Total views
11,090
On SlideShare
0
From Embeds
0
Number of Embeds
3,288
Actions
Shares
0
Downloads
0
Comments
1
Likes
16
Embeds 0
No embeds

No notes for slide

Best Practices with Oracle Data Integrator (ODI)-BGOUG 20th November 2015

  1. 1. Gürcan ORHAN Senior Manager Information Management, Technology Deloitte – Turkey 20th November 2015 1 Best Practices with Oracle Data Integrator (ODI): Flexibility
  2. 2. 20 NOVEMBER 2015 / #BGOUG15 Who Am I ? +20 years of IT experience. +12 years of DWH experience. +9 years of Oracle Data Integrator experience. +7 years of Oracle Warehouse Builder experience. Sybase Power Designer, ERwin Data Modeler, SQL Developer OBIEE, Cognos, Microstrategy, Business Objects, Tableau. Oracle Excellence Awards - Technologist of the Year 2011 : Enterprise Architect DWH & BI Chair : TROUG (Turkish Oracle User Group) Published Customer Snapshot for NODI @Oracle.com Published videos about ODI @Oracle.com (Oracle Media Network) Published OTN Podcasts about “Data Warehousing and ODI” “ODI and the Evolution of Data Integration” 3 different “2MTT”s Articles in OTech Magazine, SearchSoftwareQuality.com Annual panelist for ODTUG “Ask the Experts Panel : ODI” Presenter in OOW since 2010 (6 times in a row ⭐ ) Presenter in many OUG conferences in global Presenter in various universities in Turkey First time in Bulgaria. 😊 2
  3. 3. 20 NOVEMBER 2015 / #BGOUG15 Deloitte Turkey As a client and industry oriented professional service provider, we have been delivering high – quality and distinctive services for 29 years with more than 1.350 experienced professionals through 5 different offices in Turkey. İstanbul Office Ankara Office İzmir Office Bursa Office Çukurova Office © 2015. For information, contact Deloitte Turkey, Member of Deloitte Touche Tohmatsu Limited. 3
  4. 4. 20 NOVEMBER 2015 / #BGOUG15 Deloitte Turkey by Numbers 4 © 2015. For information, contact Deloitte Turkey, Member of Deloitte Touche Tohmatsu Limited. * Deloitte ranks number 1 among Big 4 on the ‘‘Leading Companies in Turkey 2015’’ list published by Realta Consulting and Bloomberg Businessweek Turkey Over 1.350 professionals 54% of our management team has an experience for at least a year at various Deloitte offices abroad We serve 79% of Global Fortune 500 companies The largest independent audit firm in Turkey We audit 76 of the listed companies in BIST (Istanbul Stock Exchange) Training provided to 2.500 companies and over 10.500 professionals (Deloitte Business Institute) The largest and most experienced independent audit team comprised of 267 Turkish CPAs, 95 licensed by the capital markets board Deloitte Turkey tops ‘‘Leading Companies in Turkey 2015 ’’ list among Big 4* Scholarships to more than 100 students (Deloitte Educational Foundation) Placing Top 3 in Corporate Football League and Corporate Basketball League each year
  5. 5. 20 NOVEMBER 2015 / #BGOUG15 My Definition of ODI 5
  6. 6. 20 NOVEMBER 2015 / #BGOUG15 Make sure you have;  Backup your repository  Backup your Knowledge Modules (export, duplicate)  Backup your necessary ODI development Before trying something in your environment Remember to create a zzz_Test folder and test before apply Attention …!!! 6
  7. 7. AGENDA
  8. 8. 20 NOVEMBER 2015 / #BGOUG15 select count(1) from msdb.dbo.sysjobhistory where step_id = 0 and run_status = 1 and job_id = (select job_id from msdb.dbo.sysjobs where name = 'My_BI_Job') and run_date = CAST(SUBSTRING('#V_SYSDATE',1,8) as integer) Use A Variable Within A Variable 8
  9. 9. 20 NOVEMBER 2015 / #BGOUG15 SELECT * FROM odiwd.snp_session sess, odiwd.snp_step_log step, odiwd.snp_sess_task sess_task, odiwd.snp_exp_txt exp_txt WHERE sess.sess_no = step.sess_no AND sess.sess_no = sess_task.sess_no AND step.nno = sess_task.nno AND step.i_txt_step_mess = exp_txt.i_txt AND step.step_status = 'E' AND exp_txt.txt_ord = 0 AND sess.sess_no = <%=odiRef.getSession("SESS_NO")%> SELECT NVL(MAX(ALARM_ID), 0) FROM <%=odiRef.getSchemaName("MYDB.DWH", "D")%>.TABLE_NAME SELECT NVL(MAX(ALARM_ID), 0) FROM DWH.TABLE_NAME Use A odiRef Function In A Variable 9
  10. 10. AGENDA
  11. 11. 20 NOVEMBER 2015 / #BGOUG15 Oracle’s most powerful querying attribute when in right hands. An Oracle hint is an optimizer directive that is embedded into an SQL statement to suggest to Oracle how the statement should be executed. http://psoug.org/reference/hints.html or just google «Oracle hints» Most common hints in a DWH system;  APPEND  PARALLEL  USE_HASH  USE_MERGE  FULL  INDEX  ORDERED  MERGE HINTS in Oracle 11
  12. 12. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Static Hints 12
  13. 13. 20 NOVEMBER 2015 / #BGOUG15 Step 1 : Create OPTIONS for KM’s KM’s… How To Apply semi-Dynamic Hints 13
  14. 14. 20 NOVEMBER 2015 / #BGOUG15 Step 1 : Create OPTIONS for KM’s KM’s… How To Apply semi-Dynamic Hints 14
  15. 15. 20 NOVEMBER 2015 / #BGOUG15 Step 2 : Insert this OPTIONS into KM’s insert <%=odiRef.getOption("INSERT_HINT")%> into <%=snpRef.getTable("L","TARG_NAME","A")%> select <%=odiRef.getOption("SELECT_HINT")%> <%=snpRef.getPop("DISTINCT_ROWS")%> <%=snpRef.getColList("", "[EXPRESSION]t[ALIAS_SEP] [CX_COL_NAME]", ",nt", "", "")%> from <%=snpRef.getFrom()%> INSERT /*+ APPEND PARALLEL(t3, 8) */ INTO t3 SELECT /*+ parallel(t1) parallel(t2) ordered use_hash(t2) index(t1 t1_abc) index(t2 t2_abc) */ COUNT(*) FROM t1, t2 WHERE t1.col1 = t2.col1; KM’s… How To Apply semi-Dynamic Hints 15
  16. 16. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 16 Step 1 : Create table INTEFACE_HINTS to read from database CREATE TABLE ODI.INTERFACE_HINTS ( INTERFACE_NAME VARCHAR2(255 BYTE), INTERFACE_KM VARCHAR2(255 BYTE), KM_STEP_NAME VARCHAR2(255 BYTE), KM_STEP_ORDER INTEGER, STEP_HINT_NAME VARCHAR2(255 BYTE), STEP_HINT_ORDER INTEGER, STEP_HINT_VALUE VARCHAR2(4000 BYTE) ) TABLESPACE TBS_ODI NOLOGGING NOCOMPRESS NOCACHE NOPARALLEL MONITORING;
  17. 17. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 17 Step 2 : Add 1st command (Get Hints) in Knowledge Module
  18. 18. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 18 Step 2 : Add 1st command (Get Hints) in Knowledge Module
  19. 19. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 19 Step 3 : Use Hints in SQL statements in Knowledge Module
  20. 20. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 20 Output of step “Get Hints” in ODI
  21. 21. 20 NOVEMBER 2015 / #BGOUG15 KM’s… How To Apply Variabled Dynamic Hints 21 Output of “SQL Statements” in ODI
  22. 22. AGENDA
  23. 23. 20 NOVEMBER 2015 / #BGOUG15 Right Click ➜ Insert Datatype Adding Datatypes 23
  24. 24. 20 NOVEMBER 2015 / #BGOUG15 Adding Datatypes 24 Converted To, Converted From
  25. 25. AGENDA
  26. 26. 20 NOVEMBER 2015 / #BGOUG15 Topology Manager ➜ Languages ➜ SQL ➜ Aggregate (or other) Adding Database Related Functions 26
  27. 27. 20 NOVEMBER 2015 / #BGOUG15 Adding User Functions 27 User Functions ➜ Right Click ➜ New User Function AVG CORR COUNT COVAR_POP COVAR_SAMP CUME_DIST DENSE_RANK FIRST LAST LISTAGG, MAX MIN PERCENT_RANK PERCENTILE_CONT PERCENTILE_DISC RANK STDDEV STDDEV_POP STDDEV_SAMP SUM VAR_POP VAR_SAMP VARIANCE
  28. 28. 20 NOVEMBER 2015 / #BGOUG15 Adding User Functions 28 User Functions ➜ Impact Analysis
  29. 29. AGENDA
  30. 30. 20 NOVEMBER 2015 / #BGOUG15 Running in «Asynchronous Mode» Error Handler Max. Number of Failed Child Sessions = 1 Mail body (refresh variable) Send mail Raise Error (error refresh variable) Handling Alerts 30
  31. 31. 20 NOVEMBER 2015 / #BGOUG15 Send e-mail for each script, attach the appropriate file.  Mail Server : #V_MAIL_SERVER_IP  From : This is static, user that you are sending mail.  TO : #V_ERROR_MAIL_TO (need to be refreshed in the beginning of your ETL or current package)  CC : #V_ERROR_MAIL_CC (need to be refreshed in the beginning of your ETL or current package)  BCC : #V_ERROR_MAIL_BCC (need to be refreshed in the beginning of your ETL or current package)  Subject : There are #V_MY_ERROR_COUNT errors exist in your system (Error Code = #V_MY_ERROR_CODE)  Attachment : /data/my_errors/MY_Error_#V_MY_ERROR_CODE.txt (will represent as /data/my_errors/MY_Error_1.txt initially, then will attach regarding file in the loop, every step will attach its own script file)  Message Body : #V_MY_ERROR_DESC Handling - in ETL - Data Quality 31
  32. 32. AGENDA
  33. 33. 20 NOVEMBER 2015 / #BGOUG15 Step 1 : Operating system folders and read file names Step 2 : IKM Knowledge Module (edited to fit for purpose) Step 4 : ODI Procedure to rename files for external table usage Step 3 : ETL_FILE LOG (database table) Step 6 : ODI Procedure to Get File List of operating system Step 7 : a single ODI Interface (loading and transforming) Step 8 : Some ODI Variables Step 9 : ODI Package for running everything in right order Step 5 : ODI Procedures to finish working with files File2Table… Summary (F2T) 33
  34. 34. 20 NOVEMBER 2015 / #BGOUG15 Illustration for files received from «web_logs» F2T… Prepare Folders (Step 1) 34
  35. 35. 20 NOVEMBER 2015 / #BGOUG15  Rename as : IKM SQL Control Append (Direct Load,HINT)  Copy current KM : IKM SQL Control Append  Add New Row : «Parallel DML», Transaction 0, No Commit alter session enable parallel dml  Add Options : «SELECT HINT», «INSERT HINT»  Add New fixed Row : «Commit transaction» to Transaction 0, Commit /* commit */  Modify : «Insert new rows» to Transaction 0, No Commit File2Table… Prepare IKM (Step 2) 35
  36. 36. 20 NOVEMBER 2015 / #BGOUG15 INSERT /*+ APPEND PARALLEL(4) */ INTO ODIDB.MY_TARGET_TABLE ( MY_TARGET_COLUMN_1, MY_TARGET_COLUMN_2, MY_TARGET_COLUMN_3 ) SELECT /*+ PARALLEL(MY_SOURCE_TABLE,4) FULL(MY_SOURCE_TABLE) */ MY_SOURCE_TABLE.MY_SOURCE_COLUMN_1, MY_SOURCE_TABLE.MY_SOURCE_COLUMN_2, MY_SOURCE_TABLE.MY_SOURCE_COLUMN_3 FROM ODISTG.I$MY_TARGET_TABLE MY_SOURCE_TABLE WHERE MY_SOURCE_TABLE.MY_SOURCE_COLUMN_4 = ‘USA’ AND MY_SOURCE_TABLE.MY_SOURCE_COLUMN_5 = ‘New York’ F2T… Prepare IKM (Step 2) 36
  37. 37. 20 NOVEMBER 2015 / #BGOUG15 COLUMN NAME PK NULL? DATA TYPE DEF COMMENT FILE_ID 1 N NUMBER (10) The unique identification number of file. FILE_NAME N VARCHAR2 (50 Byte) The name of file to be processed. FILE_GROUP N VARCHAR2 (20 Byte) Source system name or group with the same template. FILE_COPIED_FLAG Y NUMBER (1) 0 0 : not copied, 1 : copied successfully, 2 : error in copy. FILE_COPY_DATE Y DATE Date of file copied. FILE_READ_FLAG Y NUMBER (1) 0 0:not read, 1:read successfully, 2:error in read. FILE_READ_DATE Y DATE Date of file read. FILE_PROCESSED_FLAG Y NUMBER (1) 0 Date of file processed. FILE_PROCESSED_DATE Y DATE 0:not processed, 1:processed successfully, 2:error in process. Create table ETL_FILE_LOG F2T… Prepare log Table (Step 3) 37
  38. 38. 20 NOVEMBER 2015 / #BGOUG15  Create external table «STG.INVOICE_LOG»  Create ODI Procedure «Rename File» mv /data/invoice_logs/#FILE_NAME/data/invoice_logs /my_external_table.ext rm /data/invoice_logs/my_external_table.ext  Step 1 ; delete previous file  Step 2 ; rename next file F2T… Rename File ODI Procedure (Step 4) 38
  39. 39. 20 NOVEMBER 2015 / #BGOUG15  Create ODI Procedure – «UPDATE PROCESSED FILE» UPDATE ODI.ETL_FILE_LOG SET FILE_PROCESSED_FLAG = 1, FILE_PROCESSED_DATE = SYSDATE WHERE FILE_ID = #FILE_ID  Step 1 ; update ETL_FILE_LOG (Processed)  Step 2 ; delete processed data file rm /data/invoice_logs/my_external_table.ext  Step 3 ; delete processed log file rm /data/invoice_logs/my_external_table.log F2T… Update Processed File (Step 5) 39
  40. 40. 20 NOVEMBER 2015 / #BGOUG15 import java.lang as lang import java.sql as sql import snpsftp import java.lang.String import os import java.io.File as File #db connection driverSrc = ‘oracle.jdbc.driver.OracleDriver’ lang.Class.forName(driverSrc) #Production Environment urlSrc = ‘jdbc:oracle:thin:@<host>:<port>:<sid>’ #Development Environment #urlSrc = ‘jdbc:oracle:thin:@<host>:<port>:<sid>’ userSrc = ‘ODI’ passwdSrc = ‘ODI’ ConSrc = sql.DriverManager.getConnection(urlSrc, userSrc, passwdSrc); readDBLink = ConSrc.createStatement() syslist = os.listdir(‘<%=odiRef.getOption( “SOURCE_DIRECTORY” )%>’) for sys in syslist: str = java.lang.String(sys) if str.length() > 8: sqlDBLink = “select * from ODI.ETL_FILE_LOG where FILE_NAME = ‘” + sys + “‘” rqteDBLink = readDBLink.executeQuery(sqlDBLink) if not rqteDBLink.next(): sqlDBLink = “insert into ODI.ETL_FILE_LOG (FILE_ID, FILE_NAME, FILE_GROUP, FILE_SUB_GROUP, FILE_READ_FLAG, FILE_READ_DATE) values (ODI.SEQ_FILE_ID.NEXTVAL, ‘” + sys + “‘, ‘<source_system_name>’, ‘<file_type>’, ’1′, SYSDATE)” rqteDBLink = readDBLink.execute(sqlDBLink) ConSrc.close()  Create ODI Procedure Jyhton Technology – «GetFileList» F2T… GetFileList from OS (Step 6) 40
  41. 41. 20 NOVEMBER 2015 / #BGOUG15  Create ODI Interface (external table  db table)  Source : STG.INVOICE_LOG (based on external table  my_external_table.ext)  Target : DWH.INVOICE_LOGS  KM : IKM SQL Control Append (Direct Load, HINT) Truncate : No Select Hint : /*+ PARALLEL(4) */ Insert Hint : /*+ APPEND PARALLEL(4) NOLOGGING */ F2T… ODI Interface (Step 7) 41
  42. 42. 20 NOVEMBER 2015 / #BGOUG15  Create ODI Variable to refresh – «File_ID»  Create ODI Variable to refresh – «File_Name» SELECT NVL(MIN(FILE_ID), 0) FROM ODI.ETL_FILE_LOG WHERE FILE_READ_FLAG = 1 AND FILE_PROCESSED_FLAG = 0 AND FILE_GROUP = 'INVOICE_LOGS' SELECT FILE_NAME FROM ODI.ETL_FILE_LOG WHERE FILE_ID = #FILE_ID F2T… Prepare ODI Variables (Step 8) 42
  43. 43. 20 NOVEMBER 2015 / #BGOUG15 Get_File_List ODI Procedure FILE_ID Refresh Variable FILE_ID Evaluate Variable FILE_NAME Refresh Variable RENAME_FILE ODI Procedure INTERFACE From:Ext_Table To:DB.Table UPDATE_PROCESSED ODI Procedure UPDATE_REJECTED ODI Procedure Determine_Error_Desc ODI Procedure ODISendMail Rejected File F2T… Time to Pack-UP (Step 9) 43
  44. 44. 20 NOVEMBER 2015 / #BGOUG15 http://gurcanorhan.wordpress.com @gurcan_orhan http://tr.linkedin.com/in/gurcanorhan …Final Words… 44
  45. 45. Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its network of member firms, and their related entities. DTTL and each of its member firms are legally separate and independent entities. DTTL (also referred to as “Deloitte Global”) does not provide services to clients. Please see www.deloitte.com/about for a more detailed description of DTTL and its member firms. Deloitte provides audit, consulting, financial advisory, risk management, tax and related services to public and private clients spanning multiple industries. With a globally connected network of member firms in more than 150 countries and territories, Deloitte brings world-class capabilities and high-quality service to clients, delivering the insights they need to address their most complex business challenges. Deloitte’s more than 210,000 professionals are committed to becoming the standard of excellence. This communication contains general information only, and none of Deloitte Touche Tohmatsu Limited, its member firms, or their related entities (collectively, the “Deloitte network”) is, by means of this communication, rendering professional advice or services. No entity in the Deloitte network shall be responsible for any loss whatsoever sustained by any person who relies on this communication © 2015. For more information, contact Deloitte Touche Tohmatsu Limited.

×