Oracle Data Integrator (ODI) seems to be slow when it is installed out-of-the-box, since it has to comply with different versions of the databases and operating systems. The default installation is generally not the optimal choice. ODI is a flexible product, that can be customized for specific requirements and to implement new features of the database or operating systems. Attendees will learn how to easily create a customized ODI environment.
This presentation will demonstrate the flexibility of the Knowledge Module, configuration best practices and the best query response time tips and techniques depending on complex business requirements. It will include information about how to load an extensive number of files quickly with a special algorithm, as well as how to define new customized data types, analytical and database functions, archiving ODI logs in a timely fashion and using Oracle HINTS in a variabled and static way due to business and IT needs.
Hitchhiker's Guide to free Oracle tuning toolsBjoern Rost
Instance and SQL tuning with EM12c Cloud Control is so easy, it is not even much fun
anymore. Also, not every customer may have the appropriate license or database
edition, or all you have available remotely is a command-line login to a database.
This presentation showcases a few open-source database tuning tools such as Snapper
and ASH replacements that DBAs can use to gather and review metrics and wait events
from the command line and even in standard edition.
Hitchhiker's Guide to free Oracle tuning toolsBjoern Rost
Instance and SQL tuning with EM12c Cloud Control is so easy, it is not even much fun
anymore. Also, not every customer may have the appropriate license or database
edition, or all you have available remotely is a command-line login to a database.
This presentation showcases a few open-source database tuning tools such as Snapper
and ASH replacements that DBAs can use to gather and review metrics and wait events
from the command line and even in standard edition.
"Extended" or "Stretched" Oracle RAC has been available as a concept for a while. Oracle RAC 12c Release 2 introduces an Oracle Extended Cluster configuration, in which the cluster understands the concept of sites and extended setups. This knowledge is used to more efficiently manage "Extended Oracle RAC", whether the nodes are 0.1 mile or 10 miles apart.
The presentation was last updated on August 7th 2017 to add a reference to the new MAA White Paper: "Installing Oracle Extended Clusters on Exadata Database Machine" - http://www.oracle.com/technetwork/database/availability/maa-extclusters-installguide-3748227.pdf and to correct some minor details.
This is a recording of my Advanced Oracle Troubleshooting seminar preparation session - where I showed how I set up my command line environment and some of the main performance scripts I use!
Oracle Data Integrator 12c - Getting StartedMichael Rainey
I think it’s time for a fresh look at Oracle Data Integrator 12c. What is ODI? How has it evolved over the years and where is it going? And, of course, how do you get started with Oracle Data Integrator? I plan to share what I love about ODI, how to get started building your first ODI project, and what makes Oracle Data Integrator 12c the premier ETL and data warehousing tool on the market. It’s time to get back to the basics!
Presented at UTOUG Training Days 2017.
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsJohn Beresniewicz
RMOUG 2020 abstract:
This session will cover core concepts for Oracle performance analysis first introduced in Oracle 10g and forming the backbone of many features in the Diagnostic and Tuning packs. The presentation will cover the theoretical basis and meaning of these concepts, as well as illustrate how they are fundamental to many user-facing features in both the database itself and Enterprise Manager.
Event: Passcamp, 07.12.2017
Speaker: Stefan Kirner
Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/
Mehr Tech-Artikel: https://www.inovex.de/blog
Session aims at introducing less familiar audience to the Oracle database statistics concept, why statistics are necessary and how the Oracle Cost-Based Optimizer uses them
Analysis of Database Issues using AHF and Machine Learning v2 - SOUGSandesh Rao
Oracle Autonomous Health Framework (AHF) is Oracle’s Artificial Intelligence Operations platform for autonomous database health management. This session will focus on enhancements to current functionality and new features in 21c. We will discuss how to use the data which is derived from the Bayesian Net framework of AHF to conduct root cause analysis, telemetry and remediations for issues. You will learn to utilize these features to determine workload footprint, ongoing monitoring, early detection of anomalies and performance issues, their root causes and corrective actions, prevention of node or database failures, and targeted postmortem analysis enabling quick resolution.
Session Highlights:
• Insights into AHF enhancements to current functionality and new features in 21c
• Learn early detection of anomalies and performance issues, their root causes and corrective actions
• Targeted postmortem analysis enabling quick resolution
How to solve complex business requirements with Oracle Data Integrator?Gurcan Orhan
Business requirements are always hard to implement, develop, operate and always changeable. In this session attendees will have some fact examples of turning unstructured data into structural meaning, writing complex queries without typing anything, adding function based joins, implementing CTAS (Create Table As Select) and IAS (Insert As Select) methods and simplifying business rules, writing optimized queries to decrease operational and development costs as well as the faster loads.
In this presentation, see how you can solve some of complex business requirements with Oracle Data Integrator's flexibility and ease of usage features.
"Extended" or "Stretched" Oracle RAC has been available as a concept for a while. Oracle RAC 12c Release 2 introduces an Oracle Extended Cluster configuration, in which the cluster understands the concept of sites and extended setups. This knowledge is used to more efficiently manage "Extended Oracle RAC", whether the nodes are 0.1 mile or 10 miles apart.
The presentation was last updated on August 7th 2017 to add a reference to the new MAA White Paper: "Installing Oracle Extended Clusters on Exadata Database Machine" - http://www.oracle.com/technetwork/database/availability/maa-extclusters-installguide-3748227.pdf and to correct some minor details.
This is a recording of my Advanced Oracle Troubleshooting seminar preparation session - where I showed how I set up my command line environment and some of the main performance scripts I use!
Oracle Data Integrator 12c - Getting StartedMichael Rainey
I think it’s time for a fresh look at Oracle Data Integrator 12c. What is ODI? How has it evolved over the years and where is it going? And, of course, how do you get started with Oracle Data Integrator? I plan to share what I love about ODI, how to get started building your first ODI project, and what makes Oracle Data Integrator 12c the premier ETL and data warehousing tool on the market. It’s time to get back to the basics!
Presented at UTOUG Training Days 2017.
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsJohn Beresniewicz
RMOUG 2020 abstract:
This session will cover core concepts for Oracle performance analysis first introduced in Oracle 10g and forming the backbone of many features in the Diagnostic and Tuning packs. The presentation will cover the theoretical basis and meaning of these concepts, as well as illustrate how they are fundamental to many user-facing features in both the database itself and Enterprise Manager.
Event: Passcamp, 07.12.2017
Speaker: Stefan Kirner
Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/
Mehr Tech-Artikel: https://www.inovex.de/blog
Session aims at introducing less familiar audience to the Oracle database statistics concept, why statistics are necessary and how the Oracle Cost-Based Optimizer uses them
Analysis of Database Issues using AHF and Machine Learning v2 - SOUGSandesh Rao
Oracle Autonomous Health Framework (AHF) is Oracle’s Artificial Intelligence Operations platform for autonomous database health management. This session will focus on enhancements to current functionality and new features in 21c. We will discuss how to use the data which is derived from the Bayesian Net framework of AHF to conduct root cause analysis, telemetry and remediations for issues. You will learn to utilize these features to determine workload footprint, ongoing monitoring, early detection of anomalies and performance issues, their root causes and corrective actions, prevention of node or database failures, and targeted postmortem analysis enabling quick resolution.
Session Highlights:
• Insights into AHF enhancements to current functionality and new features in 21c
• Learn early detection of anomalies and performance issues, their root causes and corrective actions
• Targeted postmortem analysis enabling quick resolution
How to solve complex business requirements with Oracle Data Integrator?Gurcan Orhan
Business requirements are always hard to implement, develop, operate and always changeable. In this session attendees will have some fact examples of turning unstructured data into structural meaning, writing complex queries without typing anything, adding function based joins, implementing CTAS (Create Table As Select) and IAS (Insert As Select) methods and simplifying business rules, writing optimized queries to decrease operational and development costs as well as the faster loads.
In this presentation, see how you can solve some of complex business requirements with Oracle Data Integrator's flexibility and ease of usage features.
How to Handle DEV&TEST&PROD for Oracle Data IntegratorGurcan Orhan
Most of us have development teams apart from test and operation teams using the different repository environments. And there are generally 3 different ODI installations and repositories which each of the teams use separately. Chaos is usually expected and happened who will test which development and what to deploy into production.
In this session hear how ODI can handle your development hierarchy with ease of usage and in simplified/synchronized way for successful deployments.
A simple project will be built up and will be enlarged to enterprise level step by step.
Presentation by Mark Rittman, Technical Director, Rittman Mead, on ODI 11g features that support enterprise deployment and usage. Delivered at BIWA Summit 2013, January 2013.
It is the fact that Oracle Warehouse Builder (OWB) released the latest major version and final state. But business requirements are rapidly increasing. New applications are implemented in source systems and as a result new reports and new subject areas are needed urgently. It is needed to implement new features for growing business needs into our data warehouses. Resources are limited and conversion should be done as soon as possible.
In this presentation, see the most convenient methods to migrate from Oracle Warehouse Builder to Oracle Data Integrator with agile methodology without interrupting on going daily jobs as well as understanding of Oracle's OWB2ODI migration utility.
This presenation explains basics of ETL (Extract-Transform-Load) concept in relation to such data solutions as data warehousing, data migration, or data integration. CloverETL is presented closely as an example of enterprise ETL tool. It also covers typical phases of data integration projects.
Services are one of the most underutilized features of the Oracle Database. This presentation shows some use cases that may make you change your mind and motivate to implement services in one way or another.
ODI 11g - Multiple Flat Files to Oracle DB Table by taking File Name dynamica...Darshankumar Prajapati
This is a brief low level technical steps for Loading Multiple flat files data in to Oracle Table with ODI via Interface. Also Files are moved to Archive Destination.
Dear All,
Hope all are doing well!
Here we are posting same model which we have posted earlier in 11g, but now we have implemented same in ODI 12C(12.2.1.0.0) with slight changes.
Please review it and Keep ODIING !!!
Thanks,
A small big data overview i created in 3 hours on my iPad using the drawing app called Paper.
Disclaimer: The last slide is Oracle's property and I own no part of it.
Change tracking is a lightweight solution that provides an efficient change tracking mechanism for applications. Typically, to enable applications to query for changes to data in a database and access information that is related to the changes, application developers had to implement custom change tracking mechanisms. Creating these mechanisms usually involved a lot of work and frequently involved using a combination of triggers, timestamp columns, new tables to store tracking information, and custom cleanup processes.
The optimizer is the brain of any DBMS system, This presentation helps to understand MongoDB explain plan metrics and How MongoDB Optimiser scores for each query plan and choosing the proper query plan for the execution.
BigQuery JavaScript User-Defined Functions by THOMAS PARK and FELIPE HOFFA at...Big Data Spain
Performing ETL on big data can be slow, expensive and painful - but it doesn't have to be! In this session, we'll take an in-depth look at several real-world examples of computations that don't fit well with the SQL language model and how to solve them with user-defined functions in Google BigQuery.
Session presented at Big Data Spain 2014 Conference
18th Nov 2014
Kinépolis Madrid
http://www.bigdataspain.org
Event promoted by: http://www.paradigmatecnologico.com
Abstract: http://www.bigdataspain.org/2014/conference/hands-on-with-bigquery-javascript-user-defined-functions
MySQL 8 -- A new beginning : Sunshine PHP/PHP UK (updated)Dave Stokes
MySQL 8 has many new features and this presentation covers the new data dictionary, improved JSON functions, roles, histograms, and much more. Updated after SunshinePHP 2018 after feedback
SQL Saturday 108 -- Enterprise Data Mining with SQL ServerMark Tabladillo
Presented at SQL Saturday 108 Redmond, WA -- This presentation introduces SQL Server Data Mining (SSDM) for SQL Server Professionals based on the speaker's past presentation for Microsoft TechEd. Starting with SQL Server Management Studio (SSMS), the demo includes the interfaces important for professional development, including Business Intelligence Development Studio (BIDS), highlighting Integration Services, and PowerShell. The interactive demos are based on Microsoft's Contoso Retail sample data. Finally we will evaluate where Microsoft data mining can help you in a practical business environment, which may include Oracle and SAS.
SQL Saturday 109 -- Enterprise Data Mining with SQL ServerMark Tabladillo
Delivered at SQL Saturday 109 Silicon Valley, CA -- This presentation introduces SQL Server Data Mining (SSDM) for SQL Server Professionals based on the speaker's past presentation for Microsoft TechEd. Starting with SQL Server Management Studio (SSMS), the demo includes the interfaces important for professional development, including Business Intelligence Development Studio (BIDS), highlighting Integration Services, and PowerShell. The interactive demos are based on Microsoft's Contoso Retail sample data. Finally we will evaluate where Microsoft data mining can help you in a practical business environment, which may include Oracle and SAS.
Similar to Best Practices with ODI : Flexibility (20)
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf
Best Practices with ODI : Flexibility
1. Gürcan ORHAN
Enterprise Data Warehouse Architect
Ekol Logistics Corp.
30th May 2017
Best Practices with
Oracle Data Integrator (ODI): Flexibility
http://gurcanorhan.wordpress.com
@gurcan_orhan
http://tr.linkedin.com/in/gurcanorhan
2. Who Am I ?
+20 years of IT experience.
+14 years of DWH experience.
+10 years of Oracle Data Integrator experience.
+8 years of Oracle Warehouse Builder experience.
Sybase Power Designer, ERwin Data Modeler, SDDM
OBIEE, Cognos, Microstrategy, Business Objects, Qlikview, Tableau
IBM Data Stage, SAP Data Services, Informatica, etc…
Oracle Excellence Awards - Technologist of the Year 2011 :
Enterprise Architect
DWH & BI Chair : TROUG (Turkish Oracle User Group)
Published Customer Snapshot for NODI @Oracle.com
Published videos about ODI @Oracle.com (Oracle Media Network)
Published OTN Podcasts about
“Data Warehousing and ODI”
“ODI and the Evolution of Data Integration”
3 different “2MTT”s
Articles in OTech Magazine, SearchSoftwareQuality.com
Annual panelist for ODTUG “Ask the Experts Panel : ODI”
Presenter in OOW since 2010 (7 times in a row ⭐ )
Presenter in many OUG conferences in globe
Presenter in various universities in Turkey
23RD MAY 2017 / #OTNEMEATOUR
3. Ekol Germany
Warehousing
Solutions
begin with the
Kardelen Facility
1996 2003 2010 2012 2014 2016
201520132011200820021990
Acquire STS Int.
Transport
Ekol Bosnia
Ekol France
Ekol Greece
Ekol Ukraine
Ekol Spain
Ekol Bulgaria
Ekol Czech Rep.
Ekol Iran
Ekol PolandEkol Italy
Ekol Romania
Ekol HungaryAcquire
Unok/Unatsan
Rainbow
Replaced by
Quadro
(software)
Intermodal
operations Ro-Ro
operations
Established
Ekol Milestones
6. 30 MAY 2017 / #OTNEMEATOUR
My Definition of ODI
6
7. 30 MAY 2017 / #OTNEMEATOUR
Make sure you have;
Backup your repository
Backup your Knowledge Modules (export, duplicate)
Backup your necessary ODI development
Before trying something in your environment
Remember to create a zzz_Test folder and test before apply
Attention …!!!
7
9. 30 MAY 2017 / #OTNEMEATOUR
select count(1) from msdb.dbo.sysjobhistory
where step_id = 0
and run_status = 1
and job_id = (select job_id from msdb.dbo.sysjobs
where name = 'My_BI_Job')
and run_date = CAST(SUBSTRING('#V_SYSDATE',1,8) as integer)
Use A Variable Within A Variable
9
10. 30 MAY 2017 / #OTNEMEATOUR
SELECT *
FROM odiwd.snp_session sess,
odiwd.snp_step_log step,
odiwd.snp_sess_task sess_task,
odiwd.snp_exp_txt exp_txt
WHERE sess.sess_no = step.sess_no
AND sess.sess_no = sess_task.sess_no
AND step.nno = sess_task.nno
AND step.i_txt_step_mess = exp_txt.i_txt
AND step.step_status = 'E'
AND exp_txt.txt_ord = 0
AND sess.sess_no = <%=odiRef.getSession("SESS_NO")%>
SELECT NVL(MAX(ALARM_ID), 0) FROM
<%=odiRef.getSchemaName("MYDB.DWH", "D")%>.TABLE_NAME
SELECT NVL(MAX(ALARM_ID), 0) FROM DWH.TABLE_NAME
Use A odiRef Function In A Variable
10
12. 30 MAY 2017 / #OTNEMEATOUR
Oracle’s most powerful querying attribute when in right hands.
An Oracle hint is an optimizer directive that is embedded into
an SQL statement to suggest to Oracle how the statement
should be executed.
http://psoug.org/reference/hints.html or just google «Oracle hints»
Most common hints in a DWH system;
APPEND
PARALLEL
USE_HASH
USE_MERGE
FULL
INDEX
ORDERED
MERGE
HINTS in Oracle
12
13. 30 MAY 2017 / #OTNEMEATOUR
KM’s… How To Apply Static Hints
13
14. 30 MAY 2017 / #OTNEMEATOUR
Step 1 : Create OPTIONS for KM’s
KM’s… How To Apply semi-Dynamic Hints
14
15. 30 MAY 2017 / #OTNEMEATOUR
Step 1 : Create OPTIONS for KM’s
KM’s… How To Apply semi-Dynamic Hints
15
16. 30 MAY 2017 / #OTNEMEATOUR
Step 2 : Insert this OPTIONS into KM’s
insert <%=odiRef.getOption("INSERT_HINT")%>
into <%=snpRef.getTable("L","TARG_NAME","A")%>
select <%=odiRef.getOption("SELECT_HINT")%>
<%=snpRef.getPop("DISTINCT_ROWS")%>
<%=snpRef.getColList("", "[EXPRESSION]t[ALIAS_SEP]
[CX_COL_NAME]", ",nt", "", "")%>
from <%=snpRef.getFrom()%>
INSERT /*+ APPEND PARALLEL(t3, 8) */
INTO t3
SELECT /*+ parallel(t1) parallel(t2)
ordered use_hash(t2) index(t1 t1_abc)
index(t2 t2_abc) */ COUNT(*)
FROM t1, t2
WHERE t1.col1 = t2.col1;
KM’s… How To Apply semi-Dynamic Hints
16
27. 30 MAY 2017 / #OTNEMEATOUR
Topology Manager ➜ Languages ➜ SQL ➜ Aggregate (or other)
Adding Database Related Functions
27
28. 30 MAY 2017 / #OTNEMEATOUR
Adding User Functions
28
User Functions ➜ Right Click ➜ New User Function AVG
CORR
COUNT
COVAR_POP
COVAR_SAMP
CUME_DIST
DENSE_RANK
FIRST
LAST
LISTAGG,
MAX
MIN
PERCENT_RANK
PERCENTILE_CONT
PERCENTILE_DISC
RANK
STDDEV
STDDEV_POP
STDDEV_SAMP
SUM
VAR_POP
VAR_SAMP
VARIANCE
29. 30 MAY 2017 / #OTNEMEATOUR
Adding User Functions
29
User Functions ➜ Impact Analysis
31. 30 MAY 2017 / #OTNEMEATOUR
Running in «Asynchronous Mode»
Error Handler
Max. Number of Failed Child
Sessions = 1
Mail body
(refresh variable) Send mail
Raise Error
(error refresh variable)
Handling Alerts
31
32. 30 MAY 2017 / #OTNEMEATOUR
Send e-mail for each script, attach the appropriate file.
Mail Server : #V_MAIL_SERVER_IP
From : This is static, user that you are sending mail.
TO : #V_ERROR_MAIL_TO (need to be refreshed in the beginning of
your ETL or current package)
CC : #V_ERROR_MAIL_CC (need to be refreshed in the beginning of
your ETL or current package)
BCC : #V_ERROR_MAIL_BCC (need to be refreshed in the beginning of
your ETL or current package)
Subject : There are #V_MY_ERROR_COUNT errors exist in your
system (Error Code = #V_MY_ERROR_CODE)
Attachment : /data/my_errors/MY_Error_#V_MY_ERROR_CODE.txt
(will represent as /data/my_errors/MY_Error_1.txt initially, then will attach
regarding file in the loop, every step will attach its own script file)
Message Body : #V_MY_ERROR_DESC
Handling - in ETL - Data Quality
32
34. 30 MAY 2017 / #OTNEMEATOUR
Step 1 : Operating system folders and read file names
Step 2 : IKM Knowledge Module (edited to fit for purpose)
Step 4 : ODI Procedure to rename files for external table usage
Step 3 : ETL_FILE LOG (database table)
Step 6 : ODI Procedure to Get File List of operating system
Step 7 : a single ODI Interface (loading and transforming)
Step 8 : Some ODI Variables
Step 9 : ODI Package for running everything in right order
Step 5 : ODI Procedures to finish working with files
File2Table… Summary (F2T)
34
35. 30 MAY 2017 / #OTNEMEATOUR
Illustration for files received from «web_logs»
F2T… Prepare Folders (Step 1)
35
36. 30 MAY 2017 / #OTNEMEATOUR
Rename as : IKM SQL Control Append (Direct Load,HINT)
Copy current KM : IKM SQL Control Append
Add New Row : «Parallel DML», Transaction 0, No Commit
alter session enable parallel dml
Add Options : «SELECT HINT», «INSERT HINT»
Add New fixed Row : «Commit transaction» to Transaction 0,
Commit
/* commit */
Modify : «Insert new rows» to Transaction 0, No Commit
File2Table… Prepare IKM (Step 2)
36
37. 30 MAY 2017 / #OTNEMEATOUR
INSERT /*+ APPEND PARALLEL(4) */
INTO ODIDB.MY_TARGET_TABLE
(
MY_TARGET_COLUMN_1,
MY_TARGET_COLUMN_2,
MY_TARGET_COLUMN_3
)
SELECT
/*+ PARALLEL(MY_SOURCE_TABLE,4) FULL(MY_SOURCE_TABLE) */
MY_SOURCE_TABLE.MY_SOURCE_COLUMN_1,
MY_SOURCE_TABLE.MY_SOURCE_COLUMN_2,
MY_SOURCE_TABLE.MY_SOURCE_COLUMN_3
FROM ODISTG.I$MY_TARGET_TABLE MY_SOURCE_TABLE
WHERE MY_SOURCE_TABLE.MY_SOURCE_COLUMN_4 = ‘USA’
AND MY_SOURCE_TABLE.MY_SOURCE_COLUMN_5 = ‘New York’
F2T… Prepare IKM (Step 2)
37
38. 30 MAY 2017 / #OTNEMEATOUR
COLUMN NAME PK NULL? DATA TYPE DEF COMMENT
FILE_ID 1 N NUMBER (10) The unique identification number of file.
FILE_NAME N VARCHAR2 (50 Byte) The name of file to be processed.
FILE_GROUP N VARCHAR2 (20 Byte)
Source system name or group with the
same template.
FILE_COPIED_FLAG
Y NUMBER (1) 0
0 : not copied,
1 : copied successfully,
2 : error in copy.
FILE_COPY_DATE Y DATE Date of file copied.
FILE_READ_FLAG
Y NUMBER (1) 0
0:not read, 1:read successfully, 2:error
in read.
FILE_READ_DATE Y DATE Date of file read.
FILE_PROCESSED_FLAG Y NUMBER (1) 0 Date of file processed.
FILE_PROCESSED_DATE
Y DATE
0:not processed, 1:processed
successfully, 2:error in process.
Create table ETL_FILE_LOG
F2T… Prepare log Table (Step 3)
38
41. 30 MAY 2017 / #OTNEMEATOUR
import java.lang as lang
import java.sql as sql
import snpsftp
import java.lang.String
import os
import java.io.File as File
#db connection
driverSrc = ‘oracle.jdbc.driver.OracleDriver’
lang.Class.forName(driverSrc)
#Production Environment
urlSrc = ‘jdbc:oracle:thin:@<host>:<port>:<sid>’
#Development Environment
#urlSrc = ‘jdbc:oracle:thin:@<host>:<port>:<sid>’
userSrc = ‘ODI’
passwdSrc = ‘ODI’
ConSrc = sql.DriverManager.getConnection(urlSrc, userSrc, passwdSrc);
readDBLink = ConSrc.createStatement()
syslist = os.listdir(‘<%=odiRef.getOption( “SOURCE_DIRECTORY” )%>’)
for sys in syslist:
str = java.lang.String(sys)
if str.length() > 8:
sqlDBLink = “select * from ODI.ETL_FILE_LOG where FILE_NAME = ‘” + sys + “‘”
rqteDBLink = readDBLink.executeQuery(sqlDBLink)
if not rqteDBLink.next():
sqlDBLink = “insert into ODI.ETL_FILE_LOG (FILE_ID, FILE_NAME, FILE_GROUP,
FILE_SUB_GROUP, FILE_READ_FLAG, FILE_READ_DATE) values (ODI.SEQ_FILE_ID.NEXTVAL, ‘” +
sys + “‘, ‘<source_system_name>’, ‘<file_type>’, ’1′, SYSDATE)”
rqteDBLink = readDBLink.execute(sqlDBLink)
ConSrc.close()
Create ODI Procedure Jyhton Technology – «GetFileList»
F2T… GetFileList from OS (Step 6)
41
42. 30 MAY 2017 / #OTNEMEATOUR
Create ODI Interface (external table db table)
Source : STG.INVOICE_LOG
(based on external table my_external_table.ext)
Target : DWH.INVOICE_LOGS
KM : IKM SQL Control Append (Direct Load, HINT)
Truncate : No
Select Hint : /*+ PARALLEL(4) */
Insert Hint : /*+ APPEND PARALLEL(4) NOLOGGING */
F2T… ODI Interface (Step 7)
42
43. 30 MAY 2017 / #OTNEMEATOUR
Create ODI Variable to refresh – «File_ID»
Create ODI Variable to refresh – «File_Name»
SELECT NVL(MIN(FILE_ID), 0) FROM ODI.ETL_FILE_LOG
WHERE FILE_READ_FLAG = 1
AND FILE_PROCESSED_FLAG = 0
AND FILE_GROUP = 'INVOICE_LOGS'
SELECT FILE_NAME FROM ODI.ETL_FILE_LOG
WHERE FILE_ID = #FILE_ID
F2T… Prepare ODI Variables (Step 8)
43
44. 30 MAY 2017 / #OTNEMEATOUR
Get_File_List
ODI Procedure
FILE_ID
Refresh Variable
FILE_ID
Evaluate Variable
FILE_NAME
Refresh Variable
RENAME_FILE
ODI Procedure
INTERFACE
From:Ext_Table
To:DB.Table
UPDATE_PROCESSED
ODI Procedure
UPDATE_REJECTED
ODI Procedure
Determine_Error_Desc
ODI Procedure
ODISendMail
Rejected File
F2T… Time to Pack-UP (Step 9)
44