Exadata, Oracle Data Integrator
and Parallel Data Load: A Real-
World Case Study
Kellyn Pot’Vin, Sr. Technical Consultant
Who I am
• Westminster, CO
• Oracle ACE Director
• Sr. Technical Consultant, Enkitec
• Blog at http://dbakevlar.com
• Board of Directors and Directory of RMOUG
Training Days Conference in Denver, CO.
• Database Track Lead for KSCOPE 2014 in
• Tweet @DBAKevlar
• Author: Expert Enterprise Manager 12c, Pro
SQL Server 2012, Apress
• WIT, (Women In Technology) advocate
Agenda for this Session
• Discuss ODI Requirements
• Discuss Exadata Features
• Discuss “Old School” Limitations to
• Solution to Create Speed in
• Multiple Exadata Environment
• Oracle Data Integrator is New Feature as part of
recent consolidation effort.
• 15-20 consolidated databases on each
• 10 currently on production Exadata and
consolidating more monthly.
• Monitored by EM12c
• Golden Gate utilized originally for
consolidations, now commissioned to support
ODI for new project.
Oracle Data Integrator, (ODI)
• Enterprise platforms with its open and
integrated E-LT architecture
• Simple mapping wizards, user-friendly
• Integrates successfully with Enterprise
• Also integrates with weblogic, SAP, APIs and
other advanced features.
What is Change Data Capture,
• Originally self-contained, then part of Streams.
• Not all CDC is created, (built) the same.
• Transaction “aware”
• Handle Data Integrity
• Are Scalable
• Flexible and Robust
• This is your goal, practice, (testing) makes
CDC and/or just ODI?
• CDC is part of the Golden Gate piece.
• Golden Gate offers real-time “continuous
capture and delivery” of changes to your
• Once synchronized with the source, Golden
Gate can be integrated into ODI.
• Golden Gate requires a term license from
Where Does Exadata Come in?
• Smartscans- Offloading large table scans to
cell nodes are an impressive enhancement
• Storage Indexes- If a report offers an
opportunity for Exadata to create a storage
index to assist in performance, then this
feature will benefit reporting.
• HCC- Compression if data is loaded
appropriately, (APPEND should be used) and
NO UPDATING data in compressed objects!
Out of the Starting Gate with
Almost identical performance issues on each
Red is concurrency- but what is the concurrency on?
Did we Offload?
SQL Monitor Confirms Easily…
So why aren’t we seeing great performance??
From the Execution Plan
What did the SQL look like that
was created through ODI?
insert into TABLE1...
SELECT /*+parallel(4)*/ SUM(col1) ….,
CASE when col3 is null then 0 else col3 end, 0,
CASE WHEN col7 IS NULL THEN 0 ELSE col7 END, 0
FROM SOURCE_TBL1, SOURCE_TBL2, SOURCE_TBL3,
SELECT (lots of columns and sorting and
grouping) from SOURCE_TBL3 and SOURCE_TBL4)
And col5 > 0
And (col8=col12(+)) DT_col1 between DT_col2(+)
GROUP BY col6, col4, col3, col9, DT_col1;
The Cost of Temp Waits to ODI
Event Waits Time(s) (ms) Pct Type
DB CPU 4,314 42.0
direct path read temp 545,690 3,389 33.0 User I/O
direct path write temp 156,464 1,296 12.6 User I/O
Why the Temp Usage?
• Large hash joins following offload table scans.
• Summing of the data in the query.
• Sorting the data in the query.
• DOP, (Degree of Parallelism) set at table level
or as hint.
Data was not aggregated or stored in the
format easily utilized for reporting.
Let’s Talk About Performance,
PGA and Limits…
Two Types of PGA-
• Non-limited, outside of Oracle’s control, often
used for PL/SQL tables, etc.
• Limited by allocation per process set within Oracle
for hashing, sorting, etc.
• Depending on release, process type, etc., there
are approximate percentages that can be
expected for limits of PGA allocated to sort and
When You Work Outside of PGA..
Types of PGA Processing
Optimal- All fits within the PGA allocation per
Single Pass- Written once to temp tablespace.
Multi-Pass- Writte multiple times to temp
tablesapace to achive results, (least desirable)
Disk is Slow…
What is “* Cache Hit”?
When something does not “fit”, we know it’s gone
back to disk to perform the task and disk is slow.
If PGA allocation is surpassed, the processing is then
performed in the Temp Tablespace.
Where do the tempfiles for the Temp Tablespace
Temp is disk, disk is SLOW!
Tuning these Limits
Increasing PGA will of course increase the
percentage per process for some types of
sorting and hashing, but there is still a limit.
Parallel can offer some assistance by spreading
the PGA load across multiple processes- but
there is a cost to “knitting” the results back
What you should not do:
• Do not start playing with “_pga_max_size",
"_smm_max_size" and "_smm_px_max_size"
Stop Using Temp
Sorting and Hashing should be done as much
as possible within PGA.
Limit any “swapping” to temp.
Ensure you are viewing temp usage in your
How Do You Know?
AWR Report for large timeline of snapshots is Good
Place to Start…
Issue #2- So What is Causing our
18.104.22.168, PSU Jan 2013+ dynamic sampling
initiation changes for Parallel.
AWR SQL specific report, (awrsqrpt.sql)
shows that Dynamic Sampling did occur.
First Step: Consistent Behavior
• Statistics verification at each step of data load
• Parallel is needed- so how do we address
• Alter system, set dynamic_sampling=0
• ODI allows hints to be introduced at
universe- add hint, “dynamic_sampling(0)”
• Parallel controlled to verify consistent DOP.
Performance Stable- We’ve Got
it ALL COVERED!
• Identified high resource use, repeated sorts &
• Created objects to limit amount that has to be
performed in PGA by introducing:
• Rollup tables
• Materialized Views with single, night time
refresh, post ETL load.
• Identify WHAT is consuming time.
• Understand the limits on PGA/Temp
usage- Repeatedly the biggest hurdle in
• Understand that TEMP is disk and disk is
SLOW, even on Exadata.
• Identify what data you repeatedly are
aggregating and create objects to support
Fastest Growing Companies
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.