OOW13 Exadata and ODI with Parallel


Published on

Oracle Open World Presentation on Exadata with ODI and Parallel

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OOW13 Exadata and ODI with Parallel

  1. 1. Exadata, Oracle Data Integrator and Parallel Data Load: A Real- World Case Study Kellyn Pot’Vin, Sr. Technical Consultant
  2. 2. Who I am 3 • Westminster, CO • Oracle ACE Director • Sr. Technical Consultant, Enkitec • Blog at http://dbakevlar.com • Board of Directors and Directory of RMOUG Training Days Conference in Denver, CO. • Database Track Lead for KSCOPE 2014 in Seattle, Wa. • Tweet @DBAKevlar • Author: Expert Enterprise Manager 12c, Pro SQL Server 2012, Apress • WIT, (Women In Technology) advocate
  3. 3. Agenda for this Session 4 • Discuss ODI Requirements • Discuss Exadata Features • Discuss “Old School” Limitations to ETL design. • Solution to Create Speed in Reporting.
  4. 4. Environment 5 • Multiple Exadata Environment • Oracle Data Integrator is New Feature as part of recent consolidation effort. • 15-20 consolidated databases on each development Exadata. • 10 currently on production Exadata and consolidating more monthly. • Monitored by EM12c • Golden Gate utilized originally for consolidations, now commissioned to support ODI for new project.
  5. 5. Why Oracle Data Integrator 6
  6. 6. Oracle Data Integrator, (ODI) 7 • Enterprise platforms with its open and integrated E-LT architecture • Simple mapping wizards, user-friendly interface. • Integrates successfully with Enterprise Manager 12c. • Also integrates with weblogic, SAP, APIs and other advanced features.
  7. 7. ODI in Action… 8 *Courtesy of Oracle.com
  8. 8. What is Change Data Capture, (CDC)? 9 • Originally self-contained, then part of Streams. • Not all CDC is created, (built) the same. • Transaction “aware” • Recoverable • Handle Data Integrity • Are Scalable • Flexible and Robust • This is your goal, practice, (testing) makes perfect!
  9. 9. Designing in Oracle Data Integrator 10
  10. 10. CDC and/or just ODI? 11 • CDC is part of the Golden Gate piece. • Golden Gate offers real-time “continuous capture and delivery” of changes to your warehouse. • Once synchronized with the source, Golden Gate can be integrated into ODI. • Golden Gate requires a term license from Oracle.
  11. 11. Where Does Exadata Come in? 12 • Smartscans- Offloading large table scans to cell nodes are an impressive enhancement to performance. • Storage Indexes- If a report offers an opportunity for Exadata to create a storage index to assist in performance, then this feature will benefit reporting. • HCC- Compression if data is loaded appropriately, (APPEND should be used) and NO UPDATING data in compressed objects!
  12. 12. Out of the Starting Gate with ODI.. 13 Almost identical performance issues on each statement. Red is concurrency- but what is the concurrency on? Performance Tanked…
  13. 13. Did we Offload? 14 SQL Monitor Confirms Easily… So why aren’t we seeing great performance??
  14. 14. From the Execution Plan Perspective 15 Hasn’t Finished!
  15. 15. What did the SQL look like that was created through ODI? 16 insert into TABLE1... SELECT /*+parallel(4)*/ SUM(col1) …., CASE when col3 is null then 0 else col3 end, 0, CASE WHEN col7 IS NULL THEN 0 ELSE col7 END, 0 FROM SOURCE_TBL1, SOURCE_TBL2, SOURCE_TBL3, SOURCE_TBL4 SELECT (lots of columns and sorting and grouping) from SOURCE_TBL3 and SOURCE_TBL4) where (1=1) And col5 > 0 And (col8=col12(+)) DT_col1 between DT_col2(+) AND DT_col9(+) GROUP BY col6, col4, col3, col9, DT_col1;
  16. 16. Issue #1- Performance and High Temp Usage 17
  17. 17. The Cost of Temp Waits to ODI Processing 18 Event Waits Time(s) (ms) Pct Type ------------------------------ ------------ DB CPU 4,314 42.0 direct path read temp 545,690 3,389 33.0 User I/O direct path write temp 156,464 1,296 12.6 User I/O
  18. 18. Why the Temp Usage? 19 • Large hash joins following offload table scans. • Summing of the data in the query. • Sorting the data in the query. • DOP, (Degree of Parallelism) set at table level or as hint. Data was not aggregated or stored in the format easily utilized for reporting.
  19. 19. Let’s Talk About Performance, PGA and Limits… 20 Two Types of PGA- • Non-limited, outside of Oracle’s control, often used for PL/SQL tables, etc. • Limited by allocation per process set within Oracle for hashing, sorting, etc. • Depending on release, process type, etc., there are approximate percentages that can be expected for limits of PGA allocated to sort and hash processing.
  20. 20. When You Work Outside of PGA.. 21 Tempfile Processes Writing to Temp Processes Reading from Temp
  21. 21. Types of PGA Processing 22 Optimal- All fits within the PGA allocation per process. Single Pass- Written once to temp tablespace. Multi-Pass- Writte multiple times to temp tablesapace to achive results, (least desirable)
  22. 22. Disk is Slow… 23 What is “* Cache Hit”? When something does not “fit”, we know it’s gone back to disk to perform the task and disk is slow. If PGA allocation is surpassed, the processing is then performed in the Temp Tablespace. Where do the tempfiles for the Temp Tablespace reside? Temp is disk, disk is SLOW!
  23. 23. Tuning these Limits 24 Increasing PGA will of course increase the percentage per process for some types of sorting and hashing, but there is still a limit. Parallel can offer some assistance by spreading the PGA load across multiple processes- but there is a cost to “knitting” the results back together. What you should not do: • Do not start playing with “_pga_max_size", "_smm_max_size" and "_smm_px_max_size"
  24. 24. Stop Using Temp 25 Sorting and Hashing should be done as much as possible within PGA. Limit any “swapping” to temp. Ensure you are viewing temp usage in your explain/execution plans!
  25. 25. How Do You Know? 26 AWR Report for large timeline of snapshots is Good Place to Start…
  26. 26. Searching SQL History with EM12c 27
  27. 27. Issue #2- So What is Causing our Changes now? 28, PSU Jan 2013+ dynamic sampling initiation changes for Parallel. AWR SQL specific report, (awrsqrpt.sql) shows that Dynamic Sampling did occur.
  28. 28. First Step: Consistent Behavior 29 • Statistics verification at each step of data load process. • Parallel is needed- so how do we address dynamic sampling? • Alter system, set dynamic_sampling=0 • ODI allows hints to be introduced at universe- add hint, “dynamic_sampling(0)” • Parallel controlled to verify consistent DOP.
  29. 29. Performance Stable- We’ve Got it ALL COVERED! 30 • Identified high resource use, repeated sorts & sums. • Created objects to limit amount that has to be performed in PGA by introducing: • Rollup tables • Materialized Views with single, night time refresh, post ETL load. • Indexing
  30. 30. Elapsed Time, Improvements 31
  31. 31. Summary 32 • Identify WHAT is consuming time. • Understand the limits on PGA/Temp usage- Repeatedly the biggest hurdle in ETL projects. • Understand that TEMP is disk and disk is SLOW, even on Exadata. • Identify what data you repeatedly are aggregating and create objects to support reporting.
  32. 32. 34 Questions? Fastest Growing Companies in Dallas
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.