SlideShare a Scribd company logo

Online Statistics Gathering for ETL

Online Statistics Gathering for Bulk Loads - the official name of the feature - was introduced in Oracle 12.1. The idea is to gather optimizer statistics "on the fly" for direct path loads. Sounds good for ETL? In certain scenarios it makes sense but even then there are many points to consider so that it becomes a reliable part of your ETL processes. When exactly will it be working and when not? Do you prevent it yourself? Documented, undocumented cases, known bugs. Which statistics are gathered and which are not? What has to be considered with partitioned tables? Interval partitioning - special case?

1 of 33
Download to read offline
Online Statistics Gathering
for ETL
Andrej Pashchenko
@Andrej_SQL doag2018
We help to generate added value from data
Online Statistics Gathering for ETL2 20.11.2018
With over 650 specialists and IT experts in your region.
Online Statistics Gathering for ETL3 20.11.2018
16 Trivadis branches and more than
650 employees
Experience from more than 1,900
projects per year at over 800
customers
250 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
About Me
Online Statistics Gathering for ETL4 20.11.2018
Working at Trivadis GmbH, Düsseldorf
Focus on Oracle:
– Data Warehousing
– Application Development
– Application Performance
Course instructor „Oracle 12c New Features for
Developers“ und „Beyond SQL and PL/SQL“
Blog: http://blog.sqlora.com
Online Statistics Gathering for Bulk Loads
Online Statistics Gathering for ETL6 20.11.2018
new feature of Oracle 12c
for direct path writes
– Create Table As Select (CTAS)
– Bulk Inserts
into an empty segment
table and basic columns statistics are collected „on the fly“ (piggyback)
no additional table scan is required
What ETL scenarios can benefit?
Online Statistics Gathering for ETL7 20.11.2018
Data Warehouse
MartsCore
Metadata
Cleansing AreaStaging Area
? ? ??

Recommended

Properly Use Parallel DML for ETL
Properly Use Parallel DML for ETLProperly Use Parallel DML for ETL
Properly Use Parallel DML for ETLAndrej Pashchenko
 
SQL Macros - Game Changing Feature for SQL Developers?
SQL Macros - Game Changing Feature for SQL Developers?SQL Macros - Game Changing Feature for SQL Developers?
SQL Macros - Game Changing Feature for SQL Developers?Andrej Pashchenko
 
MERGE SQL Statement: Lesser Known Facets
MERGE SQL Statement: Lesser Known FacetsMERGE SQL Statement: Lesser Known Facets
MERGE SQL Statement: Lesser Known FacetsAndrej Pashchenko
 
Polymorphic Table Functions in 18c
Polymorphic Table Functions in 18cPolymorphic Table Functions in 18c
Polymorphic Table Functions in 18cAndrej Pashchenko
 
Adaptive Query Optimization in 12c
Adaptive Query Optimization in 12cAdaptive Query Optimization in 12c
Adaptive Query Optimization in 12cAnju Garg
 
Dan Hotka's Top 10 Oracle 12c New Features
Dan Hotka's Top 10 Oracle 12c New FeaturesDan Hotka's Top 10 Oracle 12c New Features
Dan Hotka's Top 10 Oracle 12c New FeaturesEmbarcadero Technologies
 
Oracle Parallel Distribution and 12c Adaptive Plans
Oracle Parallel Distribution and 12c Adaptive PlansOracle Parallel Distribution and 12c Adaptive Plans
Oracle Parallel Distribution and 12c Adaptive PlansFranck Pachot
 

More Related Content

What's hot

Indexes: Structure, Splits and Free Space Management Internals
Indexes: Structure, Splits and Free Space Management InternalsIndexes: Structure, Splits and Free Space Management Internals
Indexes: Structure, Splits and Free Space Management InternalsChristian Antognini
 
Ground Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_planGround Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_planMaria Colgan
 
Cooking Cassandra
Cooking CassandraCooking Cassandra
Cooking CassandraOpen-IT
 
Part3 Explain the Explain Plan
Part3 Explain the Explain PlanPart3 Explain the Explain Plan
Part3 Explain the Explain PlanMaria Colgan
 
Explaining the explain_plan
Explaining the explain_planExplaining the explain_plan
Explaining the explain_planarief12H
 
Histograms: Pre-12c and now
Histograms: Pre-12c and nowHistograms: Pre-12c and now
Histograms: Pre-12c and nowAnju Garg
 
R - Get Started I - Sanaitics
R - Get Started I - SanaiticsR - Get Started I - Sanaitics
R - Get Started I - SanaiticsVijith Nair
 
Sql and PL/SQL Best Practices I
Sql and PL/SQL Best Practices ISql and PL/SQL Best Practices I
Sql and PL/SQL Best Practices ICarlos Oliveira
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and NowAnju Garg
 
Adaptive Query Optimization
Adaptive Query OptimizationAdaptive Query Optimization
Adaptive Query OptimizationAnju Garg
 
Indexing Strategies for Oracle Databases - Beyond the Create Index Statement
Indexing Strategies for Oracle Databases - Beyond the Create Index StatementIndexing Strategies for Oracle Databases - Beyond the Create Index Statement
Indexing Strategies for Oracle Databases - Beyond the Create Index StatementSean Scott
 
Part4 Influencing Execution Plans with Optimizer Hints
Part4 Influencing Execution Plans with Optimizer HintsPart4 Influencing Execution Plans with Optimizer Hints
Part4 Influencing Execution Plans with Optimizer HintsMaria Colgan
 
Part1 of SQL Tuning Workshop - Understanding the Optimizer
Part1 of SQL Tuning Workshop - Understanding the OptimizerPart1 of SQL Tuning Workshop - Understanding the Optimizer
Part1 of SQL Tuning Workshop - Understanding the OptimizerMaria Colgan
 
Advanced functions in PL SQL
Advanced functions in PL SQLAdvanced functions in PL SQL
Advanced functions in PL SQLHosein Zare
 
The Database Environment Chapter 8
The Database Environment Chapter 8The Database Environment Chapter 8
The Database Environment Chapter 8Jeanie Arnoco
 
Best SAP ABAP Training Institute with Placement in Pune | Aspire
Best SAP ABAP Training Institute with Placement in Pune | AspireBest SAP ABAP Training Institute with Placement in Pune | Aspire
Best SAP ABAP Training Institute with Placement in Pune | AspireAspire Techsoft Pvt Ltd
 
Part2 Best Practices for Managing Optimizer Statistics
Part2 Best Practices for Managing Optimizer StatisticsPart2 Best Practices for Managing Optimizer Statistics
Part2 Best Practices for Managing Optimizer StatisticsMaria Colgan
 

What's hot (20)

Indexes: Structure, Splits and Free Space Management Internals
Indexes: Structure, Splits and Free Space Management InternalsIndexes: Structure, Splits and Free Space Management Internals
Indexes: Structure, Splits and Free Space Management Internals
 
Ground Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_planGround Breakers Romania: Explain the explain_plan
Ground Breakers Romania: Explain the explain_plan
 
Cooking Cassandra
Cooking CassandraCooking Cassandra
Cooking Cassandra
 
Part3 Explain the Explain Plan
Part3 Explain the Explain PlanPart3 Explain the Explain Plan
Part3 Explain the Explain Plan
 
Explaining the explain_plan
Explaining the explain_planExplaining the explain_plan
Explaining the explain_plan
 
Migration
MigrationMigration
Migration
 
Histograms: Pre-12c and now
Histograms: Pre-12c and nowHistograms: Pre-12c and now
Histograms: Pre-12c and now
 
R - Get Started I - Sanaitics
R - Get Started I - SanaiticsR - Get Started I - Sanaitics
R - Get Started I - Sanaitics
 
PLSQL Advanced
PLSQL AdvancedPLSQL Advanced
PLSQL Advanced
 
Sql and PL/SQL Best Practices I
Sql and PL/SQL Best Practices ISql and PL/SQL Best Practices I
Sql and PL/SQL Best Practices I
 
Histograms : Pre-12c and Now
Histograms : Pre-12c and NowHistograms : Pre-12c and Now
Histograms : Pre-12c and Now
 
Adaptive Query Optimization
Adaptive Query OptimizationAdaptive Query Optimization
Adaptive Query Optimization
 
Indexing Strategies for Oracle Databases - Beyond the Create Index Statement
Indexing Strategies for Oracle Databases - Beyond the Create Index StatementIndexing Strategies for Oracle Databases - Beyond the Create Index Statement
Indexing Strategies for Oracle Databases - Beyond the Create Index Statement
 
Table functions - Planboard Symposium 2013
Table functions - Planboard Symposium 2013Table functions - Planboard Symposium 2013
Table functions - Planboard Symposium 2013
 
Part4 Influencing Execution Plans with Optimizer Hints
Part4 Influencing Execution Plans with Optimizer HintsPart4 Influencing Execution Plans with Optimizer Hints
Part4 Influencing Execution Plans with Optimizer Hints
 
Part1 of SQL Tuning Workshop - Understanding the Optimizer
Part1 of SQL Tuning Workshop - Understanding the OptimizerPart1 of SQL Tuning Workshop - Understanding the Optimizer
Part1 of SQL Tuning Workshop - Understanding the Optimizer
 
Advanced functions in PL SQL
Advanced functions in PL SQLAdvanced functions in PL SQL
Advanced functions in PL SQL
 
The Database Environment Chapter 8
The Database Environment Chapter 8The Database Environment Chapter 8
The Database Environment Chapter 8
 
Best SAP ABAP Training Institute with Placement in Pune | Aspire
Best SAP ABAP Training Institute with Placement in Pune | AspireBest SAP ABAP Training Institute with Placement in Pune | Aspire
Best SAP ABAP Training Institute with Placement in Pune | Aspire
 
Part2 Best Practices for Managing Optimizer Statistics
Part2 Best Practices for Managing Optimizer StatisticsPart2 Best Practices for Managing Optimizer Statistics
Part2 Best Practices for Managing Optimizer Statistics
 

Similar to Online Statistics Gathering for ETL

ELT Publishing Tool Overview V3_Jeff
ELT Publishing Tool Overview V3_JeffELT Publishing Tool Overview V3_Jeff
ELT Publishing Tool Overview V3_JeffJeff McQuigg
 
Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007paulguerin
 
MariaDB: Engine Independent Table Statistics, including histograms
MariaDB: Engine Independent Table Statistics, including histogramsMariaDB: Engine Independent Table Statistics, including histograms
MariaDB: Engine Independent Table Statistics, including histogramsSergey Petrunya
 
SQL Tuning and VST
SQL Tuning and VST SQL Tuning and VST
SQL Tuning and VST Kyle Hailey
 
Skills Portfolio
Skills PortfolioSkills Portfolio
Skills Portfoliorolee23
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit
 
Fordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usaFordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usarukford
 
Fordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usaFordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usarukford
 
Ssis Ssas Ssrs Sp Pps Hong Bing Li
Ssis Ssas Ssrs Sp Pps Hong Bing LiSsis Ssas Ssrs Sp Pps Hong Bing Li
Ssis Ssas Ssrs Sp Pps Hong Bing LiHong-Bing Li
 
X rec extened reconciliation using excel vba
X rec   extened reconciliation using excel vbaX rec   extened reconciliation using excel vba
X rec extened reconciliation using excel vbasenthilsundaresan
 
Faster transactions & analytics with the new SQL2016 In-memory technologies
Faster transactions & analytics with the new SQL2016 In-memory technologiesFaster transactions & analytics with the new SQL2016 In-memory technologies
Faster transactions & analytics with the new SQL2016 In-memory technologiesHenk van der Valk
 
Oracle Join Methods and 12c Adaptive Plans
Oracle Join Methods and 12c Adaptive PlansOracle Join Methods and 12c Adaptive Plans
Oracle Join Methods and 12c Adaptive PlansFranck Pachot
 
325546_adding fields in CJI3 & CJI5.pdf
325546_adding fields in CJI3 & CJI5.pdf325546_adding fields in CJI3 & CJI5.pdf
325546_adding fields in CJI3 & CJI5.pdfVaishali Ketkar
 
Computer Architecture - Data Path & Pipeline Hazards
Computer Architecture - Data Path & Pipeline HazardsComputer Architecture - Data Path & Pipeline Hazards
Computer Architecture - Data Path & Pipeline HazardsThyagharajan K.K.
 
IBM Informix dynamic server 11 10 Cheetah Sql Features
IBM Informix dynamic server 11 10 Cheetah Sql FeaturesIBM Informix dynamic server 11 10 Cheetah Sql Features
IBM Informix dynamic server 11 10 Cheetah Sql FeaturesKeshav Murthy
 
Ssis sql ssrs_sp_ssas_mdx_hb_li
Ssis sql ssrs_sp_ssas_mdx_hb_liSsis sql ssrs_sp_ssas_mdx_hb_li
Ssis sql ssrs_sp_ssas_mdx_hb_liHong-Bing Li
 
Top 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsTop 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsNirav Shah
 

Similar to Online Statistics Gathering for ETL (20)

ELT Publishing Tool Overview V3_Jeff
ELT Publishing Tool Overview V3_JeffELT Publishing Tool Overview V3_Jeff
ELT Publishing Tool Overview V3_Jeff
 
Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007Myth busters - performance tuning 101 2007
Myth busters - performance tuning 101 2007
 
MariaDB: Engine Independent Table Statistics, including histograms
MariaDB: Engine Independent Table Statistics, including histogramsMariaDB: Engine Independent Table Statistics, including histograms
MariaDB: Engine Independent Table Statistics, including histograms
 
BIWorkDemos
BIWorkDemosBIWorkDemos
BIWorkDemos
 
SQL Tuning and VST
SQL Tuning and VST SQL Tuning and VST
SQL Tuning and VST
 
Skills Portfolio
Skills PortfolioSkills Portfolio
Skills Portfolio
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
 
Fordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usaFordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usa
 
Fordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usaFordrazborka.zu8.ru eec iv-usa
Fordrazborka.zu8.ru eec iv-usa
 
Ssis Ssas Ssrs Sp Pps Hong Bing Li
Ssis Ssas Ssrs Sp Pps Hong Bing LiSsis Ssas Ssrs Sp Pps Hong Bing Li
Ssis Ssas Ssrs Sp Pps Hong Bing Li
 
X rec extened reconciliation using excel vba
X rec   extened reconciliation using excel vbaX rec   extened reconciliation using excel vba
X rec extened reconciliation using excel vba
 
Faster transactions & analytics with the new SQL2016 In-memory technologies
Faster transactions & analytics with the new SQL2016 In-memory technologiesFaster transactions & analytics with the new SQL2016 In-memory technologies
Faster transactions & analytics with the new SQL2016 In-memory technologies
 
Oracle Join Methods and 12c Adaptive Plans
Oracle Join Methods and 12c Adaptive PlansOracle Join Methods and 12c Adaptive Plans
Oracle Join Methods and 12c Adaptive Plans
 
325546_adding fields in CJI3 & CJI5.pdf
325546_adding fields in CJI3 & CJI5.pdf325546_adding fields in CJI3 & CJI5.pdf
325546_adding fields in CJI3 & CJI5.pdf
 
Computer Architecture - Data Path & Pipeline Hazards
Computer Architecture - Data Path & Pipeline HazardsComputer Architecture - Data Path & Pipeline Hazards
Computer Architecture - Data Path & Pipeline Hazards
 
Etl2
Etl2Etl2
Etl2
 
IBM Informix dynamic server 11 10 Cheetah Sql Features
IBM Informix dynamic server 11 10 Cheetah Sql FeaturesIBM Informix dynamic server 11 10 Cheetah Sql Features
IBM Informix dynamic server 11 10 Cheetah Sql Features
 
Ssis sql ssrs_sp_ssas_mdx_hb_li
Ssis sql ssrs_sp_ssas_mdx_hb_liSsis sql ssrs_sp_ssas_mdx_hb_li
Ssis sql ssrs_sp_ssas_mdx_hb_li
 
Top 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tipsTop 10 Oracle SQL tuning tips
Top 10 Oracle SQL tuning tips
 
Pentaho etl-tool
Pentaho etl-toolPentaho etl-tool
Pentaho etl-tool
 

Recently uploaded

Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...DrSumathyV
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxHizkiaJastis
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsDataArchiva
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............mahetamanav24
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...ThinkInnovation
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdfAlexia Trejo
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensKondapi V Siva Rama Brahmam
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)UNCResearchHub
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelTope Osanyintuyi
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxPoonamRijal
 
What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?Denodo
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughsNikolas Markou
 

Recently uploaded (13)

Electricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptxElectricity Year 2023_updated_22022024.pptx
Electricity Year 2023_updated_22022024.pptx
 
Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...Introduction to data science.pdf-Definition,types and application of Data Sci...
Introduction to data science.pdf-Definition,types and application of Data Sci...
 
ppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptxppt penjualan berbasis online omset.pptx
ppt penjualan berbasis online omset.pptx
 
Tips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data GoalsTips to Align with Your Salesforce Data Goals
Tips to Align with Your Salesforce Data Goals
 
itc limited word file.pdf...............
itc limited word file.pdf...............itc limited word file.pdf...............
itc limited word file.pdf...............
 
Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...Unlocking New Insights Into the World of European Soccer Through the European...
Unlocking New Insights Into the World of European Soccer Through the European...
 
Choose your perfect jacket.pdf
Choose your perfect jacket.pdfChoose your perfect jacket.pdf
Choose your perfect jacket.pdf
 
Operations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample ScreensOperations Data On Mobile - inSis Mobile App - Sample Screens
Operations Data On Mobile - inSis Mobile App - Sample Screens
 
A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)A Gentle Introduction to Text Analysis :)
A Gentle Introduction to Text Analysis :)
 
Basics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft ExcelBasics of Creating Graphs / Charts using Microsoft Excel
Basics of Creating Graphs / Charts using Microsoft Excel
 
fundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptxfundamentals of digital imaging - POONAM.pptx
fundamentals of digital imaging - POONAM.pptx
 
What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?What you need to know about Generative AI and Data Management?
What you need to know about Generative AI and Data Management?
 
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
Artificial Intelligence for Vision:  A walkthrough of recent breakthroughsArtificial Intelligence for Vision:  A walkthrough of recent breakthroughs
Artificial Intelligence for Vision: A walkthrough of recent breakthroughs
 

Online Statistics Gathering for ETL

  • 1. Online Statistics Gathering for ETL Andrej Pashchenko @Andrej_SQL doag2018
  • 2. We help to generate added value from data Online Statistics Gathering for ETL2 20.11.2018
  • 3. With over 650 specialists and IT experts in your region. Online Statistics Gathering for ETL3 20.11.2018 16 Trivadis branches and more than 650 employees Experience from more than 1,900 projects per year at over 800 customers 250 Service Level Agreements Over 4,000 training participants Research and development budget: CHF 5.0 million Financially self-supporting and sustainably profitable
  • 4. About Me Online Statistics Gathering for ETL4 20.11.2018 Working at Trivadis GmbH, Düsseldorf Focus on Oracle: – Data Warehousing – Application Development – Application Performance Course instructor „Oracle 12c New Features for Developers“ und „Beyond SQL and PL/SQL“ Blog: http://blog.sqlora.com
  • 5. Online Statistics Gathering for Bulk Loads Online Statistics Gathering for ETL6 20.11.2018 new feature of Oracle 12c for direct path writes – Create Table As Select (CTAS) – Bulk Inserts into an empty segment table and basic columns statistics are collected „on the fly“ (piggyback) no additional table scan is required
  • 6. What ETL scenarios can benefit? Online Statistics Gathering for ETL7 20.11.2018 Data Warehouse MartsCore Metadata Cleansing AreaStaging Area ? ? ??
  • 7. Online Statistics Gathering for ETL8 20.11.2018 Control, Preconditions, Scope
  • 8. Switch ON/OFF, selective enable/disable Online Statistics Gathering for ETL9 20.11.2018 The behaviour is controlled by the parameter _optimizer_gather_stats_on_load The default is TRUE Hints: (NO_)GATHER_OPTIMIZER_STATISTICS These hints are only meaningful to override the parameter setting for particular SQL Do not enforce gathering without precondictions (empty table, direct path) ALTER SYSTEM(SESSION) SET "_optimizer_gather_stats_on_load" = TRUE (FALSE); INSERT /*+ APPEND GATHER_OPTIMIZER_STATISTICS */ INTO ...
  • 9. How do I know it works? Online Statistics Gathering for ETL10 20.11.2018 New step in explain plan, but not sufficient to judge It only means: the feature is turned on and will be considered Also check LAST_ANALYZED for table and NOTES for column statistics: Also reflected in (ALL|DBA)USER_TAB_STATS_HISTORY SQL> SELECT table_name, num_rows, 2 last_analyzed 3 FROM user_tab_statistics 4 WHERE table_name = 'T2'; TABLE NUM_ROWS LAST_ANALY ----- ---------- ---------- T2 10000 09-OCT-18 SQL> SELECT column_name, num_distinct, 2 num_buckets, sample_size, notes 3 FROM user_tab_col_statistics 4 WHERE table_name = 'T2'; COLUM NUM_DIST NUM_B SAMPLE_S NOTES ----- -------- ----- -------- ------------- N 10000 1 10000 STATS_ON_LOAD
  • 10. How do I know it works? Online Statistics Gathering for ETL11 20.11.2018 Oracle Documentation suggests to look at USER_TAB_MODIFICATIONS.INSERTS column. If the query returns a row indicating the number of rows loaded, then the most recent bulk load did not gather statistics automatically. INSERT /*+ APPEND */ ... SQL> begin 2 dbms_stats.FLUSH_DATABASE_MONITORING_INFO; 3 end; 4 / PL/SQL procedure successfully completed. SQL> SELECT inserts 2 FROM user_tab_modifications 3 WHERE table_name = 'T'; INSERTS ---------- 10000
  • 11. Direct Path Insert (I) Online Statistics Gathering for ETL12 20.11.2018 DB-blocks are written to disk by user server process (not DBWR), bypassing the buffer cache CTAS is always perfoming direct path Use APPEND hint for INSERT AS SELECT (IAS) Also INSERT mit APPEND_VALUES from FORALL statement in PL/SQL INSERT /*+ append */ INTO ... SELECT Hints are evil ?! Not this one! It is a non-optimizer hint and the official and only possible way to tell the DB to do a direct path insert
  • 12. Direct Path Insert (II) – Parallel DML Online Statistics Gathering for ETL13 20.11.2018 If INSERT is done in parallel (DML) the default mode ist direct path Enable parallel DML for your session or with the hint just for your SQL Specify PARALLEL clause for the table or use PARALLEL hint in your INSERT: Parallel DML would not be possible in some cases Do not rely on that and always specify APPEND hint altogether ALTER SESSION ENABLE PARALLEL DML; or INSERT /*+ enable_parallel_dml */ INTO ... ALTER TABLE tx2 PARALLEL 4; or INSERT /*+ parallel */ INTO tx2 ... INSERT /*+ append enable_parallel_dml parallel */ INTO ...
  • 13. Direct Path Insert (III) Online Statistics Gathering for ETL14 20.11.2018 There are some cases where APPEND hint can be ignored (e.g. triggers, enabled foreign keys). Reliable FK-constraints: valuable for the CBO, but not disruptive for ETL (RELY DISABLE NOVALIDATE) Trigger: redesign, drop How to proof? – LOAD AS SELECT in explain plan – You cannot read the table in the same transaction after direct path insert ORA-12838: cannot read/modify an object after modifying it in parallel ------------------------------------------ | 0 | INSERT STATEMENT | | 1 | LOAD AS SELECT | | 2 | OPTIMIZER STATISTICS GATHERING |
  • 14. Direct Path and MERGE? Online Statistics Gathering for ETL15 20.11.2018 SQL> MERGE /*+ append */ 2 INTO tx2 USING tx ON (tx.n = tx2.n) 3 WHEN NOT MATCHED THEN INSERT (N) VALUES (tx.n) ; 10,000 rows merged. ---------------------------------------- | Id | Operation | Name | ---------------------------------------- | 0 | MERGE STATEMENT | | | 1 | MERGE | TX2 | | 2 | VIEW | | |* 3 | HASH JOIN RIGHT OUTER| | | 4 | TABLE ACCESS FULL | TX2 | | 5 | TABLE ACCESS FULL | TX | ---------------------------------------- SQL> SELECT count(*) FROM tx2; ORA-12838: cannot read/modify an object after modifying it in parallel SQL> INSERT /*+ append*/ INTO tx2 2 SELECT n FROM tx 3 WHERE n NOT IN (SELECT n FROM tx2); 10,000 rows inserted. ----------------------------------------------- | Id | Operation | Name ----------------------------------------------- | 0 | INSERT STATEMENT | | 1 | LOAD AS SELECT | TX2 | 2 | OPTIMIZER STATISTICS GATHERING | |* 3 | HASH JOIN RIGHT ANTI NA | | 4 | TABLE ACCESS FULL | TX2 | 5 | TABLE ACCESS FULL | TX ----------------------------------------------- direct path write has happend
  • 15. Empty Segment Online Statistics Gathering for ETL16 20.11.2018 What does „empty“ exactly mean? The rules apply to partitions as well Case Online Statistics? New Table YES DELETE all rows NO INSERT and ROLLBACK NO TRUNCATE (DROP STORAGE) YES TRUNCATE REUSE STORAGE NO
  • 16. What kind of statistics are gathered? Online Statistics Gathering for ETL17 20.11.2018 Table Statistics Base column statistics, also virtual columns and column groups (12.2) No histograms – not often used in Stage and Cleansing but can be important in Core and Mart areas No Index Statistics – must be gathered separately. Interactions with the default statistics maintainence job can be dangerous: Data TRUNCATE DBMS_STATS Empty BULK LOAD Data Wrong index stats may confuse CBO! Empty
  • 17. Online Statistics Gathering for ETL18 20.11.2018 Restrictions and Pitfalls
  • 18. Restrictions and Pitfalls (I) Online Statistics Gathering for ETL19 20.11.2018 Restrictions are documented Check your version´s documentation. Even if you are on 12.1 also check the 12.2 documentation. It has better explanations, some restrictions were lifted in 12.2 and backported to 12.1, so check MOS and test it! Some examples: – IOT, nested tables – Statistics are locked or PUBLISH preference is set to FALSE – Multitable Insert (INSERT ALL/FIRST)
  • 19. Restrictions and Pitfalls (II) Online Statistics Gathering for ETL20 20.11.2018 Statistics are gathered only if all skipped columns have a default value (e.g. extending the tables and not changing the ETL process at the same time) Virtual columns are claimed to be a restriction in 12.1 documentation, but not in 12.2 In fact it also works in 12.1.0.2 (backport?) The presense of extended statistics prevents online statistics gathering (bug 18425876), fixed in 12.2, but also works in 12.1.0.2 (backport?)
  • 20. Online Statistics Gathering for ETL21 20.11.2018 Partitioned Tables
  • 21. Partitioned Tables (simple case) Online Statistics Gathering for ETL22 20.11.2018 Insert into an empty partitioned table: Only global table statistics are gathered No online statistics gathering for subsequent inserts (the table is not empty anymore) Even if inserting into an empty partition INSERT /*+ append*/ INTO t_part SELECT ... Table T_PART P1 P2
  • 22. Partitioned Tables (using extended syntax) Online Statistics Gathering for ETL23 20.11.2018 Insert into an empty partition, explicitly specifying partition name: Only partition level statistics are gathered Online statistics gathering is also active for subsequent inserts into yet empty partitions INSERT /*+ append*/ INTO t_part PARTITION (P1) SELECT ... Table T_PART P1 P2 INSERT /*+ append*/ INTO t_part PARTITION (P2) SELECT ...
  • 23. Partitioned Tables (using extended syntax+incremental) Online Statistics Gathering for ETL24 20.11.2018 Insert into an empty partition, explicitly specifying partition name, INCREMENTAL- preference is TRUE: Partition level statistics and synopsis for global stats are gathered Online statistics gathering is also active for subsequent inserts into yet empty partitions INSERT /*+ append*/ INTO t_part PARTITION (P1) Table T_PART P1 P2 INSERT /*+ append*/ INTO t_part PARTITION (P2) exec dbms_stats.set_table_prefs(null,'T_PART','INCREMENTAL','TRUE')
  • 24. Partitioned Tables (using extended syntax+incremental) Online Statistics Gathering for ETL25 20.11.2018 Insert into an empty partitioned table, no partition-extended syntaxl (partition name), INCREMENTAL-preference is TRUE: NO Online statistics gathering at all! Documented restriction INSERT /*+ append*/ INTO t_part Table T_PART P1 P2 INSERT /*+ append*/ INTO t_part exec dbms_stats.set_table_prefs(null,'T_PART','INCREMENTAL','TRUE')
  • 25. Partitioned Tables Online Statistics Gathering for ETL26 20.11.2018 Loading into an empty partition INCREMENTAL=TRUE INCREMENTAL=FALSE Partition name specified partition-level stats + synopsis partition-level stats No partition name specified No stats at all! global stats
  • 26. Partitioned Tables – Interval Partitioning Online Statistics Gathering for ETL27 20.11.2018 Partitions are automatically created as data arrives No online statistics in 12.1, fixed in 12.2 Partition names are system generated and are not known in advance No way for online gathering of partition level stats? Actually there is one – using extended syntax with values: But the value is hard-coded (no bind variables), so that dynamic SQL is needed INSERT /*+ append*/ INTO t_part PARTITION FOR (DATE '2018-11-20')
  • 27. Table T_PART Partitioned Tables – Partition Exchange Online Statistics Gathering for ETL28 20.11.2018 P1 INSERT /*+ append*/ INTO t_exchange P2 P3 T_EXCHANGE Table T_PART P1 P2 P3 T_EXCHANGE
  • 28. Partitioned Tables – Partition Exchange Online Statistics Gathering for ETL29 20.11.2018 Since 12.1 synopses can be gathered on non-partitioned tables This happens also with online statistcs gathering CREATE TABLE t_exchange FOR EXCHANGE WITH TABLE t_part; BEGIN dbms_stats.set_table_prefs(null,'t_exchange','INCREMENTAL','TRUE'); dbms_stats.set_table_prefs(null,'t_exchange','INCREMENTAL_LEVEL','TABLE'); END; INSERT /*+ append */ INTO t_exchange ... -- Online statistics including synopsis are gathered ALTER TABLE T_PART EXCHANGE PARTITION P3 WITH TABLE t_exchange; -- Partition statistics and synopsis are exchanged
  • 29. Online Statistics Gathering for ETL30 20.11.2018 Autonomous Data Warehouse Cloud Service (ADWC) What‘s new?
  • 30. Autonomous Data Warehouse Cloud – What‘s new? Online Statistics Gathering for ETL31 20.11.2018 Statistics are gathered automatically Unlike 12c, this works also for non-empty tables for histograms Two new undocumented parameters _optimizer_gather_stats_on_load_all (default: TRUE) _optimizer_gather_stats_on_load_hist (default: TRUE) ------------------------------------------------------ | Id | Operation | Name | ------------------------------------------------------ | 0 | INSERT STATEMENT | | | 1 | LOAD AS SELECT | TARGET | | 2 | PX COORDINATOR | | | 3 | PX SEND QC (RANDOM) | :TQ10000 | | 4 | OPTIMIZER STATISTICS GATHERING | | | 5 | PX BLOCK ITERATOR | | | 6 | TABLE ACCESS STORAGE FULL | SOURCE | ------------------------------------------------------
  • 31. Summary Online Statistics Gathering for ETL32 20.11.2018 very useful feature if loading in combination with truncate (Stage, Cleansing) becomes better with every release since 12.1 can save time and resources during ETL runs becomes more complicated in case of partitioned tables requires dynamic SQL for efficient loading in partitioned tables No support for MERGE /*+ append */ subject to check and test with your load scenario and your ETL-tool Test it!
  • 32. Online Statistics Gathering for ETL33 20.11.2018 http://blog.sqlora.com/en/tag/osg/ https://blogs.oracle.com/optimizer/efficient-statistics-maintenance-for-partitioned-tables- using-incremental-statistics-part-2 https://danischnider.wordpress.com/2018/07/11/gathering-statistics-in-the-autonomous-data- warehouse-cloud/
  • 33. Trivadis @ DOAG 2018 #opencompany Booth: 3rd Floor – next to the escalator We share our Know how! Just come across, Live-Presentations and documents archive T-Shirts, Contest and much more We look forward to your visit 20.11.2018 Online Statistics Gathering for ETL34