My presentation "Analysing and troubleshooting Parallel Execution" held at IT Tage Conference 2015, Frankfurt

2.  Independent consultant
 Performance Troubleshooting
 In-house workshops
 Cost-Based Optimizer
 Performance By Design
 Oracle ACE Director
 Member of OakTable Network

3.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

4.  Oracle Database Enterprise Edition includes
the powerful Parallel Execution feature that
allows spreading the processing of a single
SQL statement execution across multiple
worker processes
 The feature is fully integrated into the Cost
Based Optimizer as well as the execution
runtime engine and automatically distributes
the work across the so called Parallel Workers

5.  Simple generic parallelization example
Task: Compute sum of 8 numbers
1+8=9, 9+7=16, 16+9=25,...
1+8+7+9+6+2+6+3= ???
n=8 numbers, 7 computation steps required
Serial execution: 7 time units

6. Simple generic parallelization example
4 workers
3+ time units
1 + 8
= 9
9 + 7
= 16
6 + 2
= 8
6 + 3
= 9
9 + 16
= 25
8 + 9
= 17
25 + 17
= 42
Coordinator

7. Parallel Execution doesn’t mean “work smarter”
You’re actually willing to accept to “work harder”
Could also be called: “Brute force” approach

8. So with Parallel Execution there
might be the problem that it
doesn’t work “hard enough”

9.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

10.  More reasons why Oracle Parallel Execution
might not reduce runtime as expected:
 Parallel DML/DDL gotchas
 “Downgrade” at execution time (less workers
assigned than expected)
 Overhead of Parallel Execution implementation
 Limitations of Parallel Execution implementation

11. Parallel DML / DDL gotchas
 DML / DDL part can run parallel or serial
 Query part can run parallel or serial

12. Parallel CTAS but serial query
-------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | Q1,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T4 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | Q1,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN| :TQ10000 | | S->P | RND-ROBIN |
|* 6 | HASH JOIN | | | | |
| 7 | TABLE ACCESS FULL| T2 | | | |
| 8 | TABLE ACCESS FULL| T2 | | | |
-------------------------------------------------------------------------

13. Serial CTAS but parallel query
--------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | LOAD AS SELECT | T4 | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) |
|* 4 | HASH JOIN BUFFERED | | Q1,02 | PCWP | |
| 5 | PX RECEIVE | | Q1,02 | PCWP | |
| 6 | PX SEND HASH | :TQ10000 | Q1,00 | P->P | HASH |
| 7 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 8 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
| 9 | PX RECEIVE | | Q1,02 | PCWP | |
| 10 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH |
| 11 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
| 12 | TABLE ACCESS FULL| T2 | Q1,01 | PCWP | |
--------------------------------------------------------------------------

14.  Other reasons why Oracle Parallel Execution
might not scale as expected:
 Parallel DML/DDL gotchas
 “Downgrade” at execution time (less workers
assigned than expected)
 Overhead of Parallel Execution implementation
 Limitations of Parallel Execution implementation

15. “Parallel Forced Serial” Example
------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | PX COORDINATOR FORCED SERIAL| | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10003 | Q1,03 | P->S | QC (RAND) |
| 3 | HASH UNIQUE | | Q1,03 | PCWP | |
| 4 | PX RECEIVE | | Q1,03 | PCWP | |
| 5 | PX SEND HASH | :TQ10002 | Q1,02 | P->P | HASH |
|* 6 | HASH JOIN BUFFERED | | Q1,02 | PCWP | |
| 7 | PX RECEIVE | | Q1,02 | PCWP | |
| 8 | PX SEND HASH | :TQ10000 | Q1,00 | P->P | HASH |
| 9 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 10 | TABLE ACCESS FULL | T2 | Q1,00 | PCWP | |
| 11 | PX RECEIVE | | Q1,02 | PCWP | |
| 12 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH |
| 13 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
| 14 | TABLE ACCESS FULL | T2 | Q1,01 | PCWP | |
------------------------------------------------------------------------------

16.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

17. select median(id) from t2;
-----------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | SORT GROUP BY | | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM)| :TQ10000 | Q1,00 | P->S | QC (RAND) |
| 4 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 5 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
-----------------------------------------------------------------------

18. create table t3 parallel
as
select * from t2
where rownum <= 10000000;
-----------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ20001 | Q2,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T3 | Q2,01 | PCWP | |
| 4 | PX RECEIVE | | Q2,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN | :TQ20000 | | S->P | RND-ROBIN |
|* 6 | COUNT STOPKEY | | | | |
| 7 | PX COORDINATOR | | | | |
| 8 | PX SEND QC (RANDOM) | :TQ10000 | Q1,00 | P->S | QC (RAND) |
|* 9 | COUNT STOPKEY | | Q1,00 | PCWC | |
| 10 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 11 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
-----------------------------------------------------------------------------

19. create table t3 parallel
as select * from (select a.*,
lag(filler, 1) over (order by id) as prev_filler
from t2 a);
--------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ20001 | Q2,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T3 | Q2,01 | PCWP | |
| 4 | PX RECEIVE | | Q2,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN | :TQ20000 | | S->P | RND-ROBIN |
| 6 | VIEW | | | | |
| 7 | WINDOW BUFFER | | | | |
| 8 | PX COORDINATOR | | | | |
| 9 | PX SEND QC (ORDER) | :TQ10001 | Q1,01 | P->S | QC (ORDER) |
| 10 | SORT ORDER BY | | Q1,01 | PCWP | |
| 11 | PX RECEIVE | | Q1,01 | PCWP | |
| 12 | PX SEND RANGE | :TQ10000 | Q1,00 | P->P | RANGE |
| 13 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 14 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
--------------------------------------------------------------------------------

20. All these examples have one thing in common:
If the Query Coordinator (non-parallel part)
needs to perform a significant part of the overall
work, Parallel Execution won’t reduce the
runtime as expected

21.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

24.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

25.  Extended SQL tracing
 Generates one trace file per Parallel Worker process
in database server trace directory
 For cross-instance Parallel Execution this means files
spread across more than one server
 Each Parallel Worker trace file lists, among others,
the number of rows produced per plan line and
elapsed time

26.  Extended SQL tracing
Allows detecting data distribution skew
Usually requires to reproduce the issue
Tedious to collect and skim through dozens of trace
files (no tool known that automates that job)

27.  DBMS_XPLAN.DISPLAY_CURSOR +
Rowsource statistics
 Based on same internal code instrumentation as
extended SQL trace
 Feeds back rowsource statistics (rows produced on
execution plan line level, timing, I/O stats) into
Library Cache
 Very useful for analyzing serial executions

28.  DBMS_XPLAN.DISPLAY_CURSOR +
Rowsource statistics
Not enabled by default, need to reproduce
Doesn’t cope with well multiple Parallel Execution
Servers / more complex parallel plans or Cross
Instance RAC execution
No information per Parallel Execution Server
=> Therefore not able to detect distribution skew

29.  Analyzing Data Distribution Skew
 Oracle since a long time offers a special view called
V$PQ_TQSTAT
 It is populated after a successful Parallel Execution
in the session of the Query Coordinator
 It lists the amount of data send via the “Table
Queues” / “Virtual Tables”
 In theory this allows exact analysis which operations
caused an uneven data distribution

30.  V$PQ_TQSTAT skewed execution example
DFO_NUMBER TQ_ID SERVER_TYP NUM_ROWS BYTES PROCESS
---------- ---------- ---------- ---------- ---------- ----------
1 0 Producer 500807 54396692 P008
1 0 Producer 499193 54259853 P009
1 0 Consumer 500038 54332349 P010
1 0 Consumer 499962 54324196 P011
1 1 Producer 499826 57267724 P008
1 1 Producer 500174 57357253 P009
1 1 Consumer 499933 57304789 P010
1 1 Consumer 500067 57320188 P011
1 2 Producer 462069 52963939 P010
1 2 Producer 537931 61681312 P011
1 2 Consumer 500212 57346574 P008
1 2 Consumer 499788 57298677 P009
1 3 Producer 500038 57337041 P010
1 3 Producer 499962 57328456 P011
1 3 Consumer 0 48 P008
1 3 Consumer 1000000 114665449 P009
1 4 Producer 0 24 P008
1 4 Producer 1000000 116668401 P009
1 4 Consumer 1000000 116668425 QC

31.  V$PQ_TQSTAT
Allows detecting data distribution skew
Can only be queried from inside session
Doesn’t cope with well more complex parallel plans
No information about relevance of skew

32. Easy to identify whether all workers are kept
busy all the time or not
Easy to identify if there was a problem with
work distribution
Shows actual parallel degree used (“Parallel
Downgrade”)
Supports RAC

33. Reports are not persisted and will be flushed
from memory quite quickly on busy systems
No easy identification and therefore no
systematic troubleshooting which plan
operations cause a work distribution problem
Lacks some precision regarding Parallel
Execution details

34.  Real-Time SQL Monitoring allows detecting
Parallel Execution skew in the following way:
 In the “Activity” tab
 The average active sessions will be less than the DOP of
the operation, allows to detect both temporal and data
distribution skew
 In the “Parallel” tab
 The DB Time recorded per Parallel Slave will show an
uneven distribution of active work time

35.  Real-Time SQL Monitoring “Activity” tab –
skewed execution example

36.  Real-Time SQL Monitoring “Parallel” tab –
skewed execution example

37.  One very useful approach is using Active
Session History (ASH)
 ASH samples active sessions once a second
 Activity of Parallel Workers over time can
easily be analyzed
 From 11g on the ASH data even contains a
reference to the execution plan line, so a
relation between Parallel Worker activity and
execution plan line based on ASH is possible

38.  Custom queries on ASH data required for
detailed analysis
 XPLAN_ASH tool runs these queries for a
given SQL_ID
 Advantage of ASH is the availability of
retained historic ASH data via AWR on disk
 Information can be extracted even for SQL
executions as long ago as the retention
configured for AWR

39.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

40. Fixing Work Distribution Skew
 Influence Parallel Distribution: Data volume
estimates, PQ_DISTRIBUTE / PQ_[NO]MAP /
PQ_SKEW (12c+) hint
 Limit impact by influencing join order
 Rewrite queries trying to limit or avoid skew
 Remap skewed data changing the value distribution

41. Fixing Work Distribution Skew
 Automatic join skew detection in 12c (PQ_SKEW)
 Supports only parallel HASH JOINs
 Supports at present only simple probe row sources
(no result of other joins supported)
 Histogram showing popular values required (or
forced via PQ_SKEW hint)

42. ----------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | SORT AGGREGATE | | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) |
| 4 | SORT AGGREGATE | | Q1,02 | PCWP | |
|* 5 | HASH JOIN | | Q1,02 | PCWP | |
| 6 | PX RECEIVE | | Q1,02 | PCWP | |
| 7 | PX SEND HYBRID HASH | :TQ10000 | Q1,00 | P->P | HYBRID HASH|
| 8 | STATISTICS COLLECTOR | | Q1,00 | PCWC | |
| 9 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
|* 10 | TABLE ACCESS FULL | T_1 | Q1,00 | PCWP | |
| 11 | PX RECEIVE | | Q1,02 | PCWP | |
| 12 | PX SEND HYBRID HASH (SKEW)| :TQ10001 | Q1,01 | P->P | HYBRID HASH|
| 13 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
|* 14 | TABLE ACCESS FULL | T_2 | Q1,01 | PCWP | |
----------------------------------------------------------------------------------

43. Fixing Work Distribution Skew
 More information:
 http://oracle-
randolf.blogspot.com/2014/08/parallel-execution-
skew-summary.html

44. Q & A