Oracle Parallel Execution, ein Feature der Enterprise Edition, erlaubt die automatische Verteilung einer SQL-Ausführung auf mehrere Prozesse/CPUs. Dies erfordert andere Herangehensweisen, wenn es um Analyse und Troubleshooting solcher Parallel-Ausführungen geht, insbesondere, da es bei Parallel-Ausführungen zu Problemen kommen kann, die es so bei normaler, serieller Ausführung überhaupt nicht gibt. Dieser Vortrag geht auf diese speziellen Probleme ein und zeigt auf, wie man sie analysieren kann.

1. www.informatik-aktuell.de

3.  Independent consultant
 Performance Troubleshooting
 In-house workshops
 Cost-Based Optimizer
 Performance By Design
 Oracle ACE Director
 Member of OakTable Network

4.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

5.  Oracle Database Enterprise Edition includes
the powerful Parallel Execution feature that
allows spreading the processing of a single
SQL statement execution across multiple
worker processes
 The feature is fully integrated into the Cost
Based Optimizer as well as the execution
runtime engine and automatically distributes
the work across the so called Parallel Workers

6.  Simple generic parallelization example
Task: Compute sum of 8 numbers
1+8=9, 9+7=16, 16+9=25,...
1+8+7+9+6+2+6+3= ???
n=8 numbers, 7 computation steps required
Serial execution: 7 time units

7. Simple generic parallelization example
4 workers
3+ time units
1 + 8
= 9
9 + 7
= 16
6 + 2
= 8
6 + 3
= 9
9 + 16
= 25
8 + 9
= 17
25 + 17
= 42
Coordinator

8. Parallel Execution doesn’t mean “work smarter”
You’re actually willing to accept to “work harder”
Could also be called: “Brute force” approach

9. So with Parallel Execution there
might be the problem that it
doesn’t work “hard enough”

10.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

11.  Parallel Execution can only reduce runtime as
expected if all workers are kept busy
 Possibly only a few or a single worker will be
active and have to do all the work
 In this case Parallel Execution can actually be
slower than serial execution
 There is a need to measure how busy the
workers are kept

12.  Note that this measure doesn’t tell you
anything about the efficiency of the actual
operation / execution plan
 But an otherwise efficient Parallel Execution
plan can only scale if the expected number of
workers is kept busy ideally all the time
 Note that it says “can scale” – if your system
cannot scale the required resources (like I/O)
you just end up with more workers waiting

13.  More reasons why Oracle Parallel Execution
might not reduce runtime as expected:
 Parallel DML/DDL gotchas
 “Downgrade” at execution time (less workers
assigned than expected)
 Overhead of Parallel Execution implementation
 Limitations of Parallel Execution implementation

14. Parallel DML / DDL gotchas
 DML / DDL part can run parallel or serial
 Query part can run parallel or serial

15. Parallel CTAS but serial query
-------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | Q1,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T4 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | Q1,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN| :TQ10000 | | S->P | RND-ROBIN |
|* 6 | HASH JOIN | | | | |
| 7 | TABLE ACCESS FULL| T2 | | | |
| 8 | TABLE ACCESS FULL| T2 | | | |
-------------------------------------------------------------------------

16. Serial CTAS but parallel query
--------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | LOAD AS SELECT | T4 | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) |
|* 4 | HASH JOIN BUFFERED | | Q1,02 | PCWP | |
| 5 | PX RECEIVE | | Q1,02 | PCWP | |
| 6 | PX SEND HASH | :TQ10000 | Q1,00 | P->P | HASH |
| 7 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 8 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
| 9 | PX RECEIVE | | Q1,02 | PCWP | |
| 10 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH |
| 11 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
| 12 | TABLE ACCESS FULL| T2 | Q1,01 | PCWP | |
--------------------------------------------------------------------------

17.  Other reasons why Oracle Parallel Execution
might not scale as expected:
 Parallel DML/DDL gotchas
 “Downgrade” at execution time (less workers
assigned than expected)
 Overhead of Parallel Execution implementation
 Limitations of Parallel Execution implementation

18. “Parallel Forced Serial” Example
------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | PX COORDINATOR FORCED SERIAL| | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10003 | Q1,03 | P->S | QC (RAND) |
| 3 | HASH UNIQUE | | Q1,03 | PCWP | |
| 4 | PX RECEIVE | | Q1,03 | PCWP | |
| 5 | PX SEND HASH | :TQ10002 | Q1,02 | P->P | HASH |
|* 6 | HASH JOIN BUFFERED | | Q1,02 | PCWP | |
| 7 | PX RECEIVE | | Q1,02 | PCWP | |
| 8 | PX SEND HASH | :TQ10000 | Q1,00 | P->P | HASH |
| 9 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 10 | TABLE ACCESS FULL | T2 | Q1,00 | PCWP | |
| 11 | PX RECEIVE | | Q1,02 | PCWP | |
| 12 | PX SEND HASH | :TQ10001 | Q1,01 | P->P | HASH |
| 13 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
| 14 | TABLE ACCESS FULL | T2 | Q1,01 | PCWP | |
------------------------------------------------------------------------------

19.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

20.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work

21. select median(id) from t2;
-----------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | SORT GROUP BY | | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM)| :TQ10000 | Q1,00 | P->S | QC (RAND) |
| 4 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 5 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
-----------------------------------------------------------------------

22. create table t3 parallel
as
select * from t2
where rownum <= 10000000;
-----------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ20001 | Q2,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T3 | Q2,01 | PCWP | |
| 4 | PX RECEIVE | | Q2,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN | :TQ20000 | | S->P | RND-ROBIN |
|* 6 | COUNT STOPKEY | | | | |
| 7 | PX COORDINATOR | | | | |
| 8 | PX SEND QC (RANDOM) | :TQ10000 | Q1,00 | P->S | QC (RAND) |
|* 9 | COUNT STOPKEY | | Q1,00 | PCWC | |
| 10 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 11 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
-----------------------------------------------------------------------------

23. create table t3 parallel
as select * from (select a.*,
lag(filler, 1) over (order by id) as prev_filler
from t2 a);
--------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | | | |
| 1 | PX COORDINATOR | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ20001 | Q2,01 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | T3 | Q2,01 | PCWP | |
| 4 | PX RECEIVE | | Q2,01 | PCWP | |
| 5 | PX SEND ROUND-ROBIN | :TQ20000 | | S->P | RND-ROBIN |
| 6 | VIEW | | | | |
| 7 | WINDOW BUFFER | | | | |
| 8 | PX COORDINATOR | | | | |
| 9 | PX SEND QC (ORDER) | :TQ10001 | Q1,01 | P->S | QC (ORDER) |
| 10 | SORT ORDER BY | | Q1,01 | PCWP | |
| 11 | PX RECEIVE | | Q1,01 | PCWP | |
| 12 | PX SEND RANGE | :TQ10000 | Q1,00 | P->P | RANGE |
| 13 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
| 14 | TABLE ACCESS FULL| T2 | Q1,00 | PCWP | |
--------------------------------------------------------------------------------

24. All these examples have one thing in common:
If the Query Coordinator (non-parallel part)
needs to perform a significant part of the overall
work, Parallel Execution won’t reduce the
runtime as expected

25.  Two major challenges
 Can the given task be divided into sub-tasks that can
efficiently and independently be processed by the
workers? (“Parallel Unfriendly”)
 Can all assigned workers be kept busy all the time?

28.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

29.  Extended SQL tracing
 Generates one trace file per Parallel Worker process
in database server trace directory
 For cross-instance Parallel Execution this means files
spread across more than one server
 Each Parallel Worker trace file lists, among others,
the number of rows produced per plan line

30.  Extended SQL tracing
Allows detecting data distribution skew
Usually requires to reproduce the issue
Tedious to collect and skim through dozens of trace
files (no tool known that automates that job)
No information about relevance of skew

31.  DBMS_XPLAN.DISPLAY_CURSOR +
Rowsource statistics
 Based on same internal code instrumentation as
extended SQL trace
 Feeds back rowsource statistics (rows produced on
execution plan line level, timing, I/O stats) into
Library Cache
 Useful for analyzing serial executions

32.  DBMS_XPLAN.DISPLAY_CURSOR +
Rowsource statistics
Doesn’t cope with well multiple Parallel Execution
Servers / multiple DFO trees or Cross Instance RAC
execution
No information about rows produced per Parallel
Execution Server
=> Therefore not able to detect distribution skew

33.  Analyzing Data Distribution Skew
 Oracle since a long time offers a special view called
V$PQ_TQSTAT
 It is populated after a successful Parallel Execution
in the session of the Query Coordinator
 It lists the amount of data send via the “Table
Queues” / “Virtual Tables”
 In theory this allows exact analysis which operations
caused an uneven data distribution

34.  V$PQ_TQSTAT skewed execution example
DFO_NUMBER TQ_ID SERVER_TYP NUM_ROWS BYTES PROCESS
---------- ---------- ---------- ---------- ---------- ----------
1 0 Producer 500807 54396692 P008
1 0 Producer 499193 54259853 P009
1 0 Consumer 500038 54332349 P010
1 0 Consumer 499962 54324196 P011
1 1 Producer 499826 57267724 P008
1 1 Producer 500174 57357253 P009
1 1 Consumer 499933 57304789 P010
1 1 Consumer 500067 57320188 P011
1 2 Producer 462069 52963939 P010
1 2 Producer 537931 61681312 P011
1 2 Consumer 500212 57346574 P008
1 2 Consumer 499788 57298677 P009
1 3 Producer 500038 57337041 P010
1 3 Producer 499962 57328456 P011
1 3 Consumer 0 48 P008
1 3 Consumer 1000000 114665449 P009
1 4 Producer 0 24 P008
1 4 Producer 1000000 116668401 P009
1 4 Consumer 1000000 116668425 QC

35. Easy to identify whether all workers are kept
busy all the time or not
Easy to identify if there was a problem with
work distribution
Shows actual parallel degree used (“Parallel
Downgrade”)
Supports RAC

36. Reports are not persisted and will be flushed
from memory quite quickly on busy systems
No easy identification and therefore no
systematic troubleshooting which plan
operations cause a work distribution problem
Lacks some precision regarding Parallel
Execution details

37.  Real-Time SQL Monitoring allows detecting
Parallel Execution skew in the following way:
 In the “Activity” tab
 The average active sessions will be less than the DOP of
the operation, allows to detect both temporal and data
distribution skew
 In the “Parallel” tab
 The DB Time recorded per Parallel Slave will show an
uneven distribution of active work time

38.  Real-Time SQL Monitoring “Activity” tab –
skewed execution example

39.  Real-Time SQL Monitoring “Parallel” tab –
skewed execution example

40.  One very useful approach is using Active
Session History (ASH)
 ASH samples active sessions once a second
 Activity of Parallel Workers over time can
easily be analyzed
 From 11g on the ASH data even contains a
reference to the execution plan line, so a
relation between Parallel Worker activity and
execution plan line based on ASH is possible

41.  Custom queries on ASH data required for
detailed analysis
 XPLAN_ASH tool runs these queries for a
given SQL_ID
 Advantage of ASH is the availability of
retained historic ASH data via AWR on disk
 Information can be extracted even for SQL
executions as long ago as the retention
configured for AWR

42.  Parallel Execution introduction
 Major challenges
 Parallel unfriendly examples
 Distribution skew examples
 How to measure distribution of work
 Fixing work distribution skew

43. Fixing Work Distribution Skew
 Influence Parallel Distribution: Data volume
estimates, PQ_DISTRIBUTE / PQ_[NO]MAP /
PQ_SKEW (12c+) hint
 Limit impact by influencing join order
 Rewrite queries trying to limit or avoid skew
 Remap skewed data changing the value distribution

44. Fixing Work Distribution Skew
 Automatic join skew detection in 12c (PQ_SKEW)
 Supports only parallel HASH JOINs
 Supports at present only simple probe row sources
(no result of other joins supported)
 Histogram showing popular values required (or
forced via PQ_SKEW hint)

45. ----------------------------------------------------------------------------------
| Id | Operation | Name | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | |
| 1 | SORT AGGREGATE | | | | |
| 2 | PX COORDINATOR | | | | |
| 3 | PX SEND QC (RANDOM) | :TQ10002 | Q1,02 | P->S | QC (RAND) |
| 4 | SORT AGGREGATE | | Q1,02 | PCWP | |
|* 5 | HASH JOIN | | Q1,02 | PCWP | |
| 6 | PX RECEIVE | | Q1,02 | PCWP | |
| 7 | PX SEND HYBRID HASH | :TQ10000 | Q1,00 | P->P | HYBRID HASH|
| 8 | STATISTICS COLLECTOR | | Q1,00 | PCWC | |
| 9 | PX BLOCK ITERATOR | | Q1,00 | PCWC | |
|* 10 | TABLE ACCESS FULL | T_1 | Q1,00 | PCWP | |
| 11 | PX RECEIVE | | Q1,02 | PCWP | |
| 12 | PX SEND HYBRID HASH (SKEW)| :TQ10001 | Q1,01 | P->P | HYBRID HASH|
| 13 | PX BLOCK ITERATOR | | Q1,01 | PCWC | |
|* 14 | TABLE ACCESS FULL | T_2 | Q1,01 | PCWP | |
----------------------------------------------------------------------------------

46. Fixing Work Distribution Skew
 More information:
 http://oracle-
randolf.blogspot.com/2014/08/parallel-execution-
skew-summary.html

47. Q & A