Oracle Data Warehouses are typically deployed on servers with very large number of cores, and increasingly on RAC. Making efficient use of all available cores when processing data warehouse workloads is therefore critical in achieving maximal performance. To make efficient use of all cores in a data warehouse system, skilled use of parallel queries is key.

1. Maximize Data Warehouse
Performance with
Parallel Queries
Presented by: Christo Kutrovsky

2. © 2015 Pythian – Presentation for DOAG
Who Am I
2
• Oracle ACE
• 15 years in Oracle field
• Joined Pythian 2003
• Part of Pythian Consulting Group
• Special projects
• Performance tuning
• Critical services
• Presenter at: IOUG, RMOUG, UKOUG, OpenWorld
“oracle pinup”

3. © 2015 Pythian – Presentation for DOAG3 © 2011 Pythian - Confidential
Why Companies Trust Pythian
• Recognized Leader:
• Global industry-leader in remote database administration services and consulting for
Oracle, Oracle Applications, MySQL and SQL Server
• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion
and Western Union to help manage their complex IT deployments
• Expertise:
• One of the world’s largest concentrations of dedicated, full-time DBA expertise.
Employ 7 Oracle ACEs/ACE Directors.
• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle
Exadata, Oracle GoldenGate & Oracle RAC.
• Global Reach & Scalability:
• 24/7/365 global remote support for DBA and consulting, systems administration,
special projects or emergency response

4. © 2015 Pythian – Presentation for DOAG
Agenda
4
• Review Parallel Query concepts
• Review Exadata Data Flow capabilities
• The Big Sort
• The Big Group By
• The Big Join
• Final Thoughts

5. 5
Parallel Query Concepts

6. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
6
• Producer/Consumer model
• A PQ slave is either a producer or consumer, never both
Figure 8-2 Inter-operation Parallelism and Dynamic Partitioning
Oracle® Database VLDB and Partitioning Guide

7. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
7
• Producer/Consumer model
• A PQ slave is either a producer or consumer, never both
Figure 8-2 Inter-operation Parallelism and Dynamic Partitioning
Oracle® Database VLDB and Partitioning Guide

8. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
8
• A PQ slave is either a producer or consumer in relation
to other PQ slaves
PQ1
PQ2
PQ3
PQ4
PQ5
PQ6
PQ7
PQ8
S
G
A
DB

9. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
9
• But logically, step-wise, when there are two sets it’s
flow-through
PQ1
PQ2
PQ3
PQ4
PQ5
PQ6
PQ7
PQ8
S
G
A
DB
ONE STEP (1)

10. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
10
• QC is always flow-throught
PQ1
PQ2
PQ3
PQ4
PQ5
PQ6
PQ7
PQ8
S
G
A
DB
S
G
A
QC
SQLNET
CLIENTONE STEP (2)

11. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
11
• select count(*) from table;
PQ1
PQ2
PQ3
PQ4
S
G
A
DB
ONE STEP
QC
SQLNET
CLIENT

12. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
12
• Producer/consumer model
• A slave is either producer and consumer in
relation to other slaves.

13. 13
Exadata Data Flow

14. © 2015 Pythian – Presentation for DOAG
Multiple Tiers
14
• Every database has multiple tiers
• Memory
• SAN/NFS – network
• SAN Performance
• CPU Capacity
• Client side connectivity
• Each tier has it’s own nuances, properties and
bandwidth limitations

15. © 2015 Pythian – Presentation for DOAG
Exadata X2-2 Data Flow (intra)
15
• Each DB node (compute) and CELL node (storage)
• Single dual port 40 Gbit infiniband (QDR)
• 3,200 Mb/sec (full-duplex)
• Each CELL node
• Single dual port 40 Gbit infiniband (QDR)
• 3,200 Mb/sec (full-duplex)
• 12 x 4 TB SAS drives (7200 rpm)
• Total bandwidth: 1,000 - 2,000 Mb/sec
• 4 x Sun Flash Arrays
• Total bandwidth: 4,100 Mb/sec

16. © 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
16
• SQL Net Clients talk only to DB nodes
• Each DB node
• 4 x 10 gigabit network ( 4 x 1.08 Gbytes/sec)
• 2 x 10 Gigabit network (opt) ( 2 x 1.08 Gbytes/sec)
• Alternatively, can use IP over infiniband
• Exalogic
• RMAN backups
• Etc.

17. © 2015 Pythian – Presentation for DOAG
Exadata Data Flow (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET

18. © 2015 Pythian – Presentation for DOAG
DISK: 2 GB/sec + FLASH: 4 GB/sec
= 6 GB/sec
X 7 cells = 42 GB/sec
Single Cell (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
CELL 01

19. © 2015 Pythian – Presentation for DOAG
Cell output (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
CELL to SWITCH: 3.2 GB/sec
X 7 cells = 22.4 GB/sec

20. © 2015 Pythian – Presentation for DOAG
DB Node Consumption (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
SWITCH to DB NODE: 3.2 GB/sec
X 4 nodes = 12.8 GB/sec

21. © 2015 Pythian – Presentation for DOAG
Memory speed (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
DB to SGA (memory) 22 GB/sec
(very CPU dependant)
X 4 DB = 88 GB/sec

22. © 2015 Pythian – Presentation for DOAG
DB Node to Client (Half Rack)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
DB to SINGLE SQLNET Client
112 MB/sec (1 GBIT)
1.08 GB/sec (10 GBIT)

23. © 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
23
• Ethernet link aggregation requires
• Single conversation runs on a single path
• Ethernet standard
• One database sessions runs on single TCP
connection, which must run on single path
• Switches and network cards can implement different
hashing algorithms to determine paths
• Most are on IP level. Between 2 IPs the same NICs
are used, regardless of number of connections

24. © 2015 Pythian – Presentation for DOAG
Exadata Data Flow (Half Rack)
DB4 eth0
DB to THIS Client: 112 MB/sec (theoretical)
(18 MB/sec per session)
IP: 10.31.33.1
Session
Session
Session
Session
Session
Session
eth1
eth2
eth3
eth0
eth1
eth2
eth3
S
w
i
t
c
h

25. © 2015 Pythian – Presentation for DOAG
Exadata Data Flow (External)
25
• As per Exadata Documentation:
• Data Load Rate: Up to 10.5 TB/hour
• 764 MB/sec per DB node
• 3 GB/sec – can your other infrastructure provide
such performance?

26. 26
The Big Sort

27. © 2015 Pythian – Presentation for DOAG
Sample Data
27
• My favorite table – CKK
• 39 GiB of data
• 10,000,000 rows
• id, mod5_id, mod5000_id, dt_dec, dt_pos etc.
create table ckk /*parallel 8 */nologging tablespace ckbig as
select rownum id, mod(rownum,5) mod5_id, mod(rownum,5000)
mod5000_id, sysdate dt_fixed, sysdate - rownum/24/60 dt_dec, sysdate
+ rownum/24/60 dt_pos, sysdate + ora_hash(rownum,65,535)/24 dt_rand,
sysdate+mod(rownum,10) dt_mod10, rpad('x',3500,'x') filler
from (select rownum r from dual connect by level <= 10000) r1,
(select rownum r from dual connect by level <= 1000)
;

28. © 2015 Pythian – Presentation for DOAG
Parallel Sort
28
select /*+ parallel(4) */ * from ckk c1 order by id;
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 33G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) |
| 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE |
| 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
PQ1
PQ2
PQ3
PQ4
PQ5 – up ID=2.5 M
PQ6 – up ID=5.0 M
PQ7 – up ID=7.5 M
PQ8 – up ID=10 M
S
G
A
DB

29. © 2015 Pythian – Presentation for DOAG
STORAGE FULL (OFFLOADED)
DB1
INFINIBAND SWITCHES
DB2 DB3 DB4
CELL 01 CELL 02 CELL 03 CELL 04 CELL 05 CELL 06 CELL 07
SGA SGA SGA SGA
CLIENT
RESULT
SET
Decompression, Projection and Filter
Happens on EVERY CELL NODE
regardless of query parallelism.
MPP mode.

30. © 2015 Pythian – Presentation for DOAG
Parallel Sort
30
select dfo_number "d", tq_id as "t", server_type, num_rows,rpad('x',round(num_rows*10/nullif(max(num_rows) over (partition by
dfo_number, tq_id, server_type),0)),'x') as "pr", round(bytes/1024/1024) mb, process, instance i,round(ratio_to_report (num_rows)
over (partition by dfo_number, tq_id, server_type)*100) as "%", open_time, avg_latency, waits,
timeouts,round(bytes/nullif(num_rows,0)) as "b/r” from v$pq_tqstat order by dfo_number, tq_id, server_type desc, process;;
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 372 xxxxxxxxxx 2 QC 1 100
1 0 Producer 2,692,340 xxxxxxxxxx 9,161 P002 1 27
1 0 Producer 2,499,910 xxxxxxxxx 8,506 P002 2 25
1 0 Producer 2,692,340 xxxxxxxxxx 9,161 P003 2 27
1 0 Producer 2,115,410 xxxxxxxx 7,198 P003 1 21
1 0 Consumer 6,146,614 xxxxxxxxxx 20,915 P000 2 61
1 0 Consumer 781,680 x 2,660 P000 1 8
1 0 Consumer 1,968,128 xxx 6,697 P001 2 20
1 0 Consumer 1,103,578 xx 3,755 P001 1 11
1 1 Producer 13 x 0 P000 1 5
1 1 Producer 212 xxxxxxxxxx 1 P000 2 84
1 1 Producer 13 x 0 P001 2 5
1 1 Producer 13 x 0 P001 1 5
1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100

31. © 2015 Pythian – Presentation for DOAG
Parallel Sort
31
select /*+ parallel(4) */ * from ckk c1 order by id;
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 33G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (ORDER) | :TQ10001 | 10M| 33G| | Q1,01 | P->S | QC (ORDER) |
| 3 | SORT ORDER BY | | 10M| 33G| 38G| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 5 | PX SEND RANGE | :TQ10000 | 10M| 33G| | Q1,00 | P->P | RANGE |
| 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
PQ1
PQ2
PQ3
PQ4
PQ5 – up to ID=6.1 M (6.1 M)
PQ6 – up to ID=6.9 M (0.8 M)
PQ7 – up to ID=8.8 M (1.9 M)
PQ8 – up to ID=10.0 M (1.1M
S
G
A
DB

32. © 2015 Pythian – Presentation for DOAG
The Ranger
32
• The Ranger Bug
• The RANGER uses a sample from the data to
determine the boundaries for PQ sort operations
• Sometimes it’s off by a lot
• It will ALWAYS miss when the input data is already
somewhat sorted
• Limit your scalability – no matter how big your machine
is

33. © 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
33
select /*+ parallel(4) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos,
dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand;
--90 sec
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 372 xxxxxxxxxx 1 QC 1 100
1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 1 25
1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P002 2 25
1 0 Producer 2,499,910 xxxxxxxxxx 4,925 P003 1 25
1 0 Producer 2,500,030 xxxxxxxxxx 4,925 P003 2 25
1 0 Consumer 2,424,554 xxxxxxxxx 4,776 P000 2 24
1 0 Consumer 2,576,164 xxxxxxxxx 5,075 P000 1 26
1 0 Consumer 2,272,136 xxxxxxxx 4,476 P001 2 23
1 0 Consumer 2,727,146 xxxxxxxxxx 5,373 P001 1 27
1 1 Producer 218 xxxxxxxxxx 0 P000 1 76
1 1 Producer 23 x 0 P000 2 8
1 1 Producer 23 x 0 P001 2 8
1 1 Producer 23 x 0 P001 1 8
1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

34. © 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
34
select /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos,
dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by dt_rand;
--10 sec
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 100
1 0 Producer 312,520 xxxxxxxxxx 616 P016 1 3
1 0 Producer 312,520 xxxxxxxxxx 616 P016 2 3
1 0 Producer 312,520 xxxxxxxxxx 616 P017 1 3
1 0 Producer 312,520 xxxxxxxxxx 616 P017 2 3
...
1 0 Consumer 151,262 xxx 298 P003 1 2
1 0 Consumer 454,696 xxxxxxxxxx 896 P003 2 5
1 0 Consumer 302,320 xxxxxxx 596 P004 1 3
1 0 Consumer 151,731 xxx 299 P004 2 2
1 0 Consumer 303,429 xxxxxxx 598 P005 1 3
1 0 Consumer 454,650 xxxxxxxxxx 896 P005 2 5
1 0 Consumer 303,259 xxxxxxx 597 P006 2 3
1 0 Consumer 455,172 xxxxxxxxxx 897 P006 1 5
1 0 Consumer 302,945 xxxxxxx 597 P012 1 3
1 0 Consumer 303,318 xxxxxxx 598 P012 2 3
1 0 Consumer 303,245 xxxxxxx 597 P013 1 3
1 0 Consumer 304,029 xxxxxxx 599 P013 2 3
1 0 Consumer 301,937 xxxxxxx 595 P014 2 3
1 0 Consumer 454,903 xxxxxxxxxx 896 P014 1 5
1 0 Consumer 151,887 xxx 299 P015 1 2
1 1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

35. © 2015 Pythian – Presentation for DOAG
Parallel Sort – random data
35
select /*+ parallel(32) */ id, mod5_id, mod5000_id, dt_fixed, dt_dec, dt_pos,
dt_rand, dt_mod10, substr(filler,1,2000) from ckk c1 order by id;
--20 sec
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 3,872 xxxxxxxxxx 11 QC 1 100
1 0 Producer 240,400 xxxxxx 474 P016 2 2
1 0 Producer 384,640 xxxxxxxxx 758 P016 1 4
1 0 Producer 216,360 xxxxx 426 P017 1 2
1 0 Producer 192,320 xxxx 379 P017 2 2
...
1 0 Consumer 373,702 xxxx 736 P000 1 4
1 0 Consumer 372,530 xxxx 734 P000 2 4
1 0 Consumer 124,801 x 246 P001 2 1
1 0 Consumer 183,552 xx 361 P001 1 2
1 0 Consumer 734,398 xxxxxxx 1,446 P002 1 7
1 0 Consumer 22,402 44 P002 2 0
1 0 Consumer 191,810 xx 378 P003 1 2
1 0 Consumer 132,990 x 262 P003 2 1
1 0 Consumer 32,640 64 P004 1 0
1 0 Consumer 920,448 xxxxxxxxx 1,813 P004 2 9
1 0 Consumer 167,232 xx 329 P005 1 2
1 0 Consumer 110,466 x 218 P005 2 1
1 0 Consumer 1,055,669 xxxxxxxxxx 2,080 P014 2 11
1 0 Consumer 130,944 x 258 P014 1 1
1 0 Consumer 60,250 x 119 P015 2 1
1 0 Consumer 20,298 40 P015 1 0
...
1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

36. © 2015 Pythian – Presentation for DOAG
The Ranger issue
36
• Distribution can be much worse
• When using analytics or other “input” into the sort
operation that is pre-sorted
• Often 90% goes to one slave, 10% to remaining
• “Been worked on” – I am told
• Problem made even worse due to lack of large buffers

37. © 2015 Pythian – Presentation for DOAG
Writing to disk – lack of buffering
37
• When a slave is writing to disk – it cannot receive any
messages or can receive a very small number of
messages
• Results in slowdowns when Oracle has to write to TEMP
• Massive slowdown when only a few slaves have to
write to TEMP
• All producers have to wait
• Significant slowdown when all slaves have to write
to TEMP
• Cannot parallelise sufficiently to leverage
available disk bandwidth

38. 38
The Big Group By

39. © 2015 Pythian – Presentation for DOAG
Group By
39
select /*+ parallel(4) */ mod5_id, count(*) from ckk group by mod5_id;
---------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 5 | 15 | | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) |
| 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 5 | 15 | Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 5 | 15 | Q1,00 | P->P | HASH |
| 6 | HASH GROUP BY | | 5 | 15 | Q1,00 | PCWP | |
| 7 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | |
| 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | |
---------------------------------------------------------------------------------------------------
select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk group by mod5_id;
--------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 5 | 15 | | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 5 | 15 | Q1,01 | P->S | QC (RAND) |
| 3 | HASH GROUP BY | | 5 | 15 | Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 28M| Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 10M| 28M| Q1,00 | P->P | HASH |
| 6 | PX BLOCK ITERATOR | | 10M| 28M| Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 28M| Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------

40. © 2015 Pythian – Presentation for DOAG
Oracle Parallel Query
40
• But logically, step-wise, when there are two sets it’s
flow-through
PQ1
PQ2
PQ3
PQ4
S
G
A
DB
STEP (1)
GROUP
PGA
GROUP
PGA
GROUP
PGA
GROUP
PGA
PQ5
PQ6
PQ7
PQ8
GROUP
PGA
GROUP
PGA
GROUP
PGA
GROUP
PGA
S
G
A
QCSTEP (2)

41. © 2015 Pythian – Presentation for DOAG
Group by – small result set
41
select /*+ parallel(4) */ mod5_id, count(*) from ckk
group by mod5_id;
-- 4.8 sec
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 5 xxxxxxxxxx 0 P002 2 25
1 0 Producer 5 xxxxxxxxxx 0 P002 1 25
1 0 Producer 5 xxxxxxxxxx 0 P003 2 25
1 0 Producer 5 xxxxxxxxxx 0 P003 1 25
1 0 Consumer 4 xxxxx 0 P000 2 20
1 0 Consumer 8 xxxxxxxxxx 0 P000 1 40
1 0 Consumer 4 xxxxx 0 P001 2 20
1 0 Consumer 4 xxxxx 0 P001 1 20
1 1 Producer 2 xxxxxxxxxx 0 P000 1 40
1 1 Producer 1 xxxxx 0 P000 2 20
1 1 Producer 1 xxxxx 0 P001 1 20
1 1 Producer 1 xxxxx 0 P001 2 20
1 1 Consumer 5 xxxxxxxxxx 0 QC 1 100

42. © 2015 Pythian – Presentation for DOAG
Group by – small result set
42
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 2,692,340 xxxxxxxxxx 10 P004 2 27
1 0 Producer 2,307,600 xxxxxxxxx 8 P004 1 23
1 0 Producer 2,500,030 xxxxxxxxx 9 P005 2 25
1 0 Producer 2,500,030 xxxxxxxxx 9 P005 1 25
1 0 Consumer 4,000,000 xxxxxxxxxx 15 P002 2 40
1 0 Consumer 2,000,000 xxxxx 8 P002 1 20
1 0 Consumer 2,000,000 xxxxx 6 P003 2 20
1 0 Consumer 2,000,000 xxxxx 8 P003 1 20
1 1 Producer 1 xxxxx 0 P002 1 20
1 1 Producer 2 xxxxxxxxxx 0 P002 2 40
1 1 Producer 1 xxxxx 0 P003 1 20
1 1 Producer 1 xxxxx 0 P003 2 20
1 1 Consumer 5 xxxxxxxxxx 0 QC 1 100
select /*+ parallel(4) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk
group by mod5_id;
-- 4.8 sec

43. © 2015 Pythian – Presentation for DOAG
Group by – small result set
43
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 264,440 xxxxxx 1 P016 2 3
1 0 Producer 312,520 xxxxxxxx 1 P016 1 3
1 0 Producer 240,400 xxxxxx 1 P017 1 2
1 0 Producer 360,600 xxxxxxxxx 1 P017 2 4
...
1 0 Consumer 0 0 P000 2 0
1 0 Consumer 2,000,000 xxxxxxxxxx 8 P000 1 20
1 0 Consumer 0 0 P001 2 0
...
1 0 Consumer 0 0 P004 1 0
1 0 Consumer 0 0 P005 2 0
1 0 Consumer 2,000,000 xxxxxxxxxx 6 P005 1 20
1 0 Consumer 0 0 P006 2 0
1 0 Consumer 0 0 P006 1 0
1 0 Consumer 0 0 P007 2 0
1 0 Consumer 2,000,000 xxxxxxxxxx 8 P007 1 20
1 0 Consumer 2,000,000 xxxxxxxxxx 8 P008 1 20
1 0 Consumer 2,000,000 xxxxxxxxxx 8 P010 2 20
...
select /*+ parallel(32) NO_GBY_PUSHDOWN*/ mod5_id, count(*) from ckk
group by mod5_id;
-- 4.8 sec

44. © 2015 Pythian – Presentation for DOAG
Group by – small result set
44
• When the result set from a group by is small – there are
a lot of optimisations that can kick in
• PUSHDOWN saves on PQ traffic
• Receiving end has very little work to do
• Very efficient to parallelise

45. © 2015 Pythian – Presentation for DOAG
select /*+ parallel(4) */ id, count(*) from ckk
group by id;
-- 4.8 sec
Group by – large result set HASH
45
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 2,500,030 xxxxxxxx 17 P002 2 25
1 0 Producer 2,115,410 xxxxxx 14 P002 1 21
1 0 Producer 2,115,410 xxxxxx 14 P003 2 21
1 0 Producer 3,269,150 xxxxxxxxxx 21 P003 1 33
1 0 Consumer 2,499,213 xxxxxxxxxx 16 P000 1 25
1 0 Consumer 2,499,807 xxxxxxxxxx 16 P000 2 25
1 0 Consumer 2,500,981 xxxxxxxxxx 16 P001 1 25
1 0 Consumer 2,499,999 xxxxxxxxxx 16 P001 2 25
1 1 Producer 1,465 xxxxx 0 P000 2 17
1 1 Producer 2,929 xxxxxxxxxx 0 P000 1 33
1 1 Producer 2,928 xxxxxxxxxx 0 P001 1 33
1 1 Producer 1,464 xxxxx 0 P001 2 17
1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

46. © 2015 Pythian – Presentation for DOAG
select /*+ parallel(4) */ id, count(*) from ckk
group by id order by id;
-- 6.5 sec
Group by – large result set SORT
46
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 372 xxxxxxxxxx 0 QC 1 100
1 0 Producer 2,499,910 xxxxxxxxx 17 P002 1 25
1 0 Producer 2,692,340 xxxxxxxxxx 18 P002 2 27
1 0 Producer 2,500,030 xxxxxxxxx 17 P003 2 25
1 0 Producer 2,307,720 xxxxxxxxx 15 P003 1 23
1 0 Consumer 6,154,678 xxxxxxxxxx 40 P000 2 62
1 0 Consumer 793,712 x 5 P000 1 8
1 0 Consumer 1,095,514 xx 7 P001 1 11
1 0 Consumer 1,956,096 xxx 13 P001 2 20
1 1 Producer 4,391 xxxxxxxx 0 P000 1 24
1 1 Producer 5,379 xxxxxxxxxx 0 P000 2 29
1 1 Producer 4,391 xxxxxxxx 0 P001 2 24
1 1 Producer 4,391 xxxxxxxx 0 P001 1 24
1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

47. © 2015 Pythian – Presentation for DOAG
select /*+ parallel(32) */ id, count(*) from ckk
group by id;
-- 4.8 sec
Group by – large result set HASH
47
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 360,600 xxxxxxxxx 2 P016 1 4
1 0 Producer 240,400 xxxxxx 2 P016 2 2
1 0 Producer 312,520 xxxxxxxx 2 P017 1 3
1 0 Producer 336,560 xxxxxxxxx 2 P017 2 3
...
1 0 Consumer 312,829 xxxxxxxxxx 2 P000 2 3
1 0 Consumer 311,690 xxxxxxxxxx 2 P000 1 3
1 0 Consumer 312,371 xxxxxxxxxx 2 P001 1 3
1 0 Consumer 312,854 xxxxxxxxxx 2 P001 2 3
1 0 Consumer 312,255 xxxxxxxxxx 2 P002 2 3
1 0 Consumer 312,321 xxxxxxxxxx 2 P002 1 3
1 0 Consumer 313,241 xxxxxxxxxx 2 P003 2 3
1 0 Consumer 312,605 xxxxxxxxxx 2 P003 1 3
1 0 Consumer 312,202 xxxxxxxxxx 2 P004 1 3
1 0 Consumer 311,294 xxxxxxxxxx 2 P004 2 3
1 0 Consumer 312,338 xxxxxxxxxx 2 P005 1 3
1 0 Consumer 312,206 xxxxxxxxxx 2 P005 2 3
1 0 Consumer 312,898 xxxxxxxxxx 2 P006 1 3
1 0 Consumer 312,817 xxxxxxxxxx 2 P006 2 3
1 0 Consumer 312,291 xxxxxxxxxx 2 P007 1 3
1 0 Consumer 312,872 xxxxxxxxxx 2 P007 2 3
...
1 1 Consumer 200 xxxxxxxxxx 0 QC 1 100

48. © 2015 Pythian – Presentation for DOAG
Group by – large result set HASH
48
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Producer 312,520 xxxxxxxxxx 1,047 P016 1 3
1 0 Producer 312,520 xxxxxxxxxx 1,047 P016 2 3
1 0 Producer 312,520 xxxxxxxxxx 1,047 P017 1 3
...
1 0 Consumer 311,603 xxxxxxxxxx 1,044 P000 1 3
1 0 Consumer 312,470 xxxxxxxxxx 1,047 P000 2 3
1 0 Consumer 312,484 xxxxxxxxxx 1,047 P001 2 3
1 0 Consumer 312,662 xxxxxxxxxx 1,048 P001 1 3
1 0 Consumer 312,341 xxxxxxxxxx 1,047 P002 2 3
1 0 Consumer 312,692 xxxxxxxxxx 1,048 P002 1 3
1 0 Consumer 313,114 xxxxxxxxxx 1,049 P003 2 3
1 0 Consumer 311,889 xxxxxxxxxx 1,045 P003 1 3
1 0 Consumer 312,323 xxxxxxxxxx 1,047 P004 1 3
1 0 Consumer 313,310 xxxxxxxxxx 1,050 P004 2 3
1 0 Consumer 312,715 xxxxxxxxxx 1,048 P005 2 3
1 0 Consumer 313,394 xxxxxxxxxx 1,050 P005 1 3
1 0 Consumer 312,718 xxxxxxxxxx 1,048 P006 1 3
1 0 Consumer 312,090 xxxxxxxxxx 1,046 P006 2 3
1 0 Consumer 312,720 xxxxxxxxxx 1,048 P007 1 3
1 0 Consumer 311,889 xxxxxxxxxx 1,045 P007 2 3
2 ...
1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100
select /*+ parallel(32) */ id, filler, count(*) from ckk
group by id,filler;
-- 18 sec

49. © 2015 Pythian – Presentation for DOAG
Group by – large result set SORT
49
d t SERVER_TYPE NUM_ROWS pr MB PROCESS I %
1 0 Ranger 3,872 xxxxxxxxxx 23 QC 1 100
1 0 Producer 288,480 xxxxxxx 967 P016 2 3
1 0 Producer 288,480 xxxxxxx 967 P016 1 3
1 0 Producer 336,560 xxxxxxxx 1,128 P017 2 3
1 0 Producer 240,400 xxxxxx 806 P017 1 2
...
1 0 Consumer 722,115 xxxxxx 2,420 P002 2 7
1 0 Consumer 16,386 55 P002 1 0
1 0 Consumer 210,112 xx 704 P003 2 2
1 0 Consumer 112,510 x 377 P003 1 1
1 0 Consumer 940,928 xxxxxxxx 3,153 P004 1 9
1 0 Consumer 34,688 116 P004 2 0
1 0 Consumer 171,392 xx 574 P005 2 2
1 0 Consumer 110,466 x 370 P005 1 1
1 0 Consumer 163,658 x 548 P014 2 2
1 0 Consumer 1,113,523 xxxxxxxxxx 3,732 P014 1 11
1 0 Consumer 11,100 37 P015 1 0
1 0 Consumer 4,150 14 P015 2 0
...
1 1 Consumer 200 xxxxxxxxxx 1 QC 1 100
select /*+ parallel(32) */ id, filler, count(*) from ckk
group by id,filler order by id,filler;
-- 41 sec

50. © 2015 Pythian – Presentation for DOAG
Group by – summary
50
• Theoretical performance is reachable
• SORT based operations have major issues
• Especially with sorted inputs, such as from analytics
• model clause can help – partition by is hash based
• Partial disk staging causes major slowdowns
• Balanced operation is critical

51. © 2015 Pythian – Presentation for DOAG
Group by – Partitioning
51
• What about partitioning?
create table ckk_p (id, mod5_id, mod5000_id, dt_fixed,
dt_dec, dt_pos, dt_rand, dt_mod10, filler)
partition by hash (id) partitions 32 tablespace ckbig
nologging parallel 4
as
select * from ckk;

52. © 2015 Pythian – Presentation for DOAG
Group by – with partitioning
52
select /*+ parallel(4) */ id,count(*) from ckk_p group by id,filler;
------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop | TQ |IN-OUT| PQ Distrib |
------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 7071K| 23G| | | | | | |
| 1 | PX COORDINATOR | | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 7071K| 23G| | | | Q1,00 | P->S | QC (RAND) |
| 3 | PX PARTITION HASH ALL | | 7071K| 23G| | 1 | 32 | Q1,00 | PCWC | |
| 4 | HASH GROUP BY | | 7071K| 23G| 38G| | | Q1,00 | PCWP | |
| 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 | Q1,00 | PCWP | |
------------------------------------------------------------------------------------------------------------------------
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
GROUP
PGA
GROUP
PGA
GROUP
PGA
GROUP
PGA
QC
LIST
P1
P2
P3
P4
P5
P6
P7
P8

53. © 2015 Pythian – Presentation for DOAG
select /*+ parallel(64) */ * from ckk c1 order by id;
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 7071K| 23G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 7071K| 23G| | Q1,01 | P->S | QC (RAND) |
| 3 | HASH GROUP BY | | 7071K| 23G| 38G| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH |
| 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
Group by – enough partitions
53
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
GROUP
PGA
GROUP
PGA
GROUP
PGA
GROUP
PGA
QC
LIST
P1
P2
P3
P4
P5
P6
P7
P8

54. © 2015 Pythian – Presentation for DOAG
Analytics don’t speak parallel
54
select /*+ parallel(4) */ id, row_number() over (partition by id,filler order by null) r from ckk_p ;
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 32G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 32G| | Q1,01 | P->S | QC (RAND) |
| 3 | WINDOW SORT | | 10M| 32G| 38G| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 32G| | Q1,01 | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 10M| 32G| | Q1,00 | P->P | HASH |
| 6 | PX BLOCK ITERATOR | | 10M| 32G| | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
select /*+ NO_parallel */ id, row_number() over (partition by id,filler order by null) r from ckk_p ;
--------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | Pstart| Pstop |
--------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 32G| | | |
| 1 | PARTITION HASH ALL | | 10M| 32G| | 1 | 32 |
| 2 | WINDOW SORT | | 10M| 32G| 38G| | |
| 3 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 32G| | 1 | 32 |
--------------------------------------------------------------------------------------

55. 55
The Big Join

56. © 2015 Pythian – Presentation for DOAG
Test Data
56
• Copy of ckk – ckk2
• ckk_m5 – 5 rows dimension
create table ckk_m5 as select rownum mod5_id, rpad('x',1000,'x')
filler from dual connect by level <=5;
alter table ckk_m5 modify (mod5_id not null);

57. © 2015 Pythian – Presentation for DOAG
Simple Hash Join
select /*+ parallel(4) */ * from ckk c1 join ckk_m5 m5 on (c1.mod5_id = m5.mod5_id);
--------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 42G| | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 42G| Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN | | 10M| 42G| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 5 | 5020 | Q1,01 | PCWP | |
| 5 | PX SEND BROADCAST | :TQ10000 | 5 | 5020 | Q1,00 | P->P | BROADCAST |
| 6 | PX BLOCK ITERATOR | | 5 | 5020 | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK_M5 | 5 | 5020 | Q1,00 | PCWP | |
| 8 | PX BLOCK ITERATOR | | 10M| 33G| Q1,01 | PCWC | |
| 9 | TABLE ACCESS STORAGE FULL | CKK | 10M| 33G| Q1,01 | PCWP | |
--------------------------------------------------------------------------------------------------

58. © 2015 Pythian – Presentation for DOAG
Simple Hash Join
58
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
QC
CKK
_M5
0 GiB
All Hash Areas:
0 GiB

59. © 2015 Pythian – Presentation for DOAG
Simple Hash Join
59
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
STEP (2)
QC
CKK
(39 GiB)

60. © 2015 Pythian – Presentation for DOAG
Big Hash Join
select /*+ parallel(4) */ * from ckk c1join ckk2 c2 on (c1.id = c2.id)
-------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10002 | 10M| 66G| | P->S | QC (RAND) |
|* 3 | HASH JOIN BUFFERED | | 10M| 66G| 8502M| PCWP | |
| 4 | PX RECEIVE | | 10M| 33G| | PCWP | |
| 5 | PX SEND HASH | :TQ10000 | 10M| 33G| | P->P | HASH |
| 6 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | PCWP | |
| 8 | PX RECEIVE | | 10M| 33G| | PCWP | |
| 9 | PX SEND HASH | :TQ10001 | 10M| 33G| | P->P | HASH |
| 10 | PX BLOCK ITERATOR | | 10M| 33G| | PCWC | |
| 11 | TABLE ACCESS STORAGE FULL| CKK2 | 10M| 33G| | PCWP | |
-------------------------------------------------------------------------------------------------

61. © 2015 Pythian – Presentation for DOAG
Big Hash Join
61
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
QC
CKK
39 GiB
All Hash Areas:
39 GiB

62. © 2015 Pythian – Presentation for DOAG
Big Hash Join
62
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (2)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
QC
CKK
2
39 GiB
All Hash Areas:
78 GiB

63. © 2015 Pythian – Presentation for DOAG
Simple Hash Join
63
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
STEP (3)
QC

64. © 2015 Pythian – Presentation for DOAG
Hash Join summary
64
• HASH JOINS with small tables are extremely fast
• Uses BROADCAST to send copy of table to every
slave
• As much PGA work area as parallel degree * small
table size
• HASH JOINS with big tables can be extremely expensive
• Almost entire result set written to temp
• Unless – partitioning

65. © 2015 Pythian – Presentation for DOAG
Semi-Partitioned Hash Join
select /*+ parallel(32) */ * from ckk c1 join ckk_p c2 on (c1.id = c2.id)
-----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | |
| 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | |
| 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) |
| 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
| 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | |
| 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |
-----------------------------------------------------------------------------------------------------------

66. © 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
66
PQ1
PQ2
PQ3
PQ4
S
G
A
STEP (1)
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
QC
CKK
39 GiB
All Hash Areas:
39 GiB

67. © 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
67
PQ5
PQ6
PQ7
PQ8
HASH
PGA
HASH
PGA
HASH
PGA
HASH
PGA
S
G
A
STEP (2)
QC
CKK_P
(39 GiB)
LIST
P1
P2
P3
P4
P5
P6
P7
P8

68. © 2015 Pythian – Presentation for DOAG
Semi Partitioned Hash Join
68
• Dramatically reduces HASH Area sizes
• Very likely to fit in RAM – no disk
• Reduces from 3 steps to 2 steps
• Always reads the non-partitioned table into HASH areas
• Each slave reads it’s own partition

69. © 2015 Pythian – Presentation for DOAG
Partitioned Hash Join
select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c1.id = c2.id)
--------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 10M| 66G| | Q1,00 | P->S | QC (RAND) |
| 3 | PX PARTITION HASH ALL | | 10M| 66G| | Q1,00 | PCWC | |
|* 4 | HASH JOIN | | 10M| 66G| 8502M| Q1,00 | PCWP | |
| 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | |
| 6 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------------

70. © 2015 Pythian – Presentation for DOAG
Partitioned Hash Join
select /*+ parallel(4) */ * from ckk_p c1 join ckk_p c2 on (c1.id = c2.id)
--------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 10M| 66G| | Q1,00 | P->S | QC (RAND) |
| 3 | PX PARTITION HASH ALL | | 10M| 66G| | Q1,00 | PCWC | |
|* 4 | HASH JOIN | | 10M| 66G| 8502M| Q1,00 | PCWP | |
| 5 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | |
| 6 | TABLE ACCESS STORAGE FULL| CKK_P | 10M| 33G| | Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------------

71. © 2015 Pythian – Presentation for DOAG
Switching places
select /*+ parallel(32) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id);
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | |
| 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | |
| 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |
| 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) |
| 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
select /*+ parallel(32) LEADING(c2) */ * from ckk_p c1 join ckk c2 on (c1.id = c2.id);
-----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
-----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN | | 10M| 66G| 1062M| Q1,01 | PCWP | |
| 4 | PART JOIN FILTER CREATE | :BF0000 | 10M| 33G| | Q1,01 | PCWP | |
| 5 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 6 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) |
| 7 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 8 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
| 9 | PX PARTITION HASH JOIN-FILTER| | 10M| 33G| | Q1,01 | PCWC | |
| 10 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |
-----------------------------------------------------------------------------------------------------------

72. © 2015 Pythian – Presentation for DOAG
Outer Joins
select /*+ parallel(32) */ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id);
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN OUTER BUFFERED | | 10M| 66G| 1062M| Q1,01 | PCWP | |
| 4 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | |
| 5 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |
| 6 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 7 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) |
| 8 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 9 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------
select /*+ parallel(32) SWAP_JOIN_INPUTS(c2)*/ * from ckk_p c1 left join ckk c2 on (c1.id = c2.id);
----------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|E-Temp | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 10M| 66G| | | | |
| 1 | PX COORDINATOR | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10001 | 10M| 66G| | Q1,01 | P->S | QC (RAND) |
|* 3 | HASH JOIN RIGHT OUTER | | 10M| 66G| 1062M| Q1,01 | PCWP | |
| 4 | PX RECEIVE | | 10M| 33G| | Q1,01 | PCWP | |
| 5 | PX SEND PARTITION (KEY) | :TQ10000 | 10M| 33G| | Q1,00 | P->P | PART (KEY) |
| 6 | PX BLOCK ITERATOR | | 10M| 33G| | Q1,00 | PCWC | |
| 7 | TABLE ACCESS STORAGE FULL| CKK | 10M| 33G| | Q1,00 | PCWP | |
| 8 | PX PARTITION HASH ALL | | 10M| 33G| | Q1,01 | PCWC | |
| 9 | TABLE ACCESS STORAGE FULL | CKK_P | 10M| 33G| | Q1,01 | PCWP | |
----------------------------------------------------------------------------------------------------------

73. 73
The Big Join – with Index

74. © 2015 Pythian – Presentation for DOAG
Adding a few indexes
74
create index ckk$id on ckk(id)
parallel 32 tablespace ckbig nologging;
--6.5 sec
create index ckk$mod5000_id on
ckk(mod5000_id) parallel 32 tablespace
ckbig nologging compress;
--5.8 sec
create index ckk_p$mod5000_id on
ckk_p(mod5000_id) parallel 32
tablespace ckbig nologging compress
local;
--16 sec

75. © 2015 Pythian – Presentation for DOAG
Direct index access
select /*+ parallel(4) */ * from ckk where mod5000_id=100;
-----------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes|
-----------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 2000 | 6941K|
| 1 | TABLE ACCESS BY INDEX ROWID| CKK | 2000 | 6941K|
|* 2 | INDEX RANGE SCAN | CKK$MOD5000_ID | 2000 | |
-----------------------------------------------------------------------
select /*+ parallel(4) */ * from ckk_p where mod5000_id=100;
--------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 2000 | 6941K| | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 2000 | 6941K| Q1,00 | P->S | QC (RAND) |
| 3 | PX PARTITION HASH ALL | | 2000 | 6941K| Q1,00 | PCWC | |
| 4 | TABLE ACCESS BY LOCAL INDEX ROWID| CKK_P | 2000 | 6941K| Q1,00 | PCWP | |
|* 5 | INDEX RANGE SCAN | CKK_P$MOD5000_ID | 2000 | | Q1,00 | PCWP | |
--------------------------------------------------------------------------------------------------------------

76. © 2015 Pythian – Presentation for DOAG
Join with Index
select /*+ parallel(4) */ * from ckk c1 join ckk2 c2 on (c1.id = c2.id)
where c2.mod5000_id=5;
-------------------------------------------------------------------------------------------------
| Id | Operation | Name | E-Rows |E-Bytes| TQ |IN-OUT| PQ Distrib |
-------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 2017 | 13M| | | |
| 1 | PX COORDINATOR | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | | | Q1,00 | P->S | QC (RAND) |
| 3 | NESTED LOOPS | | | | Q1,00 | PCWP | |
| 4 | NESTED LOOPS | | 2017 | 13M| Q1,00 | PCWP | |
| 5 | PX BLOCK ITERATOR | | | | Q1,00 | PCWC | |
|* 6 | TABLE ACCESS STORAGE FULL| CKK2 | 2000 | 6941K| Q1,00 | PCWP | |
|* 7 | INDEX RANGE SCAN | CKK$ID | 1 | | Q1,00 | PCWP | |
| 8 | TABLE ACCESS BY INDEX ROWID| CKK | 1 | 3554 | Q1,00 | PCWP | |
-------------------------------------------------------------------------------------------------

77. © 2015 Pythian – Presentation for DOAG
Indexes summary
77
• Parallel access on direct value queries only on
partitioned tables and on local indexes
• Parallel access on global or non-partitioned only
from joins (multiple different values)

78. © 2015 Pythian – Presentation for DOAG
Final Thoughts
78
• Exadata allows Parallel Query to make sense
• High Bandwidth everywhere
• Very efficient table scans
• Optimiser sometimes picks up wrong access
plan
• There are deficiencies in SORT operations –
balancing the workloads
• There are deficiencies in handling disk write
operations in PQ Slaves
• Can be extremely fast under the right
circumstances

79. © 2009/2010 Pythian - Presentation for ABC Company79
The End
Thank You
Questions?
I blog at
http://www.pythian.com/news/author/kutrovsky/