SlideShare a Scribd company logo

Solving Slow Parallel Loads and Waits in an Exadata Data Warehouse

•
•
Tanel Poder
Tanel Poder

The document describes troubleshooting a performance issue involving parallel data loads into a data warehouse. It is determined that the slowness is due to recursive locking and buffer busy waits occurring during inserts into the SEG$ table as new segments are created by parallel CREATE TABLE AS SELECT statements. This is causing a nested locking ping-pong effect between the cache, transaction, and I/O layers as sessions repeatedly acquire and release locks and buffers.

Technology
1 of 41
Download to read offline
1 © Tanel Poder 2020 blog.tanelpoder.com Troubleshooting Another Complex Performance Issue (Part 4) Tanel Põder https://blog.tanelpoder.com/ @tanelpoder
2 © Tanel Poder 2020 blog.tanelpoder.com A bit of philosophy …
3 © Tanel Poder 2020 blog.tanelpoder.com Your task is slow because It is doing too many operations or The operations complete too slowly or both
4 © Tanel Poder 2020 blog.tanelpoder.com Measure & Profile! 1. Top SQL & wait events 2. Performance counters 3. Oracle kernel functions ASH, sqlmon sql_trace snapper sqlmon pstack perf
5 © Tanel Poder 2020 blog.tanelpoder.com Let's get started!
6 © Tanel Poder 2020 blog.tanelpoder.com A Data Warehouse data loading/preparation • A large Exadata / RAC / Oracle 11.2.0.3 reporting environment • Lots of parallel CTAS and direct path loads • High concurrency, high parallelism • Throughput bad, all kinds of waits showing up: • Other, Cluster, Configuration, Application, Concurrency, User I/O, CPU
7 © Tanel Poder 2020 blog.tanelpoder.com Parallel Degree 48, 96+, inter-instance PX • Mostly CREATE TABLE AS SELECT statements! • Creating pre-aggregated tables, etc
8 © Tanel Poder 2020 blog.tanelpoder.com Drilldown with ASH data – instance-wide • Wait event names are more informative than wait classes: SQL> @ashtop session_state,wait_class,event session_type='FOREGROUND' sysdate-1/24 sysdate %This SESSION WAIT_CLASS EVENT AAS Seconds ------ ------- -------------- ------------------------------ ------ ------- 32% ON CPU 40.6 146197 13% WAITING Cluster gc buffer busy acquire 16.8 60559 11% WAITING Configuration enq: TX - allocate ITL entry 13.9 50115 9% WAITING Cluster gc buffer busy release 11.3 40739 8% WAITING Scheduler resmgr:cpu quantum 9.9 35711 4% WAITING Application enq: TX - row lock contention 5.2 18798 3% WAITING User I/O direct path read temp 4.3 15400 3% WAITING User I/O cell smart table scan 3.7 13259 2% WAITING Concurrency buffer busy waits 3 10735 2% WAITING Other enq: CF - contention 2.1 7443 1% WAITING Application enq: TM - contention 1.7 6109
9 © Tanel Poder 2020 blog.tanelpoder.com Drilldown with ASH data – one SQL: • Let's pick one of the slow CTAS statements: SQL> @ash/dashtop session_state,event,p1text,p1 top_level_sql_id='89apzfvjba3gu' sysdate-2 sysdate %This SESSION EVENT P1TEXT P1 ------ ------- ------------------------------ ------------------ ---------- 36% WAITING gc buffer busy acquire file# 1 20% WAITING gc buffer busy release file# 1 10% WAITING enq: TX - allocate ITL entry name|mode 1415053316 8% WAITING gc buffer busy acquire file# 1 4% WAITING buffer busy waits file# 1 4% WAITING enq: TX - allocate ITL entry name|mode 1415053316 2% WAITING buffer busy waits file# 1 2% WAITING gc buffer busy release file# 1 2% WAITING enq: TX - row lock contention name|mode 1415053316 1% WAITING latch: ges resource hash list address 3.3055E+10 1% ON CPU sleeptime/senderid 268501512 ... Looks like the CTAS statements do directly experience these waits
10 © Tanel Poder 2020 blog.tanelpoder.com Wait event explanation • buffer busy waits • Buffer is physically available in local instance, but is pinned by some other local session (in incompatible mode) • gc buffer busy acquire • Someone has already requested the remote block into local instance • gc buffer busy release • Block is local, but has to be shipped out to a remote instance first (as someone in the remote instance had requested it first) • enq: TX - allocate ITL entry • Can't change block as all the block's ITL entries are held by others • Can't dynamically add more ITLs (no space in block)
11 © Tanel Poder 2020 blog.tanelpoder.com Looks like write-write contention across RAC nodes!
12 © Tanel Poder 2020 blog.tanelpoder.com Contention on what objects, blocks, what SQL? • Let's dig into the top wait event: SQL> @ashtop session_state,wait_class,p1,p2 "event='gc buffer busy acquire'" sysdate-1/24 sysdate %This SESSION WAIT_CLASS P1 P2 AAS TotalSeconds ------ ------- -------------- ---------- ---------- ---------- ------------ 8% WAITING Cluster 1 279634 1.3 4760 6% WAITING Cluster 1 279635 1.1 3916 6% WAITING Cluster 1 279629 1.1 3838 5% WAITING Cluster 1 279632 .9 3146 4% WAITING Cluster 1 279638 .7 2503 ... SQL> @sed "gc buffer busy" EVENT_NAME WAIT_CLASS PARAMETER1 PARAMETER2 PARAMETER3 ----------------------- ----------- ----------- ----------- ----------- gc buffer busy acquire Cluster file# block# class# gc buffer busy release Cluster file# block# class#
13 © Tanel Poder 2020 blog.tanelpoder.com Which object? • Translate file#, block# into segment names/numbers: • Assumes the blocks are in buffer cache SQL> SELECT objd data_object_id, COUNT(*) FROM v$bh WHERE file#=1 AND block# IN ( 279634,279635,279629,279632,279638,279636,279613,279662, 279628,279608,279653,279627,279642,279637,279643,279631, 279646,279622,279582,279649 ) GROUP BY objd ORDER BY 2 DESC; DATA_OBJECT_ID COUNT(*) -------------- ---------- 8 113 All blocks belong to a segment with data_object_id = 8
14 © Tanel Poder 2020 blog.tanelpoder.com What is segment #8? • Look up the object names: • Using DBA_OBJECTS.DATA_OBJECT_ID -> OBJECT_ID SQL> @doid 8 object_id owner object_name O_PARTITION object_type --------- --------- ----------------- ------------- ----------- 8 SYS C_FILE#_BLOCK# CLUSTER 14 SYS SEG$ TABLE 13 SYS UET$ TABLE This segment #8 is a cluster which contains SEG$ (DBA_SEGMENTS) and UET$ (DBA_EXTENTS), but UET$ isn't used anymore thanks to LMT tablespaces. That leaves SEG$
15 © Tanel Poder 2020 blog.tanelpoder.com Write contention on SEG$? • SEG$ is modified when: 1. A new segment is created (table, index, partition, etc) 2. An existing segment extends 3. A segment is dropped / moved 4. Parallel direct path loads (CTAS) can also create many temporary segments (one per PX slave) • …and merge these into final segment in the end of loading More about this later…
16 © Tanel Poder 2020 blog.tanelpoder.com More evidence!
17 © Tanel Poder 2020 blog.tanelpoder.com AWR 1 • No CPU starvation evident (checked all RAC nodes) Top 5 Timed Foreground Events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Event Waits Time(s) (ms) time Wait Class ---------------------------- ----------- -------- ----- ------ ---------- DB CPU 165,199 37.0 gc buffer busy acquire 1,260,128 68,399 54 15.3 Cluster enq: TX - allocate ITL entry 354,496 40,583 114 9.1 Configurat direct path write temp 4,632,946 37,455 8 8.4 User I/O gc buffer busy release 213,750 22,683 106 5.1 Cluster Host CPU (CPUs: 160 Cores: 80 Sockets: 8) ~~~~~~~~ Load Average Begin End %User %System %WIO %Idle --------- --------- --------- --------- --------- --------- 5.20 42.76 26.0 3.3 0.0 69.8
18 © Tanel Poder 2020 blog.tanelpoder.com AWR 2 • AWR also listed a SEG$ insert as a top SQL: Elapsed Elapsed Time Time (s) Executions per Exec (s) %Total %CPU %IO SQL Id ----------- ------------- ------------- ------ ----- ----- ------------- 100,669.1 4,444 22.65 22.6 .5 .0 g7mt7ptq286u7 insert into seg$ (file#,block#,type#,ts#,blocks,extents,minexts,maxexts, extsize,extpct,user#,iniexts,lists,groups,cachehint,hwmincr, spare1, scanhint, bitmapranges) values (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12, :13,:14,:15,:16,DECODE(:17,0,NULL,:17),:18,:19) SNAP_ID NODE SQL_ID EXECS AVG_ETIME AVG_LIO AVG_PIO ------- ---- ------------- ------ ---------- -------- -------- 24263 3 g7mt7ptq286u7 552 70.227 880.6 .0 • AWR on another node (same insert into seq$): Super-slow single row inserts into SEG$ ???
19 © Tanel Poder 2020 blog.tanelpoder.com SQL Trace confirms SEG$ insert problem: insert into seg$ (file#,block#,type#,ts#,blocks,extents,minexts,maxexts, extsize,extpct,user#,iniexts,lists,groups,cachehint,hwmincr, spare1, scanhint, bitmapranges) values (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,:15,:16,DECODE(:17,0,NULL, :17),:18,:19) call count cpu elapsed disk query current rows ------- ------ -------- ---------- -------- -------- --------- ------- Parse 0 0.00 0.00 0 0 0 0 Execute 1 0.82 75.14 3 773 1508 1 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- -------- -------- --------- ------- total 1 0.82 75.14 3 773 1508 1 Row Source Operation --------------------------------------------------- LOAD TABLE CONVENTIONAL (cr=773 pr=3 pw=0 time=75139696 us)
20 © Tanel Poder 2020 blog.tanelpoder.com SEG$ insert wait events: Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ------------------------------------ Waited ---------- ------------ gc buffer busy release 199 1.01 10.76 buffer busy waits 560 1.03 7.02 resmgr:cpu quantum 54 0.04 0.08 enq: TX - row lock contention 93 0.35 4.25 latch: ges resource hash list 400 0.01 0.98 gc buffer busy acquire 464 8.96 36.64 buffer deadlock 158 0.00 0.00 Disk file operations I/O 5 0.00 0.00 gc cr block 2-way 101 0.00 0.01 gc current block 2-way 2 0.00 0.00 cell single block physical read 3 0.00 0.00 gc current block busy 10 0.05 0.14 gc current grant 2-way 2 0.00 0.00 enq: TX - allocate ITL entry 316 0.77 13.18 latch: cache buffers chains 334 0.01 1.45 ... Why does a single row insert cause so many waits?
21 © Tanel Poder 2020 blog.tanelpoder.com Checkpoint: What do we know?
22 © Tanel Poder 2020 blog.tanelpoder.com Checkpoint 1. Lots of parallel CTAS statements, seem to wait for various RAC gc buffer busy events and enq: TX – ITL contention 2. CTAS statements create new segments, new segments cause inserts into SEG$ 3. AWR and SQL Trace report super-long elapsed times for single-row SEG$ inserts 4. It's actually the recursive SEG$ inserts that wait for the gc buffer busy and enq: TX – ITL contention events
23 © Tanel Poder 2020 blog.tanelpoder.com "Hypothesis" Could we be in a cache + transaction layer nested locking ping-pong loop?
24 © Tanel Poder 2020 blog.tanelpoder.com Multiple layers of locking & coherency ITL 2ITL 1 data2data1ccLH c2c1 ITL 4ITL 3 KCBH - block common header KTBBH - transaction common header KDBH - Data header Data block Row Level 0: VOS/service layer - Cache buffers chains latch Level 1: Cache layer Buffer pin - local pin - global cache pin Level 2: Transaction layer - ITL entry - row lock byte You must pin (and retrieve) a buffer before you can change or examine it What if you pin a buffer and find all its ITL slots busy? (with no space to add mode) The session will release the pin and start waiting for enq: TX – ITL contention event And re-get/pin again when TX – ITL wait is over!
25 © Tanel Poder 2020 blog.tanelpoder.com Raw trace file (edited): *** 2013-11-18 21:39:09.760 WAIT #1: nam='buffer busy waits' ela= 1023321 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='gc buffer busy release' ela= 25304 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 4315 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='Disk file operations I/O' ela= 10 FileOperation=2 fileno=141 filetype=2 WAIT #1: nam='gc cr block 2-way' ela= 134 p1=141 p2=5456 p3=3447 obj#=0 WAIT #1: nam='enq: TX - allocate ITL entry' ela= 29295 name|mode=1415053316 WAIT #1: nam='latch: ges resource hash list' ela= 958 address=33046235872 number=91 WAIT #1: nam='gc buffer busy acquire' ela= 3384 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 817 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='gc buffer busy acquire' ela= 344 file#=1 block#=192 class#=4 obj#=14 WAIT #1: nam='gc buffer busy acquire' ela= 2096 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 3922 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='gc cr block 2-way' ela= 124 p1=141 p2=28888 p3=3895 obj#=0 WAIT #1: nam='enq: TX - allocate ITL entry' ela= 9260 name|mode=1415053316 WAIT #1: nam='latch: ges resource hash list' ela= 683 address=33043220640 WAIT #1: nam='gc buffer busy acquire' ela= 13696 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 1312 file#=1 block#=279656 class#=1 obj#=14 ... *** 2013-11-18 21:39:10.933 WAIT #1: nam='gc buffer busy acquire' ela= 1023271 file#=1 block#=277294 class#=1 obj#=9 WAIT #1: nam='buffer deadlock' ela= 15 dba=4471598 class*10+mode=12 flag=8192 obj#=9 What's obj# 9? What's class# 4? obj# 14 is SEG$
26 © Tanel Poder 2020 blog.tanelpoder.com OBJ #9 and #14 • Look up the object names: • Using DBA_OBJECTS.DATA_OBJECT_ID -> OBJECT_ID SQL> @oid 9,14 owner object_name object_type --------------- --------------------- ------------------ SYS I_FILE#_BLOCK# INDEX SYS SEG$ TABLE The object #9 is a cluster index for cluster C_FILE#_BLOCK# (oid #8 on a previous slide SQL> @bclass 4 CLASS UNDO_SEGMENT_ID ------------------ --------------- segment header Bclass #4 is a segment header block that also stores freelist information
27 © Tanel Poder 2020 blog.tanelpoder.com So what did happen?
28 © Tanel Poder 2020 blog.tanelpoder.com Conclusion from gathered evidence 1. Lots of concurrent + highly parallel CTAS statements 2. Each PX slave created its own temporary segments (in spikes when PX query started up) 3. Spikes of concurrent SEG$ inserts simultaneously on 4 RAC nodes 4. SEG$ cluster blocks out of ITL entries 5. This caused global cache/TX locking trashing – long loops of unsuccessful attempts to pin and insert into SEG$ blocks
29 © Tanel Poder 2020 blog.tanelpoder.com Also worth noting 1. Resource manager a possible factor • Plenty of resmgr:cpu quantum waits • Lock holders, buffer pin holders might have gone to sleep – while holding the pins/locks under contention 1. Freelist-based block space allocation • SEG$ is a SYS-owned freelist-managed index cluster • Many cluster table blocks walked through when searching for space 2. ASH doesn't always show recursive SQL IDs • Attributes waits to parent (top level) statement SQL ID instead • ASH p1, p2, p3, current_obj# columns are useful for drilldown
30 © Tanel Poder 2020 blog.tanelpoder.com How to fix it? • Do it less! • What? • Do not insert/update SEG$ entries so frequently! • How? • Do not allow parallel direct load insert to create a temporary loading segment for each slave (and inserted partition)! • How? • Make sure that High Water Mark Brokering is used! • One temporary segment - thus one SEG$ insert/update/delete - per query (and inserted partition) = much less SEG$ contention. • More in following slides …
31 © Tanel Poder 2020 blog.tanelpoder.com Why is High-Water-Mark Brokering needed? • A historical problem with large uniform extent sizes: 1. Each data loading PX slave allocated its own extents (to its private temporary segment) when loading data 2. When the load was completed the private temporary segments were merged to one final table segment 3. The last extent of each private segment possibly ended up not fully used • Some were almost full, some almost empty – half-empty on average 4. Wastage = 0.5 x extent_size x PX_slaves • The more PX slaves, the more wastage • The bigger extent, the bigger wastage • References: • https://blogs.oracle.com/datawarehousing/entry/parallel_load_uniform_or_autoall ocate • http://www.oracle.com/technetwork/database/bi- datawarehousing/twpdwbestpractices-for-loading-11g-404400.pdf Problem Solved by using High-Water-Mark Brokering! But…
32 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – space wastage? SQL> @seg sales_parallel_ctas SEG_MB OWNER SEGMENT_NAME SEG_TABLESPACE_NAME ------- ------- ------------------------- -------------------- 9472 TANEL SALES_PARALLEL_CTAS TANEL_DEMO_LARGE 8896 TANEL SALES_PARALLEL_CTAS_FIX TANEL_DEMO_LARGE SQL>SELECT * FROM TABLE(space_tools.get_space_usage('TANEL', 'SALES_PARALLEL_CTAS','TABLE')); --------------------------------------------------------------------------------------- Full blocks /MB 1134669 8865 Unformatted blocks/MB 0 0 Free Space 0-25% 0 0 Free Space 25-50% 0 0 Free Space 50-75% 0 0 Free Space 75-100% 72963 570 SQL>SELECT * FROM TABLE(space_tools.get_space_usage('TANEL', 'SALES_PARALLEL_CTAS_FIX’,'TABLE')) ---------------------------------------------------------------------------------------- Full blocks /MB 1134669 8865 Unformatted blocks/MB 0 0 Free Space 0-25% 0 0 Free Space 25-50% 0 0 Free Space 50-75% 0 0 Free Space 75-100% 0 0 An identical table PX CTAS in the same tablespace with 64MB uniform extent (upper table 6.4% bigger) 570 MB worth of blocks unused in the segment? PX DOP 16 x 64 * ~0.5 = 512MB Same table loaded with INSERT /*+ APPEND */ PX DOP 16. No empty blocks
33 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – HWM brokering • Instead of each PX slave working on their own separate segment… 1. Allocate and extend only one segment 2. Slaves allocate space within the single segment as needed 3. No “holes” in tables, no space wastage problem anymore! 4. Serialized via the HV - Direct Loader High Water Mark enqueue • Parameter: • _insert_enable_hwm_brokered = true (default) • “during parallel inserts high water marks are brokered” • Problem: • In Oracle 11.2 (including 11.2.0.4) this only works with INSERT APPENDs • …and not for CTAS nor ALTER TABLE MOVE • (in one case it worked for CTAS, only if the target table was partitioned)
34 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – HWM brokering SQL> @fix 6941515 BUGNO VALUE DESCRIPTION IS_DEFAULT ------- ------ ------------------------------------------------------------ ---------- 6941515 0 use high watermark brokering for insert into single segment 1 SQL> alter session set "_fix_control"='6941515:ON'; Session altered. • Actually this is a bug, and Oracle has fixed it long time ago • But bugfix 6941515 is not enabled by default! • After enabling the bug-fix, parallel CTAS and parallel ALTER TABLE MOVE also use HWM brokering… • Unfortunately it didn’t work with ALTER TABLE MOVE if moving the whole table if the table was partitioned. DBMS_SPACE / space_tools helps to measure this! Thanks to Alex Fatkulin for spotting this!
35 © Tanel Poder 2020 blog.tanelpoder.com Additional slides (we are probably out of time)
36 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Data Loading Performance – Example 1 • Parallel Create Table As Select from Hadoop to Exadata • Buffer busy waits dominating the response time profile (SQL Monitoring) https://blog.tanelpoder.com/2013/11/06/diagnosing-buffer-busy-waits-with-the-ash_wait_chains-sql-script-v0-2/
37 © Tanel Poder 2020 blog.tanelpoder.com Data Loading Performance: ashtop.sql SQL> @ash/ashtop session_state,event,p2text,p2,p3text,p3 sql_id='3rtbs9vqukc71' "timestamp'2013-10-05 01:00:00'" "timestamp'2013-10-05 03:00:00'" %This SESSION EVENT P2TEXT P2 P3TEXT P3 ------ ------- --------------------------------- --------- -------- -------- ------- 57% WAITING buffer busy waits block# 2 class# 13 31% ON CPU file# 0 size 524288 1% WAITING external table read file# 0 size 524288 1% ON CPU block# 2 class# 13 0% ON CPU consumer 12573 0 0% WAITING cell smart file creation 0 0 0% WAITING DFS lock handle id1 3 id2 2 0% ON CPU file# 41 size 41 0% WAITING cell single block physical read diskhash# 4695794 bytes 8192 0% WAITING control file parallel write block# 1 requests 2 0% WAITING control file parallel write block# 41 requests 2 0% WAITING change tracking file synchronous blocks 1 0 0% WAITING control file parallel write block# 42 requests 2 0% WAITING db file single write block# 1 blocks 1 0% ON CPU 0 0 • Break the (buffer busy) wait events down by block#/class# block #2 ?
38 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Data Loading Performance – Example 2 • Lots of serial sessions doing single row inserts – on multiple RAC nodes • “buffer busy waits” and “gc buffer busy acquire” waits • file# = 6 • block# = 2 • class# = 13 SQL> @bclass 13 CLASS UNDO_SEGMENT_ID ------------------ --------------- file header block SQL> select file#, block#, status from v$bh where class# = 13; FILE# BLOCK# STATUS ---------- ---------- ---------- 5 2 xcur 4 2 xcur ... Block dump from disk: buffer tsn: 7 rdba: 0x00000002 (1024/2) scn: 0x0000.010b6f9b seq: 0x02 flg: 0x04 tail: 0x6f9b1d02 frmt: 0x02 chkval: 0xd587 type: 0x1d=KTFB Bitmapped File Space Header Hex dump of block: st=0, typ_found=1 A single space allocation contention point per LMT file. Bigfile tablespaces have only one file!
39 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading Performance: ash_wait_chains.sql SQL> @ash/ash_wait_chains event2 sql_id='3rtbs9vqukc71' "timestamp'2013-10-05 01:00:00'" "timestamp'2013-10-05 03:00:00'” %This SECONDS WAIT_CHAIN ------ ---------- ---------------------------------------------------------------------------- 35% 43735 -> ON CPU 15% 18531 -> buffer busy waits [file header block] -> ON CPU 7% 9266 -> buffer busy waits [file header block] -> control file parallel write 7% 8261 -> buffer busy waits [file header block] -> cell smart file creation 6% 6959 -> direct path write 5% 6707 -> buffer busy waits [file header block] -> DFS lock handle 4% 4658 -> buffer busy waits [file header block] -> local write wait 4% 4610 -> buffer busy waits [file header block] -> cell single block physical read 3% 4282 -> buffer busy waits [file header block] -> local write wait -> db file par 2% 2801 -> buffer busy waits [file header block] 2% 2676 -> buffer busy waits [file header block] -> ASM file metadata operation 2% 2092 -> buffer busy waits [file header block] -> change tracking file synchronous 2% 2050 -> buffer busy waits [file header block] -> control file sequential read The waiting session(s) The blocking session(s) Block type if buffer busy wait • Display top wait chains from ASH (who’s waiting for whom)
40 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Parallel Data Loading Performance • Reduce demand on the LMT bitmap blocks • By allocating bigger extents at a time • Use large uniform extents for fast-growing tables(paces) • The customer went with 64MB uniform extent size • The autoallocate extent management is suboptimal for very large segments • As there’d be 1 LMT space management bit per 64kB regardless of your INITIAL extent size at the segment level • The _partition_large_extents = TRUE doesn’t change this either • Large uniform extents are better for data loading and scanning!
41 © Tanel Poder 2020 blog.tanelpoder.com Thanks!!! https://blog.tanelpoder.com/seminar tanel@tanelpoder.com @tanelpoder

Recommended

Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Kyle Hailey
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder
 
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder
 
Oracle Database performance tuning using oratop
Oracle Database performance tuning using oratopOracle Database performance tuning using oratop
Oracle Database performance tuning using oratopSandesh Rao
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTanel Poder
 
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsEnkitec
 
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAsOracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAsZohar Elkayam
 
Same plan different performance
Same plan different performanceSame plan different performance
Same plan different performanceMauro Pagano
 

Recommended

Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Kyle Hailey
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder
 
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder
 
Oracle Database performance tuning using oratop
Oracle Database performance tuning using oratopOracle Database performance tuning using oratop
Oracle Database performance tuning using oratopSandesh Rao
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTanel Poder
 
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsEnkitec
 
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAsOracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAsZohar Elkayam
 
Same plan different performance
Same plan different performanceSame plan different performance
Same plan different performanceMauro Pagano
 
Low Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesLow Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesTanel Poder
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsDB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsJohn Beresniewicz
 
Tanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder
 
Oracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention TroubleshootingOracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention TroubleshootingTanel Poder
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Aaron Shilo
 
Chasing the optimizer
Chasing the optimizerChasing the optimizer
Chasing the optimizerMauro Pagano
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slidesUnderstanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slidesMohamed Farouk
 
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN Riyaj Shamsudeen
 
AWR and ASH Deep Dive
AWR and ASH Deep DiveAWR and ASH Deep Dive
AWR and ASH Deep DiveKellyn Pot'Vin-Gorman
 
Oracle db performance tuning
Oracle db performance tuningOracle db performance tuning
Oracle db performance tuningSimon Huang
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1Tanel Poder
 
Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360Carlos Sierra
 
SQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12cSQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12cTanel Poder
 
TFA Collector - what can one do with it
TFA Collector - what can one do with it TFA Collector - what can one do with it
TFA Collector - what can one do with it Sandesh Rao
 
Oracle AWR Data mining
Oracle AWR Data miningOracle AWR Data mining
Oracle AWR Data miningYury Velikanov
 
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And WhatPerformance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And Whatudaymoogala
 
Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by exampleMauro Pagano
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsJohn Kanagaraj
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the TradeCarlos Sierra
 
Oracle Database SQL Tuning Concept
Oracle Database SQL Tuning ConceptOracle Database SQL Tuning Concept
Oracle Database SQL Tuning ConceptChien Chung Shen
 
Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +OracletuningThomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuningafa reg
 
Using AWR for IO Subsystem Analysis
Using AWR for IO Subsystem AnalysisUsing AWR for IO Subsystem Analysis
Using AWR for IO Subsystem AnalysisTexas Memory Systems, and IBM Company
 

More Related Content

What's hot

Low Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesLow Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesTanel Poder
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsDB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsJohn Beresniewicz
 
Tanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder
 
Oracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention TroubleshootingOracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention TroubleshootingTanel Poder
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Aaron Shilo
 
Chasing the optimizer
Chasing the optimizerChasing the optimizer
Chasing the optimizerMauro Pagano
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slidesUnderstanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slidesMohamed Farouk
 
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN Riyaj Shamsudeen
 
Oracle db performance tuning
Oracle db performance tuningOracle db performance tuning
Oracle db performance tuningSimon Huang
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1Tanel Poder
 
Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360Carlos Sierra
 
SQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12cSQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12cTanel Poder
 
TFA Collector - what can one do with it
TFA Collector - what can one do with it TFA Collector - what can one do with it
TFA Collector - what can one do with it Sandesh Rao
 
Oracle AWR Data mining
Oracle AWR Data miningOracle AWR Data mining
Oracle AWR Data miningYury Velikanov
 
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And WhatPerformance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And Whatudaymoogala
 
Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by exampleMauro Pagano
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsJohn Kanagaraj
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the TradeCarlos Sierra
 
Oracle Database SQL Tuning Concept
Oracle Database SQL Tuning ConceptOracle Database SQL Tuning Concept
Oracle Database SQL Tuning ConceptChien Chung Shen
 

What's hot (20)

Low Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling ExamplesLow Level CPU Performance Profiling Examples
Low Level CPU Performance Profiling Examples
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentalsDB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
 
Tanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata MigrationsTanel Poder - Performance stories from Exadata Migrations
Tanel Poder - Performance stories from Exadata Migrations
 
Oracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention TroubleshootingOracle Latch and Mutex Contention Troubleshooting
Oracle Latch and Mutex Contention Troubleshooting
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
 
Chasing the optimizer
Chasing the optimizerChasing the optimizer
Chasing the optimizer
 
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slidesUnderstanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slides
 
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN
 
AWR and ASH Deep Dive
AWR and ASH Deep DiveAWR and ASH Deep Dive
AWR and ASH Deep Dive
 
Oracle db performance tuning
Oracle db performance tuningOracle db performance tuning
Oracle db performance tuning
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
 
Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360Understanding my database through SQL*Plus using the free tool eDB360
Understanding my database through SQL*Plus using the free tool eDB360
 
SQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12cSQL Monitoring in Oracle Database 12c
SQL Monitoring in Oracle Database 12c
 
TFA Collector - what can one do with it
TFA Collector - what can one do with it TFA Collector - what can one do with it
TFA Collector - what can one do with it
 
Oracle AWR Data mining
Oracle AWR Data miningOracle AWR Data mining
Oracle AWR Data mining
 
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And WhatPerformance Tuning With Oracle ASH and AWR. Part 1 How And What
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
 
Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by example
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
 
Oracle Performance Tools of the Trade
Oracle Performance Tools of the TradeOracle Performance Tools of the Trade
Oracle Performance Tools of the Trade
 
Oracle Database SQL Tuning Concept
Oracle Database SQL Tuning ConceptOracle Database SQL Tuning Concept
Oracle Database SQL Tuning Concept
 

Similar to Solving Slow Parallel Loads and Waits in an Exadata Data Warehouse

Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +OracletuningThomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuningafa reg
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11gfcamachob
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01Karam Abuataya
 
Oracle Open World Thursday 230 ashmasters
Oracle Open World Thursday 230 ashmastersOracle Open World Thursday 230 ashmasters
Oracle Open World Thursday 230 ashmastersKyle Hailey
 
Oow2007 performance
Oow2007 performanceOow2007 performance
Oow2007 performanceRicky Zhu
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and ArchitectureSidney Chen
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringNETWAYS
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksKyle Hailey
 
Awrrpt 1 3004_3005
Awrrpt 1 3004_3005Awrrpt 1 3004_3005
Awrrpt 1 3004_3005Kam Chan
 
Oracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspackOracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspackKyle Hailey
 
Debugging Ruby
Debugging RubyDebugging Ruby
Debugging RubyAman Gupta
 
OSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015: Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015: Linux Performance Profiling and Monitoring by Werner FischerNETWAYS
 
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerNETWAYS
 
The Data Center and Hadoop
The Data Center and HadoopThe Data Center and Hadoop
The Data Center and HadoopDataWorks Summit
 

Similar to Solving Slow Parallel Loads and Waits in an Exadata Data Warehouse (20)

Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +OracletuningThomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuning
 
Using AWR for IO Subsystem Analysis
Using AWR for IO Subsystem AnalysisUsing AWR for IO Subsystem Analysis
Using AWR for IO Subsystem Analysis
 
Rmoug ashmaster
Rmoug ashmasterRmoug ashmaster
Rmoug ashmaster
 
Rac 12c optimization
Rac 12c optimizationRac 12c optimization
Rac 12c optimization
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11g
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
 
Oracle Open World Thursday 230 ashmasters
Oracle Open World Thursday 230 ashmastersOracle Open World Thursday 230 ashmasters
Oracle Open World Thursday 230 ashmasters
 
Using Statspack and AWR for Memory Monitoring and Tuning
Using Statspack and AWR for Memory Monitoring and TuningUsing Statspack and AWR for Memory Monitoring and Tuning
Using Statspack and AWR for Memory Monitoring and Tuning
 
Oow2007 performance
Oow2007 performanceOow2007 performance
Oow2007 performance
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoringOSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
OSDC 2017 - Werner Fischer - Linux performance profiling and monitoring
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
Awrrpt 1 3004_3005
Awrrpt 1 3004_3005Awrrpt 1 3004_3005
Awrrpt 1 3004_3005
 
Oracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspackOracle 10g Performance: chapter 00 statspack
Oracle 10g Performance: chapter 00 statspack
 
Debugging Ruby
Debugging RubyDebugging Ruby
Debugging Ruby
 
OSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015: Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015: Linux Performance Profiling and Monitoring by Werner Fischer
 
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner FischerOSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
OSMC 2015 | Linux Performance Profiling and Monitoring by Werner Fischer
 
Iuwne10 S06 L01
Iuwne10 S06 L01Iuwne10 S06 L01
Iuwne10 S06 L01
 
The Data Center and Hadoop
The Data Center and HadoopThe Data Center and Hadoop
The Data Center and Hadoop
 
Using AWR/Statspack for Wait Analysis
Using AWR/Statspack for Wait AnalysisUsing AWR/Statspack for Wait Analysis
Using AWR/Statspack for Wait Analysis
 

More from Tanel Poder

Modern Linux Performance Tools for Application Troubleshooting
Modern Linux Performance Tools for Application TroubleshootingModern Linux Performance Tools for Application Troubleshooting
Modern Linux Performance Tools for Application TroubleshootingTanel Poder
 
GNW01: In-Memory Processing for Databases
GNW01: In-Memory Processing for DatabasesGNW01: In-Memory Processing for Databases
GNW01: In-Memory Processing for DatabasesTanel Poder
 
SQL in the Hybrid World
SQL in the Hybrid WorldSQL in the Hybrid World
SQL in the Hybrid WorldTanel Poder
 
Connecting Hadoop and Oracle
Connecting Hadoop and OracleConnecting Hadoop and Oracle
Connecting Hadoop and OracleTanel Poder
 
Oracle Exadata Performance: Latest Improvements and Less Known Features
Oracle Exadata Performance: Latest Improvements and Less Known FeaturesOracle Exadata Performance: Latest Improvements and Less Known Features
Oracle Exadata Performance: Latest Improvements and Less Known FeaturesTanel Poder
 
Oracle Database In-Memory Option in Action
Oracle Database In-Memory Option in ActionOracle Database In-Memory Option in Action
Oracle Database In-Memory Option in ActionTanel Poder
 
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder Oracle Scripts and Tools (2010)Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder Oracle Scripts and Tools (2010)Tanel Poder
 
Oracle LOB Internals and Performance Tuning
Oracle LOB Internals and Performance TuningOracle LOB Internals and Performance Tuning
Oracle LOB Internals and Performance TuningTanel Poder
 

More from Tanel Poder (8)

Modern Linux Performance Tools for Application Troubleshooting
Modern Linux Performance Tools for Application TroubleshootingModern Linux Performance Tools for Application Troubleshooting
Modern Linux Performance Tools for Application Troubleshooting
 
GNW01: In-Memory Processing for Databases
GNW01: In-Memory Processing for DatabasesGNW01: In-Memory Processing for Databases
GNW01: In-Memory Processing for Databases
 
SQL in the Hybrid World
SQL in the Hybrid WorldSQL in the Hybrid World
SQL in the Hybrid World
 
Connecting Hadoop and Oracle
Connecting Hadoop and OracleConnecting Hadoop and Oracle
Connecting Hadoop and Oracle
 
Oracle Exadata Performance: Latest Improvements and Less Known Features
Oracle Exadata Performance: Latest Improvements and Less Known FeaturesOracle Exadata Performance: Latest Improvements and Less Known Features
Oracle Exadata Performance: Latest Improvements and Less Known Features
 
Oracle Database In-Memory Option in Action
Oracle Database In-Memory Option in ActionOracle Database In-Memory Option in Action
Oracle Database In-Memory Option in Action
 
Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder Oracle Scripts and Tools (2010)Tanel Poder Oracle Scripts and Tools (2010)
Tanel Poder Oracle Scripts and Tools (2010)
 
Oracle LOB Internals and Performance Tuning
Oracle LOB Internals and Performance TuningOracle LOB Internals and Performance Tuning
Oracle LOB Internals and Performance Tuning
 

Recently uploaded

DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPathCommunity
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 

Recently uploaded (20)

DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
UiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to HeroUiPath Community: Communication Mining from Zero to Hero
UiPath Community: Communication Mining from Zero to Hero
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 

Solving Slow Parallel Loads and Waits in an Exadata Data Warehouse

  • 1. 1 © Tanel Poder 2020 blog.tanelpoder.com Troubleshooting Another Complex Performance Issue (Part 4) Tanel Põder https://blog.tanelpoder.com/ @tanelpoder
  • 2. 2 © Tanel Poder 2020 blog.tanelpoder.com A bit of philosophy …
  • 3. 3 © Tanel Poder 2020 blog.tanelpoder.com Your task is slow because It is doing too many operations or The operations complete too slowly or both
  • 4. 4 © Tanel Poder 2020 blog.tanelpoder.com Measure & Profile! 1. Top SQL & wait events 2. Performance counters 3. Oracle kernel functions ASH, sqlmon sql_trace snapper sqlmon pstack perf
  • 5. 5 © Tanel Poder 2020 blog.tanelpoder.com Let's get started!
  • 6. 6 © Tanel Poder 2020 blog.tanelpoder.com A Data Warehouse data loading/preparation • A large Exadata / RAC / Oracle 11.2.0.3 reporting environment • Lots of parallel CTAS and direct path loads • High concurrency, high parallelism • Throughput bad, all kinds of waits showing up: • Other, Cluster, Configuration, Application, Concurrency, User I/O, CPU
  • 7. 7 © Tanel Poder 2020 blog.tanelpoder.com Parallel Degree 48, 96+, inter-instance PX • Mostly CREATE TABLE AS SELECT statements! • Creating pre-aggregated tables, etc
  • 8. 8 © Tanel Poder 2020 blog.tanelpoder.com Drilldown with ASH data – instance-wide • Wait event names are more informative than wait classes: SQL> @ashtop session_state,wait_class,event session_type='FOREGROUND' sysdate-1/24 sysdate %This SESSION WAIT_CLASS EVENT AAS Seconds ------ ------- -------------- ------------------------------ ------ ------- 32% ON CPU 40.6 146197 13% WAITING Cluster gc buffer busy acquire 16.8 60559 11% WAITING Configuration enq: TX - allocate ITL entry 13.9 50115 9% WAITING Cluster gc buffer busy release 11.3 40739 8% WAITING Scheduler resmgr:cpu quantum 9.9 35711 4% WAITING Application enq: TX - row lock contention 5.2 18798 3% WAITING User I/O direct path read temp 4.3 15400 3% WAITING User I/O cell smart table scan 3.7 13259 2% WAITING Concurrency buffer busy waits 3 10735 2% WAITING Other enq: CF - contention 2.1 7443 1% WAITING Application enq: TM - contention 1.7 6109
  • 9. 9 © Tanel Poder 2020 blog.tanelpoder.com Drilldown with ASH data – one SQL: • Let's pick one of the slow CTAS statements: SQL> @ash/dashtop session_state,event,p1text,p1 top_level_sql_id='89apzfvjba3gu' sysdate-2 sysdate %This SESSION EVENT P1TEXT P1 ------ ------- ------------------------------ ------------------ ---------- 36% WAITING gc buffer busy acquire file# 1 20% WAITING gc buffer busy release file# 1 10% WAITING enq: TX - allocate ITL entry name|mode 1415053316 8% WAITING gc buffer busy acquire file# 1 4% WAITING buffer busy waits file# 1 4% WAITING enq: TX - allocate ITL entry name|mode 1415053316 2% WAITING buffer busy waits file# 1 2% WAITING gc buffer busy release file# 1 2% WAITING enq: TX - row lock contention name|mode 1415053316 1% WAITING latch: ges resource hash list address 3.3055E+10 1% ON CPU sleeptime/senderid 268501512 ... Looks like the CTAS statements do directly experience these waits
  • 10. 10 © Tanel Poder 2020 blog.tanelpoder.com Wait event explanation • buffer busy waits • Buffer is physically available in local instance, but is pinned by some other local session (in incompatible mode) • gc buffer busy acquire • Someone has already requested the remote block into local instance • gc buffer busy release • Block is local, but has to be shipped out to a remote instance first (as someone in the remote instance had requested it first) • enq: TX - allocate ITL entry • Can't change block as all the block's ITL entries are held by others • Can't dynamically add more ITLs (no space in block)
  • 11. 11 © Tanel Poder 2020 blog.tanelpoder.com Looks like write-write contention across RAC nodes!
  • 12. 12 © Tanel Poder 2020 blog.tanelpoder.com Contention on what objects, blocks, what SQL? • Let's dig into the top wait event: SQL> @ashtop session_state,wait_class,p1,p2 "event='gc buffer busy acquire'" sysdate-1/24 sysdate %This SESSION WAIT_CLASS P1 P2 AAS TotalSeconds ------ ------- -------------- ---------- ---------- ---------- ------------ 8% WAITING Cluster 1 279634 1.3 4760 6% WAITING Cluster 1 279635 1.1 3916 6% WAITING Cluster 1 279629 1.1 3838 5% WAITING Cluster 1 279632 .9 3146 4% WAITING Cluster 1 279638 .7 2503 ... SQL> @sed "gc buffer busy" EVENT_NAME WAIT_CLASS PARAMETER1 PARAMETER2 PARAMETER3 ----------------------- ----------- ----------- ----------- ----------- gc buffer busy acquire Cluster file# block# class# gc buffer busy release Cluster file# block# class#
  • 13. 13 © Tanel Poder 2020 blog.tanelpoder.com Which object? • Translate file#, block# into segment names/numbers: • Assumes the blocks are in buffer cache SQL> SELECT objd data_object_id, COUNT(*) FROM v$bh WHERE file#=1 AND block# IN ( 279634,279635,279629,279632,279638,279636,279613,279662, 279628,279608,279653,279627,279642,279637,279643,279631, 279646,279622,279582,279649 ) GROUP BY objd ORDER BY 2 DESC; DATA_OBJECT_ID COUNT(*) -------------- ---------- 8 113 All blocks belong to a segment with data_object_id = 8
  • 14. 14 © Tanel Poder 2020 blog.tanelpoder.com What is segment #8? • Look up the object names: • Using DBA_OBJECTS.DATA_OBJECT_ID -> OBJECT_ID SQL> @doid 8 object_id owner object_name O_PARTITION object_type --------- --------- ----------------- ------------- ----------- 8 SYS C_FILE#_BLOCK# CLUSTER 14 SYS SEG$ TABLE 13 SYS UET$ TABLE This segment #8 is a cluster which contains SEG$ (DBA_SEGMENTS) and UET$ (DBA_EXTENTS), but UET$ isn't used anymore thanks to LMT tablespaces. That leaves SEG$
  • 15. 15 © Tanel Poder 2020 blog.tanelpoder.com Write contention on SEG$? • SEG$ is modified when: 1. A new segment is created (table, index, partition, etc) 2. An existing segment extends 3. A segment is dropped / moved 4. Parallel direct path loads (CTAS) can also create many temporary segments (one per PX slave) • …and merge these into final segment in the end of loading More about this later…
  • 16. 16 © Tanel Poder 2020 blog.tanelpoder.com More evidence!
  • 17. 17 © Tanel Poder 2020 blog.tanelpoder.com AWR 1 • No CPU starvation evident (checked all RAC nodes) Top 5 Timed Foreground Events ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Event Waits Time(s) (ms) time Wait Class ---------------------------- ----------- -------- ----- ------ ---------- DB CPU 165,199 37.0 gc buffer busy acquire 1,260,128 68,399 54 15.3 Cluster enq: TX - allocate ITL entry 354,496 40,583 114 9.1 Configurat direct path write temp 4,632,946 37,455 8 8.4 User I/O gc buffer busy release 213,750 22,683 106 5.1 Cluster Host CPU (CPUs: 160 Cores: 80 Sockets: 8) ~~~~~~~~ Load Average Begin End %User %System %WIO %Idle --------- --------- --------- --------- --------- --------- 5.20 42.76 26.0 3.3 0.0 69.8
  • 18. 18 © Tanel Poder 2020 blog.tanelpoder.com AWR 2 • AWR also listed a SEG$ insert as a top SQL: Elapsed Elapsed Time Time (s) Executions per Exec (s) %Total %CPU %IO SQL Id ----------- ------------- ------------- ------ ----- ----- ------------- 100,669.1 4,444 22.65 22.6 .5 .0 g7mt7ptq286u7 insert into seg$ (file#,block#,type#,ts#,blocks,extents,minexts,maxexts, extsize,extpct,user#,iniexts,lists,groups,cachehint,hwmincr, spare1, scanhint, bitmapranges) values (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12, :13,:14,:15,:16,DECODE(:17,0,NULL,:17),:18,:19) SNAP_ID NODE SQL_ID EXECS AVG_ETIME AVG_LIO AVG_PIO ------- ---- ------------- ------ ---------- -------- -------- 24263 3 g7mt7ptq286u7 552 70.227 880.6 .0 • AWR on another node (same insert into seq$): Super-slow single row inserts into SEG$ ???
  • 19. 19 © Tanel Poder 2020 blog.tanelpoder.com SQL Trace confirms SEG$ insert problem: insert into seg$ (file#,block#,type#,ts#,blocks,extents,minexts,maxexts, extsize,extpct,user#,iniexts,lists,groups,cachehint,hwmincr, spare1, scanhint, bitmapranges) values (:1,:2,:3,:4,:5,:6,:7,:8,:9,:10,:11,:12,:13,:14,:15,:16,DECODE(:17,0,NULL, :17),:18,:19) call count cpu elapsed disk query current rows ------- ------ -------- ---------- -------- -------- --------- ------- Parse 0 0.00 0.00 0 0 0 0 Execute 1 0.82 75.14 3 773 1508 1 Fetch 0 0.00 0.00 0 0 0 0 ------- ------ -------- ---------- -------- -------- --------- ------- total 1 0.82 75.14 3 773 1508 1 Row Source Operation --------------------------------------------------- LOAD TABLE CONVENTIONAL (cr=773 pr=3 pw=0 time=75139696 us)
  • 20. 20 © Tanel Poder 2020 blog.tanelpoder.com SEG$ insert wait events: Elapsed times include waiting on following events: Event waited on Times Max. Wait Total Waited ------------------------------------ Waited ---------- ------------ gc buffer busy release 199 1.01 10.76 buffer busy waits 560 1.03 7.02 resmgr:cpu quantum 54 0.04 0.08 enq: TX - row lock contention 93 0.35 4.25 latch: ges resource hash list 400 0.01 0.98 gc buffer busy acquire 464 8.96 36.64 buffer deadlock 158 0.00 0.00 Disk file operations I/O 5 0.00 0.00 gc cr block 2-way 101 0.00 0.01 gc current block 2-way 2 0.00 0.00 cell single block physical read 3 0.00 0.00 gc current block busy 10 0.05 0.14 gc current grant 2-way 2 0.00 0.00 enq: TX - allocate ITL entry 316 0.77 13.18 latch: cache buffers chains 334 0.01 1.45 ... Why does a single row insert cause so many waits?
  • 21. 21 © Tanel Poder 2020 blog.tanelpoder.com Checkpoint: What do we know?
  • 22. 22 © Tanel Poder 2020 blog.tanelpoder.com Checkpoint 1. Lots of parallel CTAS statements, seem to wait for various RAC gc buffer busy events and enq: TX – ITL contention 2. CTAS statements create new segments, new segments cause inserts into SEG$ 3. AWR and SQL Trace report super-long elapsed times for single-row SEG$ inserts 4. It's actually the recursive SEG$ inserts that wait for the gc buffer busy and enq: TX – ITL contention events
  • 23. 23 © Tanel Poder 2020 blog.tanelpoder.com "Hypothesis" Could we be in a cache + transaction layer nested locking ping-pong loop?
  • 24. 24 © Tanel Poder 2020 blog.tanelpoder.com Multiple layers of locking & coherency ITL 2ITL 1 data2data1ccLH c2c1 ITL 4ITL 3 KCBH - block common header KTBBH - transaction common header KDBH - Data header Data block Row Level 0: VOS/service layer - Cache buffers chains latch Level 1: Cache layer Buffer pin - local pin - global cache pin Level 2: Transaction layer - ITL entry - row lock byte You must pin (and retrieve) a buffer before you can change or examine it What if you pin a buffer and find all its ITL slots busy? (with no space to add mode) The session will release the pin and start waiting for enq: TX – ITL contention event And re-get/pin again when TX – ITL wait is over!
  • 25. 25 © Tanel Poder 2020 blog.tanelpoder.com Raw trace file (edited): *** 2013-11-18 21:39:09.760 WAIT #1: nam='buffer busy waits' ela= 1023321 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='gc buffer busy release' ela= 25304 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 4315 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='Disk file operations I/O' ela= 10 FileOperation=2 fileno=141 filetype=2 WAIT #1: nam='gc cr block 2-way' ela= 134 p1=141 p2=5456 p3=3447 obj#=0 WAIT #1: nam='enq: TX - allocate ITL entry' ela= 29295 name|mode=1415053316 WAIT #1: nam='latch: ges resource hash list' ela= 958 address=33046235872 number=91 WAIT #1: nam='gc buffer busy acquire' ela= 3384 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 817 file#=1 block#=279627 class#=1 obj#=14 WAIT #1: nam='gc buffer busy acquire' ela= 344 file#=1 block#=192 class#=4 obj#=14 WAIT #1: nam='gc buffer busy acquire' ela= 2096 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 3922 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='gc cr block 2-way' ela= 124 p1=141 p2=28888 p3=3895 obj#=0 WAIT #1: nam='enq: TX - allocate ITL entry' ela= 9260 name|mode=1415053316 WAIT #1: nam='latch: ges resource hash list' ela= 683 address=33043220640 WAIT #1: nam='gc buffer busy acquire' ela= 13696 file#=1 block#=279656 class#=1 obj#=14 WAIT #1: nam='buffer busy waits' ela= 1312 file#=1 block#=279656 class#=1 obj#=14 ... *** 2013-11-18 21:39:10.933 WAIT #1: nam='gc buffer busy acquire' ela= 1023271 file#=1 block#=277294 class#=1 obj#=9 WAIT #1: nam='buffer deadlock' ela= 15 dba=4471598 class*10+mode=12 flag=8192 obj#=9 What's obj# 9? What's class# 4? obj# 14 is SEG$
  • 26. 26 © Tanel Poder 2020 blog.tanelpoder.com OBJ #9 and #14 • Look up the object names: • Using DBA_OBJECTS.DATA_OBJECT_ID -> OBJECT_ID SQL> @oid 9,14 owner object_name object_type --------------- --------------------- ------------------ SYS I_FILE#_BLOCK# INDEX SYS SEG$ TABLE The object #9 is a cluster index for cluster C_FILE#_BLOCK# (oid #8 on a previous slide SQL> @bclass 4 CLASS UNDO_SEGMENT_ID ------------------ --------------- segment header Bclass #4 is a segment header block that also stores freelist information
  • 27. 27 © Tanel Poder 2020 blog.tanelpoder.com So what did happen?
  • 28. 28 © Tanel Poder 2020 blog.tanelpoder.com Conclusion from gathered evidence 1. Lots of concurrent + highly parallel CTAS statements 2. Each PX slave created its own temporary segments (in spikes when PX query started up) 3. Spikes of concurrent SEG$ inserts simultaneously on 4 RAC nodes 4. SEG$ cluster blocks out of ITL entries 5. This caused global cache/TX locking trashing – long loops of unsuccessful attempts to pin and insert into SEG$ blocks
  • 29. 29 © Tanel Poder 2020 blog.tanelpoder.com Also worth noting 1. Resource manager a possible factor • Plenty of resmgr:cpu quantum waits • Lock holders, buffer pin holders might have gone to sleep – while holding the pins/locks under contention 1. Freelist-based block space allocation • SEG$ is a SYS-owned freelist-managed index cluster • Many cluster table blocks walked through when searching for space 2. ASH doesn't always show recursive SQL IDs • Attributes waits to parent (top level) statement SQL ID instead • ASH p1, p2, p3, current_obj# columns are useful for drilldown
  • 30. 30 © Tanel Poder 2020 blog.tanelpoder.com How to fix it? • Do it less! • What? • Do not insert/update SEG$ entries so frequently! • How? • Do not allow parallel direct load insert to create a temporary loading segment for each slave (and inserted partition)! • How? • Make sure that High Water Mark Brokering is used! • One temporary segment - thus one SEG$ insert/update/delete - per query (and inserted partition) = much less SEG$ contention. • More in following slides …
  • 31. 31 © Tanel Poder 2020 blog.tanelpoder.com Why is High-Water-Mark Brokering needed? • A historical problem with large uniform extent sizes: 1. Each data loading PX slave allocated its own extents (to its private temporary segment) when loading data 2. When the load was completed the private temporary segments were merged to one final table segment 3. The last extent of each private segment possibly ended up not fully used • Some were almost full, some almost empty – half-empty on average 4. Wastage = 0.5 x extent_size x PX_slaves • The more PX slaves, the more wastage • The bigger extent, the bigger wastage • References: • https://blogs.oracle.com/datawarehousing/entry/parallel_load_uniform_or_autoall ocate • http://www.oracle.com/technetwork/database/bi- datawarehousing/twpdwbestpractices-for-loading-11g-404400.pdf Problem Solved by using High-Water-Mark Brokering! But…
  • 32. 32 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – space wastage? SQL> @seg sales_parallel_ctas SEG_MB OWNER SEGMENT_NAME SEG_TABLESPACE_NAME ------- ------- ------------------------- -------------------- 9472 TANEL SALES_PARALLEL_CTAS TANEL_DEMO_LARGE 8896 TANEL SALES_PARALLEL_CTAS_FIX TANEL_DEMO_LARGE SQL>SELECT * FROM TABLE(space_tools.get_space_usage('TANEL', 'SALES_PARALLEL_CTAS','TABLE')); --------------------------------------------------------------------------------------- Full blocks /MB 1134669 8865 Unformatted blocks/MB 0 0 Free Space 0-25% 0 0 Free Space 25-50% 0 0 Free Space 50-75% 0 0 Free Space 75-100% 72963 570 SQL>SELECT * FROM TABLE(space_tools.get_space_usage('TANEL', 'SALES_PARALLEL_CTAS_FIX’,'TABLE')) ---------------------------------------------------------------------------------------- Full blocks /MB 1134669 8865 Unformatted blocks/MB 0 0 Free Space 0-25% 0 0 Free Space 25-50% 0 0 Free Space 50-75% 0 0 Free Space 75-100% 0 0 An identical table PX CTAS in the same tablespace with 64MB uniform extent (upper table 6.4% bigger) 570 MB worth of blocks unused in the segment? PX DOP 16 x 64 * ~0.5 = 512MB Same table loaded with INSERT /*+ APPEND */ PX DOP 16. No empty blocks
  • 33. 33 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – HWM brokering • Instead of each PX slave working on their own separate segment… 1. Allocate and extend only one segment 2. Slaves allocate space within the single segment as needed 3. No “holes” in tables, no space wastage problem anymore! 4. Serialized via the HV - Direct Loader High Water Mark enqueue • Parameter: • _insert_enable_hwm_brokered = true (default) • “during parallel inserts high water marks are brokered” • Problem: • In Oracle 11.2 (including 11.2.0.4) this only works with INSERT APPENDs • …and not for CTAS nor ALTER TABLE MOVE • (in one case it worked for CTAS, only if the target table was partitioned)
  • 34. 34 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading and Large Extents – HWM brokering SQL> @fix 6941515 BUGNO VALUE DESCRIPTION IS_DEFAULT ------- ------ ------------------------------------------------------------ ---------- 6941515 0 use high watermark brokering for insert into single segment 1 SQL> alter session set "_fix_control"='6941515:ON'; Session altered. • Actually this is a bug, and Oracle has fixed it long time ago • But bugfix 6941515 is not enabled by default! • After enabling the bug-fix, parallel CTAS and parallel ALTER TABLE MOVE also use HWM brokering… • Unfortunately it didn’t work with ALTER TABLE MOVE if moving the whole table if the table was partitioned. DBMS_SPACE / space_tools helps to measure this! Thanks to Alex Fatkulin for spotting this!
  • 35. 35 © Tanel Poder 2020 blog.tanelpoder.com Additional slides (we are probably out of time)
  • 36. 36 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Data Loading Performance – Example 1 • Parallel Create Table As Select from Hadoop to Exadata • Buffer busy waits dominating the response time profile (SQL Monitoring) https://blog.tanelpoder.com/2013/11/06/diagnosing-buffer-busy-waits-with-the-ash_wait_chains-sql-script-v0-2/
  • 37. 37 © Tanel Poder 2020 blog.tanelpoder.com Data Loading Performance: ashtop.sql SQL> @ash/ashtop session_state,event,p2text,p2,p3text,p3 sql_id='3rtbs9vqukc71' "timestamp'2013-10-05 01:00:00'" "timestamp'2013-10-05 03:00:00'" %This SESSION EVENT P2TEXT P2 P3TEXT P3 ------ ------- --------------------------------- --------- -------- -------- ------- 57% WAITING buffer busy waits block# 2 class# 13 31% ON CPU file# 0 size 524288 1% WAITING external table read file# 0 size 524288 1% ON CPU block# 2 class# 13 0% ON CPU consumer 12573 0 0% WAITING cell smart file creation 0 0 0% WAITING DFS lock handle id1 3 id2 2 0% ON CPU file# 41 size 41 0% WAITING cell single block physical read diskhash# 4695794 bytes 8192 0% WAITING control file parallel write block# 1 requests 2 0% WAITING control file parallel write block# 41 requests 2 0% WAITING change tracking file synchronous blocks 1 0 0% WAITING control file parallel write block# 42 requests 2 0% WAITING db file single write block# 1 blocks 1 0% ON CPU 0 0 • Break the (buffer busy) wait events down by block#/class# block #2 ?
  • 38. 38 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Data Loading Performance – Example 2 • Lots of serial sessions doing single row inserts – on multiple RAC nodes • “buffer busy waits” and “gc buffer busy acquire” waits • file# = 6 • block# = 2 • class# = 13 SQL> @bclass 13 CLASS UNDO_SEGMENT_ID ------------------ --------------- file header block SQL> select file#, block#, status from v$bh where class# = 13; FILE# BLOCK# STATUS ---------- ---------- ---------- 5 2 xcur 4 2 xcur ... Block dump from disk: buffer tsn: 7 rdba: 0x00000002 (1024/2) scn: 0x0000.010b6f9b seq: 0x02 flg: 0x04 tail: 0x6f9b1d02 frmt: 0x02 chkval: 0xd587 type: 0x1d=KTFB Bitmapped File Space Header Hex dump of block: st=0, typ_found=1 A single space allocation contention point per LMT file. Bigfile tablespaces have only one file!
  • 39. 39 © Tanel Poder 2020 blog.tanelpoder.com Parallel Data Loading Performance: ash_wait_chains.sql SQL> @ash/ash_wait_chains event2 sql_id='3rtbs9vqukc71' "timestamp'2013-10-05 01:00:00'" "timestamp'2013-10-05 03:00:00'” %This SECONDS WAIT_CHAIN ------ ---------- ---------------------------------------------------------------------------- 35% 43735 -> ON CPU 15% 18531 -> buffer busy waits [file header block] -> ON CPU 7% 9266 -> buffer busy waits [file header block] -> control file parallel write 7% 8261 -> buffer busy waits [file header block] -> cell smart file creation 6% 6959 -> direct path write 5% 6707 -> buffer busy waits [file header block] -> DFS lock handle 4% 4658 -> buffer busy waits [file header block] -> local write wait 4% 4610 -> buffer busy waits [file header block] -> cell single block physical read 3% 4282 -> buffer busy waits [file header block] -> local write wait -> db file par 2% 2801 -> buffer busy waits [file header block] 2% 2676 -> buffer busy waits [file header block] -> ASM file metadata operation 2% 2092 -> buffer busy waits [file header block] -> change tracking file synchronous 2% 2050 -> buffer busy waits [file header block] -> control file sequential read The waiting session(s) The blocking session(s) Block type if buffer busy wait • Display top wait chains from ASH (who’s waiting for whom)
  • 40. 40 © Tanel Poder 2020 blog.tanelpoder.com Case Study: Parallel Data Loading Performance • Reduce demand on the LMT bitmap blocks • By allocating bigger extents at a time • Use large uniform extents for fast-growing tables(paces) • The customer went with 64MB uniform extent size • The autoallocate extent management is suboptimal for very large segments • As there’d be 1 LMT space management bit per 64kB regardless of your INITIAL extent size at the segment level • The _partition_large_extents = TRUE doesn’t change this either • Large uniform extents are better for data loading and scanning!
  • 41. 41 © Tanel Poder 2020 blog.tanelpoder.com Thanks!!! https://blog.tanelpoder.com/seminar tanel@tanelpoder.com @tanelpoder