SlideShare a Scribd company logo
1 of 56
Download to read offline
RAC 12c optimization 
By 
Riyaj Shamsudeen 
©OraInternals Riyaj Shamsudeen
©OraInternals Riyaj Shamsudeen 2 
Me 
 20+ years using Oracle products/DBA 
 OakTable member, Oracle ACE Director 
 Certified DBA -7.0,7.3,8,8i,9i &10g 
 Specializes in RAC, performance 
tuning, EBS, & Exadata 
 Chief DBA with OraInternals 
 Co-author of a few books 
 Email: rshamsud@orainternals.com 
 Blog : orainternals.wordpress.com 
 Web: www.orainternals.com
Process architecture
RAC message flow
RAC cluster root causes 
 CPU / Memory starvation in a node. 
 Log file write/ LGWR slowness. 
 Network issues / packet drops. 
 Excessive Dynamic Remastering activities. 
 Excessive amount of global cache traffic- due to workload or 
execution plan regression. 
 Localized inserts in to a few blocks. 
 Localized, excessive accesses to a few blocks. 
©OraInternals Riyaj Shamsudeen
Golden rules of RAC diagnostics 
1. Beware of top event tunnel vision. 
2. Eliminate infrastructure as an issue. 
3. Identify problem instance(s). 
4. Review Send side metrics, and not just Receive side metrics. 
5. Use histograms to identify any abnormalities, and not just 
averages. 
©OraInternals Riyaj Shamsudeen
 Impact of side effect events can be much, much worse then 
actual root cause event itself. 
©OraInternals Riyaj Shamsudeen 
Top event tunnel vision 
 In RAC, the root cause can be hidden in other node and might 
not show up in the Top-5 wait events. 
 Performance of the instances are intertwined. 
 Performance of background processes in one node, can affect 
the application performance in the all other nodes.
Top event tunnel vision ..2 
©OraInternals Riyaj Shamsudeen 8 
Example 1
Top event tunnel vision ..3 
©OraInternals Riyaj Shamsudeen 9 
Example 2
Top event tunnel vision ..4 
©OraInternals Riyaj Shamsudeen 10 
Example 3
©OraInternals Riyaj Shamsudeen 11 
Problem node 
Example 3 (continued)
©OraInternals Riyaj Shamsudeen 
gc buffer busy 
gc current request 
FG1 
LMS 
LGWR 
Open request 
File #5, blk #8 
Flush request Post 
Block 
send 
FG2 
A BL request for file #5, 
block #8 is pending. gc buffer busy acquire 
gc buffer busy release 
FG3 
Inst 2 Inst 1 
charge time to actual 
wait event. 
Wire
©OraInternals Riyaj Shamsudeen 
gc buffer busy 
 Event ‘gc buffer busy [acquire|release]’ can be the symptoms for 
all the RAC performance root causes discussed earlier. 
 Gc buffer busy wait simply means that there is a pending request 
for a BL lock for that block already. 
 All other process wait for the BL lock to be available and charge 
the time to gc buffer busy event.
Gc buffer busy – Objects related 
Following four pages only applicable if there are no 
other events indicating systemic issues. 
 ASH or trace files can be used to identify the blocks 
suffering from excessive ‘gc buffer busy’ waits. 
 Only use this method if there is no other issue affecting 
background processes. 
 To understand gc buffer busy waits: 
 Identify the object and object type 
 Identify the block type 
 ash_gcwait_to_obj.sql can be used to identify the object 
name and type. 
©OraInternals Riyaj Shamsudeen 14
Gc buffer busy – Identify object 
©OraInternals Riyaj Shamsudeen 15 
@ash_gcwait_to_obj.sql 
INST_ID EVENT OWNER OBJECT_NAME OBJECT_TYPE CNT 
---------- ------------------------ ------ ---------------- ---------------- 
1 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 3 
2 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 10 
2 gc buffer busy release RS T_GEN_INS_01 TABLE 4
Gc buffer busy – Identify block 
@ash_gcwait_to_block.sql 
INST_ID EVENT CURRENT_FILE# CURRENT_BLOCK# CNT 
---------- ------------------------------ ------------- -------------- ---------- 
2 gc buffer busy acquire 4 103582 4 
2 gc buffer busy acquire 4 103607 3 
2 gc buffer busy acquire 4 103603 3 
1 gc buffer busy acquire 4 103582 2 
2 gc buffer busy release 4 103615 2 
1 gc buffer busy acquire 4 103583 1 
2 gc buffer busy release 4 103586 1 
2 gc buffer busy release 4 103613 1 
In this example, many different blocks are involved in ‘gc buffer 
busy’ waits. A symptom of heavy insert load. 
©OraInternals Riyaj Shamsudeen 16 
Demo: ash_gcwait_to_block.sql
Gc buffer busy – Identify block type 
 You might need to dump the block to identify the type and 
contents of the block. 
alter system dump datafile 4 block min 103582 block max 103582; 
 In this example, it is a leaf block of an index. 
Block header dump: 0x0101949e 
Object id on Block? Y 
seg/obj: 0x13266 csc: 0x00.11c65d1 itc: 9 flg: E typ: 2 - INDEX 
brn: 0 bdba: 0x1019498 ver: 0x01 opc: 0 
inc: 0 exflg: 0 
Itl Xid Uba Flag Lck Scn/Fsc 
0x01 0x0008.001.0000144f 0x00c00a1f.058a.01 -BU- 1 fsc 0x0000.011c65d4 
0x02 0x0008.015.00001449 0x00c00a37.058a.56 ---- 55 fsc 0x0000.00000000 
0x03 0x0004.008.000016c0 0x00c01232.045c.18 ---- 90 fsc 0x0000.00000000 
0x04 0x0002.005.000015c1 0x00c005d1.0402.12 ---- 73 fsc 0x0000.00000000 
0x05 0x000a.01c.0000158b 0x00c00849.0446.05 ---- 75 fsc 0x0000.00000000 
Leaf block dump 
=============== 
©OraInternals Riyaj Shamsudeen 17 
Demo: dump_blocks.sql, tracefile.sql
©OraInternals Riyaj Shamsudeen 18 
Object 
type 
Block type Possible issue(s) 
Table Segment header Freelists, freelist groups, deletes from 
one node,ASSM bugs etc 
Segment blocks Heavy insert workload, heavy update to 
few blocks, SQL performance issues 
scanning few objects aggressively. 
Index Leaf block Numerous inserts on indexed columns, 
sequence generated keys 
Root block/ 
branch block 
Insert + numerous index scans, 
concurrent inserts and deletes etc 
Undo Undo header 
block 
Numerous short transactions 
Undo block Long pending transaction and CR block 
generation 
Dictionary SEQ$ (object) Sequences with nocache or order set 
and excessive access to sequence.
Golden rules of RAC diagnostics 
1. Beware of top event tunnel vision. 
2. Eliminate infrastructure as an issue. 
3. Identify problem instance(s). 
4. Review Send side metrics, and not just Receive side metrics. 
5. Use histograms to identify any abnormalities, and not just 
averages. 
©OraInternals Riyaj Shamsudeen
©OraInternals Riyaj Shamsudeen 
gc current block 2-way 
gc current request 
FG1 
LMS 
Open request 
File #5, blk #8 
Block 
send 
Inst 2 Inst 1 
charge time to 
‘gc current block 2-way’ 
Wire
Eliminate infra-structure as an issue 
©OraInternals Riyaj Shamsudeen 21
Eliminate infra-structure as an issue 
 Wait events ‘gc cr block 2-way’ , ‘gc cr block 3-way’ , ‘gc current 
block 2-way’, and ‘gc current block 3-way’, ‘gc cr grant 2-way’, ‘gc 
cr grant 2-way’. 
 Concurrency or congestion issues are not factored in to these 
events. 
 The lowest wait time that you can possibly get for a block or a 
message transfer. 
 A wait time of <2ms is normal. In Exadata, <0.5ms 
©OraInternals Riyaj Shamsudeen 22
Golden rules of RAC diagnostics 
1. Beware of top event tunnel vision. 
2. Eliminate infrastructure as an issue. 
3. Identify problem instance(s). 
4. Review Send side metrics, and not just Receive side metrics. 
5. Use histograms to identify any abnormalities, and not just 
averages. 
©OraInternals Riyaj Shamsudeen
Identify problem-inducing instance 
 Gv$instance_cache_transfer keeps track of the RX metrics at 
both instance and class level. 
©OraInternals Riyaj Shamsudeen 24
Identify problem inducing instance ..2 
SELECT INSTANCE ||'->’ || inst_id transfer, 
class, cr_block cr_blk, 
Trunc(cr_block_time / cr_block / 1000, 2) avg_Cr, 
current_block cur_blk, 
Trunc(current_block_time / current_block / 1000, 2) avg_cur 
FROM gv$instance_cache_transfer 
WHERE cr_block > 0 AND current_block > 0 
ORDER BY INSTANCE, inst_id, class 
/ 
Receiving side analysis. 
TRANS CLASS CR_BLK AVG_CR CUR_BLK AVG_CUR 
----- ---------- ---------- ---------- ---------- ---------- 
1->2 data block 87934887 1.23 9834152 1.8 
2->1 data block 28392332 1.30 1764932 2.1 
... 
3->1 data block 12519985 11.57 2231921 21.6 
... 
3->2 undo block 4676398 8.85 320 27.82 
 Use script gc_identify_slow_inst.sql 
©OraInternals Riyaj Shamsudeen 25
Golden rules of RAC diagnostics 
1. Beware of top event tunnel vision. 
2. Eliminate infrastructure as an issue. 
3. Identify problem instance(s). 
4. Review Send side metrics, and not just Receive side metrics. 
5. Use histograms to identify any abnormalities, and not just 
averages. 
©OraInternals Riyaj Shamsudeen
©OraInternals Riyaj Shamsudeen 
Send side in an oval 
gc current request 
FG1 
LMS 
LGWR 
Open request 
File #5, blk #8 
Flush request Post 
Block 
send 
FG2 
A BL request for file #5, 
block #8 is pending. gc buffer busy acquire 
gc buffer busy release 
FG3 
Inst 2 Inst 1 
charge time to actual 
wait event. 
Wire
RX side 
Send side 
©OraInternals Riyaj Shamsudeen 28 
Sending side analysis 
gc cr block receive time= 
Time to send message to a remote LMS process by FG 
+ Time taken by LMS to build block (statistics: gc cr block build time) 
+ LMS wait for LGWR latency ( statistics: gc cr block flush time) 
+ LMS send time (Statistics: gc cr block send time) 
+ Wire latency. 
Instance 1: 
---------- 
Avg global cache cr block receive time (ms) : 222.1 
Avg global cache current block receive time (ms) : 27.5 
Instance 2: 
---------- 
Avg global cache cr block build time (ms) : 0.0 
Avg global cache cr block send time (ms) : 0.1 
Global cache log flushes for cr blocks served % : 2.7 
Avg global cache cr block flush time (ms) : 15879.9
©OraInternals Riyaj Shamsudeen 29 
Problem node 
Global Cache and Enqueue Services - Workload Characteristics 
Avg global enqueue get time (ms) : 0.1 
Avg global cache cr block receive time (ms) : 14.4 
Avg global cache current block receive time (ms): 23.4 
Avg global cache cr block build time (ms) : 0.0 
Avg global cache cr block send time (ms) : 0.0 
Global cache log flushes for cr blocks served %: 25.0 
Avg global cache cr block flush time (ms) : 2.5 
Avg global cache current block pin time (ms) : 0.0 
Avg global cache current block send time (ms): 0.0 
Global cache log flushes for current blocks served %: 6.3 
Avg global cache current block flush time (ms): 5.9
Sending side analysis ..2 
Avg message sent queue time (ms) : 0.1 
Avg message sent queue time on ksxp (ms): 0.6 
Avg message received queue time (ms): 0.0 
Avg GCS message process time (ms) : 0.0 
Avg GES message process time (ms) : 0.0 
% of direct sent messages : 46.62 
% of indirect sent messages : 43.76 
% of flow controlled messages : 9.62 
Avg message sent queue time (ms) : 0.0 
Avg message sent queue time on ksxp (ms): 0.4 
Avg message received queue time (ms) : 0.0 
Avg GCS message process time (ms) : 0.0 
Avg GES message process time (ms) : 0.0 
% of direct sent messages : 60.86 
% of indirect sent messages : 38.48 
% of flow controlled messages : 0.66 
©OraInternals Riyaj Shamsudeen 30
©OraInternals Riyaj Shamsudeen 31 
Problem node 
Global Cache and Enqueue Services - Messaging Statistics 
Avg message sent queue time (ms) : 344.7 
Avg message sent queue time on ksxp (ms): 0.3 
Avg message received queue time (ms): 0.0 
Avg GCS message process time (ms) : 0.0 
Avg GES message process time (ms) : 0.0 
% of direct sent messages : 31.76 
% of indirect sent messages : 46.12 
% of flow controlled messages : 22.13
©OraInternals Riyaj Shamsudeen 32 
Multi-tenancy 
 Buffer cache and Global cache processing is common to all 
pluggable databases. 
 A PDB generating more global cache workload can cause 
performance issues to other PDBs. 
 There doesn’t seem to be a way to constrain a database from 
overusing buffer cache resource or global cache resources. 
 Resource manager only controls CPU, PX, and I/O(exadata).
Prepare for multi-tenancy (RAC specific) 
 Keep the buffer cache as large (for OLTP). 
 Global cache overhead is reduced if the buffers are in the local 
buffer cache. 
 Make LGWR efficient. LMS waits for LGWR to ship blocks. 
 Tune the SQL statements to reduce global cache traffic. A 
misbehaving piece of application code affects the whole cluster. 
 Connect to the container database to review global cache metrics. 
©OraInternals Riyaj Shamsudeen 33
©OraInternals Riyaj Shamsudeen 34 
Resource names 
 From 12 onwards, resource names are coded with con_id. 
select resource_name from gv$ges_resource 
where resource_name like '[0x15f29][0x0],[TM]%' 
/ 
RESOURCE_NAME 
------------------------------ 
[0x15f29][0x0],[TM][ext 0x3,0x0] <- hrdb1 PDB – GSTEST table –object_id=89897 
[0x15f29][0x0],[TM][ext 0x4,0x0] <- hrdb2 PDB – GSTEST table –object_id=89897 
 Comes handy if you are debugging RAC trace files, to identify 
the PDB generating the errors.
©OraInternals Riyaj Shamsudeen 35 
GC statistics and PDBs 
 Most Global cache statistics views do not provide granularity to a 
pluggable database. For example, 
select con_id, inst_id, count(*) from gv$instance_cache_transfer 
group by con_id, inst_id 
/ 
CON_ID INST_ID COUNT(*) 
---------- ---------- ---------- 
0 2 4096 
0 1 4096 
 Review AWR reports from the container database to review 
global cache performance issues.
High Priority Processes (from 12c) 
Parameter: _high_priority_processes 
Value : LMS*|VKTM|LM*|LCK0|GCR*|DIAG 
… 
kjfmPriCheck: 0 RTLMS taken, 1 RTLMS available 
kjfmPriCheck: automatic priority changes enabled - multi-inst mode 
kjfmPriCheck: # of real time LMS increasing from 0 to 1 
… 
Parameter: _lm_lms_priority_dynamic 
Value : TRUE 
©OraInternals Riyaj Shamsudeen 36
©OraInternals Riyaj Shamsudeen 
gc cr grants 2-way 
gc cr request 
FG1 
LMS 
Open request 
File #5, blk #8 
Read from 
the disk 
Inst 2 Inst 1 
charge time to 
‘gc cr grant 2-way’ 
Wire
Event: GC cr grants 2-way 
 Time is accounted to this wait event, if the block is not in any of 
the buffer cache. 
 Trace file will indicate this wait event followed by a disk read. 
nam='gc cr grant 2-way' ela= 659 p1=1 p2=88252 p3=1 obj#=77779 
nam='db file sequential read' ela= 938 file#=1 block#=88252 blocks=1 obj#=77779 
 Typical latency is 1-2ms. Any thing above needs to be reviewed as 
these are light-wait events. 
 Process sends a request to remote master LMS process and the 
LMS process simply responds with ‘read from disk’. 
 This is another base line wait event to measure interconnect 
response time, as LMS processing for this event is minimal. 
©OraInternals Riyaj Shamsudeen 38
©OraInternals Riyaj Shamsudeen 
Why DRM? 
FG1 
LMS 
Read from 
the disk 
Inst 2 Inst 1 
charge time to 
‘gc cr grant 2-way’ 
Excessive amount of GC activity for 
One object!
©OraInternals Riyaj Shamsudeen 40 
DRM & grants 
 Dynamic Remastering feature is to reduce or eliminate this 
specific RAC Tax: gc cr grants 2-way 
 If the object is mastered locally, processes can read the block 
directly without incurring any GC waits. 
 Identify the objects suffering from excessive ‘gc cr grants 2-way’ 
events (and remaster them manually if needed) using script 
ash_gcwait_to_obj.sql. 
INST_ID OWNER OBJECT_NAME OBJECT_TYPE CNT 
---------- -------------------- -------------------------------- -------------------- ---------- 
1 AR RA_CUSTOMER_TRX_ALL TABLE 9 
1 INV MTL_MATERIAL_TRANSACTIONS TABLE 141 
1 INV MTL_SERIAL_NUMBERS TABLE 155 
1 Undo Block 253 
1 INV MTL_SYSTEM_ITEMS_B TABLE 476 
1 AP AP_CHECKS_ALL TABLE 785
©OraInternals Riyaj Shamsudeen 41 
Local vs Remote 
 In AWR report, identify the local vs remote grants also. 
Following shows that 74% of grants are remote. DRM might be 
of help here, if there are much waits for ‘gc grants 2-way’. 
gc local grants 6,570,658 3,643.6 21.8 
gc remote grants 19,262,806 10,681.7 63.9 
 Increase SGA size to reduce single block I/O reads. 
 Tuning SQL statements to reduce logical I/O, optimal indexes, 
etc can help to improve the performance.
©OraInternals Riyaj Shamsudeen 42 
DRM effect 
Top 5 Timed Events Avg %Total 
~~~~~~~~~~~~~~~~~ wait Call 
Event Waits Time (s) (ms) Time Wait Class 
------------------------------ ------------ ----------- ------ ------ ---------- 
gc buffer busy 1,826,073 152,415 83 62.0 Cluster 
CPU time 30,192 12.3 
enq: TX - index contention 34,332 15,535 453 6.3 Concurrency 
gcs drm freeze in enter server 22,789 11,279 495 4.6 Other 
enq: TX - row lock contention 46,926 4,493 96 1.8 Applicatio 
Global Cache and Enqueue Services - Workload Characteristics 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
Avg global enqueue get time (ms) : 16.8 
Avg global cache cr block receive time (ms) : 17.1 
Avg global cache current block receive time (ms): 14.9
©OraInternals Riyaj Shamsudeen 43 
DRM (11g) 
LCK 
Request queue 
LMD 
LMON 
LMS0 LMS1 LMSn 
LMD 
LMON 
LMS0 LMS1 LMSn 
LMD 
LMON 
LMS0 LMS1 LMSn 
Inst 1 Inst 2 Inst 3 
Resources 
In 11g, all resources are frozen 
during the reconfiguration.
©OraInternals Riyaj Shamsudeen 44 
DRM (12c) 
LCK 
Request queue 
LMD 
LMON 
LMS0 LMS1 LMSn 
LMD 
LMON 
LMS0 LMS1 LMSn 
LMD 
LMON 
LMS0 LMS1 LMSn 
Inst 1 Inst 2 Inst 3 
Resources 
In 12c, only set of resources in 
a window is frozen.
DRM impact is decreased in 12c by freezing a resource partition, at a time.. 
©OraInternals Riyaj Shamsudeen 45 
DRM in 12c 
* drm quiesce 
2014-02-15 16:03:45.131840 : DRM(3) resources quiesced [0-1023], rescount 1719 
2014-02-15 16:03:45.132173 : DRM(3) local converts quiesced [0-1023], lockcount 0, bucket 0 
* drm sync 1 
* drm freeze 
* drm cleanup 
* drm sync 2 
* drm replay 
* drm sync 3 
* drm fix writes 
* drm sync 4 
* drm end 
* DRM RCFG called (swin 0) 
CGS recovery timeout = 85 sec 
* drm quiesce 
2014-02-15 16:03:45.172867 : DRM(3) resources quiesced [1024-2047], rescount 1754 
2014-02-15 16:03:45.172932 : DRM(3) local converts quiesced [1024-2047], lockcount 0, bucket 0 
* drm sync 1 
* drm freeze
©OraInternals Riyaj Shamsudeen 46 
PQ trace file 
 PQ Co-ordinator trace shows Object checkpoint. 
select /*+ parallel (a 4) */ min(length(v1)) from rs.hugetable a; 
WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 866 name| 
mode=1263468550 2=65581 0=1 obj#=-1 tim=6965638229 
... 
WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 454 name| 
mode=1263468545 2=65581 0=2 obj#=-1 tim=6965638986 
... 
 PQ server process will perform direct reads to PGA. Notice that 
there is no GC waits between the reads. 
WAIT #18446741324874985288: nam='direct path read' ela= 4782 file number=4 first dba=11584 block 
cnt=16 obj#=76739 tim=3653439149 
WAIT #18446741324874985288: nam='direct path read' ela= 16157 file number=4 first dba=11600 
block cnt=16 obj#=76739 tim=3653456065 
WAIT #18446741324874985288: nam='direct path read' ela= 3675 file number=4 first dba=11616 block 
cnt=16 obj#=76739 tim=3653462065 
WAIT #18446741324874985288: nam='direct path read' ela= 8162 file number=4 first dba=11632 block 
cnt=16 obj#=76739 tim=3653471358 
WAIT #18446741324874985288: nam='direct path read' ela= 10270 file number=4 first dba=11650 
block cnt=14 obj#=76739 tim=3653482566
©OraInternals Riyaj Shamsudeen 47 
Application issues 
 There is no easy fix in 12c either  
 Avoid localized inserts.
©OraInternals Riyaj Shamsudeen 48 
Application issues 
 Hash partitioning the index/table might be helpful.
©OraInternals Riyaj Shamsudeen 49 
Application issues 
 Reduce localized accesses: 
 Frequent access to a table block such as a lookup table: Cache 
it somewhere else. 
 Aggressive scanning of smaller tables due to inefficient 
optimizer plans. 
 Aggressive nested loops joins accessing huge number of 
index leaf blocks. 
 Ordered cached sequences – Excessive SV resource contention. 
 Excessive DDL in a multi-tenant databases.
©OraInternals Riyaj Shamsudeen 50 
Use Services 
Inst 1 
PO 
FIN 
Inst 2 
PO 
FIN 
Inst 3 
PO 
FIN 
 While creating a service, you 
define a preferred and available 
instances. 
PO FIN 
Preferred 
Available 
For PO service: Instance 1 & 2 are 
preferred instances, and instance 
3 is an available instance.
PDB1 
©OraInternals Riyaj Shamsudeen 51 
Or PDBs 
Inst 1 
PDB1 
PDB2 
Inst 2 
Inst 3 
 PDB1 is open in inst1 & inst2. 
 PDB2 is open in inst2 & inst3 
PO FIN 
Online 
Failover 
PDB1 
PDB2 
PDB2
Statistics: gc lost blocks 
 This statistics tracks dropped interconnect packets. 
 For 8K block size, block transfer would require 6 packets 
of size ~1500 bytes (non-Jumbo) to be transferred. 
 Even if a packet is dropped, then the block need to be 
resent. 
 Requesting process does not know the state of the request 
as UDP layer can not reassemble packet. (UDP is a send 
and forget type of protocol.) 
 After a timeout (0.5s in 11g, ~5s in 10g) requesting 
process will declare that packet as lost, and request the 
block again. 
©OraInternals Riyaj Shamsudeen 52
Statistics: gc lost blocks - Effect 
 If the packets are lost, then the sessions can be seen waiting for 
an idle event. 
 So, higher wait times for gc cr request, gc current request, 
and gc cr multiblock request will be visible. 
 Since many blocks are requested during multi block reads, 
waits for ‘gc cr multiblock request’ will be a prominent 
wait event. 
 Ideally, should be 0. But, usually, few drops are acceptable. 
You would need to judge the impact of lost block 
reviewing other wait events. 
©OraInternals Riyaj Shamsudeen 53
Statistics: gc lost blocks - reason 
 In a nutshell, packet is dropped  
 We need to probe and understand why the packets are 
dropped. 
 Few common reasons are: (In-depth discussion in Network) 
 Insufficient UDP buffer space. 
 High CPU usage. 
 Incorrect network configuration so packets thrown away. 
 Route between the nodes has many hops. 
 Switch issues- configuration, firmware etc 
©OraInternals Riyaj Shamsudeen 54
©OraInternals Riyaj Shamsudeen 55 
HugePages 
 By default, pagesize is 4KB in Linux. PageTables size will be 
huge, in the order of, GigaBytes. 
 For huge SGA sizes, this is problematic. kswapd deamon will 
consume more CPU during memory starvation and lead to server 
hung situations. 
 CRS might not restart the server in time, this can lead to other 
nodes suffering from excessive gc buffer busy waits. 
 HugePages uses a pagesize of 2MB. This reduces the size of 
PageTable to few 100MBs. In RAC & Linux, you should consider 
using HugePages.
©OraInternals Riyaj Shamsudeen 56 
THANK YOU 
 Email: rshamsud@orainternals.com 
 Blog : orainternals.wordpress.com 
 Web: www.orainternals.com

More Related Content

What's hot

Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsEnkitec
 
Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by exampleMauro Pagano
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfOracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfSrirakshaSrinivasan2
 
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresPerformance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresJitendra Singh
 
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionOracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionMarkus Michalewicz
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationThe Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationMarkus Michalewicz
 
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder
 
What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1Satishbabu Gunukula
 
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionOracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionMarkus Michalewicz
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMarkus Michalewicz
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder
 
How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsHow to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsSandesh Rao
 
Oracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best PracticesOracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best PracticesBobby Curtis
 
Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil Nair
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Kyle Hailey
 
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAAchieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAMarkus Michalewicz
 
Oracle db performance tuning
Oracle db performance tuningOracle db performance tuning
Oracle db performance tuningSimon Huang
 

What's hot (20)

Oracle Performance Tuning Fundamentals
Oracle Performance Tuning FundamentalsOracle Performance Tuning Fundamentals
Oracle Performance Tuning Fundamentals
 
Oracle statistics by example
Oracle statistics by exampleOracle statistics by example
Oracle statistics by example
 
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdfOracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
 
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and UnderscoresPerformance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and Underscores
 
AWR and ASH Deep Dive
AWR and ASH Deep DiveAWR and ASH Deep Dive
AWR and ASH Deep Dive
 
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 VersionOracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
 
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - PresentationThe Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
 
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools shortTanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools short
 
What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1
 
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion EditionOracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion Edition
 
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19cMAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
 
One PDB to go, please!
One PDB to go, please!One PDB to go, please!
One PDB to go, please!
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
 
How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata EnvironmentsHow to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata Environments
 
Oracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best PracticesOracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best Practices
 
AWR & ASH Analysis
AWR & ASH AnalysisAWR & ASH Analysis
AWR & ASH Analysis
 
Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016
 
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
 
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAAAchieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAA
 
Oracle db performance tuning
Oracle db performance tuningOracle db performance tuning
Oracle db performance tuning
 

Viewers also liked

Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACExpert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACSolarWinds
 
Presentation capacity management for oracle exadata database machine v2
Presentation   capacity management for oracle exadata database machine v2Presentation   capacity management for oracle exadata database machine v2
Presentation capacity management for oracle exadata database machine v2xKinAnx
 
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4Fran Navarro
 
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Markus Michalewicz
 
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015Nelson Calero
 
Architecture of exadata database machine – Part II
Architecture of exadata database machine – Part IIArchitecture of exadata database machine – Part II
Architecture of exadata database machine – Part IIParesh Nayak,OCP®,Prince2®
 
Exadata x4 for_sap
Exadata x4 for_sapExadata x4 for_sap
Exadata x4 for_sapFran Navarro
 
Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2Rodrigo Almeida
 
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1jenkin
 
A Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12cA Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12cLeighton Nelson
 
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]Markus Michalewicz
 
SAP BASIS Practice Exam
SAP BASIS Practice ExamSAP BASIS Practice Exam
SAP BASIS Practice ExamIT LearnMore
 
Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714ramesh469
 
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...Markus Michalewicz
 
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewOracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewMarkus Michalewicz
 

Viewers also liked (15)

Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RACExpert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RAC
 
Presentation capacity management for oracle exadata database machine v2
Presentation   capacity management for oracle exadata database machine v2Presentation   capacity management for oracle exadata database machine v2
Presentation capacity management for oracle exadata database machine v2
 
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4
 
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013
 
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
 
Architecture of exadata database machine – Part II
Architecture of exadata database machine – Part IIArchitecture of exadata database machine – Part II
Architecture of exadata database machine – Part II
 
Exadata x4 for_sap
Exadata x4 for_sapExadata x4 for_sap
Exadata x4 for_sap
 
Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2
 
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1
 
A Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12cA Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12c
 
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
 
SAP BASIS Practice Exam
SAP BASIS Practice ExamSAP BASIS Practice Exam
SAP BASIS Practice Exam
 
Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714
 
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
 
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - OverviewOracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - Overview
 

Similar to Rac 12c optimization

Oow2007 performance
Oow2007 performanceOow2007 performance
Oow2007 performanceRicky Zhu
 
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdfriyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdfabdulhafeezkalsekar1
 
Advanced rac troubleshooting
Advanced rac troubleshootingAdvanced rac troubleshooting
Advanced rac troubleshootingRiyaj Shamsudeen
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTanel Poder
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTanel Poder
 
How to debug ocfs2 hang problem
How to debug ocfs2 hang problemHow to debug ocfs2 hang problem
How to debug ocfs2 hang problemGang He
 
Velocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard WayVelocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard WayCosimo Streppone
 
Mod03 linking and accelerating
Mod03 linking and acceleratingMod03 linking and accelerating
Mod03 linking and acceleratingPeter Haase
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01Karam Abuataya
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11gfcamachob
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksKyle Hailey
 
SCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имяSCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имяEkaterina Melnik
 
SCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the NameSCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the NamePositive Hack Days
 
A close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issuesA close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issuesRiyaj Shamsudeen
 
Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +OracletuningThomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuningafa reg
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsJohn Kanagaraj
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceAnil Nair
 
hacking-embedded-devices.pptx
hacking-embedded-devices.pptxhacking-embedded-devices.pptx
hacking-embedded-devices.pptxssuserfcf43f
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and ArchitectureSidney Chen
 

Similar to Rac 12c optimization (20)

Oow2007 performance
Oow2007 performanceOow2007 performance
Oow2007 performance
 
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdfriyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
 
Advanced rac troubleshooting
Advanced rac troubleshootingAdvanced rac troubleshooting
Advanced rac troubleshooting
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel PoderTroubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contentionTroubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contention
 
How to debug ocfs2 hang problem
How to debug ocfs2 hang problemHow to debug ocfs2 hang problem
How to debug ocfs2 hang problem
 
Velocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard WayVelocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard Way
 
Mod03 linking and accelerating
Mod03 linking and acceleratingMod03 linking and accelerating
Mod03 linking and accelerating
 
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp0111thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
 
11 Things About11g
11 Things About11g11 Things About11g
11 Things About11g
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
SCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имяSCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имя
 
SCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the NameSCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the Name
 
Using AWR for IO Subsystem Analysis
Using AWR for IO Subsystem AnalysisUsing AWR for IO Subsystem Analysis
Using AWR for IO Subsystem Analysis
 
A close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issuesA close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issues
 
Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +OracletuningThomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuning
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and AdvisorsYour tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
 
New Generation Oracle RAC Performance
New Generation Oracle RAC PerformanceNew Generation Oracle RAC Performance
New Generation Oracle RAC Performance
 
hacking-embedded-devices.pptx
hacking-embedded-devices.pptxhacking-embedded-devices.pptx
hacking-embedded-devices.pptx
 
Oracle Basics and Architecture
Oracle Basics and ArchitectureOracle Basics and Architecture
Oracle Basics and Architecture
 

More from Riyaj Shamsudeen

Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010Riyaj Shamsudeen
 
Riyaj real world performance issues rac focus
Riyaj real world performance issues rac focusRiyaj real world performance issues rac focus
Riyaj real world performance issues rac focusRiyaj Shamsudeen
 
Debunking myths about_redo_ppt
Debunking myths about_redo_pptDebunking myths about_redo_ppt
Debunking myths about_redo_pptRiyaj Shamsudeen
 
Deep review of LMS process
Deep review of LMS processDeep review of LMS process
Deep review of LMS processRiyaj Shamsudeen
 
Demystifying cost based optimization
Demystifying cost based optimizationDemystifying cost based optimization
Demystifying cost based optimizationRiyaj Shamsudeen
 
A kind and gentle introducton to rac
A kind and gentle introducton to racA kind and gentle introducton to rac
A kind and gentle introducton to racRiyaj Shamsudeen
 
Performance tuning a quick intoduction
Performance tuning   a quick intoductionPerformance tuning   a quick intoduction
Performance tuning a quick intoductionRiyaj Shamsudeen
 
pstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle databasepstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle databaseRiyaj Shamsudeen
 
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN Riyaj Shamsudeen
 
Dbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineersDbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineersRiyaj Shamsudeen
 

More from Riyaj Shamsudeen (13)

Rac introduction
Rac introductionRac introduction
Rac introduction
 
Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010
 
Riyaj real world performance issues rac focus
Riyaj real world performance issues rac focusRiyaj real world performance issues rac focus
Riyaj real world performance issues rac focus
 
Redo internals ppt
Redo internals pptRedo internals ppt
Redo internals ppt
 
Debunking myths about_redo_ppt
Debunking myths about_redo_pptDebunking myths about_redo_ppt
Debunking myths about_redo_ppt
 
Deep review of LMS process
Deep review of LMS processDeep review of LMS process
Deep review of LMS process
 
Demystifying cost based optimization
Demystifying cost based optimizationDemystifying cost based optimization
Demystifying cost based optimization
 
Px execution in rac
Px execution in racPx execution in rac
Px execution in rac
 
A kind and gentle introducton to rac
A kind and gentle introducton to racA kind and gentle introducton to rac
A kind and gentle introducton to rac
 
Performance tuning a quick intoduction
Performance tuning   a quick intoductionPerformance tuning   a quick intoduction
Performance tuning a quick intoduction
 
pstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle databasepstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle database
 
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN
 
Dbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineersDbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineers
 

Recently uploaded

Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 

Recently uploaded (20)

20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 

Rac 12c optimization

  • 1. RAC 12c optimization By Riyaj Shamsudeen ©OraInternals Riyaj Shamsudeen
  • 2. ©OraInternals Riyaj Shamsudeen 2 Me  20+ years using Oracle products/DBA  OakTable member, Oracle ACE Director  Certified DBA -7.0,7.3,8,8i,9i &10g  Specializes in RAC, performance tuning, EBS, & Exadata  Chief DBA with OraInternals  Co-author of a few books  Email: rshamsud@orainternals.com  Blog : orainternals.wordpress.com  Web: www.orainternals.com
  • 5. RAC cluster root causes  CPU / Memory starvation in a node.  Log file write/ LGWR slowness.  Network issues / packet drops.  Excessive Dynamic Remastering activities.  Excessive amount of global cache traffic- due to workload or execution plan regression.  Localized inserts in to a few blocks.  Localized, excessive accesses to a few blocks. ©OraInternals Riyaj Shamsudeen
  • 6. Golden rules of RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
  • 7.  Impact of side effect events can be much, much worse then actual root cause event itself. ©OraInternals Riyaj Shamsudeen Top event tunnel vision  In RAC, the root cause can be hidden in other node and might not show up in the Top-5 wait events.  Performance of the instances are intertwined.  Performance of background processes in one node, can affect the application performance in the all other nodes.
  • 8. Top event tunnel vision ..2 ©OraInternals Riyaj Shamsudeen 8 Example 1
  • 9. Top event tunnel vision ..3 ©OraInternals Riyaj Shamsudeen 9 Example 2
  • 10. Top event tunnel vision ..4 ©OraInternals Riyaj Shamsudeen 10 Example 3
  • 11. ©OraInternals Riyaj Shamsudeen 11 Problem node Example 3 (continued)
  • 12. ©OraInternals Riyaj Shamsudeen gc buffer busy gc current request FG1 LMS LGWR Open request File #5, blk #8 Flush request Post Block send FG2 A BL request for file #5, block #8 is pending. gc buffer busy acquire gc buffer busy release FG3 Inst 2 Inst 1 charge time to actual wait event. Wire
  • 13. ©OraInternals Riyaj Shamsudeen gc buffer busy  Event ‘gc buffer busy [acquire|release]’ can be the symptoms for all the RAC performance root causes discussed earlier.  Gc buffer busy wait simply means that there is a pending request for a BL lock for that block already.  All other process wait for the BL lock to be available and charge the time to gc buffer busy event.
  • 14. Gc buffer busy – Objects related Following four pages only applicable if there are no other events indicating systemic issues.  ASH or trace files can be used to identify the blocks suffering from excessive ‘gc buffer busy’ waits.  Only use this method if there is no other issue affecting background processes.  To understand gc buffer busy waits:  Identify the object and object type  Identify the block type  ash_gcwait_to_obj.sql can be used to identify the object name and type. ©OraInternals Riyaj Shamsudeen 14
  • 15. Gc buffer busy – Identify object ©OraInternals Riyaj Shamsudeen 15 @ash_gcwait_to_obj.sql INST_ID EVENT OWNER OBJECT_NAME OBJECT_TYPE CNT ---------- ------------------------ ------ ---------------- ---------------- 1 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 3 2 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 10 2 gc buffer busy release RS T_GEN_INS_01 TABLE 4
  • 16. Gc buffer busy – Identify block @ash_gcwait_to_block.sql INST_ID EVENT CURRENT_FILE# CURRENT_BLOCK# CNT ---------- ------------------------------ ------------- -------------- ---------- 2 gc buffer busy acquire 4 103582 4 2 gc buffer busy acquire 4 103607 3 2 gc buffer busy acquire 4 103603 3 1 gc buffer busy acquire 4 103582 2 2 gc buffer busy release 4 103615 2 1 gc buffer busy acquire 4 103583 1 2 gc buffer busy release 4 103586 1 2 gc buffer busy release 4 103613 1 In this example, many different blocks are involved in ‘gc buffer busy’ waits. A symptom of heavy insert load. ©OraInternals Riyaj Shamsudeen 16 Demo: ash_gcwait_to_block.sql
  • 17. Gc buffer busy – Identify block type  You might need to dump the block to identify the type and contents of the block. alter system dump datafile 4 block min 103582 block max 103582;  In this example, it is a leaf block of an index. Block header dump: 0x0101949e Object id on Block? Y seg/obj: 0x13266 csc: 0x00.11c65d1 itc: 9 flg: E typ: 2 - INDEX brn: 0 bdba: 0x1019498 ver: 0x01 opc: 0 inc: 0 exflg: 0 Itl Xid Uba Flag Lck Scn/Fsc 0x01 0x0008.001.0000144f 0x00c00a1f.058a.01 -BU- 1 fsc 0x0000.011c65d4 0x02 0x0008.015.00001449 0x00c00a37.058a.56 ---- 55 fsc 0x0000.00000000 0x03 0x0004.008.000016c0 0x00c01232.045c.18 ---- 90 fsc 0x0000.00000000 0x04 0x0002.005.000015c1 0x00c005d1.0402.12 ---- 73 fsc 0x0000.00000000 0x05 0x000a.01c.0000158b 0x00c00849.0446.05 ---- 75 fsc 0x0000.00000000 Leaf block dump =============== ©OraInternals Riyaj Shamsudeen 17 Demo: dump_blocks.sql, tracefile.sql
  • 18. ©OraInternals Riyaj Shamsudeen 18 Object type Block type Possible issue(s) Table Segment header Freelists, freelist groups, deletes from one node,ASSM bugs etc Segment blocks Heavy insert workload, heavy update to few blocks, SQL performance issues scanning few objects aggressively. Index Leaf block Numerous inserts on indexed columns, sequence generated keys Root block/ branch block Insert + numerous index scans, concurrent inserts and deletes etc Undo Undo header block Numerous short transactions Undo block Long pending transaction and CR block generation Dictionary SEQ$ (object) Sequences with nocache or order set and excessive access to sequence.
  • 19. Golden rules of RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
  • 20. ©OraInternals Riyaj Shamsudeen gc current block 2-way gc current request FG1 LMS Open request File #5, blk #8 Block send Inst 2 Inst 1 charge time to ‘gc current block 2-way’ Wire
  • 21. Eliminate infra-structure as an issue ©OraInternals Riyaj Shamsudeen 21
  • 22. Eliminate infra-structure as an issue  Wait events ‘gc cr block 2-way’ , ‘gc cr block 3-way’ , ‘gc current block 2-way’, and ‘gc current block 3-way’, ‘gc cr grant 2-way’, ‘gc cr grant 2-way’.  Concurrency or congestion issues are not factored in to these events.  The lowest wait time that you can possibly get for a block or a message transfer.  A wait time of <2ms is normal. In Exadata, <0.5ms ©OraInternals Riyaj Shamsudeen 22
  • 23. Golden rules of RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
  • 24. Identify problem-inducing instance  Gv$instance_cache_transfer keeps track of the RX metrics at both instance and class level. ©OraInternals Riyaj Shamsudeen 24
  • 25. Identify problem inducing instance ..2 SELECT INSTANCE ||'->’ || inst_id transfer, class, cr_block cr_blk, Trunc(cr_block_time / cr_block / 1000, 2) avg_Cr, current_block cur_blk, Trunc(current_block_time / current_block / 1000, 2) avg_cur FROM gv$instance_cache_transfer WHERE cr_block > 0 AND current_block > 0 ORDER BY INSTANCE, inst_id, class / Receiving side analysis. TRANS CLASS CR_BLK AVG_CR CUR_BLK AVG_CUR ----- ---------- ---------- ---------- ---------- ---------- 1->2 data block 87934887 1.23 9834152 1.8 2->1 data block 28392332 1.30 1764932 2.1 ... 3->1 data block 12519985 11.57 2231921 21.6 ... 3->2 undo block 4676398 8.85 320 27.82  Use script gc_identify_slow_inst.sql ©OraInternals Riyaj Shamsudeen 25
  • 26. Golden rules of RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
  • 27. ©OraInternals Riyaj Shamsudeen Send side in an oval gc current request FG1 LMS LGWR Open request File #5, blk #8 Flush request Post Block send FG2 A BL request for file #5, block #8 is pending. gc buffer busy acquire gc buffer busy release FG3 Inst 2 Inst 1 charge time to actual wait event. Wire
  • 28. RX side Send side ©OraInternals Riyaj Shamsudeen 28 Sending side analysis gc cr block receive time= Time to send message to a remote LMS process by FG + Time taken by LMS to build block (statistics: gc cr block build time) + LMS wait for LGWR latency ( statistics: gc cr block flush time) + LMS send time (Statistics: gc cr block send time) + Wire latency. Instance 1: ---------- Avg global cache cr block receive time (ms) : 222.1 Avg global cache current block receive time (ms) : 27.5 Instance 2: ---------- Avg global cache cr block build time (ms) : 0.0 Avg global cache cr block send time (ms) : 0.1 Global cache log flushes for cr blocks served % : 2.7 Avg global cache cr block flush time (ms) : 15879.9
  • 29. ©OraInternals Riyaj Shamsudeen 29 Problem node Global Cache and Enqueue Services - Workload Characteristics Avg global enqueue get time (ms) : 0.1 Avg global cache cr block receive time (ms) : 14.4 Avg global cache current block receive time (ms): 23.4 Avg global cache cr block build time (ms) : 0.0 Avg global cache cr block send time (ms) : 0.0 Global cache log flushes for cr blocks served %: 25.0 Avg global cache cr block flush time (ms) : 2.5 Avg global cache current block pin time (ms) : 0.0 Avg global cache current block send time (ms): 0.0 Global cache log flushes for current blocks served %: 6.3 Avg global cache current block flush time (ms): 5.9
  • 30. Sending side analysis ..2 Avg message sent queue time (ms) : 0.1 Avg message sent queue time on ksxp (ms): 0.6 Avg message received queue time (ms): 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 46.62 % of indirect sent messages : 43.76 % of flow controlled messages : 9.62 Avg message sent queue time (ms) : 0.0 Avg message sent queue time on ksxp (ms): 0.4 Avg message received queue time (ms) : 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 60.86 % of indirect sent messages : 38.48 % of flow controlled messages : 0.66 ©OraInternals Riyaj Shamsudeen 30
  • 31. ©OraInternals Riyaj Shamsudeen 31 Problem node Global Cache and Enqueue Services - Messaging Statistics Avg message sent queue time (ms) : 344.7 Avg message sent queue time on ksxp (ms): 0.3 Avg message received queue time (ms): 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 31.76 % of indirect sent messages : 46.12 % of flow controlled messages : 22.13
  • 32. ©OraInternals Riyaj Shamsudeen 32 Multi-tenancy  Buffer cache and Global cache processing is common to all pluggable databases.  A PDB generating more global cache workload can cause performance issues to other PDBs.  There doesn’t seem to be a way to constrain a database from overusing buffer cache resource or global cache resources.  Resource manager only controls CPU, PX, and I/O(exadata).
  • 33. Prepare for multi-tenancy (RAC specific)  Keep the buffer cache as large (for OLTP).  Global cache overhead is reduced if the buffers are in the local buffer cache.  Make LGWR efficient. LMS waits for LGWR to ship blocks.  Tune the SQL statements to reduce global cache traffic. A misbehaving piece of application code affects the whole cluster.  Connect to the container database to review global cache metrics. ©OraInternals Riyaj Shamsudeen 33
  • 34. ©OraInternals Riyaj Shamsudeen 34 Resource names  From 12 onwards, resource names are coded with con_id. select resource_name from gv$ges_resource where resource_name like '[0x15f29][0x0],[TM]%' / RESOURCE_NAME ------------------------------ [0x15f29][0x0],[TM][ext 0x3,0x0] <- hrdb1 PDB – GSTEST table –object_id=89897 [0x15f29][0x0],[TM][ext 0x4,0x0] <- hrdb2 PDB – GSTEST table –object_id=89897  Comes handy if you are debugging RAC trace files, to identify the PDB generating the errors.
  • 35. ©OraInternals Riyaj Shamsudeen 35 GC statistics and PDBs  Most Global cache statistics views do not provide granularity to a pluggable database. For example, select con_id, inst_id, count(*) from gv$instance_cache_transfer group by con_id, inst_id / CON_ID INST_ID COUNT(*) ---------- ---------- ---------- 0 2 4096 0 1 4096  Review AWR reports from the container database to review global cache performance issues.
  • 36. High Priority Processes (from 12c) Parameter: _high_priority_processes Value : LMS*|VKTM|LM*|LCK0|GCR*|DIAG … kjfmPriCheck: 0 RTLMS taken, 1 RTLMS available kjfmPriCheck: automatic priority changes enabled - multi-inst mode kjfmPriCheck: # of real time LMS increasing from 0 to 1 … Parameter: _lm_lms_priority_dynamic Value : TRUE ©OraInternals Riyaj Shamsudeen 36
  • 37. ©OraInternals Riyaj Shamsudeen gc cr grants 2-way gc cr request FG1 LMS Open request File #5, blk #8 Read from the disk Inst 2 Inst 1 charge time to ‘gc cr grant 2-way’ Wire
  • 38. Event: GC cr grants 2-way  Time is accounted to this wait event, if the block is not in any of the buffer cache.  Trace file will indicate this wait event followed by a disk read. nam='gc cr grant 2-way' ela= 659 p1=1 p2=88252 p3=1 obj#=77779 nam='db file sequential read' ela= 938 file#=1 block#=88252 blocks=1 obj#=77779  Typical latency is 1-2ms. Any thing above needs to be reviewed as these are light-wait events.  Process sends a request to remote master LMS process and the LMS process simply responds with ‘read from disk’.  This is another base line wait event to measure interconnect response time, as LMS processing for this event is minimal. ©OraInternals Riyaj Shamsudeen 38
  • 39. ©OraInternals Riyaj Shamsudeen Why DRM? FG1 LMS Read from the disk Inst 2 Inst 1 charge time to ‘gc cr grant 2-way’ Excessive amount of GC activity for One object!
  • 40. ©OraInternals Riyaj Shamsudeen 40 DRM & grants  Dynamic Remastering feature is to reduce or eliminate this specific RAC Tax: gc cr grants 2-way  If the object is mastered locally, processes can read the block directly without incurring any GC waits.  Identify the objects suffering from excessive ‘gc cr grants 2-way’ events (and remaster them manually if needed) using script ash_gcwait_to_obj.sql. INST_ID OWNER OBJECT_NAME OBJECT_TYPE CNT ---------- -------------------- -------------------------------- -------------------- ---------- 1 AR RA_CUSTOMER_TRX_ALL TABLE 9 1 INV MTL_MATERIAL_TRANSACTIONS TABLE 141 1 INV MTL_SERIAL_NUMBERS TABLE 155 1 Undo Block 253 1 INV MTL_SYSTEM_ITEMS_B TABLE 476 1 AP AP_CHECKS_ALL TABLE 785
  • 41. ©OraInternals Riyaj Shamsudeen 41 Local vs Remote  In AWR report, identify the local vs remote grants also. Following shows that 74% of grants are remote. DRM might be of help here, if there are much waits for ‘gc grants 2-way’. gc local grants 6,570,658 3,643.6 21.8 gc remote grants 19,262,806 10,681.7 63.9  Increase SGA size to reduce single block I/O reads.  Tuning SQL statements to reduce logical I/O, optimal indexes, etc can help to improve the performance.
  • 42. ©OraInternals Riyaj Shamsudeen 42 DRM effect Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class ------------------------------ ------------ ----------- ------ ------ ---------- gc buffer busy 1,826,073 152,415 83 62.0 Cluster CPU time 30,192 12.3 enq: TX - index contention 34,332 15,535 453 6.3 Concurrency gcs drm freeze in enter server 22,789 11,279 495 4.6 Other enq: TX - row lock contention 46,926 4,493 96 1.8 Applicatio Global Cache and Enqueue Services - Workload Characteristics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Avg global enqueue get time (ms) : 16.8 Avg global cache cr block receive time (ms) : 17.1 Avg global cache current block receive time (ms): 14.9
  • 43. ©OraInternals Riyaj Shamsudeen 43 DRM (11g) LCK Request queue LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn Inst 1 Inst 2 Inst 3 Resources In 11g, all resources are frozen during the reconfiguration.
  • 44. ©OraInternals Riyaj Shamsudeen 44 DRM (12c) LCK Request queue LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn Inst 1 Inst 2 Inst 3 Resources In 12c, only set of resources in a window is frozen.
  • 45. DRM impact is decreased in 12c by freezing a resource partition, at a time.. ©OraInternals Riyaj Shamsudeen 45 DRM in 12c * drm quiesce 2014-02-15 16:03:45.131840 : DRM(3) resources quiesced [0-1023], rescount 1719 2014-02-15 16:03:45.132173 : DRM(3) local converts quiesced [0-1023], lockcount 0, bucket 0 * drm sync 1 * drm freeze * drm cleanup * drm sync 2 * drm replay * drm sync 3 * drm fix writes * drm sync 4 * drm end * DRM RCFG called (swin 0) CGS recovery timeout = 85 sec * drm quiesce 2014-02-15 16:03:45.172867 : DRM(3) resources quiesced [1024-2047], rescount 1754 2014-02-15 16:03:45.172932 : DRM(3) local converts quiesced [1024-2047], lockcount 0, bucket 0 * drm sync 1 * drm freeze
  • 46. ©OraInternals Riyaj Shamsudeen 46 PQ trace file  PQ Co-ordinator trace shows Object checkpoint. select /*+ parallel (a 4) */ min(length(v1)) from rs.hugetable a; WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 866 name| mode=1263468550 2=65581 0=1 obj#=-1 tim=6965638229 ... WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 454 name| mode=1263468545 2=65581 0=2 obj#=-1 tim=6965638986 ...  PQ server process will perform direct reads to PGA. Notice that there is no GC waits between the reads. WAIT #18446741324874985288: nam='direct path read' ela= 4782 file number=4 first dba=11584 block cnt=16 obj#=76739 tim=3653439149 WAIT #18446741324874985288: nam='direct path read' ela= 16157 file number=4 first dba=11600 block cnt=16 obj#=76739 tim=3653456065 WAIT #18446741324874985288: nam='direct path read' ela= 3675 file number=4 first dba=11616 block cnt=16 obj#=76739 tim=3653462065 WAIT #18446741324874985288: nam='direct path read' ela= 8162 file number=4 first dba=11632 block cnt=16 obj#=76739 tim=3653471358 WAIT #18446741324874985288: nam='direct path read' ela= 10270 file number=4 first dba=11650 block cnt=14 obj#=76739 tim=3653482566
  • 47. ©OraInternals Riyaj Shamsudeen 47 Application issues  There is no easy fix in 12c either   Avoid localized inserts.
  • 48. ©OraInternals Riyaj Shamsudeen 48 Application issues  Hash partitioning the index/table might be helpful.
  • 49. ©OraInternals Riyaj Shamsudeen 49 Application issues  Reduce localized accesses:  Frequent access to a table block such as a lookup table: Cache it somewhere else.  Aggressive scanning of smaller tables due to inefficient optimizer plans.  Aggressive nested loops joins accessing huge number of index leaf blocks.  Ordered cached sequences – Excessive SV resource contention.  Excessive DDL in a multi-tenant databases.
  • 50. ©OraInternals Riyaj Shamsudeen 50 Use Services Inst 1 PO FIN Inst 2 PO FIN Inst 3 PO FIN  While creating a service, you define a preferred and available instances. PO FIN Preferred Available For PO service: Instance 1 & 2 are preferred instances, and instance 3 is an available instance.
  • 51. PDB1 ©OraInternals Riyaj Shamsudeen 51 Or PDBs Inst 1 PDB1 PDB2 Inst 2 Inst 3  PDB1 is open in inst1 & inst2.  PDB2 is open in inst2 & inst3 PO FIN Online Failover PDB1 PDB2 PDB2
  • 52. Statistics: gc lost blocks  This statistics tracks dropped interconnect packets.  For 8K block size, block transfer would require 6 packets of size ~1500 bytes (non-Jumbo) to be transferred.  Even if a packet is dropped, then the block need to be resent.  Requesting process does not know the state of the request as UDP layer can not reassemble packet. (UDP is a send and forget type of protocol.)  After a timeout (0.5s in 11g, ~5s in 10g) requesting process will declare that packet as lost, and request the block again. ©OraInternals Riyaj Shamsudeen 52
  • 53. Statistics: gc lost blocks - Effect  If the packets are lost, then the sessions can be seen waiting for an idle event.  So, higher wait times for gc cr request, gc current request, and gc cr multiblock request will be visible.  Since many blocks are requested during multi block reads, waits for ‘gc cr multiblock request’ will be a prominent wait event.  Ideally, should be 0. But, usually, few drops are acceptable. You would need to judge the impact of lost block reviewing other wait events. ©OraInternals Riyaj Shamsudeen 53
  • 54. Statistics: gc lost blocks - reason  In a nutshell, packet is dropped   We need to probe and understand why the packets are dropped.  Few common reasons are: (In-depth discussion in Network)  Insufficient UDP buffer space.  High CPU usage.  Incorrect network configuration so packets thrown away.  Route between the nodes has many hops.  Switch issues- configuration, firmware etc ©OraInternals Riyaj Shamsudeen 54
  • 55. ©OraInternals Riyaj Shamsudeen 55 HugePages  By default, pagesize is 4KB in Linux. PageTables size will be huge, in the order of, GigaBytes.  For huge SGA sizes, this is problematic. kswapd deamon will consume more CPU during memory starvation and lead to server hung situations.  CRS might not restart the server in time, this can lead to other nodes suffering from excessive gc buffer busy waits.  HugePages uses a pagesize of 2MB. This reduces the size of PageTable to few 100MBs. In RAC & Linux, you should consider using HugePages.
  • 56. ©OraInternals Riyaj Shamsudeen 56 THANK YOU  Email: rshamsud@orainternals.com  Blog : orainternals.wordpress.com  Web: www.orainternals.com