Submit Search
Upload
Rac 12c optimization
•
2 likes
•
1,466 views
Riyaj Shamsudeen
Follow
A deep dive discussion about optimization in a RAC 12c database.
Read less
Read more
Technology
Report
Share
Report
Share
1 of 56
Download Now
Download to read offline
Recommended
Understanding oracle rac internals part 2 - slides
Understanding oracle rac internals part 2 - slides
Mohamed Farouk
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
Migration to Oracle Multitenant
Migration to Oracle Multitenant
Jitendra Singh
Oracle RAC 19c and Later - Best Practices #OOWLON
Oracle RAC 19c and Later - Best Practices #OOWLON
Markus Michalewicz
Advanced RAC troubleshooting: Network
Advanced RAC troubleshooting: Network
Riyaj Shamsudeen
Understanding oracle rac internals part 1 - slides
Understanding oracle rac internals part 1 - slides
Mohamed Farouk
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...
Oracle Real Application Clusters 19c- Best Practices and Internals- EMEA Tour...
Sandesh Rao
Oracle Database performance tuning using oratop
Oracle Database performance tuning using oratop
Sandesh Rao
More Related Content
What's hot
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning Fundamentals
Enkitec
Oracle statistics by example
Oracle statistics by example
Mauro Pagano
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
SrirakshaSrinivasan2
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and Underscores
Jitendra Singh
AWR and ASH Deep Dive
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Markus Michalewicz
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
Markus Michalewicz
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools short
Tanel Poder
What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1
Satishbabu Gunukula
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion Edition
Markus Michalewicz
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
Markus Michalewicz
One PDB to go, please!
One PDB to go, please!
Christian Gohmann
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder
How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata Environments
Sandesh Rao
Oracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best Practices
Bobby Curtis
AWR & ASH Analysis
AWR & ASH Analysis
aioughydchapter
Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016
Anil Nair
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
Kyle Hailey
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAA
Markus Michalewicz
Oracle db performance tuning
Oracle db performance tuning
Simon Huang
What's hot
(20)
Oracle Performance Tuning Fundamentals
Oracle Performance Tuning Fundamentals
Oracle statistics by example
Oracle statistics by example
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Oracle_Multitenant_19c_-_All_About_Pluggable_D.pdf
Performance Stability, Tips and Tricks and Underscores
Performance Stability, Tips and Tricks and Underscores
AWR and ASH Deep Dive
AWR and ASH Deep Dive
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
Oracle Multitenant meets Oracle RAC - IOUG 2014 Version
The Oracle RAC Family of Solutions - Presentation
The Oracle RAC Family of Solutions - Presentation
Tanel Poder - Scripts and Tools short
Tanel Poder - Scripts and Tools short
What’s New in Oracle Database 19c - Part 1
What’s New in Oracle Database 19c - Part 1
Oracle RAC Internals - The Cache Fusion Edition
Oracle RAC Internals - The Cache Fusion Edition
MAA Best Practices for Oracle Database 19c
MAA Best Practices for Oracle Database 19c
One PDB to go, please!
One PDB to go, please!
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
How to Use EXAchk Effectively to Manage Exadata Environments
How to Use EXAchk Effectively to Manage Exadata Environments
Oracle GoldenGate 21c New Features and Best Practices
Oracle GoldenGate 21c New Features and Best Practices
AWR & ASH Analysis
AWR & ASH Analysis
Anil nair rac_internals_sangam_2016
Anil nair rac_internals_sangam_2016
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
Achieving Continuous Availability for Your Applications with Oracle MAA
Achieving Continuous Availability for Your Applications with Oracle MAA
Oracle db performance tuning
Oracle db performance tuning
Viewers also liked
Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RAC
SolarWinds
Presentation capacity management for oracle exadata database machine v2
Presentation capacity management for oracle exadata database machine v2
xKinAnx
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4
Fran Navarro
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013
Markus Michalewicz
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Nelson Calero
Architecture of exadata database machine – Part II
Architecture of exadata database machine – Part II
Paresh Nayak,OCP®,Prince2®
Exadata x4 for_sap
Exadata x4 for_sap
Fran Navarro
Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2
Rodrigo Almeida
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1
jenkin
A Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12c
Leighton Nelson
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Markus Michalewicz
SAP BASIS Practice Exam
SAP BASIS Practice Exam
IT LearnMore
Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714
ramesh469
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Markus Michalewicz
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - Overview
Markus Michalewicz
Viewers also liked
(15)
Expert performance tuning tips for Oracle RAC
Expert performance tuning tips for Oracle RAC
Presentation capacity management for oracle exadata database machine v2
Presentation capacity management for oracle exadata database machine v2
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle RAC 12c Best Practices with Appendices DOAG2013
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Oracle Exadata Maintenance tasks 101 - OTN Tour 2015
Architecture of exadata database machine – Part II
Architecture of exadata database machine – Part II
Exadata x4 for_sap
Exadata x4 for_sap
Desvendando Oracle Exadata X2-2
Desvendando Oracle Exadata X2-2
Sun Oracle Exadata Technical Overview V1
Sun Oracle Exadata Technical Overview V1
A Second Look at Oracle RAC 12c
A Second Look at Oracle RAC 12c
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
Understanding Oracle RAC 12c Internals OOW13 [CON8806]
SAP BASIS Practice Exam
SAP BASIS Practice Exam
Sap basis certification_and_interview_questions_answers_and11237206714
Sap basis certification_and_interview_questions_answers_and11237206714
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c (12.1.0.2) Operational Best Practices - A result of true colla...
Oracle RAC 12c Release 2 - Overview
Oracle RAC 12c Release 2 - Overview
Similar to Rac 12c optimization
Oow2007 performance
Oow2007 performance
Ricky Zhu
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
abdulhafeezkalsekar1
Advanced rac troubleshooting
Advanced rac troubleshooting
Riyaj Shamsudeen
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder
How to debug ocfs2 hang problem
How to debug ocfs2 hang problem
Gang He
Velocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard Way
Cosimo Streppone
Mod03 linking and accelerating
Mod03 linking and accelerating
Peter Haase
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
Karam Abuataya
11 Things About11g
11 Things About11g
fcamachob
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
Kyle Hailey
SCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имя
Ekaterina Melnik
SCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the Name
Positive Hack Days
Using AWR for IO Subsystem Analysis
Using AWR for IO Subsystem Analysis
Texas Memory Systems, and IBM Company
A close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issues
Riyaj Shamsudeen
Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuning
afa reg
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj
New Generation Oracle RAC Performance
New Generation Oracle RAC Performance
Anil Nair
hacking-embedded-devices.pptx
hacking-embedded-devices.pptx
ssuserfcf43f
Oracle Basics and Architecture
Oracle Basics and Architecture
Sidney Chen
Similar to Rac 12c optimization
(20)
Oow2007 performance
Oow2007 performance
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
riyaj_advanced_rac_troubleshooting_rmoug_2010_ppt.pdf
Advanced rac troubleshooting
Advanced rac troubleshooting
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Troubleshooting Complex Performance issues - Oracle SEG$ contention
How to debug ocfs2 hang problem
How to debug ocfs2 hang problem
Velocity 2012 - Learning WebOps the Hard Way
Velocity 2012 - Learning WebOps the Hard Way
Mod03 linking and accelerating
Mod03 linking and accelerating
11thingsabout11g 12659705398222 Phpapp01
11thingsabout11g 12659705398222 Phpapp01
11 Things About11g
11 Things About11g
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
SCADA Strangelove: взлом во имя
SCADA Strangelove: взлом во имя
SCADA Strangelove: Hacking in the Name
SCADA Strangelove: Hacking in the Name
Using AWR for IO Subsystem Analysis
Using AWR for IO Subsystem Analysis
A close encounter_with_real_world_and_odd_perf_issues
A close encounter_with_real_world_and_odd_perf_issues
Thomas+Niewel+ +Oracletuning
Thomas+Niewel+ +Oracletuning
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
New Generation Oracle RAC Performance
New Generation Oracle RAC Performance
hacking-embedded-devices.pptx
hacking-embedded-devices.pptx
Oracle Basics and Architecture
Oracle Basics and Architecture
More from Riyaj Shamsudeen
Rac introduction
Rac introduction
Riyaj Shamsudeen
Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010
Riyaj Shamsudeen
Riyaj real world performance issues rac focus
Riyaj real world performance issues rac focus
Riyaj Shamsudeen
Redo internals ppt
Redo internals ppt
Riyaj Shamsudeen
Debunking myths about_redo_ppt
Debunking myths about_redo_ppt
Riyaj Shamsudeen
Deep review of LMS process
Deep review of LMS process
Riyaj Shamsudeen
Demystifying cost based optimization
Demystifying cost based optimization
Riyaj Shamsudeen
Px execution in rac
Px execution in rac
Riyaj Shamsudeen
A kind and gentle introducton to rac
A kind and gentle introducton to rac
Riyaj Shamsudeen
Performance tuning a quick intoduction
Performance tuning a quick intoduction
Riyaj Shamsudeen
pstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle database
Riyaj Shamsudeen
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN
Riyaj Shamsudeen
Dbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineers
Riyaj Shamsudeen
More from Riyaj Shamsudeen
(13)
Rac introduction
Rac introduction
Riyaj: why optimizer_hates_my_sql_2010
Riyaj: why optimizer_hates_my_sql_2010
Riyaj real world performance issues rac focus
Riyaj real world performance issues rac focus
Redo internals ppt
Redo internals ppt
Debunking myths about_redo_ppt
Debunking myths about_redo_ppt
Deep review of LMS process
Deep review of LMS process
Demystifying cost based optimization
Demystifying cost based optimization
Px execution in rac
Px execution in rac
A kind and gentle introducton to rac
A kind and gentle introducton to rac
Performance tuning a quick intoduction
Performance tuning a quick intoduction
pstack, truss etc to understand deeper issues in Oracle database
pstack, truss etc to understand deeper issues in Oracle database
A deep dive about VIP,HAIP, and SCAN
A deep dive about VIP,HAIP, and SCAN
Dbms plan - A swiss army knife for performance engineers
Dbms plan - A swiss army knife for performance engineers
Recently uploaded
20150722 - AGV
20150722 - AGV
Jamie (Taka) Wang
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
GDSC PJATK
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
Brian Pichman
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
DianaGray10
Nanopower In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
Pedro Manuel
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
IES VE
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
DianaGray10
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
Md Hossain Ali
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
D Cloud Solutions
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
Jamie (Taka) Wang
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
Seth Reyes
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
DianaGray10
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
Matsuo Lab
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
Mahmoud Rabie
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
DianaGray10
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
DianaGray10
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
bruanjhuli
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
DianaGray10
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Commit University
Recently uploaded
(20)
20150722 - AGV
20150722 - AGV
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Nanopower In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Rac 12c optimization
1.
RAC 12c optimization
By Riyaj Shamsudeen ©OraInternals Riyaj Shamsudeen
2.
©OraInternals Riyaj Shamsudeen
2 Me 20+ years using Oracle products/DBA OakTable member, Oracle ACE Director Certified DBA -7.0,7.3,8,8i,9i &10g Specializes in RAC, performance tuning, EBS, & Exadata Chief DBA with OraInternals Co-author of a few books Email: rshamsud@orainternals.com Blog : orainternals.wordpress.com Web: www.orainternals.com
3.
Process architecture
4.
RAC message flow
5.
RAC cluster root
causes CPU / Memory starvation in a node. Log file write/ LGWR slowness. Network issues / packet drops. Excessive Dynamic Remastering activities. Excessive amount of global cache traffic- due to workload or execution plan regression. Localized inserts in to a few blocks. Localized, excessive accesses to a few blocks. ©OraInternals Riyaj Shamsudeen
6.
Golden rules of
RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
7.
Impact of
side effect events can be much, much worse then actual root cause event itself. ©OraInternals Riyaj Shamsudeen Top event tunnel vision In RAC, the root cause can be hidden in other node and might not show up in the Top-5 wait events. Performance of the instances are intertwined. Performance of background processes in one node, can affect the application performance in the all other nodes.
8.
Top event tunnel
vision ..2 ©OraInternals Riyaj Shamsudeen 8 Example 1
9.
Top event tunnel
vision ..3 ©OraInternals Riyaj Shamsudeen 9 Example 2
10.
Top event tunnel
vision ..4 ©OraInternals Riyaj Shamsudeen 10 Example 3
11.
©OraInternals Riyaj Shamsudeen
11 Problem node Example 3 (continued)
12.
©OraInternals Riyaj Shamsudeen
gc buffer busy gc current request FG1 LMS LGWR Open request File #5, blk #8 Flush request Post Block send FG2 A BL request for file #5, block #8 is pending. gc buffer busy acquire gc buffer busy release FG3 Inst 2 Inst 1 charge time to actual wait event. Wire
13.
©OraInternals Riyaj Shamsudeen
gc buffer busy Event ‘gc buffer busy [acquire|release]’ can be the symptoms for all the RAC performance root causes discussed earlier. Gc buffer busy wait simply means that there is a pending request for a BL lock for that block already. All other process wait for the BL lock to be available and charge the time to gc buffer busy event.
14.
Gc buffer busy
– Objects related Following four pages only applicable if there are no other events indicating systemic issues. ASH or trace files can be used to identify the blocks suffering from excessive ‘gc buffer busy’ waits. Only use this method if there is no other issue affecting background processes. To understand gc buffer busy waits: Identify the object and object type Identify the block type ash_gcwait_to_obj.sql can be used to identify the object name and type. ©OraInternals Riyaj Shamsudeen 14
15.
Gc buffer busy
– Identify object ©OraInternals Riyaj Shamsudeen 15 @ash_gcwait_to_obj.sql INST_ID EVENT OWNER OBJECT_NAME OBJECT_TYPE CNT ---------- ------------------------ ------ ---------------- ---------------- 1 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 3 2 gc buffer busy acquire RS T_GEN_INS_01_N1 INDEX 10 2 gc buffer busy release RS T_GEN_INS_01 TABLE 4
16.
Gc buffer busy
– Identify block @ash_gcwait_to_block.sql INST_ID EVENT CURRENT_FILE# CURRENT_BLOCK# CNT ---------- ------------------------------ ------------- -------------- ---------- 2 gc buffer busy acquire 4 103582 4 2 gc buffer busy acquire 4 103607 3 2 gc buffer busy acquire 4 103603 3 1 gc buffer busy acquire 4 103582 2 2 gc buffer busy release 4 103615 2 1 gc buffer busy acquire 4 103583 1 2 gc buffer busy release 4 103586 1 2 gc buffer busy release 4 103613 1 In this example, many different blocks are involved in ‘gc buffer busy’ waits. A symptom of heavy insert load. ©OraInternals Riyaj Shamsudeen 16 Demo: ash_gcwait_to_block.sql
17.
Gc buffer busy
– Identify block type You might need to dump the block to identify the type and contents of the block. alter system dump datafile 4 block min 103582 block max 103582; In this example, it is a leaf block of an index. Block header dump: 0x0101949e Object id on Block? Y seg/obj: 0x13266 csc: 0x00.11c65d1 itc: 9 flg: E typ: 2 - INDEX brn: 0 bdba: 0x1019498 ver: 0x01 opc: 0 inc: 0 exflg: 0 Itl Xid Uba Flag Lck Scn/Fsc 0x01 0x0008.001.0000144f 0x00c00a1f.058a.01 -BU- 1 fsc 0x0000.011c65d4 0x02 0x0008.015.00001449 0x00c00a37.058a.56 ---- 55 fsc 0x0000.00000000 0x03 0x0004.008.000016c0 0x00c01232.045c.18 ---- 90 fsc 0x0000.00000000 0x04 0x0002.005.000015c1 0x00c005d1.0402.12 ---- 73 fsc 0x0000.00000000 0x05 0x000a.01c.0000158b 0x00c00849.0446.05 ---- 75 fsc 0x0000.00000000 Leaf block dump =============== ©OraInternals Riyaj Shamsudeen 17 Demo: dump_blocks.sql, tracefile.sql
18.
©OraInternals Riyaj Shamsudeen
18 Object type Block type Possible issue(s) Table Segment header Freelists, freelist groups, deletes from one node,ASSM bugs etc Segment blocks Heavy insert workload, heavy update to few blocks, SQL performance issues scanning few objects aggressively. Index Leaf block Numerous inserts on indexed columns, sequence generated keys Root block/ branch block Insert + numerous index scans, concurrent inserts and deletes etc Undo Undo header block Numerous short transactions Undo block Long pending transaction and CR block generation Dictionary SEQ$ (object) Sequences with nocache or order set and excessive access to sequence.
19.
Golden rules of
RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
20.
©OraInternals Riyaj Shamsudeen
gc current block 2-way gc current request FG1 LMS Open request File #5, blk #8 Block send Inst 2 Inst 1 charge time to ‘gc current block 2-way’ Wire
21.
Eliminate infra-structure as
an issue ©OraInternals Riyaj Shamsudeen 21
22.
Eliminate infra-structure as
an issue Wait events ‘gc cr block 2-way’ , ‘gc cr block 3-way’ , ‘gc current block 2-way’, and ‘gc current block 3-way’, ‘gc cr grant 2-way’, ‘gc cr grant 2-way’. Concurrency or congestion issues are not factored in to these events. The lowest wait time that you can possibly get for a block or a message transfer. A wait time of <2ms is normal. In Exadata, <0.5ms ©OraInternals Riyaj Shamsudeen 22
23.
Golden rules of
RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
24.
Identify problem-inducing instance
Gv$instance_cache_transfer keeps track of the RX metrics at both instance and class level. ©OraInternals Riyaj Shamsudeen 24
25.
Identify problem inducing
instance ..2 SELECT INSTANCE ||'->’ || inst_id transfer, class, cr_block cr_blk, Trunc(cr_block_time / cr_block / 1000, 2) avg_Cr, current_block cur_blk, Trunc(current_block_time / current_block / 1000, 2) avg_cur FROM gv$instance_cache_transfer WHERE cr_block > 0 AND current_block > 0 ORDER BY INSTANCE, inst_id, class / Receiving side analysis. TRANS CLASS CR_BLK AVG_CR CUR_BLK AVG_CUR ----- ---------- ---------- ---------- ---------- ---------- 1->2 data block 87934887 1.23 9834152 1.8 2->1 data block 28392332 1.30 1764932 2.1 ... 3->1 data block 12519985 11.57 2231921 21.6 ... 3->2 undo block 4676398 8.85 320 27.82 Use script gc_identify_slow_inst.sql ©OraInternals Riyaj Shamsudeen 25
26.
Golden rules of
RAC diagnostics 1. Beware of top event tunnel vision. 2. Eliminate infrastructure as an issue. 3. Identify problem instance(s). 4. Review Send side metrics, and not just Receive side metrics. 5. Use histograms to identify any abnormalities, and not just averages. ©OraInternals Riyaj Shamsudeen
27.
©OraInternals Riyaj Shamsudeen
Send side in an oval gc current request FG1 LMS LGWR Open request File #5, blk #8 Flush request Post Block send FG2 A BL request for file #5, block #8 is pending. gc buffer busy acquire gc buffer busy release FG3 Inst 2 Inst 1 charge time to actual wait event. Wire
28.
RX side Send
side ©OraInternals Riyaj Shamsudeen 28 Sending side analysis gc cr block receive time= Time to send message to a remote LMS process by FG + Time taken by LMS to build block (statistics: gc cr block build time) + LMS wait for LGWR latency ( statistics: gc cr block flush time) + LMS send time (Statistics: gc cr block send time) + Wire latency. Instance 1: ---------- Avg global cache cr block receive time (ms) : 222.1 Avg global cache current block receive time (ms) : 27.5 Instance 2: ---------- Avg global cache cr block build time (ms) : 0.0 Avg global cache cr block send time (ms) : 0.1 Global cache log flushes for cr blocks served % : 2.7 Avg global cache cr block flush time (ms) : 15879.9
29.
©OraInternals Riyaj Shamsudeen
29 Problem node Global Cache and Enqueue Services - Workload Characteristics Avg global enqueue get time (ms) : 0.1 Avg global cache cr block receive time (ms) : 14.4 Avg global cache current block receive time (ms): 23.4 Avg global cache cr block build time (ms) : 0.0 Avg global cache cr block send time (ms) : 0.0 Global cache log flushes for cr blocks served %: 25.0 Avg global cache cr block flush time (ms) : 2.5 Avg global cache current block pin time (ms) : 0.0 Avg global cache current block send time (ms): 0.0 Global cache log flushes for current blocks served %: 6.3 Avg global cache current block flush time (ms): 5.9
30.
Sending side analysis
..2 Avg message sent queue time (ms) : 0.1 Avg message sent queue time on ksxp (ms): 0.6 Avg message received queue time (ms): 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 46.62 % of indirect sent messages : 43.76 % of flow controlled messages : 9.62 Avg message sent queue time (ms) : 0.0 Avg message sent queue time on ksxp (ms): 0.4 Avg message received queue time (ms) : 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 60.86 % of indirect sent messages : 38.48 % of flow controlled messages : 0.66 ©OraInternals Riyaj Shamsudeen 30
31.
©OraInternals Riyaj Shamsudeen
31 Problem node Global Cache and Enqueue Services - Messaging Statistics Avg message sent queue time (ms) : 344.7 Avg message sent queue time on ksxp (ms): 0.3 Avg message received queue time (ms): 0.0 Avg GCS message process time (ms) : 0.0 Avg GES message process time (ms) : 0.0 % of direct sent messages : 31.76 % of indirect sent messages : 46.12 % of flow controlled messages : 22.13
32.
©OraInternals Riyaj Shamsudeen
32 Multi-tenancy Buffer cache and Global cache processing is common to all pluggable databases. A PDB generating more global cache workload can cause performance issues to other PDBs. There doesn’t seem to be a way to constrain a database from overusing buffer cache resource or global cache resources. Resource manager only controls CPU, PX, and I/O(exadata).
33.
Prepare for multi-tenancy
(RAC specific) Keep the buffer cache as large (for OLTP). Global cache overhead is reduced if the buffers are in the local buffer cache. Make LGWR efficient. LMS waits for LGWR to ship blocks. Tune the SQL statements to reduce global cache traffic. A misbehaving piece of application code affects the whole cluster. Connect to the container database to review global cache metrics. ©OraInternals Riyaj Shamsudeen 33
34.
©OraInternals Riyaj Shamsudeen
34 Resource names From 12 onwards, resource names are coded with con_id. select resource_name from gv$ges_resource where resource_name like '[0x15f29][0x0],[TM]%' / RESOURCE_NAME ------------------------------ [0x15f29][0x0],[TM][ext 0x3,0x0] <- hrdb1 PDB – GSTEST table –object_id=89897 [0x15f29][0x0],[TM][ext 0x4,0x0] <- hrdb2 PDB – GSTEST table –object_id=89897 Comes handy if you are debugging RAC trace files, to identify the PDB generating the errors.
35.
©OraInternals Riyaj Shamsudeen
35 GC statistics and PDBs Most Global cache statistics views do not provide granularity to a pluggable database. For example, select con_id, inst_id, count(*) from gv$instance_cache_transfer group by con_id, inst_id / CON_ID INST_ID COUNT(*) ---------- ---------- ---------- 0 2 4096 0 1 4096 Review AWR reports from the container database to review global cache performance issues.
36.
High Priority Processes
(from 12c) Parameter: _high_priority_processes Value : LMS*|VKTM|LM*|LCK0|GCR*|DIAG … kjfmPriCheck: 0 RTLMS taken, 1 RTLMS available kjfmPriCheck: automatic priority changes enabled - multi-inst mode kjfmPriCheck: # of real time LMS increasing from 0 to 1 … Parameter: _lm_lms_priority_dynamic Value : TRUE ©OraInternals Riyaj Shamsudeen 36
37.
©OraInternals Riyaj Shamsudeen
gc cr grants 2-way gc cr request FG1 LMS Open request File #5, blk #8 Read from the disk Inst 2 Inst 1 charge time to ‘gc cr grant 2-way’ Wire
38.
Event: GC cr
grants 2-way Time is accounted to this wait event, if the block is not in any of the buffer cache. Trace file will indicate this wait event followed by a disk read. nam='gc cr grant 2-way' ela= 659 p1=1 p2=88252 p3=1 obj#=77779 nam='db file sequential read' ela= 938 file#=1 block#=88252 blocks=1 obj#=77779 Typical latency is 1-2ms. Any thing above needs to be reviewed as these are light-wait events. Process sends a request to remote master LMS process and the LMS process simply responds with ‘read from disk’. This is another base line wait event to measure interconnect response time, as LMS processing for this event is minimal. ©OraInternals Riyaj Shamsudeen 38
39.
©OraInternals Riyaj Shamsudeen
Why DRM? FG1 LMS Read from the disk Inst 2 Inst 1 charge time to ‘gc cr grant 2-way’ Excessive amount of GC activity for One object!
40.
©OraInternals Riyaj Shamsudeen
40 DRM & grants Dynamic Remastering feature is to reduce or eliminate this specific RAC Tax: gc cr grants 2-way If the object is mastered locally, processes can read the block directly without incurring any GC waits. Identify the objects suffering from excessive ‘gc cr grants 2-way’ events (and remaster them manually if needed) using script ash_gcwait_to_obj.sql. INST_ID OWNER OBJECT_NAME OBJECT_TYPE CNT ---------- -------------------- -------------------------------- -------------------- ---------- 1 AR RA_CUSTOMER_TRX_ALL TABLE 9 1 INV MTL_MATERIAL_TRANSACTIONS TABLE 141 1 INV MTL_SERIAL_NUMBERS TABLE 155 1 Undo Block 253 1 INV MTL_SYSTEM_ITEMS_B TABLE 476 1 AP AP_CHECKS_ALL TABLE 785
41.
©OraInternals Riyaj Shamsudeen
41 Local vs Remote In AWR report, identify the local vs remote grants also. Following shows that 74% of grants are remote. DRM might be of help here, if there are much waits for ‘gc grants 2-way’. gc local grants 6,570,658 3,643.6 21.8 gc remote grants 19,262,806 10,681.7 63.9 Increase SGA size to reduce single block I/O reads. Tuning SQL statements to reduce logical I/O, optimal indexes, etc can help to improve the performance.
42.
©OraInternals Riyaj Shamsudeen
42 DRM effect Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time Wait Class ------------------------------ ------------ ----------- ------ ------ ---------- gc buffer busy 1,826,073 152,415 83 62.0 Cluster CPU time 30,192 12.3 enq: TX - index contention 34,332 15,535 453 6.3 Concurrency gcs drm freeze in enter server 22,789 11,279 495 4.6 Other enq: TX - row lock contention 46,926 4,493 96 1.8 Applicatio Global Cache and Enqueue Services - Workload Characteristics ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Avg global enqueue get time (ms) : 16.8 Avg global cache cr block receive time (ms) : 17.1 Avg global cache current block receive time (ms): 14.9
43.
©OraInternals Riyaj Shamsudeen
43 DRM (11g) LCK Request queue LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn Inst 1 Inst 2 Inst 3 Resources In 11g, all resources are frozen during the reconfiguration.
44.
©OraInternals Riyaj Shamsudeen
44 DRM (12c) LCK Request queue LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn LMD LMON LMS0 LMS1 LMSn Inst 1 Inst 2 Inst 3 Resources In 12c, only set of resources in a window is frozen.
45.
DRM impact is
decreased in 12c by freezing a resource partition, at a time.. ©OraInternals Riyaj Shamsudeen 45 DRM in 12c * drm quiesce 2014-02-15 16:03:45.131840 : DRM(3) resources quiesced [0-1023], rescount 1719 2014-02-15 16:03:45.132173 : DRM(3) local converts quiesced [0-1023], lockcount 0, bucket 0 * drm sync 1 * drm freeze * drm cleanup * drm sync 2 * drm replay * drm sync 3 * drm fix writes * drm sync 4 * drm end * DRM RCFG called (swin 0) CGS recovery timeout = 85 sec * drm quiesce 2014-02-15 16:03:45.172867 : DRM(3) resources quiesced [1024-2047], rescount 1754 2014-02-15 16:03:45.172932 : DRM(3) local converts quiesced [1024-2047], lockcount 0, bucket 0 * drm sync 1 * drm freeze
46.
©OraInternals Riyaj Shamsudeen
46 PQ trace file PQ Co-ordinator trace shows Object checkpoint. select /*+ parallel (a 4) */ min(length(v1)) from rs.hugetable a; WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 866 name| mode=1263468550 2=65581 0=1 obj#=-1 tim=6965638229 ... WAIT #18446741324874976768: nam='enq: KO - fast object checkpoint' ela= 454 name| mode=1263468545 2=65581 0=2 obj#=-1 tim=6965638986 ... PQ server process will perform direct reads to PGA. Notice that there is no GC waits between the reads. WAIT #18446741324874985288: nam='direct path read' ela= 4782 file number=4 first dba=11584 block cnt=16 obj#=76739 tim=3653439149 WAIT #18446741324874985288: nam='direct path read' ela= 16157 file number=4 first dba=11600 block cnt=16 obj#=76739 tim=3653456065 WAIT #18446741324874985288: nam='direct path read' ela= 3675 file number=4 first dba=11616 block cnt=16 obj#=76739 tim=3653462065 WAIT #18446741324874985288: nam='direct path read' ela= 8162 file number=4 first dba=11632 block cnt=16 obj#=76739 tim=3653471358 WAIT #18446741324874985288: nam='direct path read' ela= 10270 file number=4 first dba=11650 block cnt=14 obj#=76739 tim=3653482566
47.
©OraInternals Riyaj Shamsudeen
47 Application issues There is no easy fix in 12c either Avoid localized inserts.
48.
©OraInternals Riyaj Shamsudeen
48 Application issues Hash partitioning the index/table might be helpful.
49.
©OraInternals Riyaj Shamsudeen
49 Application issues Reduce localized accesses: Frequent access to a table block such as a lookup table: Cache it somewhere else. Aggressive scanning of smaller tables due to inefficient optimizer plans. Aggressive nested loops joins accessing huge number of index leaf blocks. Ordered cached sequences – Excessive SV resource contention. Excessive DDL in a multi-tenant databases.
50.
©OraInternals Riyaj Shamsudeen
50 Use Services Inst 1 PO FIN Inst 2 PO FIN Inst 3 PO FIN While creating a service, you define a preferred and available instances. PO FIN Preferred Available For PO service: Instance 1 & 2 are preferred instances, and instance 3 is an available instance.
51.
PDB1 ©OraInternals Riyaj
Shamsudeen 51 Or PDBs Inst 1 PDB1 PDB2 Inst 2 Inst 3 PDB1 is open in inst1 & inst2. PDB2 is open in inst2 & inst3 PO FIN Online Failover PDB1 PDB2 PDB2
52.
Statistics: gc lost
blocks This statistics tracks dropped interconnect packets. For 8K block size, block transfer would require 6 packets of size ~1500 bytes (non-Jumbo) to be transferred. Even if a packet is dropped, then the block need to be resent. Requesting process does not know the state of the request as UDP layer can not reassemble packet. (UDP is a send and forget type of protocol.) After a timeout (0.5s in 11g, ~5s in 10g) requesting process will declare that packet as lost, and request the block again. ©OraInternals Riyaj Shamsudeen 52
53.
Statistics: gc lost
blocks - Effect If the packets are lost, then the sessions can be seen waiting for an idle event. So, higher wait times for gc cr request, gc current request, and gc cr multiblock request will be visible. Since many blocks are requested during multi block reads, waits for ‘gc cr multiblock request’ will be a prominent wait event. Ideally, should be 0. But, usually, few drops are acceptable. You would need to judge the impact of lost block reviewing other wait events. ©OraInternals Riyaj Shamsudeen 53
54.
Statistics: gc lost
blocks - reason In a nutshell, packet is dropped We need to probe and understand why the packets are dropped. Few common reasons are: (In-depth discussion in Network) Insufficient UDP buffer space. High CPU usage. Incorrect network configuration so packets thrown away. Route between the nodes has many hops. Switch issues- configuration, firmware etc ©OraInternals Riyaj Shamsudeen 54
55.
©OraInternals Riyaj Shamsudeen
55 HugePages By default, pagesize is 4KB in Linux. PageTables size will be huge, in the order of, GigaBytes. For huge SGA sizes, this is problematic. kswapd deamon will consume more CPU during memory starvation and lead to server hung situations. CRS might not restart the server in time, this can lead to other nodes suffering from excessive gc buffer busy waits. HugePages uses a pagesize of 2MB. This reduces the size of PageTable to few 100MBs. In RAC & Linux, you should consider using HugePages.
56.
©OraInternals Riyaj Shamsudeen
56 THANK YOU Email: rshamsud@orainternals.com Blog : orainternals.wordpress.com Web: www.orainternals.com
Download Now