Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Oracle RAC 12c Release 2 and Clusterware
Internals
Anil Nair
Sr. Principal Product Manager,
Oracle Real Application Clusters (RAC)
Dec 25th, 2018
@RACMasterPM,
http://www.linkedin.com/in/anil-nair-01960b6
http://www.slideshare.net/AnilNair27/
20 Years of Continuous Innovation

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

• Please remember to provide
feedback
– Session ID Session ID: 1369
– Oracle RAC 12 Release 2 and
Clusterware Internals
• Reach out to us with follow up
questions, suggestions etc. via
Twitter @RACMasterPM
Linkedin http://www.linkedin.com/in/anil-nair-
01960b6
Feedback is important for successful collaboration

Program Agenda
Cluster Domain Architecture
RAC New Features
Sample Problems and Resolution
FAQ
1
2
4
3
4

Program Agenda
Cluster Domain Architecture1
2
5
3
4

Oracle RAC 12c Rel. 2 Cluster Domain
• Simplified Management
– Fleet Management for installation,
update, patching and maintenance
• Reduced Local Overhead
– Member Clusters benefit from the
consolidation of common services
on the Domain Services Cluster
• Improved IO Performance
– Utilizing consolidated shared storage
6
Centralized Management for Cluster Estates “too big to manage” otherwise

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7
Cluster Domain
Application
Member Cluster
Uses ASM
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service
Database
Member Cluster
Uses IO Service
Domain Services Cluster
(DSC)
Shared ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

The DSC – The Heart of the Cluster Domain
Domain Services Cluster (DSC)
IO
Service
ASM
Service
TFA
Service
Management
Service
Shared ASM
ACFS
Service
RHP
Service
• The DSC hosts services that are consumed by Member Clusters, including:
– Management Service for centralized and simplified management
– Trace File Analyzer (TFA) for centralized diagnostics
– Rapid Home Provisioning (RHP) for software fleet management
– Storage Services (ACFS, ASM direct or indirect over IO Service)

IO
Service
ASM
Service
Shared ASM
ACFS
Service
RHP
Service
9
The DSC Management Service
TFA
Service
Management
Service
Applied Machine Learning for Database Diagnostics
• Efficient diagnosis using Machine Learning
• Automatically performs corrective actions to
prevent possible issues
• Provides simple alerts & recommendations for
issues that require manual intervention Subject Matter
ExpertLog
ASH
Metrics
ML
Knowledge
Extraction
Model
Generation
Human
Supervision
Application
Optimized
Models
Feedback

Rapid Home Provisioning Service
IO
Service
ASM
Service
Shared ASM
ACFS
Service
RHP
Service
TFA
Service
Management
Service
• Provision new pools onto base machines
• DB and GI: provision, scale, patch, upgrade
• Custom workflow framework
• Notification model
• Audit capabilities
RHP
Service
Member
Cluster
Member
Cluster
Member
Cluster
Fleet Management installation, update, patching and maintenance

Domain Services Cluster (DSC) Availability
• Services provided by the DSC are
unaffected by nodes joining/
leaving the DSC cluster
• DSC can be patched and upgraded
independently without affecting
the services provided by DSC
• Use Member Clusters for user
databases (not the DSC)

Cluster Domain
Application
Member Cluster
Uses ASM
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service
Database
Member Cluster
Uses IO Service
(DSC)
Shared ASM
IO
Service
ACFS
Service
ASM
Service
TFA
Service
Management
Service
RHP
Service

• A Member Cluster is a Standalone
Cluster utilizing shared services on
the Domain Services Cluster
• It automatically benefits from the
management, TFA & RHP service.
– ASM services are optional
– and can be utilized as needed
13
Member Clusters = Standalone Cluster + Benefits
Database
Member Cluster
Uses local ASM
Database
Member Cluster
Uses ASM
Service of DSC
Database
Member Cluster
Uses ASM IO
Service

Same Tools and Commands for all Types of Deployments
[GRID]> crsctl get cluster name
CRS-6724: Current cluster name is 'SolarCluster'
[GRID]> crsctl get cluster class
CRS-41008: Cluster class is 'Standalone Cluster'
[GRID]> crsctl get cluster type
CRS-6539: The cluster type is 'flex'.
[GRID]> crsctl get cluster name
CRS-6724: Current cluster name is 'SalesCluster'
[GRID]> crsctl get cluster class
CRS-41008: Cluster class is ‘Database Member Cluster'
[GRID]> crsctl get cluster type
CRS-6539: The cluster type is 'flex'.

• Easy conversion from
Standalone to Member Clusters
• Member Cluster GI version can be
higher or equal to the GI version on
the Domain Services Cluster
15
Convert a Standalone Cluster to Member Cluster
Convert
Database
Member Cluster
Uses local ASM

Program Agenda
RAC New Features
1
2
16
3
4

All Deployments Benefit from Oracle RAC Features
PDB Isolation Hang
Manager
Choose the “right path”
Zero Impact Patching
Scalable Sequences

1010101000010011100101001
1011100101001110010100101
0010010100001111010100101
1010101000010011100101001010010100100111001010010100100
1011100101001110010100101111110100111001010010111101101
0010010100001111010100101001010100100110101001010101000
18
• Using Oracle Multitenant, PDBs can be opened
as singletons (in one database instance only), in
a subset of instances or all in instances at once.
• If certain PDBs are only opened on some
instances, Pluggable Database Isolation
– improves performance by
• Reducing DLM operations for
PDBs not open in all instances.
• Optimizing block operations based
on in-memory block separation.
– improves availability by
• Ensuring that instance failures of instances
only hosting singleton PDBs will not impact
other instances of the same RAC-based CDB.
Optimized Singleton Workload Scaling
Pluggable Database Isolation
NEW IN 12.2PRIOR TO 12.2 Inter Cluster Messages

Database
Member Cluster
Uses ASM Service
19
Pluggable Database and Service Isolation
Provides higher Isolation in Member Cluster Consolidated Environments

Hang Manager
• Hang Manager “manages” database
process(es) that are not progressing
– Manage = detection and resolution
– Considers cross-layer hangs
• I.E: Hangs caused by a blocked ASM resource.
• In addition, deadlocks are now
resolved by Hang Manager
• User defined control via PL/SQL
• Early Warning exposed via (V$ view)
20
Resolving cross-layer hangs in the Cluster Domain
Database
Member Cluster
Uses ASM IO
Service
IO
Service
ASM
Service

Resolving cross-layer hangs in the Cluster Domain
Provides higher Availability in Member Cluster Consolidated Environments
Database
Member Cluster
Uses ASM Service
W W
W
B

Oracle RAC Always Chooses the “Right Path”
22
• RAC determines the optimal path
to serve blocks – network or disk
• SSDs and NVMe storage technology
continue to drive down latency
• e.g. flash storage may provide better
access times to data than the private
network under high load
• RAC takes those statistics into account
Query
BlockBlock
Network
congestion

Database
Member Cluster
Uses ASM Service
23
The “Right Path” Ensures Performance on Member Clusters
Effectively reducing the impact of competing workloads overwhelming the I/O on the DSC
B
u
s
y

Zero Impact Patching
• Zero Impact Patching enables patching of the Oracle Grid Infrastructure without
interrupting database operations.
• Patches are applied out-of-place and in a rolling fashion with one node being patched at
a time while the database instance(s) on that node remain up and running.
• Zero Impact Patching supports Oracle Real Application Clusters (RAC) databases on
clusters with two or more nodes.
24
Never take down a database instance to patch Grid Infrastructure

Operational Details of Zero Impact Patching
• During patch application, the stack is monitored by a special CSSD running out of place
• This CSSD performs Heartbeat operations
• Once the patch is done, the special CSSD concedes control to the CSSD running out of
the patched home
• The steps are performed on all the nodes until the entire cluster is patched.
• Any interruptions in normal activity during patching will trigger the CSSD to evict the
node.
25

• Sequences cause Index block
contention during heavy OLTP
especially on tables which use
sequence values as keys
• This is not RAC specific but the
problem gets amplified in RAC
environments
• The effects of this issue is mitigated
via Scalable Sequences
26
Scalable Sequences

• New option in Create/Alter Sequence
SCALE {Extend |NoExtend | NoScale }
• When Scale is specified an internal
algorithm uses the instance id and
other session specific information to
generate an offset
• DBAs need to alter existing sequences
• DBA/User/ALL_sequences will reflect
the additional attributes
27
Scalable Sequences Operational details

Program Agenda
1
2
28
3
4

29
Problem 1 – Node Eviction

Node Eviction – Node 1 ocssd.trc [ Part 1 ]
• 2010-08-13 17:00:22.818: [ CSSD][4106599328]
clssnmPollingThread: node anair2 (2) at 50% heartbeat fatal,
removal in 14.520 seconds
• 2010-08-13 17:00:29.833: [ CSSD][4106599328]
• 2010-08-13 17:00:37.337: [ CSSD][4106599328]
clssnmPollingThread: Removal started for node anair2 (2), flags
• 2010-08-13 17:00:37.340: [ CSSD][4085619616]clssnmCheckSplit:
Node 2, anair2, is alive, DHB (1281744040, 1396854) more than disk
timeout of 27000 after the last NHB (1281744011, 1367154)
30
Network heartbeat missing
from node 2 for 15
consecutive seconds
Network heartbeat is still
missing
Finally eviction starts
Node 2 is still updating the
Voting disks

• 2010-08-13 17:00:37.340: [
CSSD][4085619616](:CSSNM00007:) clssnmrEvict: Evicting
node 2, anair2, from the cluster in incarnation 169934272,
node birth incarnation 169934271, death incarnation
169934272, stateflags 0x24000
• 2010-08-13 17:01:07.705: [
CSSD][4043389856]clssgmCMReconfig: reconfiguration
successful, incarnation 169934272 with 1 nodes, local node
number 1, master node number 1
31
Eventually Node 2 eviction
process starts
Concluding with a
reconfiguration

• 2010-08-13 17:00:26.213: [ CSSD][4073040800]
• 2010-08-13 17:00:40.702: [ CSSD][4073040800]
clssnmPollingThread: Removal started for node anair1 (1), flags
• 2010-08-13 17:00:40.706: [ CSSD][4052061088]clssnmCheckSplit:
Node 1, anair1, is alive, DHB (1281744036, 1243744) more than disk
timeout of 27000 after the last NHB (1281744007, 1214144)
32
Huh!
So who is right?
It also detects that
Node 1 is still updating
the Voting disks

• 2010-08-13 17:00:40.707: [
CSSD][4052061088](:CSSNM00008:)clssnmCheckDskInfo:
Aborting local node to avoid splitbrain. Cohort of 1 nodes
with leader 2, anair2, is smaller than cohort of 1 nodes led
by node 1, anair1, based on map type 2
• 2010-08-13 17:00:40.707: [ CSSD]
[4052061088]###################################
2010-08-13 17:00:40.707: [ CSSD][4052061088]clssscExit:
CSSD aborting from thread clssnmRcfgMgrThread
2010-08-13 17:00:40.707: [ CSSD]
[4052061088]###################################
33
it correctly detects and
aborts local node to
prevent split brain
And does the right thing

So what is the conclusion ?
34
Hmmm….I think it is the network.

What does OS Watcher say?
• netstat does not show any issues
# grep "zzz|udpInOverflows|ipReasmFails"
• OSW data itself is missing possibly due to scheduling issues
• Just prior to the issue Top reports
top - 13:23:52 up 25 days, 21:08, 1 user, load average: 3.43, 3.01, 3.02
Cpu(s): 16.8%us, 23.2%sy, 0.0%ni, 56.5%id, 3.1%wa, 0.1%hi, 0.3%si,
Mem: 74027752k total, 73689744k used, 338008k free, 1516k
Swap: 16771852k total, 9069988k used, 7701864k free, 25836528k
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1049 root 11 -5 0 0 0 R 55.0 0.0 0:57.42 [kswapd0]
35
No issues seen in netstat &
traceroute
Oh no! It is swapping
Why is the OSW data
missing?

36
Problem 2 – Slow I/O

• Start the Database on all instances to
simulate physical reads
• Simulate I/O on shared storage
– Loading data (sqlldr,expdp) OR
– Swingbench
• **behavior may depend on your test setup, HCA, HBA
etc
37
Simulate I/O performance issue
swingbench swingbench

chactl query diagnosis-db
38
$ chactl query diagnosis -db sales -start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_1) [detected]
2017-03-31 10:01:10.0 Database sales DB Control File IO Performance (sales_2) [detected]
2017-03-31 10:01:13.0 Database sales DB CPU Utilization (sales_2) [detected]
2017-03-31 10:01:33.0 Database salesDB Log File Switch (sales_1) [detected]
Consolidates and displays
information from all instances

• iostat confirms I/O performance issue
archive/oswiostat/xxxxxxxx_iostat_17.03.31.1000.dat
• Increase in Reads/writes
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 4.00 0.00 32.00 8.00 0.00 0.25 0.25 0.10
xvdb 0.00 0.00 6.50 2.00 146.00 21.00 19.65 0.00 0.24 0.24 0.20
• ---
• ---
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
xvda 0.00 0.00 0.00 144.00 434.00 53.00 12.00 0.00 0.45 1.13 0.10
xvdb 0.00 0.00 6.50 219.00 414.00 67.00 13.65 0.00 0.33 9.44 0.20
39
There is more overall I/O as
seen by the increase in the
number of writes, reads
* values may change
depending on test env.
**Output has been formatted
for presentation

40
Problem 3 – CPU bound

• Ensure Grid Infrastructure is
running for at least an hour
• Run some normal load
• Simulate excessive CPU using a CPU
stressing program on 3 nodes
– stress -- C program
41
Simulate CPU load
stress stress stress

chactl query diagnosis -cluster
42
$chactl query diagnosis –start "2017-03-31 10:00:50" -end "2017-03-31 10:25:50“
2017-03-31 10:01:25.0 Host rwsxxxxx9 Host Memory Consumption [detected]
Problem: Host Memory Consumption
Description: CHA detected that more memory than expected is consumed on this server. The
memory is not allocated by sessions of this database.
Cause: The Cluster Health Advisor (CHA) detected an increase in memory consumption by other
databases or by applications not connected to a database on this node.
Action: Identify the top memory consumers by using the Cluster Health Monitor (CHM).
Note: This is a 4 node cluster but the memory
consumption issue is reported on 3 of the 4 nodes

chatcl actions & resolutions in text or html format
43

• OSWatcher confirms chactl diagnosis
• mpstat (platform dependent)
zzz ***Fri Mar 31 10:10:29 PST 2017
10:10:29 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
10:10:29 all 74.01 0.00 7.95 12.20 0.00 0.13 0.13 0.00 5.10
10:10:29 0 44.89 0.00 8.79 12.09 0.00 0.00 0.00 0.00 31.23
10:10:29 1 56.00 0.00 4.00 18.00 0.00 0.00 0.00 0.00 23.00
• Top also reports the CPU stress program
Tasks: 454 total, 4 running, 450 sleeping, 0 stopped, 0 zombie
Cpu(s): 94.9%us, 4.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20752 racusr 20 0 2498m 45m 39m R 85.9 0.3 752:32.40 stress
44
System is indeed CPU
starved
* values may change
depending on test env.
**Output has been formatted
for presentation

45
Problem 4 – Why did my Database
Instance move to different node?

• Provides common view of cluster-
wide activities in a co-ordinated
fashion
• Customer readable summary of all
actions in a cluster
• Each activity is a summary of the
issue
– Further details are in the trace files
46
Clusterware Activity Log
On Friday, my instances
were running on Nodes
1 & 2, but today it is
only running on Node 1

crsctl query calog -- Syntax
$ crsctl query calog -h
Usage:
crsctl query calog [-aftertime <after_timestamp>] [-beforetime <before_timestamp>] [-duration <time_interval> | -
follow] [-filter <filter_expression>] [-fullfmt | -xmlfmt]
Lists the cluster activity log activities matching the specified criteria
Where
after_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR
"HH24:MI:SS[.FF][[+-]HH:MM]"
before_timestamp Time in format "YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]" OR "YYYY-MM-DD" OR
"HH24:MI:SS[.FF][[+-]HH:MM]"
time_interval Time interval in format "DD HH:MM:SS"
filter_expression Refer to product documentation.
-follow Gives continuous output
-fullfmt Detail output format
-xmlfmt XML output format

crsctl query calog output
$ crsctl query calog -aftertime "2017-03-08 15:09:46.522-07:00"
2017-04-12 20:05:04.668000 : Attempting to start 'ora.anair1.vip' on ‘anair1' :
14920191617156230/1194/11 :
…..
2017-04-12 20:05:06.559000 : Attempting to start 'ora.LISTENER.lsnr' on ‘anair1' :
14920191617156230/1194/16
…
2017-04-12 20:05:32.038000 : Start of 'ora.FRA.dg' on ‘anair1' succeeded : 14920191617156230/1194/27
:
2017-04-12 20:05:32.040000 : Attempting to start 'ora.sales.db' on ‘anair1' :
14920191617156230/1194/28 :
2017-04-12 20:05:59.415000 : Start of 'ora.sales.db' on ‘anair1' succeeded :
14920191617156230/1194/30 :
-- Format of output records is:
DATE & TIME (YYYY-MM-DD HH24:MI:SS[.FF][[+-]HH:MM]): Event text: ACTID
Possibly network issues
caused VIP relocation

crsctl query calog filter output using actid
$ crsctl query calog -filter "actid == 14920191617156230/2449732/1"
2017-08-03 16:30:24.678000 : Attempting to start 'ora.sscdb.db' on ‘anair1' : 14920191617156230/2449732/1 :
2017-08-03 16:30:24.698000 : Start of 'ora.sscdb.db' on ‘anair1' succeeded : 14920191617156230/2449732/1 :
$ crsctl query calog -filter "actid ~= “14920191617156230”
2017-08-03 16:25:20.658000 : Stop of 'ora.sscdb.test.svc' on anair1' succeeded : 14920191617156230/2449007/2 :
-filter: Use ~= or == on actid to find related actions

crsctl query calog filtered output using name
$crsctl query calog -filter "entity_name == ora.sscdb.db"
2017-04-12 18:59:28.694000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/4 :
2017-04-12 18:59:28.800000 : Resource 'ora.sscdb.db' has been modified. : 14919830844562257/9074/5 :
…
2017-04-12 19:32:39.631000 : Attempting to stop 'ora.sscdb.db' on ‘anair2' : 14919830844562257/9853/2 :
2017-04-12 19:32:42.572000 : Stop of 'ora.sscdb.db' on ‘anair2' succeeded : 14919830844562257/9853/4 :
2017-04-12 19:45:01.175000 : Attempting to stop 'ora.sscdb.db' on ‘anair1' : 14919830844562257/10151/4 :
2017-04-12 19:45:04.176000 : Stop of 'ora.sscdb.db' on ‘anair1' succeeded : 14919830844562257/10151/7 :
50

Additional examples for querying calog
• crsctl query calog -duration "00 00:02:00“ << last two minutes >>
• crsctl query calog -duration "00 02:00:00“ << last two hours >>
• crsctl query calog –follow << like tail… >>
• crsctl query calog -aftertime "2017-03-08 17:49:12.397" –follow
• crsctl query calog -filter "entity_name == foo“ << activity on foo >>
• crsctl query calog -filter "actid == 145567015239630019/49543/1"
• crsctl query calog –xmlfmt << in xml format >>
51

Program Agenda
FAQ
1
2
52
3
4

Frequently asked Question # 1
Why does MGMT
DB need so much
space?

Cluster Type Redundancy MGMT DG
(GB)
(2 Node DSC with 4 Member
Clusters of 2 Nodes each)
External 188
Normal 376
High 564
Flex 376
Standalone Cluster
(4 Node Cluster)
External 38
Normal 76
High 114
Flex 76
• Oracle GI 12c Release 2 feature AHF
(Autonomous Health Framework)
collects, Co-relates & stores diagnostics
data from OS and DB in MGMT
• In DSC, one PDB per member cluster is
provisioned to store member cluster
diagnostics data
• The data is used by AHF components like
cluster health advisor to both prevent
and help diagnose issues
54
GIMR space requirements

Should I continue
to use
Oswatcher?

Should I continue to use OSWatcher?
• Hopefully by now, the value benefits of AHF is clear
• AHF continues to be enhanced to improve
– Diagnosing large number of deployments
– Correlating data to speed diagnosis
– Preventing issues in the first place
– Utilizing latest technologies like Machine learning
• AHF is mandatory (12.2+) but customers can choose to use OSW
along with AHF
56
Learn more about Autonomous Health Framework
https://www.slideshare.net/MarkusMichalewicz/cl
uster-health-advisor-cha-deep-dive-by-mark-
scardina

• opatch automatically patches
MGMT database if required
• clients of MGMT connect using
encrypted credentials
• MGMT listener automatically
maintained by the clusterware
agent
57
But it is still one
more database
for me to manage

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf

Similar to Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf(20)

Recently uploaded

Recently uploaded(20)

Rac 12c rel2_operational_best_practices_sangam_2017_as_pdf