Real Application Cluster (RAC) allows multiple computers to simultaneously run Oracle RDBMS while accessing a single database, providing clustering. RAC provides high availability, scalability, and ease of administration by making multiple instances transparent to users. Nodes must have identical environments. Oracle Clusterware manages node additions and removals. Instances from different nodes write to the same physical database. The presentation covers RAC architecture, components, startup sequence, single instance configuration, node eviction, and tips for monitoring and improving the RAC environment.

1. Real Application Cluster
RAC - Savior of DBA
Presenter : Nikhil Kumar

2. 2
WHO AM I?
Nikhil Kumar (DBA Manager)
6 Years of Experience in Oracle Databases and Apps.
Oracle Certified Professional Oracle 9i and 11g.
Worked on Mission critical Telecom (Vodafone and IDEA),
Financial ERP, Manufacturing and Government Domain.

3. 3
AGENDA
1. Introduction to Real Application Cluster
2. RAC - Savior of DBA
3. Oracle Clusterware (Platform on Platform)
4. RAC Startup sequence
5. RAC Architecture
6. RAC Components
7. Single Instance on RAC
8. Node Eviction
9. Important Log directories in RAC.
10. Tips to monitor and improve the RAC environment.

4. 4
I AM RAC
Real Application Cluster
 RAC allow multiple computers to run Oracle RDBMS
simultaneously while accessing a single database ,thus
providing clustering. Multiple Instance for single
database.
 Set of interconnected servers (nodes), acting as a single
server. Transparent to Users.
 High Availability ,Scalability, Ease of Administration.
 Nodes should be Identical to fulfill the cluster creation
environment (OS, Users and Network Segment etc).
 CRS manages node addition and removal(One Node to
100 Nodes).
 Instances of from different nodes writes same physical
database.
 RAC architecture enables users and applications to
benefit from the processing power of multiple machines.

5. 5
 Control file, Data file, Redo log files, Temp
files and SPFILE on shared storage.
 Every Instance will have its own Redo log files
undo Segments.
 Every instance has its own set of background
of process on clusterware level and database
level.
 All cache(DB buffer cache, Data dictionary
cache and Library cache) are synchronized by
cache fusion and resource managed globally.
 Backup and Recovery process can be
performed from any node of database.
 Users can connect to any node of server.
I AM RAC Cont…

6. 6
Project Detail:-
 Mission critical OLTP database(Telecom or Banking)
Database.
 Database availability SLA is 99.99% or 100%..
Maintenance Requirement:-
 OS Patching or Schedule bounce of OS.
 Database maintenance patch(CPU or PSU) .
 Static database parameter change (Due to bug or requirement
by the time).
 Hardware upgrade or change.
Hardisk failure, power failure or system failure.
 Prevention from Single point of failure?
Can we fulfill these requirement by single instance
database ?
RAC - The Savior of DBA

7. Oracle Clusterware (Platform on Platform)
Oracle clusterware is infrastructure which provides platform to Oracle
database run on shared mode (Active- Active).
Oracle Clusterware is acting as platform on OS platform to provide database availability in shared mode.

8. Oracle Clusterware Cont..
Benefits of clusterware:-
 Oracle clusterware detect issue and evict the problematic node to resolve the issues.
 Oracle clusterware restart the services if they stop for any reason.
 Oracle clusterware add addition node as per business requirement(Pay as you grow).
 Oracle clusterware add services or single database for automatic maintenance.
 Oracle clusterware can failover required services on surviving node to provide high availability.
 Oracle Clusterware eliminate planned and unplanned downtime(24X7 database availability).
 Automatic Load Balancing
 Scale and speedup.

9. 9
INIT spawns init.ohasd (with respawn) which
in turn starts the OHASD process . This
daemon spawns 4 processes.
Level 1: OHASD Spawns:
cssdagent - Agent responsible for spawning CSSD.
orarootagent - Agent responsible for managing all root owned ohasd resources.
oraagent - Agent responsible for managing all oracle owned ohasd resources.
cssdmonitor - Monitors CSSD and node health (along wth the cssdagent).
Level 2: OHASD rootagent spawns:
CRSD - Primary daemon responsible for managing cluster resources.
CTSSD - Cluster Time Synchronization Services Daemon
Diskmon
ACFS (ASM Cluster File System) Drivers
Level 2: OHASD oraagent spawns:
MDNSD - Used for DNS lookup
GIPCD - Used for inter-process and inter-node communication
GPNPD - Grid Plug & Play Profile Daemon
EVMD - Event Monitor Daemon
ASM - Resource for monitoring ASM instances
Level 3: CRSD spawns:
orarootagent - Agent responsible for managing all root owned crsd resources.
oraagent - Agent responsible for managing all oracle owned crsd resources.
Level 4: CRSD rootagent spawns:
Network resource - To monitor the public network
SCAN VIP(s) - Single Client Access Name Virtual Ips
Node VIPs - One per node
ACFS Registery - For mounting ASM Cluster File System
GNS VIP (optional) - VIP for GNS
Level 4: CRSD oraagent spawns:
ASM Resouce - ASM Instance(s) resource
Diskgroup - Used for managing/monitoring ASM diskgroups.
DB Resource - Used for monitoring and managing the DB and instances
SCAN Listener - Listener for single client access name, listening on SCAN VIP
Listener - Node listener listening on the Node VIP
Services - Used for monitoring and managing services
ONS - Oracle Notification Service
eONS - Enhanced Oracle Notification Service
GSD - For 9i backward compatibility
GNS (optional) - Grid Naming Service - Performs name resolution
For more information
Please refer 11gR2
Clusterware and Grid
Home What
You Need to Know
(Doc ID 1053147.1)
RAC Startup Sequence

10. RAC Architecture

11. 11
Clusterware Components
Oracle Clusterware high availability components :-
1. OCR
2. Voting Disk
Oracle Clusterware Network configuration:-
1. SCAN :- SCAN stands for “Single Client Access Name”. This is new and mandatory feature introduced in RAC11GR2.
It provides single name to client to access the database running on the cluster. We don’t need to change the
configuration of client tnsname.ora file if add or remove nodes in the cluster.
Network requirement for SCAN:- SCAN requires one single name to resolve 3 IP addresses fixed in DNS on the basis
of round robin algorithm.
SCAN is given at time of grid installation interview phase. After installation 3 scan listeners are get created, which can
relocate to nodes according to load.
SCAN listeners are run from grid home and directly depends on VIP.
For more information Grid Infrastructure Single Client Access Name (SCAN) Explained (Doc ID 887522.1)

12. 12
Clusterware Components Cont..
Tradition entry of TNSNAMES.ORA file:-
New entry of tnsname.ora:-
Scan Parameter defined in database:-
Parameter defined on the database level:-

13. 13
Clusterware Components Cont..
2. VIP :- Virtual IP address which used mainly for failover of connection in case of node eviction. VIP can float on
nodes. We put the VIP address in Local listener parameter.
3. Public IP :- Public IP is address is physical machine or node.
4. Private IP :- Private IP used of internal communication of interconnected node which is connected to high speed switch.
Interconnect :- Cluster interconnect is very important private network used for intercommunication of nodes.
Heartbeat (network ping) and memory channel between the nodes. Network pings are performed by
cssd service.
Helps is cache fusion.
Wait events can be seen in AWR report if low speed switch is being used(100gbs switch is recommended).

14. 14
Process of fusing buffer cache of more than one instance
to fulfill the block requirement of other nodes is called
Cache Fusion.
Cache Fusion uses a high-speed IPC interconnect to
provide cache-to-cache transfers of data blocks between
instances in a cluster. This data block shipping eliminates
the disk I/O and optimizes read/write concurrency.
Processes involves in Cache fusion are :-
Global Cache Service:- Process responsible for
transferring block from one node to other.
Global Enqueue Service :- Holds the information about
lock on the buffer. It also performs distributed deadlock
detections.
Global Resource Directory :- GRD is present on each
instance of the cluster. It keeps the list of buffer on which
node they are mastered.
If objects is being used very frequently one specific node,
then that node becomes the master of that object and
same information passed to all GRD of the nodes. When
same block is being requested by user request then
information is read from GRD to locate the master node to
fetch the block for fast retrieval.
Cache Fusion

15. 15
Oracle Cluster Registry (OCR)
Oracle Cluster Registry stores configuration information of oracle clusterware and RAC database resources (Database,
VIP, Listener, Disk group, Scan and other services). OCR is created at time of clusterware installation. Voting disk and
OCR automatic backup kept together in a single file.
Automatic backup of Voting disk and OCR happen after every four hours, end of the day, end of the week
OCR is managed by mainly thee command utilities:-
a) ocrcheck
b) ocrconfig
c) ocrdump

16. 16
It manage information about node membership. Each voting
disk must be accessible by all nodes in the cluster.
Its primary purpose is help in the situation when private
network communication fails.
Clusterware has 2 types of heartbeats, Which is being
monitored by CSSD service in 2 way communication:-
1. Network Hearbeat Private Interconnect
2. Disk Heartbeat Voting disk based communication
Each node in the cluster is pinged every second.
Voting Disk

17. 17
Voting Disk Cont…
Some facts about Voting Disk:-
Voting disk contain two types of data:
Static :- Node membership information in the cluster.
Dynamic: Disk Heartbeat Logging.
We can dynamically add or replace voting disk..
Voting disk got identity since 11.2.0.1, Thus backup of Voting disk using “dd” command not supported anymore.
Voting disk and OCR can be keep in same disk-group or different disk-group
You must have root or sudo privilege account to manage it

18. 18
Node Eviction
 Node Eviction is one of feature of RAC to prevent whole cluster from getting hanged or down.
 If any node is not passing heat-beat across other node or voting disk, then that node will be evicted from the cluster.
This is done to avoid from Spit Brain condition.
 This is one of important process of RAC to prevent the database consistency, So that no node can write data
independently on database files.
Which node gets evicted:-
Voting and heartbeat communication is used to determine the node. Once it is determine which cluster needs to evicted,
then CSSD is requested to kill itself to evict the node from cluster. If CSSD is hang or not responded that
OCSSDMONITOR take over and kill itself to evict the node.
Which node survives:-
In 2 node cluster instance with lowest node number will survive.
In more than 2 node or n node cluster a biggest subcluster will survive. A subcluster which has access to maximum number
of voting disk in the cluster.
See Also Mos note 1050693.1 - Troubleshooting 11.2 clusterware node evictions.
See Also Mos note 1549954.1 - RAC Node Eviction Troubleshooting Tool

19. 19
Re-Bootless Node Eviction
 Prior to 11gr2 clusterware node eviction means “reboot” of the problematic node.
 From 11GR2 node eviction mean – graceful reboot of clusterware instead of system/node reboot for failures like
network interconnect heartbeat and network interconnect.
 All processes of clusterware and database will be killed(mainly IO processes) of problematic node. Once all processes
get stopped successfully . OHASD service restart the cluster.
Scenario when reboot of node will be required:-
 If successful check of kill fail for I/O process then node reboot will be performed.
 If cssd service get killed during this process, then reboot will be performed.
 If clusterware stack shutdown for long time then reboot will be performed.

20. 20
Single Instance on RAC
Suppose currently you have small business and near future you are expected to grow in business or requirement and you
think today I don’t need RAC database but after 2 months I may require to migrate on RAC
Migration factors:- Migration complication from single instance machine to RAC machine.
Performance issue.
Extra Resource utilization
What should I do ?
 Well answer is you can have single instance on oracle clusterware, when business grow in term of users and
transaction with time and you feel that you are facing performance issue with single instance database. You can covert
that database from non-cluster database to cluster database by using following tools:-
1. rconfig utility.
2. dbca
3. EM grid.
All above mentioned tools can be used to change the non-clusterware database clusterware database.

21. 21
Important Log Locations
Clusterware daemon logs are all under
<GRID_HOME>/log/<nodename>. Structure under
<GRID_HOME>/log/<nodename>:
alert<NODENAME>.log - look here first for most clusterware
issues
./admin:
./agent:
./agent/crsd:
./agent/crsd/oraagent_oracle:
./agent/crsd/ora_oc4j_type_oracle:
./agent/crsd/orarootagent_root:
./agent/ohasd:
./agent/ohasd/oraagent_oracle:
./agent/ohasd/oracssdagent_root:
./agent/ohasd/oracssdmonitor_root:
./agent/ohasd/orarootagent_root:
./client:
./crsd:
./cssd:
./ctssd:
./diskmon:
./evmd:
./gipcd:
./gnsd:
./gpnpd:
./mdnsd:
./ohasd:
./racg:
./racg/racgeut:
./racg/racgevtf:
./racg/racgmain:
./srvm:
The cfgtoollogs dir under <GRID_HOME> and $ORACLE_BASE
contains other important logfiles. Specifically for rootcrs.pl and
configuration assistants like ASMCA, etc...
ASM logs live under $ORACLE_BASE/diag/asm/+asm/<ASM
Instance Name>/trace
The diagcollection.pl script under <GRID_HOME>/bin can be used
to automatically collect important files for support. Run this as the
root user.
Important log directories on RAC

22. 22
Tips to monitor and improve the RAC environment.
There are so many tool and which can improve your configuration of RAC.
1. ORAchk Tool: Proactively scans for the most impactful problems across the various layers of your stack. It perform check on OS Level,
Cluster level, database level and Network Level and provide solution according to that.
It comes with clusterware binaries but recommended to download the latest version form oracle support.
Also refer ORAchk - Oracle Configuration Audit Tool (Doc ID 1268927.2)
2. OSWatcher and/or CHM Cluster health monitor :- These tools are used to monitor the operating and clusterware level process and
record them in log according to your retention policy.
Also refer OSWatcher (Includes: [Video]) (Doc ID 301137.1)
3. RAC AWR and ADDM reports:- RAC comes with few more AWR report functionality.
awrgrpt:- It gives you the detail information of all nodes of the cluster in single report.
awrgdrpt:- Its gives you the difference report b/w 2 awr reports to deal with performance issue.
addmrpt:- Gives you the recommendation to improve the SQL query, database and cluster component.

23. Q/A

24. Thank You