RAC - The Savior of DBA


Published on

Introduction to Real Application Cluster
RAC - Savior of DBA
Oracle Clusterware (Platform on Platform)
RAC Startup sequence
RAC Architecture
RAC Components
Single Instance on RAC
Node Eviction
Important Log directories in RAC.
Tips to monitor and improve the RAC environment.

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

RAC - The Savior of DBA

  1. 1. Real Application Cluster RAC - Savior of DBA Presenter : Nikhil Kumar
  2. 2. 2 WHO AM I? Nikhil Kumar (DBA Manager) 6 Years of Experience in Oracle Databases and Apps. Oracle Certified Professional Oracle 9i and 11g. Worked on Mission critical Telecom (Vodafone and IDEA), Financial ERP, Manufacturing and Government Domain.
  3. 3. 3 AGENDA 1. Introduction to Real Application Cluster 2. RAC - Savior of DBA 3. Oracle Clusterware (Platform on Platform) 4. RAC Startup sequence 5. RAC Architecture 6. RAC Components 7. Single Instance on RAC 8. Node Eviction 9. Important Log directories in RAC. 10. Tips to monitor and improve the RAC environment.
  4. 4. 4 I AM RAC Real Application Cluster  RAC allow multiple computers to run Oracle RDBMS simultaneously while accessing a single database ,thus providing clustering. Multiple Instance for single database.  Set of interconnected servers (nodes), acting as a single server. Transparent to Users.  High Availability ,Scalability, Ease of Administration.  Nodes should be Identical to fulfill the cluster creation environment (OS, Users and Network Segment etc).  CRS manages node addition and removal(One Node to 100 Nodes).  Instances of from different nodes writes same physical database.  RAC architecture enables users and applications to benefit from the processing power of multiple machines.
  5. 5. 5  Control file, Data file, Redo log files, Temp files and SPFILE on shared storage.  Every Instance will have its own Redo log files undo Segments.  Every instance has its own set of background of process on clusterware level and database level.  All cache(DB buffer cache, Data dictionary cache and Library cache) are synchronized by cache fusion and resource managed globally.  Backup and Recovery process can be performed from any node of database.  Users can connect to any node of server. I AM RAC Cont…
  6. 6. 6 Project Detail:-  Mission critical OLTP database(Telecom or Banking) Database.  Database availability SLA is 99.99% or 100%.. Maintenance Requirement:-  OS Patching or Schedule bounce of OS.  Database maintenance patch(CPU or PSU) .  Static database parameter change (Due to bug or requirement by the time).  Hardware upgrade or change. Hardisk failure, power failure or system failure.  Prevention from Single point of failure? Can we fulfill these requirement by single instance database ? RAC - The Savior of DBA
  7. 7. Oracle Clusterware (Platform on Platform) Oracle clusterware is infrastructure which provides platform to Oracle database run on shared mode (Active- Active). Oracle Clusterware is acting as platform on OS platform to provide database availability in shared mode.
  8. 8. Oracle Clusterware Cont.. Benefits of clusterware:-  Oracle clusterware detect issue and evict the problematic node to resolve the issues.  Oracle clusterware restart the services if they stop for any reason.  Oracle clusterware add addition node as per business requirement(Pay as you grow).  Oracle clusterware add services or single database for automatic maintenance.  Oracle clusterware can failover required services on surviving node to provide high availability.  Oracle Clusterware eliminate planned and unplanned downtime(24X7 database availability).  Automatic Load Balancing  Scale and speedup.
  9. 9. 9 INIT spawns init.ohasd (with respawn) which in turn starts the OHASD process . This daemon spawns 4 processes. Level 1: OHASD Spawns: cssdagent - Agent responsible for spawning CSSD. orarootagent - Agent responsible for managing all root owned ohasd resources. oraagent - Agent responsible for managing all oracle owned ohasd resources. cssdmonitor - Monitors CSSD and node health (along wth the cssdagent). Level 2: OHASD rootagent spawns: CRSD - Primary daemon responsible for managing cluster resources. CTSSD - Cluster Time Synchronization Services Daemon Diskmon ACFS (ASM Cluster File System) Drivers Level 2: OHASD oraagent spawns: MDNSD - Used for DNS lookup GIPCD - Used for inter-process and inter-node communication GPNPD - Grid Plug & Play Profile Daemon EVMD - Event Monitor Daemon ASM - Resource for monitoring ASM instances Level 3: CRSD spawns: orarootagent - Agent responsible for managing all root owned crsd resources. oraagent - Agent responsible for managing all oracle owned crsd resources. Level 4: CRSD rootagent spawns: Network resource - To monitor the public network SCAN VIP(s) - Single Client Access Name Virtual Ips Node VIPs - One per node ACFS Registery - For mounting ASM Cluster File System GNS VIP (optional) - VIP for GNS Level 4: CRSD oraagent spawns: ASM Resouce - ASM Instance(s) resource Diskgroup - Used for managing/monitoring ASM diskgroups. DB Resource - Used for monitoring and managing the DB and instances SCAN Listener - Listener for single client access name, listening on SCAN VIP Listener - Node listener listening on the Node VIP Services - Used for monitoring and managing services ONS - Oracle Notification Service eONS - Enhanced Oracle Notification Service GSD - For 9i backward compatibility GNS (optional) - Grid Naming Service - Performs name resolution For more information Please refer 11gR2 Clusterware and Grid Home What You Need to Know (Doc ID 1053147.1) RAC Startup Sequence
  10. 10. RAC Architecture
  11. 11. 11 Clusterware Components Oracle Clusterware high availability components :- 1. OCR 2. Voting Disk Oracle Clusterware Network configuration:- 1. SCAN :- SCAN stands for “Single Client Access Name”. This is new and mandatory feature introduced in RAC11GR2. It provides single name to client to access the database running on the cluster. We don’t need to change the configuration of client tnsname.ora file if add or remove nodes in the cluster. Network requirement for SCAN:- SCAN requires one single name to resolve 3 IP addresses fixed in DNS on the basis of round robin algorithm. SCAN is given at time of grid installation interview phase. After installation 3 scan listeners are get created, which can relocate to nodes according to load. SCAN listeners are run from grid home and directly depends on VIP. For more information Grid Infrastructure Single Client Access Name (SCAN) Explained (Doc ID 887522.1)
  12. 12. 12 Clusterware Components Cont.. Tradition entry of TNSNAMES.ORA file:- New entry of tnsname.ora:- Scan Parameter defined in database:- Parameter defined on the database level:-
  13. 13. 13 Clusterware Components Cont.. 2. VIP :- Virtual IP address which used mainly for failover of connection in case of node eviction. VIP can float on nodes. We put the VIP address in Local listener parameter. 3. Public IP :- Public IP is address is physical machine or node. 4. Private IP :- Private IP used of internal communication of interconnected node which is connected to high speed switch. Interconnect :- Cluster interconnect is very important private network used for intercommunication of nodes. Heartbeat (network ping) and memory channel between the nodes. Network pings are performed by cssd service. Helps is cache fusion. Wait events can be seen in AWR report if low speed switch is being used(100gbs switch is recommended).
  14. 14. 14 Process of fusing buffer cache of more than one instance to fulfill the block requirement of other nodes is called Cache Fusion. Cache Fusion uses a high-speed IPC interconnect to provide cache-to-cache transfers of data blocks between instances in a cluster. This data block shipping eliminates the disk I/O and optimizes read/write concurrency. Processes involves in Cache fusion are :- Global Cache Service:- Process responsible for transferring block from one node to other. Global Enqueue Service :- Holds the information about lock on the buffer. It also performs distributed deadlock detections. Global Resource Directory :- GRD is present on each instance of the cluster. It keeps the list of buffer on which node they are mastered. If objects is being used very frequently one specific node, then that node becomes the master of that object and same information passed to all GRD of the nodes. When same block is being requested by user request then information is read from GRD to locate the master node to fetch the block for fast retrieval. Cache Fusion
  15. 15. 15 Oracle Cluster Registry (OCR) Oracle Cluster Registry stores configuration information of oracle clusterware and RAC database resources (Database, VIP, Listener, Disk group, Scan and other services). OCR is created at time of clusterware installation. Voting disk and OCR automatic backup kept together in a single file. Automatic backup of Voting disk and OCR happen after every four hours, end of the day, end of the week OCR is managed by mainly thee command utilities:- a) ocrcheck b) ocrconfig c) ocrdump
  16. 16. 16 It manage information about node membership. Each voting disk must be accessible by all nodes in the cluster. Its primary purpose is help in the situation when private network communication fails. Clusterware has 2 types of heartbeats, Which is being monitored by CSSD service in 2 way communication:- 1. Network Hearbeat Private Interconnect 2. Disk Heartbeat Voting disk based communication Each node in the cluster is pinged every second. Voting Disk
  17. 17. 17 Voting Disk Cont… Some facts about Voting Disk:- Voting disk contain two types of data: Static :- Node membership information in the cluster. Dynamic: Disk Heartbeat Logging. We can dynamically add or replace voting disk.. Voting disk got identity since, Thus backup of Voting disk using “dd” command not supported anymore. Voting disk and OCR can be keep in same disk-group or different disk-group You must have root or sudo privilege account to manage it
  18. 18. 18 Node Eviction  Node Eviction is one of feature of RAC to prevent whole cluster from getting hanged or down.  If any node is not passing heat-beat across other node or voting disk, then that node will be evicted from the cluster. This is done to avoid from Spit Brain condition.  This is one of important process of RAC to prevent the database consistency, So that no node can write data independently on database files. Which node gets evicted:- Voting and heartbeat communication is used to determine the node. Once it is determine which cluster needs to evicted, then CSSD is requested to kill itself to evict the node from cluster. If CSSD is hang or not responded that OCSSDMONITOR take over and kill itself to evict the node. Which node survives:- In 2 node cluster instance with lowest node number will survive. In more than 2 node or n node cluster a biggest subcluster will survive. A subcluster which has access to maximum number of voting disk in the cluster. See Also Mos note 1050693.1 - Troubleshooting 11.2 clusterware node evictions. See Also Mos note 1549954.1 - RAC Node Eviction Troubleshooting Tool
  19. 19. 19 Re-Bootless Node Eviction  Prior to 11gr2 clusterware node eviction means “reboot” of the problematic node.  From 11GR2 node eviction mean – graceful reboot of clusterware instead of system/node reboot for failures like network interconnect heartbeat and network interconnect.  All processes of clusterware and database will be killed(mainly IO processes) of problematic node. Once all processes get stopped successfully . OHASD service restart the cluster. Scenario when reboot of node will be required:-  If successful check of kill fail for I/O process then node reboot will be performed.  If cssd service get killed during this process, then reboot will be performed.  If clusterware stack shutdown for long time then reboot will be performed.
  20. 20. 20 Single Instance on RAC Suppose currently you have small business and near future you are expected to grow in business or requirement and you think today I don’t need RAC database but after 2 months I may require to migrate on RAC Migration factors:- Migration complication from single instance machine to RAC machine. Performance issue. Extra Resource utilization What should I do ?  Well answer is you can have single instance on oracle clusterware, when business grow in term of users and transaction with time and you feel that you are facing performance issue with single instance database. You can covert that database from non-cluster database to cluster database by using following tools:- 1. rconfig utility. 2. dbca 3. EM grid. All above mentioned tools can be used to change the non-clusterware database clusterware database.
  21. 21. 21 Important Log Locations Clusterware daemon logs are all under <GRID_HOME>/log/<nodename>. Structure under <GRID_HOME>/log/<nodename>: alert<NODENAME>.log - look here first for most clusterware issues ./admin: ./agent: ./agent/crsd: ./agent/crsd/oraagent_oracle: ./agent/crsd/ora_oc4j_type_oracle: ./agent/crsd/orarootagent_root: ./agent/ohasd: ./agent/ohasd/oraagent_oracle: ./agent/ohasd/oracssdagent_root: ./agent/ohasd/oracssdmonitor_root: ./agent/ohasd/orarootagent_root: ./client: ./crsd: ./cssd: ./ctssd: ./diskmon: ./evmd: ./gipcd: ./gnsd: ./gpnpd: ./mdnsd: ./ohasd: ./racg: ./racg/racgeut: ./racg/racgevtf: ./racg/racgmain: ./srvm: The cfgtoollogs dir under <GRID_HOME> and $ORACLE_BASE contains other important logfiles. Specifically for rootcrs.pl and configuration assistants like ASMCA, etc... ASM logs live under $ORACLE_BASE/diag/asm/+asm/<ASM Instance Name>/trace The diagcollection.pl script under <GRID_HOME>/bin can be used to automatically collect important files for support. Run this as the root user. Important log directories on RAC
  22. 22. 22 Tips to monitor and improve the RAC environment. There are so many tool and which can improve your configuration of RAC. 1. ORAchk Tool: Proactively scans for the most impactful problems across the various layers of your stack. It perform check on OS Level, Cluster level, database level and Network Level and provide solution according to that. It comes with clusterware binaries but recommended to download the latest version form oracle support. Also refer ORAchk - Oracle Configuration Audit Tool (Doc ID 1268927.2) 2. OSWatcher and/or CHM Cluster health monitor :- These tools are used to monitor the operating and clusterware level process and record them in log according to your retention policy. Also refer OSWatcher (Includes: [Video]) (Doc ID 301137.1) 3. RAC AWR and ADDM reports:- RAC comes with few more AWR report functionality. awrgrpt:- It gives you the detail information of all nodes of the cluster in single report. awrgdrpt:- Its gives you the difference report b/w 2 awr reports to deal with performance issue. addmrpt:- Gives you the recommendation to improve the SQL query, database and cluster component.
  23. 23. Q/A
  24. 24. Thank You