The document discusses new features in Oracle High Availability 11gR2. It describes key features such as Grid Infrastructure, which combines Oracle Clusterware and Automatic Storage Management. It also covers out-of-place upgrades, redundant interconnect, easier addition and removal of nodes, Automatic Cluster File System, cluster-wide commands, Single Client Access Name, and Oracle RAC One Node.
The document discusses best practices for upgrading an Oracle database environment. It recommends upgrading in the following order:
1. Upgrade the clusterware and ASM
2. Install the new RDBMS software
3. Upgrade the databases to the new release
4. Perform post-upgrade steps
The document provides context on when an upgrade may be necessary or beneficial versus staying on an existing version. It also includes a compatibility matrix showing supported upgrade paths between different Oracle releases.
Since the manageability of RMAN backup, restore and recovery operations are nearly identical for nonclustered and clustered databases, the objective of this presentation is summarize you how RMAN can be best utilized in a RAC database.
Node management in Oracle Clusterware involves monitoring nodes and evicting nodes if necessary to prevent split-brain situations. The CSSD process monitors nodes through network heartbeats over the private interconnect and disk heartbeats using the voting disks. If a node fails to respond within the configured time limits for either heartbeat, it will be evicted from the cluster. Eviction involves sending a "kill request" to the node over the remaining communication channels to forcibly remove it. With Oracle Clusterware 11.2.0.2, reboots of nodes can be avoided by gracefully shutting down the Oracle Clusterware stack instead of an immediate reboot when fencing a node.
Real Application Cluster (RAC) allows multiple computers to simultaneously run Oracle RDBMS while accessing a single database, providing clustering. RAC provides high availability, scalability, and ease of administration by making multiple instances transparent to users. Nodes must have identical environments. Oracle Clusterware manages node additions and removals. Instances from different nodes write to the same physical database. The presentation covers RAC architecture, components, startup sequence, single instance configuration, node eviction, and tips for monitoring and improving the RAC environment.
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsMarkus Michalewicz
Oracle Open World (OOW) 2014 Presentation by Jim Williams (Oracle ASM Product Manager) on Oracle Flex ASM - What's New and Best Practices. The presentation provides an overview of enhancements (What's New) in Oracle ASM 12c, especially with respect to Oracle Flex ASM, and provides best practices which can be applied in any environment (Flex or Standard ASM). This presentation has also more background information for some of the configuration recommendations that I made in my "Oracle RAC (12.1.0.2) Operational Best Practices" presentation.
Understanding Oracle RAC 12c Internals as presented during Oracle Open World 2013 with Mark Scardina.
This is part two of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
This is an Oracle RAC 12c New Features Overview list used on the OOW13 Oracle RAC and QoS demo booth. It might be useful as a brief overview as to what is new in 12c RAC.
The document discusses new features in Oracle High Availability 11gR2. It describes key features such as Grid Infrastructure, which combines Oracle Clusterware and Automatic Storage Management. It also covers out-of-place upgrades, redundant interconnect, easier addition and removal of nodes, Automatic Cluster File System, cluster-wide commands, Single Client Access Name, and Oracle RAC One Node.
The document discusses best practices for upgrading an Oracle database environment. It recommends upgrading in the following order:
1. Upgrade the clusterware and ASM
2. Install the new RDBMS software
3. Upgrade the databases to the new release
4. Perform post-upgrade steps
The document provides context on when an upgrade may be necessary or beneficial versus staying on an existing version. It also includes a compatibility matrix showing supported upgrade paths between different Oracle releases.
Since the manageability of RMAN backup, restore and recovery operations are nearly identical for nonclustered and clustered databases, the objective of this presentation is summarize you how RMAN can be best utilized in a RAC database.
Node management in Oracle Clusterware involves monitoring nodes and evicting nodes if necessary to prevent split-brain situations. The CSSD process monitors nodes through network heartbeats over the private interconnect and disk heartbeats using the voting disks. If a node fails to respond within the configured time limits for either heartbeat, it will be evicted from the cluster. Eviction involves sending a "kill request" to the node over the remaining communication channels to forcibly remove it. With Oracle Clusterware 11.2.0.2, reboots of nodes can be avoided by gracefully shutting down the Oracle Clusterware stack instead of an immediate reboot when fencing a node.
Real Application Cluster (RAC) allows multiple computers to simultaneously run Oracle RDBMS while accessing a single database, providing clustering. RAC provides high availability, scalability, and ease of administration by making multiple instances transparent to users. Nodes must have identical environments. Oracle Clusterware manages node additions and removals. Instances from different nodes write to the same physical database. The presentation covers RAC architecture, components, startup sequence, single instance configuration, node eviction, and tips for monitoring and improving the RAC environment.
Oracle Flex ASM - What’s New and Best Practices by Jim WilliamsMarkus Michalewicz
Oracle Open World (OOW) 2014 Presentation by Jim Williams (Oracle ASM Product Manager) on Oracle Flex ASM - What's New and Best Practices. The presentation provides an overview of enhancements (What's New) in Oracle ASM 12c, especially with respect to Oracle Flex ASM, and provides best practices which can be applied in any environment (Flex or Standard ASM). This presentation has also more background information for some of the configuration recommendations that I made in my "Oracle RAC (12.1.0.2) Operational Best Practices" presentation.
Understanding Oracle RAC 12c Internals as presented during Oracle Open World 2013 with Mark Scardina.
This is part two of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
This is an Oracle RAC 12c New Features Overview list used on the OOW13 Oracle RAC and QoS demo booth. It might be useful as a brief overview as to what is new in 12c RAC.
This document provides an overview of setting up an Oracle 11gR2 Real Application Clusters (RAC) environment. It discusses system requirements, storage options like SAN and NAS, the Single Client Access Name (SCAN), and components like the Oracle Cluster Registry (OCR) and voting disk. It also explains Oracle Automatic Storage Management (ASM), extent distribution, and provides step-by-step instructions and references for installing Oracle 11gR2 Clusterware and database software on a RAC configuration.
The document discusses Oracle Real Application Clusters (RAC) architecture and internals. A typical RAC configuration includes multiple nodes connected to a public network, interconnect, and shared storage. Oracle Grid Infrastructure manages the clusterware and Automatic Storage Management. It provides high availability of databases and other applications by enabling them to run on multiple nodes and utilize the shared storage. The document covers various RAC components like VIPs, listeners, SCAN, client connectivity, node membership, and the interconnect.
This document provides instructions for installing and configuring a two-node Oracle RAC cluster using virtual machines on a single laptop or desktop computer. It details the necessary hardware requirements, including a minimum of 8GB RAM and 40GB free disk space. The software components needed are Windows 7 64-bit as the host operating system, along with VirtualBox for virtualization, Putty and VNC Viewer for remote access, and Oracle Grid Infrastructure and database software. The document provides a step-by-step guide to setting up the virtualized environment, installing Linux on the virtual machines, configuring the Grid Infrastructure and database software, and testing the overall cluster configuration.
Migrating from Single Instance to RAC Data guard Fuad Arshad
The document discusses migrating from a single instance Oracle database to an Oracle RAC database using Data Guard. Key steps included:
1) Configuring the new Oracle RAC environment with 3 nodes and validating the installation.
2) Installing Oracle Grid Infrastructure and database software, addressing known issues during installation.
3) Setting up the old single instance database as a physical standby using Data Guard to duplicate the database to the new RAC nodes.
4) Configuring the new RAC database and notifying the clusterware of the physical standby configuration, allowing migration with only 1 hour of downtime.
MIgrating from Single Instance to RAC via DataguardFuad Arshad
This document discusses Walgreen Company's migration from a single instance Oracle database to a RAC configuration and upgrading the database version using Oracle Data Guard. The key steps taken were:
1) Building new RAC servers and networking infrastructure and validating the environment using tools like RAC Check.
2) Setting up the primary database for Data Guard by enabling archiving and adding redo threads.
3) Configuring the standby RAC cluster by duplicating the primary database and registering instances with SRVCTL.
4) Validating that archive logs were shipping properly between primary and standby before upgrading the database using DBUA or manual steps.
Expert performance tuning tips for Oracle RACSolarWinds
In Oracle RAC 12c here have been significant enhancements to scalability and high availability, with features such as Flex Clusters, Flex ASM, Application Continuity and Transaction Guard, to name just a few. Learn how to make the most of these features, including:
*Operational support enhancements to SRVCTL
*CRSCTL commands
*ADR support for Grid Infrastructure
*Enterprise Manager
*Other support tools such Orachk and TFA analyzer
Automatic Storage Management allows Oracle databases to use disk storage that is managed as an integrated cluster file system. It provides functions like striping, mirroring, and rebalancing of data across storage disks. The document outlines new features in Oracle Exadata and Automatic Storage Management including Flex ASM, which eliminates the requirement for an ASM instance on every server, and Flex Disk Groups, which provide file groups and enable quota management and redundancy changes for databases. It also discusses enhancements to disk offline and online operations and rebalancing.
Oracle 12c RAC On your laptop Step by Step Implementation Guide 1.0Yury Velikanov
The document provides instructions for setting up a two-node Oracle 12c RAC environment within Oracle VirtualBox on a Windows laptop. The main steps include:
1. Configuring VirtualBox with a host-only network and installing Oracle Linux 6 on the first virtual machine.
2. Creating shared virtual disks for the ASM storage and installing Oracle Grid Infrastructure.
3. Cloning the first virtual machine to create the second node, and installing the Oracle 12c database software.
This allows users to test an Oracle 12c RAC sandbox environment locally without requiring additional physical hardware.
RAC-Installing your First Cluster and DatabaseNikhil Kumar
RAC - Installing your First RAC
Abstract : Oracle Real Application Clusters have been one of the hottest technologies in the market since 2001 prior this is know OPS in 8i. Oracle has brought revolution in the field of database by enhancing RAC technologies in it each version. This presentation will give introduction of RAC and features introduced in each version of RAC. This presentation contains the demo of building Oracle clusterware from the scratch. Also we will discuss the new components and its features during installation. This presentation and demo will be done on version 11GR2. Which will be used as a base for our next presentation Viz. Upgradation of RAC 11GR2 to 12C RAC.
This presentation will give brief insight information of RAC infrastructure setup. Sometimes DBA doesn’t fully aware of prerequisite and verification steps that needs to perform before installing clusterware, So this session will cover thing to consider before installing clusterware and best practices followed during the whole process.
Agenda
Introduction of RAC
Installation of Clusterware.
Creating diskgroup / Adding disk to Diskgroup using ASMCA.
Creation of ACFS Volume.
Installation of RAC Database using DBCA.
Oracle RAC 12c Practical Performance Management and Tuning as presented during Oracle Open World 2013 with Michael Zoll.
This is part three of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
This part concludes the main part of the "reindeer series" except for one bonus track "Oracle Multitenant meets Oracle RAC 12c" (available via SlidesShare, too).
Oracle RAC 12c Best Practices with Customer Example (Sanger) as presented during Oracle Open World 2013 (OOW13).
This is part one of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
The document discusses Walgreen Company's migration from a single instance Oracle database to a RAC configuration using Oracle Data Guard for zero downtime. It describes the analysis that showed the existing architecture could not handle growth. The strategy involved building RAC servers, setting up Data Guard replication between the primary and standby RAC clusters, and using DBUA for an in-place upgrade with no downtime. Things to watch out for included validating the environment, understanding XA changes in 11.2.0.2, and monitoring post-migration.
This document provides information about AST Corporation, an Oracle partner specialized in Oracle technologies. It discusses AST's Oracle certifications and awards. It also lists the Oracle products and services that AST provides support for, including their Oracle Platinum partnership. The remainder of the document describes setting up an Oracle RAC database in a virtualized environment using Oracle VM VirtualBox, including downloading software, installing operating systems on the virtual machines, configuring networking and storage, installing Oracle Grid Infrastructure and the database, and testing the RAC configuration.
Oracle Clusterware and Private Network Considerations - Practical Performance...Guenadi JILEVSKI
This document discusses Oracle Real Application Clusters (RAC) performance management. It covers RAC fundamentals and infrastructure, analyzing the impact of cache fusion, private interconnect considerations, common problems and symptoms, and diagnostics. The presentation addresses topics like global buffer cache, wait events, session and system statistics, IPC configuration, and network packet processing. It provides advice on tuning applications, the buffer cache, interconnect setup, and avoiding unnecessary parsing or locking to improve RAC performance.
Oracle RAC 12c has been touted as the best release so far and with reason. There have been significant enhancements to scalability and high availability with features such as Flex Clusters, Flex ASM, Application Continuity and Transaction Guard to name a few. While these cool features grab the headlines, there are others that are not highlighted but can make significant impact on DBA productivity.
In this session we will take a second look at some of these features, including operational support enhancements to srvctl, crsctl commands, ADR support for Grid Infrastructure and tools such orachk and tfa. We will also explore some of the new functionality introduced in 12.1.0.2.0.
Oracle RAC is a clustered version of the Oracle database that uses a shared disk architecture. It allows multiple instances of the database to run concurrently on multiple nodes, providing high availability and scalability. The document discusses how clients can connect to Oracle RAC using SCAN, which provides a single virtual IP address and listener for the entire cluster, making client connections easier to manage. It also covers how SCAN works with load balancing and provides failover between instances in the cluster.
Understanding oracle rac internals part 1 - slidesMohamed Farouk
This document discusses Oracle RAC internals and architecture. It provides an overview of the Oracle RAC architecture including software deployment, processes, and resources. It also covers topics like VIPs, networks, listeners, and SCAN in Oracle RAC. Key aspects summarized include the typical Oracle RAC software stack, local and cluster resources, how VIPs and networks are configured, and the role and dependencies of listeners.
This document provides information on upgrading Oracle Clusterware from version 11gR2 to 12cR1. It begins with an introduction to the presenter and their experience. The agenda then outlines discussing introduction to Clusterware, prerequisites for upgrade, differences between traditional and Flex clusters, the upgrade process, recovering from failures, downgrade process, and tips for monitoring the RAC environment.
The document discusses best practices for upgrading an Oracle database environment. It recommends upgrading in the following order:
1. Upgrade the clusterware and ASM
2. Install the new RDBMS software
3. Upgrade the databases to the new release
4. Perform post-upgrade steps
The document provides context on when an upgrade may be necessary or beneficial versus staying on an existing version. It also includes a compatibility matrix showing supported upgrade paths between different Oracle releases.
The document discusses Oracle 12c's new "multi-process multi-threaded" model. This new feature allows Oracle database processes on Linux/Unix systems to run as operating system threads rather than processes. This reduces resource consumption by eliminating redundant memory and CPU usage from separate processes. Background processes and local client connections now run as threads within larger processes. Remote clients still use dedicated processes that connect via a connection broker thread.
This document provides an overview of setting up an Oracle 11gR2 Real Application Clusters (RAC) environment. It discusses system requirements, storage options like SAN and NAS, the Single Client Access Name (SCAN), and components like the Oracle Cluster Registry (OCR) and voting disk. It also explains Oracle Automatic Storage Management (ASM), extent distribution, and provides step-by-step instructions and references for installing Oracle 11gR2 Clusterware and database software on a RAC configuration.
The document discusses Oracle Real Application Clusters (RAC) architecture and internals. A typical RAC configuration includes multiple nodes connected to a public network, interconnect, and shared storage. Oracle Grid Infrastructure manages the clusterware and Automatic Storage Management. It provides high availability of databases and other applications by enabling them to run on multiple nodes and utilize the shared storage. The document covers various RAC components like VIPs, listeners, SCAN, client connectivity, node membership, and the interconnect.
This document provides instructions for installing and configuring a two-node Oracle RAC cluster using virtual machines on a single laptop or desktop computer. It details the necessary hardware requirements, including a minimum of 8GB RAM and 40GB free disk space. The software components needed are Windows 7 64-bit as the host operating system, along with VirtualBox for virtualization, Putty and VNC Viewer for remote access, and Oracle Grid Infrastructure and database software. The document provides a step-by-step guide to setting up the virtualized environment, installing Linux on the virtual machines, configuring the Grid Infrastructure and database software, and testing the overall cluster configuration.
Migrating from Single Instance to RAC Data guard Fuad Arshad
The document discusses migrating from a single instance Oracle database to an Oracle RAC database using Data Guard. Key steps included:
1) Configuring the new Oracle RAC environment with 3 nodes and validating the installation.
2) Installing Oracle Grid Infrastructure and database software, addressing known issues during installation.
3) Setting up the old single instance database as a physical standby using Data Guard to duplicate the database to the new RAC nodes.
4) Configuring the new RAC database and notifying the clusterware of the physical standby configuration, allowing migration with only 1 hour of downtime.
MIgrating from Single Instance to RAC via DataguardFuad Arshad
This document discusses Walgreen Company's migration from a single instance Oracle database to a RAC configuration and upgrading the database version using Oracle Data Guard. The key steps taken were:
1) Building new RAC servers and networking infrastructure and validating the environment using tools like RAC Check.
2) Setting up the primary database for Data Guard by enabling archiving and adding redo threads.
3) Configuring the standby RAC cluster by duplicating the primary database and registering instances with SRVCTL.
4) Validating that archive logs were shipping properly between primary and standby before upgrading the database using DBUA or manual steps.
Expert performance tuning tips for Oracle RACSolarWinds
In Oracle RAC 12c here have been significant enhancements to scalability and high availability, with features such as Flex Clusters, Flex ASM, Application Continuity and Transaction Guard, to name just a few. Learn how to make the most of these features, including:
*Operational support enhancements to SRVCTL
*CRSCTL commands
*ADR support for Grid Infrastructure
*Enterprise Manager
*Other support tools such Orachk and TFA analyzer
Automatic Storage Management allows Oracle databases to use disk storage that is managed as an integrated cluster file system. It provides functions like striping, mirroring, and rebalancing of data across storage disks. The document outlines new features in Oracle Exadata and Automatic Storage Management including Flex ASM, which eliminates the requirement for an ASM instance on every server, and Flex Disk Groups, which provide file groups and enable quota management and redundancy changes for databases. It also discusses enhancements to disk offline and online operations and rebalancing.
Oracle 12c RAC On your laptop Step by Step Implementation Guide 1.0Yury Velikanov
The document provides instructions for setting up a two-node Oracle 12c RAC environment within Oracle VirtualBox on a Windows laptop. The main steps include:
1. Configuring VirtualBox with a host-only network and installing Oracle Linux 6 on the first virtual machine.
2. Creating shared virtual disks for the ASM storage and installing Oracle Grid Infrastructure.
3. Cloning the first virtual machine to create the second node, and installing the Oracle 12c database software.
This allows users to test an Oracle 12c RAC sandbox environment locally without requiring additional physical hardware.
RAC-Installing your First Cluster and DatabaseNikhil Kumar
RAC - Installing your First RAC
Abstract : Oracle Real Application Clusters have been one of the hottest technologies in the market since 2001 prior this is know OPS in 8i. Oracle has brought revolution in the field of database by enhancing RAC technologies in it each version. This presentation will give introduction of RAC and features introduced in each version of RAC. This presentation contains the demo of building Oracle clusterware from the scratch. Also we will discuss the new components and its features during installation. This presentation and demo will be done on version 11GR2. Which will be used as a base for our next presentation Viz. Upgradation of RAC 11GR2 to 12C RAC.
This presentation will give brief insight information of RAC infrastructure setup. Sometimes DBA doesn’t fully aware of prerequisite and verification steps that needs to perform before installing clusterware, So this session will cover thing to consider before installing clusterware and best practices followed during the whole process.
Agenda
Introduction of RAC
Installation of Clusterware.
Creating diskgroup / Adding disk to Diskgroup using ASMCA.
Creation of ACFS Volume.
Installation of RAC Database using DBCA.
Oracle RAC 12c Practical Performance Management and Tuning as presented during Oracle Open World 2013 with Michael Zoll.
This is part three of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
This part concludes the main part of the "reindeer series" except for one bonus track "Oracle Multitenant meets Oracle RAC 12c" (available via SlidesShare, too).
Oracle RAC 12c Best Practices with Customer Example (Sanger) as presented during Oracle Open World 2013 (OOW13).
This is part one of the Oracle RAC 12c "reindeer series" used for OOW13 Oracle RAC-related presentations.
The document discusses Walgreen Company's migration from a single instance Oracle database to a RAC configuration using Oracle Data Guard for zero downtime. It describes the analysis that showed the existing architecture could not handle growth. The strategy involved building RAC servers, setting up Data Guard replication between the primary and standby RAC clusters, and using DBUA for an in-place upgrade with no downtime. Things to watch out for included validating the environment, understanding XA changes in 11.2.0.2, and monitoring post-migration.
This document provides information about AST Corporation, an Oracle partner specialized in Oracle technologies. It discusses AST's Oracle certifications and awards. It also lists the Oracle products and services that AST provides support for, including their Oracle Platinum partnership. The remainder of the document describes setting up an Oracle RAC database in a virtualized environment using Oracle VM VirtualBox, including downloading software, installing operating systems on the virtual machines, configuring networking and storage, installing Oracle Grid Infrastructure and the database, and testing the RAC configuration.
Oracle Clusterware and Private Network Considerations - Practical Performance...Guenadi JILEVSKI
This document discusses Oracle Real Application Clusters (RAC) performance management. It covers RAC fundamentals and infrastructure, analyzing the impact of cache fusion, private interconnect considerations, common problems and symptoms, and diagnostics. The presentation addresses topics like global buffer cache, wait events, session and system statistics, IPC configuration, and network packet processing. It provides advice on tuning applications, the buffer cache, interconnect setup, and avoiding unnecessary parsing or locking to improve RAC performance.
Oracle RAC 12c has been touted as the best release so far and with reason. There have been significant enhancements to scalability and high availability with features such as Flex Clusters, Flex ASM, Application Continuity and Transaction Guard to name a few. While these cool features grab the headlines, there are others that are not highlighted but can make significant impact on DBA productivity.
In this session we will take a second look at some of these features, including operational support enhancements to srvctl, crsctl commands, ADR support for Grid Infrastructure and tools such orachk and tfa. We will also explore some of the new functionality introduced in 12.1.0.2.0.
Oracle RAC is a clustered version of the Oracle database that uses a shared disk architecture. It allows multiple instances of the database to run concurrently on multiple nodes, providing high availability and scalability. The document discusses how clients can connect to Oracle RAC using SCAN, which provides a single virtual IP address and listener for the entire cluster, making client connections easier to manage. It also covers how SCAN works with load balancing and provides failover between instances in the cluster.
Understanding oracle rac internals part 1 - slidesMohamed Farouk
This document discusses Oracle RAC internals and architecture. It provides an overview of the Oracle RAC architecture including software deployment, processes, and resources. It also covers topics like VIPs, networks, listeners, and SCAN in Oracle RAC. Key aspects summarized include the typical Oracle RAC software stack, local and cluster resources, how VIPs and networks are configured, and the role and dependencies of listeners.
This document provides information on upgrading Oracle Clusterware from version 11gR2 to 12cR1. It begins with an introduction to the presenter and their experience. The agenda then outlines discussing introduction to Clusterware, prerequisites for upgrade, differences between traditional and Flex clusters, the upgrade process, recovering from failures, downgrade process, and tips for monitoring the RAC environment.
The document discusses best practices for upgrading an Oracle database environment. It recommends upgrading in the following order:
1. Upgrade the clusterware and ASM
2. Install the new RDBMS software
3. Upgrade the databases to the new release
4. Perform post-upgrade steps
The document provides context on when an upgrade may be necessary or beneficial versus staying on an existing version. It also includes a compatibility matrix showing supported upgrade paths between different Oracle releases.
The document discusses Oracle 12c's new "multi-process multi-threaded" model. This new feature allows Oracle database processes on Linux/Unix systems to run as operating system threads rather than processes. This reduces resource consumption by eliminating redundant memory and CPU usage from separate processes. Background processes and local client connections now run as threads within larger processes. Remote clients still use dedicated processes that connect via a connection broker thread.
This document provides steps to configure replication between an Oracle database and a MySQL database using Oracle GoldenGate. It outlines installing GoldenGate on the source Oracle and target MySQL databases, setting up the necessary directories, creating and loading sample tables on the Oracle source, and starting the GoldenGate Manager process on both databases. The replication process will then capture changes on the Oracle source and replicate them to the MySQL target in real-time.
The document discusses NoSQL APIs in MySQL. It provides an overview of the memcached caching system and the history of the HandlerSocket protocol. It then describes the NoSQL interface introduced in MySQL 5.6, which allows for memcached-style operations on MySQL data. It notes that MySQL 5.7 further improved the performance and scalability of this interface.
This document summarizes the differences between Oracle RAC 11g and 12c. It provides an agenda that covers key areas such as multitenant architecture, Oracle 11g and 12c configurations, RAC 12c configurations, ASM 11g vs 12c, and improvements to DBCA in 12c. The document also references external links with more detailed information on topics like how to determine which hub node a leaf node uses, RAC networks, node eviction behavior, and IPv6 support.
This document discusses MySQL Fabric, which is a framework for managing a farm of MySQL servers to provide high availability and sharding capabilities. It describes how MySQL Fabric allows for easy management of MySQL servers, including load balancing, read/write splitting, distributed transactions, global updates, and sharding of tables. It also covers how application connectors can be made aware of MySQL Fabric to properly route queries and transactions to the backend MySQL servers.
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Nelson Calero
Each new version of the Oracle database includes improvements in the upgrade and patching utilities, forcing us to update our procedures to incorporate these changes.
The Fleet Provisioning & Patching (FPP, formerly RHP) utility, together with the change in its licensing announced at OOW 2019 that makes it free in RAC, now makes it possible to centrally manage the software life cycle.
This presentation shows examples of how to use FPP and different configuration options.
This document provides instructions for replicating data from an Oracle multitenant container database (CDB) to another CDB using Oracle GoldenGate. It outlines prerequisites, tasks to prepare the databases and environment, and steps for initial load and ongoing replication of data changes in near real-time. Key steps include creating GoldenGate users, adding supplemental logging, configuring Extract and Replicat processes, and monitoring replication status. The goal is to familiarize the reader with setting up a basic Oracle to Oracle replication setup using GoldenGate in a multitenant environment.
RMOUG2016 - Resource Management (the critical piece of the consolidation puzzle)Kristofferson A
This document discusses resource management in Oracle databases. It begins with an introduction of the speaker and his company, Accenture Enkitec Group. It then covers various aspects of resource management including the consolidation and resource management lifecycle, new features in Oracle 12c such as instance caging and threaded execution, barriers to adopting resource management, and a systematic approach to implementing resource management. Real-world scenarios are also discussed.
This document provides an overview of a presentation on tuning Oracle GoldenGate for performance. It includes details on the speaker and their company. The presentation covers GoldenGate concepts, tools for monitoring performance, a case study, and recommendations. Performance issues were identified in the case study including replication lag of over 1 hour. The recommendations provide tuning tips for Integrated Extract and Replicat parameters.
UKOUG Tech15 - Deploying Oracle 12c Cloud Control in Maximum Availability Arc...Zahid Anwar (OCM)
Common Cloud Control deployments can sometimes be exposed to single points of failure. In this presentation we will be discussing these pitfalls and how, through deploying Cloud Control within the Maximum Availability Architecture can provide a robust system. Aimed at a technical audience - we will dive into giving High Availability and Disaster Recovery for the OMS repository and OMS Web Tier through the use of RAC, Web Tier Clustering, Data Guard and Storage Replication. We will take our audience through the simple but effective steps required for this type of deployment in addition to the license implications of using Maximum Availability Architecture including what Oracle give you for free under a restricted-use license. This presentation is based on a recent project completed by our speaker Zahid Anwar. This project saw Zahid provide Maximum Availability Architecture for Cloud Control which was monitoring 6, critical X4-2 Eighth Exadata Machines.
Database As A Service: OEM + ODA (OOW 15 Presentation)Bobby Curtis
Bobby Curtis presented on building a cloud platform using Oracle Enterprise Manager 12c and Oracle Database Appliance. The presentation covered client requirements for provisioning databases quickly and reducing costs, an overview of Oracle Enterprise Manager 12c and its cloud architecture, configuring zones and resource pools, creating service templates, executing provisioning requests using REST APIs, and integrating with application code using cURL calls. The agenda also included putting the pieces together and a question/answer session.
Cloug Troubleshooting Oracle 11g Rac 101 Tips And TricksScott Jenner
The document provides an overview of troubleshooting techniques for Oracle 11g Real Application Clusters (RAC). It discusses proactive checks that can be performed to monitor the health of an 11g RAC environment, including verifying the status of RAC processes, the clusterware, and the automatic storage management (ASM). It also covers common 11g RAC problems such as offline clusterware resources, failed vote disks or OCR disks, and node reboot issues. Techniques for root cause analysis of problems are presented, including examining RAC log files.
MySQL InnoDB Cluster and Group Replication in a NutshellFrederic Descamps
This document outlines the agenda and steps for a hands-on tutorial on MySQL InnoDB Cluster and Group Replication. The agenda includes preparing the workstation by setting up virtual machines, an overview of MySQL InnoDB Cluster and Group Replication, migrating from a master-slave topology to Group Replication, monitoring Group Replication, and application interaction with Group Replication. The first lab demonstrates the current master-slave setup. The migration plan involves installing MySQL InnoDB Cluster on a new server, restoring a backup, setting up asynchronous replication on the new server, adding it to the Group Replication group, pointing the application to a new node, and stopping asynchronous replication after catch up.
My MySQL and NoSQL presentation from the NoSQL Search event in Copenhagen: http://nosqlroadshow.com/nosql-cph-2013/speaker/Ted+Wennmark
MySQL offers solutions to implement NoSQL concepts like auto-sharding, key-value access or asynchronous operations. This adds all known solutions from the SQL world to the NoSQL space.
The combined approach of SQL and NoSQL gives developers the choice to select whatever features from both worlds they need.
In this talk we take a deeper look at key-value access to MySQL and MySQL Cluster, auto-sharding and scalability of MySQL Cluster, mapping of schemaless key value access to a relational data model and the performance of NoSQL access to MySQL.
What we unlearned_and_learned_by_moving_from_m9000_to_ssc_ukoug2014Philippe Fierens
The document discusses moving databases from 3 Oracle M9000 servers to a new Oracle SPARC SuperCluster (SSC) system. It describes the key phases of the project including lifting and shifting the databases from the M9000s to application domains on the SSC, making use of the SSC's integrated storage cells, upgrading databases from Oracle 9i and 10g to 11g, and consolidating databases. It also covers issues encountered such as performance problems after the initial migration and regressions encountered during patching cycles. The document provides details on configuring features on the SSC like RAC One Node, Data Guard, and database resource management.
Mastering Aurora PostgreSQL Clusters for Disaster RecoveryMydbops
The presentation "Mastering Aurora PostgreSQL Clusters for Disaster Recovery" by Bhuvanesh, Co-Founder & CTO of ShellKode, at the Mydbops OpenSource Database Meetup 14 covers advanced topics in managing Aurora PostgreSQL clusters for disaster recovery purposes.
Bhuvanesh discusses key features of Aurora, such as its decoupled storage and compute layers, auto scaling capabilities, and native replication, highlighting its benefits over traditional RDS instances. He also explores Aurora Global Databases, explaining how they enable replication of data across regions for geo-span applications with low latency.
The presentation includes architecture details, such as physical and log replication, and managed failover options for ensuring high availability. Bhuvanesh shares real-world experiences and best practices for managing Aurora clusters, including handling replication lag and TLS certificate management.
Some design patterns and concepts for industrial grade deployment of Drupal on Solaris, plus a specific example of an interesting Drupal site deployed on Solaris
Similar to Managing troubleshooting cluster_360dgrees (20)
Executive Directors Chat Leveraging AI for Diversity, Equity, and InclusionTechSoup
Let’s explore the intersection of technology and equity in the final session of our DEI series. Discover how AI tools, like ChatGPT, can be used to support and enhance your nonprofit's DEI initiatives. Participants will gain insights into practical AI applications and get tips for leveraging technology to advance their DEI goals.
How to Make a Field Mandatory in Odoo 17Celine George
In Odoo, making a field required can be done through both Python code and XML views. When you set the required attribute to True in Python code, it makes the field required across all views where it's used. Conversely, when you set the required attribute in XML views, it makes the field required only in the context of that particular view.
How to Fix the Import Error in the Odoo 17Celine George
An import error occurs when a program fails to import a module or library, disrupting its execution. In languages like Python, this issue arises when the specified module cannot be found or accessed, hindering the program's functionality. Resolving import errors is crucial for maintaining smooth software operation and uninterrupted development processes.
This slide is special for master students (MIBS & MIFB) in UUM. Also useful for readers who are interested in the topic of contemporary Islamic banking.
Main Java[All of the Base Concepts}.docxadhitya5119
This is part 1 of my Java Learning Journey. This Contains Custom methods, classes, constructors, packages, multithreading , try- catch block, finally block and more.
How to Manage Your Lost Opportunities in Odoo 17 CRMCeline George
Odoo 17 CRM allows us to track why we lose sales opportunities with "Lost Reasons." This helps analyze our sales process and identify areas for improvement. Here's how to configure lost reasons in Odoo 17 CRM
বাংলাদেশের অর্থনৈতিক সমীক্ষা ২০২৪ [Bangladesh Economic Review 2024 Bangla.pdf] কম্পিউটার , ট্যাব ও স্মার্ট ফোন ভার্সন সহ সম্পূর্ণ বাংলা ই-বুক বা pdf বই " সুচিপত্র ...বুকমার্ক মেনু 🔖 ও হাইপার লিংক মেনু 📝👆 যুক্ত ..
আমাদের সবার জন্য খুব খুব গুরুত্বপূর্ণ একটি বই ..বিসিএস, ব্যাংক, ইউনিভার্সিটি ভর্তি ও যে কোন প্রতিযোগিতা মূলক পরীক্ষার জন্য এর খুব ইম্পরট্যান্ট একটি বিষয় ...তাছাড়া বাংলাদেশের সাম্প্রতিক যে কোন ডাটা বা তথ্য এই বইতে পাবেন ...
তাই একজন নাগরিক হিসাবে এই তথ্য গুলো আপনার জানা প্রয়োজন ...।
বিসিএস ও ব্যাংক এর লিখিত পরীক্ষা ...+এছাড়া মাধ্যমিক ও উচ্চমাধ্যমিকের স্টুডেন্টদের জন্য অনেক কাজে আসবে ...
This presentation includes basic of PCOS their pathology and treatment and also Ayurveda correlation of PCOS and Ayurvedic line of treatment mentioned in classics.
The simplified electron and muon model, Oscillating Spacetime: The Foundation...RitikBhardwaj56
Discover the Simplified Electron and Muon Model: A New Wave-Based Approach to Understanding Particles delves into a groundbreaking theory that presents electrons and muons as rotating soliton waves within oscillating spacetime. Geared towards students, researchers, and science buffs, this book breaks down complex ideas into simple explanations. It covers topics such as electron waves, temporal dynamics, and the implications of this model on particle physics. With clear illustrations and easy-to-follow explanations, readers will gain a new outlook on the universe's fundamental nature.
it describes the bony anatomy including the femoral head , acetabulum, labrum . also discusses the capsule , ligaments . muscle that act on the hip joint and the range of motion are outlined. factors affecting hip joint stability and weight transmission through the joint are summarized.
2. Disclaimer
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 2
This views/content in this slides are those of the author and
do not necessarily reflect that of Oracle Corporation and/or its
affiliates/subsidiaries. The material in this document is for
informational purposes only and is published with no
guarantee or warranty, express or implied..
This material should not be reproduced or used without the
authors' written permission.
3. About me
Syed Jaffer Hussain
Database Support Manager
Over 20 years IT hands-on experience
14+ years as an Oracle DBA
Technologist of the year, DBA 2011.
Oracle ACE Director
Oracle 10g Certified Master(OCM)
Oracle 10g RAC Certified Expert
OCP v8i,9i,10g & 11g
ITIL v3 Foundation Certified
Co-Authored - Oracle 11g R1/R2 Real Application Clusters Essentials
- Expert Oracle RAC (in-progress)
Twitter: @sjaffarhussain
http://jaffardba.blogspot.com
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 3
4. Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 4
A famous personality in the Oracle community once
compared an Oracle DBA with a Pilot!!!
Foreword
5. Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 5
Any deployment (installation/upgrade) is a single shot
Foreword
6. Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 6
Where as
Foreword
7. Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 7
Administration and Troubleshooting is never ending
Foreword
8. What is covered
o What's new in 11gR2 Clusterware – Key new features at a glance
o Oracle 11gR2 Clusterware software stack
o Clusterware start-up sequence
o Cluster logs & directory tree structure
o Analyzing Cluster logs
o Cluster logs rotation/retention policy
o Troubleshooting Cluster start-up failures
o Debugging/Tracing Clusterware components
o Tools & Utilities – how to pick the right one
o References
o Q & A
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 8
9. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 9
Oracle Grid Infrastructure
• Clusterware and ASM binaries are installed together in a single home
directory: Grid Home (GI)
10. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 10
• OCR can also be stored in ASM diskgroup
• Upto five (05) copies of OCR files
11. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 11
Oracle Local Registry – OLR
• Independent OLR copy for each node
• Not shared between nodes
• Stores local node configuration details required by OHASD
• Configured upon installation/upgrade
• Facilitates the CRS startup process when OCR/VD stored in ASM
• ocrcheck -local
• Located under $GRID_HOME/cdata/hostname/hostname.olr
• $GRID_HOME/bin/ocrconfig –local –manualbakup/restore
12. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 12
• Voting Disk (files) can also be stored in ASM diskgroup
• VD copies can’t resides in multiple ASM diskgroups
13. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 13
• crsctl start cluster –all -- starts cluster on all nodes
• crsctl stop cluster –all -- stops cluster on all nodes
• crsctl check cluster –all -- verify cluster health on all
nodes
Clusterized cluster-aware commands
14. Key new features at a glance
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 14
Complete redesign of Cluster daemon in 11gR2, ohasd introduction
Replaces RACG layer with Agents
New Services
Grid Plug and Play (GPnP)
Cluster Time Synchronization Service (CTSS)
Grid Name Service
Cluster can be started in exclusive mode for maintenance purpose
./crsctl start res –t –init ---- list all Clusterware daemon resources
27. Cluster logs & directory structure
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 27
• Writes Clusterware stack all important alerts messages
• Posts Cluster stack start/stop messages
• nodes eviction messages
• OLR events
• Voting and OCR disk related messages
• Active nodes list
• Preferably the first log file to review upon any cluster issues
GRID_HOME/log/host_name/alterrac1.log
28. Cluster logs & directory structure
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 28
GRID_HOME/log/host_name/alterrac1.log
[ohasd(10937)]CRS-1301:Oracle High Availability Service started on node rac1.
[cssd(19712)]CRS-1713:CSSD daemon is started in exclusive mode
[ohasd(19506)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server ‘rac1’
[cssd(19951)]CRS-1605:CSSD voting file is online: /dev/rdsk/c0t13d2; details in
/u00/app/11.2.0/grid_1/log/rac1/cssd/ocssd.log.
2013-04-23 16:42:28.906: [ CSSD][6]clssnmvFindInitialConfigs: No voting files found
2013-04-23 16:42:28.906: [ CSSD][6](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not
found. Retrying discovery in 15 seconds
[cssd(7945)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details
at (:CSSNM00070:) in /u00/app/11
.2.0/grid/log/usdbt01/cssd/ocssd.log
[/u00/app/11.2.0/grid_1/bin/oraagent.bin(19914)]CRS-5815:Agent
'/u00/app/11.2.0/grid_1/bin/oraagent_oracle' could not find any base
type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:9:2} in
/u00/app/11.2.0/grid_1/log/usdbp01/agent/ohasd/oraagent_oracle/oraagent_oracle.log.
[cssd(19951)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1,rac2.
[cssd(3726)]CRS-1625:Node rac2, number 2, was manually shut down
[cssd(3726)]CRS-1612:Network communication with node rac2 (2) missing for 50% of timeout interval.
Removal of this node from cluster in 14.145 seconds
29. Cluster logs & directory structure
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 29
• Cluster Ready Services daemon (CRSD) maintains this log
• Any cluster resources Start/stop/failure occurrence are written
• Review the log when you have resources failures, unable to start etc
• 2013-04-05 02:05:05.379: [ CRSPE][46] {13:24208:51894} Resource ora.prddb.db has been updated in
the PE data model:6000000004f82610
• 2013-04-05 02:08:15.371: [ CRSPE][46] {0:33:627} State change received from rac1 for ora.prddb.db
• 2013-04-05 02:08:22.698: [ CRSPE][46] {0:33:628} CRS-2672: Attempting to start
'ora.prddb.prddb_srv.svc' on ‘rac1‘
• 2013-04-23 17:32:46.340: [ OCRRAW][1]proprioini: all disks are not OCR/OLR formatted
• 2013-04-23 17:32:46.340: [ OCRRAW][1]proprinit: Could not open raw device
GRID_HOME/log/host_name/crsd.log
30. Cluster logs & directory structure
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 30
• Cluster Synchronization daemon (CSSD) maintains this log
• Busiest log file
• Records node inter-communication messages
• Heart beat missing and node eviction messages
2013-04-23 16:14:14.712: [ CSSD][6]clssnmvDiskVerify: Successful discovery of 0 disks
2013-04-23 16:14:14.712: [ CSSD][6]clssnmCompleteInitVFDiscovery: Completing initial voting file
discovery
2013-04-23 16:14:14.712: [ CSSD][6]clssnmvFindInitialConfigs: No voting files found
2013-04-23 16:14:14.713: [ CSSD][6](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not
found. Retrying discovery in 15 seconds
• 2013-04-17 11:33:23.286: [GIPCHALO][7] gipchaLowerProcessNode: bootstrap node considered dead
because of idle connection time 600014 ms, node 60000000019645b0 { host ‘rac2', haName 'CSS_crs',
srcLuid 8c78ad11-53582d91, dstLuid 2f6e4604-5e451051 numInf 1, contigSeq 6712447, lastAck 6697196,
lastValidAck 6712447, sendSeq [6697202 : 6697202], createTime 2417887052, sentRegister 1,
localMonitor 1, flags 0x808 }
• 2013-04-20 15:13:18.953: [ CSSD][54]clssnmSendingThread: sending status msg to all nodes
• 2013-04-20 15:13:18.954: [ CSSD][54]clssnmSendingThread: sent 4 status msgs to all nodes
• 2013-04-12 17:35:55.351: [ CSSD][49]clssnmvReadDskHeartbeat: Reading DHBs to get the latest info
for node rac1, 17 LATSvalid 0 uniqueness 1348227938
• [cssd(7335)]CRS-1612:Network communication with node rac2 (02) missing for 50% of timeout interval. Removal of this node from cluster in 14.397
seconds2013-03-15 17:02:44.964
[cssd(7335)]CRS-1611:Network communication with node rac2 (02) missing for 75% of timeout interval. Removal of this node from cluster in 7.317
seconds2013-03-15 17:02:50.024
[cssd(7335)]CRS-1610:Network communication with node rac2 (02) missing for 90% of timeout interval. Removal of this node from cluster in
GRID_HOME/log/host_name/ocssd.log
31. Cluster logs & directory structure
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 31
• Oracle High Availability Service (OHASD) maintains this log
• High Availability service messages are written
• Review the log when you have issues whist running root.sh/rootupgrd.sh
• If the service unable to start or becomes unhealthy due to OLR problems
• Loads default debugging levels
2013-04-17 11:32:47.096: [ default][1] OHASD Daemon Starting. Command string :reboot
2013-04-17 11:32:47.125: [ default][1] Initializing OLR
2013-04-17 11:32:47.255: [ OCRRAW][1]proprioo: for disk 0
(/u00/app/12.1.0/grid_1/cdata/rac2.olr), id match (1), total id sets,
need recover (0), my votes (0), total votes (0), commit_lsn (3118), lsn (3118)
2013-04-17 11:32:47.368: [ default][1] Loading debug levels...
2013-04-17 11:32:47.803: [ clsdmt][13]Creating PID [6401] file for home /u00/app/12.1.0/grid_1
host usdbp10 bin ohasd to /u00/app/12.1.0/grid_1/ohasd/init/
GRID_HOME/log/host_name/ohasd.log
33. Cluster logs rotation/retention policy
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 33
Operating System logs
HPUX - /var/adm/syslog/syslog.log
AIX - /bin/errpt –a
Linux - /var/log/messages
Windows - Refer .TXT log files under Application/System log using Windows Event Viewer
Solaris - /var/adm/messages
34. Cluster logs rotation/retention policy
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 34
Managing Clusterware log files manually is not recommended...
35. Cluster logs rotation/retention policy
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 35
They are governed and managed automatically ……
36. Cluster logs rotation/retention policy
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 36
• Most Clusterware log files follow the 10x10 rule as part of automatic
rotation/retention policy and governed automatically.
• 10 copies of cssd.log files with 50M retained and rotated subsequently.
• ohasd, evmd, crsd etc logs also retain 10 copies with 10M size.
• The policy doesn’t applies to the alterHOSTNAME.log file.
37. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 37
$GRID_HOME/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
38. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 38
$GRID_HOME/bin/crsctl check crs
$GRID_HOME/bin/crsctl check cluster
CRS-4639: Could not contact Oracle High Availability Services
CRS-4124: Oracle High Availability Services startup failed
CRS-4000: Command Check failed, or completed with errors
OR
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
39. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 39
Oracle High Availability Service (ohasd) start-up failures – common causes
CRS-4639: Could not contact Oracle High Availability Services
CRS-4124: Oracle High Availability Services startup failed
CRS-4000: Command Check failed, or completed with errors
1. Verify whether the cluster auto start-up is configured or not?
• crsctl config has
• /var/opt/oracle/scls_scr/hostname/root or
/etc/oracle/scls_scr/hostname/root
• Verify OS run level
• Check whether ohasd daemon process is up or not: ps –ef |grep ohasd
2. Verify ohasd auto-start pointer in the /etc/init and /etc/inittab files ?
• h1:3:respawn:/sbin/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
3. Verify the OLR availability, corruption and accessibility on the local node
• Review the ohasd.log file for more details
4. Verify whether the ohasd agents are up or not – for unhealthy cluster
• ps –ef |grep oraagent|orarootagent|cssdagent|cssdmonitor
• Review the ohasd.log file
5. Verify Grid Infrastructure location permission
• Compare with a good node location
40. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 40
Oracle High Availability Service (ohasd) start-up failures – troubleshooting common
causes
1. Enable Cluster auto start-up
• crsctl enable has|crs
• crsctl start crs/cluster
2. Put the following line in the respective OS files
• h1:3:respawn:/sbin/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
3. Restore or set permissions on the OLR
• Restore from the recent OLR backup, if the file is corrupted
• Reset appropriate permission on the local node for the file
4. Verify whether the ohasd agents are up or not – for unhealthy cluster status
• Set permission if needed
• If binaries are corrupted, restore them from a latest backup
5. Reset permissions or restore from the recent backups
6. Additionally, remove/rename the files from the /var/tmp/.oracle,
/usr/tmp/.oracle or /tmp/.oracle locations
41. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 41
Cluster Synchronization Service (cssd) start-up failures – common causes
CRS-4530: Communications failure contacting Cluster Synchronization Services
daemon:
1. Verify the following:
• GPnP profile accessibility
• Voting disk files accessibility
• Check the underlying network (private network) for any connectivity
issues
2. Verify whether the daemon status on the OS
• ps –ef |grep ocssd.bin
42. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 42
Cluster Synchronization Service (cssd) start-up failures – troubleshooting common
causes
1. Review the ocssd.log file to diagnose the issue:
• Review the ocssd.log file if the daemon is able to access the GPnP
profile
• Run, crsctl query css votedisk to verify whether the voting disk files
are accessible
• If Voting disk permissions are lost, reset them
• Resolve underlying network issues for any heart-beat issues and bring up
the interconnect resource:
./crsctl start res ora.cluster_interconnect.haip –init
2. Start the process manually
• Try to start the daemon process manually it is not up or unhealthy:
./crsctl start res ora.cssd –init
43. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 43
Cluster Ready Service (crsd) start-up failures – common causes
CRS-4535: Cannot communicate with Cluster Ready Services:
1. Verify the following:
• Oracle Cluster Registry (OCR) accessibility
./ocrcheck
• Look for any Grid Home ownership and permission changes
• Check for the OCR mirror copy issues
• Verify and validate underlying network (private network)
2. Verify whether the daemon status on the OS
• ps –ef |grep crsd.bin
• crsctl stat res –t –init, look for ora.crsd status
3. Verify crsd agents
• ps –ef |grep oraagent|orarootagent
44. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 44
Cluster Ready Service (crsd) start-up failures – troubleshooting common causes
1. Take the following action:
• Review the crsd.log file
• Take appropriate steps to resolve ownership, privilege issues on the OCR
files.
• Compare with the a good node, and restore the directory
• ./ocrcheck
• Verify and validate underlying network (private network)
2. Verify whether the daemon status on the OS
• Restart the process manually
./crsctl start res ora.crsd -init
3. Verify the following:
• ./crs_stat –t
4. Ensure sufficient free space available under the $GRID_HOME to avoid cluster
unhealthy issues.
46. Troubleshooting Cluster start-up failures
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 46
Troubleshooting other clusterware process
./crsctl start res ora.cluster_interconnect.haip –init
./crsctl start res ora.cssd –init
The following output will be displayed at your screen:
CRS-2679: Attempting to clean 'ora.cssdmonitor' on 'rac1'
CRS-2681: Clean of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
47. Debugging/Tracing Cluster components
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 47
• Flexibility to modify the default tracing/logging levels for any Clusterware main and sub-
processes
• Range from 1 – 5, 0 value disables the tracing level
•ohasd.log file also writes the default trace levels message when Oracle High Availability
service daemon starts up on the local node
• crsctl get log {css|crs|evm} ALL – lists existing trace levels for the modules
• crsctl lsmodules – list the module detals
• crsctl lsmodules {css|crs|evm}
48. Debugging/Tracing Cluster components
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 48
Default trace levels:
./crsctl get log css all
Get CSSD Module: BCCM Log Level: 2
Get CSSD Module: CLSF Log Level: 0
Get CSSD Module: CLSINET Log Level: 0
Get CSSD Module: CSSD Log Level: 2
Get CSSD Module: GIPCBCCM Log Level: 2
Get CSSD Module: GIPCCM Log Level: 2
Get CSSD Module: GIPCGM Log Level: 2
Get CSSD Module: GIPCNM Log Level: 2
Get CSSD Module: GPNP Log Level: 1
Get CSSD Module: OLR Log Level: 0
Get CSSD Module: SKGFD Log Level: 0
Default moduels:
./crsctl lsmodules
Usage:
crsctl lsmodules
{mdns|gpnp|css|crf|crs|ctss|evm|gipc}
where
mdns multicast Domain Name Server
gpnp Grid Plug-n-Play Service
css Cluster Synchronization
Services
crf Cluster Health Monitor
crs Cluster Ready Services
ctss Cluster Time Synchronization
Service
evm EventManager
gipc Grid Interprocess
Communications
49. Debugging/Tracing Cluster components
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 49
The following enable various tracing levels:
./crsctl set log crs crsmain=3
./crsctl set log crs crsmain=3,crsevt=4
./crsctl set log crs all=5
./crsctl set log res ora.prddb.db:5
The following examples explains how to set tracing levels on the OS:
export ORA_CRSDEBUG_ALL=1 --sets debugging level 1 to all modules
export ORA_CRSDDEBUG_CRS=2 --sets debugging level 2 to CRS module
50. Debugging/Tracing Cluster components
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 50
The following disable tracing:
./crsctl set log crs crsmain=0
./ crsctl set log res ora.prddb.db:0
./ crsctl set log res ora.crs:0 -init
The following examples explains how to set tracing levels on the OS:
export ORA_CRSDEBUG_ALL=1 --sets debugging level 1 to all modules
export ORA_CRSDDEBUG_CRS=2 --sets debugging level 2 to CRS module
52. Tools & Utilities - how to pick the right one
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 52
53. Tools & Utilities - Diagnostic Collection Script
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 53
Diagcollection.pl:
• Located under $GRID_HOME/bin location
• Is a tool that gathers required Clusterware diagnostic information in a bunch of trace
files from various resources: CRS logs, trace & core files, OCR data etc.
• Can collect diagnostic infromation at different layers and homes: Cluster, Oracle RDBMS,
Core , Oracle Base etc
• All the information will be then zipped into a few zip files
• Duration required to gather the information is directly propositional to the levels used
• Upload these files to My Oracle Support for issue investigation
55. Tools & Utilities - Cluster Health Monitor (CHM)
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 55
• Is a tool designed, developed to detect and analyze OS, Cluster resources failures etc.
• Formerly known as Instantaneous Problem Detector for OS(IPD/OS) .
• Pre 11gR2 versions, you need to download the tool from OTN.
• With 11gR2, it is the integral part of the software and integrated closely with GI.
• ora.crf CHM resource introduced | crsctl stat res –t -init
• Not available on some platforms.
• Can be used on RAC and non-RAC environments.
• Collects OS real-time (every second, 5 sec from 11203) statistics : memory, swap, I/O,
net work etc
• Stores real-time monitoring metrics in the CHM repository.
• Historical data can be used to diagnose: node eviction, instance hang, server perf. etc
• Contains two services:
System Monitoring Service (osysmond)
• runs on every node, monitor and collect OS metrics and send data to
OloggeredCluster Loger Service (ologgered)
•Stores the information received from the nodes in the respository
•Runs in one node as master service and standby service on other nodes
• CHM vs OSWatcher:
CHM takes less CPU, and less overhead on the node, OSWatcher doesn’t run
When the server CPU is heavily used
56. Tools & Utilities - Cluster Health Monitor (CHM)
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 56
• Consumes less than %5 CPU/core, minimal overhead on the server
• Takes 1GB space by default across all nodes.
• Apprx. 0.5GB data per day.
• Data can be kept for 3 days.
• ./oclumon manage –get repsize
• oclumon – a command-line tool, used to manage CHM repository
• Stores in a management repository database with 12c.
57. Tools & Utilities - OSWatcher Block Box (osbb)
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 57
• Is a tool captures OS performance metrics and stores in the statistical data in a file
• vmstat, netstat, top, tracerouts, ps, iostat etc
• Available on MOS.
• On RAC, need to configure, schedule on individual nodes
• Supports most UNIX/LINUX platforms.
• ./startoswbb.sh (default interval/retention, 30 sec/48 hrs)
• ./startoswbb.sh 60 10 (60 seconds interval, 10 hrs data retention)
• ./stoposwbb.sh
• Review the dat file in the /archive directory
58. Tools & Utilities - oratop
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 58
• Is a OS top-like utility on Linux Platforms
• Provides nearly real-time database monitoring capabilities for a RAC and Non-RAC dbs
for 11.2.0.3 or higher.
• A very light weight utility, consumes 0.20% memory and <1% CPU.
• Download the oratop.zip from MOS, set chmod 755.
• Db init parameters: statistics_level = TYPICAL, timed_statistics = TRUE must be set
• Need to input username/password, connects as system user when no credentials
provides.
• Set the following on the OS
$ ORACLE_UNQNAME=<dbname>
$ ORACLE_SID=<instance_name1>
$ ORACLE_HOME=<db_home>
$ export LD_LIBRARY_PATH=$ORACLE_HOME/lib
$ export PATH=$ORACLE_HOME/bin:$PATH
• Download the oratop.zip from MOS, set chmod 755
• Needs a TNS to the source database
• Need privileges on, v_$SESSION, v_$SYSMETRIC, v_$INSTANCE, v_$PROCESS,
V_$SYSTEM_EVENT etc
59. Tools & Utilities - oratop
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 59
• Live window : lists 5 TOP wait events, top Oracle sessions of i/o, memory, db load,
• Provides database blocking details.
• Press q/Q or Control+C to abort
./oratop -i 5 / as sysdba - interval every 5 seconds
./oratop -i 5 username/password@tns_alias
Databases
Top 5 DB events
Processes
Header
• % db - (values > 99%)
• %CU - (load > 2 x cpu counts & host cpu > 99)
• HLD - (load > 2 * cpu counts and aas > cpu
counts)
• IORL - (value > 20ms)
• %FR - (value < 1%)
• ASW - (value = session counts, USN)
• AAS - (value > cpu counts)
• DBW - (value > 50%)
• EVENT – Active wait event
• PGA – potential unusual memory growth
• BLOCKER - a blocking session with (wait time >
5 minutes)
60. Tools & Utilities – RACcheck v.2.2.1
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 60
• A RAC configuration auditing utility that audits various important configuration settings:
Cluster, ASM, Grid Infrastructure etc
• Audits, OS Kernal parameters/Packages, 11.2.0.3 upgrade readiness etc
• Download the raccheck.zip from MOS, chmod to 755.
• Ability to compare between two outputs.
• All the recommendations/output written to a HTML file.
• The output include overall health check rating – out of 100, bug fixes, patch
recommendations etc.
• Upload the .zip file if MOS ask to do so.
61. Tools & Utilities – RACcheck v.2.2.1
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 61
Examples:
./raccheck – follow the interactive steps
./raccheck –u –o pre|post
./raccheck –h
./raccheck –s
./raccheck -diff report1 report2
Usage : ./raccheck [-abvhpfmsuSo:c:rt:]
-a All (Perform best practice check and recommended patch check)
-b Best Practice check only. No recommended patch check
-h Show usage
-v Show version
-p Patch check only
-m exclude checks for Maximum Availability Architecture
-u Run raccheck to check pre-upgrade or post-upgrade best
practices.-o pre or -o post is mandatory with -u option like ./raccheck -u -o pre
-f Run Offline.Checks will be performed on data already
-o Argument to an option. if -o is followed by
v,V,Verbose,VERBOSE or Verbose, it will print checks which
passs on the screen
if -o option is not specified,it will print only failures on
screen. for eg: raccheck -a -o v -r To include High availability best practices also in regular
healthcheck eg ./racchekck -r(not applicable for exachk) -c Pass specific module or component to check
best practice
for. By default it will check for components indentified fr
62. Tools & Utilities – Hang analysis/system state
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 62
• HANGANALYZE helps detecting the cause of database hang
• Advised to run the HANGANLYZE when a database suffers from hang, performance
degradation, latching issues etc
• Available since 8.1.6, provides cluster-wide analysis from 9i
Examples:
SQL> sqlplus " / as sysdba"
SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug setinst all
SQL> oradebug –g def hanganalyze 3
-- wait 90 seconds
SQL> oradebug –g def hanganalyze 3
SQL> oradebug tracefile_name
SQL> exit
63. Tools & Utilities – Hang analysis/system state
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 63
HANGANLYZE Level:
10 Dump all processes
5 Level 4 + Dump all processes involved in wait chains (NLEAF state)
4 Level 3 + Dump leaf nodes (blockers) in wait chains (LEAF,LEAF_NW,IGN_DMP state)
3 Level 2 + Dump only processes thought to be in a hang (IN_HANG state)
-- recommended
1-2 Only HANGANALYZE output, no process dump at all
64. Tools & Utilities – Hang analysis/system state
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 64
Review the trace file:
HANG ANALYZE (section)
CYCLES
list process dependencies for deadlock/hung state
BLOCKER OF MANY SESSIONS:
When a session block too many session, you will have this section, when a session block
10 or more sessions
STATE OF NODES | OPEN CHAINS | OTHER CHAINS
65. Tools & Utilities – Hang analysis/system state
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 65
Dumping system state
When a database in a complete hung state and you can’t connect to the database as / as
sysdba, when memory leaks are suspected, use the following:
sqlplus – prelim / as sysdba
SQL> oradebug setmypid
SQL> oradebug unlimit;
SQL> oradebug – g all dump systemstate 10|266
Wait for 60 seconds
SQL> oradebug -g all dump systemstate 10|266
Review/upload the trace file to MOS
66. References
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 66
RACcheck - RAC Configuration Audit Tool [ID 1268927.1]
Oracle Premier Support - Oracle Database Support News - Issue November, 2012 Volume 22 [ID 1513219.1
Troubleshoot Grid Infrastructure Startup Issues [ID 1050908.1]
Oracle Clusterware CRSD OCSSD EVMD Log Rotation Policy [ID 557204.1]
CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage [ID 783456.1]
Remote Diagnostic Agent (RDA) 4 - Getting Started [ID 314422.1]
Data Gathering for Troubleshooting Oracle Clusterware (CRS or GI) And Real Application Cluster (RAC) Issues [ID 289690.1]
67. A big thank you all
for
listening ...
Presented by : Syed Jaffer Hussain RedGate/AllThingsOracle Slide # 67
You can write me at sjaffarhussain@gmail.com