Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version

3,377 views

Published on

Oracle RAC 12c Best Practices with some collaborative twist.

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,377
On SlideShare
0
From Embeds
0
Number of Embeds
38
Actions
Shares
0
Downloads
422
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
  2. 2. Oracle Real Application Clusters (RAC) 12c Collaborate Best Practices Markus Michalewicz Director of Product Management Oracle Real Application Clusters (RAC) @OracleRACpm http://www.linkedin.com/in/markusmichalewicz http://www.slideshare.net/MarkusMichalewicz
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 Oracle RAC – Family of Solutions For more information see: http://www.slideshare.net/MarkusMichalewicz/ http://www.slideshare.net/MarkusMichalewicz/the-oracle-real-application-clusters-rac-family-of-solutions
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6 Standardize on Clusters Clusters increase flexibility and HA regardless of DB types used later High Availability Flexibility License Costs Re-configuration to enable cluster
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 Standardize on Oracle RAC For all the flexibility you need Flexibility Scalability Online upgrade Online (rolling) Patch and PSU Application
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 Standardize on Oracle RAC and Oracle Multitenant The new standard for Oracle Database Consolidation Consolidation Agility More on Fiday, Apr 11th, Session 2548 Oracle Multinenant meets RAC – Level 3, Lido 3103, 11am
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 Preparation of the Servers !  Setup of the Servers include: –  Setting up network connections !  Public and private with switch –  Setting up shared storage for all servers !  Shared storage required –  Setting up the OS image – here OL 6.4 !  Suggestion: –  Use cloning to deploy OS images fastervixencomet dancerdasher OS OL 6.4 OS OL 6.4 OS OL 6.4OS OL 6.4 OS image setup and more
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 Preparation of the Servers !  Recommendations: –  Use OL 6.4+ UEK –  Use the Oracle preinstall package !  Available for –  Oracle Database 11g Rel. 2 –  Oracle Database 12c Rel. 1 –  Additional steps to perform after using the pre-install package for an Oracle RAC cluster: !  Add a “grid” user & “dba” group !  Align UIDs / GUIDs across servers !  Set SELinux to “permissive”, if possible !  Check large / huge pages settings Kernel and Oracle preinstall package [root@dasher ~]# uname –a Linux dasher 2.6.39-400.17.1.el6uek.x86_64 #1 SMP Fri Feb 22 18:16:18 PST 2013 x86_64 x86_64 x86_64 GNU/Linux #Get the pre-install package [root@dasher Desktop]# yum list oracle-* oracle-rdbms-server-11gR2-preinstall.x86_64 1.0-7.el6 ol6_latest oracle-rdbms-server-12cR1-preinstall.x86_64 1.0-8.el6 ol6_latest
  12. 12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 Install Oracle Grid Infrastructure 12c What is the default when keeping the preselection in steps 1-4…? !  GI for a Cluster Installation !  Standard Cluster !  Typical Installation
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 Install Oracle Grid Infrastructure 12c Step 3: Double-check the cluster topology to deploy!
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 Oracle Flex Cluster The new standard for resource management in the cluster Hub Nodes Leaf Nodes !  Oracle Flex Cluster: –  The new standard for flexible resource management in the cluster !  Use it as demand requires it !  Inherits all Flex ASM HA capabilities !  Will support more use cases in future –  Efficient High Availability and resource management across tiers !  Lightweight cluster stack on leaf nodes –  Benefit from virtualization and physical deployments in one cluster
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 Install Oracle Grid Infrastructure 12c For now, “Standard Cluster” is the default given the preselection !  Oracle Flex Cluster –  Can consist of only HUB-nodes !  If a Leaf-node has access to shared storage, it can perform a role change to a HUB-node –  A Standard Cluster can be converted to a Flex Cluster at any time after the installation !  Once a cluster is configured as a Flex Cluster this configuration is irreversible without a re- configuration of the cluster. !  Recommendation: –  An all-HUB-node Flex Cluster behaves the same as a Standard Cluster using Flex ASM. –  Either can be used, but if in doubt, you can choose standard cluster and convert later. More Information in Appendix A
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 Install Oracle Grid Infrastructure 12c Step 4: Do not accept the default – choose “Advanced Installation” !  Typical Installation –  Does not provide an option to use “Flex ASM” –  Does not provide an option to create the “Mgmt DB / Grid Infrastructure Management Repository” !  Enhancement Requests for later releases have been filed already !  Advanced Option –  Recommended for all configurations
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17 Install Oracle Grid Infrastructure 12c “Advanced Installation” – SCAN & GNS requirement !  Recommendations: –  SCAN remains to be required for the cluster –  Choose cluster-specific unique SCAN names –  OUI will try to enforce it as much as possible !  http://www.oracle.com/technetwork/products/ clustering/overview/scan-129069.pdf –  Choose “Configure GNS” if: !  You want to use dynamic IP management (DHCP assigned / IPv6 Stateless Address Autoconfiguration) for client connectivity –  In this case, GNS requirements remain as with previous version for standard cluster !  You chose to install an Oracle Flex Cluster
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 Install Oracle Grid Infrastructure 12c “Advanced Installation” – Network Interface Usage !  Recommendations: –  Use “Redundant Interconnect Usage” feature !  Bonding can be used underneath if required !  MOS notes explain further configuration – e.g.: –  MOS note 1210883.1 - 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip • It is recommended to align the ASM network with the private interconnect. • Using different subnets for the private interconnect is required.
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 Install Oracle Grid Infrastructure 12c “Advanced Installation” – Grid Infrastructure Management Repository !  Grid Infrastructure Management Repository (GIMR) –  A Single Instance Oracle Database 12c –  It will be installed on one of the nodes !  It is managed as a failover database –  It contains the Cluster Health Monitor (CHM) data –  It is stored in the first ASM disk group created –  As part of the configuration, Quality of Service Management (QoS) required resources are added to the cluster configuration (e.g. OC4J) !  Recommendation: configure the GIMR!
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20 Install Oracle Grid Infrastructure 12c “Advanced Installation” – Storage Options !  For Standard Cluster Installations three storage options are available: 1.  Standard ASM !  ASM configuration as known from previous releases 2.  Oracle Flex ASM !  Recommended option available with Oracle Grid Infrastructure 12c –  Flex Cluster requires Flex ASM 3.  Shared File System !  Alternative
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21 Oracle ASM 12c – Overview Oracle ASM 12c Standard Deployment ASM Cluster Pool of Storage Disk Group BDisk Group AShared Disk Groups Wide File Striping One to One Mapping of ASM Instances to Servers ASM Instance Database Instance ASM Disk RAC Cluster Node4Node3Node2Node1 Node5ASM ASM ASM ASM ASM ASM Instance Database Instance DBA DBA DBCDBB DBBDBB
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22 Introducing Oracle Flex ASM Removal of One to One Mapping and HA ASM Cluster Pool of Storage Disk Group BDisk Group AShared Disk Groups Wide File Striping Databases share ASM instances ASM Instance Database Instance ASM Disk RAC Cluster Node5Node4Node3Node2Node1 Node5 runs as ASM Client to Node4 Node1 runs as ASM Client to Node2 Node1 runs as ASM Client to Node4 Node2 runs as ASM Client to Node3 ASM ASM ASM ASM Instance DBA DBA DBCDBB DBBDBB
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 Install Oracle Grid Infrastructure 12c Use Oracle Flex ASM for storage management !  Recommendations: –  Use Oracle Flex ASM for storage !  For better HA of database instances !  Reserve per-node spare memory [GRID]> ps -ef |grep pmon grid 3822 1 0 08:05 ? 00:00:00 asm_pmon_+ASM1 grid 4865 1 0 08:06 ? 00:00:00 mdb_pmon_-MGMTDB grid 9592 1 0 08:09 ? 00:00:00 apx_pmon_+APX1 grid 19114 19003 0 09:22 pts/0 00:00:00 grep pmon More Information in Appendix B
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 Install Oracle Grid Infrastructure 12c “Advanced Installation” – Create ASM Disk Group !  Recommendations: –  Create Infrastructure Disk Group (DG) first !  Use it to host OCR / Voting Disk –  Choose Redundancy accordingly –  Normal(+) redundancy is recommended –  Create Data and FRA disk groups post installation and as required. !  Optional: Use it to host shared DB homes by creating an ACFS file system on top of it –  http://www.oracle.com/technetwork/products/cloud-storage/ oop-patching-of-acfs-shared-oh-1885763.pdf –  ADVM volumes can be of lower redundancy than the underlying DG
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 Install Oracle Grid Infrastructure 12c “Advanced Installation” – Prerequisite Checks: Use FixUp Scripts!
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26 New Memory Requirements Just something to consider [GRID]> srvctl status asm ASM is running on comet,dasher,dancer [GRID]> srvctl status mgmtdb Database is enabled Instance -MGMTDB is running on node dasher [GRID]> ps -ef |grep pmon grid 3227 1 0 13:54 ? 00:00:02 asm_pmon_+ASM1 grid 3868 1 0 13:55 ? 00:00:01 mdb_pmon_-MGMTDB grid 7364 1 0 13:57 ? 00:00:01 apx_pmon_+APX1 oracle 7917 1 0 13:57 ? 00:00:02 ora_pmon_raccdb1_4 [RAC]> df -h Filesystem Size Used Avail Use% Mounted on ... tmpfs 1.7G 1.3G 475M 73% /dev/shm ... Physical memory requirements with Oracle RAC 12c: 4GB
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28 Install the Oracle Database Home Nothing special – just a quick walk through Not the default…
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29 Install the Oracle Database Home A few things to point out ACFS based shared home …
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30 Before Creating Databases Create Server Pools and additional disk groups as required [GRID]$ srvctl add srvpool –serverpool backoffice –min 1 –max 2 –i 1 [GRID]$ srvctl add srvpool –serverpool frontoffice –min 1 –max 4 –i 2 Create additional disk groups as required.
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 Create an Oracle Database What is the default when keeping the preselection in steps 1-4…? !  Policy-managed !  Oracle RAC-based !  Non-CDB Database
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33 Create an Oracle RAC Database A few tips and tricks… Create an empty multitenant CDB.
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34 Create an Oracle RAC Database A few tips and tricks… This is the default…, but not recommended !  Recommendations: –  Use a different disk group than the “database file location”-disk group as the Fast Recovery Area. –  Automatic Storage Management (ASM) recommendation remains unchanged !  Use it for any current production database !  New in Oracle Database 12c: ACFS can be used to host database files –  Use ACFS for database files, if you want to !  manage “all data” on a file system !  use advanced functionality
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35 Milestone What has been installed and configured so far? vixen srvpool: frontofficecomet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice OS OL 6.4 OS OL 6.4 OS OL 6.4OS OL 6.4 Oracle GI for a cluster Oracle GI for a cluster Oracle GI for a clusterOracle GI for a cluster Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_2 raccdb1_1 [RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer [RAC]>srvctl status srvpool -serverpool backoffice -detail Server pool name: backoffice Active servers count: 2 Active server names: comet,dancer NAME=comet STATE=ONLINE NAME=dancer STATE=ONLINE [RAC]> srvctl status srvpool Server pool name: Free Active servers count: 0 Server pool name: Generic Active servers count: 0 Server pool name: backoffice Active servers count: 2 Server pool name: frontoffice Active servers count: 2
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36 Create Pluggable Databases Create HR PDB on running instances vixen srvpool: frontofficecomet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI for a cluster Oracle GI for a cluster Oracle GI for a clusterOracle GI for a cluster Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_2 raccdb1_1 Create a PDB on a node that hosts a multintenant CDB instance
  37. 37. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  38. 38. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38 Post-Creation Steps Create services to balance the workload vixen srvpool: frontofficecomet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI for a cluster Oracle GI for a cluster Oracle GI for a clusterOracle GI for a cluster Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_2 raccdb1_1 [RAC]> srvctl add service -d raccdb1 -s hrsvc -pdb hr - serverpool backoffice [RAC]> srvctl add service -d raccdb1 -s crmsvc -pdb crm - serverpool frontoffice !  Use Dynamic Database Services –  For easier management of multitenant CDBs –  For better workload management in the cluster !  Start services after creation and as required Details on FRI, Apr 11th, Session 2548 Oracle Multinenant meets RAC – Level 3, Lido 3103, 11am More Information in Appendix C
  39. 39. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration –  Testing –  Monitoring &Analysis !  Summary and Q&A !  Appendices A-C
  40. 40. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40 Engineered Systems Optimized and already tested Oracle Exadata Database Machine Oracle Database Appliance Quarter Rack Half Rack Full Rack • IB Interconnect • Low Overhead IPC Prot. Preconfigured
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41 Oracle Real Application Testing Gather and deploy workload on any database type… ERP CRM DW
  42. 42. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42 Standardize on Oracle RAC (One Node) Here: to simplify Oracle RAC testing … Flexibility Scalability Re-configuration to enable cluster Online upgrade
  43. 43. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43 Standardize on Oracle RAC (One Node) Oracle RAC testing made simple: 1.  Standardize on Oracle RAC (One Node) 2.  Run tests on Oracle RAC One Node instance –  RAC One Node behaves basically like a single instance 3.  Online Upgrade to Oracle RAC for testing only 4.  Run tests on Oracle RAC 5.  Decide whether the application provides service goals when load balanced over multiple active instances –  Identify bottlenecks and tuning actions as required To simplify Oracle RAC testing vixencomet RON RON_2 Online upgrade RON_1 AWR --- ASH AWR --- ASH Decide Oracle GI Oracle RAC Oracle GI Oracle RAC
  44. 44. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration –  Testing –  Monitoring &Analysis !  Summary and Q&A !  Appendices A-C
  45. 45. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45 Post-Install Steps Check your environment regularly using CVU and ORAcheck #CVU is run regularly by default [GRID]> srvctl status cvu CVU is enabled and running on node comet !  CVU is run regularly as part of the GI stack !  Use ORAcheck (formerly RACcheck) the Oracle RAC Configuration Audit Tool –  For details see MOS note 1268927.1 !  Checks the Oracle Database: –  Standalone Database –  Grid Infrastructure & RAC –  Maximum Availability Architecture (MAA) Validation –  Upgrade Readiness Validation –  Golden Gate !  E-Business Suite !  Oracle SUN Systems
  46. 46. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46 Monitoring Essentials dasher dancer vixencomet Use Cluster Health Monitor for OS (CHM/OS) 1.  First failure diagnostics: Collects as much system metrics and data as feasible 2.  Collects system metrics on all nodes in the cluster synchronized 3.  Persistent storage of all collected data in a database 4.  Integrated with Oracle Grid Infrastructure starting with 11.2.0.2 (12c: uses GIMR) 5.  Robust collection in situations of resource starvation 6.  Data dump or visualization LOGGERD sysmond sysmond Oracle GI Oracle GI Oracle GI Oracle GI sysmond sysmond sysmond LOGGERD
  47. 47. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47 Log Collection Essentials dasher dancer vixencomet Runtime and Emergency Data Collections !  Save database statistics –  For baselines and reference: export AWR !  For diagnostics: –  Always get AWR and ASH reports !  For emergencies and “last gasp” –  get ASH dump (e.g. oradebug ashdunp level 0) !  CHM/OS –  oclumon dumpnodeview -n pecdb08 pecdb09 -s "2011-09-30 15:00:00" -e "2011-09-30 15:30:00" -v –  oclumom –manage –report <ret. time in secs> !  CSS logs from all nodes –  Node and instance evictions –  Interconnect issues (e.g. gc blocks lots ) Oracle GI Oracle RAC Oracle GI Oracle RAC Oracle GI Oracle RAC Oracle GI Oracle RAC LOGs LOGs LOGs LOGs
  48. 48. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48 Log Collection Essentials dasher comet Use Trace File Analyzer (TFA) to collect Data !  Goals –  Improved comprehensive first failure diagnostics collection –  Efficient collection, packaging and transfer of data for customers –  Reduce round trips between customers and Oracle –  Supports 10.2, 11.1, 11.2 and above –  Included in the 11.2.0.4 patch set and future versions Oracle GI Oracle RAC Oracle GI Oracle RAC LOGs LOGs TFA
  49. 49. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.49 Log Collection Essentials dasher comet Use Trace File Analyzer (TFA) to collect Data !  Approach –  Collect for all relevant components (OS, Grid Infrastructure, ASM, RDBMS) –  One command to collect all information –  Prune large files based on temporal criteria –  Collect time relevant IPS (incident) packages on RAC nodes –  Collect time relevant CHM/OS, OSWatcher data on all Oracle RAC nodes –  On-demand (default) and Event Driven diagnostic collections !  TFA Collector – MOS note ID 1513912.1 Oracle GI Oracle RAC Oracle GI Oracle RAC LOGs LOGs TFA
  50. 50. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.50 Log Collection Essentials dasher comet Use Trace File Analyzer (TFA) to collect Data !  TFA Version 3.2 Updates (07Apr14): 1.  First version of the Log Analyzer. 2.  Addition of –notrim for diagnostic collections. 3.  Support for TFA on zLinux. 4.  Collection of ExaWatcher data. 5.  Sundiag collection 6.  Aging of TFA log files restricting maximum size. 7.  Multiple performance enhancements 8.  Critical Bug fixes !  TFA Collector – MOS note ID 1513912.1 Oracle GI Oracle RAC Oracle GI Oracle RAC LOGs LOGs TFA
  51. 51. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.51 TFA – Usage Example Sending diagcollect request to host : myhost2 Getting list of files satisfying time range [Tue Sep 03 14:17:43 PDT 2013, Tue Sep 03 18:17:43 PDT 2013] myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswiostat/myhost1_iostat_13.09.03.1500.dat.gz myhost1: Zipping File: /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log Trimming file : /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log with original file size : 109kB myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswtop/myhost1_top_13.09.03.1500.dat.gz myhost1: Zipping File: /opt/oracle/oak/log/myhost1/oak/oakd.log Trimming file : /opt/oracle/oak/log/myhost1/oak/oakd.log with original file size : 9.2MB myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/gipcd/gipcd.log myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log Trimming file : /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log with original filesize 4.3MB myhost1: Zipping File: /var/log/messages …Truncated for brevity myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswslabinfo/myhost1_slabinfo_13.09.03.1800.dat Collecting ADR incident files... Total Number of Files checked : 10543 Total Size of all Files Checked : 3.9GB Number of files containing required range : 68 Total Size of Files containing required range : 129MB Number of files trimmed : 10 Total Size of data prior to zip : 144MB Saved 63MB by trimming files Zip file size : 8.6MB Total time taken : 47s. Logs are collected to: /opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost1.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip /opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost2.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip OS Watcher files Pruning 47 seconds! – 1 command, 2 nodes, 4 databases, ASM, Clusterware, OS Relevant files only $ sudo ./tfactl diagcollect 144MB pruned and compressed down to 8.6MB One simple command ADR Incident files
  52. 52. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.52 Analysis Essentials Time Series Analysis – AWR Reports are not enough due to average… Occurrence of an interconnect problem ~18 secs avg !!!! !  Solution –  ASH reports Every 1 hour or out-of-space AWR Circular buffer in SGA (2MB per CPU) DBA_HIST_ACTIVE_SESS_HISTORYV$ACTIVE_SESSION_HISTORY Session state objects MMON Lite (MMNL) V$SESSION V$SESSION_WAIT Write 1 out of 10 samples Variable length rows Direct path inserts
  53. 53. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.53 Post-Install Steps Monitor best practices notes and announcements !  ORAcheck - Oracle Configuration Audit Tool –  For details see MOS note 1268927.1 !  Best practices My Oracle Support (MOS) notes: –  MOS note 810394.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) !  MOS note 811306.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Linux) !  MOS note 811280.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) –  Note: Engineered Systems with the same OS will have additional recommendations !  MOS note 811271.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) !  MOS note 811303.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (HP-UX) !  MOS note 811293.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (AIX) !  MOS note 1452965.2 – –  Information Center: Oracle Scalability Grid Infrastructure / Clusterware and Real Application Clusters (RAC)
  54. 54. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.54 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  55. 55. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.55 More Information – Extended RAC and More !  Presentation: Oracle RAC on Extended Distance Clusters –  http://www.slideshare.net/MarkusMichalewicz/oracle-rac-on-extended-distance-clusters !  WP: Oracle Real Application Clusters on Extended Distance Clusters –  http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf !  WP: Using standard NFS to support a third voting file for extended cluster configurations –  http://www.oracle.com/technetwork/database/database-technologies/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf !  Presentation: Oracle Clusterware Node Management and Voting Disks –  http://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voting-disks !  Presentation: Oracle Database with RAC 12c HA BP TCOUG –  http://www.slideshare.net/MarkusMichalewicz/oracle-database-with-rac-12c-ha-bp-tcoug
  56. 56. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.56 Questions and Answers
  57. 57. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.57 Agenda !  Oracle RAC 12c and Standardization !  Installation of Oracle Grid Infrastructure 12c !  Installation of the Oracle Database Home !  Oracle RAC Database Creation using DBCA !  Post-install Steps and Configuration !  Summary and Q&A !  Appendices A-C
  58. 58. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.58 Appendix A: Converting to an Oracle Flex Cluster with Node Conversion Example
  59. 59. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.59 Oracle Flex Cluster Some configuration details !  Oracle Flex Cluster –  Will be the basis for more uses cases in future releases (for resource mgmt) –  Can consist of only HUB-nodes –  If a Leaf-node has access to shared storage, it can perform a role change to a HUB-node –  A Standard Cluster can be converted to a Flex Cluster at any time after the installation –  Once a cluster is configured as a Flex Cluster this configuration is irreversible without a re- configuration of the cluster. !  More Information: –  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/ bigcluster.htm#CWADD92560 [GRID]> crsctl get cluster mode config Cluster is configured as "standard" [GRID]> crsctl get cluster mode status Cluster is running in "standard" mode [GRID]> olsnodes -s -a -t dasher Active Hub Unpinned vixen Active Hub Unpinned dancer Active Hub Unpinned comet Active Hub Unpinned •  Not a Flex Cluster yet. •  An all-Hub Flex Cluster is basically a Standard Cluster
  60. 60. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.60 Oracle Flex Cluster Converting to Flex Cluster !  More Information: –  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/ bigcluster.htm#CHDJGBII !  Flex Cluster implementations require GNS !  Converting a Flex Cluster requires a cluster restart –  Use “crsctl stop crs” followed by “crsctl start crs” as root –  Hence the difference between “status” and “config” –  Nodes can be restarted in a rolling fashion: [GRID]> srvctl config gns PRKF-1110 : Neither GNS server nor GNS client is configured on this cluster [GRID]> more /etc/hosts |grep gns 10.1.1.99 mygns mygns.localdomain #Change to root [GRID]> su Password: [GRID]> srvctl add gns -vip mygns [GRID]> srvctl start gns [GRID]> crsctl set cluster mode flex Cluster mode set to "flex » [GRID]> crsctl get cluster mode status Cluster is running in "standard" mode [GRID]> crsctl get cluster mode config Cluster is configured as "flex" [GRID]> date ; hostname ; crsctl get cluster mode status Sat Sep 14 14:43:45 PDT 2013 dasher Cluster is running in "flex" mode [GRID]> date ; hostname ; crsctl get cluster mode status Sat Sep 14 14:43:37 PDT 2013 dancer Cluster is running in "standard" mode
  61. 61. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.61 Oracle Flex Cluster Standard to Flex Cluster conversion – what has changed? vixen srvpool: frontofficecomet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1 [GRID]> crsctl get cluster mode status Cluster is running in "flex" mode [GRID]> olsnodes -a -s -t dasher Active Hub Unpinned vixen Active Hub Unpinned dancer Active Hub Unpinned comet Active Hub Unpinned [RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer Instance raccdb1_3 is running on node vixen Instance raccdb1_4 is running on node dasher !  Nothing at first glance… –  An all-Hub Flex Cluster behaves like a Standard Cluster –  When converting from Standard Cluster all nodes are converted into HUB-nodes
  62. 62. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.62 Oracle Flex Cluster Changing node roles – server vixen becomes a Leaf node !  More Information: –  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/ bigcluster.htm#BABHCEAE [GRID]> hostname vixen [GRID]> srvctl config vip -node vixen VIP exists: network number 1, hosting node vixen VIP Name: vixen-vip VIP IPv4 Address: 10.1.1.33 VIP IPv6 Address: [GRID]> crsctl get node role config Node 'vixen' configured role is 'hub' #Change to root … [GRID]> su Password: [GRID]> crsctl set node role leaf CRS-4408: Node 'vixen' configured role successfully changed; restart Oracle High Availability Services for new role to take effect. [GRID]> crsctl stop crs [GRID]> crsctl start crs [GRID]> crsctl get node role config Node 'vixen' configured role is 'leaf' [GRID]> srvctl config vip -node vixen VIP exists: network number 1, hosting node vixen VIP Name: vixen-vip VIP IPv4 Address: 10.1.1.33 VIP IPv6 Address:
  63. 63. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.63 Oracle Flex Cluster Node role change to Leaf – what has changed? vixen srvpool: FREEcomet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | LEAFOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_4 raccdb1_2 raccdb1_1 [RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer Instance raccdb1_4 is running on node dasher [RAC]> srvctl status server -servers vixen -detail Server active pools: Free Server name: vixen Server state: ONLINE Server state details: !  Server vixen has converted to a Leaf node !  Leaf nodes do not host DB instances in 12.1 !  Server vixen moved to Server Pool “FREE” as Server Pool “frontoffice” is a database pool –  Note: all nodes in this example have access to shared storage, which means only the node role changes – storage access is preserved.
  64. 64. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.64 Oracle Flex Cluster Revert role change – server vixen becomes a HUB again comet srvpool: backoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle RAC Oracle RAC Oracle RAC raccdb1 raccdb1_4 raccdb1_2 raccdb1_1 vixen srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_3 #As root [GRID]> crsctl set node role hub CRS-4408: Node 'vixen' configured role successfully changed; restart Oracle High Availability Services for new role to take effect. [GRID]> crsctl stop crs [GRID]> crsctl start crs [RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer Instance raccdb1_3 is running on node vixen Instance raccdb1_4 is running on node dasher
  65. 65. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.65 Appendix B: Oracle Flex ASM – Details and Test Scenarios
  66. 66. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.66 Oracle Flex ASM Configuration details !  Flex ASM introduces new local resources: –  At least one ASM listener –  One “proxy_advm” (per node) !  Used for ACFS access to Flex ASM instances !  Connections from a database instance to an ASM instance are based on SQLnet using listeners. –  The listener directs the connection to the least loaded ASM instance based on the load metric it maintains. –  The connection details are fetched from CSS global data –  The ASM instance to which the database instance connects to is listed in the database alert log: !  NOTE: ASMB connected to ASM instance +ASM1 (Flex mode; client id 0x10004) –  The userid and password supplied are also managed automatically. They are supplied while establishing the session, not while connecting. [GRID]> crsctl stat res -t ------------------------------------------------------------------------------- Name Target State Server State details ------------------------------------------------------------------------------- Local Resources ------------------------------------------------------------------------------- ora.ASMNET1LSNR_ASM.lsnr ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE ONLINE ONLINE vixen STABLE ... ora.DATA.dg ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE OFFLINE OFFLINE vixen STABLE ... ora.INFRA.dg ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE OFFLINE OFFLINE vixen STABLE ... ora.proxy_advm ONLINE ONLINE comet STABLE ONLINE ONLINE dancer STABLE ONLINE ONLINE dasher STABLE ONLINE ONLINE vixen STABLE
  67. 67. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.67 Oracle Flex ASM Administering Flex ASM !  Check ASM mode using asmcmd !  Check where ASM instances are running using srvctl !  Check the configuration of ASM using srvctl !  Modify the ASM configuration using srvctl !  Cluster ASM listener: –  Flex ASM can have its own network or overlap with the private interconnect network used by RAC DB instances –  It is recommended to configure a full overlap of the private interconnect and the ASM network in current use cases: [GRID]> asmcmd ASMCMD> showclustermode ASM cluster : Flex mode enabled [GRID]> srvctl status asm ASM is running on comet,dasher,dancer [GRID]> srvctl config asm -detail ASM home: /u01/app/12.1.0/grid Password file: +INFRA/orapwASM ASM listener: LISTENER ASM is enabled. ASM instance count: 3 Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM [GRID]> srvctl modify asm -count all [GRID]> srvctl status asm ASM is running on comet,dasher,dancer [GRID]> srvctl start asm [GRID]> srvctl status asm ASM is running on comet,vixen,dasher,dancer
  68. 68. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.68 Oracle Flex ASM Some test scenarios… !  A failure of an ASM instance is without effect for the databases using ASM disk groups (here: -MGMTDB) –  If an ASM instance is running on all nodes (count all) the ASM instance is restarted upon failure –  If an ASM instance is running on a subset of nodes the ASM instance might be restarted on another node –  The client using the ASM instance may change !  A failure of an ASM instance is without effect for the APX (ADVM proxy) running on the same node !  A failure of the APX will lead to a restart of the database instance (on the node) that uses an ACFS file system. !  More information: –  http://docs.oracle.com/cd/E16655_01/server.121/e17612/ asminst.htm#OSTMG95329 [GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4093 1 0 11:35 ? 00:00:00 asm_pmon_+ASM1 grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4 [GRID]> kill -9 4093 [GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4 [GRID]> ps -ef |grep pmon grid 3840 1 0 09:11 ? 00:00:01 mdb_pmon_-MGMTDB grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1 oracle 5092 1 0 11:36 ? 00:00:00 ora_pmon_raccdb1_4 grid 5748 1 0 11:39 ? 00:00:00 asm_pmon_+ASM1 grid 5793 19013 0 11:39 pts/1 00:00:00 grep pmon [GRID]> asmcmd lsct data DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX1 DATA [GRID]> ssh vixen /u01/app/12.1.0/grid/bin/asmcmd lsct data DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX2 DATA +APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX3 DATA
  69. 69. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.69 Appendix C: Policy-Managed Databases and with Oracle RAC 12c
  70. 70. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.70 Policy-Managed Databases with Oracle RAC 12c Database Services vixen srvpool: frontoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle RAC Oracle RAC Oracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 [GRID]> srvctl status serverpool Server pool name: frontoffice Active Servers count: 2 Server pool name: backoffice Active Servers count: 2 [RAC]> srvctl status service -d raccdb1 Service crmsvc is running on nodes: dasher,vixen Service hrsvc is running on nodes: comet,dancer !  Database Services –  Ensure that workload is hosted in the respective server pool with the cardinality defined as part of the policy definition. comet srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_1
  71. 71. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.71 Policy-Managed Databases with Oracle RAC 12c Improved HA Management - Startup raccdb1 [GRID]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 Server pool name: Free Importance: 0, Min: 0, Max -1 !  Defining a Service Startup Order –  Ensure services are started in specified groups and specified order. –  Leverages Serverpool Min and Importance properties dasher srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_4 1st vixen srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_3 2nd comet srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_1 4th dancer srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_2 3rd
  72. 72. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.72 dancer srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_2 dancer srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_2 Policy-Managed Databases with Oracle RAC 12c Improved HA Management – New Failover Strategy raccdb1 [GRID]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [GRID]> srvctl status serverpool Server pool name: frontoffice Active Servers count: 2 Server pool name: backoffice Active Servers count: 1 dasher srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_4 vixen srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_3 comet srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_1 !  Servers “Move” to Replace Failed node –  Protects against cascade failures due to load –  Ensures workload isolation between pools –  Less important workloads shutdown transactionally
  73. 73. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.73 dancer srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_2 dancer srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_2 Policy-Managed Databases with Oracle RAC 12c Improved HA Management – Manage Last Service Standing raccdb1 [RAC]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 2 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [RAC]> srvctl status service –db Service crmsrv is running on nodes comet,vixen Service hrsvc is not running dasher srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_4 vixen srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_3 comet srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_1 !  Business Critical Services survive multiple failures –  Most important pool always gets the servers –  Services preserved across multiple failures –  Less important workloads shutdown transactionally comet srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_1
  74. 74. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.74 Policy-Managed Databases with Oracle RAC 12c Improved HA Management – Dynamic Provisioning vixen srvpool: frontoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUB Oracle RAC Oracle RAC Oracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 [GRID]> srvctl modify serverpool –serverpool backoffice –max 1 [RAC]> srvctl config serverpool Server pool name: frontoffice Importance: 10, Min: 2, Max 4 Server pool name: backoffice Importance: 5, Min: 1, Max 1 [RAC]> srvctl status service –db Service crmsrv is running on nodes dasher,dancer,vixen Service hrsvc is not running comet srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_1 !  Add Servers Just-In Time to meet demand –  Server Pools sized via Min and Max properties –  Dynamically controlled by QoS Management –  Planned control via Clusterware or QoSM Policies dancer srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_2
  75. 75. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.75 Policy-Management with Oracle RAC 12c Support for Multiple Policies tracking business objectives !  More Information: –  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/pbmgmt.htm !  New in Oracle Grid Infrastructure 12c –  Server Categories !  Server Categories use server attributes to allow for an active use of differently sized servers –  Policy Sets !  Policy Sets allow for dynamic adjustment to demand changes in an atomic transaction Server Categories •  NAME •  ACL •  EXPRESSION •  … Server Attributes •  NAME •  MEMORY_SIZE! •  CPU_COUNT •  CPU_CLOCK_RATE! •  CPU_HYPERTHREADING! •  CPU_EQUIVALENCY • …! [GRID]> crsctl modify policyset –attr "LAST_ACTIVATED_POLICY=NightTime“ CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded CRS-2673: Attempting to start 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2672: Attempting to start 'ora.raccdb1.hrsvc.svc' on 'dancer‘ … QoS Management OR Oracle Clusterware - CRSCTL
  76. 76. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.76 Policy-Management with Oracle RAC 12c Setting up policy sets – Provision Server Pools and creating a PolicySet !  Add another server pool “backup” !  Set up policy set with 3 server pools & 3 policies as follows: –  DayTime: !  frontoffice uses three servers (MIN_SIZE=3) !  backoffice uses one server (MIN_SIZE=1) !  backup does not run during daytime (MIN_SIZE=0) –  NightTime: !  frontoffice uses one server (MIN_SIZE=1) !  backoffice uses two servers (MIN_SIZE=2) !  backup uses only one server (MIN_SIZE=1) –  Weekend: !  frontoffice uses one server (MIN_SIZE=1) !  backoffice uses one server (MIN_SIZE=1) !  backup uses two servers (MIN_SIZE=2) [GRID]> srvctl add serverpool –serverpool backup –min 0 –max 2 –importance 20 [GRID] srvctl status serverpool Server pool name: frontoffice Active Servers count: 3 Server pool name: backoffice Active Servers count: 1 Server pool name: backup Active Servers count: 0 [GRID]> crsctl modify policyset -file /u01/app/…/policyset.txt
  77. 77. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.77 Policy-Management with Oracle RAC 12c Using Policy Sets – part 1: Activate a policy !  Activate the policy for the weekend [GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=Weekend" [GRID]> srvctl config serverpool Server pool name: Free Importance: 0, Min: 0, Max: -1 Category: Candidate server names: Server pool name: Generic Importance: 0, Min: 0, Max: -1 Category: Candidate server names: Server pool name: backoffice Importance: 0, Min: 1, Max: 1 Category: Candidate server names: Server pool name: backup Importance: 1, Min: 2, Max: 2 Category: Candidate server names: Server pool name: frontoffice Importance: 0, Min: 1, Max: 1 Category: Candidate server names:
  78. 78. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.78 Policy-Management with Oracle RAC 12c Using Policy Sets – part 2: check the result vixen srvpool: frontofficecomet srvpool: backup dancer srvpool: backupdasher srvpool: backoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1 [RAC]> srvctl status database -d raccdb1 Instance raccdb1_1 is running on node comet Instance raccdb1_2 is running on node dancer Instance raccdb1_3 is running on node vixen Instance raccdb1_4 is running on node dasher [RAC]> srvctl status service -d raccdb1 Service backup is running on nodes: comet,dancer Service crmsvc is running on nodes: vixen Service hrsvc is running on nodes: dasher !  With an activated weekend policy
  79. 79. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.79 Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – part 1 !  Assume you have 2 servers that have better IO –  Use these servers for backups whenever possible !  Here comet and dancer have better IO by definition !  What you need to do: –  Set up a server category that identifies the servers –  Add the use of the server category to the server pool !  Define the server pools that utilize the category and during which policy activation it shall be used. !  You need to restart the cluster stack on the servers that you modify in this fashion [GRID]> su Password: [GRID]> crsctl set server label IOplus ... #On dancer [GRID]> crsctl set server label Ioplus [GRID]> crsctl get server label CRS-4972: Current SERVER_LABEL parameter value is Ioplus [GRID]> crsctl status server comet dancer –f Comet Dancer NAME=comet MEMORY_SIZE=3338 CPU_COUNT=1 CPU_CLOCK_RATE=2132 CPU_HYPERTHREADING=0 CPU_EQUIVALENCY=1000 DEPLOYMENT=other CONFIGURED_CSS_ROLE=hub RESOURCE_USE_ENABLED=1 SERVER_LABEL=IOplus ... NAME=dancer MEMORY_SIZE=3338 CPU_COUNT=1 CPU_CLOCK_RATE=2247 CPU_HYPERTHREADING=0 CPU_EQUIVALENCY=1000 DEPLOYMENT=other CONFIGURED_CSS_ROLE=hub RESOURCE_USE_ENABLED=1 SERVER_LABEL=IOplus ...
  80. 80. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.80 Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – part 2 !  What you need to do: –  Set up a server category that identifies the servers - done –  Add the use of the server category to the server pool !  Define the server pools that utilize the category and during which policy activation it shall be used !  Note: Add the server category usage to the server pool(s) that require it & specify the policies that should be altered accordingly –  In this example the backup server pool is the one that will benefit the most and the policies will only start this server pool as required. [GRID]> crsctl add category moreIO –attr "EXPRESSION='SERVER_LABEL co IOplus'" [GRID]> crsctl status category moreIO NAME=moreIO ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-- ACTIVE_CSS_ROLE=hub EXPRESSION=(SERVER_LABEL co IOplus) [GRID]> crsctl modify serverpool ora.backup -attr "SERVER_CATEGORY=moreIO" -all_policies
  81. 81. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.81 Policy-Management with Oracle RAC 12c Adding Server Categories to the picture – comparison Before After [GRID]> crsctl status policy DayTime POLICY NAME=DayTime DESCRIPTION= SERVERPOOL NAME=Free ... SERVERPOOL NAME=ora.backoffice IMPORTANCE=0 MAX_SIZE=1 MIN_SIZE=1 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.backup IMPORTANCE=0 MAX_SIZE=0 MIN_SIZE=0 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.frontoffice IMPORTANCE=0 MAX_SIZE=3 MIN_SIZE=3 SERVER_CATEGORY= SERVER_NAMES= [GRID]> crsctl status policy DayTime POLICY NAME=DayTime DESCRIPTION= SERVERPOOL NAME=Free ... SERVERPOOL NAME=ora.backoffice IMPORTANCE=0 MAX_SIZE=1 MIN_SIZE=1 SERVER_CATEGORY= SERVER_NAMES= SERVERPOOL NAME=ora.backup IMPORTANCE=0 MAX_SIZE=0 MIN_SIZE=0 SERVER_CATEGORY=moreIO SERVER_NAMES= SERVERPOOL NAME=ora.frontoffice IMPORTANCE=0 MAX_SIZE=3 MIN_SIZE=3 SERVER_CATEGORY= SERVER_NAMES=
  82. 82. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.82 Policy-Management with Oracle RAC 12c Using Policy Sets and the effect of Server Categories vixen srvpool: frontofficecomet srvpool: backup dancer srvpool: backupdasher srvpool: backoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1 vixen srvpool: backofficecomet srvpool: frontoffice dancer srvpool: frontofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1
  83. 83. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.83 [GRID]> crsctl modify policyset –attr "LAST_ACTIVATED_POLICY=DayTime“ [GRID] srvctl status serverpool Server pool name: frontoffice Active Servers count: 3 Server pool name: backoffice Active Servers count: 1 Server pool name: backup Active Servers count: 0 [RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dasher,vixen,comet Service hrsvc is running on nodes: dancer Policy-Management with Oracle RAC 12c Using Policy Sets means changing policies on a push of a button raccdb1 vixen srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_3 dasher srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_4 comet srvpool: backup Oracle GI | HUB Oracle RAC raccdb1_1 dancer srvpool: backup Oracle GI | HUB Oracle RAC raccdb1_2 dancer srvpool: backoffice Oracle GI | HUB Oracle RAC raccdb1_2 comet srvpool: frontoffice Oracle GI | HUB Oracle RAC raccdb1_1
  84. 84. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.84 Policy-Management with Oracle RAC 12c Using Policy Sets – changing policies on the push of a button is fast … vixen srvpool: frontofficecomet srvpool: backup dancer srvpool: backupdasher srvpool: backoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1 vixen srvpool: backofficecomet srvpool: frontoffice dancer srvpool: frontofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1 [GRID]> date; crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=DayTime"; date Mon Sep 16 19:26:42 PDT 2013 CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded Mon Sep 16 19:26:43 PDT 2013 " Time to execute: 1 second!
  85. 85. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.85 Policy-Management with Oracle RAC 12c What-If evaluation of policy changes [RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend Stage Group 1: ------------------------------------------------------------------------------- Stage Number Required Action ------------------------------------------------------------------------------- 1 Y Server 'comet' will be moved from pools [ora.frontoffice] to pools [ora.backup] Y Server 'dancer' will be moved from pools [ora.frontoffice] to pools [ora.backup] Y Resource 'ora.raccdb1.crmsvc.svc' (1/1) will be in state [OFFLINE] Y Resource 'ora.raccdb1.crmsvc.svc' (2/1) will be in state [OFFLINE] 2 Y Resource 'ora.raccdb1.backup.svc' (1/1) will be in state [ONLINE|INTERMEDIATE] on server[comet] Y Resource 'ora.raccdb1.backup.svc' (2/1) will be in state [ONLINE|INTERMEDIATE] on server [dancer] vixen srvpool: frontofficecomet srvpool: frontoffice dancer srvpool: backofficedasher srvpool: frontoffice Oracle GI | HUB Oracle GI | HUB Oracle GI | HUBOracle GI | HUB Oracle RAC Oracle RAC Oracle RACOracle RAC raccdb1 raccdb1_3 raccdb1_4 raccdb1_2 raccdb1_1
  86. 86. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.86 What-If with Oracle RAC 12c What-If evaluation of policy changes – in various levels [RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend -admin -l 'serverpools' NAME = Free ACTIVE_SERVERS = NAME = Generic ACTIVE_SERVERS = NAME = ora.backoffice ACTIVE_SERVERS = vixen NAME = ora.backup ACTIVE_SERVERS = comet dancer NAME = ora.frontoffice ACTIVE_SERVERS = dasher [RAC]> srvctl status service -d raccdb1 Service backup is not running. Service crmsvc is running on nodes: dancer,vixen,dasher Service hrsvc is running on nodes: comet [GRID]> crsctl eval activate policy Weekend -admin -l 'resources' -------------------------------------------------------------------------------- Name Target State Server Effect -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.mgmtdb 1 ONLINE ONLINE dasher ora.raccdb1.backup.svc 1 ONLINE ONLINE comet Started 2 ONLINE ONLINE dancer Started ora.raccdb1.crmsvc.svc 1 ONLINE OFFLINE Stopped 2 ONLINE OFFLINE Stopped 3 ONLINE ONLINE dasher ora.raccdb1.db 1 ONLINE ONLINE comet 2 ONLINE ONLINE dancer 3 ONLINE ONLINE vixen 4 ONLINE ONLINE dasher ora.raccdb1.hrsvc.svc 1 ONLINE ONLINE vixen 2 ONLINE OFFLINE --------------------------------------------------------------------------------
  87. 87. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.87 Policy-Management with Oracle RAC 12c Information on each step on the way Before After [GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=NightTime“ CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'comet' succeeded [GRID]> crsctl modify policyset -attr "LAST_ACTIVATED_POLICY=DayTime" CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer' CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet' CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer' CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet' CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded
  88. 88. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.88

×