• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Understanding Oracle RAC 11g Release 2 Internals
 

Understanding Oracle RAC 11g Release 2 Internals

on

  • 3,542 views

This is version covers Oracle RAC 11g Release 2 Internals only.

This is version covers Oracle RAC 11g Release 2 Internals only.

Statistics

Views

Total Views
3,542
Views on SlideShare
3,063
Embed Views
479

Actions

Likes
2
Downloads
586
Comments
0

2 Embeds 479

http://blog.ksdaemon.ru 451
http://www.linkedin.com 28

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Understanding Oracle RAC 11g Release 2 Internals Understanding Oracle RAC 11g Release 2 Internals Document Transcript

    • 1/20/20121 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Understanding Oracle RAC Internals Markus Michalewicz (Markus.Michalewicz@oracle.com) Senior Principal Product Manager Oracle RAC and Oracle RAC One Node2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 1
    • 1/20/2012 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Agenda• The Oracle RAC Architecture• VIPs and Networks• Listeners and SCAN – and Services• Client Connectivity• Node Membership• The Interconnect• Installation and Upgrade4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 2
    • 1/20/2012 The Oracle RAC Architecture Typical hardware architecture Network Public Lan Client Node 1 Node 2 Node N-1 Node N Interconnect with switch SAN switch Shared Storage5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The Oracle RAC Architecture Typical software deployment 11g Release 2 Oracle Real Application Clusters (RAC) Oracle Grid Infrastructure OS OS OS OS Shared Storage6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 3
    • 1/20/2012 The Oracle RAC Architecture Typical software deployment 11g Release 2 Oracle Real Application Clusters (RAC) Oracle Automatic Storage Management (ASM) Oracle Clusterware OS OS OS OS Oracle ASM managed Shared Storage7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The Oracle RAC Architecture Oracle RAC 11g Release 2 per node layer • Cloud File System (aka ASM Cluster File File System - ACFS) DB Instance • For all other files Oracle RAC • 11.2.0.3: For Backup Sets ASM Instance Oracle Grid Infrastructure • Oracle Clusterware files HA Framework OS OS • Voting Files (disks) Node Membership • Cluster Registry (OCR) OS • Database Files • Redo / Archive Logs Oracle ASM managed • Control Files Shared Storage • etc.8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 4
    • 1/20/2012 The Oracle RAC Architecture Oracle Grid Infrastructure 11g Release 2 process overview ASM Instance Oracle Grid Infrastructure HA Framework OS OS Node Membership • My Oracle Support (MOS) OS • Note 1053147.1 - 11gR2 Clusterware and Grid Home - What You Need to Know • Note 1050908.1 - How to Troubleshoot Grid Infrastructure Startup Issues 9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The Oracle RAC Architecture Local and Cluster Resources (11.2.0.2 list) LISTENER_SCAN1 LISTENER_SCAN2 LISTENER_SCAN3 ora.oc4j If database is deployed: ora.SCAN1.VIP ora.SCAN2.VIP ora.SCAN3.VIP ora.cvu • ora.Dbname.db ora.node1.vip ora.node2.vip ora.node3.vip ora.node4.vip • ora.Dbname.Srvcname.svc ora.asm ora.asm ora.asm ora.asm ora.dgname.dg ora.dgname.dg ora.dgname.dg ora.dgname.dg Command to list: ora.listener.lsnr ora.listener.lsnr ora.listener.lsnr ora.listener.lsnr • crsctl stat res -t ora.ons ora.ons ora.ons ora.ons ora.gsd ora.gsd ora.gsd ora.gsd ora.net1.network ora.net1.network ora.net1.network ora.net1.network ora.registry.acfs ora.registry.acfs ora.registry.acfs ora.registry.acfs Oracle Grid Infrastructure OS OS OS OS10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 5
    • 1/20/2012 VIPs and Networks11 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. VIPs and Networks Oracle VIPs • SCAN and node VIPs are cluster resources • SCAN VIPs can float in the cluster • Node VIPs per default run on specific nodes rac1 rac2 LISTENER_SCAN1 LISTENER_SCAN2 ora.SCAN1.VIP ora.SCAN2.VIP ora.rac1.vip ora.rac2.vip • All VIPs on the public network (SCAN, node, user or application VIPs) depend on the Oracle … … network resource. ora.listener.lsnr ora.listener.lsnr … … • The network resource monitors ora.net1.network ora.net1.network the public network interface(s). • IOCTL based checks vs. default gateway ping Oracle Grid Infrastructure OS OS • Default gateway checks as backup • racgvip for backward compatibility only12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 6
    • 1/20/2012 VIPs and Networks Node VIP Details [GRID]> crsctl status res ora.rac1.vip -p NAME=ora.rac1.vip TYPE=ora.cluster_vip_net1.type ...1 AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% ... rac1 rac2 CHECK_INTERVAL=1 … …2 CHECK_TIMEOUT=30 … … ... ora.rac1.vip ora.rac2.vip START_DEPENDENCIES=hard(ora.net1.network)3 pullup(ora.net1.network) … … ... ora.listener.lsnr ora.listener.lsnr STOP_DEPENDENCIES=hard(ora.net1.network) … … ... ora.net1.network ora.net1.network USR_ORA_VIP=rac1-vip VERSION=11.2.0.2.0 Oracle Grid Infrastructure OS OS13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. VIPs and Networks The network resource • The Oracle network resource monitors the public network interface(s) (e.g. eth0). • The default is network number 1 (net1) • There is one network resource per subnet: rac1 rac2 … … [GRID]> srvctl config network … … Network exists: ora.rac1.vip ora.rac2.vip 1/192.168.0.0/255.255.255.0/eth0, type static … … ora.listener.lsnr ora.listener.lsnr • New in 11.2.0.2: … … • Direct management of the network resource ora.net1.network ora.net1.network • srvctl verb network Oracle Grid Infrastructure • Network type support (static, dhcp, mixed) OS OS14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 7
    • 1/20/2012 VIPs and Networks How to enable multiple networks in the cluster?1 [GRID]> srvctl add network –h Adds a network configuration to the Oracle Clusterware. Usage: srvctl add network -k <net_num> network number (default number is 1) -S <subnet>/<netmask>/[if1[|if2...]] NET address spec for network -w <network_type> The network type (static, dhcp, mixed) … … ora.net1.network ora.net1.network -h Print usage ora.net2.network ora.net2.network -v Verbose output … … Oracle Grid Infrastructure • For example: srvctl add network -k 2 -S 10.1.1.0/255.255.255.0 OS OS15 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. VIPs and Networks How to enable multiple networks in the cluster?2 [GRID]> srvctl add vip -h Adds a VIP to the Oracle Clusterware. Usage: srvctl add vip -n <node_name> Node name -A <vip_name|ip>/<netmask>/[if1[|if2...]] ora.rac1.vip ora.rac2.vip VIP address spec for node applications ora.rac1-vip2.vip ora.rac2-vip2.vip -k <net_num> network number (default number is 1) … … -h Print usage ora.net1.network ora.net1.network ora.net2.network ora.net2.network -v Verbose output … … Oracle Grid Infrastructure • For example: srvctl add vip -n rac1 -k 2 -A rac1-vip2/255.255.255.0 OS OS srvctl add vip -n rac2 -k 2 -A rac2-vip2/255.255.255.016 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 8
    • 1/20/2012 VIPs and Networks Application / user and node VIPs1 [GRID]> crsctl stat res ora.rac1-vip2.vip -p NAME=ora.rac1-vip2.vip TYPE=ora.cluster_vip_net2.type ... HOSTING_MEMBERS=rac1 This is a node vip! ora.rac1.vip ora.rac2.vip ora.rac1-vip2.vip ora.rac2-vip2.vip ... START_DEPENDENCIES=hard(ora.net2.network) pullup(ora.net2.network) … … ora.net1.network ora.net1.network ... ora.net2.network ora.net2.network STOP_DEPENDENCIES=hard(ora.net2.network) … … ... Oracle Grid Infrastructure USR_ORA_VIP=rac1-vip2 OS OS VERSION=11.2.0.2.017 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. VIPs and Networks Application / user and node VIPs2 [GRID]> appvipcfg create -network=1 -ip 192.168.0.66 -vipname=myVIP -user=root This is an application vip! ora.rac1.vip ora.rac2.vip [GRID]> appvipcfg create ora.rac1-vip2.vip ora.rac2-vip2.vip -network=2 -ip 10.1.1.88 myVIP myNewVIP -vipname=myNewVIP … … -user=root ora.net1.network ora.net1.network ora.net2.network ora.net2.network … … Oracle Grid Infrastructure OS OS18 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 9
    • 1/20/2012 VIPs and Networks Application / user and node VIPs3 [GRID]> crsctl stat res myVIP -p NAME=myVIP TYPE=app.appvip_net1.type ... HOSTING_MEMBERS=rac1 This is an application vip! ora.rac1.vip ora.rac2.vip ora.rac1-vip2.vip ora.rac2-vip2.vip ... RESTART_ATTEMPTS=0 myVIP myNewVIP ... … … ora.net1.network ora.net1.network USR_ORA_VIP=192.168.0.66 ora.net2.network ora.net2.network VERSION=11.2.0.2.0 … … Oracle Grid Infrastructure • You can modify it as needed! OS OS19 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners and SCAN20 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 10
    • 1/20/2012 Listeners Listeners and dependencies • In Oracle RAC 11g Release 2 there have been three major changes regarding listeners: rac1 rac2 LISTENER_SCAN1 LISTENER_SCAN2 1. Listeners per default now run from the ora.SCAN1.VIP ora.SCAN2.VIP Oracle Grid Infrastructure (GRID) home ora.rac1.vip ora.rac2.vip 2. Listeners are managed by an Oracle agent … … ora.listener.lsnr ora.listener.lsnr 3. SCAN Listeners were introduced … … ora.net1.network ora.net1.network • A listener depends on a VIP Oracle Grid Infrastructure • A listener depends indirectly on a network OS OS21 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners The default LISTENER [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle  Grid Software Owner Home: <CRS home> rac1 rac2 End points: TCP:1521 … … ora.LISTENER.lsnr ora.LISTENER.lsnr … … ora.net1.network ora.net1.network Oracle Grid Infrastructure OS OS22 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 11
    • 1/20/2012 Listeners The default LISTENER FAQ • Can you define another port? - YES rac1 rac2 • Can you define another owner? – YES • Can you define another home? – YES, if really needed. … … ora.LISTENER.lsnr ora.LISTENER.lsnr … … • Can I add another listener on another port? - YES ora.net1.network ora.net1.network Oracle Grid Infrastructure OS OS23 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The default LISTENER Can you define another port? - YES1 [GRID]> srvctl modify listener -l LISTENER -p TCP:1545 [GRID]> srvctl config listener Name: LISTENER rac1 rac2 Network: 1, Owner: oracle Home: <CRS home> End points: TCP:1545 … … ora.LISTENER.lsnr ora.LISTENER.lsnr • Check: <CRS HOME>/network/admin/endpoints_listener.ora before: … … LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= ora.net1.network ora.net1.network (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip) (PORT=1521))(ADDRESS=(PROTOCOL=TCP) Oracle Grid Infrastructure (HOST=192.168.0.51)(PORT=1521)(IP=FIRST)))) OS OS # line added by Agent24 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 12
    • 1/20/2012 The default LISTENER Can you define another port? - YES2 [GRID]> srvctl modify listener -l LISTENER -p TCP:1545 [GRID]> srvctl config listener Name: LISTENER rac1 rac2 Network: 1, Owner: oracle Home: <CRS home> End points: TCP:1545 … … ora.LISTENER.lsnr ora.LISTENER.lsnr • Check: <CRS HOME>/network/admin/endpoints_listener.ora after restart: … … LISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= ora.net1.network ora.net1.network (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip) (PORT=1545))(ADDRESS=(PROTOCOL=TCP) Oracle Grid Infrastructure (HOST=192.168.0.51)(PORT=1545)(IP=FIRST)))) OS OS # line added by Agent25 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The default LISTENER Can you define another owner? - YES [GRID]> srvctl modify listener -l LISTENER -u root You need to be root… [GRID]> srvctl config listener Name: LISTENER rac1 rac2 Network: 1, Owner: root  For demonstration purposes only… Home: <CRS home> End points: TCP:1545 … … ora.LISTENER.lsnr ora.LISTENER.lsnr … … ora.net1.network ora.net1.network Oracle Grid Infrastructure OS OS26 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 13
    • 1/20/2012 The default LISTENER Can you define another home? - YES [GRID]> srvctl modify listener -l LISTENER -o /u01/app/oracle/product/11.2.0/dbhome_1 [GRID]> srvctl config listener Name: LISTENER rac1 rac2 Network: 1, Owner: oracle Home: /u01/app/oracle/product/11.2.0/dbhome_1 End points: TCP:1545 … … ora.LISTENER.lsnr ora.LISTENER.lsnr … … ora.net1.network ora.net1.network • This may not be required. Oracle Grid Infrastructure • Instead, create a new listener from a separate home, if needed and use the SRVCTL command as shown, but try using the GI home. OS OS27 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners Can I add another listener on another port? - YES [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle Home: <CRS home> rac1 rac2 End points: TCP:1521 Name: LISTENER2011 Network: 1, Owner: oracle … … Home: <CRS home> ora.LISTENER.lsnr ora.LISTENER.lsnr End points: TCP:2011 ora.LISTENERK2.lsnr ora.LISTENERK2.lsnr ora.net1.network ora.net1.network • MOS Note 220970.1 - RAC: Frequently Asked Questions Oracle Grid Infrastructure • How to use SCAN and node listeners with different ports? OS OS • Add additional node listeners to the system using different ports28 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 14
    • 1/20/2012 Listeners Can I add another listener on another network? - YES [GRID]> srvctl add listener -l ListenerK2 -p 1544 -k 2 [GRID]> srvctl config listener Name: LISTENER Network: 1, Owner: oracle rac1 rac2 Home: <CRS home> End points: TCP:1545 … … Name: LISTENERK2 ora.LISTENER.lsnr ora.LISTENER.lsnr Ora.LISTENERK2.lsnr ora.LISTENERK2.lsnr Network: 2, Owner: oracle ora.net1.network ora.net1.network Home: <CRS home> ora.net2.network ora.net2.network End points: TCP:1544 … … Oracle Grid Infrastructure • MOS Note 1063571.1 - How to Configure A Second OS OS Listener on a Separate Network in 11.2 Grid Infrastructure29 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners Remember – it’s just another listener …1 [GRID]> srvctl config listener Name: JUSTALISTENER Network: 1, Owner: root Home: <CRS home> rac1 rac2 End points: TCP:1522 LISTENER_SCAN1 LISTENER_SCAN2 ... ora.SCAN1.VIP ora.SCAN2.VIP ora.rac1.vip ora.rac2.vip [GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora … … JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= ora.LISTENER.lsnr ora.LISTENER.lsnr (ADDRESS=(PROTOCOL=IPC)(KEY=JUSTALISTENER)))) # line added by Agent … … ora.net1.network ora.net1.network [GRID]> vi /u01/app/11.2.0/grid/network/admin/endpoints_listener.ora JUSTALISTENER_RAC1=(DESCRIPTION=(ADDRESS_LIST= Oracle Grid Infrastructure (ADDRESS=(PROTOCOL=TCP)(HOST=rac1-vip)(PORT=1522)) (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.51)(PORT=1522)(IP=FIRST)))) OS OS30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 15
    • 1/20/2012 Listeners Remember – it’s just another listener …2 [GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10)  For demonstration purposes only… (KEY=JUSTALISTENER)))) # line added by Agent ... rac1 rac2 [GRID]> srvctl stop listener -l JustAListener LISTENER_SCAN1 LISTENER_SCAN2 ora.SCAN1.VIP ora.SCAN2.VIP [GRID]> srvctl start listener -l JustAListener ora.rac1.vip ora.rac2.vip [GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora … … ... ora.LISTENER.lsnr ora.LISTENER.lsnr [GRID]> vi /u01/app/11.2.0/grid/network/admin/listener.ora … … ora.net1.network ora.net1.network JUSTALISTENER=(DESCRIPTION=(ADDRESS_LIST= (ADDRESS=(PROTOCOL=IPC)(RATE_LIMIT=10) Oracle Grid Infrastructure (KEY=JUSTALISTENER)))) # line added by Agent OS OS31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. SCAN The basics and overview • In Oracle RAC 11g Release 2 SCAN Listeners were introduced SCAN • There is one SCAN setup per cluster • Read about SCAN: rac1 rac2 rac3 LISTENER_SCAN1 LISTENER_SCAN2 LISTENER_SCAN3 • http://www.oracle.com/technetwork/database/ ora.SCAN1.VIP ora.SCAN2.VIP ora.SCAN3.VIP clustering/overview/scan-129069.pdf … … … • MOS Note 887522.1 – … … … 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained ora.LISTENER.lsnr ora.LISTENER.lsnr ora.LISTENER.lsnr … … … ora.net1.network ora.net1.network ora.net1.network • SCAN consists of a bundle containing: 1. SCAN IP Oracle Grid Infrastructure OS OS OS 2. Associated (dependency) SCAN Listener32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 16
    • 1/20/2012 SCAN The SCAN bundle1 • 3 SCANs are the default for HA and LB • Regardless of number of nodes SCAN • You can define less or more, if really needed. • SCAN listener resources run with rac1 rac2 rac3 an active dispersion dependency LISTENER_SCAN1 LISTENER_SCAN2 LISTENER_SCAN3 • If you use more nodes in the cluster than ora.SCAN1.VIP ora.SCAN2.VIP ora.SCAN3.VIP … … … SCAN listeners are defined, no node should run more than one SCAN bundle at a time … … … ora.LISTENER.lsnr ora.LISTENER.lsnr ora.LISTENER.lsnr • If you use less nodes in the cluster than … … … SCAN listeners are defined, there will be ora.net1.network ora.net1.network ora.net1.network nodes running more than one SCAN bundle at a time. Oracle Grid Infrastructure • SCAN VIP moves with the listener, if possible. OS OS OS 33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. SCAN The SCAN bundle2 [GRID]> srvctl config scan SCAN name: cluster1, Network: 1/192.168.0.0/255.255.255.0/eth0 SCAN VIP name: scan1, IP: /cluster1.us.oracle.com/192.168.0.41 SCAN rac1 rac2 [GRID]> srvctl modify scan LISTENER_SCAN1 … Modifies the SCAN name. ora.SCAN1.VIP … … … Usage: srvctl modify scan … … -n <scan_name> Domain name qualified SCAN name ora.LISTENER.lsnr ora.LISTENER.lsnr -h Print usage … … ora.net1.network ora.net1.network • http://www.oracle.com/technetwork/database/ Oracle Grid Infrastructure clustering/overview/scan-129069.pdf page 2 OS OS 34 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 17
    • 1/20/2012 SCAN The SCAN bundle3 [GRID]> srvctl add scan -h Adds a SCAN VIP to the Oracle Clusterware. Usage: srvctl add scan -n <scan_name> SCAN -n <scan_name> Domain name qualified SCAN name rac1 rac2 -k <net_num> network number (default number is 1) LISTENER_SCAN1 … -S <subnet>/<netmask>/[if1[|if2...]] ora.SCAN1.VIP … NET address spec for network … … -h Print usage … … ora.LISTENER.lsnr ora.LISTENER.lsnr … … • Note: SCAN can only operate on one network / in one subnet. ora.net1.network ora.net1.network Oracle Grid Infrastructure OS OS35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. SCAN The SCAN FAQ1 • From MOS Note 220970.1 – RAC: Frequently Asked Questions • How to use SCAN and node listeners with different ports? SCAN • With Oracle RAC 11g Release 2 using SCAN is the default. rac1 rac2 • As with other listeners, there is no direct communication LISTENER_SCAN1 … between the node (listeners) and the SCAN listeners. ora.SCAN1.VIP … … … • Listeners are only aware of the instances and services served, since the instances (PMON) register themselves and the services … … they host with the listeners. ora.LISTENER.lsnr ora.LISTENER.lsnr … … • The instances use the LOCAL and REMOTE Listener ora.net1.network ora.net1.network parameters to know with which listeners to register. • Listeners used for a client connection to Oracle RAC should be Oracle Grid Infrastructure managed by Oracle Clusterware and should be listening on an OS OS Oracle managed VIP.36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 18
    • 1/20/2012 SCAN The SCAN FAQ2 • Can you define another port? - YES • See MOS Note 220970.1 – RAC: Frequently Asked Questions SCAN • How to use SCAN and node listeners with different ports? rac1 rac2 • Use srvctl modify scan_listener –p <newPort> LISTENER_SCAN1 … • Can you define another owner? – NO ora.SCAN1.VIP … … … • Can you define another home? – NO … … ora.LISTENER.lsnr ora.LISTENER.lsnr • Can you have more than one node listener with SCAN? – YES … … • Can the SCAN and the node listener ports differ? - YES ora.net1.network ora.net1.network • See also MOS Note 972500.1 – How to Oracle Grid Infrastructure Modify SCAN Setting or SCAN Listener Port after Installation OS OS 37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners and Instance Registration Listener changes may require instance changes • See MOS Note 220970.1 – RAC: Frequently Asked Questions • How to use SCAN and node listeners with different ports? SCAN • Whenever you change an instance parameter configuration rac1 rac2 a change to the LOCAL_LISTENER or REMOTE_LISTENER DB Instance DB Instance per database (instance) may be required. LISTENER_SCAN1 … • Note that using server-side TNSANMES aliases to ora.SCAN1.VIP … resolve the listener configuration is not recommended. … … • The use of the TNSNAMES connector string is the default: ora.LISTENER.lsnr ora.LISTENER.lsnr … … local_listener = ‘(DESCRIPTION= (ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011)) Oracle Grid Infrastructure )) OS OS 38 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 19
    • 1/20/2012 Listeners and Instance Registration EXAMPLE1 • Taken from MOS Note 220970.1 – RAC: FAQ • How to use SCAN and node listeners with different ports? • Add additional node listeners to the system using different ports SCAN • srvctl add listener -l LISTENER2011 -p 2011 rac1 rac2 • srvctl start listener -l LISTENER2011 DB Instance DB Instance • srvctl status listener -l LISTENER2011 LISTENER_SCAN1 … • lsnrctl status LISTENER2011 ora.SCAN1.VIP … … … [GRID]> srvctl config listener ora.LISTENER.lsnr ora.LISTENER.lsnr Name: LISTENER2011 … … Network: 1, Owner: oracle Oracle Grid Infrastructure Home: <CRS home> OS OS End points: TCP:201139 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners and Instance Registration EXAMPLE2 • Taken from MOS Note 220970.1 – RAC: FAQ • How to use SCAN and node listeners with different ports? • Add additional node listeners to the system using different ports SCAN rac1 rac2 • New node listener does not serve any database instance DB Instance DB Instance • You must alter the LOCAL_LISTENER parameters LISTENER_SCAN1 … for each instance accordingly: ora.SCAN1.VIP … alter system set local_listener =‘ … … (DESCRIPTION= (ADDRESS_LIST= ora.LISTENER.lsnr ora.LISTENER.lsnr (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.61)(PORT=2011)))) … … scope=BOTH SID=OCRL1; alter system register; Oracle Grid Infrastructure OS OS40 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 20
    • 1/20/2012 Listeners and Instance Registration EXAMPLE3 • Taken from MOS Note 220970.1 – RAC: FAQ • How to use SCAN and node listeners with different ports? • Add additional node listeners to the system using different ports SCAN rac1 rac2 • Note: Use spaces between the various segments of the command DB Instance DB Instance • Reason: the database agent in Oracle Clusterware currently LISTENER_SCAN1 … determines whether the local_listener or remote_listener have ora.SCAN1.VIP … been manually set by a string comparison operation. … … • If the string looks unaltered it will reset these parameters ora.LISTENER.lsnr ora.LISTENER.lsnr with the default values that it determines on instance startup. … … • Slight modifications are therefore required. ER 11772838 Oracle Grid Infrastructure has been filed to allow for a more convenient mechanism. OS OS41 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners and Instance Registration Using listeners in different subnets / networks • SCAN can only operate in one network / subnet rac1 rac2 DB Instance DB Instance • If listeners in different subnets are used, LISTENER_SCAN1 … • use LISTENER_NETWORKS: ora.SCAN1.VIP … http://download.oracle.com/docs/cd/E11882_01 … … … … /server.112/e17110/initparams115.htm#REFRN10313 ora.LISTENER.lsnr ora.LISTENER.lsnr • Note: Listeners specified by the LISTENER_NETWORKS Ora.LISTENERK2.lsnr ora.LISTENERK2.lsnr parameter should not be used in the LOCAL_LISTENER and ora.net1.network ora.net1.network REMOTE_LISTENER parameters. Otherwise, cross registration will happen and connections will be redirected cross networks. ora.net2.network ora.net2.network … … Oracle Grid Infrastructure • For more information, see also MOS Note 1063571.1 – How to Configure A Second OS OS Listener on a Separate Network in 11.2 Grid Infrastructure42 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 21
    • 1/20/2012 Listeners, SCAN and Services43 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Listeners and Service Registration A summary • Services depend on a database • Per default, the database instance will register all services it hosts with all listeners it registers with. SCAN rac1 rac2 • Services operate within the following boundaries: DB Instance DB Instance • Network (indirectly via dependency) ora.Dbname.db ora.Dbname.db • Server Pools (in policy-managed database environments) ora.Dbname.Srvcname.svc ora.Dbname.Srvcname.svc … … ora.LISTENER.lsnr ora.LISTENER.lsnr • Currently there is no support for service failover … … • Between Server Pools Oracle Grid Infrastructure • Between networks OS OS44 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 22
    • 1/20/2012 Services and Dependencies Dependency on DB and Server Pool [GRID]> srvctl config service -d ORCL -s MyService1 Service name: MyService Service is enabled SCAN Server pool: ORCL rac1 rac2 Cardinality: 1 Disconnect: false DB Instance DB Instance Service role: PRIMARY ora.ORCL.db ora.ORCL.db ora.ORCL.MyService.svc ora.Dbname.Srvcname.svc Management policy: AUTOMATIC … … ... ora.LISTENER.lsnr ora.LISTENER.lsnr Preferred instances: ORCL_1 … … Available instances: Oracle Grid Infrastructure OS OS45 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Services and Dependencies Start / Stop dependencies [GRID]> crsctl stat res ora.orcl.myservice.svc -p2 ... DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=service) PROPERTY(SERVICE_NAME=%GEN_SERVICE_NAME%) SCAN PROPERTY(DB_UNIQUE_NAME=CONCAT(PARSE(%NAME%, ., rac1 rac2 2), STAT(ora.orcl.db, USR_ORA_DOMAIN), .)) ELEMENT(INSTANCE_NAME=STAT(ora.orcl.db, DB Instance DB Instance GEN_USR_ORA_INST_NAME)) ora.ORCL.db ora.Dbname.db ... ora.ORCL.MyService.svc ora.Dbname.Srvcname.svc START_DEPENDENCIES=hard(ora.orcl.db,type:ora.cluster … … _vip_net1.type) weak(type:ora.listener.type) ora.LISTENER.lsnr ora.LISTENER.lsnr pullup(type:ora.cluster_vip_net1.type) … … pullup:always(ora.orcl.db) dispersion(type:ora.service.type) Oracle Grid Infrastructure STOP_DEPENDENCIES=hard(intermediate:ora.orcl.db,inte OS OS rmediate:type:ora.cluster_vip_net1.type) ...46 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 23
    • 1/20/2012 Client Connectivity47 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity Direct or indirect connect • Connect Time Load Balancing (CTLB) • Connect Time Connection Failover (CTCF) • Runtime Connection Load Balancing (RTLB) • Runtime Connection Failover (RTCF) BATCH Production Email SCAN Connection Pool48 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 24
    • 1/20/2012 Client Connectivity Connect Time Connection Failover jdbc:oracle:thin:@MySCAN:1521/Email PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email))) BATCH Production Email MySCAN Connection Pool49 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity Runtime Time Connection Failover PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email) ...)) BATCH Production Email MySCAN Connection Pool50 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 25
    • 1/20/2012 Client Connectivity Runtime Time Connection Failover PMRAC = (DESCRIPTION = (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email) (FAILOVER_MODE= (TYPE=select)(METHOD=basic)(RETRIES=180)(DELAY=5)))) BATCH Production Email MySCAN ? Connection Pool51 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity More information • If problems occur, see: • Note 975457.1 – How to Troubleshoot Connectivity Issues with 11gR2 SCAN Name • For more advanced configurations, see: • Note 1306927.1 – Using the TNS_ADMIN variable and changing the default port number of all Listeners in an 11.2 RAC for an 11.2, 11.1, and 10.2 Database ?? BATCH Production Email MySCAN ? Connection Pool52 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 26
    • 1/20/2012 Client Connectivity Two ways to protect the client 1. Transparent Application Failover (TAF) 2. Fast Application Notification (FAN) • Tries to make the client unaware of a failure • FAN wants to inform clients ASAP • Provides means of CTCF and RTCF • Client can react to failure asap • Allows for pure ‘selects’ (reads) to continue • Expects clients to re-connect on failure (FCF) • Write transactions need to be re-issued • Sends messages about changes in the cluster • The Application needs to be TAF aware BATCH Production Email MySCAN ? Connection Pool53 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity and Service Definition Define settings on the server • HA (and LB) settings [GRID]> srvctl config service -d ORCL -s MyService can be defined per service Service name: MyService • Clients connecting to the service will ... adhere to the settings considering the DTP transaction: false client used. AQ HA notifications: false Failover type: NONE BATCH Failover method: NONE Production Email MySCAN TAF failover retries: 0 TAF failover delay: 0 Connection Load Balancing Goal: LONG Runtime Load Balancing Goal: NONE TAF policy specification: BASIC54 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 27
    • 1/20/2012 Client Connectivity Use a FAN aware connection pool1• If a connection pool is used • The clients (users) get a physical connection to the connection pool • The connection pool creates a physical connection to the database • It is a direct client to the database • Internally the pool maintains logical connections BATCH Production Connection Pool Email MySCAN55 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity Use a FAN aware connection pool2• The connection pool • Invalidates connections to one instance • Re-establishes new logical connections • May create new physical connections • Prevent new clients to be misrouted • The application needs to handle the transaction failure that might have occurred. BATCH Production Connection Pool Email MySCAN56 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 28
    • 1/20/2012 Client Connectivity The Load Balancing (LB) cases • Connect Time Load Balancing (CTLB) • Runtime Connection Load Balancing (RTLB) • On the Client Side • On the Server Side BATCH Production MySCAN Email Connection Pool57 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Client Connectivity Connect Time Load Balancing (CTLB) – on the client side PMRAC = (DESCRIPTION = (FAILOVER=ON)(LOAD_BALANCE=ON) (ADDRESS = (PROTOCOL = TCP)(HOST = MySCAN)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = Email))) BATCH Production Email MySCAN Connection Pool58 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 29
    • 1/20/2012 Client Connectivity Connect Time Load Balancing (CTLB) – on the server side • Traditionally, PMON dynamically registers the services to the specified listeners with: • Service names for each running instance of the database and instance names for the DB • The listener is updated with the load information for every instance and node as follows: • 1-Minute OS Node Load Average all 30 secs. • Number of Connections to Each Instance • Number of Connections to Each Dispatcher BATCH Production Email MySCAN Connection Pool59 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. ? Client Connectivity Use FAN for the Load Balancing cases • Connect Time Load Balancing (CTLB) • Connect Time Connection Failover (CTCF) • Runtime Connection Load Balancing (RTLB) RAC Database • Runtime Connection Failover (RTCF) 30% connections I’m busy Instance1 10% connections I’m very busy I’m idle Instance2 60% connections Instance360 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 30
    • 1/20/2012 Client Connectivity Use FAN for the Load Balancing cases • Connect Time Load Balancing (CTLB) • Runtime Connection Load Balancing (RTLB) • Also via AQ (Advanced Queuing) based notifications • Background is always the Load Balancing Advisory RAC Database 30% connections I’m busy Instance1 10% connections MySCAN • For more information, see: I’m very busy • Oracle® Real Application I’m idle Instance2 Clusters Administration and Deployment Guide 11g Release 2: 5 Introduction to Automatic Workload Management 60% connections Instance361 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Node Membership62 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 31
    • 1/20/2012 Oracle Clusterware Architecture Node Membership Processes and Basics Public Lan Public Lan Main processes involved: Private Lan / Interconnect • CSSD (ora.cssd) Oracle Clusterware • CSSDMONITOR • was: oprocd CSSD CSSD CSSD • now: ora.cssdmonitor SAN SAN Network Network Voting Disk63 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle Clusterware Architecture What does CSSD do? • Monitors nodes using 2 communication channels: – Private Interconnect  Network Heartbeat – Voting Disk based communication  Disk Heartbeat • Evicts (forcibly removes nodes from a cluster) nodes dependent on heartbeat Oracle Clusterware feedback (failures) CSSD “Ping” CSSD “Ping”64 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 32
    • 1/20/2012 Oracle Clusterware Architecture Interconnect basics – network heartbeat • Each node in the cluster is “pinged” every second • Nodes must respond in css_misscount time (defaults to 30 secs.) – Reducing the css_misscount time is generally not supported • Network heartbeat failures “Ping” will lead to node evictions – CSSD-log: [date / time] [CSSD][1111902528] clssnmPollingThread: node mynodename CSSD CSSD (5) at 75% heartbeat fatal, removal in 6.770 seconds65 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle Clusterware Architecture Voting Disk basics – disk heartbeat1 • Each node in the cluster “pings” (r/w) the Voting Disk(s) every second • Nodes must receive a response in (long / short) diskTimeout time – IF I/O errors indicate clear accessibility problems  timeout is irrelevant • Disk heartbeat failures will lead to node evictions – CSSD-log: … [CSSD] [1115699552] >TRACE: clssnmReadDskHeartbeat: CSSD CSSD node(2) is down. rcfg(1) wrtcnt(1) LATS(63436584) Disk lastSeqNo(1) “Ping”66 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 33
    • 1/20/2012 Oracle Clusterware Architecture Voting Disk basics – Structure2 • Voting Disks contain dynamic and static data: – Dynamic data: disk heartbeat logging – Static data: information about the nodes in the cluster • With 11.2.0.1 Voting Disks got an “identity”: – E.g. Voting Disk serial number: [GRID]> crsctl query css votedisk 1. 2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA] Node information Disk Heartbeat Logging • Voting Disks must therefore not be copied using “dd” or “cp” anymore67 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle Clusterware Architecture Voting Disk basics – Simple Majority rule3 • Oracle supports redundant Voting Disks for disk failure protection • “Simple Majority Rule” applies: – Each node must “see” the simple majority of configured Voting Disks at all times in order not to be evicted (to remain in the cluster)  trunc(n/2+1) with n=number of voting disks configured and n>=1 CSSD CSSD “Ping”68 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 34
    • 1/20/2012 Oracle Clusterware Architecture Simple Majority rule – in extended clusters • http://www.oracle.com/goto/rac – Using standard NFS to support a third voting file for extended cluster configurations (PDF) CSSD CSSD • Same principles apply • Voting Disks are just geographically dispersed69 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle Clusterware Architecture Voting Disks in Oracle ASM does not change their usage [GRID]> crsctl query css votedisk 1. 2 1212f9d6e85c4ff7bf80cc9e3f533cc1 (/dev/sdd5) [DATA] 2. 2 aafab95f9ef84f03bf6e26adc2a3b0e8 (/dev/sde5) [DATA] 3. 2 28dd4128f4a74f73bf8653dabd88c737 (/dev/sdd6) [DATA] Located 3 voting disk(s).• Oracle ASM auto creates 1/3/5 Voting Files – Voting Disks reside in one diskgroup only – Based on Ext/Normal/High redundancy and on Failure Groups in the Disk Group – Per default there is one failure group per disk – ASM will enforce the required number of disks – New failure group type: Quorum Failgroup70 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 35
    • 1/20/2012 Oracle Clusterware Architecture Oracle Cluster Registry (OCR) placement in Oracle ASM • The OCR is managed like a datafile in ASM (new type) • It adheres completely to the redundancy settings for the diskgroup (DG) • There can be more than one OCR location in more than one DG (DG:OCR  1:1) • Recommendation is 2 OCR locations, 1 in DATA, 1 in FRA for example71 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Oracle Clusterware Architecture Backup of Clusteware files is fully automatic (11.2+) • Clusterware Files (managed in ASM) enables fully Automatic Backups: • The Voting Disks are backed up into the OCR • Any configuration change in the cluster (e.g. node addition) triggers a new backup of the Voting Files. • A single, failed Voting Disks is restored by ASM automatically within a Disk Group – no action required • Note: Do not use DD to back up the Voting Disks anymore! • The OCR is backed up automatically every 4 hours • Manual Backups can be taken as required • ONLY IF all Voting Disks are corrupted or failed AND (all copies of) the OCR are also corrupted or unavailable THEN manual interference would be required – the rest is automatic.72 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 36
    • 1/20/2012 Fencing Basics Why are nodes evicted? • Evicting (fencing) nodes is a preventive measure (it’s a good thing)! • Nodes are evicted to prevent consequences of a split brain: – Shared data must not be written by independently operating nodes – The easiest way to prevent this is to forcibly remove a node from the cluster 1 2 CSSD CSSD73 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Fencing Basics How are nodes evicted? – STONITH • Once it is determined that a node needs to be evicted, – A “kill request” is sent to the respective node(s) – Using all (remaining) communication channels • A node (CSSD) is requested to “kill itself”  “STONITH like” – “STONITH” foresees that a remote node kills the node to be evicted 1 2 CSSD CSSD74 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 37
    • 1/20/2012 Fencing Basics EXAMPLE: Network heartbeat failure • The network heartbeat between nodes has failed – It is determined which nodes can still talk to each other – A “kill request” is sent to the node(s) to be evicted  Using all (remaining) communication channels  Voting Disk(s)  A node is requested to “kill itself”; executer: typically CSSD 1 2 CSSD CSSD 275 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Fencing Basics What happens, if CSSD is stuck? • A node is requested to “kill itself” • See also: MOS note • BUT CSSD is “stuck” or “sick” (does not execute) – e.g.: 1050693.1 - – CSSD failed for some reason Troubleshooting 11.2 – CSSD is not scheduled within a certain margin Clusterware Node Evictions (Reboots)  OCSSDMONITOR (was: oprocd) will take over and execute 1 2 CSSD CSSDmonitor CSSD 2 CSSD76 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 38
    • 1/20/2012 Fencing Basics How can nodes be evicted? • Oracle Clusterware 11.2.0.1 and later supports IPMI (optional) – Intelligent Platform Management Interface (IPMI) drivers required • IPMI allows remote-shutdown of nodes using additional hardware – A Baseboard Management Controller (BMC) per cluster node is required 1 CSSD CSSD77 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Fencing Basics EXAMPLE: IPMI based eviction on heartbeat failure • The network heartbeat between the nodes has failed – It is determined which nodes can still talk to each other – IPMI is used to remotely shutdown the node to be evicted 1 CSSD78 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 39
    • 1/20/2012 Fencing Basics Which node gets evicted? • Voting Disks and heartbeat communication is used to determine the node • In a 2 node cluster, the node with the lowest node number should survive • In a n-node cluster, the biggest sub-cluster should survive (votes based) 1 2 CSSD CSSD79 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Fencing Basics Cluster members can escalate a kill request • Cluster members (e.g Oracle RAC instances) can request Oracle Clusterware to kill a specific member of the cluster • Oracle Clusterware will then attempt to kill the requested member Oracle RAC Oracle RAC DB Inst. 1 DB Inst. 2 Oracle Clusterware Inst. 1: kill inst. 280 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 40
    • 1/20/2012 Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued, which leads to the eviction of the node, on which the particular member currently resides Oracle RAC Oracle RAC DB Inst. 1 DB Inst. 2 Oracle Clusterware Inst. 1: kill inst. 281 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued, which leads to the eviction of the node, on which the particular member currently resides Oracle RAC Oracle RAC DB Inst. 1 DB Inst. 2 Oracle Clusterware Inst. 1: kill inst. 282 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 41
    • 1/20/2012 Fencing Basics Cluster members can escalate a kill request • Oracle Clusterware will then attempt to kill the requested member • If the requested member kill is unsuccessful, a node eviction escalation can be issued, which leads to the eviction of the node, on which the particular member currently resides Oracle RAC DB Inst. 1 Oracle Clusterware83 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Re-Bootless Node Fencing With 11.2.0.2 onwards, fencing may not mean re-boot • Until Oracle Clusterware 11.2.0.2, fencing meant “re-boot” • With Oracle Clusterware 11.2.0.2, re-boots will be seen less, because: – Re-boots affect applications that might run an a node, but are not protected – Customer requirement: prevent a reboot, just stop the cluster – implemented... App X App Y RAC DB RAC DB Inst. 1 Inst. 2 CSSD Oracle Clusterware CSSD84 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 42
    • 1/20/2012 Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less: – Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • It starts with a failure – e.g. network heartbeat or interconnect failure App X App Y RAC DB RAC DB Inst. 1 Inst. 2 CSSD Oracle Clusterware CSSD85 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less: – Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • Then IO issuing processes are killed; it is made sure that no IO process remains – For a RAC DB mainly the log writer and the database writer are of concern App X App Y RAC DB Inst. 1 CSSD Oracle Clusterware CSSD86 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 43
    • 1/20/2012 Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less: – Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • Once all IO issuing processes are killed, remaining processes are stopped – IF the check for a successful kill of the IO processes, fails → reboot App X App Y RAC DB Inst. 1 CSSD Oracle Clusterware CSSD87 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less: – Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • Once all remaining processes are stopped, the stack stops itself with a “restart flag” App X App Y RAC DB Inst. 1 Oracle Clusterware CSSD OHASD88 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 44
    • 1/20/2012 Re-Bootless Node Fencing How it works… • With Oracle Clusterware 11.2.0.2, re-boots will be seen less: – Instead of fast re-booting the node, a graceful shutdown of the stack is attempted • OHASD will finally attempt to restart the stack after the graceful shutdown App X App Y RAC DB Inst. 1 Oracle Clusterware CSSD OHASD89 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Re-Bootless Node Fencing EXCEPTIONS • With Oracle Clusterware 11.2.0.2, re-boots will be seen less, unless…: – IF the check for a successful kill of the IO processes fails → reboot – IF CSSD gets killed during the operation → reboot – IF cssdmonitor (oprocd replacement) is not scheduled → reboot – IF the stack cannot be shutdown in “short_disk_timeout”-seconds → reboot App X App Y RAC DB RAC DB Inst. 1 Inst. 2 CSSD Oracle Clusterware CSSD90 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 45
    • 1/20/2012 The Interconnect91 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The Interconnect Heartbeat and “memory channel” between instances Network Public Lan Node 1 Node 2 Node N-1 Node N Client Interconnect with switch SAN switch92 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 46
    • 1/20/2012 The Interconnect Redundant Interconnect Usage1 • Redundant Interconnect Usage can be used as a bonding alternative – It works for “private networks” only; the nodeVIPs use a different approach – It enables HA and Load Balancing for up to 4 NICs per server (on Linux / Unix) – It can be used by Oracle Databases 11.2.0.2 and Oracle Clusterware 11.2.0.2 – It uses so called HAIPs that are assigned to the private networks on the server – The HAIPs will be used by the database and ASM instances and processes Node 1 Node 2 HAIP1 HAIP3 HAIP2 HAIP493 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. The Interconnect Redundant Interconnect Usage2 • A multiple listening endpoint approach is used – The HAIPs are taken from the “link-local” (Linux / Unix) IP range (169.254.0.0) – To find the communication partners, multicasting on the interconnect is required – With 11.2.0.3 Broadcast is a fallback alternative (BUG 10411721) – Multicasting is still required on the public lan for MDNS for example. – Details in My Oracle Support (MOS) Note with Doc ID 1212703.1: 11.2.0.2 Grid Infrastructure Install or Upgrade may fail due to Multicasting Node 1 Node 2 HAIP1 HAIP3 HAIP2 HAIP494 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 47
    • 1/20/2012 The Interconnect Redundant Interconnect Usage and the HAIPs • If a network interface fails, the assigned HAIP is failed over to a remaining one. • Redundant Interconnect Usage allows having networks in different subnet • You can either have one subnet for all networks or a different one for each • You can also use VLANs with the interconnect. For more information see: • Note 1210883.1 - 11gR2 Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip • Note 220970.1 - RAC: Frequently Asked Questions - How to use VLANs in Oracle RAC? AND Are there any issues for the interconnect when sharing the same switch as the public network by using VLAN to separate the network? Node 1 Node 2 HAIP1 HAIP3 HAIP1 HAIP2 HAIP4 HAIP395 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Installation and Upgrade96 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 48
    • 1/20/2012 Installation Patch Sets are also more than BUG fixe with 11.2.0.x • Upgrade + New Features • Upgrade + Full Installation • And (always) “Out-of-place” • Patch uses new Oracle Home • First patch set: 11.2.0.2 • 11.2.0.3 works the same • Less new features97 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Installation Installation is meant to be simpler with 11g Release 2 • Oracle Universal Installer (OUI) is the main installation instrument. • Installation tips and tricks in: • MOS note 810394.1 (generic) • See: RAC Platform Specific Starter Kits and Best Practices • Step by Step for various versions available. • See also: RAC Platform Generic Load Testing and System Test Plan Outline • For an Oracle VM based deployment see: • Oracle RAC OVM Templates on OTN • Details in MOS note 1185244.1 – Pre-Built Grid Infrastructure and Oracle RAC Templates For Oracle VM98 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 49
    • 1/20/2012 Upgrade and Update Upgrade to 11g Release 2 (11.2.0.3) recommended • The Upgrade to 11g Release 2 from previous versions is generally recommended • Things to consider: • MOS note 1312225.1 - Things to Consider Before Upgrading to 11.2.0.2 Grid Infrastructure • MOS note 1320966.1 - Things to Consider Before Upgrade to 11.2.0.2 Database Performance • MOS note 1363369.1 – Things to Consider Before Upgrading to 11.2.0.3 Grid Infrastructure/ASM99 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. 0 50