The document discusses moving an existing Oracle cluster to a Flex ASM architecture. It provides steps to convert the existing cluster mode to Flex Cluster mode using the ASM Configuration Assistant (ASMCA). This involves stopping and restarting ASM and other cluster resources on each node, and configuring hub and leaf nodes with direct or indirect access to shared storage respectively. The summaries show the conversion was successful, with ASM instances restarting automatically on different nodes and the database instance remaining available during a test where ASM was killed on one node. It also shows how to change a node's role between hub and leaf.

1. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
In Oracle Flex ASM architecture, Oracle ASM instances can run only a small
number of cluster nodes. These ASM instances are connected by all the
database instances on the cluster to provide storage access for all these
database instances. When an ASM instance fails, the database instances
reconnect to other ASM instances for the failed ASM instance. This features
supports the platform for variety of application which includes RAC with large
number of nodes. There is a new type of network, ASM network with Flex ASM
which is used for communication between ASM and its clients. ASM network is
accessible from all nodes. In oracle Flex Cluster, the nodes are arranged in two
ways i.e. Hub Node and Leaf Node.
Hub Nodes: tightly connected with each other like standard configuration and
have direct access to the shared storage. The maximum 64 hub nodes will be
configured in a cluster.
Leaf nodes: Leaf nodes can be added as many as more and can host different
applications other than hub nodes. This type of nodes doesn't require direct
access to shared storage instead they can request data from Hub nodes.
The Hub nodes can be run or configured as member of cluster without any leaf
nodes but leaf nodes must have Hub node to become member of cluster.
Check current ASM Status
[oracle@rac1 bin]$ ./crsctl get cluster mode status
Cluster is running in "standard" mode
[oracle@rac1 bin]$ ./srvctl status asm -detail
ASM is running on rac1,rac2
ASM is enabled.
[oracle@rac1 bin]$ ./srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
[oracle@rac1 bin]$ asmcmd showclustermode
ASM cluster : Flex mode disabled
1
Tested By - Monowar Mukul, OCM 11g DBA

2. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
oracle@rac1 bin]$ asmca
Select ASM Instance tab
2
Tested By - Monowar Mukul, OCM 11g DBA

3. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
click Convert to Oracle Flex ASM
click OK
3
Tested By - Monowar Mukul, OCM 11g DBA

4. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
4
Tested By - Monowar Mukul, OCM 11g DBA

5. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
Run the script on local node as a root user :
/u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
Output:
[root@rac1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/converttoFlexASM.sh
PRCR-1086 : resource ora.proxy_advm is already registered
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
:
:
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac1'
CRS-2676: Start of 'ora.storage' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
Cluster bounced in node rac1
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded
CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac1'
CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac1'
CRS-2676: Start of 'ora.scan2.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac1'
CRS-2676: Start of 'ora.scan3.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac2' succeeded
5
Tested By - Monowar Mukul, OCM 11g DBA

6. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
CRS-2677: Stop of 'ora.FRA.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2'
CRS-2677: Stop of 'ora.ons' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2'
:
:
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac2'
CRS-2676: Start of 'ora.storage' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
Cluster bounced in node rac2
Check the mode again
--------------------------Check the clustermode again:
-------------------------------------[oracle@rac1 bin]$ asmcmd showclustermode
ASM cluster : Flex mode enabled
Check Configuration
--------------------------[oracle@rac1 bin]$ ./srvctl config asm
ASM home: /u01/app/12.1.0/grid
Password file: +DATA/orapwASM
ASM listener: LISTENER
ASM instance count: ALL
Cluster ASM listener: ASMNET1LSNR_ASM
On First Node:
[oracle@rac1 bin]$ ps -ef|grep pmon
oracle 15633
1 0 07:41 ?
00:00:00 asm_pmon_+ASM1
oracle 15985
1 0 07:42 ?
00:00:00 apx_pmon_+APX1
oracle 21959 7848 0 08:00 pts/1 00:00:00 grep pmon
Check the instance status:
[oracle@rac1 bin]$ ./srvctl status database -d orcl
Instance orcl_1 is running on node rac2
I see instance is running on second node based on server policy.
Check on Second Node:
[oracle@rac2 ~]$ ps -ef | grep pmon
oracle 14453
1 0 07:43 ?
00:00:00 asm_pmon_+ASM2
6
Tested By - Monowar Mukul, OCM 11g DBA

7. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
oracle 17339
1 0 07:44 ?
00:00:00 apx_pmon_+APX2
oracle 20393
1 0 07:56 ?
00:00:00 ora_pmon_orcl_1
oracle 22522 22466 0 08:17 pts/1 00:00:00 grep pmon
Note: New +APX proxy instances are running on nodes for Flex ASM
Check Network status
------------------------------[oracle@rac2 ~]$ netstat -ntl | grep 1521
tcp
tcp
tcp
tcp
0
0
0
0
0
0
0
0
192.168.2.24:1521
192.168.2.12:1521
192.168.2.102:1521
192.168.0.102:1521
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
0.0.0.0:*
LISTEN
LISTEN
LISTEN
LISTEN
Now for testing I am going to kill ASM on cluster node 2
[oracle@rac2 ~]$ ps -ef | grep pmon
oracle 14453 1 0 07:43 ?
00:00:00 asm_pmon_+ASM2
oracle 17339 1 0 07:44 ?
00:00:00 apx_pmon_+APX2
oracle 19985 22466 0 08:30 pts/1 00:00:00 grep pmon
oracle 20393 1 0 07:56 ?
00:00:00 ora_pmon_orcl_1
[oracle@rac2 ~]$ kill -9 14453 17339
RAC database instance on node 2 is still running.
[oracle@rac2 ~]$ ps -ef | grep pmon
oracle 20393 1 0 07:56 ?
00:00:00 ora_pmon_orcl_1
oracle 32397 22466 0 08:31 pts/1 00:00:00 grep pmon
Check the ASM status
----------------------------[oracle@rac2 ~]$ crsctl stat res ora.asm -v
NAME=ora.asm
TYPE=ora.asm.type
LAST_SERVER=rac1
STATE=ONLINE on rac1
TARGET=ONLINE
CARDINALITY_ID=1
CREATION_SEED=223
RESTART_COUNT=0
FAILURE_COUNT=0
FAILURE_HISTORY=
ID=ora.asm 1 1
INCARNATION=1
LAST_RESTART=11/16/2013 07:38:40
LAST_STATE_CHANGE=11/16/2013 07:42:54
STATE_DETAILS=
INTERNAL_STATE=STABLE
INSTANCE_COUNT=2
LAST_SERVER=rac2
STATE=OFFLINE
TARGET=ONLINE
CARDINALITY_ID=2
CREATION_SEED=223
RESTART_COUNT=1
FAILURE_COUNT=0
FAILURE_HISTORY=
7
Tested By - Monowar Mukul, OCM 11g DBA

8. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
ID=ora.asm 2 1
INCARNATION=1
LAST_RESTART=NEVER
LAST_STATE_CHANGE=11/16/2013 08:31:03
STATE_DETAILS=
INTERNAL_STATE=STARTING
INSTANCE_COUNT=2
Check again
-------------------[oracle@rac2 ~]$ ps -ef | grep pmon
oracle 927 1 0 08:31 ?
00:00:00 asm_pmon_+ASM2
oracle 1842 1 0 08:31 ?
00:00:00 apx_pmon_+APX2
oracle 2783 22466 0 08:35 pts/1 00:00:00 grep pmon
oracle 20393 1 0 07:56 ?
00:00:00 ora_pmon_orcl_1
So ASM has restarted and the database instance has remained stable.
Flex Cluster Role Management
------------------------------------------
[root@rac1 bin]# ./crsctl status server rac1 -f
NAME=rac1
MEMORY_SIZE=3854
CPU_COUNT=1
CPU_CLOCK_RATE=2
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
SERVER_LABEL=
PHYSICAL_HOSTNAME=
STATE=ONLINE
ACTIVE_POOLS=Generic ora.TESTP
STATE_DETAILS=
ACTIVE_CSS_ROLE=hub
I am going to change the 1st node as a leaf node:
[root@rac1 bin]# crsctl set node role leaf [or hub]
CRS-4408: Node 'rac1' configured role successfully changed; restart Oracle High Availability Services for new role to
take effect.
# crsctl stop crs
# crsctl start crs
Then check again
----------------------
./crsctl status server rac1 -f
NAME=rac1
MEMORY_SIZE=3854
CPU_COUNT=1
CPU_CLOCK_RATE=2
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=leaf
RESOURCE_USE_ENABLED=1
SERVER_LABEL=
PHYSICAL_HOSTNAME=
STATE=ONLINE
ACTIVE_POOLS=Generic ora.TESTP
8
Tested By - Monowar Mukul, OCM 11g DBA

9. 12c ASM: Moving the Existing Cluster Mode to Flex Cluster
STATE_DETAILS=
ACTIVE_CSS_ROLE=leaf
9
Tested By - Monowar Mukul, OCM 11g DBA