Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper

  1. 1. ManageabilityCCONFIGURINGONFIGURING OORACLERACLE EENTERPRISENTERPRISE MMANAGERANAGER CCLOUDLOUD CCONTROLONTROL 1212CCFORFOR HHIGHIGH AAVAILABILITYVAILABILITYLeighton L. Nelson, MercyABSTRACTOracle Enterprise Manager Cloud Control 12c provides a complete infrastructure management solution for databases,applications and hardware. Being such a key component in the enterprise naturally leads to concerns about redundancy andhigh availability. If something does go wrong within the configuration, it should be able to be recovered in the shortestpossible time, thus minimizing disruptions to manageability and monitoring of the enterprise infrastructure. It is with thesepoints in mind that Oracle Enterprise Manager has been designed to be able to meet high service and operating levels usingdifferent high availability mechanisms.TARGET AUDIENCEThis paper is for anyone who needs to implement high availability solutions for Enterprise Manager Cloud Control 12c.Typically this will be database administrators or Enterprise Manager administrators.EXECUTIVE SUMMARYUsers should be able to understand the following concepts in this paper:1. Understand the different levels of high availability configurations for Enterprise Manager Cloud Control 12c2. Implement a level 1,2 or 3 high availability configuration3. Implement a backup/recovery solution for Enterprise Manager Cloud Control 12cBACKGROUNDThe following article will outline how to configure various HA configurations for Oracle Enterprise Manager using one of thepre-defined MAA levels. We will also look at how to backup and recover the environment to maintain high availability.TECHNICAL DISCUSSIONS AND EXAMPLESOracle has defined four levels of high availability for Enterprise Manager Cloud Control. These are summarized in Table 1below in order of increased cost and complexity. Only levels 1 to 3 will be covered in this paper.Table 1. Levels of Enterprise Manager High Availability.Description Load Balancer RequiredMin No. of NodesRecommended No. ofNodesCost1 OMS and Repository on separatehosts.No redundancy.No1 2$2 OMS installed on shared storage withVIP used in active/passive failover.Repository Database using local DataGuard.No2 4$$3 Multiple OMSs in active/activeconfiguration. Repository using RACYes , At primary site3 5$$$1 Session # 326
  2. 2. ManageabilityDatabase with local Data Guard4 Primary OMS in active/activeconfiguration with RAC Repositorydatabase. Standby OMS at DR site inactive/active configuration. StandbyRAC database at DR siteYes. At pimary andstandby sites4 8$$$$Protection levels and components protected.OMS Host Failure OMS Storage Failure Database Host Failure Database Storage Failure Site Failure/Disaster RecoveryLevel1No No No No NoLevel2Yes No Yes Yes NoLevel3Yes Yes Yes Yes NoLevel4Yes Yes Yes Yes YesMANAGEMENT REPOSITORYThe repository is the persistent store for monitoring data uploaded by agents. It stores metrics and configuration data from allmonitored targets. Loss of the repository will result in management server failure. It is recommended to enable database highavailability features such as Real Application Clusters (RAC) or RAC One Node, Automatic Storage Management (ASM) fordatafile storage (depending on redundancy levels and backend RAID configuration) and Data Guard for the Repository.Additionally, the Management Service should be configured to use the Single Client Access Name (SCAN) and a non-defaultRAC service name if the repository is configured as a RAC database.Figure 1 shows an example of an OMS configured with a RAC repository database using the SCAN name. For additionaldetails on configuring SCAN see the Oracle® Clusterware Administration and Deployment Guide 11g Release 2 (11.2).Figure 1. Repository database configured using SCAN2 Session # 326
  3. 3. ManageabilityA physical standby database is recommended to provide disaster recovery in case of a failure at the primary site. Data on theprimary repository will be kept in sync with the standby database. When configuring a physical standby for the repositorydatabase, use similar hardware and resources as the primary site so that there aren’t any performance implications in the eventof a failover/switchover. Use Enterprise Manager to create a standby database from the primary database.MANAGEMENT SERVICEThe Management Service or OMS provides an interface to users via the Enterprise Manager Console and processes data fromagents. A loss of the OMS will result in a complete Enterprise Manager outage – Agent upload, jobs, incidents andnotifications will all stop to function. The Oracle Weblogic Nodemanager and the Oracle Process Management andNotification Service (OPMN) will attempt to restart the Management Service automatically if it is down. While this providessome benefit, it will not protect the OMS if the host is down. At a minimum, the OMS and repository should be installed onseparate hosts if possible. Multiple OMSs can be deployed behind a server load balancer to provide protection against a singlehost being down. Also, you can opt to install the OMS on a shared filesystem, which will provide cold failover in case of theloss of a single host. We will look at each of these options for protecting the OMS in further detail.LEVEL 1 – SEPARATE OMS AND REPOSITORY HOSTS/NO REDUNDANCYA level 1 configuration is comprised of a single OMS and repository with each installed on its own host. It provides the leastprotection, as failure of any of these hosts will result in a complete outage of the Enterprise Manager system. Considerationshould be given to the proximity between the OMS and repository as high network latency between the two can have anegative impact on performance. This is recommended for small environments (number of targets < 1000 and number ofagents < 100).Figure 2 below is a diagram of a Level 1 High Availability configuration. Agents upload directly to the management servicehost, while users interact with the OMS via the Enterprise Manager console or the command line EMCLI directly to thephysical OMS host. If either the management service or repository database hosts become unavailable, there will be acomplete outage resulting in loss of monitoring for targets. Keeping each component on its own server however, reduces theoverhead of resources from impacting each other. For example, increasing the database parameters sga_max_size and sga_targetcould cause a negative impact on the performance of the OMS since it reduces the amount of memory available to theoperating system. Likewise, changes to the JVM heap size will also impact the resources available for the management server.In addition, it lays the basis for a scalable environment as business requirements dictate.3 Session # 326
  4. 4. ManageabilityFigure 2. Level 1 High Availability Configuration with OMS and Repository on separate serversLEVEL 2 - ACTIVE/PASSIVE OMS AND DATA GUARD REPOSITORYIn order to reduce OMS downtime during planned or unplanned outage, some redundancy should be introduced into theconfiguration. A level 2 configuration uses a shared filesystem for the management service to achieve an active/passive orcold failover cluster solution. The filesystem is shared between two or more hosts and is only active on one host at a time.The steps below should be performed as a pre-requisite to a level 2 high availability configuration.The shared filesystem for the OMS can be installed on a general-purpose cluster file system including NFS, Oracle ClusterFile System (OCFS2), and Oracle Automatic Storage Management (ASM) Cluster File System (ACFS). If NFS is used as theshared storage, then ensure the correct mount options are set in /etc/fstab (/etc/filesystems on AIX) to prevent potentialI/O issues. Specifically, the rsize and wsize should be set.The example below shows and entry in the /etc/fstab file on a Linux server where the NFS share is mounted on a filernamed filer1 under the /vol1/oms_share directory.filer:/vol1/oms_share /u01/app/oms_share nfs rw,bg,rsize=32768,wsize=32768,hard,nointr,tcp,noac,vers=3,timeo=600 0 0• Binaries for the OMS along with the inventory should be installed on the shared filesystem.• Setup the virtual hostname and IP address (VIP) using Oracle Clusterware or third-party solutions. Failover isachieved by using the virtual hostname for the OMS along with a unique IP address, which resolves to the hostname.• Configure the repository database using local physical standby with Data Guard (see Figure 5).4 Session # 326
  5. 5. ManageabilityFigure 3. Level 2 Enterprise Manager Cloud Control 12c High Availability Configuration with Active/Passive OMS and local standby.In our example, we will use Oracle Clusterware to create and manage the virtual hostname. OCFS2 is chosen as the sharedfilesystem.Note OCFS v1 is not supported as shared storage for the OMS.The following prerequisites should be configured on all hosts before installing the Management Service.The OS user id should be identical on each host where the OMS will be installed.• Environment variables should be set• Operating System Time Zone variable TZ .export TZ=America/New_York• PERL5LIB variable should be unset to avoid being associated with the incorrect PERL libraries• Install Oracle Clusterware on both servers (see Oracle Grid Infrastructure Installation Guide)• Install and configure OCFS2• Prepare Database for use with OMSBefore installing the OMS, a virtual hostname that maps to a unique static IP address should be available (which means an IPaddress that is currently not used, in the same subnet as the other Enterprise Manager components). A VIP is configured onthe public subnet used for accessing the server. If the server that hosts the VIP goes down, Oracle Clusterware relocates it toan available member of the cluster. Likewise, if maintenance needs to be performed on a server hosting the VIP, it can bemanually relocated to another server in the cluster.You should use the appvipcfg utility in Oracle Clusterware 11gR2 to create application VIPs. The VIP is created with a set ofpre-defined settings suitable for an application VIP such placement policy and the failback option. See Oracle ClusterwareAdministration and Deployment Guide11gR2 documentation for details on using appvipcfg.5 Session # 326
  6. 6. ManageabilityUsing the steps below, create an application VIP for the OMS.1. After installing Oracle Clusterware 11gR2, login as the root user and issue the following command:GRID_HOME/bin/appvipcfg create -network=1 -ip= -vipname=omsvip -user=rootAn example output of running appvipcfg is as follows:[root@oms1 bin]# /u01/app/11.2.0/grid/bin/appvipcfg create -network=1 -ip= -vipname=omsvip -user=rootProduction Copyright 2007, 2008, Oracle.All rights reserved2012-10-28 03:30:29: Creating Resource Type2012-10-28 03:30:29: Executing /u01/app/11.2.0/grid/bin/crsctl add typeapp.appvip_net1.type -basetype ora.cluster_vip_net1.type -file/u01/app/11.2.0/grid/crs/template/appvip.type2012-10-28 03:30:29: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add typeapp.appvip_net1.type -basetype ora.cluster_vip_net1.type -file/u01/app/11.2.0/grid/crs/template/appvip.type2012-10-28 03:30:37: Create the Resource2012-10-28 03:30:37: Executing /u01/app/11.2.0/grid/bin/crsctl add resource omsvip -typeapp.appvip_net1.type -attr"USR_ORA_VIP=,START_DEPENDENCIES=hard(,STOP_DEPENDENCIES=hard(,ACL=owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x,,APPSVIP_FAILBACK="2012-10-28 03:30:37: Executing cmd: /u01/app/11.2.0/grid/bin/crsctl add resource omsvip-type app.appvip_net1.type -attr"USR_ORA_VIP=,START_DEPENDENCIES=hard(,STOP_DEPENDENCIES=hard(,ACL=owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x,,APPSVIP_FAILBACK="This creates a VIP on network 1 which is defined as IP address The VIP name is omsvip and it it owned by theroot user.2. Allow the Oracle Grid Infrastructure software owner (e.g. grid) to run the script to start the VIP. As root execute thefollowing:GRID_HOME/bin/crsctl setperm resource omsvip -u user:grid:r-x3. Start the VIP as the grid user:GRID_HOME/bin/crsctl start resource omsvipExample:[grid@oms1 ~]$ $GRID_HOME/bin/crsctl start resource omsvipCRS-2672: Attempting to start omsvip on oms1CRS-2676: Start of omsvip on oms1 succeeded4. Check the status of the VIPGRID_HOME/bin/crsctl status resource omsvipNAME=omsvipTYPE=app.appvip_net1.typeTARGET=ONLINESTATE=ONLINE on oms15. The virtual hostname is defined in DNS, and should resolve to the application VIP address created using the stepsabove. Check if the virtual hostname and VIP are resolvable using nslookup or the dig command.$ nslookup omsvipThis should resolve to a unique IP address of the virtual hostname on every node in the cluster.6. Also do a reverse lookup of the IP address.6 Session # 326
  7. 7. Manageability$nslookup <virtual IP address>7. Verify that the IP address returned from the nslookup output is running on the OMS host.ifconfig –a|grep <virtual IP address>8. Once we have verified that the virtual hostname and VIP have been configured correctly we can then proceed withthe OMS installation. The following should be performed before starting the installer:a. Create a new ORACLE_HOME for the OMS on the shared storage on all nodes in the cluster$ mkdir –p /u01/app/oms_shareb. Create an Oracle Inventory directory under the ORACLE_HOME for the OMS on all nodes$ mkdir /u01/app/oms_share/oraInventoryc. Create the inventory pointer in the oraInventory directory$ vi oraInst.locThe oraInst.loc should contain path to the inventory and the group of the software owner for the OMS. For example:inventory_loc=/u01/app/oracle/oms_share/oraInventoryinst_group=oinstall9. Next, we proceed with the installation by specifying the ORACLE_HOSTNAME environment variable as our virtualhostname and pointing to the shared inventory location.runInstaller -invPtrloc /u01/app/oms_share/oraInst.loc –debug10. Install the OMS on the first host by following the installation steps as described in the Oracle Enterprise ManagerCloud Control 12c Basic Installation Guide. You only need to complete the installation once. Since the location is shared, thebinaries will be accessible from another host that shares the filesystem.Once the OMS has been successfully installed and is up and running, if the host were to go down then the VIP will beautomatically relocated to another node. The management service can then be manually started on any remaining node in thecluster on which the VIP is running.1. To manually relocate the VIP to another host in the cluster issue the following command.crsctl relocate res omsvip2. Check if the IP address associated with the VIP is running on the relocated hostifconfig –a|grep <vip>The repository database should be reachable from other hosts in the cluster and the listener should be up and running.3. Set the ORACLE_HOSTNAME environment variable to the virtual hostname. Continuing with our example we usethe command below.export ORACLE_HOSTNAME=omsvip.example.com4. Start the OMS on the new node$OMS_HOME/bin/emctl start omsAlternatively, Oracle Clusterware can be configured to fully manage the OMS by creating start, check, stop, clean and abortroutines that tell it how to operate on the OMS. Details on this configuration are outside the scope of this chapter. See OracleClusterware 11gR2 Administration Guide for details.With a cold-failover solution in place for the OMS, you are protected from the failure of a single host. However, time toperform failover could range from a few minutes to hours, depending on whether it is done manually or automated usingOracle Clusterware or other methods. The repository also needs to be protected, as it is now a single point of failure. Asmentioned earlier, a local Data Guard setup consisting of a single physical standby is highly recommended. The standbydatabase should be configured on a separate host from the management servers and primary database. However, it may bepossible to create both the primary and the physical standby on another OMS host to keep costs down. In the event of aplanned or unplanned outage of the primary repository, the physical standby can be switched or failed over to the standby ona remaining host. Please note that the host will now become a single point of failure.7 Session # 326
  8. 8. ManageabilityAs a pre-requisite to creating a standby database using Enterprise Manager, the destination host should have an EnterpriseManager Agent installed and should be monitored by Enterprise Manager. Also, if ASM is used as database storage, it shouldbe monitored along with the listener.See Enterprise Manager Administration Guide for details on creating a standby database.In addition, it is recommended to use Data Guard Broker to manage the Data Guard operations such as failover, switchoverand health checks. The Data Guard Broker simplifies the management of databases in such a configuration by providing botha GUI interface via Enterprise Manager or the command line utility Data Guard Broker Line Manager dgmgrl. A fulldiscussion of Data Guard and Data Guard Broker is outside the scope of this chapter. See the Oracle Data Guard Conceptsand Administration 11g Release2 (11.2) documentation for details. In order to manage role change operations for theEnterprise Manager Repository, the latter method should be used, as the Enterprise Manager Cloud Control 12c systemwould not be available to complete the operations.It is also possible to configure the Management Service so that no configuration changes are required once the repositorydatabase changes role i.e. during switchover/failover. Using Oracle Database 11gR2 you can configure services using thesrvctl command line utility that will only be active when the database is assuming the primary role. The example belowillustrates the steps for configuring a database service in a Data Guard configuration for use with the OMS.1. Create a database service called emrepsrvc on the primary database emrepprim for use with OMS.$ srvctl add service -d emrepprim -s emrepsrvc -l PRIMARY -q FALSE -e NONE -m NONE -w 0-z 02. Verify the configuration of the service$ srvctl config service -d emreptst -s emrepsrvcService name: emrepsrvcService is enabledCardinality: SINGLETONDisconnect: falseService role: PRIMARYManagement policy: AUTOMATICDTP transaction: falseAQ HA notifications: falseFailover type: NONEFailover method: NONETAF failover retries: 0TAF failover delay: 0Connection Load Balancing Goal: LONGRuntime Load Balancing Goal: NONETAF policy specification: NONEEdition:3. Start and check the status of the service on the primary$ srvctl start service -d emreptst -s emrepsrvc$ srvctl status service -d emreptst -s emrepsrvcService emrepsrvc is running4. Create the same service on the standby database for use when the role changes to a primary.$ srvctl add service -d emrepsby -s emrepsrvc -l PRIMARY -q FALSE -e NONE -m BASIC -w 0-z 0$ srvctl config service -d emrepsby -s emrepsrvcService name: emrepsrvcService is enabledCardinality: SINGLETONDisconnect: falseService role: PRIMARYManagement policy: AUTOMATICDTP transaction: falseAQ HA notifications: falseFailover type: NONE8 Session # 326
  9. 9. ManageabilityFailover method: BASICTAF failover retries: 0TAF failover delay: 0Connection Load Balancing Goal: LONGRuntime Load Balancing Goal: NONETAF policy specification: NONEEdition:5. Reconfigure the OMS to use the new service in the connect descriptor:$OMS_HOME/bin/emctl config oms -store_repos_details -repos_conndesc(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost2)(PORT=1521)))(CONNECT_DATA=( -repos_user sysmanOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Enter Repository Users Password :Successfully updated datasources and stored repository details in Credential Store.If there are multiple OMSs in this environment, run this store_repos_details command onall of them.And finally, restart all the OMSs using emctl stop oms -all and emctl start oms.6. Stop and Restart the OMSa. To stop the OMS, use emctl stop oms -all:$emctl stop oms -allOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Stopping WebTier...WebTier Successfully StoppedStopping Oracle Management Server...Oracle Management Server Successfully StoppedAdminServer Successfully StoppedOracle Management Server is Downb. To restart, use emctl start oms:$emctl start omsOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Starting Oracle Management Server...Starting WebTier...WebTier Successfully StartedOracle Management Server Successfully StartedOracle Management Server is UpVerify that the new DG connection string is in use$emctl config oms -list_repos_detailsOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Repository Connect Descriptor : (DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost2)(PORT=1521)))(CONNECT_DATA=( User : sysmanLEVEL 3 - ACTIVE/ACTIVE OMS WITH SLB AND RAC DATA GUARD REPOSITORYIn the previous section, we determined that there would be some downtime while the OMS is failed over to another host.Some environments cannot tolerate such downtime, and as such an increased level of availability is required. Fortunately, thiscan be achieved by using multiple management servers coupled with a RAC database as a repository. RAC provides both highavailability and scalability for the database. One may also consider an active/passive RAC One Node configuration.Additionally, a local physical standby is used to protect the database in the event of a database storage failure. The9 Session # 326
  10. 10. ManageabilityManagement Services are located behind a Server Load Balancer (SLB). The SLB then directs traffic from the EnterpriseManager Console and Management Agents to an available OMS. Each management server can be installed on separate hostsfrom the RAC nodes (see Figure 4). However, you may need to balance the costs of having separate servers and the level ofprotection needed for such as configuration. The management servers and repository databases should be in close proximityto each other to reduce network latency. This may dictate that the management servers and RAC database instances co-existon the servers.Figure 4. Level 3 High Availability with multiple OMSs configured behind a server load balancer and a RAC Database Management Repository.This level of availability also provides the ability to scale based on business requirements. More OMS servers can be added toscale out while nodes can be added to RAC database to scale the repository.The steps required in setting up a level 3 high availability configuration are listed below.1. Install Primary OMS with RAC Repository2. Configure SLB3. Add Repository Targets4. Configure Software Library5. Add Second OMS6. Create Standby Database7. Convert Standby Database to RACAfter an initial installation of the first OMS, the agents and users connect via the physical hostname:$OMS_HOME/bin/emctl status oms -detailsOracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 OracleCorporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :Console Server Host : oem1.example.comHTTP Console Port : 7790HTTPS Console Port : 780310 Session # 326
  11. 11. ManageabilityHTTP Upload Port : 4890HTTPS Upload Port : 4904EM Instance Home : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1OMS Log Directory Location : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/logOMS is not configured with SLB or virtual hostnameAgent Upload is locked.OMS Console is locked.Active CA ID: 1Console URL: URL: Domain InformationDomain Name : GCDomainAdmin Server Host: oem1.example.comManaged Server InformationManaged Server Instance Name: EMGC_OMS1Managed Server Instance Host: oem1.example.comWebTier is UpOracle Management Server is UpWhen a Server Load Balancer is configured, agent and console traffic is directed to multiple Management Services by the SLB.The SLB configuration should be done after the installation of the first OMS.Server Load Balancer ConfigurationThe configuration of the SLB may vary depending on the manufacturer of the device. However, there are a number ofrequirements for the SLB. These are listed below.• Virtual server ports - Secure Upload, Agent Registration, Secure Console, Unsecure Console. If the OMS is onlyconfigured for secure upload and agent traffic, then only the Secure Upload and Secure Console virtual server portsare required.• Persistence – HTTP and HTTPS traffic between the Enterprise Manager console and OMS require persistence or“stickiness” so that requests sent to one OMS don’t switch to another during the same session. Not havingpersistence could result in users constantly having to log in as their sessions are handled by different managementservices.• Application monitoring – Checks the health of the management services so that requests aren’t sent to one that iseither unavailable or degraded.In addition to the above requirements, some devices “may also require additional settings such as F5 BIG-IP TCP Profiles.Table 2 shows an example management port configuration using only secure upload and agent registration ports.Table 2. Management Service PortsCloudControlServiceTCP Port Monitor Name Persistent Pool Name LoadBalancingVirtual ServerNameVirtualServer PortSecureUpload4899 mon_ccsu4899 None pool_cssu4899 RoundRobinvs_ccsu4899 4899SecureConsole7803 mon_cssc7803 Source IP pool_cssc7803 RoundRobinvs_cssc7803 44311 Session # 326
  12. 12. ManageabilityThe above configuration assumes that the secure upload port was configured using port 4899 and the secure console port wasconfigured using port 7803. The ports are configured during the OMS installation. Verify that you provide the same portsused during the Enterprise Manager Cloud Control 12c installation. A virtual hostname and IP address are also registered inDNS to allow clients to connect to the SLB.Next we need to create the following items.• Pools – group of two or more OMS servers that are load balanced, with each pool running a different managementservice• Health Monitors – determines that the service is running and available to accept connections• Virtual Servers – a unique IP address and port that represents a pool of serversOnce the SLB has been configured, the next step is to configure the OMS to use the SLB. To do this we need to resecure theManagement Service to regenerate the certificate.emctl secure oms –sysman_pwd <sysman_password> -reg_pwd <agent_reg_password> -host<virtualhostname> -secure_port 4899 -slb_port 4899 -slb_console_port 443 -console –lock_consoleThe example below illustrates securing the OMS with the SLB virtual hostname using the HTTPS Upload using port 4904and HTTPS Console port 443. The console is also locked to prevent non-HTTPS traffic from accessing it.$OMS_HOME/bin/emctl secure oms -sysman_pass -reg_pwd regpass -host -secure_port 4904 -slb_port 4904 -slb_console_port 443 -reset -console -lock_consoleOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.Securing OMS... Started.Securing OMS... SuccessfulRestart OMSAfter running the above commands, the OMS will need to be restarted.emctl stop oms –allemctl start omsTo verify that the OMS has been successfully secured, issue the commandThe output of the command above should indicate that SLB or virtual hostname as well as the ports for the SLB HTTPSupload and HTTPS Console.$ $OMS_HOME/bin/emctl status oms -detailsOracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 OracleCorporation. All rights reserved.Enter Enterprise Manager Root (SYSMAN) Password :Console Server Host : oem1.example.comHTTP Console Port : 7790HTTPS Console Port : 7803HTTP Upload Port : 4890HTTPS Upload Port : 4904EM Instance Home : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1OMS Log Directory Location : /u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log12 Session # 326
  13. 13. ManageabilitySLB or virtual hostname: slb.example.comHTTPS SLB Upload Port : 4904HTTPS SLB Console Port : 443Agent Upload is locked.OMS Console is locked.Active CA ID: 1Console URL: URL: Domain InformationDomain Name : GCDomainAdmin Server Host: oem1.example.comManaged Server InformationManaged Server Instance Name: EMGC_OMS1Managed Server Instance Host: oem1.example.comWebTier is UpOracle Management Server is UpUsers can now access the server using the virtual hostname Port 443 is the default HTTPS port soit is not necessary to specify the port number. If another port is selected as the secure port, it should be specified as part ofthe URL. Any agents that were previously deployed and configured to upload to the physical hostname of the OMS will berequired to be resecured also. Use the following command to resecure the agents.emctl secure agent –emdWalletSrcUrl securing the agent check the status to verify that it is uploading to the SLB Upload port by checking theREPOSITORY_URL property.Likewise, if EMCLI was previously configured it also needs to be reconfigured to use the SLB.emcli setup –url= –username=sysmanADD ADDITIONAL OMSThe next step is to install additional management services. This can either be done using the Add Additional OMSDeployment Procedure (recommended) or using Silent Mode. See the Oracle Enterprise Manager Cloud Control AdvancedInstallation and Configuration guide for details on using Silent Mode Installation.The deployment procedure simplifies the process of deploying additional management servers to meet high availabilityrequirements. It automates the steps required to prepare and install additional management services by collecting input via awizard driven interface. It will clone the existing Middleware home including the OMS configuration based on the collectedinput. Any additional servers should also meet the same requirements for installing an OMS. See the Chapter 8 of the OracleEnterprise Manager Cloud Control Basic Installation Guide for pre-requisites for the additional Management Service.Note Any new servers that are intended to be used as an OMS should already have the agent deployed. The deploymentprocedures will not clone an agent to the target.1. From the Enterprise Manager console go to Enterprise, Provisioning and Patching and select Procedure Library (seeFigure 5).13 Session # 326
  14. 14. ManageabilityFigure 5 Add Procedure Library2. Select the check box next to the Add Management Service procedure and click Launch (see Figure 6). It is possible tosort the columns by the name of the Procedure by clicking if it is not visible in the list.Figure 6. Add Management Service Deployment Procedure3. The next step will list the pre-requisites that should be met before continuing with the procedure (see Figure 7).Select all the check boxes associated with the pre-requisites already met. If any are not met you should resolve them firstbefore continuing with the processors. Click Next when done.14 Session # 326
  15. 15. ManageabilityFigure 7. Add Oracle Management Service Pre-requisite checks4. On the Select Destination screen, choose a managed host that will be used as the additional OMS (see Figure 8). Thepath for the Middleware home will be identical as that on the first OMS. Enter a path for the Instance base location or acceptthe defaults. Choose or enter the credentials of the user who owns the software on both the source and destination. 8. Add OMS, Select Destination5. Next we will choose how to transfer the cloned home from the source OMS host to the destination host and selectthe staging locations (see Figure 9). The staging locations should have a minimum of 4GB free. If a shared filesystem such asNFS is selected no staging locations are required. For FTP and HTTP(S) options, there also needs to be 8GB of free space inthe operating system temporary location (/tmp in Unix or Linux or C:Temp in Windows). See the chapter on AddingAdditional Oracle Management Service of the Oracle Enterprise Manager Cloud Control Basic Installation Guide for detailson specifying the temporary location. In addition, we need to provide the ports to be used for the destination managementservice. They will default to the source management service ports. It is recommended to keep the ports on all managementservices identical if possible.15 Session # 326
  16. 16. ManageabilityFigure 9. Add OMS, Options6. On the Post Creation Steps screen, shown in Figure 10, it will list the requirements for setting up the SLB to includethe new OMS host if it has not been done as yet. The script will also be required to be executed. Optionally, you mayprovide an email address to receive post installation steps.Figure 10. Add OMS, Post Creation Steps7. You will be presented with a summary screen where you can review your inputs (see Figure 11). Click Finish to createthe deployment procedure job. This could take a long time to run depending on your hardware resources.16 Session # 326
  17. 17. ManageabilityFigure 11. Add OMS, ReviewIf any step fails you should review it and perform the necessary corrective actions before resuming or retrying. If an emailaddress was provided for the post-installation steps notification, you will be provided with steps to configure the SLB with thenewly added OMS and execute the script.The following new targets are discovered automatically in the Enterprise Manager console:• Oracle WebLogic Server• Oracle Web Tier• Application deployments• Oracle Management Service• Oracle Management Agent• Enterprise Manager Cloud Control hostAny other existing targets on the host should be promoted via Add Manual Target or Auto Discovery Results.After the additional management service has been configured, the next step in the high availability configuration involvescreating a local physical standby database for the repository using Data Guard. The Oracle MAA architecture recommends aRAC physical standby, but a single instance standby can be used as well. The standby database will protect against storage,media corruption or any incident, which results in a loss of the primary database.The standby database can be added using the same procedure as described in the level 2 high availability section. UsingEnterprise Manager Cloud Control, it is not possible to create a RAC standby. However, using deployment procedures, asingle instance standby database can be converted to a RAC standby database.BACKUPIn a highly available Enterprise Manager Cloud Control configuration, the three components – OMS, Repository and Agent—are individually configured to reduce downtime. While it is recommended that the highest level of availability beimplemented, the cost and required resources associated may be prohibitive. Also, in any IT framework, failure of17 Session # 326
  18. 18. Manageabilitycomponents is likely to happen over time. As such, no high availability architecture is complete without discussing backupand recovery strategies.Backing up the Enterprise Manager ecosystem requires a solution that encompasses the repository, OMS, agents and softwarelibrary. Each component is individually backed up based on its available methods. The management repository database storesmetrics and data provided by agents. It should be backed up according to recommended backup strategies for Oracledatabases using RMAN.REPOSITORY BACKUPThe first step in configuring a backup strategy for the repository database is to ensure that it is in archivelog mode. This willallow consistent online backups to be taken and also facilitates point-in-time recovery of the database. Hot backups should betaken regularly using recommended backup strategies. Database high availability features such as the fast recovery area andflashback database should also be enabled to allow for faster recovery of the database in the event of a failure. Backups canbe configured using Enterprise Manager Cloud Control console. Consider using Enterprise Manager’s Recommended BackupStrategy option to backup the repository database.Using Oracle Enterprise Manager Cloud Control to schedule the backup job also enables you to get notifications about thejob status as well as view backup information including run times and time of last backup.ORACLE MANAGEMENT SERVICE BACKUPThe Oracle Management Service is responsible for communicating with Management Agents and uploading data to theManagement Repository. It includes the Oracle Fusion Middleware home, the OMS Home and the WebTier (OHS) OracleHome in addition to Plug-in Homes. The OMS filesystem should be backed up periodically to preserve any changes such aspatches, updates or configuration changes. Use the exportconfig command to backup the OMS instance home(MW_HOME/gc_inst) including the Weblogic Server, WebTier and Administration Server. To create a backup of theinstance and all sub-components, use the following syntax.emctl exportconfig oms [-sysman_pwd <sysman password>][-dir <backup dir>] Specify directory to store backup file[-oms_only] Specify OMS-only backup on Admin Server host[-keep_host] Specify to backup hostname if no slb defined(Use this option only if recovery will be doneon machine that responds to this hostname)The following example backs up the OMS configuration to a shared directory. An archive is created with a .bka extension.$OMS_HOME/bin/emctl exportconfig oms -sysman_pwd oracle12c -dir /mnt/backupOracle Enterprise Manager Cloud Control 12c Release 2Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.ExportConfig started...Machine is Admin Server host. Performing Admin Server backup...Exporting emoms properties...Exporting secure properties...Exporting configuration for pluggable modules...Preparing archive file...Backup has been written to file: /mnt/backup/opf_ADMIN_20121104_224806.bkaThe export file contains sensitive data.Please ensure that it is kept secure.ExportConfig completed successfully!In addition to backing up the OMS configuration, a backup of the Oracle inventory and output of the opatch lsinventory –detail command should also be performed.The example below generates the opatch inventory output and writes it to a shared filesystem.$OMS_HOME/OPatch/opatch lsinventory –detail > /mnt/backups/opatch_bkup.log18 Session # 326
  19. 19. ManageabilityThe backup of the OMS can either be done manually or scheduled via the Enterprise Manager Cloud Control Job system.MANAGEMENT AGENT BACKUPThe Management agents are stateless so no backups are necessary. Instead, a reference gold image of the agent should becreated for all platforms being monitored. Any patches and customizations should be included in the gold configuration. Inthe event that a Management Agent is lost, it should be installed from the reference gold image.RECOVERYPerforming regular backups of each Enterprise Manager Cloud Control component is important. However, of even moresignificance is the ability to restore the components successfully if needed. Enterprise Manager Recovery may require recoveryof several components including the repository, management service and agents depending on the nature of the problem. Forexample, recovering the management service in a scenario where the host is lost may require additional steps compared withrestoring the management service to the original host. We will examine these scenarios below.REPOSITORY RECOVERYThe Cloud Control console will be unavailable in the event that the OMS is down. To restore the repository database, theRMAN command line utility should be used. Using RMAN you can restore the database using either a full recovery or apoint-in-time recovery (see Oracle Database Backup and Recovery Guide). In the case of the latter, you may need toresynchronize agents that are out of sync with the repository. To resynchronize the agent using emctl use the following syntax.emctl resync repos (-full|-agentlist "agent names") [-name "resync name"] [-sysman_pwd "sysman password"]This command should be executed after the management repository has been restored and before starting the OMS.The steps required to make Enterprise Manager Cloud Control operational after restoring the repository database will varydepending on whether a full or incomplete recovery is performed as well as if the recovery was done on a the same host or adifferent host. In each scenario, the repository database should be recovered after stopping the OMS.OMS Reconfiguration is required whenever the repository database is recovered on a different host. This stores the newdatabase connection description in the OMS. We use the following command to reconfigure the repository database in theOMS.emctl config oms -store_repos_details (-repos_host <host> -repos_port <port> -repos_sid <sid> | -repos_conndesc<connect descriptor>) -repos_user <username> [-repos_pwd <pwd>]1. Stop all the OMSsemctl stop oms2. Reconfigure the repository connection string on each of the OMSsemctl config oms -store_repos_details3. Restart AdminServer and all the OMSs usinga. emctl stop oms -allb. emctl start omsRecovering the repository on a different host also requires relocating the Management Repository database target to adifferent Agent running on the new host. This can only be done if an Agent already exists on the host and no other databasehas been discovered by it. Use the syntax below to relocate the target to a new host.emctl config repos [-sysman_pwd <sysman password>][-agent <new agent>] Specify new destination agent for repository target[-host <new host>] Specify new hostname for repository target[-oh <new oracle home>] Specify new OracleHome for repository target19 Session # 326
  20. 20. Manageability[-conn_desc [<jdbc connect descriptor>]]Update Connect Descriptor with value if specified,else from value stored in[-ignore_timeskew] ignores timeskew on agentThe monitoring configuration for the OMS and Repository Target should also be updated using the command below.emctl config emrep [-sysman_pwd <sysman password>][-agent <new agent>] Specify new destination agent for emrep target[-conn_desc [<jdbc connect descriptor>]]Update Connect Descriptor with value if specified,else from value stored in[-ignore_timeskew] ignores timeskew on agentAfter the database has been recovered and reconfiguration performed, the next step would be to log in to the EnterpriseManager Cloud Control console and verify that all operations have been restored.ORACLE MANAGEMENT SERVICE RECOVERYRecovering an OMS requires recovery of both the software homes (Fusion Middleware) and the instance homes (gc_inst).The software homes can either be recovered from a filesystem backup or can be re-installed using the Install Software Onlyoption on the same or a different host using in the Enterprise Manager Cloud Control 12c installer. It is required that theOMS Home is in the same location as the OMS Home being recovered.All Plug-ins that existed in the OMS are required to be installed for the recovery to succeed. The SQL query below should berun as the SYSMAN user.SELECT epv.display_name, epv.plugin_id, epv.version, epv.rev_version,decode(su.aru_file,null, Media/External,||aru_file||?aru=||aru_id||chr(38)||patch_file=||aru_file) URLFROM em_plugin_version epv, em_current_deployed_plugin ecp, em_su_entities suWHERE epv.plugin_type NOT IN (BUILT_IN_TARGET_TYPE, INSTALL_HOME)AND ecp.dest_type=2AND epv.plugin_version_id = ecp.plugin_version_idAND su.entity_id = epv.su_entity_id;The example output below shows the plug-ins, version, revision and their URLs. The URL will display if downloaded viaSelf-Update. Otherwise the status will be Media/External.SQL>DISPLAY_NAME PLUGIN_ID VERSION REV_VERSION URL------------------------------ ------------------ ----------- ----------------------------Oracle MOS (My Oracle Support) oracle.sysman.mos 0Media/ExternalOracle Fusion Middleware oracle.sysman.emas 0Media/ExternalOracle Database oracle.sysman.db 20120804Media/ExternalOracle Exadata oracle.sysman.xa 0Media/ExternalIf any additional plug-ins are listed they should be downloaded into a single directory and then rename the extensions to a .opar. Use the “Install Software Only” option to install the Middleware and OMS Oracle Home components if notrestoring from a filesystem backup. After the software has been re-installed or restored, the next step is to install theadditional plug-ins (if any). Execute the script located in OMS_HOME/sysman/install by specifying the –20 Session # 326
  21. 21. ManageabilityPluginLocation flag to select the location where the downloaded plug-ins are kept. In the case of a Software Only installation,all patches previously applied will have to be re-done.After restoring the Software Homes the next step would be to restore or recreate the OMS. This is done using the omscautility and specifying the path of the backup generated by the emctl exportconfig command.omsca recover –as –ms -nostart –backup_file <exportconfig file>The steps required to recover the OMS may vary depending on whether or not a SLB is in use or multiple managementservices are configured as well as if recovery is done on the same host or a different host. The steps below are required forrestoring a single OMS on the same host without a SLB. OMS instance will be recovered using the OMS configurationbackup taken by emctl exportconfig.1. Cleanup the failed host.2. Verify that the Software Library is available.3. Restore the software home by restoring a backup of the filesystem or run the installer in using the Software Onlyoption. (See Oracle Enterprise Manager Cloud Control Advanced Installation Guide).4. Execute omsca in recovery mode and specify the location of the backup file taken previously.5. Recover Agents if necessary.6. Start the OMS7. Verify that the OMS is workingSee the Enterprise Manager Cloud Control Administrator’s Guide for details other OMS Recovery scenarios.MANAGEMENT AGENT RECOVERYThe recovery of the Management Agent involves re-installing it, preferably from a reference install or performing a filesystemrestore from a previous backup. It should be cloned with the existing patches and customizations. The Agent should also beinstalled using the same port. After it has been re-installed a resynchronization should be performed from the AgentResynchronization page in Enterprise Manager Cloud Control.Note The agent is blocked by the OMS after re-installation to prevent targets from overwriting data from previousconfigurations. Use the Resynchronize Agent button to resynchronize and unblock the Agent.Agent recovery in a typical scenario after it is lost usually follows the steps below.1. Remove existing Agent Oracle Home using the OUI to clean up the inventory2. Install a new Agent or clone from reference install using the same port and path. The filesystem can also be restoredfrom a previous backup.3. Perform Agent Resynchronization from the Enterprise Manager Cloud Control Agent home page.4. Reconfigure User-defined Metrics if necessary5. Verify the status of the Agenta. emctl status agentb. emctl upload agent21 Session # 326
  22. 22. ManageabilityAPPENDICESREFERENCESMAA Best Practices - Enterprise ManagerHow To Configure Enterprise Manager for High Availability [ID 330072.1]OTN Whitepaper: Configuring OMS High Availability with F5 Big-IP Local Traffic ManagerOracle Enterprise Manager Cloud Control Administrators Guide 12c Release 1 (, Part II - Enterprise Manager HighAvailability.22 Session # 326