Cloning Release 12: Oracle Applications Part II: Rapid Clone techniquesMake sure pre-clone steps for Release 12 with Oracle E-Business Suite arecompleted.Now that our pre-clone is complete, its time to do the actual cloning operation.With Release 12 for Oracle Applications, Oracle providesa useful perl script called adcfgclone.pl as part of Rapid Clone that performs mostof the heavy lifting for cloning the E-Business Suite.Note: 406982.1 - Cloning Oracle Applications Release 12 with Rapid CloneNote: 603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12In a nutshell the cloning process is as follows:========================================================I. Clone database tier from source to target environmentII. Clone applications tier from source to targetCloning Release 12 Oracle E-Business Suite with Rapid CloneOk so we ran the adpreclone.pl scripts from last time and finished the prep workfor the cloning. So how do we do the actual cloning or refresh for Release 12?Actually the answer is that it depends on your environment. You can do a hot cloneusing the Oracle Applications Management Pack (AMP) with Oracle EnterpriseManager (OEM) Grid Control to cut down on downtime window. However thatrequires additional licensing and installation time. Or you could use RMAN or anOracle hot backup to take the copy of your source database tier and clone to thetarget environment. However, for our example, we will use a simple cold backupto copy over the apps tier and database tier from our Source EBS environment toour target system.As I mentioned above, for cloning an Oracle Applications environment, you woulduse your backup as well as copy commands manually or via a series of UNIX orLinux shell scripts to copy over the apps and db tier as part of the cloning.
However, in my test environment, I do not have an extra 300Gb of disk spaceso we will use the Unix/Linux mv (move) commands to simulate the cloneprocedures.If you decide to use the Oracle Recovery Manager (RMAN) utility to clone thedatabase the procedure will be different.Reference the Oracle documentation on usage for RMAN for details.Some details on how to use RMAN to do this are listed below:Note: 362473.1 - Cloning E-Business Suite Using Hot Backup for Minimal Downtimeof Source EnvironmentOracle® Database Backup and Recovery Basics 10g Release 2 (10.2)http://download.oracle.com/docs/cd/B19306_01/backup.102/b14192/toc.htmOracle® Database Backup and Recovery Advanced Users Guide 10g Release 2(10.2)http://download.oracle.com/docs/cd/B19306_01/backup.102/b14191/toc.htmIn my opinion, you should use the RMAN Duplicate database command to take a hotbackup of the database tier.This works well and details are provided inNote: 228257.1 RMAN Duplicate Database Feature in Oracle9i / 10G and 11GNote: If your database tier server is on 11gR1 or 11gR2 the new features for RMANsuch as encrypted backups, Data Recovery Advisor (DRA), et al may also be of useto your backup and recovery strategy at future date. I have a blogpost on 11g RMAN new features in a previous blog post if you have questions onhow this works with RMAN and 11g.
Ok so lets get started and cloning! First, we start with cloning the database tier.On the database tier, we login as the oracle OS user and shutdown both thelistener and database:I. Clone Source database tier to target database tier serverSince we are simulating a clone procedure we create the new clone directory fordatabase and applications tiers as follows:mkdir -p /dbtier12/CLONE
mkdir -p /appstier12/CLONEBefore we use Rapid Clone script (adcfgclone.pl) we need to copy over thedatabase and application tier files.First we copy the application tier file systemLog on to the source system application tier nodes as the APPLMGR user and shutdown the application tier server processes. Copy the following application tierdirectories from the source node to the target application tier node:$APPL_TOP$COMMON_TOPApplications Technology Stack:Oracle 10gAS 10.1.2 Tools ORACLE_HOMEOracle 10gAS 10.1.3 Web ($IAS_ORACLE_HOME)So this means that for the Apps tier you need to copy over the $INST_TOP,$COMMON_TOP, $APPL_TOP, 10.1.2 ORACLE_HOME, and 10.1.3 ORACLE_HOMEdirectories and filesCopy the database node file system
Log on to the source system database node as the ORACLE user:1. Perform a normal shutdown of the source system database2. Copy the database (.dbf) files from the source system to the target system3. Copy the source database ORACLE_HOME to the target system4. Start the source Applications system database and application tier processesFor the database tier this includes the datafiles, parameter file (init.ora, spfile)and tnsnames.ora filesIn this test environment, I do not have 300+ Gb of disk space so we simulated thecopy with the UNIX/Linux mv command. However in a real world environment youwould use the cp or scp command to transfer files from source to target systems.You would need to shutdown the database and take a cold backup of the databaseand apps tiers unless you do hot cloning and hot backup.So now on the target we have the following clone directories:1) Apps tier for CLONE
/apptier12/CLONE base directory for apps tier-> 10.1.2 for CLONE ORACLE_HOME 10.1.2-> 10.1.3 for CLONE ORACLE_HOME 10.1.3-> appl for CLONE APPL_TOP-> comn for CLONE COMMON_TOP2) db tier for CLONE/dbtier12/CLONECLONE 10g ORACLE_HOME -> /dbtier12/CLONE/10.2.0CLONE datafiles /dbtier12/CLONE/db/apps_st/data/dbtier12/CLONE/db/tech_stWe will need to keep track of this information for the upcoming step when we runthe clone scripts!Do not set the environment files for the database tier! We need a clean slate forthe database tier clone procedure.Note: if you decide to do a copy of the db and apps tier files from your source totarget system, you will need to start the database tier and apps tier services onthe source system. In this example we did a Unix/Linux move (mv) to simulate thecopyso we do not start the apps tier and db tier since both source and target are onthe same virtual machine.Run the clone script for db tier this is called adcfgclone.pl we can take a sneakpeak at it with vi or emacs or whatever your favorite editor
perl adcfgclone.pl dbTierHow the clone script flowsadcfgclone.pl calls the script adchkutil.sh which then calls forth the scriptadclone.pl and adclonectx.plNote: if you do not have PD ksh installed the work around to get adcfgclone.plworking is to edit the adchkutil.sh script to comment out the reference in the
script to look for the PD ksh shell.Clone source database tier to target database tiercd /dbtier12/CLONE/10.2.0/appsutil/clone/binperl adcfgclone.pl dbTierenter the password for APPS then hit the keyenter the details for target system at the promptsIn our case we take the defaults since we are cloning from source to target onsame system as shown in the example belowOnce we enter the choices for the prompt settings the clone will begin for the dbtier. You will need to enter settings for your target clone datafile location(DATA_TOP) as well as database tier ORACLE_HOME location, and other settingssuch as port pool.
Time to go for a coffee break! It will take some time and depends on the size ofyour database. I have scripts for cloning that I usually fire off at night time andbabysit to make sure all runs smoothly.You can tail the log file to keep track of your cloning operation. I like to dump theclone output to a logfile as part of a script to verify things.When it finishes you should see a successful screen like below:
Rapid Clone starts the listener and database back up once the clone is created. Wecan run a few simple commands to verify that our new database clone is onlineand available:My clone took about 90 minutes for the database tier since I have a very slowserver running in a virtual machine. Your mileage may vary based on hardware, OSand available resources. Now lets clone the apps tier and finish our cloning!
Clone Source applications tier to target applications tierNow that our database tier has been cloned its time to clone the apps tier to thenew target! We run the adfgclone.pl script on the source apps tier as shown below:For our example, The cloning script for the apps tier lives under thefollowing directory:$ cd $INST_TOP/clone/binLogin as the applmgr OS user and we run it with the following syntax:perl adcfgclone.pl appsTierCloning the Applications TierSince I do not have the PD ksh installed it errors and then I need to edit theadchkutl.sh script which is called by adcfgclone.pl like last time so that it does not
try to search for the PD ksh.Rapid Clone uses a questionaire to prompt us for the details for the source totarget cloning for the applications tier similar to the questionaire in the cloningscript (adcfgclone.pl) that we ran earlier to clone the database tier.We enter the following for the target apps tier system prompts:target system base directory: /appstier12/CLONEtarget system Forms ORACLE_HOME /appstier12/CLONE/10.1.2target system Web ORACLE_HOME /appstier12/CLONE/10.1.3choose 0 for port pool like we did earlier for db tier cloneA complete list is given below:Now we want to keep a close watch on the logs to make sure no errors occurduring the cloning for the apps tier in which we tail the logfile:
Are you ready for another coffee break? I sure hope so because I am! So lets grab amocha and maybe a donut while this clone runs for the next few hours. I usuallyrun a clone script at night time and check it every half hour until its done.In between the coffee and donut break, I check the status and its at 50% andbefore I can drink the java or eat my donut a frantic Oracle project manager andfunctional analyst are waiting for me in my DBA cube!I calm the functional team down and fix their performance issue. Was resourceproblem with JVM for the apps tier for another environment. I will blog on asolution to this in future post. OK so we check our clone and enter additionalprompts after we had to restart it after it failed due to port issue that I fixed andre-started the clone.
Our clone finishes and all of the app tier services are started automatically byRapid Clone (adcfgclone.pl) script.
And we have verification without errors.Troubleshooting Errors during Rapid Clone - AppstierRecall that earlier our clone failed at 74% during the cloning for the applicationstier when we ran adcfgclone.pl appsTier so I checkedthe logfile under/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/log to find the
specific failure since it failed during the ApplyAppsTier phaseAfter looking at the logfile ApplyAppsTier_02251717.log I found the root causewhy my clone failed was due to unable to connect to the listener and database.The port pool needs to be same on database and applications tierso if you set 0 as port pool during adcfgclone.pl dbTier for db tier you MUST USEsame port pool for clone with apps tier or the clone will fail during Appstier applyphase!!!For Linux, you can use lsof command to find ports in use and use kill command tofree up these ports.After the cloning scripts have run successfully, the apps and db tiers will bestarted automatically by the Rapid Clone scripts.I then reviewed the logs under/appstier12/CLONE/comn/clone/binfor both the source database and applications tiers and verified that all applicationand database services are available.Next we need to verify that, users can login to OAM as cloned environment.Sure enough, I hit the dreaded 500 Internal Error when I tried to login to the OAMwebsite.Here are some good troubleshooting notes if you have 500 Internal errors whenattempt to access the OAM login pageNote: 761869.1 500 Internal Server ErrorNote: 813523.1 500 Internal ServerNote: 443353.1 Guest Password mismatchNote: 564536.1 ADSTRTAL.SH returns error TIMEOUT OUT Interrupted Exception
Note: 331322.1Note: 391406.1 How to get a clean AutoconfigNote: 387859.1 Autoconfig R12I used these to resolve the issue which involved changing settings for the contextfile to allow JVM to function correctly without consuming all server resources aswell as bouncing the app and database tiers on the target clone server.After all was said and done, I verified that things worked for my new clone.As a final sanity check, I checked the database and application tier servers for thenew cloned environment.
Good the cloned database is functional and available. Lastly, I checked that all ofthe application tier services were online.Finishing Tasks for Cloning Operations1. Workflow settings for new clone2. Verify login and accounts work correctly3. Check status for SSO and OIDWhile we did not check on workflow, this is one area that you must address in your
cloned environments. Also, run tests to check that the authentication is workingcorrectly with accounts that use SSO and OID. If you do not perform these checksthen users will have problems logging into the cloned systems after a refresh.Congratulations! You now can perform the cloning and refresh tasks that arecritical skills to have as a hardcore Oracle Apps DBA!