Oracle Release 12 E-Business Suite Patching Best Practices

Tips and Tricks for Patching R12 Oracle Applications E-Business Suite Ben Prusinski Abstract This session will provide best practices to patch your Oracle R12 EBS environments to ensure maximum availability and performance. Introduction Oracle Application database administrators spend a large amount of time patching environments for the E-Business suite. As such, the traditional method of using the suite of application tools such as opatch for the database tier and adpatch for application tier present unique challenges in terms of availability in mission critical corporate information systems that require 24x7x365 uptime. Patching is usually scheduled for off hours maintenance windows by the Oracle Applications DBA. However, in terms of availability, adpatch requires that the applications tier be shutdown in most cases or that the DBA place the EBS environment in what is called “maintenance mode” which means that business users will be unable to process their transactions and/or reports while a patch is being applied to the EBS environment. Second, with hundreds of patches applied by the DBA team to Oracle Applications EBS environments, the book keeping tasks of recording patches applied and the status of said patch not only is tedious but can be problematic. Third, patching is a complex affair and is daunting to even an experienced Apps DBA. My goal in this paper is to present not only best practices but tips to help avoid painful downtime as well as to ease the nature of patching for the EBS environment as well as to present finding on new features in Release 12 for the Oracle E-Business suite to provide assistance in the complex nature of patching your environments. Best practices for patching Oracle R12 EBS As an Oracle Applications DBA, one of the biggest challenges with patching the E-Business Suite is how to apply patches with minimum downtime. As business environments operate around the globe, we cannot afford costly maintenance windows during business hours. Collaborate 10 Copyright ©2010 Ben Prusinski 1

Zero or Less Downtime for Patching R12 EBS One method to minimize downtime is to deploy a high availability solution such as Data Guard with your Oracle R12 EBS infrastructure. Oracle Data Guard has been certified for R12 release and provides you with the ability to not only reduce potential downtime when patching the database tier but also affords you with a solid data protection component for disaster recovery and business continuity. However, while Data Guard provides you with the ability to avoid taking an outage or downtime while patching the database tier with opatch, you still need to find a way to avoid downtime for patches that are applied to the applications tier for the E-Business Suite. This is where the hotpatch option and restricted maintenance options come into play for Autopatch with EBS for the applications tier. Getting Help with adpatch With dozens of options, even the most seasoned Oracle Apps DBA is challenged to recall every feature available with adpatch from the command line. Fortunately, help is available by specifying the adpatch help=y as shown in the below figure to provide a listing. $ adpatch help=y usage: adpatch [help=y] adpatch Pre-Install Mode [preinstall=y|n] adpatch Test Mode [apply=y|n] adpatch Non-Interactive mode [defaultsfile=<$APPL_TOP/admin/SID/defaultsfile>] [logfile=<logfile>][interactive=y|n][workers=<workers>] [patchtop=<patchtop>][driver=<driver_file>][restart=y|n] [abandon=y|n][wait_on_failed_job=y|n][stdin=y|n] adpatch [localworkers=<localworkers>][workers=<workers>] [printdebug=y|n][parallel_index_threshold=<threshold_value>] Collaborate 10 Copyright ©2010 Ben Prusinski 2

[order=<order>] [flags=<flags>][options=<options>] adpatch Upload Patch History From Filesystem [uploadph=y|n] where * help=y - Prints help information about adpatch options. * preinstall - To run adpatch in Pre-Install Mode. Default - No. * apply - To run adpatch in Test Mode. Default - Yes. * uploadph - To upload the patch history from the filesystem to the database and exit. This feature can be used when the system is not in maintenance mode. Default - No. * interactive - Invokes adpatch in Non-Interactive mode when "interactive=no" is specified. Default - Yes. Following are usually specified in Non-Interactive Mode, however not all are compulsory. Some are also valid for interactive mode. * defaultsfile - Defaults file name. Default - none. No default file read or written. * stdin - If "stdin=y" then password will be prompted. Default - No. * logfile - Adpatch log file name. Default - none. Adpatch prompts for this value. * workers - Specifies the number of workers to run. Default - none. Adpatch prompts for this value. * patchtop - Top-level directory for the current patch. Default - none. Adpatch prompts for this value. * driver - Name of the patch driver file. Default - none. Adpatch prompts for this value. Collaborate 10 Copyright ©2010 Ben Prusinski 3

* backup - Absolute Path for backup. Default - patchtop. * restart - To restart an existing session. Default - none. Adpatch prompts for this value. * abandon - To abandon the previous failed session. Default - No. * wait_on_failed_job - To prevent adpatch from exiting, when all workers have failed in Non-Interactive Mode. Default - No. * localworkers - Used in Distributed AD to specify the number of workers to be run on the current machine. Default - Value specified for workers. * printdebug - To display extra debugging information. Default - No. * parallel_index_threshold - Threshold block count in each table, which when exceeded for a table causes its indexes to be created using parallel slaves. Default - none. Adpatch does not create indexes in parallel. * order = To organize list of jobs to run in a phase. Default - forward. * forward - The order in which jobs are listed in the patch driver file. * backward - Opposite to how the jobs are listed in the patch driver file. * flags = Generic flags passed to adpatch * hidepw - To omit the "HIDEPW:" comments in the log file. Default - hidepw. * trace - To log all database operations to a trace file. Collaborate 10 Copyright ©2010 Ben Prusinski 4

Default - notrace. * logging - To create indexes in LOGGING or NOLOGGING mode. Default - nologging. * options = Generic options passed to adpatch. * checkfile - To skip running exec, SQL, and exectier commands if they are recorded as already run. Default - checkfile. * compiledb - To compile invalid objects in the database after running actions in the database driver. Default - compiledb. * compilejsp - To compile out-of-date JSP files, if the patch has copy actions for at least one JSP file. Default - compilejsp. * copyportion - To run commands found in a copy driver. Default - copyportion. * databaseportion - To run commands found in a database driver. Default - databaseportion. * generateportion - To run commands found in a generate driver. Default - generateportion. * integrity - To perform patch integrity checking. Default - nointegrity. * autoconfig - To run AutoConfig. Default - autoconfig. * actiondetails - To get the details of actions turned off. Default - actiondetails. * parallel - To run actions that update the database or actions that generate files in parallel. Default - parallel. * prereq - To perform prerequisite patch checking prior to running patch driver files. Collaborate 10 Copyright ©2010 Ben Prusinski 5

Default - noprereq. * revcache - To load the package revision cache (if adpatch determines it is needed). Default - revcache. * validate - To connect to all registered Oracle Applications schemas at the start of the patch. Default - novalidate. * hotpatch - A patch which can be applied in Normal mode. Default - nohotpatch. * phtofile - The patch history will be saved to files. Default - nophtofile. Figure 1: Using adpatch help=y Use Hotpatch Option with adpatch for Applications Tier One issue that DBAs struggle with when they apply a patch to the applications tier when using adpatch utility is how to avoid impacting users during business hours. In the event of a patch that requires a shutdown of the app tier services, you don’t have a choice. However, for patches that do not require a shutdown of the app tier, you still need to worry about whether or not to enable maintenance mode for adpatch. The hotpatch option allows you to avoid the requirement to place the application tier into maintenance mode thus reducing downtime impact to business users. $ adpatch options=hotpatch Figure 2: Using hotpatch option with adpatch Hotpatch is a useful option with patching the E-Business Suite for Release 12, however, caution must be exercised when using it.. Essentially, it does allow you to put patches on the system with users on the system and the applications tier online. HOWEVER, here are the caveats for it: 1. It is technically not supported by Oracle support. 2. If you elect to use it, it should only be used on patches that do not run autoconfig or update the templates Collaborate 10 Copyright ©2010 Ben Prusinski 6

3. You need to know what files are being updated and make sure users are not in those files (ie if it just updates a single form you don't want anyone using the form) 4. It will run slower than if you place the applications tier into maintenance mode and shutdown application tier services. I recommend that you first test a patch in a sandbox or non-production environment to assess the full impact of patches for the E-Business Suite. Recently I had an experience with using hotpatch on a live test system in which the patch failed and as a result the only recourse was to do a recovery and rollback of the patch. As such, while hotpatch has useful functionailty, in most cases there are better ways to improve availability and performance for your patching strategy with the E-Business Suite. Some of these options would include: 1. Using a staged applications tier filesystem with a shared APPL_TOP 2. Distributed AD 3. Merge patch (admrgpch) We will discuss these alternatives in the following section. Use a Staged Applications Filesystem Tier with Shared APPL_TOP One effective way to reduce downtime and maintenance windows for patching activities besides using the hotpatch option with Auto Patch is to make usage of a stage applications filesystem with a shared APPL_TOP as covered in Oracle Metalink Note 743025.1. Use Shared APPL_TOP A traditional multi- node installation requires that you deploy an applications tier file system for each node in the E-Business Suite system. A shared APPL_TOP installation allows you to install both of the APPL_TOP and COMMON_TOP file systems to a shared disk resource mounted to each node in the applications tier system. For example, you can use iSCSCI, NFS or another file sharing configuration to deploy a shared APPL_TOP with the E-Business Suite for Release 12. These nodes can then be used to provide for all standard application tier services, such as Forms, Web, and Concurrent batch processing. If any changes are made to the shared APPL_TOP file Collaborate 10 Copyright ©2010 Ben Prusinski 7

system these changes will be propogated and visible to all nodes in the system. Use a Shared application tier file system In a shared application tier file system installation, the APPL_TOP, the COMMON_TOP, and the Applications technology stack are installed on shared disks that are mounted to each node in the system. These nodes can be used to provide standard application tier services, such as Forms, Web, and Concurrent processing. Any changes made in the shared application tier file system are immediately visible on all nodes. Staged APPL_TOP to minimize downtime What is a staged applications filesystem you might be wondering? To answer this question is simple. A staged APPL_TOP is a configuration of the applications system that reflects an exact copy of your Production system, including all versions of the APPL_TOPs including a copy of the current production database. How does this provide for maximum uptime and to minimize downtime? The way it works is that patches are first applied to the staged system, while your production system remains online. After all of the patches have been successfully applied to the test systems, the downtime for the production environment is initiated. As such, the staged APPL_TOP is used both to run the database update into the production as well as to synchronize the production APPL_TOP environment. The staged APPL_TOP is the applications tier filesystem that has been copied from the gold copy of the production system in which patches are applied and then copied back to the production environment. It is also used at the same time to update the production environment. To create the staged filesystem with shared/staged APPL_TOP you must take this copy of production APPL_TOP and apply to the staging system. To do this use the Rapid Clone utility with the E-Business Suite. Also, be sure to take a current snapshot using adadmin utility. On the new environment, your staged APPL_TOP name should be the same as that for your production system’s APPL_TOP so that the database driver will be able to update the patch history information correctly. Collaborate 10 Copyright ©2010 Ben Prusinski 8

Using Distributed AD for Auto Patch Performance with Release 12 An additional option to enhance performance for patching tasks within the E-Business Suite is to use distributed AD functionality. This uses multiple parallel processing to spread the load of apply the patch across multiple nodes for an Oracle Applications environment. It is especially useful for multi- node configurations. Now let’s explore the method to setup distributed AD. First we need to start Auto Patch (adpatch) or AD Controller (adctrl) with the distributed options. On one of the application tier nodes, start either one of these utilities with the following option: localworkers=<number of workers on node> workers=<total number or workers> For example, to run an AutoPatch session with a total of twelve workers (four workers on the local node and eight workers on a remote node): $ adpatch workers=12 localworkers=8 Start AD Controller on remote node(s) For each additional shared application tier file system node, start an AD Controller session with the additional distributed command line option: $ adctrl distributed=y Note: after you specify the parameters to adctrl, AD Controller will prompt for the worker number(s) to start. For example, To start workers 4 through 12 on a second node, enter "4 5 6 7 8 9 10 11 12" or "4-12". Note: workers must be in contiguous groups, meaning that you cannot specify a group of 1,3,5 on one node and 2,4,6 on another. The worker range on the node where the adpatch is running must start with 1, and must have at least one worker. The following is an example of starting a three-node session with a total of twenty workers: Node 1: $ adpatch localworkers=5 workers=15 Collaborate 10 Copyright ©2010 Ben Prusinski 9

Node 2: $ adctrl distributed=y Enter the worker range: 1-10 Node 3: $ adctrl distributed=y Enter the worker range: 11-15 Now let’s walk through an example of how to setup and use distributed AD for Oracle Applications Release 12. First we need to start a new AD controller (adctrl) session as applmgr user account in another terminal session. Figure 2a: Starting AD controller for Distributed AD with Release 12 Oracle Applications Next AD controller prompts us for the total worker range: Collaborate 10 Copyright ©2010 Ben Prusinski 10

Figure 2b: Worker range for Distributed AD with Oracle EBS Release 12 Based on current memory and CPU allocation we enter the number of workers to be used during our patching session. Multiple Workers for Faster Patch Time The default number of adworkers for a server is based on the number of physical CPU processors on the server. Oracle calculates the default number of adworkers by 2xCPUs plus 2. A good rule of thumb to use for the number of workers for adpatch is twice the number of physical CPU processors plus four in the application tier server. For instance, if you have a Linux application server with 4 physical CPUs, you could set the number of default adworkers to 8+4 for a total number of 12 adworkers thus improving the performance for adpatch operations. I recommend Collaborate 10 Copyright ©2010 Ben Prusinski 11

that you test performance to find the ideal number of adworkers without impacting server performance. $ adpatch workers=12 Figure 3: setting number of workers for adpatch Use a patch spreadsheet and logfile identification As busy Apps DBAs, we often have to apply dozens or even hundreds of patches each year. Keeping track of each patch, when it was applied and the rationale behind the patch can be challenging. A patch spreadsheet is crucial for the Apps DBA to be successful in tracking patch management. An example of a good patch spreadsheet should contain the patch number, bug fixed by the patch, product group for the patch, date applied, as well as the environment to which the patch has been applied. Below is a good example of a patch spreadsheet. Company XYZ Patch Log Product Patchset / Pre- Post-Requisite For Name Patch # Requisite Patch # For Patch PROD QA DEV Time for patch Approval # execution Financials 683635 1/1/2010 12/1/2009 11/1/2009 65 minutes Big Boss Figure 4: Patch Spreadsheet log example So, let’s put all of this together with a quick example using adpatch $ adpatch driver=u2344555.drv options=hotpatch workers=10 logfile=234455_BSP_DEV.log Review the Pre-requisites for a new patch Before apply a patch that has been requested by the functional or development team, it is wise to first review the Readme file and any pre-requisites or co-requisites for the patch. The readme file contains the details as shown in the example below. Collaborate 10 Copyright ©2010 Ben Prusinski 12

README for 9165901 Patch Details Abstract Assignment change events are not captured when changes are done using API Update 9165901 Product Advanced Benefits Release R12 Platform Generic Platform Built JAN-21-2010 03:07:53 Instructions Instructions For Applying Thi s Patch Preparation Tas ks The tasks in this section can be completed without taking any Applications ser vices or users offline. Required Code levels This patch requires additional code levels to have been successfully applied to your system before it can be applied. These code levels, as at the time that this patch was built, are listed below. Please check OracleMetalink to determine if any of these have been superseded · R12.AD.A.delta.6 · R12.ATG_PF.A.delta.6 · R12.HR_PF.A.delta.6 Pre-install Tas ks You must shut down all Application tier services before performing the tasks in this section. T he re are no t asks to be p e rfo rm e d in th is se c t io n. Apply The Patch Apply patch This patch contains the following unified driver file to be applied with AutoPatch: u9165901.drv (This is the unified driver) Post-install Tas ks You must complete the tasks in this section before star ting up Application tier ser vices. Collaborate 10 Copyright ©2010 Ben Prusinski 13

T he re are no t asks to be p e rfo rm e d in th is se c t io n. Finis hing Tas ks You may complete the tasks in this section at any time after the update, without taking any services or users offline. T he re are no t asks to be p e rfo rm e d in th is se c t io n. Addit ional Infor mat ion T he re are no t asks to be p e rfo rm e d in th is se c t io n. Description Functional Description: --------------------------------------------------------------------------- -- When api is used to update assignment details, the extract change events are not logged. Technical Description: --------------------------------------------------------------------------- -- Trigger on assignment table does not log events when changes are done using api as the :old and :new values are same. Extract's procedure ben_ext_chlg.log_asg_chg should be called from assignment rhi to log events. Packages modified: --------------------------------------------------------------------------- -- ben_ext_chlg (overloaded procedure log_asg_chg) per_asg_upd (extract's call added in pre_update) Use a defaults file for patch maintenance A defaults file allows you to kick off a patch session with adpatch without being prompted for options during a patching session. I recommend the use of defaults since you can reduce complexity when migrating patches between environments. For instance, let’s say you apply patch 9165901 to a development environment, you could create a defaults file that can be used to migrate this patch to QA and production more efficiently once it has been applied and tested in development. $ adpatch defaultsfile=$APPL_TOP/admin/TEST/defs.txt The defaultsfile should take the value of $APPL_TOP/admin/<SID> directory Collaborate 10 Copyright ©2010 Ben Prusinski 14

You need to run adpatch until Oracle prompts you for the directory of the patch uploaded. Then you need to issue abort to exit. This will now allow you to run a non- interactive session using the defaultsfile for adpatch. $ adpatch defaultsfile=$APPL_TOP/admin/TEST/defs.txt Logfile=mylog.log Patchtop=$APPL_TOP/patch/77744 Workers=12 Interactive=no Figure 5: Using defaultsfile with adpatch for R12 Be Careful with Merging Patches While support is available for admrgpch, I recommend that you exercise caution when merging patches for Oracle Application EBS environments. There have been cases that cause errors to occur in R12 EBS after patches have been merged successfully even when supported by Oracle patch documentation. My advice is to thoroughly test patch merging in a non- impact environment such as sandbox system before apply to critical environments to make sure that you work out all of the kinks with patch merging. Using the admrgpch utility is simple. You need to create a source directory under your $PATCH_TOP directory and upload the patches to be used for the merge task. Then create a second directory for the target directory under the $PATCH_TOP directory and run the utility with the manifest file per the Oracle documentation. An example of available options for admrgpch is provided below. Collaborate 10 Copyright ©2010 Ben Prusinski 15

Figure 6: Help options for AD Merge Patch with Release 12 Oracle E-Business Suite Check for Invalids and Recompile after patching One common issue that occurs during patching is that many objects become invalid. While adpatch contains a post-apply step to recompile these invalids, usually there will still be invalid objects after applying a new patch to the system. As such, you should run adadmin to recompile the APPS schema as well as utlrp.sql to recompile invalid objects outside of the APPS schema. This is especially important for objects in the SYS and SYSTEM schemas that may be invalid as a result of applying a new patch. Checking for Invalids: SQL> select owner, object_name, object_type from dba_objects where status = ‘INVALID’; Collaborate 10 Copyright ©2010 Ben Prusinski 16

Compiling Invalid Objects: Below are some options for compiling invalid objects. · Use adadmin to run ‘Compile Apps Schema’. This is helpful to use if there are a large number of invalid objects because it compiles the invalids for the entire APPS schema. For example, you can choose this option as shown below to recompile invalids after a patch has been applied. AD Administration Main Menu -------------------------------------------------- 1. Generate Applications Files menu 2. Maintain Applications Files menu 3. Compile/Reload Applications Database Entities menu 4. Maintain Applications Database Entities menu 5. Change Maintenance Mode 6. Exit AD Administration Collaborate 10 Copyright ©2010 Ben Prusinski 17

Enter your choice [6] : 3 Compile/Reload Applications Database Entities --------------------------------------------------- 1. Compile APPS schema 2. Compile menu information 3. Compile flexfields 4. Reload JAR files to database 5. Return to Main Menu Enter your choice [5] : 1 sqlplus -s APPS/***** @/TEST/bin/apps/apps_st/appl/ad/12.0.0/sql/adutlrcmp.sql APPLSYS ***** APPS ***** ***** 0 0 NONE FALSE Figure 6b: Using adadmin to recompile invalid objects Behind the scenes, adadmin will execute a script called adutlrcmp.sql against the APPLSYS and APPS schemas to recompile invalid objects. If we view this script, we can determine that Oracle is query the database tier for invalid objects for the APPS schema and then executes the recompile of these invalids via the SYS.UTL_RECOMP.RECOMP_SERIAL package as shown below. prompt Running utl_recomp.recomp_parallel(&&6), if it exists prompt Collaborate 10 Copyright ©2010 Ben Prusinski 18

variable status_message varchar2(1000); variable run_it varchar2(10); select owner, count(*) num_invalid from dba_objects where status='INVALID' group by owner; declare dummy varchar2(10); cursor CHECK_FOR_PKG is select 'x' from dba_objects where owner='SYS' and object_name='UTL_RECOMP' and object_type='PACKAGE'; if CHECK_FOR_PKG%NOTFOUND then :status_message := 'UTL_RECOMP package does not exist: will not run it'; :run_it := 'FALSE'; else :status_message := 'Running UTL_RECOMP.RECOMP_PARALLEL...'; :run_it := 'TRUE'; end if; close CHECK_FOR_PKG; end; Collaborate 10 Copyright ©2010 Ben Prusinski 19

/ for i in 1..l_schema_name_tab.count loop if (l_invobj_count_tab(i) < l_inv_obj_threshold) then c_statement := 'begin SYS.UTL_RECOMP.RECOMP_SERIAL('|| ' '''||l_schema_name_tab(i)||'''); end;'; else c_statement := 'begin SYS.UTL_RECOMP.RECOMP_PARALLEL(&&6, '|| ' '''||l_schema_name_tab(i)||'''); end;'; end if; execute immediate c_statement; end loop; :status_message := 'Successfully ran UTL_RECOMP.RECOMP_PARALLEL'; end if; select owner, count(*) num_invalid from dba_objects where status='INVALID' group by owner; prompt Collaborate 10 Copyright ©2010 Ben Prusinski 20

prompt Done running utl_recomp.recomp_parallel prompt commit; exit; Figure 7: Internals for the adutlrcmp.sql script to recompile invalid objects Compile additional invalids in database tier with utlrp.sql For the database tier which may contain invalid objects, you can also run the utlrp.sql script which is helpful for compiling all invalids within the database tier. This script needs to be run as SYS privileged account and lives under $ORACLE_HOME/rdbms/admin on the database tier server for R12 EBS. Collaborate 10 Copyright ©2010 Ben Prusinski 21

Manual recompile for invalids When all else fails and neither adadmin nor utlrp.sql succeed in recompilation of the invalid database schema objects, you need to perform a manual recompile with the ALTER COMPILE commands as shown below to resolve the invalid objects. Examples: ALTER PACKAGE AR_MYPACKG compile body; ALTER PACKAGE AR_MYPACKG compile; Review log files for Auto Patch with R12 EBS To ensure that no errors occurred in a patching session, you should review the log files generated by adpatch. I recommend a naming convention to simplify the tracking and location of these logs for future reference. For example, in a patching session you can use adpatch logfile=driver_id_<SID>_date to give a meaningful name to your patch log files. For instance: $ adpatch driver=777444.drv logfile=777444_DEV_01012010.log By default, adpatch will place the log files that are generated during the patch process under the $APPL_TOP/admin/<ORACLE_SID>/log directory. Test Patch Before Deployment to Production It is critical to take the time to test any new patch for EBS in a non-production sandbox or test environment to avoid potential issues that may arise with the patch. Another option is to run adpatch in test mode with the noapply option. Rolling back a failed patch is difficult if not impossible in most cases and you really do not want to impact the business with an expensive and costly recovery process if the patch blows up in a critical system. One way to test a patch without impact to an environment is to use the adpatch option apply=n as shown below which will run adpatch and show the results without actually applying the patch. If you do not have the luxury of a sandbox environment, this is a helpful way to deploy a patch at least impact to minimize potential issues with a new patch. $ adpatch apply=n Collaborate 10 Copyright ©2010 Ben Prusinski 22

The test mode for adpatch operates much like an interactive patch session for adpatch except that no files are copied to the installation area from the patch directory nor are any SQL scripts executed and no relinking is performed by adpatch. Review notes and pre-requisites for patch Many times a particular patch will have either pre-requisite patches or steps that MUST be performed before applying the patch. It is important to review these in advance before you apply a patch to avoid issues. By taking the time to understand the Readme files for each patch, you avoid many problems during the patch process. Tools and techniques for patching R12 EBS Oracle Applications. With the latest release of the E-Business Suite for Release 12 of Oracle Applications, additional features have been added and enhanced to improve patch tasks. The Patch Wizard is a powerful tool that was introduced in the 11.5.10.2 (11i) release of the E-Business Suite. Another tool is the automation perl script admsi that performs many pre and post patch maintenance tasks that were manual processes that the DBA had to do by hand in previous releases. Inside the Patch Wizard First introduced in 11.5.10.2 (11i) release of the E-Business Suite for Oracle Applications, the patch wizard has been enhanced with additional functionality to simplify the patch tasks that an Applications DBA performs on a regular basis. To access the Patch Wizard, you need to login to OAM (Oracle Applications Manager) graphical interface as SYSADMIN responsibility account. The navigation path is located under site map-> maintenance-> Patch Wizard Collaborate 10 Copyright ©2010 Ben Prusinski 23

Figure 8: Patch Wizard Tasks for Oracle Release 12 E-Business Suite The main screen for the Patch Wizard provides you with a variety of options to customize your patch strategy and deployments. For instance, you can setup a staging directory for patch downloads, recommendations and analysis. This reduces time required that one would need to do manually in the past. Collaborate 10 Copyright ©2010 Ben Prusinski 24

Figure 9: Configure Staging Directory with Oracle Release 12 E-Business Suite We also have an option to obtain all recommended patches based on codebase and code level as part of the filter options for the Patch Wizard. Collaborate 10 Copyright ©2010 Ben Prusinski 25

Figure 10: Using Patch Filters with Patch Wizard for Oracle Release 12 E-Business Suite This is useful for quarterly patch maintenance cycles as part of the strategy for patch maintenance tasks. It provides us with a method to track and manage patches and codelevels based on the built in intelligence via the Patch Wizard. Another excellent tool available with the Patch Wizard is the ability to assess the impact of patches before we apply them. This is critical in the event that the DBA does not have the luxury of a sandbox or test environment to apply patches. Collaborate 10 Copyright ©2010 Ben Prusinski 26

Figure 11: Using Patch Wizard for Recommended Patches After we choose the patch to analyze before we apply it to our environment we can generate the impact analysis report. Figure 12: Confirm request to analyze patch Collaborate 10 Copyright ©2010 Ben Prusinski 27

Oracle then submits the request to the concurrent management processing system. Once processed we can review the output detailed recommendations for our patch with E- Business Suite for Release 12. Collaborate 10 Copyright ©2010 Ben Prusinski 28

The Patch Bundle provides a list of XML based log files that contain the recommendations for patches and maintenance packs to be applied to the current E-Business environment for Release 12. OAM- Patch Timing reports When running adpatch you can use the patch wizard feature in OAM for Release 12 to obtain patch run time estimates via the patch timing reports. This is excellent tool to measure the requirement for a maintenance window for patches that require a downtime with the E-Business Suite. In the below example, we run a hotpatch session and examine the run time as shown below. Figure 12: Timing Report for Patch Wizard with Oracle Applications Release 12 E-Business Suite Collaborate 10 Copyright ©2010 Ben Prusinski 29

If we drill down deeper to the specifics by clicking on the details button we can observer runtimes for each phase of the patch that is currently in process. This is useful because we can measure the execution times for the C, D, G and U drivers for patches with the E-Business Suite. Furthermore, if a patch hangs or fails during execution, we can pinpoint the step where the patch failed for additional investigation. For example we can plan the maintenance window for production after testing in sandbox environment once we have gathered the metrics from the timing reports. Collaborate 10 Copyright ©2010 Ben Prusinski 30

We can assess how long it takes for all phases of the driver with the patch to execute along with any problems that occur during execution. In addition we can review the run time details for the patch applied below. Collaborate 10 Copyright ©2010 Ben Prusinski 31

The diagnostics provided by the timing report feature of the Patch Wizard is an invaluable resource for busy Oracle Applications DBAs who need an accurate way to plan maintenance windows and patch activities with the E-Business Suite. New features for Patching Oracle R12 EBS Applications The ability to monitor patch status and progress in real time is one of the new features in Release 12 for patching EBS environments with Autopatch that is helpful to Oracle Application DBAs. By using the admonitor account, you can track the status for each step in the patching process with adpatch. There are required setup details to enable this new functionality. Another feature is to use the restricted mode with Apache to allow maintenance tasks to be performed while the system is unavailable and a patch is being applied to the applications tier. Collaborate 10 Copyright ©2010 Ben Prusinski 32

Enable Restricted Mode with Apache As part of the patching strategy with Release 12 for the E-Business Suite, you may need a way to still have limited access to the Oracle Applications Suite while performing maintenance tasks. Normally for patches with the E-Business Suite you need to place the applications tier into maintenance mode and shutdown all of the application tier services. While hotpatch is available for patches that do not require a shutdown of the app tier services, larger patches such as release updates (RUP) and maintenance packs (MP) require the shutdown of these application services. Here is where the restricted mode for Apache comes into play as a useful feature. Let’s examine the steps to setup the restricted mode for Release 12 of the E-Business Suite. In Restricted Mode, only valid database users are allowed to login into OAM via a special URL and are allowed to access a limited set of features. The database role AD_MONITOR_ROLE has access to all the required database objects for Restricted Mode features. However if a valid database user who does not have the AD_MONITOR_ROLE may have further limited access to OAM functionality based on the database objects to which this user has access. How to Implement Restricted Mode with Release 12 E-Business Suite Use OAM Autoconfig editor to edit the variable "s_trusted_admin_client_nodes" to include the list of client machines that can access OAM in restricted mode. Run the Auto Config script (adautocfg.sh) on the APPL_TOP to ensure that the new settings take effect. In OAM for Release 12, navigate following the path as SYSADMIN responsibility to SiteMap > System Configuration > AutoConfig > Edit Parameters > Choose "System" parameters and then drill down to the "oa_we b_server" title search down for s_trusted_admin_client_nodes Enter comma separated values consisting of fully qualified node names (use IP address only if the address the address is static IP for a client) of client machines which will be allowed to administer E-business instance. Make sure the server can resolve the machine names. Note: to verify that the trusted client has been added, check the $iAS_TOP/Apache/Apache/conf/apps.conf for the entries that have been added for the "RESTRICT to the trusted.conf" Collaborate 10 Copyright ©2010 Ben Prusinski 33

Example: include "/u00/oracle/prodora/iAS/Apache/Apache/conf/trusted.conf" <IfDefine RESTRICT> <Location /servlets/weboamLocal> Order deny,allow Deny from all Allow from system1-sun Allow from system2-us2.us.oracle.com Allow from system3-us.us.oracle.com </Location> </IfDefine> Configure ad_monitor account Ensure that you have enabled the the monitoring user account by unlocking the "ad_monitor" user account and setting the password by using the following commands $ sqlplus system/manager SQL> alter user ad_monitor account unlock; SQL> commit; By default this ad_monitor password is set to "lizard" the first time the password is unlocked - change the password Shutdown the application Services and Configure Apache (txkrun) Use the script $INST_TOP/admin/scripts/adstpall.sh to shutdown the application tier services. Run the following script to change the Apache mode to restricted: $FND_TOP/bin/txkrun.pl –script=ChangeApacheMode Enter ‘Restrict’ when prompted by the script. Restart the apache server in Restricted Mode using the 'adaprstctl' script Collaborate 10 Copyright ©2010 Ben Prusinski 34

$ adaprstctl start Launch OAM Restricted Mode using the OAM Restricted Mode URL: http://servername.domain:[port]/servlets/weboamLocal/oam/oamLogin If an error occurs when accessing the URL such as “You are not authorized to view this page” HTTP - error 403 - Forbidden Check the Internet Explorer Tools > Internet Options > Connections > LAN Settings - Uncheck Automatically detect settings - Check Use Automatic configuration script - Uncheck Proxy Server - Delete Cookies - Delete files and all files content Login using the ad_monitor account and new password You are now in the OAM Restricted Mode. You can now access utilities on the Maintenance tab of the Site Map. You can monitor progress of your patching utilities here. After patch completion, bring down the Apache Server in Restricted Mode using the script 'adaprstctl'. Start the system using the script $INSTANCE_TOP/admin/scripts/<context name>/adstrtal.sh. Apache and items such as concurrent managers, forms will be started. Collaborate 10 Copyright ©2010 Ben Prusinski 35

Maintenance Notification Feature for Patch Downtime Release 12 of the E-Business Suite for Oracle Applications provides an excellent notification tool through the Oracle Applications Manager (OAM) GUI interface to schedule downtime events. First we create a downtime schedule by logging into OAM as SYSADMIN responsibility by navigation path of sitemap->maintenance->maintain downtime schedules as shown below. Now we can setup the schedule to notify users when the system will be unavailable for the patching. Collaborate 10 Copyright ©2010 Ben Prusinski 36

Since no schedules exist yet we need to create a new one as shown below. If we need to change the date and time of the maintenance notification we can also do so from the Schedule Downtime screen. Collaborate 10 Copyright ©2010 Ben Prusinski 37

To verify status we can confirm and observer the scheduled maintenance window. The nice addition to this tool is that users can be directed to the message before and during the patch maintenance activity is performed so that they understand the current work in progress. Collaborate 10 Copyright ©2010 Ben Prusinski 38

This provides you the busy DBA with the benefit to reassure the technical user and management community that your EBS systems are not down because of a failure but rather due to scheduled maintenance window. It builds confidence and trust and alerts critical users before hand that the system will be offline due to maintenance and not a failure. How to Disable Restricted Mode for Apache – Post Maintenance Tasks No discussion would be complete without mention on how to disable the restricted mode for Apache to return things back to normal after our patch activities have been completed. To do so you need to run the txkrun script as shown below: $FND_TOP/bin/txkrun.pl –script=ChangeApacheMode Enter the mode for Apache as ‘Normal’ Restart application tier services after the script has run by using $INST_TOP/admin/adstrtal.sh . Collaborate 10 Copyright ©2010 Ben Prusinski 39

Patch Automation with Patch Application Assistant (PAA) The patch automation tool for Release 12 is a new feature available under the $AD_TOP/bin directory in the form of a perl script called admi.pl. It provides a host of automation tasks that previously had to be manually performed by the Oracle Applications DBA. The Patch Application Assistant (PAA) can be run in either command line mode (CLI) or via the GUI graphical interface. To start the Patch Application Assistant (PAA) in graphical mode, grant X- Window access to the applmgr account and execute as shown below. Collaborate 10 Copyright ©2010 Ben Prusinski 40

From the main Patch Application Assistant (PAA) menu screen we can perform various patch related tasks: view instructions for a new patch, view generic instructions or look at all incomplete tasks that still need to be completed from earlier patches that failed to run to successful completion. For example, one of our previous patches did not run to completion. So we can use the PAA to provide us with remaining tasks that need to be performed for this patch. The PAA instructs us to first run the adgrants.sql script for this patch. After we run adgrants.sql it wants us to perform some tasks for the hrglobal driver of the patch as shown below. Collaborate 10 Copyright ©2010 Ben Prusinski 41

After we have performed these tasks to wrap up the patching, we can check the boxes in the PAA and save for future reference. Now let’s examine how to use the Patch Application Assistant in the command line mode (CLI). First we need to get some help on the available options possible with the Patch Application Assistant. Collaborate 10 Copyright ©2010 Ben Prusinski 42

The –help parameter generates all of the switches and usage for the PAA utility. To use the command line version for a patch that requires it, we run the utility as shown below from $AD_TOP/bin directory. Since there were no manual steps to be completed for the patch, the Patch Assistant notifies us of this event so we can run it via the Auto Patch (adpatch) utility. The patch_metadata.xml file contains the steps to be processed by the Patch Assistant. Collaborate 10 Copyright ©2010 Ben Prusinski 43

References My Oracle Support Formerly Oracle Metalink (www.metalink.oracle.com) or http://support.oracle.com Note: 734025.1 Using a Staged Applications System to Reduce Patching Downtime Note: 236469.1 Using Distributed AD Note: 756063.1 How to Deal with the adpatch Peformance issue : Slow, Hang or Crash ? Note: 841218.1 Release 12: Why are One-Off Patches so Large? Note: 749117.1 OAM Patch Wizard Shows Incorrect Patch Impact Analysis Result Note: 976188.1 Patch Wizard Utility Note: 976688.1 Patch Wizard FAQ Note: 862116.1 How to Eliminate Invalid Patch Wizard Recommendations Note: 352910.1 Patch Wizard Not Recommending Patches Note: 741129.1 How to Use Patch Wizard Without Setting up Internet Connection Note: 62418.1 - PATCHING/PATCHSET FREQUENTLY ASKED QUESTIONS Note: 443761.1 - How to check if a certain Patch was applied to Oracle Applications instance? Note: 459156.1 - Oracle Applications Patching FAQ for Release 12 Note: 130608.1 - AdPatch Basics Note:60766.1 - Patch Installation FAQ (Part 1) Note: 461709.1 - Oracle E-Business Suite Upgrade Guide – Plan Note: 314540.1 What Log Files are Ge nerated with Adpatch, and Where are They Located? Note: 364236.1 Managing Downtime in Restricted Mode from OAM Note: 789450.1 What is The URL When Apache Restricted Mode Is Enabled on Release 12 Collaborate 10 Copyright ©2010 Ben Prusinski 44

Oracle Release 12 E-Business Suite Patching Best Practices

by mixxalot on May 25, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 3

Statistics

Oracle Release 12 E-Business Suite Patching Best Practices — Document Transcript