Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

  • Got a new Iphone 6 in just 7 days completing surveys and offers! Now I'm just a few days away from completing and receiving my samsung tablet! Highly recommended! Definitely the best survey site out there! ◆◆◆
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this


  1. 1. Online Patching Challenges with EBS 12.2.5 Sean Braymen & Miklos Lorant
  2. 2. About National Instruments Founded: 1976 Corporate Headquarters: Austin, Texas Industry: Test and Measurement Our Mission: We equip engineers and scientists with systems that accelerate productivity, innovation, and discovery. Revenue: $1.23 billion in 2015 Global Operations: Approximately 7,100 employees; operations in almost 50 countries IT Development Centers: • Austin, Texas • Debrecen, Hungary • Penang, Malaysia • San José, Costa Rica • Shanghai, China
  3. 3. Agenda • Review of 12.2 Architecture and ADOP Procedures • 12.2 Architectural Changes and Philosophy • Online Patching o File System Editioning o Edition Based Redefinition o ADOP Patching Cycle • Custom Development Standards • Full vs Runtime Compliance • Custom Development Options • Lessons Learned • Database Object Grants • DB_Domain • Executing Autoconfig • Administering Application Nodes • Challenges with Online Patching • ADOP - Useful Utilities
  4. 4.
  5. 5. Copy the Code, NOT the Data • EBS uses both the file system and the database to store the code and data that make up the application • Code o Stored both on the File system & In the Database o Any code object changed in a patch is copied • Data o Stored both on the File system & In the Database o Application data is NOT copied by a patch
  6. 6. Downtime Redefined as “Cutover” • Cutover is the time taken to switch users from the production system to the newly patched copy • Cutover changes the unit of measure for downtime • Measured in minutes NOT hours or days • Cutover time is very predictable • The time taken to bounce the Middle Tiers Changing the Unit of Measure for Downtime
  7. 7. Downtime Limited to Short Cutover • Users are logged off the production Instance • When users reconnect they are directed to the newly patched system E-Business Suite 12.2.3 Patches •12.2.4 •12.2.5 •…. E-Business Suite 12.2.3 Production Patched Production
  8. 8. • Downtime Patches • No online users • Wall clock time very important • Consumes all resources • Upgrade designed to run as fast as possible • Online Patches • User remain online • Wall clock time is no longer an overriding concern • Online users share resources • Data upgrades designed to not affect the running application Changing the Patching Paradigm
  9. 9. Online Patching - High level Architecture
  10. 10. 12.2 Architecture Dual File System and Edition-Based Redefinition Synchronization Managed Automatically Edition-Based Redefinition Non-Editioned File System Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP Oracle HTTP Server (OHS) WebLogic Server (WLS) Run File System Developer 10.1.2 COMMON_TOP APPL_TOP INST_TOP Oracle HTTP Server (OHS) WebLogic Server (WLS) Patch File System PATCH_TOP APPL_TOP_NE LOGS
  11. 11. File System Editioning
  12. 12. • Run file system • Used by online users • Stores a complete copy of all Applications and Middle Tier code • Logically mapped to either fs1 or fs2 • Patch file system • Used by patching tools • Stores a complete copy of all Applications and Middle Tier code • Logically mapped to either fs1 or fs2 • Non-Editioned file system • Used for data files e.g.: data import/export files, log files, report output files • Only stores data files Online Patching uses a Dual File System fs1 and fs2 switch Run and Patch designation during the cutover phase of an Online Patching cycle fs1 Run fs1 Cutover fs1fs2 PatchPatch fs2 Run
  13. 13. Database Edition-Based Redefinition
  14. 14. 11gR2 Edition-Based Redefinition (EBR) • Key enabling technology for Online Patching • Enables the online upgrade of the application definition • Allows an application to efficiently store multiple copies of its code and schema in the same database • Provides an isolation mechanism that allows pre-upgrade and post-upgrade schemas to co-exist • Changes to database objects are made in the isolation of an “Edition” • Changes to database objects do not affect the running application
  15. 15. Online Patching is Enabled by 11gR2 Edition-Based Redefinition Multiple Copies of Database Code Objects Can Coexist • Client code connects to an “edition” of the database • Run Edition o Used by online users o Never changed by a patch • Patch Edition o Used by patching tools o Changes do not affect the running application • Patch Edition becomes the Run Edition at Cutover 15 DBA Business Users
  16. 16. Online Patching Cycle
  17. 17. Online Patching Cycle - Overview Understanding the Online Patching Cycle • The Basics • Remove obsolete objects Cleanup • Restart application on Patch Edition Cutover • Compile invalid Objects • Wait for a good downtime window Finalize • Apply one or more patches to the Patch Edition Apply • Copy the production application code • Create a new Patch Edition in the database Prepare Users Online Users OnlineUsers Offline • Online Patching is used to apply all patches in 12.2 • Online Patching cycle includes 5 major phases • Application is only offline during the Cutover phase
  18. 18. Custom Development with Online Patching
  19. 19. Full Compliance vs Runtime Compliance • Full Compliance • Development meets all standards to support runtime and online patching operations • Specialized deploy methodology to implement changes in the patch edition/file system o See Doc ID 1577661.1 for all online patching compliance standards • All Oracle eBusiness Suite products meet full compliance • Runtime Compliance • A subset of the full compliance standards that allow a product to operate correctly in R12.2 • Customizations and third-party tools may only meet runtime compliance
  20. 20. Pro/Cons of Online Patching Full Compliance • Oracle’s Deployment Methodology • Doc ID 1577661.1 – Section 1.4 • Standard application development is executed directly against the run edition • Option #1: Apply Customizations as Online Patch. When application development is completed, a custom patch is created and deployed in an ADOP cycle • Option #2: Apply customizations as a downtime patch. Custom code is added to run edition in a downtime window • Advantages • Higher guarantee of system stability • Disadvantages: • Changes to the Applications are less agile – Both paths require downtime • Option #1 requires the creation of patches. Steps vary based on AD/TXK patchset level and database objects being modified. See Part 2 of Doc ID 1577661.1
  21. 21. Disadvantage of Runtime Compliance • Deployment Methodology • Deploy code directly to the run edition • Advantages • Higher degree of flexibility. Code changes can be done without system wide downtime • Removes the learning curve for developers • Disadvantages • Extra maintenance required before ADOP cycle. Must execute a process to synchronize run and patch file systems o Delta 7 and below: fs_clone or custom process o Delta 8: Possibly taken care of in the prepare phase • Higher risk of system stability issues during code deploys
  22. 22. Lessons Learned
  23. 23. Database Object Grants • Doc ID 1987947.1 • Symptoms • Grants cannot be performed in the run edition when the application is being used • Granting privileges on an object may cause object invalidations in the current edition • Problem Description • When grants are given to "Stub objects" it invalidates all dependent stub objects. • “Stub objects" are simply pointers to an actual object definition from an ancestor edition that is still being inherited by the given edition.
  24. 24. Database Object Grants Solutions • Option #1: Execute an ADOP cycle when granting privileges on the APPS schema to other users • Option #2: Leverage new API for granting privileges with minimum invalidations • API: AD_ZD.GRANT_PRIVS
  25. 25. Database Object Grants • Syntax: ad_zd.grant_privs (X_PERMISSIONS in varchar2, X_OBJECT_NAME in varchar2, X_GRANTEE in varchar2, X_OPTIONS in varchar2 default NULL); • X_PERMISSIONS: comma-seperated permission list • X_OBJECT_NAME: name of APPS object or APPS synonym • X_GRANTEE: grantee schema or role • X_OPTIONS: grant options, example: 'WITH GRANT OPTION'
  26. 26. DB_Domain • Specifies the logical location of the database within the network structure • Rapid Clone and ADOP are making assumptions about the DB_Domain • Assumption #1: DB_Name + DB_Domain is less than 8 characters o If this condition is not met, Rapid clone on the Application tier will not execute o Workaround: Unset DB_Domain, run rapid clone, reset DB_Domain, run Autoconfig • Assumption #2: The DB_Domain has the same value as the host domain o In our case: – DB_Domain = WORLD – Host Domain = o ADOP builds a JDBC connect string during the ADOP patching cycle o Workaround: Create a service for db_name + host_domain –
  27. 27. ExecutingAutoconfig • Application services should be down while running autoconfig • Autoconfig has to be executed on ALL application tier nodes • Autoconfig cannot be executed in parallel on multiple application tier nodes • While the autoconfig script will show a successful completion, data is not properly inserted into FND_OAM_CONTEXT_FILES • This will lead to potential issues during the next ADOP cycle in which nodes may need to be abandoned
  28. 28. AdministeringApplication Nodes • Adding Application Nodes • Doc ID 1375769.1 • Application nodes cannot be added or removed when application services are running • Major steps • Setup ssh between Primary APPS Tier and New Node • Run appsTier for both patch and run file system • Create Oracle Inventory • Prepare Pair File for adclonectx
  29. 29. AdministeringApplication Nodes Example: [Instance Specific] s_temp=/opt/apps/r12/d3ebiz/apps/inst/fs2/inst/apps/d3ebiz_d3ebizui1/temp s_contextname=d3ebiz_d3ebizui1 s_hostname=d3ebizui1 s_cphost=d3ebizcm1 s_webhost=d3ebizui1 s_config_home=/opt/apps/r12/d3ebiz/apps/inst/fs2/inst/apps/d3ebiz_d3ebizui1 s_inst_base=/opt/apps/r12/d3ebiz/apps/inst s_display=d3ebizui1:0.0 s_forms-c4ws_display=d3ebizui1:0.0 s_ohs_instance=EBS_web_d3ebiz_OHS2 s_webport=8000 s_http_listen_parameter=8000 s_https_listen_parameter=4443
  30. 30. AdministeringApplication Nodes Example: [Services] s_web_applications_status=enabled s_web_entry_status=enabled s_apcstatus=enabled s_root_status=enabled s_batch_status=enabled s_other_service_group_status=disabled s_adminserverstatus=disabled s_web_admin_status=disabled
  31. 31. AdministeringApplication Nodes Execute the Add Node Utility /opt/apps/r12/<db_name>/apps/<run_fs>/FMW_Home/webtier/p erl/bin/perl ./ addnode contextfile=<full path of the cm1 node's Run File System Context file> pairsfile=<full path of custom pairsfile> dualfs=yes
  32. 32. AdministeringApplication Nodes • Changing apps.conf and mod_wl_ohs.conf perl $FND_TOP/patch/115/bin/ -contextfile=<full path of the <host> Run File System Context file> -configoption=addMS -oacore=<new host><oacore port number> -oafm=<new host><oafm port number> -forms=<new host><forms port number> -formsc4ws=<new host><formsc4ws port number>
  33. 33. AdministeringApplication Nodes • Change context file entries (if needed), eg: Change following entries from enabled to disabled on CM nodes s_oacorestatus s_formsstatus s_oafmstatus • Run autoconfig
  34. 34. AdministeringApplication Nodes • Adding Managed Servers • Doc ID 1905593.1 • New managed servers can be added when the applications are running • However, they will not be used until the applications are stopped and autoconfig is executed perl $AD_TOP/patch/115/bin/ ebs-create-managedserver -contextfile=$CONTEXT_FILE -managedsrvname=forms_server17 -servicetype=forms -managedsrvport=7412 -logfile=/tmp/addforms_server.log
  35. 35. AdministeringApplication • E-Business Suite Technology Codelevel Checker (ETCC) • Rapid Install StartCD 50 for EBS 12.2 comes with a database • Since Feb. 22, 2016, Rapid Install StartCD 51 is available. It comes with DB. You can download as Patch 22066363 • Since Sept 15, 2016, 12.2.6 is available. New Installation and Upgrade (upgrading to Oracle E-Business Suite Release 12.2.6 from Release 11i, 12.0, or 12.1) customers should use StartCD (Doc ID 2114016.1) • Weblogic upgrade to was not possible till we applied all ETCC recommended patches
  36. 36. Challenges with Online Patching • Serial number in context file should match the value of database copy. If not, adop can fail (Doc ID 1916658.1) • adop phase=fs_clone • In our environment, performance is at least 3 hours • The addition of more application nodes increases the time for the procedure to complete • Recommended to be executed after any configuration changes • Not all files get transferred on slave nodes o • adop fs_clone allnode=yes|no • adop fs_clone force=yes
  37. 37. Challenges with Online Patching • How it supposed to look like (eg.:): SSO_SERVER_RELEASE=11 SSO_SERVER_URL= SSO_SERVER_TYPE=OAM LOG_CONFIG_FILE=DEFAULT APPL_SERVER_ID=3DA0E4FD00B9625FE05400144FFB48B032374020223902728214531018323218 OAM_LOGOUT_URL=/accessgate/logout CONNECTION_REF=jdbc/OAEADatasourceDS WEBGATE_LOGOUT= oracle.apps.fnd.sso.WebEntries=DEFAULT • How it was looking like (everywhere, except master node): SSO_SERVER_RELEASE= SSO_SERVER_URL= SSO_SERVER_TYPE= LOG_CONFIG_FILE=DEFAULT APPL_SERVER_ID= OAM_LOGOUT_URL= CONNECTION_REF= WEBGATE_LOGOUT= oracle.apps.fnd.sso.WebEntries=DEFAULT
  38. 38. Online Patching – Useful Utilities • adopmon - utility to monitor the progress of adop session. • adopreports - Online Patching Diagnostic Reports utility, $AD_TOP/bin/adopreports, can be used to help diagnose issues or simply gather information about the status of your system. • - before prepare, cutover or fs_clone, this tool can validate the instance if it is ready for those phases perl $AD_TOP/patch/115/bin/ -contextfile=$RUN_CONTEXT_FILE - patchctxfile=$PATCH_CONTEXT_FILE -phase=fs_clone -logloc=/tmp • adopscanlog - utility makes the work easier and analyzes adop log directories for errors and warnings
  39. 39. Questions