2. 2ni.com
About National Instruments
Founded: 1976
Corporate Headquarters: Austin, Texas
Industry: Test and Measurement
Our Mission:
We equip engineers and scientists with systems that
accelerate productivity, innovation, and discovery.
Revenue: $1.23 billion in 2015
Global Operations: Approximately 7,100 employees;
operations in almost 50 countries
IT Development Centers:
• Austin, Texas
• Debrecen, Hungary
• Penang, Malaysia
• San José, Costa Rica
• Shanghai, China
3. 3ni.com
Agenda
• Review of 12.2 Architecture and ADOP Procedures
• 12.2 Architectural Changes and Philosophy
• Online Patching
o File System Editioning
o Edition Based Redefinition
o ADOP Patching Cycle
• Custom Development Standards
• Full vs Runtime Compliance
• Custom Development Options
• Lessons Learned
• Database Object Grants
• DB_Domain
• Executing Autoconfig
• Administering Application Nodes
• Challenges with Online Patching
• ADOP - Useful Utilities
5. 5ni.com
Copy the Code, NOT the Data
• EBS uses both the file system and the database to store the code and data that
make up the application
• Code
o Stored both on the File system & In the Database
o Any code object changed in a patch is copied
• Data
o Stored both on the File system & In the Database
o Application data is NOT copied by a patch
6. 6ni.com
Downtime Redefined as “Cutover”
• Cutover is the time taken to switch users from the production system to the
newly patched copy
• Cutover changes the unit of measure for downtime
• Measured in minutes NOT hours or days
• Cutover time is very predictable
• The time taken to bounce the Middle Tiers
Changing the Unit of Measure for Downtime
7. 7ni.com
Downtime Limited to Short Cutover
• Users are logged off the production Instance
• When users reconnect they are directed to the newly patched system
E-Business Suite
12.2.3
Patches
•12.2.4
•12.2.5
•….
E-Business Suite
12.2.3
Production Patched Production
8. 8ni.com
• Downtime Patches
• No online users
• Wall clock time very important
• Consumes all resources
• Upgrade designed to run as fast
as possible
• Online Patches
• User remain online
• Wall clock time is no longer an
overriding concern
• Online users share resources
• Data upgrades designed to not
affect the running application
Changing the Patching Paradigm
12. 12ni.com
• Run file system
• Used by online users
• Stores a complete
copy of all Applications
and Middle Tier code
• Logically mapped to
either fs1 or fs2
• Patch file system
• Used by patching tools
• Stores a complete
copy of all Applications
and Middle Tier code
• Logically mapped to
either fs1 or fs2
• Non-Editioned file
system
• Used for data files
e.g.: data import/export
files, log files, report
output files
• Only stores data files
Online Patching uses a Dual File System
fs1 and fs2 switch Run and Patch designation during the cutover phase of an Online
Patching cycle
fs1
Run
fs1
Cutover
fs1fs2
PatchPatch
fs2
Run
14. 14ni.com
11gR2 Edition-Based Redefinition (EBR)
• Key enabling technology for Online Patching
• Enables the online upgrade of the application definition
• Allows an application to efficiently store multiple copies of its code and schema in the same
database
• Provides an isolation mechanism that allows pre-upgrade and post-upgrade
schemas to co-exist
• Changes to database objects are made in the isolation of an “Edition”
• Changes to database objects do not affect the running application
15. 15ni.com
Online Patching is Enabled by 11gR2 Edition-Based Redefinition
Multiple Copies of Database Code Objects Can Coexist
• Client code connects to an “edition”
of the database
• Run Edition
o Used by online users
o Never changed by a patch
• Patch Edition
o Used by patching tools
o Changes do not affect the running application
• Patch Edition becomes the
Run Edition at Cutover
15
DBA
Business Users
17. 17ni.com
Online Patching Cycle - Overview
Understanding the Online Patching Cycle
• The Basics
• Remove
obsolete
objects
Cleanup
• Restart
application on
Patch Edition
Cutover
• Compile invalid
Objects
• Wait for a good
downtime
window
Finalize
• Apply one or
more patches to
the Patch Edition
Apply
• Copy the
production
application
code
• Create a new
Patch Edition
in the database
Prepare
Users Online Users OnlineUsers Offline
• Online Patching is used to apply all patches in 12.2
• Online Patching cycle includes 5 major phases
• Application is only offline during the Cutover phase
19. 19ni.com
Full Compliance vs Runtime Compliance
• Full Compliance
• Development meets all standards to
support runtime and online patching
operations
• Specialized deploy methodology to
implement changes in the patch
edition/file system
o See Doc ID 1577661.1 for all online
patching compliance standards
• All Oracle eBusiness Suite products
meet full compliance
• Runtime Compliance
• A subset of the full compliance
standards that allow a product to
operate correctly in R12.2
• Customizations and third-party tools
may only meet runtime compliance
20. 20ni.com
Pro/Cons of Online Patching Full Compliance
• Oracle’s Deployment Methodology
• Doc ID 1577661.1 – Section 1.4
• Standard application development is executed directly against the run edition
• Option #1: Apply Customizations as Online Patch. When application development is
completed, a custom patch is created and deployed in an ADOP cycle
• Option #2: Apply customizations as a downtime patch. Custom code is added to run edition in
a downtime window
• Advantages
• Higher guarantee of system stability
• Disadvantages:
• Changes to the Applications are less agile – Both paths require downtime
• Option #1 requires the creation of patches. Steps vary based on AD/TXK patchset level and
database objects being modified. See Part 2 of Doc ID 1577661.1
21. 21ni.com
Disadvantage of Runtime Compliance
• Deployment Methodology
• Deploy code directly to the run edition
• Advantages
• Higher degree of flexibility. Code changes can be done without system wide downtime
• Removes the learning curve for developers
• Disadvantages
• Extra maintenance required before ADOP cycle. Must execute a process to synchronize
run and patch file systems
o Delta 7 and below: fs_clone or custom process
o Delta 8: Possibly taken care of in the prepare phase
• Higher risk of system stability issues during code deploys
23. 23ni.com
Database Object Grants
• Doc ID 1987947.1
• Symptoms
• Grants cannot be performed in the run edition when the application is being used
• Granting privileges on an object may cause object invalidations in the current edition
• Problem Description
• When grants are given to "Stub objects" it invalidates all dependent stub objects.
• “Stub objects" are simply pointers to an actual object definition from an ancestor edition
that is still being inherited by the given edition.
24. 24ni.com
Database Object Grants
Solutions
• Option #1: Execute an ADOP cycle when granting privileges on the APPS
schema to other users
• Option #2: Leverage new API for granting privileges with minimum
invalidations
• API: AD_ZD.GRANT_PRIVS
25. 25ni.com
Database Object Grants
• Syntax:
ad_zd.grant_privs
(X_PERMISSIONS in varchar2,
X_OBJECT_NAME in varchar2,
X_GRANTEE in varchar2,
X_OPTIONS in varchar2 default NULL);
• X_PERMISSIONS: comma-seperated permission list
• X_OBJECT_NAME: name of APPS object or APPS synonym
• X_GRANTEE: grantee schema or role
• X_OPTIONS: grant options, example: 'WITH GRANT OPTION'
26. 26ni.com
DB_Domain
• Specifies the logical location of the database within the network structure
• Rapid Clone and ADOP are making assumptions about the DB_Domain
• Assumption #1: DB_Name + DB_Domain is less than 8 characters
o If this condition is not met, Rapid clone on the Application tier will not execute
o Workaround: Unset DB_Domain, run rapid clone, reset DB_Domain, run Autoconfig
• Assumption #2: The DB_Domain has the same value as the host domain
o In our case:
– DB_Domain = WORLD
– Host Domain = natinst.com
o ADOP builds a JDBC connect string during the ADOP patching cycle
o Workaround: Create a service for db_name + host_domain
– d1ebiz.natinst.com
27. 27ni.com
ExecutingAutoconfig
• Application services should be down while running autoconfig
• Autoconfig has to be executed on ALL application tier nodes
• Autoconfig cannot be executed in parallel on multiple application tier nodes
• While the autoconfig script will show a successful completion, data is not properly
inserted into FND_OAM_CONTEXT_FILES
• This will lead to potential issues during the next ADOP cycle in which nodes may need to
be abandoned
28. 28ni.com
AdministeringApplication Nodes
• Adding Application Nodes
• Doc ID 1375769.1
• Application nodes cannot be added or removed when application services are running
• Major steps
• Setup ssh between Primary APPS Tier and New Node
• Run adpreclone.pl appsTier for both patch and run file system
• Create Oracle Inventory
• Prepare Pair File for adclonectx
31. 31ni.com
AdministeringApplication Nodes
Execute the Add Node Utility
/opt/apps/r12/<db_name>/apps/<run_fs>/FMW_Home/webtier/p
erl/bin/perl ./adclonectx.pl
addnode contextfile=<full path of the cm1 node's Run
File System Context file>
pairsfile=<full path of custom pairsfile>
dualfs=yes
32. 32ni.com
AdministeringApplication Nodes
• Changing apps.conf and mod_wl_ohs.conf
perl $FND_TOP/patch/115/bin/txkSetAppsConf.pl
-contextfile=<full path of the <host> Run File System Context file>
-configoption=addMS
-oacore=<new host>.natinst.com:<oacore port number>
-oafm=<new host>.natinst.com:<oafm port number>
-forms=<new host>.natinst.com:<forms port number>
-formsc4ws=<new host>.natinst.com:<formsc4ws port number>
33. 33ni.com
AdministeringApplication Nodes
• Change context file entries (if needed), eg:
Change following entries from enabled to disabled on CM nodes
s_oacorestatus
s_formsstatus
s_oafmstatus
• Run autoconfig
34. 34ni.com
AdministeringApplication Nodes
• Adding Managed Servers
• Doc ID 1905593.1
• New managed servers can be added when the applications are running
• However, they will not be used until the applications are stopped and autoconfig is
executed
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl
ebs-create-managedserver -contextfile=$CONTEXT_FILE
-managedsrvname=forms_server17 -servicetype=forms
-managedsrvport=7412 -logfile=/tmp/addforms_server.log
35. 35ni.com
AdministeringApplication
• E-Business Suite Technology Codelevel Checker (ETCC)
• Rapid Install StartCD 50 for EBS 12.2 comes with a 11.2.0.3 database
• Since Feb. 22, 2016, Rapid Install StartCD 51 is available. It comes with 12.1.0.2 DB.
You can download as Patch
22066363
• Since Sept 15, 2016, 12.2.6 is available. New Installation and Upgrade (upgrading to
Oracle E-Business Suite Release 12.2.6 from Release 11i, 12.0, or 12.1) customers
should use StartCD 12.2.0.51. (Doc ID 2114016.1)
• Weblogic 11.1.1.7 upgrade to 11.1.1.9 was not possible till we applied all ETCC
recommended patches
36. 36ni.com
Challenges with Online Patching
• Serial number in context file should match the value of database copy. If not,
adop can fail (Doc ID 1916658.1)
• adop phase=fs_clone
• In our environment, performance is at least 3 hours
• The addition of more application nodes increases the time for the procedure to complete
• Recommended to be executed after any configuration changes
• Not all files get transferred on slave nodes
o oaea_wls.properties
• adop fs_clone allnode=yes|no
• adop fs_clone force=yes
37. 37ni.com
Challenges with Online Patching
• How it supposed to look like (eg.:):
SSO_SERVER_RELEASE=11
SSO_SERVER_URL=https://winnie-app.natinst.com:443
SSO_SERVER_TYPE=OAM
LOG_CONFIG_FILE=DEFAULT
APPL_SERVER_ID=3DA0E4FD00B9625FE05400144FFB48B032374020223902728214531018323218
OAM_LOGOUT_URL=/accessgate/logout
CONNECTION_REF=jdbc/OAEADatasourceDS
WEBGATE_LOGOUT=
oracle.apps.fnd.sso.WebEntries=DEFAULT
• How it was looking like (everywhere, except master node):
SSO_SERVER_RELEASE=
SSO_SERVER_URL=
SSO_SERVER_TYPE=
LOG_CONFIG_FILE=DEFAULT
APPL_SERVER_ID=
OAM_LOGOUT_URL=
CONNECTION_REF=
WEBGATE_LOGOUT=
oracle.apps.fnd.sso.WebEntries=DEFAULT
38. 38ni.com
Online Patching – Useful Utilities
• adopmon - utility to monitor the progress of adop session.
• adopreports - Online Patching Diagnostic Reports utility,
$AD_TOP/bin/adopreports, can be used to help diagnose issues or simply
gather information about the status of your system.
• txkADOPValidations.pl - before prepare, cutover or fs_clone, this tool can
validate the instance if it is ready for those phases
perl $AD_TOP/patch/115/bin/txkADOPValidations.pl -contextfile=$RUN_CONTEXT_FILE -
patchctxfile=$PATCH_CONTEXT_FILE -phase=fs_clone -logloc=/tmp
• adopscanlog - utility makes the work easier and analyzes adop log
directories for errors and warnings
Online patching has 5 cycles:
First, the system is prepared by creating a copy of production code and creating a new edition in the database
Then, you apply one or more patches to the patch edition with adop
Next, the finalize phase is executed which includes compiling invalids
Then you wait for a time to cutover from the run filesystem and edition to the updated patch filesystem and edition
Cutover – is in red as this is the only time the system is down
You execute cutover, when it is convenient for your environment
Finally you can execute the cleanup phase any time after cutover
