Large Scale EBS 12.2 Upgrade: Licking the Wounds

1 © 2019 FUJITSU
Unclassified
Large Scale EBS 12.2 Upgrade: Licking the Wounds
Michael Salt , Oracle Solution Architect
UKOUG
Applications Unlimited 2019

2
Safe Harbour Statement
“The views expressed in this presentation are
those of the author and do not necessarily reflect
the views of Fujitsu U.K. nor Oracle Corporation”
UKOUG Applications Unlimited 2019

3
About Me…
 24 Years Oracle Experience (v7+)
• Developer
• DBA / E-Biz DBA
• Oracle Solution Architect
• E-Business Suite Consultant
 21 Years E-Business Suite Experience (10.5+)
 21 Years Consultancy Experience:
• Sun Microsystems
• Fujitsu
 DBA 1.5
 SIG Attendance:
• RDBMS
• RAC
• Application Server
• E-Biz
 Regular Presenter At UKOUG Annual Conference
 Fujitsu Distinguished Engineer (2018)

4
About FUJITSU . . . Where we work
Approximately 140,000 Fujitsu colleagues working with customers in over 100 countries (March 2018)
*EMEIA : Europe,Middle East,India and Africa.
Oceania
3,000
EMEIA*
27,000
Asia
12,000
Japan
92,000
Americas
6,000

5
About FUJITSU . . . Working with world class partners

6
About FUJITSU . . . Working with world class customers
Public Sector Manufacturing Communications Health
Retail Financial Services Transport & Logistics Utilities

© Copyright FUJITSU 2019
7
Oracle Infrastructure and Applications
Simplify Adopt Innovate
Fujitsu Multi Cloud Services
Oracle IAAS and PAAS
Partner of the Year 2019
Oracle ACS and Premier
Support Partner Of The Year
2019
Oracle System Partner of
the Year 2019

8 UKOUG Applications Unlimited 2019
Introduction & Background
12.2

Possible Reasons for Upgrading to 12.2.X ?
Regulatory Reasons
New Product Features
New Technical Features
Better Look and Feel
Longer Term Support
Integration with Fusion Apps / SaaS
Fun of the Challenge
Applicable to our client Not Applicable to our client
AKA Online Patching!!
Enterprise Command Centres
HTML / Customisable

Roadmap from Oracle (July 2017)
https://www.oracle.com/assets/ebs-roadmap-4739519.pdf

Roadmap from Oracle (October 2018)
http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf

Implications of New Roadmap
http://www.oracle.com/us/products/applications/ebs-suite-12-2-announcement-5172019.pdf
Oracle is today announcing a “Continuous Innovation” release model for Oracle E-Business Suite 12.2,
delivering ongoing applications and underlying technology stack updates without a major upgrade. At the
same time, we are announcing Premier Support for Oracle E-Business Suite 12.2 through at least 2030. This
means that Oracle E-Business Suite customers already running on 12.2 can continue to access new
applications functionality and separately uptake later versions of underlying technology for years to come,
without the time and expense of a major release upgrade.

How it Works . . .

Part 1: The Upgrade Process
(Non-Technical)
12.2

Client Architecture (Production, Simplified)
• FUJITSU M10-4S
• 74 x 3GHz SPARC64™ X cores
• 2TB RAM
• Big SAN (not shown)
• FUJITSU M10-1
• 5 x 2.8GHz SPARC64™ X cores
• 150GB RAM
• ORACLE RDBMS 11.2.0.5
• EBS v12.1.3 Conc Manager Tier
• EBS v12.1.3 “Middle Tier”
• Large User Base

Client Upgrade Path
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
AS 10.1.3
Applications Tier
v12.1.3
Developer 10.1.2
COMMON_TOP
APPL_TOP
INST_TOP
Applications Tier
V12.2.X
Oracle HTTP Server (OHS)
WebLogic Server (WLS)
Fusion Middleware 11g
DB UPGRADE?

Upgrade Approach: Phase A (Proof of Concept)
Iteration A1
 Take Recent Clone of Production
 Follow the Oracle Upgrade Documentation
 Document as you go!
 Fix / Resolve Issues as you go!
Iteration A2 . . An
 Take a new Clone of Production
 Follow your own documentation
 Fix / Resolve Issues as you go!
 Refine Documentation
Hand-over for Functional Testing
 Formulate List of Patches
 Update Master Documentation
Our Appendix!!

Upgrade Approach: Phase B (Refinement & Timing)
Review Oracle Updates
 New Patches & Documents
Iteration B1
 Take new Clone of Production
 Use Production-Like Hardware
 Use Different Resources
 Follow & Refine own documentation
 Conduct Performance Testing
Iteration B2 . . Bn
 Repeat of above until clean runs achieved
 Add all timing information to documentation
 Tune the migration process!!
Hand-over for Functional Testing

Upgrade Approach: Phase C (Dress Rehearsals)
Plan for Production-Like Upgrade Schedule
Iteration C1
 Take new Clone of Production
 Use Production-Like Hardware
 Use Multiple Resources
 Follow your own Documentation
 Verify Timings in Documentation
 Refine documentation
Iteration C2 . . Cn (optional)
 Repeat of above until clean runs achieved
 Establish Final Baseline (no more patches)
 Last iteration should be done 24x7
Hand-over for Final Functional Testing

Upgrade Approach: Phase D (The Real Thing)
Perform Pre-Upgrade Steps in Production
Shut down Production
Back up Production
Perform Upgrade Steps
 Use Multiple Resources (24x7 Tag Teams)
 Follow your own documentation
Internal Testing & Backup
 Technical & Functional
 Take Backup / Snapshot
External User Testing
 Small set of Functional Product Champions
Go / No-Go!

Upgrade Gotchas Part 1
Process Can Take Many Months
Oracle Things Change Frequently
 AD/TXK DELTA Patches
 Release Update Packs (RUPs)
 Upgrade Patches
 Database Patches
 Application Patches
 Etc etc
Project Scope Creep
 New Products Added
 Database Upgrade
Project Resources
OBSOLETE!
OBSOLETE!
OBSOLETE!
OBSOLETE!
OBSOLETE!
CLIENT DECISION

Upgrade Gotchas Part 2
Staging Software Area – Invalid Symbolic Link
 References to “/d1/R12/ab/apps/V122X39”
 Fixed manually
Too Much Memory (yes, I said too much!!)
 If >2TB Free Space under “/tmp” it causes problems!
 Problem with old 10.1 Oracle Universal Installer, used by:
• Rapid Wizard Installer (fails after several hours)
• Adop “fs_clone” Operations (fails after an hour)
applmgr> $ORACLE_HOME/oui/bin/runInstaller -help
Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ...
applmgr> echo $?
255
applmgr> export TEMP=/var/tmp ## Required to fix 10.1 OUI
### Now re-check OUI
applmgr> $ORACLE_HOME/oui/bin/runInstaller -help
Preparing to launch Oracle Universal Installer from /tmp/OraInstall . . . Please wait ...
Oracle Universal Installer, Version 10.1.0.6.0 Production
. . .
applmgr> echo $?
0
Problem!
OK

Moving Goalposts – Some Things to watch for . . .
Changes To Watch For . . .
Migration Phase
A B C D
Upgrade Documentation (1320300.1 / 133070x.1 / 1349240.1 / E48839-nn / etc)   ? 
Latest JDK Updates (1530033.1 / 1439822.1)    
Consolidated Update Patch (CUPn – Section 3 of 1320300.1)   ? 
Pre-Install Patches Report [Quarterly] (1448102.2)   ? 
Latest Release Update Pack (RUP 12.2.n - Section 2 of 1320300.1)  ? ? 
Latest AD/TXK Delta Rollup Patches (1583092.1 / 1617461.1)    
Database PSUs & Overlay Patches (1147107.1 / ETCC Patch 17537119)    
Latest Patches and Technology Bug Fixes (1594274.1 / ETCC Patch 17537119)    
VERY COOL UTILITY!

Upgrade Process Tuning
Determine Target
 How much downtime is acceptable to client?
 Target Timeline (Bank Holiday Weekends)
 How much testing time to be built into plan?
First Iterations (Early Learning)
 Fix-on-Fail vs Raise Calls with Oracle
 Tasks Run Serially
 Focus on Long-Running tasks (30mins+)
 Look out for Performance Patches
Later Iterations (Get Smarter)
 Determine Steps which can be done in Advance
 Determine Steps which can be run in Parallel
 Test, Time, Tune, Repeat . . .

Upgrade Process Tuning Example 1 (bsu.sh)
WebLogic Patching Utility
During Upgrade, needs to be run many times *
 Remove Operations: 8 + Install Operations: 7
 Before Tuning: 5 mins per operation (approx)
 Time Taken: 75 mins!!
Done in Preparation Phase, but. . . future?
Two Steps For Tuning
 Edit “bsu.sh” & Increase Heap Size: 512MB  2GB+
 Apply Patch 27238412 (replaced 26664545) [Doc ID 2271366.1]
After Tuning:
 Operations now take approx 1min
 Total Time Reduced to 15 mins (Saving 1 hour!!)
* Numbers may vary: This was from our upgrade!

Upgrade Process Tuning Example 2 (Scale-up)
We will see later that we need multiple “OACORE” JVMs
Method for adding these is cumbersome (see below)
Developed Scripted Approach
Also re-used the same script for Cloning (Post Go-Live)
Managing Configuration of Oracle HTTP Server and Web Application Services in Oracle EBS Release 12.2 (Doc ID 1905593.1)
1). Logon to first target node
2). Run:
$ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl
ebs-create-managedserver -contextfile=<CONTEXT_FILE>
-managedsrvname=oacore_server2 -servicetype=oacore
-managedsrvport=7203 -
logfile=<APPLRGF>/TXK/addMS_oacoreserver2.log
3). Wait a few mins for start-up / shutdown of managed server
4). Run:
$ perl <FND_TOP>/patch/115/bin/txkSetAppsConf.pl -
contextfile=<CONTEXT_FILE>
-configoption=addMS -oacore=testserver.example.com:7203
5). Repeat for each managed server (x42 for us)
1). From Primary Application Tier Node, Run:
applmgr> cd /stage/R12.2/XTRA_SCRIPTS
applmgr> add_new_managed_servers.sh 7 <node1> <node2> <node3>
<node4> <node5> <node6>
. . .
ManagedServer oacore_server8 created successfully.
. . .
Complete, check logfiles to ensure success.

Final Upgrade Process: Prepare Phase (High Level)
Download and Prepare All Software into Staging Area
DB Upgrade: Preparation Phase
 Install 12c RDBMS Software
 Apply Latest PSU and Associated Patches
 Prepare 12c Init.ora Files (Upgrade & Live Running)
Apps Upgrade: Preparation Phase
 Prepare Apps Staging Area
 Run Pre-Upgrade Tasks
 Create Upgrade Filesystems
 Patch the Apps Tech Stacks
 Merge AD12.2 Upgrade Driver
 Merge Consolidated Upgrade Patches
 Prepare Online Patching Enablement Patch and Report

Final Upgrade Process: Execute Phase (High Level)
System Close & Backup/Snap
DB Upgrade: Execute Phase
 Run Pre-Upgrade Tool
 Run “dbua” Upgrade Assistant
 Run Manual Upgrade Post-Steps
Backup/Snap
Apps Upgrade: Execute Phase
 Run Pre-Upgrade Tasks (Tech and Functional)
 Close Down App
 Apply Manual Work-Arounds
 Run 12.2 Upgrade Steps (inc AD/TXK + R.U.P. etc)
 Run Customisation Tasks
 Run Post-Upgrade Tasks (inc Scale-out and Scale-up)

Final Upgrade Process: Hand-Over Phase (High Level)
System Close
Full System Backup / Snapshot
Open System for Internal Testing
Open System for Limited End-User Testing
Open System Fully
Take cover . . . !

Upgrade Summary
What Worked Well What Didn’t Work Well
Fully Documented Process Relatively Early Adopters (Bugs & Changes)
Iterative Structured Approach Project Scope Creep (DB Upgrade)
Performance Testing* & Tuning Release Update Pack 12.2.6 vs 12.2.7
Functional Testing Managing Resources
Upgrade Completed On Time On Cut-Over Go Live Issues! (More on this later!!)
Automated / Scripted Many Steps No Party!!

Part 2: EBS 12.2 Tuning (Technical)
12.2

E-Business Suite R12.2: Application Tuning
Test Harness
 Fully Sized Database
 Production-Like Hardware
 Load Simulator (Silk Performer)
Key Tuneable Parameters
 Database Parameters
 OHS Parameters
 Number of “OACORE” JVMs
 Connection Pool Settings
 WebLogic Logging
 JVM Parameters
Other Items For Tuning
 Inefficient “Profiles” (later slide)
12.2

EBS Tuning: Database Parameters
Based on the Previous “init.ora”
Changes made for 11g  12c DB Upgrade
Changes made for 12.1.3  12.2.X App Upgrade
Refer to notes: [396009.1] & [174605.1]
 Use “bde_chk_cbo.sql” provided in [174605.1]
Some changes needed just for Upgrade Period (see below)
WebLogic Connection Pool reduced backend connections
 Enables Reduction to “Processes” & “Sessions” vs 12.1.3
 See later slide on this!
12.2
ora[12c]$ cd $ORACLE_HOME/dbs
ora[12c]$ vi init$ORACLE_SID.ora
#############################################################################
# Added For 12.2 Upgrade - Remove Afterwards
#############################################################################
log_buffer=100M
job_queue_processes=30
parallel_max_servers = 60
max_dump_file_size = UNLIMITED # Prevents delays in adop enab (Doc ID 1946877.1)

EBS Tuning: OHS Parameters
12.2
vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/httpd.conf
under worker MPM. . . set:
PARAMETER NEW VALUE DEFAULT VALUE COMMENTS
ServerLimit 8 25
ThreadLimit 128 unset <--- This is needed to support ThreadsPerChild
StartServers 8 5
MaxClients 1024 625
MinSpareThreads 100 25 <--- Gives warning when close to max-clients (<100 free)
MaxSpareThreads 1024 75 <--- Will prevent any server shutdowns
ThreadsPerChild 128 25
MaxRequestsPerChild 0 0 <--- No sign of memory leak on httpd.worker processes, so OK!
KeepAliveTimeout 15 15 <--- Default (unchanged)
MaxKeepAliveRequests 0 0 <--- Default (unchanged)
Note: Also need to change "AcceptMutex fcntl" to AcceptMutex pthread
And comment out the LockFile entries!

EBS Tuning: “OACORES” & DB Connection Pool
Trials Revealed ~300 Concurrent Users* per JVM was OK
Found 300 JDBC Connections per JVM worked well
Connection Pool Harvesting (tweaked)
Do the maths . .
12.2
*What’s a concurrent user? Your mileage will vary!

EBS Tuning: Load Balancer “Fencing”
By Default, Oracle will use software load balancing
 Incoming connection to OHS redirects to any managed server!
 Can result in a lot of cross-tier network traffic
 Not necessary if using Hardware Load Balancer
 Less control in some maintenance scenarios
Our solution was to “fence” . . .
 After scale-out / scale-up . . .
12.2
applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/mod_wl_ohs.conf
[Ensure OHS2 File has only WebLogicCluster commands referring to <Apps Node 1>]
etc
applmgr> vi $FMW_HOME/webtier/instances/*OHS[2-7]/config/OHS/*/apps.conf
[Ensure OHS2 File has only BalancerMember commands referring to <Apps Node 1>]
etc

EBS Tuning: WebLogic Logging
By Default, WebLogic Logging is Verbose and Duplicated:
12.2
[OACORE Server Logs – Standard Out]
applmgr> wc -l oacore_serverX.out
1831 oacore_serverX.out
applmgr> egrep -c 'at |Truncated.|Caused By:|java:' oacore_serverX.out
1282
[OACORE Server Logs – Normal Log File]
applmgr> wc -l oacore_serverX.log
17895 oacore_serverX.log
applmgr> egrep -c ' at |java:|Caused By:' oacore_serverX.log
3639
applmgr> grep -c "<Info>" oacore_serverX.log
13741
[AdminServer Logs – Domain Log File]
applmgr> wc -l EBS_domain_XXXXX.log
59713 EBS_domain_XXXXX.log
applmgr> grep -c "<Error>" EBS_domain_XXXXX.log
872
applmgr> egrep -c ' at |ServletException|java:|Caused By:' EBS_domain_XXXXX.log
55491
Majority (70%) of Standard Out
Messages are Truncated Stack Dumps
Majority (75%) of Standard Log
Messages are “INFO” Level
Majority (94%) of Domain Log Messages
are “ERROR” and Stack Dumps

EBS Tuning: WebLogic Logging (2)
Here’s how to reduce this overhead:
12.2
Change From
“Info” to “Notice”
Select
Select
Change From
“Trace” to “Notice”

EBS Tuning: WebLogic Logging (3)
Here’s how to reduce this overhead (cont):
12.2
Change From
“Notice” to “Warning”
Change From
“Notice” to “Critical”
Reduce From
10 to 1
UNTICK!

EBS Tuning: JVM Parameters (OACORE)
During Trials, experimented with many JVM parameters:
 Increased Heap Size 1GB  3GB
 Used Garbage First (G1) Collector
 Set “DevPollDisabled” (doesn’t scale without this on some platforms)
 Used “Tiered Compilation” (more on this later)
12.2

EBS Tuning: Changing JVM Parameters - Gotcha
In 12.1.3:
 Change JVM Parameters via “oacore_jvm_start_options”
 Run Auto-Config
In 12.2:
 We have same parameter – yay!
 However, only used when creating a new managed server!
 Therefore need to do this before scale-out & scale-up!
To Change in 12.2 after Managed Server Created:
 Edit in WebLogic Console under Config  Server Start  Arguments
 Or, edit the “$EBS_DOMAIN_HOME/config/config.xml” file
• Be Careful! – Take backup first!
 Or, use the “adProvisionEBS.pl ebs-set-managedsrvproperty” utility
• Quite Messy, all arguments listed in full, refer to Doc ID 1905593.1
12.2

First Encounter with Problematic Profiles
Discovered During Load Testing
 Good Performance for first few hours of test
 Performance Rapidly Declined
 Observed the following:
 Increasing size of result cache, 200MB  500MB did not fix issue!
12.2
Latch Name Tot Wait (s) Get Requests Miss % SLEEPS
------------------------------------ ------------ ------------ ------ ---------
Result Cache: RC Latch 4000631.1 1268891679 125.4 59713271
row cache objects 76299.2 7095624871 13.9 50778810
cache buffers chains 1570.9 1.3298E+11 0.2 4289651
In memory undo latch 752.9 643026407 0.8 1069567
parameter table management 479.2 95434614 5.9 479927
NAMESPACE STATUS CACHE_ID
---------- --------- ------------------------------
NAME
--------------------------------------------------------------------------------
NUM_RESULTS AVG_SCAN_PER_ENTRY MAX_SCAN_PER_ENTRY TOT_SCAN_CNT TOT_BLK_CNT
----------- ------------------ ------------------ ------------ -----------
PLSQL Published a77mb1r8mfppd0ktc3712ykmt2
"APPS"."FND_PROFILE"::11."GET_USER_VALUE"#4b60a496cb9dc4ee #505
206195 22 638031 4486880 206195
ICX: Language
ICX:Session Timeout
FND: Override Directory
PA: Tasks to Display for Expenditure Entry
etc etc

Problematic Profiles – Root Cause (1)
Oracle’s Code Doesn’t Scale!
Here’s how “FND_PROFILE.VALUE” function works:
 As of: $Header: AFPFPROB.pls 120.43.12020000.7 2016/06/23
 Moves from User  Responsibility  Application  . . .  Site
 Each Sub-Function Uses RESULT_CACHE
• get_user_value
• get_resp_value
• get_appl_value
• etc
12.2
** The algorithm is such:
** 1. Using the profile option information acquired at declaration,
** determine if the profile exists in FND_PROFILE_OPTIONS. If it does
** not, return NULL.
** 2. If it exists and the user_enabled_flag is Y, get user level value.
** 3. If there was no user-level value, proceed to check the applicable
** levels based on the profile option hierarchy.
** 4. If no value was obtained in all the other levels, check the site
** level if it is enabled for the profile option.
*Refer to Bug 20198791 & Doc ID 2099303.1 for some background information

Problem for Systems with many Active Users / Resps
In R12.2 We Have Around 10,000 Profile Options
 Even if only 50 of these were actually enabled at user / resp level
 By ½ of all users and ½ of all responsibilities
 That would be over 2 Million Result Cache entries (blocks)
 Each Result Cache Block is 1kb in size, so that’s . .
12.2

Problem is not:
 Number of Profile Options that are Set
 Paradoxically, better if set at User / Resp levels!
Problem is:
 Number of profile options that have ability to be set
 . . . as NULL results still stored in result cache
 . . . so what to do?
Initial Solution was simple:
 Disable Result Cache for “User Level” Lookups
 Not documented*
12.2
ora[12c]$ vi $ORACLE_HOME/dbs/init$ORACLE_SID.ora
#############################################################################
# Added For 12.2 - Enable Only After DB Upgrade Complete Inc Patch 17305006
#############################################################################
_result_cache_black_list=(a77mb1r8mfppd0ktc3712ykmt2)
*The black_list feature has some refs, see bug 17305006, PSU merge patches needed before DB 12.2

Part 3: Post Go-Live (Technical)
12.2

Day 1 – Slow System & Complete Panic

Day 1 – Main Issues
Integration Issues
 Some custom static files needed amending
 GOTCHA: Files changed under “OA_HTML” Need WebLogic Bounce!
Lots of Performance Bugs
 Especially
• Email Centre – Stuck Threads on Apps Servers
• Timecards – High “Log File Sync” due to excessive logging
 None of these appeared during testing!
 Load Test Suite focused on “Majority” Transactions
 Some “esoteric features” did not scale!
Return of the “Profiles” Issue
 This was “Major Issue” for us!
 Engaged with Senior Levels in Oracle
12.2

Week 1 – Example Issue: Timecards*
Lots of messages appearing in “FND_LOG_MESSAGES”
Piece of very bad code located (after manual search)
Even though mobile timecards not in use!
Common Code (clearly shows debugging):
No Debug Check! So executed for every timesheet
AND, “write_log” function had Explicit Commit!
12.2
$Header: hxclockapi.pkb 120.6.12020000.4 2014/01/30 09:54:17 asrajago ship
-- Bug 18113601
hxc_mob_utils.write_log('p_resource_id :'||p_resource_id);
hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type);
hxc_mob_utils.write_log('p_start_time : '||p_start_time);
hxc_mob_utils.write_log('p_stop_time : '||p_stop_time);
hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id);
hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn);
hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time);
hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id);
-- Bug : Timecard Mobile
-- Check if this timecard is locked in In Transit Tables.
hxc_mob_utils.write_log(dbms_utility.format_call_Stack);
hxc_mob_utils.write_log(p_process_locker_type);
hxc_mob_utils.write_log('Checking mobile lock ');
hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));
*This issue was present as of 12.2.7 Codeline!

Week 1 – Example Issue: Timecards* (2)
We Told Oracle of the Fault in their Code!
Fujitsu Proposed Interim Solution:
2 Lines of Code
 Big Difference!
12.2
IF g_debug THEN
-- Bug 18113601
hxc_mob_utils.write_log('p_resource_id :'||p_resource_id);
hxc_mob_utils.write_log('p_process_locker_type :'||p_process_locker_type);
hxc_mob_utils.write_log('p_start_time : '||p_start_time);
hxc_mob_utils.write_log('p_stop_time : '||p_stop_time);
hxc_mob_utils.write_log('p_time_building_block_id : '||p_time_building_block_id);
hxc_mob_utils.write_log('p_time_building_block_ovn : '||p_time_building_block_ovn);
hxc_mob_utils.write_log('p_expiration_time : '||p_expiration_time);
hxc_mob_utils.write_log('p_row_lock_id : '||p_row_lock_id);
hxc_mob_utils.write_log(dbms_utility.format_call_Stack);
hxc_mob_utils.write_log(p_process_locker_type);
hxc_mob_utils.write_log('Checking mobile lock ');
hxc_mob_utils.write_log('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));
END IF;

Week 1 – Example Issue: Timecards* (3)
Oracle’s Final Solution:
Moved away from problematic “write_log”
Included the “debug” check first!
12.2
$Header: hxclockapi.pkb 120.6.12020000.7 2018/01/04 16:59:42 asrajago ship $
IF g_debug THEN
hr_utility.trace('p_resource_id :'||p_resource_id);
hr_utility.trace('p_process_locker_type :'||p_process_locker_type);
hr_utility.trace('p_start_time : '||p_start_time);
hr_utility.trace('p_stop_time : '||p_stop_time);
hr_utility.trace('p_time_building_block_id : '||p_time_building_block_id);
hr_utility.trace('p_time_building_block_ovn : '||p_time_building_block_ovn);
hr_utility.trace('p_expiration_time : '||p_expiration_time);
hr_utility.trace('p_row_lock_id : '||p_row_lock_id);
hr_utility.trace(dbms_utility.format_call_Stack);
hr_utility.trace(p_process_locker_type);
hr_utility.trace('Checking mobile lock ');
hr_utility.trace('Checking mobile lock '||p_start_time||'-'||TO_CHAR(p_stop_time,'dd-MON-YYYY HH24:MI'));
END IF;

Month 1 – Main Issue: Profiles
Still issues with Result Cache (esp for “RESP”)
Added “RESP” and “SITE” into Black List
Result Cache Performance Improved
System Just About Coped, But . . .
12.2
Elapsed Elapsed Time
Time (s) Executions per Exec (s) %Total %CPU %IO SQL Id
---------------- -------------- ------------- ------ ------ ------ -------------
250,607.2 113 2,217.76 21.5 25.9 .0 6sc8jn1bssj0c
select count(*) from per_workforce_x hr,per_phones pho,hr_lookups hrl,hr_locati
ons loc,hr_territories_v tr WHERE hr.person_id = pho.parent_id(+) AND pho.phon
e_type(+) = 'W1' AND pho.parent_table(+) = 'PER_ALL_PEOPLE_F' AND sysdate BETWEE
N pho.date_from(+) AND nvl(pho.date_to(+), sysdate) AND pho.phone_type = hrl
149,828.2 129,441,219 0.00 12.9 8.0 .0 7qwsx7rw0s3a5
SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_USER_V WHERE PROFILE_OPTION_
ID = :B3 AND APPLICATION_ID = :B2 AND LEVEL_VALUE = :B1 AND PROFILE_OPTION_VALUE
IS NOT NULL
54,806.2 156,747,772 0.00 4.7 25.3 .0 gj84g2yx56u2q
SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_SITE_V WHERE PROFILE_OPTION_
ID = :B2 AND APPLICATION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL
49,114.6 108,659,530 0.00 4.2 25.1 .0 9k7gd9pn661pj
SELECT PROFILE_OPTION_VALUE FROM FND_PROFILE_VALUES_RESP_V WHERE PROFILE_OPTION_
ID = :B4 AND APPLICATION_ID = :B3 AND LEVEL_VALUE = :B2 AND LEVEL_VALUE_APPLICAT
ION_ID = :B1 AND PROFILE_OPTION_VALUE IS NOT NULL

Month 1 – Main Issue: Profiles (2)
During peak times, system went into meltdown!
 Huge Mutex Issues – Massive ramp up in CPU
 Needed Complete Bounce to resolve!
Oracle Advised “Beef Up” RC & remove black lists!
 We were reluctant, but client insisted!
 So we did it in the evening . . . but . . .
12.2
Total Wait Wait % DB Wait
Event Waits Time (sec) Avg(ms) time Class
------------------------------ ----------- ---------- ---------- ------ --------
latch free 54,089,624 181.9K 3.36 77.3 Other
DB CPU 39.2K 16.7
db file sequential read 2,273,797 9988.5 4.39 4.2 User I/O
log file sync 79,076 2933.1 37.09 1.2 Commit
Pct Avg Wait Pct
Get Get Slps Time NoWait NoWait
Latch Name Requests Miss /Miss (s) Requests Miss
------------------------ -------------- ------ ------ ------ ------------ ------
Result Cache: RC Latch 550,997,581 26.2 0.4 2.E+05 1 0.0
PL/SQL Entry Subprogram % Activity
----------------------------------------------------------------- ----------
APPS.PAY_ARCHIVE.PROCESS_EMPLOYEE 55.13
APPS.FND_PROFILE.VALUE 53.07

Oracle keen to get to bottom of Result Cache Issue!
1st Problem: Invalidations
 Single change to any profile value invalidates ALL profile caches!
 But, we have many changes every day . . .
12.2
SQL> select APPLICATION_ID,PROFILE_OPTION_ID,LEVEL_ID,LEVEL_VALUE,
2 to_char(LAST_UPDATE_DATE,'HH24:MI:SS')
3 from apps.fnd_profile_option_values
4 where LAST_UPDATE_DATE>sysdate-(8/24)
5* order by LAST_UPDATE_DATE;
APPLICATION_ID PROFILE_OPTION_ID LEVEL_ID LEVEL_VALUE TO_CHAR(
-------------- ----------------- ---------- ----------- --------
0 13791 10004 315365 08:43:12
690 4306 10004 359689 09:24:39
178 7646 10004 359689 09:24:39
690 4306 10004 283772 09:31:48
. . .
690 4306 10004 365973 11:00:46
690 4292 10004 365973 11:00:46
. . .
178 3798 10004 481853 14:07:23
0 3773 10004 481853 14:07:35
178 4108 10004 481853 14:08:55
800 1204 10004 241598 15:22:18
49 rows selected.
Show all profile
changes in last 8 hours
Almost all changes are
“USER” Level (10004)
Sometimes changes
are bunched together!
Lots of changes!

How to cut-down on Invalidations?
 Can’t stop users from making changes!
 However, found a “Password Sync” job:
• Synchronising between SSO and E-biz
• Developed by Oracle Consulting
• Ran Every 15 minutes and made profile changes!!
• Changed to run once in the evening
• Improved situation, but ad hoc user changes still prevalent
12.2

I Suggested Design Change to Oracle:
 Replace “FND_PROFILE_OPTION_VALUES” Table with:
• FND_PROFILE_OPTION_VALUES_USER
• FND_PROFILE_OPTION_VALUES_RESP
• FND_PROFILE_OPTION_VALUES_SITE
• etc
 Add “FND_PROFILE_OPTION_VALUES” View
• Based on “UNION ALL” of all of the above
• For backwards compatibility
 Change “FND_PROFILE” RC functions:
• Each would select from their own Specific Table
 Would give Result Cache independence
 Even better* option would be Partitioned Table!
 Oracle said . . . NO! Too much change for them!
12.2
*Need to check result cache invalidation based on partition?

2nd Problem: Too many Result Cache Entries
 Remember: NULL Entries are held in cache
 Remember: User  Responsibility  Application  . . .  Site
 Problem gets worse as you fix first problem!
Major Performance Problems if Result Cache gets full
 Oracle’s solution was to increase RC size
Poor Performance when lots of entries in Result Cache
 Especially when invalidations occurring!
 Due to long hash chain length.
12.2

How to reduce number of Result Cache entries?
 Check which Profiles hitting cache hardest
 Check Profiles not in use at various levels
 Where not in use  DISABLE
12.2
ICX: Language disable RESP/USER
ICX:Session Timeout disable USER only
FND: Override Directory disable RESP/USER
PA: Tasks to Display for Expenditure Entry disable RESP/USER
RCV: Fail All ASN Lines if One Line Fails disable RESP/USER
Achieved 10% Reduction
in RC Entries with just
these 5!

How to manage Result Cache when getting full?
 Manual Methods:
 Fujitsu Developed Automatic Solution:
• Monitor Result Cache on periodic basis (e.g. 5mins)
• Invalidate Specific Cache Entries when >50,000 (approx 50MB)
12.2
SQL> exec dbms_result_cache.flush();  Whole Cache
SQL> exec dbms_result_cache.INVALIDATE_OBJECT(<cache_id>);  Single Entry
Wed Apr 18 16:23:13 2018
FUJITSU RC MONITOR
Invalidating large (63796) cache id: a77mb1r8mfppd0ktc3712ykmt2
Wed Apr 18 16:34:16 2018
Thread 1 cannot allocate new log, sequence 651
Private strand flush not complete
Beginning log switch checkpoint up to RBA [0x28b.2.10], SCN: 287205410218
Wed Apr 18 16:34:19 2018
Thread 1 advanced to log sequence 651 (LGWR switch)
Wed Apr 18 16:38:14 2018
FUJITSU RC MONITOR
Wed Apr 18 16:39:22 2018
Completed checkpoint up to RBA [0x28b.2.10], SCN: 287205410218
Wed Apr 18 16:53:15 2018
FUJITSU RC MONITOR

So what happened next?
Fujitsu Spent Time Looking at “Mutex Meltdown”
Issue suspected as “hot objects” [Ref: 2194448.1]
Three things done for “USER” and “RESP” Queries:
 1). Make SQLs as fast as possible with Indexing & Baselines
 2). Set “Hot Object Copies” parameter to 16
 3). Add SQLs as “HOT” during database start-up:
12.2
### Run MarkHot Procedure To Ensure Multiple Copies Of USER/RESP Busy SQL
sqlplus -s / as sysdba <<EOF
set heading off
select APPS.FND_PROFILE.VALUE_SPECIFIC('BOM:STANDARD_ITEM_ACCESS',1234,1234)
from dual;
begin
sys.dbms_shared_pool.markhot (hash=>'cb73920f7392208c7b731d3df80c0d45',
namespace=>0, global=>true);
sys.dbms_shared_pool.markhot (hash=>'39966b9bcb73f953991ded4d686306b1',
namespace=>0, global=>true);
end;
/
exit;
EOF

New Solution Implemented
 Black Lists added again for “USER” and “RESP” Queries
 Good, Consistent and Stable Performance
Hidden Snares
 Non-Standard Solution (we are using 2 sticking plasters)
 Relies on “black_list” Procedure
• Not standard until DB12.2
• Have to request new merge patch with each PSU
• Changes in Oracle code could generate new signatures!
 Relies on “markhot” Procedure
• Beware manual DB bounces without using script
• Changes in Oracle code could generate new signatures!
 Code is still Inefficient
• Exasperated with use of “HR_SECURITY”
12.2

Future Plans For Profiles Issue
Move back towards Result Cache
Trim down as many unnecessary profiles as possible
 60% of hits already removed!
Keep pushing Oracle for Enhanced Profiles Solution
Use new Fujitsu RC Monitor to maintain stability
Keep tuning “HR Security” problematic queries
12.2

Online Patching
12.2
HEY – Wait a minute!
This is a 12.2 Presentation and we have
NOT spoken about online patching!!!

Much Later – Online Patching Issue
12.2
NOTE:
Logarithmic Scale!
*Bug 22568797 - latch: row cache objects contention with editions.
*Bug 20161956 - Adop:prepare Should Throw Warning If No. Of Editions Exceed Threshold Limit

Much Later – Online Patching Issue (2)
Oracle’s internal guides say . . .
 We need to actualize when we have >20 Editions
However, for busy systems need lower threshold!
 Patch 22568797 Should Help, but . . .
 We now “actualize_all” after every patching session!
12.2
SQL> select count(*) from dba_editions;
COUNT(*)
----------
5
SQL> select count(*) from dba_objects;
COUNT(*)
----------
459168
SQL> select count(*) from obj$;
COUNT(*)
----------
1203883
COUNT(*)
----------
1
COUNT(*)
----------
459164
COUNT(*)
----------
460160

How we patch with “actualize_all”
12.2
$ adop phase=prepare
$ adop phase=apply patches=xxx,xxx...
$ adop phase=finalize
[bring down system manually]
$ adop phase=cutover mtrestart=no
$ . /home/$LOGNAME/.profile
$ adop phase=cleanup
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cleanup cleanup_mode=full
SQL> ?/rdbms/admin/utlrp
[bring up system manually]
SQL> exec dbms_stats.gather_fixed_objects_stats();
SQL> exec dbms_stats.gather_dictionary_stats();
Can take over 2 hours (fs_clone)
10mins  25mins
~10mins (much faster than “mtrestart”)
5mins  10mins
10mins++ (depending on number of patches)
30mins+
35mins+
10mins+
25mins+
Can take over 3 hours!
10mins  30mins
~15mins
~8mins
~30mins
~10mins

How we patch with “actualize_all”
12.2
$ adop phase=apply patches=xxx,xxx...
$ adop phase=finalize
[bring down system manually]
$ adop phase=cleanup
$ adop phase=actualize_all
$ adop phase=finalize finalize_mode=full
$ adop phase=cleanup cleanup_mode=full
SQL> ?/rdbms/admin/utlrp
[bring up system manually]
SQL> exec dbms_stats.gather_fixed_objects_stats();
SQL> exec dbms_stats.gather_dictionary_stats();
~6 Hours for Cut-Over
Rather than ~1 Hour
without Actualize!

Patches Required before using “ACTUALIZE_ALL”:
 25099339:
• Trigger On Editioning View Not Firing When Updated Via Inline View
 27997875
• Regression Of Base Bug 25919622 (Invalid Objects Observed In Ebs
Environment After Drop Edition ORA$BASE)
Beware, you may see:
But*, if you check the logs:
12.2
The cleanup phase completed successfully.
[ERROR] ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx
ORA-04052: error occurred when looking up remote object MYOBJECT@XXXXXXXXX
ORA-00604: error occurred at recursive SQL level 3
ORA-12541: TNS:no listener, SQL[V_20180920_1105]: drop PACKAGE "APPS"."xxxobject"
[ERROR] Could not drop covered object [V_20180920_1105] APPS.xxxobjectxxx
(PACKAGE): ORA-04045: errors during recompilation/revalidation of APPS.xxxobjectxxx
*Many possible reasons why clean-up may silently fail, example above is reference to remote object

Belt and Braces Check:
12.2
COUNT(*)
----------
5
COUNT(*)
----------
459168
COUNT(*)
----------
1203883
SQL> select * from dba_editions;
EDITION_NAME PARENT_EDITION_NAME usable
--------------------------------------------------------------------------------
V_20180920_1105 V_20180630_1205 YES
V_20180922_1132 V_20180920_1105 YES
V_20180630_1205 YES
COUNT(*)
----------
1
COUNT(*)
----------
459164
COUNT(*)
----------
460160

Much Later – JVM Tuning
Remember this from slide 38?
We hadn’t noticed . .
12.2
Warnings reported in oacore_serverX.out
Java HotSpot(TM) 64-Bit Server VM warning:
CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning:
Try increasing the code cache size using -XX:ReservedCodeCacheSize=

Much Later – JVM Tuning (2)
Fair Re-Trial, Compare Apples with Apples
Non-Tiered (Default) Compilation is Best!
 But, you may need to set -XX:ReservedCodeCacheSize=nnn
 We also needed to increase our heap size 3GB  4GB
12.2

Much Later – Quarterly PSUs
Great idea to maintain latest PSUs (eg. DB and WLS)
But, be careful not to go “too recent”*
Example Ref: 2443497.1
Download Latest ETCC Patch 17537119
Simple check of mappings file . . .
 E.g. ETCC Patch: 29th Nov 2018:
12.2
applmgr> cd $INSTALL_BASE/fs_ne/etcc/mw
applmgr> grep WLS_HOME txk_R1220_MT_mappings.xml | tail -1
<Technology name="WLS_HOME" version="10.3.6.0.181016">
applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem
applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | tail -1
<Technology name="DB_HOME" version="12.1.0.2.181016ProactiveBP">
applmgr> cd $INSTALL_BASE/fs_ne/etcc/db/onprem
applmgr> grep "DB_HOME" txk_R1220_DB_mappings.xml | grep 12.1 | grep -v Pro | tail -1
<Technology name="DB_HOME" version="12.1.0.2.181016">
*Wait ~1 month after release! Example Ref: 2443497.1 – Download latest EBS Technology Codelevel Checker

Part 4: Wrap Up!
12.2

Be Aware (bigger software stack / more ports)

Cool Features of R12.2
WebLogic Server
 Good visibility of System Performance
 Even better when integrated with OEM!
 Faster Start-up / Shutdown!
 Better connection pooling!

WebLogic Admin Console (Summary of Servers)
QUICK HEALTH VIEW
USEFUL!

Dealing with Stuck Threads
12.2

Thanks For Your Time!
Any Questions?
Contact: michael.salt@uk.fujitsu.com

Large Scale EBS 12.2 Upgrade: Licking the Wounds

Recommended

Recommended

More Related Content

Similar to Large Scale EBS 12.2 Upgrade: Licking the Wounds

Similar to Large Scale EBS 12.2 Upgrade: Licking the Wounds (20)

Recently uploaded

Recently uploaded (20)

Large Scale EBS 12.2 Upgrade: Licking the Wounds