Ben Prusinski is presenting on Oracle R12 E-Business Suite performance tuning. He will cover methodology, best practices, and techniques from basic to advanced. The presentation includes tuning at the infrastructure, application, and database levels with a focus on a holistic approach. Specific areas that will be discussed are concurrent manager tuning including queue size, sleep cycle, cache size, and number of processes.

1. Ben Prusinski
ben@ben-oracle.com

2. About the Speaker
 Ben Prusinski , Oracle ACE Director
 Oracle R12 EBS OCP
 More than 14 years experience with Oracle
 Author of three books on Oracle Database Technology

3. Agenda
 Oracle R12 E-Business Suite Performance Tuning
 Focus on Methodology
 Best Practices
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning

4. Methodology
•Load testing before, during, and after Go-Live
•Functional Requirements and Testing
•Map the Business and Functional to the Technical
Environment
•Business people think different than technical people
•Goal: Unify as one to resolve tuning issues (Method R)
•Stress Testing
•Integration Testing and Unit Test

5. Methodology: Load Testing
 Why do we Load test?
 Load testing allows you to generate metrics before,
during and post-Go Live to optimize performance and
availability
 Fact: you really do not know the true performance
until you test!
 Tools for Load testing: Oracle 11g Real Application Test
(RAT) Suite excellent for database tier
 Oracle 10g/11g Application Management Pack (AMP)
useful for measuring application tier performance
metrics.

6. Methodology: Functional to
Technical Specifications and Testing
 Eliminate the disconnect between the functional,
development and technical teams as soon as possible
in the deployment process to reduce bottlenecks in
performance.
 Establish stake holders to review and test
requirements from functional to tech teams
 Dependent on business requirements such as modules
deployed (eg: AP, AR, GL for Oracle Financials) to
really understand what to test and measure for
performance tuning!

7. How to Tune?
Knowing is half the battle!
Start with business problem eg) users slow response time for invoices processed
(AP/AR modules) with Oracle Financials
Drill down from macro to micro areas
Never assume!
Myth : The database is slow or “The database is guilty until proven innocent!”
Fact: it could be the application server or concurrent manager issue

8. Infrastructure First
 Hardware Tuning
 Storage and Disk Tuning
 Network Tuning
 Operating System Performance
Oracle R12 EBS Technology Stack
 Concurrent Manager Tuning
 Oracle R12 EBS Module tuning: AR, AP, GL, HR, etc.
 Oracle 10g Application Server Tuning- Apache, Java tuning
 Servlet vs. Socket mode (11i versus R12 EBS)
Database Tuning
 Instance tuning: SGA, PGA sizing is critical for database tier performance
 Statistics- FND_STATS
Application Tuning
 SQL, PL/SQL, Oracle R12 Customizations and Code Performance
Focus on Methodology
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning
Key Lesson: Do not mistake the forest for the trees and avoid tunnel vision!
Areas to Tune: Start Here!

9. Holistic Picture
 Understand the “real” problem first!
 Cary Millsap, Method R
 Oak Table “Battle against any Guess (BAAG)”- Alex Gorbachev
 Avoid the Three Blind Men and Elephant Fallacy
 Drill down to the root cause and analyze
 Duplicate the problem with test cases if possible Oracle R12 E-Business
Suite Performance Tuning
 Focus on Methodology not “Silver Bullets”
 Best Practices
 Holistic Tuning Approach
 Proactive Versus Reactive Tuning Methods
 Basic techniques to Advanced Tuning
 MOS Note: 69565.1 : A Holistic Approach to Performance Tuning
Oracle Applications Systems

10. Hardware Tuning
 Common area overlooked when sizing Oracle R12 EBS
 Starting point when sizing hardware:
 Get sizing recommendations for vendors and compare
benchmarks for performance
 Big centralized servers versus many servers, which is
best?
 For application tier, recommend multiple smaller
servers with hardware load balancers
 For database tier, advise large single server.

11. Virtualization and Performance
 Virtual servers with VMWare and Oracle VM require
additional hardware resources to optimize
performance
 Common question:
Why is my Oracle R12 EBS Slow on Virtual Servers?
 Answer: Virtual Servers consume additional memory
resources beyond just the application
 Watch out for issues with virtualization and
performance
 VMWare not officially supported by Oracle

12. Hardware and Performance: Disk I/O tuning
 Solid State Disk offers best overall disk performance
but is more expensive.
 Optimize performance with Disk I/O with RAID 0+1 or
RAID 1+0 and Avoid RAID 5 for best overall disk
performance.
 Implement Automatic Storage Management (ASM) for
Oracle 10g and Oracle 11g on database tier for
performance and availability
 Usually vendor centric depends on storage vendor eg)
EMC, SUN, HP, Hitachi, etc.

13. Hardware and Performance: Memory and CPU
 More memory not always better but you do want to
ensure that you have enough
 Sizing recommendations from vendors helps
 Initial load testing is best bet to assess if you have
sufficient memory and CPU resources.
 Tools such as top, sar, vmstat, iostat and Oracle
Enterprise Manager (OEM) are useful yardsticks to
measure resources and consumption
 Multiple cores (ie: dual core) excellent for CPU
performance but may have additional Oracle licensing
requirements.

14. Operating System Performance
 Tuning the operating system a common area that is
neglected with performance analysis!
 Varies based on platform: Linux Windows!
 MOS Note: 224176.1
How to use OS commands to diagnose Database
Performance issues?

15. Tuning Linux For Oracle R12 EBS
 Start simple with top and iostat
 Check memory and configuration for kernel settings
with dmesg and sysctl commands.
 Drill down if necessary with strace utility.

16. Linux Performance Tips
Network Performance:
 MOS Note: 274953.1 Tuning TCP/IP parameter in Linux Box for
SQLNET
Use netstat and ping to identify latency issues with networks
 MOS Note: 560590.1 How and When to use the net.ipv4.tcp_rmem and
net.ipv4.tcp_wmem Linux kernel Parameters
Kernel Tuning:
 MOS Note: 434351.1 Linux Kernel: The SLAB Allocator
Disk Tuning:
 MOS Note: 175980.1 Tuning Disk Throughput Using hdparm in Linux
 sar and iostat are your friends for identifying disk I/O contention
issues.
 Also check for disk I/O issues with SAN tools (varies with vendor such
as EMC, Hitachi, etc).

17. UNIX Performance Tips
 My Oracle Support (MOS) available from
http://support.oracle.com
 MOS Note: 144638.1 Relationship Between Common
Init.ora Parameters and Unix, Linux Kernel Parameters
 Verify that kernel parameters are set correctly and verify
with the vendor platform for UNIX.
 Assess performance with top, iostat, vmstat, sar with UNIX
systems.
 OS Watcher tool:
MOS Note: 301137.1 OS Watcher User Guide

18. Unix Performance Tuning
Find the performance bottleneck with top and ps to drill
down to the root cause
 Solaris: use truss and DTRACE for tuning system calls and
processes
http://www.sun.com/bigadmin/features/articles/dtrace_truss.jsp
 HP-UX: use the sam utility for tuning
http://docs.hp.com/en/B2355-90692/sam.1M.html
IBM AIX: SMIT (System Management Information Tool)
is useful tool for performance tuning
http://www.ibm.com/developerworks/aix/library/au-smit/index.html
Note: you may need root OS privileges to run these tools or
elevated privileges.

19. Linux Performance Tuning: top
top - 15:05:08 up 33 days, 21:49, 10 users, load average: 0.09, 0.05, 0.01
Tasks: 492 total, 2 running, 490 sleeping, 0 stopped, 0 zombie
Cpu(s): 16.9%us, 1.1%sy, 0.0%ni, 81.2%id, 0.7%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 4044596k total, 3683136k used, 361460k free, 272104k buffers
Swap: 2064376k total, 76k used, 2064300k free, 1488836k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9455 applmgr 15 0 219m 112m 13m R 27.2 2.9 6233:08 npviewer.bin
10914 applmgr 19 0 546m 50m 7704 S 1.9 1.3 0:28.01 java
13701 applmgr 15 0 13020 1316 720 R 1.9 0.0 0:00.01 top
1 root 15 0 10348 704 588 S 0.0 0.0 0:01.51 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.23 ksoftirqd/0
4 root 10 -5 0 0 0 S 0.0 0.0 0:01.17 events/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
46 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
50 root 10 -5 0 0 0 S 0.0 0.0 0:05.01 kblockd/0

20. Linux Performance Tuning: iostat
$ iostat
Linux 2.6.18-194.8.1.0.1.el5 (app01.ben.com) 09/19/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.37 0.00 99.22
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 0.08 0.29 1.79 21019459 131483716
sda1 0.00 0.00 0.00 2496 28
sda2 0.08 0.29 1.79 21016659 131483688
sdb 0.00 0.00 0.00 23141 5072
sdb1 0.00 0.00 0.00 22253 5072
dm-0 0.23 0.29 1.79 21000250 131361616
dm-1 0.00 0.00 0.00 16120 122072
$ iostat -x
Linux 2.6.18-194.8.1.0.1.el5 (db01.ben.com) 09/19/2010
avg-cpu: %user %nice %system %iowait %steal %idle
0.29 0.00 0.13 0.37 0.00 99.22
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm
%util
sda 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05
0.02
sda1 0.00 0.00 0.00 0.00 0.00 0.00 23.37 0.00 3.22 2.70
0.00
sda2 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05
0.02

21. Windows Performance Tips
 Recommend using system monitoring tools available
with Windows 2003 and Windows 2008 Server
 Sysinternal tools excellent for diagnosis as well
http://technet.microsoft.com/en-us/sysinternals/default.aspx
MOS Note: 231159.1: How to configure Windows to
improve performance database oracle
MOS Note: 1054009.1 Instructions for setting up the
Performance Monitor Counters (PERFMON) for
Windows
Perfmon (Performance Monitor) excellent tool for
tuning Windows environments.

22. Using Perfmon for Windows Performance Tuning

23. Tuning Concurrent Managers
 Concurrent Managers source of many performance
issues.
 Useful My Oracle Support Notes: 104452.1:
Troubleshooting Concurrent Manager
 My Oracle Support Note: 1057802.1
Best Practices for Performance for Concurrent
Managers in E-Business Suite
 Myth: More concurrent managers are better
 Keep it simple and use only few concurrent managers

24. Tuning Concurrent Managers
 OAM excellent starting point to find performance
bottlenecks for concurrent managers and other tuning
issues with Oracle R12 EBS.

25. Using OAM for tuning Concurrent Processing
 Check for long running concurrent requests for potential
bottlenecks by using Oracle Application Manager

26. Tuning Concurrent Managers
Oracle Applications Manager (OAM) to measure performance with Concurrent
Managers for throughput to get the big picture

27. Tuning Internal Concurrent Manager (ICM)
Tuning the Internal Concurrent Manager (ICM)
Performance is based on the following parameters for Oracle R12
 PMON sleep cycle — The number of sleep cycles that ICM
waits between polling status for failures in concurrent
processing. You should change the PMON cycle to a lower value
if your concurrent managers experience chronic terminations.
 Queue Size — is the number of PMON sleep cycles that the
ICM waits before polling for new concurrent managers.
 Sleep Time — parameter that indicates the time in seconds that
the ICM waits before polling for requests that are waiting to run.

28. Concurrent Manager Tuning
 Tune queue size for concurrent managers
 Tune sleep cycle (PMON)
 Tune cache size
 Tune number of processes
 Configure Specialization Rules
 Define Workshifts to balance performance

29. Queue Size Tuning: Concurrent Managers
 Queue size
 Default queue size is too small after R12 installation
 How to tune?
 View statistics from scripts
 Focus on tuning for ICM and Standard Managers first

30. How to Tune Sleep Cycle for PMON?
During Peak Usage for business hours:
1) Set the sleep cycle for PMON to low value when
number of concurrent requests is high.
Why? With low sleep cycle for PMON, less need for
concurrent managers to check for new requests thus
increasing throughput.
During Off hours with low business activity:
2) Recommend setting PMON sleep cycle to higher
value during non-business hours such as night time
with value such as 5 minutes.

31. Tuning PMON Sleep Cycle
How to set value for PMON sleep cycle?
 Startup parameter in adcmctl.sh script for Oracle R12
EBS
Located under $ADMIN_SCRIPTS_HOME or
$INST_TOP/admin/scripts/ directory in
applications tier.
 Parameters:
adcmctl.sh {start|stop|abort|status} [<APPS username/APPS
password>] [sleep=<seconds>] [restart=<N|minutes>]
[pmon=<iterations>] [quesiz=<pmon_iterations>] [diag=Y|N]
[wait=Y|N]
 Tune by modify the pmon and sleep parameters
 You can also use OAM to set these values as well.

32. Finding value for PMON sleep cycle
 Login to OAM as SYSADMIN then System Administrator
responsibility then navigate System Administrator->
Concurrent: Manager->Define->WorkShifts

33. Tuning cache size for Concurrent Processing
How to Tune cache size for concurrent managers?
 Increase cache size (number of requests cached) to
twice the number of target processes for each manager.
Example
If we have standard manager with 1 target process and a
cache value of 5, it will read five requests, and then
attempt to run those five requests before reading any
new requests.
Recommendation
 Use a value of 1 when defining a manager that has
long running jobs, and a higher value of 4-5 for
managers that run shorter transaction jobs.

34. Cache Size and Performance for Concurrent Processing
 Tune by setting in OAM for Oracle R12 environment

35. Tuning Concurrent Managers Number of Processes
 Examine the number of processes for Concurrent Managers
 Check the columns for actual, target and running in OAM
Login as SYSADMIN then navigate path Concurrent:Manager-> Administer

36. Specialization Rules for Concurrent Manager Performance
 Use INCLUDE and EXCLUDE rules to optimize performance since Concurrent Managers are
batch type processes
 Login as SYSADMIN and navigate System Administrator Concurrent:Manager->Define in OAM
Located under Specialization Rules

37. Setting values for number of processes with Concurrent Managers
 Located under in OAM under navigation path
System Administrator-> Concurrent: Manager->Define->WorkShifts

38. Specialization Rules Continued
 Use combined specialization rules for complex tasks

39. Workshifts for Performance with Concurrent Processing
 Setup different workshifts for balancing performance
load with different critical tasks
 Example: schedule AP during day and nightly
processing on different workshifts
 Login to OAM as SYSADMIN then navigate path
System Administrator Concurrent:Manager->
WorkShifts

40. Example: Workshifts with Concurrent Processing

41. Concurrent Manager Tuning
 Check for high number of requests
 Script to find high number of concurrent requests
SELECT v.USER_CONCURRENT_QUEUE_NAME,COUNT(PHASE_CODE) v1
FROM APPS.FND_CONCURRENT_QUEUES_V1 v,
APPS.FND_CONCURRENT_WORKER_REQUESTS r
WHERE r.queue_application_id = 0
AND r.PHASE_CODE = 'P' -- Pending Concurrent Requests
AND r.HOLD_FLAG != 'Y' -- Concurrent Requests not on hold
AND r.REQUESTED_START_DATE <= SYSDATE -- No Future Concurrent jobs
AND r.CONCURRENT_QUEUE_ID=v.CONCURRENT_QUEUE_ID
GROUP BY v.USER_CONCURRENT_QUEUE_NAME
HAVING COUNT (PHASE_CODE) >=20
/
 Use Oracle Application Manager (OAM) charts to measure
performance

42. Useful Scripts for Tuning Concurrent Managers
 Some Key Scripts for monitoring the concurrent managers:
afcmstat.sql Shows all defined managers, capacity, process id, etc.
afimchk.sql Status for Internal Concurrent Manager (ICM) & PMON
afcmcreq.sql Shows concurrent manager basics
afrqwait.sql Shows pending, held and scheduled requests
afrqstat.sql Concurrent request execution times and status
afqpmrid.sql Process id (PID) for FNDLIBR processes
afimlock.sql Find concurrency and locking issues
Execute these while connected to SQL*PLUS as APPS schema.
 The above scripts are located under the $FND_TOP/sql directory

43. Additional Scripts for Tuning Concurrent Managers and Requests
select
f.application_short_name app,
substr(p.user_concurrent_program_name,1,55) description,
substr(p.concurrent_program_name,1,20) program,
r.priority,
count(*) cnt,
sum(actual_completion_date - actual_start_date) * 24 elapsed,
avg(actual_completion_date - actual_start_date) * 24 average,
max(actual_completion_date - actual_start_date) * 24 max,
min(actual_completion_date - actual_start_date) * 24 min,
stddev(actual_completion_date - actual_start_date) * 24 stddev,
stddev(actual_start_date - requested_start_date) * 24 wstddev,
sum(actual_start_date - requested_start_date) * 24 waited,
avg(actual_start_date - requested_start_date) * 24 avewait,
c.request_class_name type
from fnd_concurrent_queues fcq,
fnd_concurrent_queue_content fcqc,
fnd_concurrent_request_class c,
fnd_application f,
fnd_concurrent_programs_vl p,
fnd_concurrent_requests r
where r.program_application_id = p.application_id
and r.concurrent_program_id = p.concurrent_program_id
and r.status_code in ('C','G','E')
and p.application_id = f.application_id
and r.program_application_id = f.application_id
and r.request_class_application_id = c.application_id(+)
and r.concurrent_request_class_id = c.request_class_id(+)
and r.request_class_application_id = fcqc.type_application_id(+)
and r.concurrent_request_class_id = fcqc.type_id(+)
and fcqc.queue_application_id = fcq.application_id(+)
and fcqc.concurrent_queue_id = fcq.concurrent_queue_id(+)
group by
c.request_class_name,
f.application_short_name,
p.concurrent_program_name,
p.user_concurrent_program_name,
r.priority
/
TOTAL AVG MAX MIN RUN WAIT #WAITED AVG REQ APP DESCRIPTION PROGRAM PRI R
UN HOURS HOURS HOURS HOURS STDDEV STDDEV HOURS WAIT TYPE
------ ---------------------------------------- -------------------- ---- ------ -- ------- ------ ------ ------ ------- --
----- --------- -------- ------------- --
SQLGL Periods - Open Period: Child Process GLOOAPC 50 1 2 .00 .00 .00 .00 .00
.00 .10 .01
BOM Cost Manager CMCTCM 50 1 ####### ###### # ##### ###### .00
.00 -16403.89 ########
ALR Check Event Alert ALECTC 1 8 .02 .00 .00 .00 .00
.01 .16 .02
OFA Asset Inventory Report FAS410 1 2 .01 .01 .01 .00 .00
.00 .01 .00
SQLGL Open Encumbrance Year GLEOYR 1 2 .09 .05 .05 .04 .01
.00 .03 .01

44. Manually Gather Statistics for Oracle R12 EBS
 FND_STATS for schema statistics collection
Example: Collect statistics for AP schema:
exec fnd_stats.gather_schema_statistics('AP');
 Collect stats for all schemas for Oracle R12 EBS:
exec fnd_stats.gather_schema_statistics('ALL');
 How to gather stats for temp tables with Oracle R12 EBS:
exec fnd_stats.gather_table_stats('<schema>','<temp_table_name>');
 Example to collect stats for temp tables in AR schema:
exec fnd_stats.gather_table_stats('AR','temp_ap');

45. Statistics Collection for Oracle R12 EBS
Why?
 Because cost based optimizer requires best execution
path with latest current statistics.
 Reference Point on how to collect statistics
My Oracle Support http://support.oracle.com
Note: 419728.1 How To Gather Statistics On Oracle
Applications 11.5.10(and above) - Concurrent
Process,Temp Tables, Manually
Do not use DBMS_STATS with Oracle R12 EBS to
collect statistics!

46. Concurrent Manager Statistics
Be sure to Gather Table Statistics on these CM tables:
 FND_CONCURRENT_PROCESSES
 FND_CONCURRENT_PROGRAMS
 FND_CONCURRENT_REQUESTS
 FND_CONCURRENT_QUEUES.

47. Gather Stats for Oracle R12 EBS
 Either run FND_STATS or Concurrent Request via OAM
System Administrator|Concurrent Requests|

48. Options for STATS Collection
• Options to set parameters for estimate percent,
parallelization, schema for Concurrent request to gather
statistics for Oracle R12 E-Business Suite
• Recommend that you do not set estimate percent too high
otherwise the load to collect stats will impact performance
and collection time.

49. Tuning Application Tech Stack
 Oracle 10g Application Server tuning issues
 Oracle HTTP Server (OHS): “Apache” web server
 Timeout settings for Apache
 Under httpd.conf file
 Configuration settings for JVM with OC4J under both
the Oracle R12 EBS Context file for application tech
stack and opmn.xml values.

50. Tuning Apache OHS
Oracle R12 EBS Application Tech Stack
 Values to check for the Apache httpd.conf file located under
$IAS_ORACLE_HOME/Apache/Apache/conf directory
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
KeepAliveTimeout 15

51. Application Tech Stack Tuning
 Tune values for timeouts to prevent and resolve errors
in performance for network delays for forms, oacore,
OC4J
 Avoid and resolve infamous “Uninterrupted Exception
150 error” when starting and stopping app tier services
 Values to check in Oracle R12 EBS Context file
 Consider shared APPL_TOP and staged APPL_TOP for
performance
 Implement Parallel Concurrent Processing (PCP) so
distribute performance load for Concurrent processing

52. Application Tech Stack tuning
 Performance metrics for OC4J groups
 Use server for concurrent processing server,
application server in split tier configuration to provide
more performance and availability
 Consider load balancer: hardware or software to
improve performance and availability
 Consider deploying Oracle RAC for scale out and scale
up in performance if you need it for large deployments
 Oracle Application Management Pack (AMP) for
tuning and monitoring Oracle R12 EBS environment

53. Load Balancing for Application tier performance
 Hardware Load Balancing options: Cisco and Big IP
some vendors that offer solutions
 Software Load Balancing
 Load balancing is complex and requires coordination
with network, security and systems administration
teams and the database team to implement
successfully.
 Reference Point:
My Oracle Support Note 727171.1:
Implementing Load Balancing On Oracle E-Business Suite -
Documentation For Specific Load Balancer Hardware

54. Software Load Balancing for Oracle R12 EBS
Application Tier performance
 Not as robust as hardware load balancers but more
cost effective.
 Uses web cache option for Oracle 10gAS application
server
 My Oracle Support Note: 380486.1
Installing and Configuring Web Cache 10g and Oracle
E-Business Suite 12

55. Software Load Balancing Oracle R12 EBS
 Requires configuration at both 10g Application Server level and within
context file for Oracle R12 EBS
The following autoconfig variables must be set within the Oracle R12 EBS
application tier context file:
 s_webentryhost- webentry points to Oracle 10gAS Web Cache 10g
Server
 s_webentrydomain- webentry domain to domain name for Oracle
10gAS Web Cache 10g Server
 s_active_webport-active webport to port for Oracle 10gAS Server
Web Cache where it listens for client requests
 s_webentryurlprotocol - webentry protocol to protocol values
that clients will use to access Oracle 10gAS Server Web Cache 10g
 s_login_page - login page to
hostname:port/OA_HTML/AppsLogin

56. Java Tuning for Oracle R12 EBS
 Tuning Java components is essential for performance
with Oracle R12 EBS
 Oracle R12 EBS is dependent on Java technology
 Default values are too low for Java with OC4J settings
 Watch out for issues with garbage collection
 Avoid full garbage collection (GC) as this impacts
performance!
 My Oracle Support Note: 567551.1:
Troubleshooting : Configuring various JVM tuning
parameters for Oracle E-Business suite 11i and R12

57. Tuning Java for R12 EBS: JVM Sizing
 How many JVMs (Java Virtual Machines) ?
OACoreGroup
 For 2-tier environment with application middle tier on separate host server and database server
 Best not to have more than 100 active users per JVM/OC4J instance for middle tier server.
No more than one active JVM/OC4J instance per CPU
 Best practice: there should be ratio of 1 JVM to 2 CPUs for middle tier server
 Use top and other system tools to check that sufficient memory is available. You want avoid memory
swapping issues.
 Common mistake for application middle tier server to not be sized correctly for memory and CPU!
FormsGroup
 For the forms servlet, the actual JVM/servlet usage is minimal, and a single JVM can normally handle
up to 100-250 forms users.
 JVM settings are configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs,
s_forms_servlet_nprocs and s_xmlsvcs_nprocs

58. Java Tuning: Oracle R12 EBS
Heap configuration for JVMs with middle tier
 OACORE starting point for tuning J VM heap sizes:
-Xmx512M -Xms256M -XX:MaxPermSize=128M
-XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB
 Goal is to minimize garbage collection frequency and to reduce user response
time by minimizing full garbage collection.
 For JDK 1.5, if the JVM detects that you have a server with multiple CPUs it
will enable Parallel Throughput Garbage Collector by default.
 Rule of thumb: number of GC (garbage collection) threads defaults to number
of CPUs per machine.
 If you run multiple JVMs on the same server, or if you have more than 2 CPUs,
to avoid excessive GC, reduce the number of GC threads:
-XX:+UseParallelGC -XX:ParallelGCThreads=2
 Two GC threads should be enough for most environments
 For heap sizes over 1GB, you can test with 4 GC threads and measure
performance.
 FormsGroup values for Garbage Collection (GC) as rule of thumb:
-Xmx256M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2
Key Oracle R12 EBS Application Middle Tier Server Autoconfig Variables for Java Heap tuning with JVMs:
 s_oacore_jvm_start_option
 s_forms_jvm_start_options

59. Tools for Measuring JVM Performance with
Oracle R12 EBS
 Excellent free tool to graph performance for JVM with
Oracle R12 EBS and garbage collection.
 http://www.tagtraum.com/gcviewer.html

60. Tuning Java for Oracle R12 EBS
 My Oracle Support Note: 362851.1: -
Guidelines to setup the JVM in Apps E-Business Suite 11i and R12
JVM Behavioral Options:
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#BehavioralOptions
JVM Options List:
http://blogs.sun.com/watt/resource/jvm-options-list.html

61. Monitoring Java Performance: Oracle R12 EBS
 Oracle 10g Grid Control (OEM) with Application
Management Pack (AMP).
Note: These items require additional cost licenses.
 MOS Note 557194.1 provides a script called
monitor_jdbc_conn.sql to monitor JDBC
connections.

62. Java Tuning for Oracle R12 EBS
 Collect metrics to analyze current performance
 Good information from ADDM and AWR in 11g
Statistic Total per Second per Trans
-------------------------------- ------------------ -------------- -------------
java call heap collected bytes 683,300,160 1,300.0 877.8
java call heap collected count 3,693,358 7.0 4.7
java call heap gc count 356 0.0 0.0
java call heap live object count 24,810 0.1 0.0
java call heap live object count 142,439 0.3 0.2
Instance Activity Stats DB/Inst: SBOX/SBOX Snaps: 274-420
java call heap live size 2,358,144 4.5 3.0
java call heap live size max 33,260,184 63.3 42.7
java call heap object count 56,992 0.1 0.1
java call heap object count max 345,992 0.7 0.4
java call heap total size 42,729,472 81.3 54.9
java call heap total size max 115,343,360 219.5 148.2
java call heap used size 6,144,496 11.7 7.9
java call heap used size max 72,864,528 138.6 93.6

63. Database Tuning for Oracle R12 EBS
 Similar to tuning regular Oracle 11g database but
unique in how statistics collected
 Remember to use FND_STATS and NOT
DBMS_STATS to collect and update statistics!
 Watch out for concurrency and locking issues for
runaway user processes!
 Schedule backup jobs off hours to avoid impact
business users
 Don’t forget to use tools like AWR and STATSPACK if
possible.

64. Database Tuning
 Suggested values for database initialization parameters for
Oracle R12 EBS
 My Oracle Support Note: 396009.1
Database Initialization Parameters for Oracle
Applications Release 12
 Oracle R12 by default installs either a 10gR2 or an 11gR1
database for fresh (new) installation.
 For example: Oracle 12.0 to 12.04 uses 10.2.x database
 While Oracle 12.1.1 and later release install Oracle 11gR1
database (11.1.x).
 Watch out for upgrade issues from 11i to R12 EBS for
database!

65. Database Parameters for Oracle R12 EBS
Parameter Name
Development or
Test Instance
11-100 Users 101-500 Users 501-1000 Users 1001-2000 Users
processes 200 200 800 1200 2500
sessions 400 400 1600 2400 5000
sga_target 1G 1G 2G 3G 14G
shared_pool_size
(csp)
N/A N/A N/A 1800M 3000M
shared_pool_reser
ved_size (csp)
N/A N/A N/A 180M 300M
shared_pool_size
(no csp)
400M 600M 800M 1000M 2000M
shared_pool_reser
ved_size (no csp)
40M 60M 80M 100M 100M
pga_aggregate_tar
get
1G 2G 4G 10G 20G
Total Memory
Required
~ 2 GB ~ 3 GB ~ 6 GB ~ 13 GB ~ 25 GB

66. Upgrade Performance considerations for Oracle R12 EBS
 When you upgrade from older releases such as 10.x and 11i
to Oracle R12 must account for performance items
 Rule based Optimizer (RBO) in 10.x and early 11i releases to
the Cost Based Optimizer (CBO) in R12
 Different way of tuning for stats and database
 Consider changes for SGA and PGA sizing
 Migrate from old tablespace model to Oracle Application
Managed Tablespaces (OATM)
 My Oracle Support Note: 761570.1
Database Preparation Guidelines for an E-Business
Suite Release 12.1.1 Upgrade

67. Q&A Discussion

68. Thank You!
 Please provide feedback
 Email: ben@ben-oracle.com
 My blog has further tips on Oracle R12 EBS
 http://oracle-magician.blogspot.com
 I will be available in the OTN ACE Director Office
Hours this week for questions