Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oracle R12 EBS Performance Tuning

20,679 views

Published on

presentation from Oracle OpenWorld 2010 conference.

Published in: Technology, News & Politics

Oracle R12 EBS Performance Tuning

  1. 1. Ben Prusinski ben@ben-oracle.com
  2. 2. About the Speaker  Ben Prusinski , Oracle ACE Director  Oracle R12 EBS OCP  More than 14 years experience with Oracle  Author of three books on Oracle Database Technology
  3. 3. Agenda  Oracle R12 E-Business Suite Performance Tuning  Focus on Methodology  Best Practices  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning
  4. 4. Methodology •Load testing before, during, and after Go-Live •Functional Requirements and Testing •Map the Business and Functional to the Technical Environment •Business people think different than technical people •Goal: Unify as one to resolve tuning issues (Method R) •Stress Testing •Integration Testing and Unit Test
  5. 5. Methodology: Load Testing  Why do we Load test?  Load testing allows you to generate metrics before, during and post-Go Live to optimize performance and availability  Fact: you really do not know the true performance until you test!  Tools for Load testing: Oracle 11g Real Application Test (RAT) Suite excellent for database tier  Oracle 10g/11g Application Management Pack (AMP) useful for measuring application tier performance metrics.
  6. 6. Methodology: Functional to Technical Specifications and Testing  Eliminate the disconnect between the functional, development and technical teams as soon as possible in the deployment process to reduce bottlenecks in performance.  Establish stake holders to review and test requirements from functional to tech teams  Dependent on business requirements such as modules deployed (eg: AP, AR, GL for Oracle Financials) to really understand what to test and measure for performance tuning!
  7. 7. How to Tune? Knowing is half the battle! Start with business problem eg) users slow response time for invoices processed (AP/AR modules) with Oracle Financials Drill down from macro to micro areas Never assume! Myth : The database is slow or “The database is guilty until proven innocent!” Fact: it could be the application server or concurrent manager issue
  8. 8. Infrastructure First  Hardware Tuning  Storage and Disk Tuning  Network Tuning  Operating System Performance Oracle R12 EBS Technology Stack  Concurrent Manager Tuning  Oracle R12 EBS Module tuning: AR, AP, GL, HR, etc.  Oracle 10g Application Server Tuning- Apache, Java tuning  Servlet vs. Socket mode (11i versus R12 EBS) Database Tuning  Instance tuning: SGA, PGA sizing is critical for database tier performance  Statistics- FND_STATS Application Tuning  SQL, PL/SQL, Oracle R12 Customizations and Code Performance Focus on Methodology  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning Key Lesson: Do not mistake the forest for the trees and avoid tunnel vision! Areas to Tune: Start Here!
  9. 9. Holistic Picture  Understand the “real” problem first!  Cary Millsap, Method R  Oak Table “Battle against any Guess (BAAG)”- Alex Gorbachev  Avoid the Three Blind Men and Elephant Fallacy  Drill down to the root cause and analyze  Duplicate the problem with test cases if possible Oracle R12 E-Business Suite Performance Tuning  Focus on Methodology not “Silver Bullets”  Best Practices  Holistic Tuning Approach  Proactive Versus Reactive Tuning Methods  Basic techniques to Advanced Tuning  MOS Note: 69565.1 : A Holistic Approach to Performance Tuning Oracle Applications Systems
  10. 10. Hardware Tuning  Common area overlooked when sizing Oracle R12 EBS  Starting point when sizing hardware:  Get sizing recommendations for vendors and compare benchmarks for performance  Big centralized servers versus many servers, which is best?  For application tier, recommend multiple smaller servers with hardware load balancers  For database tier, advise large single server.
  11. 11. Virtualization and Performance  Virtual servers with VMWare and Oracle VM require additional hardware resources to optimize performance  Common question: Why is my Oracle R12 EBS Slow on Virtual Servers?  Answer: Virtual Servers consume additional memory resources beyond just the application  Watch out for issues with virtualization and performance  VMWare not officially supported by Oracle
  12. 12. Hardware and Performance: Disk I/O tuning  Solid State Disk offers best overall disk performance but is more expensive.  Optimize performance with Disk I/O with RAID 0+1 or RAID 1+0 and Avoid RAID 5 for best overall disk performance.  Implement Automatic Storage Management (ASM) for Oracle 10g and Oracle 11g on database tier for performance and availability  Usually vendor centric depends on storage vendor eg) EMC, SUN, HP, Hitachi, etc.
  13. 13. Hardware and Performance: Memory and CPU  More memory not always better but you do want to ensure that you have enough  Sizing recommendations from vendors helps  Initial load testing is best bet to assess if you have sufficient memory and CPU resources.  Tools such as top, sar, vmstat, iostat and Oracle Enterprise Manager (OEM) are useful yardsticks to measure resources and consumption  Multiple cores (ie: dual core) excellent for CPU performance but may have additional Oracle licensing requirements.
  14. 14. Operating System Performance  Tuning the operating system a common area that is neglected with performance analysis!  Varies based on platform: Linux Windows!  MOS Note: 224176.1 How to use OS commands to diagnose Database Performance issues?
  15. 15. Tuning Linux For Oracle R12 EBS  Start simple with top and iostat  Check memory and configuration for kernel settings with dmesg and sysctl commands.  Drill down if necessary with strace utility.
  16. 16. Linux Performance Tips Network Performance:  MOS Note: 274953.1 Tuning TCP/IP parameter in Linux Box for SQLNET Use netstat and ping to identify latency issues with networks  MOS Note: 560590.1 How and When to use the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem Linux kernel Parameters Kernel Tuning:  MOS Note: 434351.1 Linux Kernel: The SLAB Allocator Disk Tuning:  MOS Note: 175980.1 Tuning Disk Throughput Using hdparm in Linux  sar and iostat are your friends for identifying disk I/O contention issues.  Also check for disk I/O issues with SAN tools (varies with vendor such as EMC, Hitachi, etc).
  17. 17. UNIX Performance Tips  My Oracle Support (MOS) available from http://support.oracle.com  MOS Note: 144638.1 Relationship Between Common Init.ora Parameters and Unix, Linux Kernel Parameters  Verify that kernel parameters are set correctly and verify with the vendor platform for UNIX.  Assess performance with top, iostat, vmstat, sar with UNIX systems.  OS Watcher tool: MOS Note: 301137.1 OS Watcher User Guide
  18. 18. Unix Performance Tuning Find the performance bottleneck with top and ps to drill down to the root cause  Solaris: use truss and DTRACE for tuning system calls and processes http://www.sun.com/bigadmin/features/articles/dtrace_truss.jsp  HP-UX: use the sam utility for tuning http://docs.hp.com/en/B2355-90692/sam.1M.html IBM AIX: SMIT (System Management Information Tool) is useful tool for performance tuning http://www.ibm.com/developerworks/aix/library/au-smit/index.html Note: you may need root OS privileges to run these tools or elevated privileges.
  19. 19. Linux Performance Tuning: top top - 15:05:08 up 33 days, 21:49, 10 users, load average: 0.09, 0.05, 0.01 Tasks: 492 total, 2 running, 490 sleeping, 0 stopped, 0 zombie Cpu(s): 16.9%us, 1.1%sy, 0.0%ni, 81.2%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4044596k total, 3683136k used, 361460k free, 272104k buffers Swap: 2064376k total, 76k used, 2064300k free, 1488836k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9455 applmgr 15 0 219m 112m 13m R 27.2 2.9 6233:08 npviewer.bin 10914 applmgr 19 0 546m 50m 7704 S 1.9 1.3 0:28.01 java 13701 applmgr 15 0 13020 1316 720 R 1.9 0.0 0:00.01 top 1 root 15 0 10348 704 588 S 0.0 0.0 0:01.51 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.23 ksoftirqd/0 4 root 10 -5 0 0 0 S 0.0 0.0 0:01.17 events/0 5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 46 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 50 root 10 -5 0 0 0 S 0.0 0.0 0:05.01 kblockd/0
  20. 20. Linux Performance Tuning: iostat $ iostat Linux 2.6.18-194.8.1.0.1.el5 (app01.ben.com) 09/19/2010 avg-cpu: %user %nice %system %iowait %steal %idle 0.29 0.00 0.13 0.37 0.00 99.22 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 0.08 0.29 1.79 21019459 131483716 sda1 0.00 0.00 0.00 2496 28 sda2 0.08 0.29 1.79 21016659 131483688 sdb 0.00 0.00 0.00 23141 5072 sdb1 0.00 0.00 0.00 22253 5072 dm-0 0.23 0.29 1.79 21000250 131361616 dm-1 0.00 0.00 0.00 16120 122072 $ iostat -x Linux 2.6.18-194.8.1.0.1.el5 (db01.ben.com) 09/19/2010 avg-cpu: %user %nice %system %iowait %steal %idle 0.29 0.00 0.13 0.37 0.00 99.22 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05 0.02 sda1 0.00 0.00 0.00 0.00 0.00 0.00 23.37 0.00 3.22 2.70 0.00 sda2 0.00 0.15 0.01 0.07 0.29 1.79 26.25 0.00 3.77 2.05 0.02
  21. 21. Windows Performance Tips  Recommend using system monitoring tools available with Windows 2003 and Windows 2008 Server  Sysinternal tools excellent for diagnosis as well http://technet.microsoft.com/en-us/sysinternals/default.aspx MOS Note: 231159.1: How to configure Windows to improve performance database oracle MOS Note: 1054009.1 Instructions for setting up the Performance Monitor Counters (PERFMON) for Windows Perfmon (Performance Monitor) excellent tool for tuning Windows environments.
  22. 22. Using Perfmon for Windows Performance Tuning
  23. 23. Tuning Concurrent Managers  Concurrent Managers source of many performance issues.  Useful My Oracle Support Notes: 104452.1: Troubleshooting Concurrent Manager  My Oracle Support Note: 1057802.1 Best Practices for Performance for Concurrent Managers in E-Business Suite  Myth: More concurrent managers are better  Keep it simple and use only few concurrent managers
  24. 24. Tuning Concurrent Managers  OAM excellent starting point to find performance bottlenecks for concurrent managers and other tuning issues with Oracle R12 EBS.
  25. 25. Using OAM for tuning Concurrent Processing  Check for long running concurrent requests for potential bottlenecks by using Oracle Application Manager
  26. 26. Tuning Concurrent Managers Oracle Applications Manager (OAM) to measure performance with Concurrent Managers for throughput to get the big picture
  27. 27. Tuning Internal Concurrent Manager (ICM) Tuning the Internal Concurrent Manager (ICM) Performance is based on the following parameters for Oracle R12  PMON sleep cycle — The number of sleep cycles that ICM waits between polling status for failures in concurrent processing. You should change the PMON cycle to a lower value if your concurrent managers experience chronic terminations.  Queue Size — is the number of PMON sleep cycles that the ICM waits before polling for new concurrent managers.  Sleep Time — parameter that indicates the time in seconds that the ICM waits before polling for requests that are waiting to run.
  28. 28. Concurrent Manager Tuning  Tune queue size for concurrent managers  Tune sleep cycle (PMON)  Tune cache size  Tune number of processes  Configure Specialization Rules  Define Workshifts to balance performance
  29. 29. Queue Size Tuning: Concurrent Managers  Queue size  Default queue size is too small after R12 installation  How to tune?  View statistics from scripts  Focus on tuning for ICM and Standard Managers first
  30. 30. How to Tune Sleep Cycle for PMON? During Peak Usage for business hours: 1) Set the sleep cycle for PMON to low value when number of concurrent requests is high. Why? With low sleep cycle for PMON, less need for concurrent managers to check for new requests thus increasing throughput. During Off hours with low business activity: 2) Recommend setting PMON sleep cycle to higher value during non-business hours such as night time with value such as 5 minutes.
  31. 31. Tuning PMON Sleep Cycle How to set value for PMON sleep cycle?  Startup parameter in adcmctl.sh script for Oracle R12 EBS Located under $ADMIN_SCRIPTS_HOME or $INST_TOP/admin/scripts/ directory in applications tier.  Parameters: adcmctl.sh {start|stop|abort|status} [<APPS username/APPS password>] [sleep=<seconds>] [restart=<N|minutes>] [pmon=<iterations>] [quesiz=<pmon_iterations>] [diag=Y|N] [wait=Y|N]  Tune by modify the pmon and sleep parameters  You can also use OAM to set these values as well.
  32. 32. Finding value for PMON sleep cycle  Login to OAM as SYSADMIN then System Administrator responsibility then navigate System Administrator-> Concurrent: Manager->Define->WorkShifts
  33. 33. Tuning cache size for Concurrent Processing How to Tune cache size for concurrent managers?  Increase cache size (number of requests cached) to twice the number of target processes for each manager. Example If we have standard manager with 1 target process and a cache value of 5, it will read five requests, and then attempt to run those five requests before reading any new requests. Recommendation  Use a value of 1 when defining a manager that has long running jobs, and a higher value of 4-5 for managers that run shorter transaction jobs.
  34. 34. Cache Size and Performance for Concurrent Processing  Tune by setting in OAM for Oracle R12 environment
  35. 35. Tuning Concurrent Managers Number of Processes  Examine the number of processes for Concurrent Managers  Check the columns for actual, target and running in OAM Login as SYSADMIN then navigate path Concurrent:Manager-> Administer
  36. 36. Specialization Rules for Concurrent Manager Performance  Use INCLUDE and EXCLUDE rules to optimize performance since Concurrent Managers are batch type processes  Login as SYSADMIN and navigate System Administrator Concurrent:Manager->Define in OAM Located under Specialization Rules
  37. 37. Setting values for number of processes with Concurrent Managers  Located under in OAM under navigation path System Administrator-> Concurrent: Manager->Define->WorkShifts
  38. 38. Specialization Rules Continued  Use combined specialization rules for complex tasks
  39. 39. Workshifts for Performance with Concurrent Processing  Setup different workshifts for balancing performance load with different critical tasks  Example: schedule AP during day and nightly processing on different workshifts  Login to OAM as SYSADMIN then navigate path System Administrator Concurrent:Manager-> WorkShifts
  40. 40. Example: Workshifts with Concurrent Processing
  41. 41. Concurrent Manager Tuning  Check for high number of requests  Script to find high number of concurrent requests SELECT v.USER_CONCURRENT_QUEUE_NAME,COUNT(PHASE_CODE) v1 FROM APPS.FND_CONCURRENT_QUEUES_V1 v, APPS.FND_CONCURRENT_WORKER_REQUESTS r WHERE r.queue_application_id = 0 AND r.PHASE_CODE = 'P' -- Pending Concurrent Requests AND r.HOLD_FLAG != 'Y' -- Concurrent Requests not on hold AND r.REQUESTED_START_DATE <= SYSDATE -- No Future Concurrent jobs AND r.CONCURRENT_QUEUE_ID=v.CONCURRENT_QUEUE_ID GROUP BY v.USER_CONCURRENT_QUEUE_NAME HAVING COUNT (PHASE_CODE) >=20 /  Use Oracle Application Manager (OAM) charts to measure performance
  42. 42. Useful Scripts for Tuning Concurrent Managers  Some Key Scripts for monitoring the concurrent managers: afcmstat.sql Shows all defined managers, capacity, process id, etc. afimchk.sql Status for Internal Concurrent Manager (ICM) & PMON afcmcreq.sql Shows concurrent manager basics afrqwait.sql Shows pending, held and scheduled requests afrqstat.sql Concurrent request execution times and status afqpmrid.sql Process id (PID) for FNDLIBR processes afimlock.sql Find concurrency and locking issues Execute these while connected to SQL*PLUS as APPS schema.  The above scripts are located under the $FND_TOP/sql directory
  43. 43. Additional Scripts for Tuning Concurrent Managers and Requests select f.application_short_name app, substr(p.user_concurrent_program_name,1,55) description, substr(p.concurrent_program_name,1,20) program, r.priority, count(*) cnt, sum(actual_completion_date - actual_start_date) * 24 elapsed, avg(actual_completion_date - actual_start_date) * 24 average, max(actual_completion_date - actual_start_date) * 24 max, min(actual_completion_date - actual_start_date) * 24 min, stddev(actual_completion_date - actual_start_date) * 24 stddev, stddev(actual_start_date - requested_start_date) * 24 wstddev, sum(actual_start_date - requested_start_date) * 24 waited, avg(actual_start_date - requested_start_date) * 24 avewait, c.request_class_name type from fnd_concurrent_queues fcq, fnd_concurrent_queue_content fcqc, fnd_concurrent_request_class c, fnd_application f, fnd_concurrent_programs_vl p, fnd_concurrent_requests r where r.program_application_id = p.application_id and r.concurrent_program_id = p.concurrent_program_id and r.status_code in ('C','G','E') and p.application_id = f.application_id and r.program_application_id = f.application_id and r.request_class_application_id = c.application_id(+) and r.concurrent_request_class_id = c.request_class_id(+) and r.request_class_application_id = fcqc.type_application_id(+) and r.concurrent_request_class_id = fcqc.type_id(+) and fcqc.queue_application_id = fcq.application_id(+) and fcqc.concurrent_queue_id = fcq.concurrent_queue_id(+) group by c.request_class_name, f.application_short_name, p.concurrent_program_name, p.user_concurrent_program_name, r.priority / TOTAL AVG MAX MIN RUN WAIT #WAITED AVG REQ APP DESCRIPTION PROGRAM PRI R UN HOURS HOURS HOURS HOURS STDDEV STDDEV HOURS WAIT TYPE ------ ---------------------------------------- -------------------- ---- ------ -- ------- ------ ------ ------ ------- -- ----- --------- -------- ------------- -- SQLGL Periods - Open Period: Child Process GLOOAPC 50 1 2 .00 .00 .00 .00 .00 .00 .10 .01 BOM Cost Manager CMCTCM 50 1 ####### ###### # ##### ###### .00 .00 -16403.89 ######## ALR Check Event Alert ALECTC 1 8 .02 .00 .00 .00 .00 .01 .16 .02 OFA Asset Inventory Report FAS410 1 2 .01 .01 .01 .00 .00 .00 .01 .00 SQLGL Open Encumbrance Year GLEOYR 1 2 .09 .05 .05 .04 .01 .00 .03 .01
  44. 44. Manually Gather Statistics for Oracle R12 EBS  FND_STATS for schema statistics collection Example: Collect statistics for AP schema: exec fnd_stats.gather_schema_statistics('AP');  Collect stats for all schemas for Oracle R12 EBS: exec fnd_stats.gather_schema_statistics('ALL');  How to gather stats for temp tables with Oracle R12 EBS: exec fnd_stats.gather_table_stats('<schema>','<temp_table_name>');  Example to collect stats for temp tables in AR schema: exec fnd_stats.gather_table_stats('AR','temp_ap');
  45. 45. Statistics Collection for Oracle R12 EBS Why?  Because cost based optimizer requires best execution path with latest current statistics.  Reference Point on how to collect statistics My Oracle Support http://support.oracle.com Note: 419728.1 How To Gather Statistics On Oracle Applications 11.5.10(and above) - Concurrent Process,Temp Tables, Manually Do not use DBMS_STATS with Oracle R12 EBS to collect statistics!
  46. 46. Concurrent Manager Statistics Be sure to Gather Table Statistics on these CM tables:  FND_CONCURRENT_PROCESSES  FND_CONCURRENT_PROGRAMS  FND_CONCURRENT_REQUESTS  FND_CONCURRENT_QUEUES.
  47. 47. Gather Stats for Oracle R12 EBS  Either run FND_STATS or Concurrent Request via OAM System Administrator|Concurrent Requests|
  48. 48. Options for STATS Collection • Options to set parameters for estimate percent, parallelization, schema for Concurrent request to gather statistics for Oracle R12 E-Business Suite • Recommend that you do not set estimate percent too high otherwise the load to collect stats will impact performance and collection time.
  49. 49. Tuning Application Tech Stack  Oracle 10g Application Server tuning issues  Oracle HTTP Server (OHS): “Apache” web server  Timeout settings for Apache  Under httpd.conf file  Configuration settings for JVM with OC4J under both the Oracle R12 EBS Context file for application tech stack and opmn.xml values.
  50. 50. Tuning Apache OHS Oracle R12 EBS Application Tech Stack  Values to check for the Apache httpd.conf file located under $IAS_ORACLE_HOME/Apache/Apache/conf directory # Timeout: The number of seconds before receives and sends time out. # Timeout 300 # # KeepAlive: Whether or not to allow persistent connections (more than # one request per connection). Set to "Off" to deactivate. # KeepAlive On # # MaxKeepAliveRequests: The maximum number of requests to allow # during a persistent connection. Set to 0 to allow an unlimited amount. # We recommend you leave this number high, for maximum performance. # MaxKeepAliveRequests 100 # # KeepAliveTimeout: Number of seconds to wait for the next request from the # same client on the same connection. KeepAliveTimeout 15
  51. 51. Application Tech Stack Tuning  Tune values for timeouts to prevent and resolve errors in performance for network delays for forms, oacore, OC4J  Avoid and resolve infamous “Uninterrupted Exception 150 error” when starting and stopping app tier services  Values to check in Oracle R12 EBS Context file  Consider shared APPL_TOP and staged APPL_TOP for performance  Implement Parallel Concurrent Processing (PCP) so distribute performance load for Concurrent processing
  52. 52. Application Tech Stack tuning  Performance metrics for OC4J groups  Use server for concurrent processing server, application server in split tier configuration to provide more performance and availability  Consider load balancer: hardware or software to improve performance and availability  Consider deploying Oracle RAC for scale out and scale up in performance if you need it for large deployments  Oracle Application Management Pack (AMP) for tuning and monitoring Oracle R12 EBS environment
  53. 53. Load Balancing for Application tier performance  Hardware Load Balancing options: Cisco and Big IP some vendors that offer solutions  Software Load Balancing  Load balancing is complex and requires coordination with network, security and systems administration teams and the database team to implement successfully.  Reference Point: My Oracle Support Note 727171.1: Implementing Load Balancing On Oracle E-Business Suite - Documentation For Specific Load Balancer Hardware
  54. 54. Software Load Balancing for Oracle R12 EBS Application Tier performance  Not as robust as hardware load balancers but more cost effective.  Uses web cache option for Oracle 10gAS application server  My Oracle Support Note: 380486.1 Installing and Configuring Web Cache 10g and Oracle E-Business Suite 12
  55. 55. Software Load Balancing Oracle R12 EBS  Requires configuration at both 10g Application Server level and within context file for Oracle R12 EBS The following autoconfig variables must be set within the Oracle R12 EBS application tier context file:  s_webentryhost- webentry points to Oracle 10gAS Web Cache 10g Server  s_webentrydomain- webentry domain to domain name for Oracle 10gAS Web Cache 10g Server  s_active_webport-active webport to port for Oracle 10gAS Server Web Cache where it listens for client requests  s_webentryurlprotocol - webentry protocol to protocol values that clients will use to access Oracle 10gAS Server Web Cache 10g  s_login_page - login page to hostname:port/OA_HTML/AppsLogin
  56. 56. Java Tuning for Oracle R12 EBS  Tuning Java components is essential for performance with Oracle R12 EBS  Oracle R12 EBS is dependent on Java technology  Default values are too low for Java with OC4J settings  Watch out for issues with garbage collection  Avoid full garbage collection (GC) as this impacts performance!  My Oracle Support Note: 567551.1: Troubleshooting : Configuring various JVM tuning parameters for Oracle E-Business suite 11i and R12
  57. 57. Tuning Java for R12 EBS: JVM Sizing  How many JVMs (Java Virtual Machines) ? OACoreGroup  For 2-tier environment with application middle tier on separate host server and database server  Best not to have more than 100 active users per JVM/OC4J instance for middle tier server. No more than one active JVM/OC4J instance per CPU  Best practice: there should be ratio of 1 JVM to 2 CPUs for middle tier server  Use top and other system tools to check that sufficient memory is available. You want avoid memory swapping issues.  Common mistake for application middle tier server to not be sized correctly for memory and CPU! FormsGroup  For the forms servlet, the actual JVM/servlet usage is minimal, and a single JVM can normally handle up to 100-250 forms users.  JVM settings are configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs and s_xmlsvcs_nprocs
  58. 58. Java Tuning: Oracle R12 EBS Heap configuration for JVMs with middle tier  OACORE starting point for tuning J VM heap sizes: -Xmx512M -Xms256M -XX:MaxPermSize=128M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB  Goal is to minimize garbage collection frequency and to reduce user response time by minimizing full garbage collection.  For JDK 1.5, if the JVM detects that you have a server with multiple CPUs it will enable Parallel Throughput Garbage Collector by default.  Rule of thumb: number of GC (garbage collection) threads defaults to number of CPUs per machine.  If you run multiple JVMs on the same server, or if you have more than 2 CPUs, to avoid excessive GC, reduce the number of GC threads: -XX:+UseParallelGC -XX:ParallelGCThreads=2  Two GC threads should be enough for most environments  For heap sizes over 1GB, you can test with 4 GC threads and measure performance.  FormsGroup values for Garbage Collection (GC) as rule of thumb: -Xmx256M -Xms128M -XX:MaxPermSize=128M -XX:NewRatio=2 Key Oracle R12 EBS Application Middle Tier Server Autoconfig Variables for Java Heap tuning with JVMs:  s_oacore_jvm_start_option  s_forms_jvm_start_options
  59. 59. Tools for Measuring JVM Performance with Oracle R12 EBS  Excellent free tool to graph performance for JVM with Oracle R12 EBS and garbage collection.  http://www.tagtraum.com/gcviewer.html
  60. 60. Tuning Java for Oracle R12 EBS  My Oracle Support Note: 362851.1: - Guidelines to setup the JVM in Apps E-Business Suite 11i and R12 JVM Behavioral Options: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp#BehavioralOptions JVM Options List: http://blogs.sun.com/watt/resource/jvm-options-list.html
  61. 61. Monitoring Java Performance: Oracle R12 EBS  Oracle 10g Grid Control (OEM) with Application Management Pack (AMP). Note: These items require additional cost licenses.  MOS Note 557194.1 provides a script called monitor_jdbc_conn.sql to monitor JDBC connections.
  62. 62. Java Tuning for Oracle R12 EBS  Collect metrics to analyze current performance  Good information from ADDM and AWR in 11g Statistic Total per Second per Trans -------------------------------- ------------------ -------------- ------------- java call heap collected bytes 683,300,160 1,300.0 877.8 java call heap collected count 3,693,358 7.0 4.7 java call heap gc count 356 0.0 0.0 java call heap live object count 24,810 0.1 0.0 java call heap live object count 142,439 0.3 0.2 Instance Activity Stats DB/Inst: SBOX/SBOX Snaps: 274-420 java call heap live size 2,358,144 4.5 3.0 java call heap live size max 33,260,184 63.3 42.7 java call heap object count 56,992 0.1 0.1 java call heap object count max 345,992 0.7 0.4 java call heap total size 42,729,472 81.3 54.9 java call heap total size max 115,343,360 219.5 148.2 java call heap used size 6,144,496 11.7 7.9 java call heap used size max 72,864,528 138.6 93.6
  63. 63. Database Tuning for Oracle R12 EBS  Similar to tuning regular Oracle 11g database but unique in how statistics collected  Remember to use FND_STATS and NOT DBMS_STATS to collect and update statistics!  Watch out for concurrency and locking issues for runaway user processes!  Schedule backup jobs off hours to avoid impact business users  Don’t forget to use tools like AWR and STATSPACK if possible.
  64. 64. Database Tuning  Suggested values for database initialization parameters for Oracle R12 EBS  My Oracle Support Note: 396009.1 Database Initialization Parameters for Oracle Applications Release 12  Oracle R12 by default installs either a 10gR2 or an 11gR1 database for fresh (new) installation.  For example: Oracle 12.0 to 12.04 uses 10.2.x database  While Oracle 12.1.1 and later release install Oracle 11gR1 database (11.1.x).  Watch out for upgrade issues from 11i to R12 EBS for database!
  65. 65. Database Parameters for Oracle R12 EBS Parameter Name Development or Test Instance 11-100 Users 101-500 Users 501-1000 Users 1001-2000 Users processes 200 200 800 1200 2500 sessions 400 400 1600 2400 5000 sga_target 1G 1G 2G 3G 14G shared_pool_size (csp) N/A N/A N/A 1800M 3000M shared_pool_reser ved_size (csp) N/A N/A N/A 180M 300M shared_pool_size (no csp) 400M 600M 800M 1000M 2000M shared_pool_reser ved_size (no csp) 40M 60M 80M 100M 100M pga_aggregate_tar get 1G 2G 4G 10G 20G Total Memory Required ~ 2 GB ~ 3 GB ~ 6 GB ~ 13 GB ~ 25 GB
  66. 66. Upgrade Performance considerations for Oracle R12 EBS  When you upgrade from older releases such as 10.x and 11i to Oracle R12 must account for performance items  Rule based Optimizer (RBO) in 10.x and early 11i releases to the Cost Based Optimizer (CBO) in R12  Different way of tuning for stats and database  Consider changes for SGA and PGA sizing  Migrate from old tablespace model to Oracle Application Managed Tablespaces (OATM)  My Oracle Support Note: 761570.1 Database Preparation Guidelines for an E-Business Suite Release 12.1.1 Upgrade
  67. 67. Q&A Discussion
  68. 68. Thank You!  Please provide feedback  Email: ben@ben-oracle.com  My blog has further tips on Oracle R12 EBS  http://oracle-magician.blogspot.com  I will be available in the OTN ACE Director Office Hours this week for questions

×