Your SlideShare is downloading. ×
Ebs performance tuning session  feb 13 2013---Presented by Oracle
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Ebs performance tuning session feb 13 2013---Presented by Oracle

1,092
views

Published on

Published in: Technology, News & Politics

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,092
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
70
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 2. E-Business Suite Performance Tuning (11.5.x/12.x.x) Oracle Support is pleased to Welcome you to  the WebCast. The Session will commence shortly.  Audio Conf Details:  Audio Conference ID: # 89316108 Passcode: 2013   India Toll-Free Dial-In Number: 0008001006550 Operator Assisted International Dial-In Number: + 61285249600   Thanks for joining 2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 3. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 4. Performance Tuning E-Business Suite 11.5.x/12.x.x Kandasamy Nainamalai Senior Principal Support Engineer - Applications Technology Group 4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 5. Program Agenda • • • • • • • • 5 Applications Architecture Defining & Isolating issue How to approach a performance issue SQL Trace and TKPROF output AWR/Statspack Report Best Practices - Maximizing the performance Announcement : Upcoming training “Concurrent Manager – Best Practice” Q&A Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 6. <Insert Picture Here> Oracle Applications Architecture 6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 7. Oracle Applications Architecture – – – • • • 7 3 Tier architecture. Distributed Computing & Internet Based Computing Applications components are distributed among multiple levels, or tiers, of machines. Database Tier Application / Middle Tier Desktop Tier Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 8. 3-Tier Architecture 8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 9. <Insert Picture Here> Defining & Isolating issue 9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 10. Defining & Isolating issue What is slow ? 1. 2. 3. 10 Entire system is slow Sub-system is slow • Forms Based Applications • Web Based Applications • Concurrent Manager Particular process is slow. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 11. Defining & Isolating issue • • When is slow – Month end/yearend | – Peak hours |  Concurrency, due to Load (or) H/W is not sized. – All the time | Was it working fine earlier? – – • Who are all affected? – – 11 All of sudden • List out recent Changes – Patch, Gather Schema, New RAC-node/modules/users, Data load, OS, Network/DNS Deteriorated over a period All users Specific to some module. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 12. Defining & Isolating issue • • • 12 By this time, We should be in a position to isolate which component is the source of performance issue. Once we identify the component , then we need to investigate what could be the problem. How to go about doing that ? Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 13. <Insert Picture Here> How to approach a performance issue 13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 14. <Insert Picture Here> Approaching performance issue Gathering Required information 14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 15. Approaching performance issue Desktop – – – – 15 Though its a thin client; browser has jinitator as a plug-in. It uses Java for the presentation Layer. It requires resource at Desktop. Check Enough resource CPU/RAM Ensure no heavy applications running when accessing Applications. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 16. Approaching performance issue Concurrent Manger – – – – – – – – – 16 Concurrent jobs are CPU intensive. Hence ensure enough resource CPU/RAM available on the DB tier. Check RAC & PCP enabled. Check concurrent manager definition for Sleep time & Cache are set correctly Check resource intensive batch requests scheduled during peak hours. (Separate manager for long running process with less number of managers ) Check there are short running requests waiting for longer time to process (Resolution : specialized managers) Check Fnd_concurrent_requests & Processes are purged & defragment regularly. Avoid enabling an excessive number of standard or specialized managers. Note 1057802.1 Best Practices for Performance for Concurrent Managers Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 17. Approaching performance issue Forms Server – – – – – – – 17 Memory Intensive. Ensure RAM & CPU Check Sar & top ( check for spinning f60webmx) Check FORMS60_CATCHTERM FORMS_RECORD_GROUP_MAX  Debug/FRD enabled Check load balancing enabled and work effectively Forms Patch set level with IO patches. Servlet Vs Socket Mode Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 18. Approaching performance issue Applications – – – – – – – 18 How many concurrent users are there ? What are all the modules used? • Any resource intensive module is used. When was gather schema statistics request run last ? Diagnostics profiles are disabled as this will consume lots of resource Sign on Audit is disabled if not needed. Audit on other tables have been enabled (additional overhead) Purging is done periodically. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 19. Approaching performance issue Database – – – – DB process are basically CPU Intensive Check adequate resource (CPU/RAM) is available Monitor Sar / vmstat output (first thing to do) • Wio is high then need to check I/O setup & SQL queries • CPU Is high then check for CPU intensive queries. Monitor top OS process at OS level • – – – 19 Top (HP) / prstat (Sun) / ps aux (AIX) init.ora setup (Refer 216205.1 written by Dev). init.ora check Version of the Database & performance patch applied. AWR / Statspack Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 20. Approaching performance issue Web Tier (OC4J, Apache & Jserv) – – – – – – Check JVM is hanging (due to Out of Memory issue). Check Heap size and how many JVM thread are defined. Disable statement/debug level logs when not needed.  AOL/J Database Connection Pool Status (Doc ID 278868.1) Review DBC file configuration Review Apache/Jserv configuration Network – – – – 20 It plays vital role if Applications accessed through VPN/WAN Check for sufficient Bandwidth (ping host –l 1024 –t) Latency should be as less as possible. Network Test ((Doc ID 556738.1) Socket Vs Servlet Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 21. <Insert Picture Here> SQL Trace and TKPROF output When specific process takes time 21 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 22. Different trace levels – – 22 4 possible levels: • Level 1 : Regular Trace / SQL Trace • Level 4 : Trace with Binds • Level 8 : Trace with Waits • Level 12 : Trace with Binds and Waits Level 8 is preferred option for analyzing performance issues as Level 12 is too expensive. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 23. Create trace file - Form – – – 23 Trace menu availability: 11.5.7 or higher Trace file name contains application User Name Trace in Pre 11.5.7 • Identify the Session/Process Id then enable trace as below. • ORADEBUG SETOSPID &&pid • ORADEBUG EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 8 • rem ORADEBUG EVENT 10046 TRACE NAME CONTEXT off Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 24. 24 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 25. Concurrent Report Tracing Report Level Vs Program level – Set “Enable Trace” flag in the Concurrent Program Definition Form. (Program level) – Enable trace for a specific request (SRS form –> Debug Options) (Request level) • • 25 The profile “Concurrent: Allow Debugging” should be set to “Yes.” Recommended way to enable trace. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 26. Enabling Trace at the program Level 26 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 27. Enabling Trace at the Request Level 27 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 28. Create a trace file – Self Service After tuning JVM , if you need to take DB trace if web pages are slow. There are two options. – – – 28 Set Profile Option “FND: Diagnostics” to Yes Alternative: Profile Option called ‘Initialization SQL Statement Custom’ What to choose from the above two options ? Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 29. 29 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 30. Create a trace file – Anywhere – – – – – Profile Option : Initialization SQL Statement - Custom Can always be used, e.g.: • Concurrent Programs • Background processes like Workflow • ... Set on User level only SQL code will execute for each new DB session which it initialized for an application user SQL Statement for a level 12 trace: begin fnd_ctl.fnd_sess_ctl ('','','TRUE','TRUE','LOG','ALTER SESSION SET tracefile_identifier=''MYTRACE'' EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 12'''); end; 30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 31. 31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 32. What is TKPROF ? – – – – 32 Formatted report of SQL Trace file Includes all SQL statements with statistical info Allows to identify the performance problem by using different sort methods Invoke tkprof from Database Tier (10g/11g) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 33. Getting TKPROF output tkprof tracefile.trc output.txt sort=fchela,exeela,prsela – – – 33 Invoke tkprof from Database Tier Upload raw trace and tkprof output i.e. output.txt and tracefile.trc Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 34. total time 34 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. # rows
  • 35. Identify culprit SQL statement – – – Identify Culprit SQL statement is within TKPROF files Most relevant is TKPROF – Sorted by elapsed time In general the first SQL statement is the culprit Tip: Always look for ‘large numbers’ 35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 36. Gather Enhanced Explain Plan – – – – – – Isolate the culprit SQL statement in a text file Gather Enhanced Explain Plan using SQLTXPLAIN.SQL: Note.215187.1 Do NOT replace BIND variables with literals! Creates a TXT report and trace file Gathers information e.g.: • tables and indexes used • last_analyzed dates Sample Query Tuning • Any full table scan • • • • • 36 is used. See any index can be used Check Statistics collected for the tables involved Temp, interface tables are purged and de fragmented Any index which is not appropriate being used Rebuild all index for the tables involved Patch (for Higher version of code ) which resolves it Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 37. 37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 38. <Insert Picture Here> AWR/Statspack Report 38 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 39. DEMO • • Demo AWR Report If you see any DML is consuming more elapsed time. Get the explain plan using sql_id as below.you can run the following commands to obtain the runtime statistics, as well as past plans recorded in AWR: • • 39 SQL> SELECT * FROM table(dbms_xplan.display_cursor('&sql_id',null,'ALLSTATS')) ; SQL> SELECT * FROM table(DBMS_XPLAN.DISPLAY_AWR('&sql_id',null,null,'ALL')); Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 40. AWR/ Statspack Report Top 5 Events section This section shows the Top 5 timed events that must be considered to focus the tuning efforts. Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Event Wait Waits % Total Time (cs) Wt Time -------------------------------------------- ------------ ------------ ------db file sequential read 1,797,963 11,207,291 42.67 db file scattered read 792,958 9,651,032 36.75 buffer busy waits 158,725 1,554,054 5.92 31,317 994,507 3.79 15,489,369 497,062 1.89 log file sync SQL*Net break/reset to client ------------------------------------------------------------40 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 41. AWR / Statspack Report Handling “db file scattered read” This wait happens when a session is waiting for a multi block IO to complete. This typically occurs during full table scans or index fast full scans. Oracle reads up to DB_FILE_MULTIBLOCK_READ_COUNT consecutive blocks at a time and scatters them into buffers in the buffer cache. • • • 41 Check if there are any full table scans.In this case query tuning should be used to optimize the SQL. See if partitioning can be used to reduce the amount of data you need to scan. Review SGA Database Buffer Cache Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 42. Best Practices for Maximizing the Performance • Techstack upgrade (for Performance & High-Availability) Ø Ø Ø • Upgrade EBS to 12.1.3 (Note 269.1) Ø Ø Ø Ø • 42 Upgrade Database to 11.2.0.3 (Note 1058763.1 & Doc ID 881505.1) OracleAS 10g Release 3 Patch Set 5 (10.1.3.5.0) (Note 454811.1) Forms bundle patch on10.1.2.3.0 (Doc ID 437878.1) New functionality Changes in business requirements Release 12 provides upgrade path to Fusion Applications Support - Premier Support for 11.5.10 ended in November 2010 Apply Recommended Performance Patches (Doc ID 244040.1) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 43. Best Practices for Maximizing the Performance • Regular Health check to be performed • Database – – – – • Sufficient resource CPU/RAM available (20 - 30% free RAM/CPU is healthy) Sar & Top process has to be examined (Store it ; can be used later to compare) Ensure init.ora parameters are as per standard AWR/Statspack report examined Forms Server – – Sufficient resource RAM & SWAP available Spinning of forms process set FORMS_RECORD_GROUP_MAX=10000 (Doc ID 745711.1) Disable Cancel Query “FND: Enable Cancel Query” to ‘No’ • Tweak FORMS60_CATCHTERM & FORMS_CATCHTERM Servlet Vs Socket Mode https://blogs.oracle.com/stevenChan/entry/which_is_better_forms_servlet_or_socket_mode • • – • 43 Note 384241.1 - Using Forms Socket Mode with Oracle E-Business Suite Release 12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 44. Best Practices for Maximizing the Performance • Concurrent Manager – – – – – • Tune using Best Practice Concurrent Manager (Note 1057802.1) Sufficient resource CPU/RAM available Sleep time (30 secs) & Cache size (equal to number of process) Specialized Manger are used if needed. Purge / de-fragment Concurrent tables Applications – 11g Database , Apply Apps Patch (13443215 & 12830671 ) to speed to Gather schema Stats • – – – – • 44 Gather schema statistics collected at regular interval with 10 – 40 % Diagnostics profiles/FRD/Traces are disabled if not needed. Hardware Sizing. Sign-on Audit & Audit tables are disabled if not needed. Purge & De-fragment interface/transaction tables periodically. (Note 752322.1) Purge portal is available in OAM that can be used to purge Apps. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 45. Best Practices for Maximizing the Performance • Network – Audit Network usage. – Tool to help identify the source of of the problem • – – • Oracle E-Business Suite Network Utilities: Best Practices [ID 556738.1] Note 967992.1 - Symmetrical Network Acceleration with Oracle E-Business Suite Release 12 Note 786582.1 - Performance Degradation in Forms in R12 When not Using Swan Color Scheme Web Tier (OC4J, Apache & Jserv) • • Ensure heapsize is set properly (Doc ID 362851.1 ) Tune s_oacore_jvm_start_option (R12) (Doc ID 362851.1) – • • 45 $ grep s_oacore_jvm_start_option $CONTEXT_FILE (OS command to check it out) Tune oacore_nprocs( R12) Tune ApJServGroup OACoreGroup in jserv.properties (R11i ) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 46. Summary • Performance Tuning • Monitor Resource (CPU/ I/O /RAM) Utilization. • Ensure Init.ora standards are met (Note 174605.1) • AWR Report for Database wide performance issue • Best Practice Concurrent Manager (Note 1057802.1) • • • • 46 Purge & Defragment interface/transaction tables. (Note 752322.1) Gather Schema Statistics at regular Interval. Apply FND_STATS  Patch (13443215 or 12830671 ) to speed to Gather schema Stats (relevant for 11g Database). Upgrade Applications & Techstack. Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 47. References • • • • • A Holistic Approach to Performance Tuning Oracle Applications Systems (Doc ID 69565.1) Best Practices for Performance for Concurrent Managers in EBS (Doc ID 1057802.1) Reducing Your EBS Data Footprint using Archiving, Purging, and 752322.1) ILM (Doc ID bde_chk_cbo.sql - EBS initialization parameters - Healthcheck (Doc ID 174605.1) Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite (Doc ID 1121043.1) • • 47 Oracle E-Business Suite Recommended Performance Patches (Doc ID 244040.1) https://blogs.oracle.com/stevenChan/entry/which_is_better_forms_servlet_or_socket_mod Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 48. References • • 48 JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1). Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 49. References Upgrading E-Business Database • • • • • 49 Upgrade Advisor: E-Business Suite (EBS) Upgrade from 11.5.10.2 to 12.1.3 (Doc ID 269.1) Interoperability Notes EBS R12 with Database 11gR2 (Doc ID 1058763.1) Interoperability Notes Oracle EBS 11i with Oracle Database 11gR2 (11.2.0) (Doc ID 881505.1) Upgrading to the Latest OracleAS 10g 10.1.3.x Patch Set in Oracle E-Business Suite Release 12 (Doc ID 454811.1) Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12 (Doc ID 437878.1) Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 50. You can visit us at https://communities.oracle.com 50 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 51. Announcement : Upcoming Webcast in this series Ø Ø Ø Training on “EBS Concurrent Manager Performance - Best Practices” When : 13th March 2013 ( 11:30 to 1:00 pm) Agenda Ø Ø Ø Ø 51 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Overview of Concurrent Manager Enhancing Performance - Best Practices Ø Tuning Concurrent Queue Parameters Ø Specialized Concurrent manager Ø Output Post Processor – OPP Ø Tuning PCP/RAC Ø Transaction Manager (TM) Common / Typical issues and resolution Ø Purging Concurrent records Ø Running cmclean.sql Ø OPP and TM related issues. Ø Resolving concurrent requests Trouble-shooting Concurrent manager
  • 52. Q&A 52 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 53. 53 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.
  • 54. 54 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.