Best practices for_large_oracle_apps_r12_implementations


Published on

SANGAM-13, Hyderabad, India, 8-9 Nov 2013

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Best practices for_large_oracle_apps_r12_implementations

  1. 1. Best Practices For Large Oracle Apps R12 Implementations Ajith Narayanan Dell IT, Bangalore Hyderabad , 9th Nov 2013
  2. 2. Disclaimer The views/contents in this slides are those of the author and do not necessarily reflect that of Oracle Corporation and/or its affiliates/subsidiaries. The material in this document is for informational purposes only and is published with no guarantee or warranty, express or implied..
  3. 3. Who Am I? Ajith Narayanan Software Development Advisor Dell IT, Bangalore. 9 years of Oracle [APPS] DBA experience. Blogger :- Website Chair: - OracleRACSIG (2011 -2013) Member:- IOUG, OAUG & AIOUG, RACSIG (
  4. 4. Agenda Why do we need to implement best practices? Application Tier ================= 1. R12 Tech Stack 2. Forms 3. Response Time/CPU Issues 4. Web Applications 5. Concurrent Manager 6. Workflow Engine 7. Taking Diagnostics Help 8. CBO Statistics 9. Application Patching Best practices 10. Network & Latency Check
  5. 5. Agenda Database Tier ============== 1. CPU Utilization Observations and analysis 2. High DB_IO observation and correlation with CPU usage. 3. Taking Diagnostics Help 4. Identify Top SQLs 5. Effective usage of AWR, ADDM Reports 6. Fragmented Interface tables 7. Object to be pinned – (Frequently used database objects) 8. RAC Check (From Oracle RAC assurance team) 9. Cluster Callout scripts to capture cluster events and notification through e-mail. 10. ADR Command Interpreter (ADRCI) 11. ADG for offloading reporting workload Looking At Issues Proactively & From Greater Heights References Q&A
  6. 6. Why do we need to implement best practices? • Oracle Applications R12 project with 8 Node middle tiers and 16 Node RAC database tiers. Database Size 40+ TB size and with a yearly database growth approximately 2 TB • Makes our environment proactive & simple in terms of manageability. • Helps in environment support & maintenance cost reduction. • Time & effort saving . • And Of Course!! Makes everybody’s life easier
  7. 7. Application Tier R12 Tech Stack: - Make sure to upgrade to the latest certified technology stack OC4J: Oracle AS 10g Forms: Oracle AS E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1) Useful Articles for the Oracle E-Business Suite Technology Stack for Technical Consultant / DBA (MOS Note - 828157.1) Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)
  8. 8. Application Tier
  9. 9. Application Tier Forms: - Few checks for improved performance are Socket Mode for Internal users (R12: Refer to Note 384241.1.) Servlet mode is default in R12 and recommended for external access. Forms dead client detection (DCD) and Abnormal termination handler. Value specified in minutes : FORMS_TIMEOUT=10 Terminates fwebmx processes for dead clients. Do not set FORMS_CATCHTERM Disable Cancel Query Cancel Query increases middle-tier CPU as well as DB CPU Refer to MetaLink Note 138159.1 on how to enable and tune Cancel query related parameters To Disable Cancel Query Set the Profile “FND: Enable Cancel Query” to ‘No’
  10. 10. Application Tier Response Time/CPU Issues: - When users complain about response time, configure Apache to log the time it takes to service a request. Access logs monitoring should be good way to keep a tab on the response times of user requests. Configure Apache to log the time it takes to service a request Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf LogFormat "%h %T Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
  11. 11. Application Tier Web Applications:Ensure the users are trained to use the Logout or Home global buttons when completing their transactions. Do not train them to use the browser close (“x”) link. Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.) Releases process resources sooner First, rule out any SQL issues:- Note 357597.1 on how to enable SQL trace for OA Framework applications , If there are no database related issues, then you need to analyze the JVM (GC logs: Configure JVM sizing). Pool Monitor can be used to monitor the Framework Applications as well as the JVM utilization Navigation :- Application Module Pool -> Memory Utilization -> JVM Properties -> Invoke with -> Login-> Diagnostics -> Show Pool Monitor
  12. 12. Application Tier Concurrent Manager – (EBS Concurrent Processing Analyzer) The EBS Concurrent Processing Analyzer is a self-service health-check script that reviews the overall Concurrent Processing footprint and analyzes the current configurations and settings for the environment, providing feedback and recommendations on best practices. APPS Workload (Concurrent, FORM, ICX) observations. Mandatory Concurrent Programs. Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1] Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)
  13. 13. Application Tier Workflow Engine (EBS Workflow Analyzer) The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint, analyzes the configurations environment, provides feedback and makes recommendations on best practices and areas of concern. It provides immediate analysis and output of the EBS Workflow environment. Workflow Process Stuck & Work Flow Checks How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1) 11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc ID 1369938.1) How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)
  14. 14. Application Tier Taking Diagnostics Help Oracle Application Object Library Best Practices: Database Security Tests Health Check Test Oracle Application Object Library Best Practices: E-Business Suite Security Tests Health Check Test Oracle Application Object Library Best Practices: E-Business Suite Diagnostic Tests Health Check Test CBO Statistics Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1) bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)
  15. 15. Application Tier Application Patching Best practices Through patch analysis with timing reports on sand box & non-prods environments before proceeding with the Prod patching. Patching Best Practices And Reducing Downtime (Doc ID 225165.1) See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“ How to Find E-Business Suite Recommended Patches (Note 1400757.1) Network & Latency Check Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)
  16. 16. Database Tier CPU Utilization Observations and analysis For more than 30 minutes continuously is a candidate for RCA. High DB_IO observation and correlation with CPU usage. I/O Thresholds: Logical RDS > 300K/sec, Physical RDS> 3000/Sec are candidate for RCA. LIO is Expensive? How many of you think RAM access is 10,000 times faster than Physical disk access? In real world, LIO is only 25-100 times cheaper than PIO - …not 1,000s or 10,ooos Reason – Internal locks & latch serialization mechanisms involved. Targeting only PIO counts(or high cache hit ratios) during SQL optimization is an important pitfall to avoid.Even with no PIOs, a query can still be outrageously inefficient -LIO are a critical component of query cost
  17. 17. Database Tier Taking Diagnostics Help Oracle Application Object Library Best Practices: E-Business Suite Database Diagnostic Tests. Effective usage of AWR, ADDM Reports Identify Top SQL, Latch Spin Counts etc., Memory Advisors Identify top wait events (System, User IO, Cluster, Concurrency, Configuration, Applications, Network) Fragmented Interface tables Object to be pinned – (Frequently used database objects) RAC Check (From Oracle RAC assurance team) RACcheck - RAC Configuration Audit Tool (Doc ID 1268927.1)
  18. 18. Database Tier Cluster Callout scripts to capture cluster events and notification through e-mail. ADR Command Interpreter (ADRCI) ADRCI has multiple features such as Analyze diagnostic data within the Automatic Diagnostic Repository (ADR). Create a package with incident / problem information for Oracle Support Run the Health Checks Helps in purging old diagnostics trace & dump files Note: - I’ve heard many DBAs saying that they can understand shell scripts, but that’s not enough, DBAs should be creative enough to write beautiful shell scripts to automate their environments and utilize the Oracle tools & utilities in efficient manner.(It actually develops DBAs general problem solving capabilities) ADG for offloading reporting workload Business Continuity for Oracle E-Business Release 12 Using Oracle 11g Physical Standby Database (Doc ID 1070033.1) Using Active Data Guard Reporting with Oracle E-Business Suite Release 12.1 and Oracle Database 11g(Doc ID 1070491.1)
  19. 19. ADRCI Commands ajithpathiyil1:/home/oracle[RAC2]$ adrci ADRCI: Release - Production on Sat Nov 9 00:36:43 2013 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. ADR base = "/u01/app/oracle" adrci> show home ADR Homes: diag/tnslsnr/ajithpathiyil1/listener_scan1 diag/tnslsnr/ajithpathiyil1/listener diag/rdbms/rac/RAC2 diag/asm/+asm/+ASM2 adrci>
  20. 20. ADRCI Commands adrci> show problem ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2: ************************************************************************* PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME -------------------- ----------------------------------------------------------- -------------------- --------------------------------------1 ORA 7445 [qcstda()+720] 17097 2013-11-09 00:42:09.740000 +05:30 1 rows fetched adrci> show incident ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME -------------------- ----------------------------------------------------------- ---------------------------------------17097 ORA 7445 [qcstda()+720] 2013-11-09 00:42:09.740000 +05:30 1 rows fetched
  21. 21. ADRCI Commands adrci> show incident -mode detail -p "incident_id=17097" adrci> show trace /u01/app/oracle/diag/rdbms/rac/RAC2/incident/incdir_17097/RAC2_ora_9226_i17097.trc adrci> show control ADR Home = /u01/app/oracle/diag/rdbms/rac/RAC2: ************************************************************************* ADRID SHORTP_POLICY LONGP_POLICY LAST_MOD_TIME LAST_AUTOPRG_TIME LAST_MANUPRG_TIME ADRDIR_VERSION ADRSCHM_VERSION ADRSCHMV_SUMMARY ADRALERT_VERSION CREATE_TIME -------------------- -------------------- -------------------- ---------------------------------------- ---------------------------------------- --------------------------------------- -------------------- -------------------- -------------------- -------------------- ---------------------------------------4067289953 720 8760 2013-06-28 05:37:42.235895 +05:30 1 2 76 1 2013-06-28 05:37:42.235895 +05:30 1 rows fetched adrci> show tracefile -rt 09-NOV-13 00:26:32 diag/rdbms/rac/RAC2/trace/RAC2_vktm_6761.trc 09-NOV-13 00:20:15 diag/rdbms/rac/RAC2/trace/alert_RAC2.log 09-NOV-13 00:20:13 diag/rdbms/rac/RAC2/trace/RAC2_diag_6772.trc 09-NOV-13 00:20:12 diag/rdbms/rac/RAC2/incident/incdir_17097/RAC2_ora_9226_i17097.trc 09-NOV-13 00:20:12 diag/rdbms/rac/RAC2/trace/RAC2_ora_9226.trc .. .adrci> purge -age 2880 -type trace
  22. 22. ADRCI Commands 1. Create a package (A logical container for the diagnostic data) 2. Add diagnostic data into the package (from an incident or manually by adding trace files) 3. Generate a 'physical' package (file on the filesystem) 4. Finalize the package Step 1. adrci> ips create package problem 1 correlate all Created package 1 based on problem id 1, correlation level all adrci> ips generate package 2 in "/home/oracle" Step 2. adrci>ips add incident 32578 package 3; Added incident 32578 to package 3 To manually add a tracefile into the package adrci ips add file <path to tracefile> package 3 Step 3. adrci> ips generate package 1 in "/home/oracle" Generated package 1 in file /home/oracle/, mode complete Step 4. adrci> ips finalize package 3 When reviewing a database incident, even if you may not forward it on to Oracle Support it is helpful to generate the package zip file then unzipping if for your own review.
  23. 23. OCM(Oracle Configuration Manager) Oracle Configuration Manager helps us with lesser headaches, Since the details of our configuration changes, Accurate planning & upgrade are proactively foreseen by Oracle support. Also, by pairing My oracle support with OCM, we can leverage our Oracle support, with latest support information, Online communities, Security recommendations & Product alerts. Oracle Configuration Manager is used to collect client configuration information and upload it to the Oracle repository. When client configuration data is uploaded on a regular basis, customer support representatives can analyze this data and provide better service to the customers. If you want to install OCM in an Oracle E-Business Suite Release 12 installation, you must use Rapid Wizard and / or AutoConfig to install and configure Oracle Configuration Manager. This process, along with known issues and other considerations for E-Business Suite users, is described in the (DOC ID 406369.1) Information Center: Configuration Management (Doc ID 1457157.2) OCM (Oracle Configuration Management) RDA (Remote Diagnostic Agent) EM (Enterprise Manager Cloud Control 12c) MOS (My Oracle Support)
  24. 24. Looking At Issues Proactively & From Greater Heights No time for implementing Oracle recommended best practices? Implemented Oracle recommended best practices in phased manner.
  25. 25. References References 1. My Oracle Support 2. Steven Chan’s blog (
  26. 26. Q&A
  27. 27. Thank You Blog – E-Mail - Twitter - @ajithpathiyil Facebook - ajithpathiyil