cmandtracing-1560359.pdf

Performance Tuning Oracle E-Business Suite
Part 2: Concurrent Processing and Tracing
Andy Tremayne, Senior Architect & Uday Moogala, Senior Principal
Applications Performance Oracle Development

<Insert Picture Here>
Agenda
• Tuning the Concurrent Manager
• Tracing

Concurrent Manager
• 50% of performance tuning is in the business!
– Review long-running auto resubmitted jobs
– Review short/long-running jobs – selective parameters?
– Don’t purge jobs that the users then have to run again
• Points to note
– More managers/workers != more throughput
– Avoid enabling an excessive number of managers/workers
• New systems: a start point is 1-2 target process per CPU
– Balance with resources required by online activity
Introduction

Concurrent Manager
• Conflict Resolution Manager
– Carefully consider reducing the CRM sleep time to 5 ~ 10 secs
• Purge issues
– 2GB limit on any output files… esp. reports.log (Doc ID 842850.1)
– Set Concurrent: Force Local Output File Mode" to "Yes“
– Patch 7530490 for R12 or 7834670 for 11i (Doc ID 822368.1)
• High throughput: Keep out and log on fast disks
– Separate if throughput seems slow esp. on NFS Filers
• Use IPC or bequeath if the CM and DB are co-located
General Points

Concurrent Manager
• Use queues in RAC
– System Profile: Concurrent:TM Transport Type: Pipe/Queue
– Failover
– Available in 11i (Doc ID 413636.1)
– For R12 (or 11i post Oracle 10.1) AQ is largely self-tuning
• Single instance can still use pipes
– Some customers have reported better performance
– Description of how pipes work Doc ID 105133.1
Pipes vs Queue

Concurrent Manager
• Synchronous processing: PO Approvals…
• PO, CRP, INV, AR, and OE all ship TMs
– Set the profile Concurrent:Wait for Available TM to 1 (second)
– Set the sleep time on the TMs to a high number (e.g. 10 minutes)
– Setting timeouts too low may return a TIMEOUT before completion
• But the job then completes
• Tracing: set 'Initialization SQL - Custom‘
– fnd_transaction.synchronous(rpc_timeout,:2,:3,'PO','RCVTPO'….
WAIT #420: nam='pipe get' ela= 2929821 handle address=29374777744 buffer
length=4096 timeout=172800 obj#=4292320 tim=4590000269038
Transaction Managers

Concurrent Manager
Concurrent
Requests
At least 1 job
Time


Select for Update
Select for Update
Select for Update
Select for Update
Select for Update
Select for Update




Job Status
A
B
C
D
E
Job Run
Job Run
Job Run
Job Run
Select Runnable
Select Runnable
Select Runnable
Select Runnable
A
B
A
B
A
B
A
B
A B
A B
A B
A B
A
B
A
B
A
B
A
B
Lock?
Lock?
Lock?
Lock?
Running

X
X
X
Select for Update
Select for Update
Select for Update




Running
X
X
Update Set A Running
Update Set B Running
Running
Select Runnable
Select Runnable
C
D
C
D
C
D
C
D
C D
C D


Lock?
Lock?
Update Set C Running

X
Running
Select Runnable
Select for Update
Update Set A Complete
Running

Select for Update
Select Runnable
Complete
Update Set D Running
E
E
E 
Update Set E Running
Lock?

Running
Update Set D Complete
Complete
None
Worker 1
Worker 2
Worker 3
Worker 4
FND_CONCURRENT_REQUESTS
Sleep
Concurrent Manager

Concurrent Manager
• Avoid short sleep times
– *Enough for a couple of minutes of work
– **Set cache size to at least twice the number of workers
• Specialization rules and work shifts
– Bind specific jobs to specific time windows
– Profile Option: Concurrent: Active Request Limit
– Specialize if there are too many jobs in a specific category
Workload Management Strategy
Fast
Medium
Slow
Long
Critical
Cache
Enough*
2xworkers
= 2xworkers**
= workers**
?
Sleep
15 - 20
60
60
60
30 or 60
Workers
Lots
A few
Not many
Very few
?
Avg. Duration
10s
5 mins
30 mins
2 hours +
N/A

Concurrent Manager
How do you know your strategy needs review?
• More workers = more accesses
• 9301929 (fix 12.1.4) modifies fnd_concurrent_requests_u2
– Adds program_application_id and concurrent_program_id

Concurrent Manager
• 11.5.10
– 4 Node RAC: 40 CPUs per node
– 5,000 concurrent users
– In addition to “standard processing”
• 400,000 concurrent requests per day (24,000 per hour peak)
• Typically run times 2 ~ 3 seconds
• Using “Fast, Medium and Slow” strategy
– CPU running at 85% on all nodes and jobs were queuing
Implementation Example: The Problem

Concurrent Manager
• 4 Specialized Managers
– 15 Workers, Cache 30, Sleep15s
– 24,000 jobs/hr = 400 jobs/worker/hr = ~800~1200 secs/hr/worker
• 4 Managers/15 Workers = 1 manager/60 workers
– Each scan caches 30 jobs = 60~90 secs work
– Divide that by 60 processes – avg. 1 sec work/worker
– Add back the other managers
– Scan 2+ million row table 2 – 3 times/second
Implementation Example: The Workload

Concurrent Manager
• Get enough work for a few minutes of work
• Customer still had things to work out (in red)
• CPU Improvement: 85% => 25% with no backlog
Implementation Example: The Strategy
Number Manager Workers Cache Sleep Description
1 Fast 30 100 15 Jobs < 2 mins
1 Intermediate 15 70 60 Jobs >2 <40 mins
1 Slow 2 20 180 Jobs > 40 mins
4 Very Fast 30 400 10 Fast specialized < 3secs
3 Standard managers 30 20 60 Mixed workload
3 Other specialized 5 50 60 Other Specialized jobs
20 Other (specialized) 50 ~ ~ Various

Concurrent Manager
• Goal: Direct some workload components to specific RAC
nodes to maximize scalability by minimizing inter-node
communication and synchronization work
– Put parent and all child processes on the same RAC node
– RAC/PCP Doc ID1359612.1
• May cause an overload one node
– Decision: load balance or affinitize
– Can also happen on Failover
– Recommended: set the profile option ‘Concurrent: PCP Instance
Check ‘ to OFF
Affinity

Concurrent Manager
Affinity
• PCP: 1 – 1 correspondence of CM nodes to RAC nodes
– Set <s_cp_twotask> per CM node to tie to a specific RAC instance
– Define primary/secondary CM nodes for failover
• Use dedicated queues
– Use specialization rules for programs with high scalability needs
– Group by application, program, user, request_type; examples:
• OM, Receivables Super User, Create Accounting
• WF background engines Node to ITEM_TYPE assignment
• 12.1 define node affinity at the program level
– Concurrent | Program | Define ( See Doc ID 1129203.1)

Concurrent Manager
• Forms
– Set Profile Option "Database Instance" at App or Resp level
which can be tied to two_task value or to a service
• Self Service
– Set Profile Options “App%Agent” to web tier hosts configured
for specific services …or
– Set Profile Option "Applications Database ID" to node specific
DBC file name
Extending Affinity to the Applications Tier

Concurrent Manager
• 16TB DB 4 Node Cluster, R12, Oracle 11.1.0.7
• 15K concurrent peak users
Affinity Examples
Node 1
Order Management
Install Base
Shipping
Education
Contracts
Node 2
Payroll
HR (with self-service)
Benefits
OTL (self-service)
Node 3
Financials –
GL, AP, PO, FA
Self-Service –
iProc, Expenses
DBI
Node 4
Accounts Receivable
Project Accounting
Imaging

Concurrent Manager
• Large AR Telco system: Batch Affinity
– Node 1: AutoInvoice,Create Accounting-Invoices
– Node 2: AutoReceipt, Remittance, Create Accounting- Receipts
• Order Management Benchmark: Batch Affinity
– Node 1: Order Management, Shipping, WF background- OE Item Type
– Node 2: Financials, Procurement, WF for PO item types
• Incentive Comp system: No Affinity
– 4 node cluster, commodity HW; nodes too small to handle required OIC batch
workload in one node
– Scaled to customer’s needs with RAC to required throughputs, BUT …
– Needed custom partitioning, and several development and custom code
optimizations, index changes/drops and other tuning
Affinity Examples

Eg) Network Traffic received
= 232.31*8k=1.9 Mb/sec
Concurrent Manager
• Baseline & Monitor AWR
– Global Cache Load Profile: “Estimated Interconnect Traffic”
– Global Cache and Enqueue Services - Messaging Statistics
• “Average message sent queue time on ksxp”
• IPC efficiency > 2ms = interconnect/CPU capacity/issue
Interconnect Messaging Performance – Instance Level

Concurrent Manager
• E-Business Suite Technology-Related Sessions at OpenWorld 2011
• https://blogs.oracle.com/stevenChan/entry/e_business_suite_technology_related
Interconnect Messaging Performance – Instance Level

Concurrent Manager
Interconnect Messaging Performance – Statement Level

Concurrent Manager
• Much of the functionality has now been built into 12.1
• Procedure: How to get the REQUEST_ID into a shell script
Enhancing The Concurrent Manager (Doc ID 164085.1)

Concurrent Manager
• Program Level
– Not recommended
– Enable the trace/profiling option in the concurrent program definition
• Request Level
– Ensure “Concurrent: Allow Debugging” is set to ‘Yes’
– Use the Debug Options button in the Request Submission Form
• Tracing Documents
– 280391.1 How to Retrieve SID Information For a Running Request
Tracing

Doc ID Document
1359612.1 Webcast 'E-Business Suite - RAC & Parallel Concurrent Processing
1367676.1 Webcast 'E-Business Suite - Concurrent Manager Performance
164085.1 Enhancing and Automating Oracle Applications Concurrent Processing
1057802.1 Best Practices for Performance for Concurrent Managers
1304305.1 E-Business Concurrent Processing Information Center
Concurrent Manager
Useful References

Agenda
• Tuning the Concurrent Manager
• Tracing

Tracing
• Good idea to create a user called “Trace”
– Collecting Diagnostic Data for Performance Issues in Oracle E-
Business Suite - Doc ID 1121043.1
– Enhanced Performance Diagnostics for Oracle E-Business Suite
Release 12.1.3 – Doc ID 1362660.1
• For 11g apply tracing related patches
– 7522002, 8342329, 11687336, and 8328200
• SQL Trace Level 8 (with waits)
– Raw trace and TKPROF
General Points

Tracing
• Forms: Help  Diagnostics Trace
• OA HTML: Diagnostics  Tracing
• Catch-All
– Set the profile: “Initialization SQL Statement - Custom"
BEGIN
FND_CTL.FND_SESS_CTL('','','TRUE','TRUE','','ALTER SESSION
SET TRACEFILE_IDENTIFIER=''TESTING''
MAX_DUMP_FILE_SIZE=unlimited STATISTICS_LEVEL = ALL
EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 8''');
END;
Catch-All

Tracing
• max_dump_file_size=unlimited
– Ensure there is sufficient disk space in the tracing directory
• user_dump_dest or if diagnostic_dest is set in Oracle11g
– (diagnostic_dest)/diag/rdbms/(dbname)/(instname)/trace
– background_dump_dest for parallel slave processes
• STATISTICS_LEVEL=ALL
Parameters

Tracing
• Run TKPROF from the correct Oracle home
– Use the sort options = fchela,exeela,prsela
• Checks
– Check file contains runtime execution plans and row source stats
– Check the last line: “elapsed seconds in trace file“
Plan Completeness

Tracing
Rows Row Source Operation
------- ---------------------------------------------------
0 NESTED LOOPS (cr=5 pr=0 pw=0 time=0 us cost=4 size=71 card=1)
1 TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTIONS (cr=3 pr=0 pw=0 time=0 us cost=2 size=44 card=1)
1 INDEX UNIQUE SCAN FND_PROFILE_OPTIONS_U2 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 215619)
0 TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTION_VALUES (cr=2 pr=0 pw=0 time=0 us cost=2 size=27 card=1)
0 INDEX RANGE SCAN FND_PROFILE_OPTION_VALUES_U1 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 224902)
Rows Execution Plan
------- ---------------------------------------------------
0 SELECT STATEMENT MODE: ALL_ROWS
0 NESTED LOOPS
1 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTIONS' (TABLE)
1 INDEX MODE: ANALYZED (UNIQUE SCAN) OF 'FND_PROFILE_OPTIONS_XX_1' (INDEX (UNIQUE))
0 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTION_VALUES' (TABLE)
0 INDEX MODE: ANALYZED (RANGE SCAN) OF 'FND_PROFILE_OPTION_VALUES_U1' (INDEX (UNIQUE))
Plan Completeness
Explain option == ‘explain plan for’ on SQL*Plus
Actual runtime plan needed for diagnosis
• New tracing option: plan_stat=adaptive (patch 8328200)
– Sampled statistics are dumped for every minute of DB time

Tracing
• Trace Analyzer (TRCA)
– Doc ID 224270.1
– Reads an Event 10046 SQL Trace file
– Provides a comprehensive report for performance analysis and
tuning
• Interpreting Raw SQL_TRACE files Doc ID 39817.1
Event 10046

Tracing
• SQLTXPLAIN (aka SQLT)
– Maintained by Oracle Support - often requested
– Analyses why the execution plan was chosen
• Always use the latest version
– Doc ID 215187.1
• XTRACT report is preferred output
– Focuses on actual SQL execution runtime data
SQLT XTRACT for Poorly Performing SQL

Tracing
SQLT XTRACT for Poorly Performing SQL

Tracing
SQLT XTRACT - Completeness
With
Statistics_level = ALL

Tracing
• Doc ID 1362660.1/Patch 12544073
• Trace a session
– Run Gather Diagnostic Data for Traced Sessions
• AWR and ADDM Reports
• ASH Report
• Session Statistics
• Trace Analyzer output, which includes SQLT output for top SQLs
• Object statistics when a SQL statement is traced, but SQLT output is not chosen or not installed
• SQLT output (requires SQLT tool – Doc ID 215187.1
• Real-Time SQL Monitor Report when a SQL statement is traced (in Oracle Database 11g and higher)
• SQL Detail Report when a SQL Statement is traced (in Oracle Database 11gR2 and higher)
• Also use to capture information for individual SQL(s)
Enhanced Tracing in R12.1.3

Tracing
• AWR (Automatic Workload Repository) Report
• ADDM Report
• ASH (Active Session History) Report
• Real-Time SQL Monitoring Report (11g+)
– Parallel queries and queries taking > 5 seconds
• Operating system statistics
– OSW Document Doc ID 301137.1
– OSWg Document Doc ID 461053.1
System Level Performance Information

Tracing
Real-Time SQL Monitoring Report

Tracing
• AWR is to an entire SYSTEM
– TRCA is to a PROCESS
– SQLT is to ONE SQL
• ASH
– Session, Module, SQL etc
• Process
– SQLTrace followed by TKPROF and/or TRCA
• ONE SQL
– SQLT using XTRACT method
Takeaway

Doc ID Document
1121043.1 Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite
1362660.1 Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3
301137.1 OSW Document
461053.1 OSWg Document
215187.1 SQLTXPLAIN (SQLT)
224270.1 Trace Analyzer (TRCA)
39817.1 Interpreting Raw SQL_TRACE files
280391.1 How to Retrieve SID Information For a Running Request
Tracing
Useful References

PC and Network Performance
• PC
– Windows 7 vs Windows XP
– Longevity of old clients
• Network
– Real world network maps
– WAN emulation
– WAN and VSAT optimization
March 22nd

Q&A

cmandtracing-1560359.pdf

Recommended

Recommended

More Related Content

Similar to cmandtracing-1560359.pdf

Similar to cmandtracing-1560359.pdf (20)

Recently uploaded

Recently uploaded (20)

cmandtracing-1560359.pdf