This document discusses tuning the concurrent manager and tracing in Oracle E-Business Suite. It provides tips for tuning the concurrent manager such as reviewing job durations, balancing managers and workers, using affinity, and implementing specialized workload strategies. It also covers setting up tracing to troubleshoot performance issues, including setting the necessary initialization parameters and analyzing the trace files.

1. 1 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

2. 2 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Performance Tuning Oracle E-Business Suite
Part 2: Concurrent Processing and Tracing
Andy Tremayne, Senior Architect & Uday Moogala, Senior Principal
Applications Performance Oracle Development

3. 3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
<Insert Picture Here>
Agenda
• Tuning the Concurrent Manager
• Tracing

4. 4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• 50% of performance tuning is in the business!
– Review long-running auto resubmitted jobs
– Review short/long-running jobs – selective parameters?
– Don’t purge jobs that the users then have to run again
• Points to note
– More managers/workers != more throughput
– Avoid enabling an excessive number of managers/workers
• New systems: a start point is 1-2 target process per CPU
– Balance with resources required by online activity
Introduction

5. 5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Conflict Resolution Manager
– Carefully consider reducing the CRM sleep time to 5 ~ 10 secs
• Purge issues
– 2GB limit on any output files… esp. reports.log (Doc ID 842850.1)
– Set Concurrent: Force Local Output File Mode" to "Yes“
– Patch 7530490 for R12 or 7834670 for 11i (Doc ID 822368.1)
• High throughput: Keep out and log on fast disks
– Separate if throughput seems slow esp. on NFS Filers
• Use IPC or bequeath if the CM and DB are co-located
General Points

6. 6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Use queues in RAC
– System Profile: Concurrent:TM Transport Type: Pipe/Queue
– Failover
– Available in 11i (Doc ID 413636.1)
– For R12 (or 11i post Oracle 10.1) AQ is largely self-tuning
• Single instance can still use pipes
– Some customers have reported better performance
– Description of how pipes work Doc ID 105133.1
Pipes vs Queue

7. 7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Synchronous processing: PO Approvals…
• PO, CRP, INV, AR, and OE all ship TMs
– Set the profile Concurrent:Wait for Available TM to 1 (second)
– Set the sleep time on the TMs to a high number (e.g. 10 minutes)
– Setting timeouts too low may return a TIMEOUT before completion
• But the job then completes
• Tracing: set 'Initialization SQL - Custom‘
– fnd_transaction.synchronous(rpc_timeout,:2,:3,'PO','RCVTPO'….
WAIT #420: nam='pipe get' ela= 2929821 handle address=29374777744 buffer
length=4096 timeout=172800 obj#=4292320 tim=4590000269038
Transaction Managers

8. 8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
Concurrent
Requests
At least 1 job
Time

9. 9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Select for Update
Select for Update
Select for Update
Select for Update
Select for Update
Select for Update




Job Status
A
B
C
D
E
Job Run
Job Run
Job Run
Job Run
Select Runnable
Select Runnable
Select Runnable
Select Runnable
A
B
A
B
A
B
A
B
A B
A B
A B
A B
A
B
A
B
A
B
A
B
Lock?
Lock?
Lock?
Lock?
Running

X
X
X
Select for Update
Select for Update
Select for Update




Running
X
X
Update Set A Running
Update Set B Running
Running
Select Runnable
Select Runnable
C
D
C
D
C
D
C
D
C D
C D


Lock?
Lock?
Update Set C Running

X
Running
Select Runnable
Select for Update
Update Set A Complete
Running

Select for Update
Select Runnable
Complete
Update Set D Running
E
E
E 
Update Set E Running
Lock?

Running
Update Set D Complete
Complete
None
Worker 1
Worker 2
Worker 3
Worker 4
FND_CONCURRENT_REQUESTS
Sleep
Concurrent Manager

10. 10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Avoid short sleep times
– *Enough for a couple of minutes of work
– **Set cache size to at least twice the number of workers
• Specialization rules and work shifts
– Bind specific jobs to specific time windows
– Profile Option: Concurrent: Active Request Limit
– Specialize if there are too many jobs in a specific category
Workload Management Strategy
Fast
Medium
Slow
Long
Critical
Cache
Enough*
2xworkers
= 2xworkers**
= workers**
?
Sleep
15 - 20
60
60
60
30 or 60
Workers
Lots
A few
Not many
Very few
?
Avg. Duration
10s
5 mins
30 mins
2 hours +
N/A

11. 11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
How do you know your strategy needs review?
• More workers = more accesses
• 9301929 (fix 12.1.4) modifies fnd_concurrent_requests_u2
– Adds program_application_id and concurrent_program_id

12. 12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• 11.5.10
– 4 Node RAC: 40 CPUs per node
– 5,000 concurrent users
– In addition to “standard processing”
• 400,000 concurrent requests per day (24,000 per hour peak)
• Typically run times 2 ~ 3 seconds
• Using “Fast, Medium and Slow” strategy
– CPU running at 85% on all nodes and jobs were queuing
Implementation Example: The Problem

13. 13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• 4 Specialized Managers
– 15 Workers, Cache 30, Sleep15s
– 24,000 jobs/hr = 400 jobs/worker/hr = ~800~1200 secs/hr/worker
• 4 Managers/15 Workers = 1 manager/60 workers
– Each scan caches 30 jobs = 60~90 secs work
– Divide that by 60 processes – avg. 1 sec work/worker
– Add back the other managers
– Scan 2+ million row table 2 – 3 times/second
Implementation Example: The Workload

14. 14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Get enough work for a few minutes of work
• Customer still had things to work out (in red)
• CPU Improvement: 85% => 25% with no backlog
Implementation Example: The Strategy
Number Manager Workers Cache Sleep Description
1 Fast 30 100 15 Jobs < 2 mins
1 Intermediate 15 70 60 Jobs >2 <40 mins
1 Slow 2 20 180 Jobs > 40 mins
4 Very Fast 30 400 10 Fast specialized < 3secs
3 Standard managers 30 20 60 Mixed workload
3 Other specialized 5 50 60 Other Specialized jobs
20 Other (specialized) 50 ~ ~ Various

15. 15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Goal: Direct some workload components to specific RAC
nodes to maximize scalability by minimizing inter-node
communication and synchronization work
– Put parent and all child processes on the same RAC node
– RAC/PCP Doc ID1359612.1
• May cause an overload one node
– Decision: load balance or affinitize
– Can also happen on Failover
– Recommended: set the profile option ‘Concurrent: PCP Instance
Check ‘ to OFF
Affinity

16. 16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
Affinity
• PCP: 1 – 1 correspondence of CM nodes to RAC nodes
– Set <s_cp_twotask> per CM node to tie to a specific RAC instance
– Define primary/secondary CM nodes for failover
• Use dedicated queues
– Use specialization rules for programs with high scalability needs
– Group by application, program, user, request_type; examples:
• OM, Receivables Super User, Create Accounting
• WF background engines Node to ITEM_TYPE assignment
• 12.1 define node affinity at the program level
– Concurrent | Program | Define ( See Doc ID 1129203.1)

17. 17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Forms
– Set Profile Option "Database Instance" at App or Resp level
which can be tied to two_task value or to a service
• Self Service
– Set Profile Options “App%Agent” to web tier hosts configured
for specific services …or
– Set Profile Option "Applications Database ID" to node specific
DBC file name
Extending Affinity to the Applications Tier

18. 18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• 16TB DB 4 Node Cluster, R12, Oracle 11.1.0.7
• 15K concurrent peak users
Affinity Examples
Node 1
Order Management
Install Base
Shipping
Education
Contracts
Node 2
Payroll
HR (with self-service)
Benefits
OTL (self-service)
Node 3
Financials –
GL, AP, PO, FA
Self-Service –
iProc, Expenses
DBI
Node 4
Accounts Receivable
Project Accounting
Imaging

19. 19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Large AR Telco system: Batch Affinity
– Node 1: AutoInvoice,Create Accounting-Invoices
– Node 2: AutoReceipt, Remittance, Create Accounting- Receipts
• Order Management Benchmark: Batch Affinity
– Node 1: Order Management, Shipping, WF background- OE Item Type
– Node 2: Financials, Procurement, WF for PO item types
• Incentive Comp system: No Affinity
– 4 node cluster, commodity HW; nodes too small to handle required OIC batch
workload in one node
– Scaled to customer’s needs with RAC to required throughputs, BUT …
– Needed custom partitioning, and several development and custom code
optimizations, index changes/drops and other tuning
Affinity Examples

20. 20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Eg) Network Traffic received
= 232.31*8k=1.9 Mb/sec
Concurrent Manager
• Baseline & Monitor AWR
– Global Cache Load Profile: “Estimated Interconnect Traffic”
– Global Cache and Enqueue Services - Messaging Statistics
• “Average message sent queue time on ksxp”
• IPC efficiency > 2ms = interconnect/CPU capacity/issue
Interconnect Messaging Performance – Instance Level

21. 21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• E-Business Suite Technology-Related Sessions at OpenWorld 2011
• https://blogs.oracle.com/stevenChan/entry/e_business_suite_technology_related
Interconnect Messaging Performance – Instance Level

22. 22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
Interconnect Messaging Performance – Statement Level

23. 23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Much of the functionality has now been built into 12.1
• Procedure: How to get the REQUEST_ID into a shell script
Enhancing The Concurrent Manager (Doc ID 164085.1)

24. 24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Concurrent Manager
• Program Level
– Not recommended
– Enable the trace/profiling option in the concurrent program definition
• Request Level
– Ensure “Concurrent: Allow Debugging” is set to ‘Yes’
– Use the Debug Options button in the Request Submission Form
• Tracing Documents
– 280391.1 How to Retrieve SID Information For a Running Request
Tracing

25. 25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Doc ID Document
1359612.1 Webcast 'E-Business Suite - RAC & Parallel Concurrent Processing
1367676.1 Webcast 'E-Business Suite - Concurrent Manager Performance
164085.1 Enhancing and Automating Oracle Applications Concurrent Processing
1057802.1 Best Practices for Performance for Concurrent Managers
1304305.1 E-Business Concurrent Processing Information Center
Concurrent Manager
Useful References

26. 26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
<Insert Picture Here>
<Insert Picture Here>
Agenda
• Tuning the Concurrent Manager
• Tracing

27. 27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• Good idea to create a user called “Trace”
– Collecting Diagnostic Data for Performance Issues in Oracle E-
Business Suite - Doc ID 1121043.1
– Enhanced Performance Diagnostics for Oracle E-Business Suite
Release 12.1.3 – Doc ID 1362660.1
• For 11g apply tracing related patches
– 7522002, 8342329, 11687336, and 8328200
• SQL Trace Level 8 (with waits)
– Raw trace and TKPROF
General Points

28. 28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• Forms: Help  Diagnostics Trace
• OA HTML: Diagnostics  Tracing
• Catch-All
– Set the profile: “Initialization SQL Statement - Custom"
BEGIN
FND_CTL.FND_SESS_CTL('','','TRUE','TRUE','','ALTER SESSION
SET TRACEFILE_IDENTIFIER=''TESTING''
MAX_DUMP_FILE_SIZE=unlimited STATISTICS_LEVEL = ALL
EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 8''');
END;
Catch-All

29. 29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• max_dump_file_size=unlimited
– Ensure there is sufficient disk space in the tracing directory
• user_dump_dest or if diagnostic_dest is set in Oracle11g
– (diagnostic_dest)/diag/rdbms/(dbname)/(instname)/trace
– background_dump_dest for parallel slave processes
• STATISTICS_LEVEL=ALL
Parameters

30. 30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• Run TKPROF from the correct Oracle home
– Use the sort options = fchela,exeela,prsela
• Checks
– Check file contains runtime execution plans and row source stats
– Check the last line: “elapsed seconds in trace file“
Plan Completeness

31. 31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
Rows Row Source Operation
------- ---------------------------------------------------
0 NESTED LOOPS (cr=5 pr=0 pw=0 time=0 us cost=4 size=71 card=1)
1 TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTIONS (cr=3 pr=0 pw=0 time=0 us cost=2 size=44 card=1)
1 INDEX UNIQUE SCAN FND_PROFILE_OPTIONS_U2 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 215619)
0 TABLE ACCESS BY INDEX ROWID FND_PROFILE_OPTION_VALUES (cr=2 pr=0 pw=0 time=0 us cost=2 size=27 card=1)
0 INDEX RANGE SCAN FND_PROFILE_OPTION_VALUES_U1 (cr=2 pr=0 pw=0 time=0 us cost=1 size=0 card=1)(object id 224902)
Rows Execution Plan
------- ---------------------------------------------------
0 SELECT STATEMENT MODE: ALL_ROWS
0 NESTED LOOPS
1 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTIONS' (TABLE)
1 INDEX MODE: ANALYZED (UNIQUE SCAN) OF 'FND_PROFILE_OPTIONS_XX_1' (INDEX (UNIQUE))
0 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'FND_PROFILE_OPTION_VALUES' (TABLE)
0 INDEX MODE: ANALYZED (RANGE SCAN) OF 'FND_PROFILE_OPTION_VALUES_U1' (INDEX (UNIQUE))
Plan Completeness
Explain option == ‘explain plan for’ on SQL*Plus
Actual runtime plan needed for diagnosis
• New tracing option: plan_stat=adaptive (patch 8328200)
– Sampled statistics are dumped for every minute of DB time

32. 32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• Trace Analyzer (TRCA)
– Doc ID 224270.1
– Reads an Event 10046 SQL Trace file
– Provides a comprehensive report for performance analysis and
tuning
• Interpreting Raw SQL_TRACE files Doc ID 39817.1
Event 10046

33. 33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• SQLTXPLAIN (aka SQLT)
– Maintained by Oracle Support - often requested
– Analyses why the execution plan was chosen
• Always use the latest version
– Doc ID 215187.1
• XTRACT report is preferred output
– Focuses on actual SQL execution runtime data
SQLT XTRACT for Poorly Performing SQL

34. 34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
SQLT XTRACT for Poorly Performing SQL

35. 35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
SQLT XTRACT - Completeness
With
Statistics_level = ALL

36. 36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• Doc ID 1362660.1/Patch 12544073
• Trace a session
– Run Gather Diagnostic Data for Traced Sessions
• AWR and ADDM Reports
• ASH Report
• Session Statistics
• Trace Analyzer output, which includes SQLT output for top SQLs
• Object statistics when a SQL statement is traced, but SQLT output is not chosen or not installed
• SQLT output (requires SQLT tool – Doc ID 215187.1
• Real-Time SQL Monitor Report when a SQL statement is traced (in Oracle Database 11g and higher)
• SQL Detail Report when a SQL Statement is traced (in Oracle Database 11gR2 and higher)
• Also use to capture information for individual SQL(s)
Enhanced Tracing in R12.1.3

37. 37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• AWR (Automatic Workload Repository) Report
• ADDM Report
• ASH (Active Session History) Report
• Real-Time SQL Monitoring Report (11g+)
– Parallel queries and queries taking > 5 seconds
• Operating system statistics
– OSW Document Doc ID 301137.1
– OSWg Document Doc ID 461053.1
System Level Performance Information

38. 38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
Real-Time SQL Monitoring Report

39. 39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracing
• AWR is to an entire SYSTEM
– TRCA is to a PROCESS
– SQLT is to ONE SQL
• ASH
– Session, Module, SQL etc
• Process
– SQLTrace followed by TKPROF and/or TRCA
• ONE SQL
– SQLT using XTRACT method
Takeaway

40. 40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Doc ID Document
1121043.1 Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite
1362660.1 Enhanced Performance Diagnostics for Oracle E-Business Suite Release 12.1.3
301137.1 OSW Document
461053.1 OSWg Document
215187.1 SQLTXPLAIN (SQLT)
224270.1 Trace Analyzer (TRCA)
39817.1 Interpreting Raw SQL_TRACE files
280391.1 How to Retrieve SID Information For a Running Request
Tracing
Useful References

41. 41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
PC and Network Performance
• PC
– Windows 7 vs Windows XP
– Longevity of old clients
• Network
– Real world network maps
– WAN emulation
– WAN and VSAT optimization
March 22nd

42. 42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Q&A

43. 43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

44. 44 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.