• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Internals of concurent managers
 

Internals of concurent managers

on

  • 836 views

This is my old presentation about internals of Concurrent managers in e-Business Suite. Presented at OUG Harmony 2011 in Helsinki, Finland.

This is my old presentation about internals of Concurrent managers in e-Business Suite. Presented at OUG Harmony 2011 in Helsinki, Finland.

Statistics

Views

Total Views
836
Views on SlideShare
836
Embed Views
0

Actions

Likes
0
Downloads
42
Comments
1

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Internals of concurent managers Internals of concurent managers Presentation Transcript

    • Inside Concurrent Managers OUG Harmony 2011© 2011 Tieto Corporation Māris Elsiņš Lead Oracle [Applications] DBA Tieto Latvia, Maris.Elsins@tieto.com
    • Who I am?• 9 years in IT • 3 years – PL/SQL developer • 6 years – Oracle [Apps] DBA (started with 11.5.7 and 8.1.7)• Certificates • 10g OCM • 9i / 10g / 11g OCP • 11i Applications Database Administrator OCP • 11i System Administrator OCE• Conferences • UKOUG 2007/2008/2010 • LVOUG 2009/2010 • EMEA Harmony 2010/2011 • Current employer – Tieto Latvia • All kinds of oracle DBA tasks - patching, upgrade, performance tuning, troubleshooting, planning and implementation of backup and recovery procedures, cross platform migration, etc • Planning of sydtem architecture, design and implementation of HA solutions (RAC, Data Guard, Custom cold failover) • Implementation of system specific monitoring, automation of routine tasks • Technical project planning and coordination, management of team of DBAs2 © 2011 Tieto Corporation 2010-11-29
    • Purpose of this session• Provide background knowledge for successful troubleshooting of concurrent manager problems• What this presentation is about? • Life cycle of a concurrent request • Implementation details of the most interesting phases of a life cycle of a concurrent request • Internals of Concurrent Managers and Conflict Resolution Managers • DB objects that can be used for troubleshooting• Why it is important? • DBA has to be quick when solving real issues • Knowing the process before the problems happen decreases solving time • Not everything is visible using the GUI • Querying DB objects is faster then navigating through Forms • Understanding the process is important when tuning the setup of Concurrent Managers3 © 2011 Tieto Corporation 2010-11-29
    • Most interesting DB objects• Objects • FND_CONCURRENT_REQUESTS (FCR) • FND_CONCURRENT_PROGRAMS (FCP) • FND_CONCURRENT_QUEUES (FCQ) • FND_CONCURRENT_PROCESSES (FCPROC) • FND_CONCURRENT_PROGRAM_SERIAL (FCPS) • FND_CONC_RELEASE_CLASSES (FCRC) • FND_LOOKUPS (FL) • FND_CONCURRENT_WORKER_REQESTS (FCWR) • FND_CRM_HISTORY (FCH) • FND_CONC_WAITING_REQUESTS• Some examples how the tables are used will be uncovered in this presentation4 © 2011 Tieto Corporation 2010-11-29
    • Types of concurrent managers • Types of concurrent managers • Internal Concurrent Manager • Service Manager • Concurrent Manager • Conflict Resolution Manager • Internal Monitor • Transaction Manager • Scheduler/Prereleaser Manager • … • Today the accent is on • Concurrent Manager • Conflict Resolution Manager5 © 2011 Tieto Corporation 2010-11-29
    • Life cycle of a concurrent request INACTIVE (I) PENDING (P) RUNNING (R) COMPLETED (C) • Disabled (U) • Normal (I) • Normal (R) • Normal (C) • On Hold (H) • Standby (Q) • Paused (W) • Error (E) • No Manager (M) • Scheduled (P) • Resuming (B) • Warning (G) • Waiting (Z) • Terminating (T) • Cancelled (D) • Terminated (X)FND_LOOKUPS contains the values of Phase/Status Codes«Oracle® E-Business Suite System Administrators Guide – Maintenance»describes the meaning of each phase and status 6 © 2011 Tieto Corporation 2010-11-29
    • Typical life cycle of most requestsPending / Scheduled• Request is scheduled for the execution in future Pending / Standby • Time to execute the request has arrived • Request waits for to be evaluated by the Conflict Resolution Manager Pending / Normal • Request is allowed to be executed • Request is waiting to be picked up by Concurrent Manager Running / Running • Request is beeing executed by a concurrent manager Completed / Normal7 © 2011 Tieto Corporation 2010-11-29
    • Phase/Status representation in tables• FND_CONCURRENT_REQUESTS • PHASE_CODE • STATUS_CODE• Is that so simple? NO! • Phase = Inactive practically not used • Inactive/Disabled: FCR.PHASE_CODE=„P‟, FCP.ENABLED_FLAG=„N‟ • Inactive/On Hold: FCR.PHASE_CODE=„P‟, FCR.HOLD_FLAG=„Y‟ • Inactive/No manager: FCR.PHASE_CODE=„P‟ and nonexistance of specific rows in FND_CONCURRENT_WORKER_REQESTS view • Pending/Scheduled is actually • STATUS_CODE=‘Q’+(FCR.REQUESTED_START_DATE>SYSDATE)• Why so complicated? • No need to update statuses too often8 © 2011 Tieto Corporation 2010-11-29
    • Pending / Scheduled9 © 2011 Tieto Corporation 2010-11-29
    • Schedule types• As Soon as Possible/Once FCR.REQUESTED_START_DATE• Periodically/On Specific Days FND_CONC_RELEASE_CLASSES (FCRC) • Join with FCR on column RELEASE_CLASS_ID • FCRC.DATE1 - “Start at” field in the form • FCRC.DATE2 - “End at” field in the form • FCRC.CLASS_TYPE – «P» for «Periodically», «S» for «On Specific days» • FCRC.CLASS_INFO – actual schedule data• FCR.REQUESTED_START_DATE always contains the time of next execution, only this field used to determine when the request should be run by the concurrent manager10 © 2011 Tieto Corporation 2010-11-29
    • Periodic schedulesFCRC.CLASS_INFO contents• Values like «X:Y:Z» • X – number of months/weeks/days/hours/minutes the request has to be rescheduled from prior run. • Y – time units: “M” – months, “D” – days, “H” – hours, “N” – minutes • Z – rescheduling type: «S» – from the start of the prior run, «C» – from the completion of the prior run.• Samples • 30:N:S – Repeat every 30 minutes from the start of the prior run • 5:N:C – Repeat every 5 minutes from the completion of the prior run • 12:H:S – Repeat every 12 hours from the start of the prior run11 © 2011 Tieto Corporation 2010-11-29
    • «On Specific Days» schedulesFCRC.CLASS_INFO contents• Values like «XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYZZZZZZZ» • X, Y, Z contains values 0 or 1, representing if the option is selected or not • X – digit in each position represent dates 1 – 31 • Y – Last Day of month • Z – day of week, Su - Sa• Samples • 000000000000000000000000000000000000001 – Days of week: Sa • 111111111000000000000000000000000111110 – Dates: 1 2 3 4 5 6 7 8 9. Days of week: Mo Tu We Th Fr • 000000000000000000000000000000010000000 – Last day of month12 © 2011 Tieto Corporation 2010-11-29
    • Reporting the schedules• Easy to schedule requests, but hard to keep track• Check if there are no obsolete schedules that can be removed or tuned• Might be few repeting schedules of the same concurrent program can be consolidated into one13 © 2011 Tieto Corporation 2010-11-29
    • Pending / Standby14 © 2011 Tieto Corporation 2010-11-29
    • Conflict Resolution Manager (CRM)• CRM resolves conflicts in execution of requests enforced by incompatibility rules• Simplified CRM workflow • Startup – load all incompatibility rules 1. Loaded from FND_CONCURRENT_PROGRAM_SERIAL 2. Stored in memory by FNDCRM (CRM binary executable) • Each itereation 1. If FCQ.CONTROL_CODE=„V‟, reload incompatibility rules (?) 2. Lock STATUS_CODE for all Pending/Normal and Pending/Scheduled requests 3. Check each Pending request against the incompatibility rules and verify there are running concurrent managers that can process them (FND_CONCURRENT_WORKER_REQUESTS) 1. «Release» Pending/Standby requests, which don‟t break any rules 2. «Return» Pending/Normal requests, which break some rules (?) 4. Record statistics in FND_CRM_HISTORY 5. Sleep for «Sleep Seconods»15 © 2011 Tieto Corporation 2010-11-29
    • Interesting DB objects• FND_CONCURRENT_WORKER_REQUESTS • View returning mappings of requests and concurrent managers able to execute them • Definition of the view contains hardcoded IDs according to the Specialization Rules of Concurrent Managers • Rebuilt by «Build Concurrent Request Queue View» request on change of Specilaization rules• FND_CRM_HISTORY • Good information for tuning and troubleshooting • Records statistics for each CRM run, stotistics include: • REQUESTS_EXAMINED • REQUESTS_STANDBY • REQUESTS_RELEASED • REQUESTS_RETURNED • Purged by «Purge Concurrent Request and/or Manager Data», leaving 1 day of history (R12.1.3)• Check FCR.CRM_RELEASE_DATE to find when the request has been released for execution by CRM16 © 2011 Tieto Corporation 2010-11-29
    • Graphing FND_CRM_HISTORY17 © 2011 Tieto Corporation 2010-11-29
    • Why is my request still pending?• Check the status! • Pending/Scheduled – time for execution has not come yet • Pending/Normal –waiting to be picked up by concurrent manager • The request has just been released • All concurrent manager processes are busy executing requests • Long request execution queue • Pending/Standby – Waiting for CRM to be released SELECT WREQID, PHASE, STATUS, WHY FROM FND_CONC_WAITING_REQUESTS WHERE REQID = &REQUEST_ID --582065818 © 2011 Tieto Corporation 2010-11-29
    • Pending / Normal19 © 2011 Tieto Corporation 2010-11-29
    • Workflow of «Concurrent Managers»20 © 2011 Tieto Corporation 2010-11-29
    • Building SQL for querying the requests queueSelect R.Rowid … From Fnd_Concurrent_Requests R And (P.Execution_Method_Code != S OR Where R.Hold_Flag = N (R.PROGRAM_APPLICATION_ID, And R.Status_Code = I R.CONCURRENT_PROGRAM_ID) IN And R.Requested_Start_Date <= Sysdate ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) And (R.Node_Name1 is null or AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN (R.Node_Name1 is not null and ((510, 40032), FND_DCP.target_node_mgr_chk(R.request_id) = 1)) (510, 40033), AND EXISTS (510, 42156), (Select Null (510, 42157), From Fnd_Concurrent_Programs P (530, 43793), Where P.Enabled_Flag = Y (530, 43794), And R.Program_Application_Id = P.Application_Id (535, 42626), And R.Concurrent_Program_Id = P.Concurrent_Program_Id (535, 42627), AND EXISTS (535, 42628)) AND (Select Null ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND From Fnd_Oracle_Userid O R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR Where R.Oracle_Id = O.Oracle_Id (R.REQUEST_CLASS_APPLICATION_ID, AND EXISTS (Select Null R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2))))) From Fnd_Conflicts_Domain C ORDER BY NVL(R.priority, 999999999), R.Priority_Request_ID, Where P.Run_Alone_Flag = C.RunAlone_Flag R.Request_ID And R.CD_Id = C.CD_Id))… Implementation ofConcurrentrulesrulewithRequestprograms Distributed only «Pending/Normal»implementation Execution order Processing requests «Run Alone» flag implementationexecutables (Subroutines) Exclusion specializationfor concurrentConflict type Exclusion «Immediate» type concurrent domains Query Specialization for for21 © 2011 Tieto Corporation 2010-11-29
    • Building SQL for querying the requests queue• The query is built at the startup of the concurrent manager• It hardcodes all specialization rules for the manager• Any changes to specialization rules force restart of the concurrent manager processes (and runs «Build Concurrent Request Queue View» concurrent program too) • Be careful! It does restart automatically! What happens if there are long running requests? • Use «request types», define one for each custom concurrent manager, and one for Standard manager exclusions (DEMO?)22 © 2011 Tieto Corporation 2010-03-26
    • Locking the STATUS_CODESELECT ... … FROM fnd_concurrent_requests R, And R.Requested_Start_Date <= Sysdate fnd_concurrent_programs P, And (R.Enforce_Seriality_Flag = N OR fnd_application A, (C.RunAlone_Flag = P.Run_Alone_Flag And fnd_user U, (P.Run_Alone_Flag = N OR Not Exists fnd_oracle_userid O, (Select Null fnd_conflicts_domain C, From Fnd_Concurrent_Requests Sr fnd_concurrent_queues Q, Where Sr.Status_Code In (R, T) fnd_application A2, And Sr.Enforce_Seriality_Flag = Y fnd_executables E, And Sr.CD_id = C.CD_Id)))) fnd_conc_request_arguments X And Q.Running_Processes <= Q.Max_ProcessesWHERE R.Status_code = I And R.Rowid = :reqname And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or And ((P.Execution_Method_Code != S OR (R.OPS_INSTANCE = (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN decode(:dcp_on, 1, FND_CONC_GLOBAL.OPS_INST_NUM, ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) ANDR.OPS_INSTANCE))) ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN And R.Request_ID = X.Request_ID(+) ((510, 40032), And R.Program_Application_Id = P.Application_Id(+) (510, 40033), And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) (510, 42156), And R.Program_Application_Id = A.Application_Id(+) (510, 42157), And P.Executable_Application_Id = E.Application_Id(+) (530, 43793), And P.Executable_Id = E.Executable_Id(+) (530, 43794), And P.Executable_Application_Id = A2.Application_Id(+) (535, 42626), And R.Requested_By = U.User_Id(+) (535, 42627), And R.Cd_Id = C.Cd_Id(+) (535, 42628)) AND And R.Oracle_Id = O.Oracle_Id(+) ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND And Q.Application_Id = :q_applid R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR And Q.Concurrent_Queue_Id = :queue_id (R.REQUEST_CLASS_APPLICATION_ID, And (P.Enabled_Flag is NULL OR P.Enabled_Flag = Y) R.CONCURRENT_REQUEST_CLASS_ID) NOT IN And R.Hold_Flag = N„ ((0, 2)))))… FOR UPDATE OF R.status_code NoWait23 © 2011 Tieto Corporation 2010-11-29
    • Locking the STATUS_CODE• Query for locking the STATUS_CODE reimplements the same validation criteria to make sure situation has not changed• All processes of a concurrent manager use the same query to fetch the «cache size» number of requests • As more processes of the same manager are run, as higher the competition for requests («ORA-00054: resource busy and acquire with NOWAIT specified», or 0 rows updated by the query if the status has been changed already) • As higher the competition, as faster runs out the list of cached request ids for each manager • As sooner the list of cached queries runs out, as more often FCR is queried• We want to query FCR as saldom as possible • Not hard to get to point where FCR queries are TOP SQLs in DB • Even more important if you have RAC • The key is to minimize the number of concurrent manager processes • Cache size and sleep seconds have some effect24 © 2011 Tieto Corporation 2010-11-29
    • Cache size and Sleep seconds• No silver bullets for tuning • «Sleep seconds» • sleep time of Conflict Resolution Manager affects how soon the request will be passed to execution • are spent only when no requests are in pending/normal status in FCR • should be chosen based on the average time the request is allowed to spend in the queue • Average time 20s and 5 managers? == 100s sleep seconds?• «Cache size» • Large cache sizes make changes of request priorities less effective (do you use different priorities) • Small cache size might be better for Long-running requests • Larger cache sizes might be better for Short-running requests that have few concurrent manager instances. • Large cache size increases the number of failed attempts to lock the status code.25 © 2011 Tieto Corporation 2010-11-29
    • Running / Normal26 © 2011 Tieto Corporation 2010-11-29
    • How to find processes and sessions?select r.request_id req_id, from applsys.fnd_concurrent_requests r, r.phase_code p, gv$session ss, r.status_code s, gv$process pp, (select node_name || : gv$instance gi, from applsys.fnd_concurrent_processes cp applsys.fnd_concurrent_processes cp where concurrent_process_id = r.controlling_manager) || where request_id = &request_id r.os_process_id cp_process, and ss.audsid(+) = r.oracle_session_id gi.INSTANCE_NAME || : || ss.sid || , || ss.serial# and pp.inst_id(+) = ss.inst_idinst_sid_serial#, and pp.addr(+) = ss.paddr gi.HOST_NAME || : || pp.spid db_process, and gi.inst_id(+) = ss.inst_id ss.program, and cp.concurrent_process_id(+) = r.controlling_manager ss.status, ss.sql_id || : || ss.sql_child_number sql_id_chld, ss.event, ss.WAIT_TIME, ss.STATE • For completed requests CP_PROCESS field is still visible27 © 2011 Tieto Corporation 2010-11-29
    • Do we still have time for some bonus material?28 © 2011 Tieto Corporation 2010-11-29
    • Troubleshooting performance of aconcurrent program• Trace it! – but how?• How do we switch on tracing for a concurrent program and what are we looking for to get a complete picture? • waits information • bind variables information • row source operation statistics • easy to switch on • transparent to the end-user • traces only what we want (only the concurrent program) • trace the program execution from the beginning• Lot of options...29 © 2011 Tieto Corporation 2010-11-29
    • Tracing the concurrent program 1• Concurrent -> Program -> Define -> «Enable Trace»• Binds30 © 2011 Tieto Corporation 2010-11-29
    • Tracing the concurrent program 2• Profile Option “Concurrent: Allow Debugging”• transparent to the end-user31 © 2011 Tieto Corporation 2010-11-29
    • Tracing the concurrent program 3• Profile Option “SQL Statement - Custom“• BEGIN FND_CTL.FND_SESS_CTL(,, , TRUE,,ALTER SESSION SET TRACEFILE_IDENTIFIER=||||USERID |||| EVENTS =|||| 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 ||); END;• easy to switch on• traces only what we want (only the concurrent program)32 © 2011 Tieto Corporation 2011-05-19
    • Tracing the concurrent program 4• «Catch» the session (v$session) and enable tracing manually • dbms_monitor.session_trace_enable • dbms_system.set_sql_trace_in_session • ...• trace the program execution from the beginning33 © 2011 Tieto Corporation 2011-05-19
    • Best way I have found so far...• Concurrent manager sets MODULE information (concurrent program short name) for the DB session when the execution of the request starts and unsets MODULE information when it ends.• Need to find the service_name and module from v$sessions beforeselect r.request_id, s.service_name, s.modulefrom fnd_concurrent_requests r, gv$session s where r.status_code = R and s.AUDSID = r.oracle_session_id;begin dbms_monitor.serv_mod_act_trace_enable(service_name => TEST, module_name => e:SYSADMIN:cp:fnd/FNDCPRT_PLSQL, action_name => dbms_monitor.all_actions, waits => true, binds => true, plan_stat => ALL_EXECUTIONS); --<==starting from 11gend;• What‟s enabled? DBA_ENABLED_TRACES• Find the trace file! • find -mmin -20 -exec grep -l 391541 {} ; • select oracle_process_id from fnd_concurrent_requests where request_id=391541;34 © 2011 Tieto Corporation 2011-05-19
    • Where to get more information?• OTN - Oracle E-Business Suite System Administrators Guide Documentation Set• http://etrm.oracle.com – ER Diagrams and information about the DB Objects• http://appsdbalife.wordpress.com – Comment and ask the questions, I will answer!• Twitter @MarisElsins Thank you! ?35 © 2011 Tieto Corporation 2010-11-29
    • Māris Elsiņš© 2011 Tieto Corporation Lead Oracle [Applications] DBA Tieto Latvia, Maris.Elsins@tieto.com