Tracing Parallel Execution (UKOUG 2006)

3,810 views

Published on

Some of the quirks of tracing the activities of a task which uses PX

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,810
On SlideShare
0
From Embeds
0
Number of Embeds
37
Actions
Shares
0
Downloads
178
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Tracing Parallel Execution (UKOUG 2006)

  1. 1. Tracing PX Slaves Doug Burns [email_address] http://oracledoug.com
  2. 2. Agenda <ul><li>Introduction </li></ul><ul><li>Test Environment </li></ul><ul><li>Tracing Parallel Execution </li></ul><ul><li>Trace Output </li></ul><ul><li>Consolidation </li></ul><ul><li>Timing Issues </li></ul><ul><li>Conclusion </li></ul>
  3. 3. Introduction <ul><li>Who (or what) am I ? </li></ul><ul><ul><li>Scottish </li></ul></ul><ul><ul><li>Predominantly a DBA </li></ul></ul><ul><ul><li>Training and Consultancy </li></ul></ul><ul><li>Current Assignment </li></ul><ul><ul><li>BSkyB </li></ul></ul><ul><ul><li>Very Cool Projects and Hardware </li></ul></ul><ul><ul><li>Less Cool Release Management </li></ul></ul><ul><li>http://oracledoug.com </li></ul><ul><ul><li>Blog </li></ul></ul>
  4. 4. Introduction <ul><li>Previous Related Papers </li></ul><ul><ul><li>Suck It Dry - Tuning Parallel Execution </li></ul></ul><ul><ul><li>How Many Slaves? – PX and the Magic of 2 </li></ul></ul><ul><ul><li>Available at http://oracledoug.com </li></ul></ul><ul><li>Objectives </li></ul><ul><ul><li>Answer tracing questions that cropped up </li></ul></ul><ul><ul><li>Show the detail </li></ul></ul>
  5. 5. <ul><li>Why Parallel Execution? </li></ul><ul><ul><li>Increasing Volumes of Data </li></ul></ul><ul><ul><li>Increasing User Expectations </li></ul></ul><ul><ul><li>More Powerful Hardware </li></ul></ul><ul><li>Parallel Execution (PX) splits a single large task into multiple smaller tasks which are handled by separate processes running concurrently. </li></ul><ul><ul><li>Full Table Scans </li></ul></ul><ul><ul><li>Sorts </li></ul></ul><ul><ul><li>Index Creation, Direct Path inserts etc … </li></ul></ul>Introduction
  6. 6. Test Environment <ul><li>Example </li></ul><ul><ul><li>Small Table </li></ul></ul><ul><ul><ul><li>Easier to wade through trace files </li></ul></ul></ul><ul><ul><ul><li>Quick execution </li></ul></ul></ul><ul><ul><ul><li>Not representative of real applications </li></ul></ul></ul><ul><ul><ul><li>Force PX using Hints </li></ul></ul></ul><ul><ul><li>TEST_TAB </li></ul></ul><ul><ul><ul><li>2,048,000 rows </li></ul></ul></ul><ul><ul><ul><li>Average row length – 109 bytes </li></ul></ul></ul><ul><ul><ul><li>8Kb Blocks/ 32M Extents / 256 Mb Segment </li></ul></ul></ul><ul><ul><ul><li>No Indexes </li></ul></ul></ul>
  7. 7. Test Environment <ul><li>Oracle 9i Release 2 </li></ul><ul><ul><li>The most common database at work </li></ul></ul><ul><ul><li>Tracing similar to earlier versions </li></ul></ul><ul><li>Oracle 10g Release 2 </li></ul><ul><ul><li>Principles are the same </li></ul></ul><ul><ul><li>Improved tracing functionality </li></ul></ul><ul><li>Single CPU / Single HDD </li></ul><ul><ul><li>Trying to introduce wait events </li></ul></ul>
  8. 8. Test Environment <ul><li>The Query </li></ul><ul><ul><ul><ul><li>SELECT /*+ parallel(tt1, 2) */ </li></ul></ul></ul></ul><ul><ul><ul><ul><li>MOD(tt1.pk_id, 2), COUNT(*) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>FROM test_tab tt1 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>GROUP BY MOD(tt1.pk_id,2) </li></ul></ul></ul></ul><ul><ul><ul><ul><li>ORDER BY MOD(tt1.pk_id,2); </li></ul></ul></ul></ul><ul><ul><li>Degree of Parallelism (DOP) = 2 </li></ul></ul><ul><ul><li>Two Sets of PX Slaves Requested </li></ul></ul><ul><ul><ul><li>One set for Full Table Scan </li></ul></ul></ul><ul><ul><ul><li>One set for Grouping Operation </li></ul></ul></ul>
  9. 9. Tracing Parallel Execution <ul><li>Enable Tracing </li></ul><ul><ul><li>10g </li></ul></ul><ul><ul><li>alter session set tracefile_identifier=‘doug_test'; </li></ul></ul><ul><ul><li>exec dbms_session.set_identifier(‘doug_test'); </li></ul></ul><ul><ul><li>exec dbms_monitor.client_id_trace_enable( </li></ul></ul><ul><ul><li>client_id => ‘doug_test'); </li></ul></ul><ul><ul><li>9i </li></ul></ul><ul><ul><li>alter session set tracefile_identifier=‘doug_test'; </li></ul></ul><ul><ul><li>alter session set events '10046 trace name context forever, level 8'; </li></ul></ul><ul><ul><li>tracefile_identifier not as useful for PX </li></ul></ul><ul><ul><ul><li>Only applies to the Query Co-ordinator (QC) process </li></ul></ul></ul><ul><ul><ul><li>Makes sense as the PX slave doesn’t ‘belong’ to the session </li></ul></ul></ul>
  10. 10. Tracing Parallel Execution <ul><li>Run Query </li></ul><ul><li>Disable Tracing </li></ul><ul><ul><li>10g </li></ul></ul><ul><ul><li>exec dbms_monitor.client_id_trace_disable( </li></ul></ul><ul><ul><li>client_id => ‘doug_test'); </li></ul></ul><ul><ul><li>9i </li></ul></ul><ul><ul><li>alter session set events '10046 trace name context off'; </li></ul></ul><ul><li>Several Trace files produced </li></ul><ul><ul><li>Remember – this is for the lowest DOP of 2 </li></ul></ul>
  11. 11. Tracing Parallel Execution
  12. 12. Tracing Parallel Execution <ul><li>user_dump_dest </li></ul><ul><ul><ul><li>test1020_ora_17239_ doug_test .trc </li></ul></ul></ul><ul><li>background_dump_dest </li></ul><ul><ul><ul><li>test1020_ p000 _17219.trc </li></ul></ul></ul><ul><ul><ul><li>test1020_ p001 _17221.trc </li></ul></ul></ul><ul><ul><ul><li>test1020_ p002 _17223.trc </li></ul></ul></ul><ul><ul><ul><li>test1020_ p003 _17225.trc </li></ul></ul></ul><ul><ul><li>No mention of tracefile_identifier </li></ul></ul><ul><ul><li>Slave numbers could be different </li></ul></ul>
  13. 13. Tracing Parallel Execution <ul><li>Issues </li></ul><ul><ul><li>Difficult to get a consolidated view of the user action </li></ul></ul><ul><ul><li>PX Slaves are shared background processes </li></ul></ul><ul><ul><ul><li>How to ensure that you’re looking at the correct slaves? </li></ul></ul></ul><ul><ul><ul><li>How to ensure that you don’t include unrelated activity? </li></ul></ul></ul><ul><ul><li>9i - With a little difficulty and rm! </li></ul></ul><ul><ul><li>10g – dbms_monitor and trcsess </li></ul></ul>
  14. 14. Trace Output – 9i <ul><li>Query Co-ordinator (QC) </li></ul><ul><ul><li>Unix process pid: 5790, image: oracle@ISP4400.localdomain (TNS V1-V3) </li></ul></ul><ul><ul><li>*** SESSION ID:(8.283) 2006-09-12 10:54:33.245 </li></ul></ul><ul><ul><li>APPNAME mod='SQL*Plus' mh=3669949024 act='' ah=4029777240 </li></ul></ul>
  15. 15. Trace Output – 9i <ul><li>Followed by tons of recursive calls </li></ul><ul><ul><li>Enabling Tracing, CBO work, etc. </li></ul></ul><ul><ul><li>PARSING IN CURSOR #1 len=130 dep=0 uid=23 oct=3 lid=23 tim=1130912963021298 hv=3017045374 ad='959ffae8' </li></ul></ul><ul><ul><li>SELECT /*+ parallel(tt1, 2) */ </li></ul></ul><ul><ul><li>MOD(tt1.pk_id, 2), COUNT(*) </li></ul></ul><ul><ul><li>FROM test_tab1 tt1 </li></ul></ul><ul><ul><li>GROUP BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>ORDER BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>END OF STMT </li></ul></ul><ul><ul><li>PARSE #1:c=10998,e=12643,p=0,cr=7,cu=3,mis=1,r=0,dep=0,og=4,tim=1130912963021287 </li></ul></ul><ul><ul><li>WAIT #1: nam='enqueue' ela= 2998428 p1=1413677062 p2=65544 p3=0 </li></ul></ul>
  16. 16. Trace Output – 9i (QC Contd) <ul><li>Now to set up the slave sets </li></ul><ul><ul><li>WAIT #1: nam='rdbms ipc reply' ela= 122 p1=5 p2=21474836 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam= 'process startup ' ela= 13 p1=80 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='process startup' ela= 34572 p1=80 p2=1 p3=1 </li></ul></ul><ul><ul><li>WAIT #1: nam=' PX Deq: Join ACK ' ela= 1983 p1=268500992 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Join ACK' ela= 1479 p1=268500993 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='process startup' ela= 19432 p1=80 p2=2 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='process startup' ela= 26803 p1=80 p2=3 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Join ACK' ela= 2930 p1=268500994 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Join ACK' ela= 1557 p1=268500995 p2=1 p3=0 </li></ul></ul>
  17. 17. Trace Output – 9i (QC Contd) <ul><li>Set up queues and wait for slaves to parse statements </li></ul><ul><ul><li>WAIT #1: nam=' PX qref latch ' ela= 4 p1=1 p2=1 p3=-1821541616 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 10385 p1=1 p2=2 p3=-1821541616 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 1182 p1=1 p2=3 p3=-1821541616 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 29903 p1=1 p2=4 p3=-1821541616 </li></ul></ul><ul><ul><li>WAIT #1: nam=' PX Deq: Parse Reply ' ela= 3277 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Parse Reply' ela= 1992474 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Parse Reply' ela= 960495 p1=200 p2=2 p3=0 </li></ul></ul>
  18. 18. Trace Output – 9i (QC Contd) <ul><li>Start waiting for responses and take out PX qref latch when communicating </li></ul><ul><ul><li>WAIT #1: nam=' PX Deq: Execute Reply ' ela= 210184 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execute Reply' ela= 42955 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execute Reply' ela= 1636370 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam=' PX qref latch ' ela= 11128 p1=1 p2=1 p3=-1821543064 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execute Reply' ela= 501343 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch ' ela= 11011 p1=1 p2=1 p3=-1821541616 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execute Reply' ela= 1424051 p1=200 p2=1 p3=0 </li></ul></ul>
  19. 19. Trace Output – 9i (QC Contd) <ul><li>Some wrap-up towards the end </li></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 2 p1=0 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=10 p2=2 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 98523 p1=10 p2=3 p3=0 </li></ul></ul><ul><ul><li>FETCH #1:c=1000,e=98953,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033637953 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 2351 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=200 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 200 p1=200 p2=2 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Signal ACK' ela= 1031 p1=200 p2=1 p3=0 </li></ul></ul>
  20. 20. Trace Output – 9i <ul><li>Slaves 0 and 1 – Sort/Group </li></ul><ul><li> Instance name: TEST92 </li></ul><ul><ul><li>Redo thread mounted by this instance: 1 </li></ul></ul><ul><ul><li>Oracle process number: 9 </li></ul></ul><ul><ul><li>Unix process pid: 5794, image: oracle@ISP4400.localdomain (P000) </li></ul></ul><ul><ul><li>*** SESSION ID:(13.34) 2006-09-12 10:55:19.741 </li></ul></ul><ul><ul><li><<snipped> </li></ul></ul><ul><ul><li>PARSING IN CURSOR #1 len=98 dep=0 uid=23 oct=3 lid=23 tim=1130913007567950 hv=4220356981 ad='959d6c34' </li></ul></ul><ul><ul><li>SELECT /*+ CIV_GB */ A1.C0,COUNT(SYS_OP_CSR(A1.C1,0)) FROM :Q3000 A1 GROUP BY A1.C0 ORDER BY A1.C0 </li></ul></ul><ul><ul><li>END OF STMT </li></ul></ul><ul><ul><li>PARSE #1:c=11998,e=14410,p=0,cr=2,cu=0,mis=1,r=0,dep=0,og=4,tim=1130913007567939 </li></ul></ul>
  21. 21. Trace Output – 9i (Sort Contd) <ul><li>Cycle round waiting for data to come back from other slave set </li></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 3 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1992813 p1=268566527 p2=2 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1992862 p1=268566527 p2=3 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1993857 p1=268566527 p2=4 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1993834 p1=268566527 p2=5 p3=0 </li></ul></ul>
  22. 22. Trace Output – 9i (Sort Contd) <ul><li>EXEC #1:c=0,e=248,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913033435536 </li></ul><ul><ul><li>WAIT #1: nam='PX Deq: Table Q Sample' ela= 98589 p1=10 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Table Q Sample' ela= 2042 p1=200 p2=2 p3=0 </li></ul></ul><ul><ul><li>FETCH #1:c=1000,e=101033,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913033536644 </li></ul></ul><ul><ul><li>WAIT #1: nam= 'PX Deq Credit: send blkd ' ela= 511 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>FETCH #1:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033537700 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1543 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 98633 p1=268566527 p2=1 p3=0 </li></ul></ul>
  23. 23. Trace Output – 9i <ul><li>Slaves 2 and 3 – FTS </li></ul><ul><ul><li>PARSING IN CURSOR #1 len=208 dep=0 uid=23 oct=3 lid=23 tim=1130913007579832 hv=721990200 ad='959d0330' </li></ul></ul><ul><ul><li>SELECT /*+ PIV_GB */ MOD(A1.C0,2) C0,SYS_OP_MSR(COUNT(*)) C1 FROM (SELECT /*+ NO_EXPAND ROWID(A2) */ A2.&quot;PK_ID&quot; C0 FROM &quot;TESTUSER&quot;.&quot;TEST_TAB1&quot; PX_GRANULE(0, BLOCK_RANGE, DYNAMIC) A2) A1 GROUP BY MOD(A1.C0,2) </li></ul></ul><ul><ul><li>END OF STMT </li></ul></ul><ul><ul><li>PARSE #1:c=9999,e=9226,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=4,tim=1130913007579821 </li></ul></ul>
  24. 24. Trace Output – 9i FTS (Contd) <ul><li>Go out and read data (with Direct Path Reads) </li></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1992624 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 994085 p1=268566527 p2=2 p3=0 </li></ul></ul><ul><ul><li>EXEC #1:c=0,e=180,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913010567272 </li></ul></ul><ul><ul><li>WAIT #1: nam= 'direct path read ' ela= 50 p1=5 p2=10 p3=128 </li></ul></ul><ul><ul><li>WAIT #1: nam='direct path read' ela= 14 p1=5 p2=138 p3=128 </li></ul></ul><ul><ul><li>WAIT #1: nam='direct path read' ela= 17 p1=5 p2=266 p3=128 </li></ul></ul><ul><ul><li><snipped> </li></ul></ul><ul><ul><li>WAIT #1: nam='direct path read' ela= 32 p1=5 p2=522 p3=128 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 11253 p1=268566527 p2=1 p3=0 </li></ul></ul>
  25. 25. Trace Output – 9i FTS (Contd) <ul><li>Wrap-up </li></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 10387 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>FETCH #1:c=11410266,e=22810016,p=16263,cr=16486,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913033377363 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 59137 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>FETCH #1:c=0,e=86,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033436946 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 20 p1=268566527 p2=1 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 389 p1=268566527 p2=2 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 101813 p1=268566527 p2=3 p3=0 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 100783 p1=268566527 p2=1 p3=0 </li></ul></ul>
  26. 26. Trace Output – 10g <ul><li>Query Co-ordinator (QC) </li></ul><ul><li>Unix process pid: 17239 , image: oracle@ISP4400.localdomain (TNS V1-V3) </li></ul><ul><li>*** ACTION NAME:() 2006-08-20 10:59:33.605 </li></ul><ul><li>*** MODULE NAME:(SQL*Plus) 2006-08-20 10:59:33.605 </li></ul><ul><li>*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.605 </li></ul><ul><li>*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.605 </li></ul><ul><li>*** SESSION ID:( 1095.5 ) 2006-08-20 10:59:33.605 </li></ul><ul><li>Followed by tons of recursive calls </li></ul><ul><ul><li>Enabling Tracing, CBO work, etc. </li></ul></ul>
  27. 27. Trace Output – 10g (QC Contd) <ul><li>Essentially the same </li></ul><ul><li>PARSING IN CURSOR #4 len=129 dep=0 uid=27 oct=3 lid=27 tim=1128972630780520 hv=3411843401 ad='77e67d54' </li></ul><ul><li>SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*) </li></ul><ul><li>FROM test_tab1 tt1 </li></ul><ul><li>GROUP BY MOD(tt1.pk_id,2) </li></ul><ul><li>ORDER BY MOD(tt1.pk_id,2) </li></ul><ul><li>END OF STMT </li></ul><ul><li>PARSE #4:c=267960,e=270534,p=20,cr=447,cu=3,mis=1,r=0,dep=0,og=1,tim=1128972630780508 </li></ul>
  28. 28. Trace Output – 10g (QC Contd) <ul><li>Note the different timed event format </li></ul><ul><li>WAIT #4: nam='PX Deq: Execute Reply' ela= 31 sleeptime/senderid=200 passes=1 p3=0 obj#=10907 tim=1129017680317433 </li></ul><ul><li>WAIT #4: nam='PX Deq: Execute Reply' ela= 282420 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 tim=1129017680599950 </li></ul><ul><li>WAIT #4: nam='PX qref latch' ela= 57346 function=1 sleeptime=2042549608 qref=0 obj#=10907 tim=1129017680657526 </li></ul><ul><li>WAIT #4: nam='PX Deq: Execute Reply' ela= 916249 sleeptime/senderid=200 passes=2 p3=0 obj#=10907 tim=1129017681574125 </li></ul><ul><li>WAIT #4: nam='PX qref latch' ela= 1594 function=1 sleeptime=2042549608 qref=0 obj#=10907 tim=1129017681575903 </li></ul>
  29. 29. Trace Output – 10g <ul><li>Slaves 0 and 1 – Sort / Group </li></ul><ul><ul><li>Instance name: TEST1020 </li></ul></ul><ul><ul><li>Redo thread mounted by this instance: 1 </li></ul></ul><ul><ul><li>Oracle process number: 12 </li></ul></ul><ul><ul><li>Unix process pid: 17219 , image: oracle@ISP4400.localdomain ( P000 ) </li></ul></ul><ul><ul><li>*** 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951 </li></ul></ul>
  30. 30. Trace Output – 10g (Sort contd) <ul><li>Change to a shared cursor model </li></ul><ul><ul><li>PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 tim=1129017679619468 hv=3411843401 ad='77e66bd4' </li></ul></ul><ul><ul><li>SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*) </li></ul></ul><ul><ul><li>FROM test_tab1 tt1 </li></ul></ul><ul><ul><li>GROUP BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>ORDER BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>END OF STMT </li></ul></ul><ul><ul><li>PARSE #1:c=1000,e=731,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129017679619454 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 52088 function=1 sleeptime=2042551468 qref=0 obj#=-1 tim=1129017679672279 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1954088 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1129017681626630 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 1954772 sleeptime/senderid=268566527 passes=2 p3=0 obj#=-1 tim=1129017683581534 </li></ul></ul>
  31. 31. Trace Output – 10g <ul><li>Slaves 2 and 3 - FTS </li></ul><ul><ul><li>Instance name: TEST1020 </li></ul></ul><ul><ul><li>Redo thread mounted by this instance: 1 </li></ul></ul><ul><ul><li>Oracle process number: 12 </li></ul></ul><ul><ul><li>Unix process pid: 17219 , image: oracle@ISP4400.localdomain ( P000 ) </li></ul></ul><ul><ul><li>*** 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** CLIENT ID:( doug_test ) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938 </li></ul></ul><ul><ul><li>WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951 </li></ul></ul>
  32. 32. Trace Output – 10g (FTS Contd) <ul><ul><li>PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 tim=1129017679653431 hv=3411843401 ad='77e66bd4' </li></ul></ul><ul><ul><li>SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*) </li></ul></ul><ul><ul><li>FROM test_tab1 tt1 </li></ul></ul><ul><ul><li>GROUP BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>ORDER BY MOD(tt1.pk_id,2) </li></ul></ul><ul><ul><li>END OF STMT </li></ul></ul><ul><ul><li>PARSE #1:c=1000,e=649,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,tim=1129017679653417 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 18705 function=1 sleeptime=2042549980 qref=0 obj#=-1 tim=1129017679672803 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX Deq: Execution Msg' ela= 500 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1129017679673500 </li></ul></ul><ul><ul><li>WAIT #1: nam='direct path read' ela= 46 file number=6 first dba=10 block cnt=118 obj#=10907 tim=1129017679688488 </li></ul></ul><ul><ul><li>WAIT #1: nam='PX qref latch' ela= 11838 function=1 sleeptime=2042549980 qref=0 obj#=10907 tim=1129017679700708 </li></ul></ul>
  33. 33. Consolidation <ul><li>trcsess utility (10g) </li></ul><ul><ul><li>Consolidates all trace files associated with a specific session identifier </li></ul></ul><ul><ul><li>Consolidated trace file looks like any standard trace file </li></ul></ul><ul><ul><li>Can be processed using tkprof to produce statement-level summary information </li></ul></ul><ul><ul><li>trcsess output=output_file clientid=session_identifier /oracle/admin/TEST1020/udump/*.trc /oracle/admin/TEST1020/bdump/*.trc </li></ul></ul><ul><ul><li>tkprof trcsess_output_file output_file sort=prsela,fchela,exeela </li></ul></ul><ul><ul><li>Output is in cursor order, not strict time order </li></ul></ul>
  34. 34. Timing Issues <ul><li>Example Timings </li></ul><ul><ul><li>Wall Clock elapsed time 13.56 seconds </li></ul></ul><ul><ul><ul><li>call count cpu elapsed </li></ul></ul></ul><ul><ul><ul><li>------- ------ -------- ---------- </li></ul></ul></ul><ul><ul><ul><li>Parse 5 0.01 0.02 </li></ul></ul></ul><ul><ul><ul><li>Execute 5 12.98 51.57 </li></ul></ul></ul><ul><ul><ul><li>Fetch 2 0.01 12.81 </li></ul></ul></ul><ul><ul><ul><li>------- ------ -------- ---------- </li></ul></ul></ul><ul><ul><ul><li>total 12 13.01 64.41 </li></ul></ul></ul><ul><ul><li>Picking up the wrong trace files? </li></ul></ul><ul><ul><li>trcsess bug? </li></ul></ul>
  35. 35. Timing Issues <ul><li>Wait Time also ‘too high’ </li></ul><ul><li>Event waited on Times Max. Wait Total Waited </li></ul><ul><li>---------------------------------------- Waited ---------- ------------ </li></ul><ul><li>PX qref latch 31 0.09 0.76 </li></ul><ul><li>PX Deq: Execution Msg 49 1.95 27.76 </li></ul><ul><li>direct path read 281 0.00 0.00 </li></ul><ul><li>db file sequential read 2 0.00 0.00 </li></ul><ul><li>PX Deq: Parse Reply 2 0.00 0.00 </li></ul><ul><li>SQL*Net message to client 2 0.00 0.00 </li></ul><ul><li>PX Deq: Execute Reply 39 0.92 12.15 </li></ul><ul><li>PX Deq: Table Q qref 1 0.00 0.00 </li></ul><ul><li>SQL*Net message from client 2 0.00 0.00 </li></ul><ul><li>PX Deq: Signal ACK 6 0.00 0.01 </li></ul><ul><li>enq: PS - contention 1 0.00 0.00 </li></ul><ul><li>PX Deq: Table Q Sample 2 0.00 0.00 </li></ul><ul><li>PX Deq Credit: send blkd 3 0.00 0.00 </li></ul>
  36. 36. Timing Issues <ul><li>This is a consolidated trace file </li></ul><ul><ul><li>Multiple concurrent processes </li></ul></ul><ul><ul><li>Some running on multiple CPUs </li></ul></ul><ul><ul><ul><li>There is more CPU time available than wall clock time </li></ul></ul></ul><ul><ul><li>Some just waiting </li></ul></ul><ul><ul><ul><li>Systems have an infinite capacity for waiting </li></ul></ul></ul><ul><ul><li>Will vary, depending on the actual DOP acquired </li></ul></ul><ul><ul><ul><li>The DOP acquired could vary </li></ul></ul></ul><ul><ul><ul><li>How many slaves are available in the pool? </li></ul></ul></ul><ul><ul><ul><li>parallel_adaptive_multi_user </li></ul></ul></ul>
  37. 37. Timing Issues <ul><li>Exactly the same job, but DOP 16 </li></ul><ul><ul><li>Wall Clock elapsed time = 16.60 seconds </li></ul></ul><ul><ul><ul><li>Slightly higher than 13.56 seconds </li></ul></ul></ul><ul><ul><ul><li>call count cpu elapsed </li></ul></ul></ul><ul><ul><ul><li>------- ------ -------- ---------- </li></ul></ul></ul><ul><ul><ul><li>Parse 33 0.03 0.16 </li></ul></ul></ul><ul><ul><ul><li>Execute 33 13.42 430.24 </li></ul></ul></ul><ul><ul><ul><li>Fetch 2 0.02 12.28 </li></ul></ul></ul><ul><ul><ul><li>------- ------ -------- ---------- </li></ul></ul></ul><ul><ul><ul><li>total 68 13.48 442.69 </li></ul></ul></ul><ul><ul><li>Much higher elapsed time </li></ul></ul><ul><ul><li>Couple of seconds additional wait time </li></ul></ul>
  38. 38. Conclusion <ul><li>Tracing PX is a little more tricky that tracing serial processes </li></ul><ul><ul><li>A bit like trying to trace multiple users, or an entire server? </li></ul></ul><ul><li>10g dbms_monitor package and trcsess utility help </li></ul><ul><ul><li>Consolidated timings need to be interpreted differently </li></ul></ul><ul><li>More papers available at http://oracledoug.com </li></ul><ul><li>Next presentation – How Many Slaves – Hall 7b at 16:00 </li></ul>
  39. 39. Tracing PX Slaves Doug Burns [email_address] http://oracledoug.com

×