These are the warmup slides for the Wildcard 13 conference (Riga, Latvia, September 13th.2013 Join the discussion with oracle professionals, get your problems solved and help others! Bring your questions and problems with you to discuss them in a larger group of oracle professionals. We'll discuss anything you have related to Oracle Databases - performance tuning, coding standards and instrumentation, configuration issues, database design, migration strategies, system architectures, upgrade issues, etc. The chances are: The question will be answered or the problem will be solved. You'll have more ideas to explore and try to address the issue. You'll spend fun time helping others by sharing your experience. You'll get a free beer for your courage to join the discussion.

1. Roundtable Discussion with
Oracle Professionals
Ask an Expert and Be an Expert
Jože Senegačnik Māris Elsiņš Andrejs Vorobjovs

2. Jože Senegačnik
• Owner of Dbprof d.o.o.
• First experience with Oracle Version 4.1 in 1988
• 25+ years of experience with Oracle RDBMS.
• Proud member of the OakTable Network www.oaktable.net
• Oracle ACE Director
• VP of Slovenian OUG (SIOUG) board
• CISA – Certified IS auditor
• Blog about Oracle: http://joze-senegacnik.blogspot.com
• Co-author of the OakTable book “Expert Oracle Practices”
by Apress (Jan 2010)
• PPL(A) / IR(SE) – private pilot license, instrument rating
• Blog about flying: http://jsenegacnik.blogspot.com
• Blog about Building Ovens, Baking and Cooking:
http://senegacnik.blogspot.com
© 2013 Pythian2

3. Māris Elsiņš
3 © 2013 Pythian
Google: Elsins Oracle
Twitter, LinkedIn, Blog, Slideshare
Oracle [Apps] DBA – 8 years {PL/SQL Developer 3 years}
Oracle Certified Master – 10g {9i/10g/11g/11i/R12 OCP}
Speaker at Oracle Conferences – 14+ {since 2007}
Pythian – {since 2011}
FAN OF BAAG!

4. © 2013 Pythian4

5. Roundtable Discussion!
Courage = Beer
Questions = Presents
Beer + Presents = Fun
© 2013 Pythian5

6. Warm-up 1: troubleshooting!
• 10.2.0.3
• Suddenly, we started getting lots of “ORA-07445:
exception encountered: core dump
[qercoStart()+156]…”
• No code changes in the DB
• MOS: there are bugs for SQLs with:
– Using ROWNUM < x condition in the where clause
– Using ROWNUM condition and FULL OUTER joins
– Using ROWNUM condition with UNION ALL set operation
© 2013 Pythian6

7. © 2013 Pythian7
Warm-up 1: troubleshooting!
Māris Elsiņš

8. Warm-up 1: troubleshooting!
• There is a “ROWNUM < x”
• There’s no “FULL OUTER”
• There’s no “No UNION ALL”
© 2013 Pythian8

9. Warm-up 1: troubleshooting!
• Try all workarounds listed in MOS bugs:
– Flushed the shared pool, eh?
– bouncing the database, eeeh??
– setting “_complex_view_merging”=false, uhh! Eh…
• None of the bugs is our case. What’s common?
© 2013 Pythian9

10. Warm-up 1: troubleshooting!
© 2013 Pythian10

11. Warm-up 1: troubleshooting!
© 2013 Pythian11

12. Warm-up 1: troubleshooting!
© 2013 Pythian12

13. Warm-up 1: troubleshooting!
© 2013 Pythian13

14. Warm-up 1: troubleshooting!
© 2013 Pythian14

15. Warm-up 1: troubleshooting!
• Oracle introduced native FULL OUTER JOIN
operation in 10.2.0.5. Before that, it was
implemented using the UNION ALL operation.
• “OR” and “IN” + “OR Expansion” transformation
= UNION ALL (CONCATENATION)
© 2013 Pythian15

16. My query – Statistics changed and
“OR Expansion” was applied
© 2013 Pythian16

17. Warm-up 1: troubleshooting!
SQL> ALTER SESSION SET EVENTS '10053 trace name context forever,level 1';
SQL> alter session set tracefile_identifier=CR758708_2;
SQL> alter session set max_dump_file_size=unlimited;
SQL> explain plan for SELECT CBMD.CBMD_BASE_MDL_NUMBER, ....
$ more test_ora_17805_CR758708_2.trc
*************************************
PARAMETERS WITH DEFAULT VALUES
******************************
...
_px_broadcast_fudge_factor = 100
_ordered_nested_loop = true
_no_or_expansion = false
optimizer_index_cost_adj = 100
optimizer_index_caching = 0
© 2013 Pythian17

18. Warm-up 1: troubleshooting!
• Alter system set “_no_or_expansion” = true;
• And I deserve a beer!
© 2013 Pythian18

19. © 2013 Pythian19
Warm-up 2: troubleshooting!
Māris Elsiņš

20. Warm-up 2: troubleshooting!
• eBusiness Suite.
• Oracle Forms.
• frmweb memory footprint is under 200M.
• Sudden spikes to 1.2G
• 7 processes = Unusable system 
© 2013 Pythian20

21. Warm-up 2: troubleshooting!
• Initial troubleshooting efforts
– Forms FRD trace. Too hard to read and low level
tracing caused issues to end users.
– Gdb / Gcore – memory dumps – reading a 1.2 Gb
memory dumps is “too exciting” (may be I just can’t do
it properly”
© 2013 Pythian21

22. Warm-up 2: troubleshooting!
• Hypothesis and the plan:
- Forms process grows because it receives huge
amount of data from the database. How to confirm?
- Find the query. How to find it?
- Check MOS
- Fix the bug
© 2013 Pythian22

23. Warm-up 2: troubleshooting!
• R12.1.3 has proper instrumentation for oracle
sessions!
• exec
dbms_monitor.CLIENT_ID_TRACE_ENABLE('B
FONROUGE', waits=>false, binds=>true);
© 2013 Pythian23

24. Warm-up 2: troubleshooting!
• Recursive and non-Recursive SQLs:
PARSE #140120325469424:c=0,e=39,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1514468521,tim=1377002345178916
PARSE #140120325223680:c=0,e=45,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=2933345646,tim=1377002345179984
PARSE #140120325374992:c=0,e=40,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1377002345184564
PARSE #140120323759152:c=0,e=35,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=2782854086,tim=1377002345185121
PARSE #140120325363448:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=1057973958,tim=1377002345185491
PARSE #140120325362576:c=0,e=34,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=1,plh=749271464,tim=1377002345185890
PARSE #140120325535560:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=3,og=4,plh=2020579421,tim=1377002345186987
PARSE #140120325534544:c=1000,e=718,p=0,cr=0,cu=2,mis=1,r=0,dep=3,og=4,plh=0,tim=1377002345189805
PARSE #140120325373648:c=0,e=36,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=1895590010,tim=1377002345192930
PARSE
#140120325364640:c=0,e=106,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1388734953,tim=1377002345852910
PARSE
#140120325526432:c=0,e=166,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4065726397,tim=1377002347109204
• dep=0 = non-recursive = called by the form.
© 2013 Pythian24

25. Warm-up 2: troubleshooting!
$ tkprof PROD_ora_14363.trc PROD_ora_14363.trc.tkp sort=fchcnt sys=no
$ grep "Parsing user id" PROD_ora_14363.trc.tkp | more
Parsing user id: 173
Parsing user id: 173 (recursive depth: 1)
Parsing user id: 173 (recursive depth: 2)
Parsing user id: 173 (recursive depth: 1)
Parsing user id: 173
© 2013 Pythian25

26. Warm-up 2: troubleshooting!
$ more PROD_ora_14363.trc.tkp
********************************************************************************
SQL ID: 2f4375b2hn94s Plan Hash: 603069092
SELECT *
FROM
( SELECT l.address1, l.address1||decode(l.address2,null,null,';'||
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 3 0.00 0.00 0 0 0 0
Execute 3 0.01 0.01 0 0 0 0
Fetch 3840 33.82 34.76 0 619685 0 253415
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3846 33.84 34.79 0 619685 0 253415
© 2013 Pythian26

27. Warm-up 2: troubleshooting!
$ egrep "^*** MODULE|2f4375b2hn94s" PROD_ora_14363.trc | grep -B1 "2f4375b2hn94s”
*** MODULE NAME:(e:CSF:frm:CSXSRISR) 2013-08-21 03:46:26.026
PARSING IN CURSOR #140415761994912 len=2289 dep=0 uid=173 oct=3 lid=173
tim=1377071356826929 hv=3305776280 ad='262dad2180' sqlid='2f4375b2hn94s’
• Submitted a SR – still under being worked on.
© 2013 Pythian27

28. © 2013 Pythian28
Warm-up 3: Performance!
Measurement errors
Jože Senegačnik

29. I/O Bottleneck Problem
Response Time Component Time % Elap AvgEla
---------------------------------------- ----------- ------- ---------
CPU service 3934.97s 48.39% 0.000716
un-accounted for time 1363.01s 16.76%
db file sequential read 1122.00s 13.80% 0.032253
gc buffer busy 451.73s 5.56% 0.011746
log buffer space 451.64s 5.55% 0.123974
buffer busy waits 176.79s 2.17% 0.029579
gc cr block 2-way 156.49s 1.92% 0.003287
gc cr grant 2-way 100.20s 1.23% 0.006541
latch: cache buffers chains 98.92s 1.22% 0.005708
gc current grant 2-way 69.68s 0.86% 0.006728
latch: library cache 30.10s 0.37% 0.010030
row cache lock 28.95s 0.36% 0.018727
gc current block 2-way 26.72s 0.33% 0.003828
gc cr block busy 19.35s 0.24% 0.006802
gc current grant busy 15.30s 0.19% 0.004999
latch: row cache objects 14.28s 0.18% 0.006165
gc cr block 3-way 11.73s 0.14% 0.002952
gc current block 3-way 11.34s 0.14% 0.003440
log file sync 10.71s 0.13% 0.315066
enq: SQ - contention 9.14s 0.11% 0.060911

30. Timings for Single Block Reads – 3 node RAC
Single Block Read Times From AWR
0
5
10
15
20
25
30
35
40
45
500:00
1:40
3:20
5:00
6:40
8:20
10:00
11:40
13:20
15:00
16:40
18:20
20:00
21:40
23:20
Snapshot ID - 20 minutes interval
SingleBlockReadTimein
Milliseconds
Inst 1

31. Timings for Single Block Reads – 3 node RAC
Single Block Read Times From AWR
0
5
10
15
20
25
30
35
40
45
500:00
1:40
3:20
5:00
6:40
8:20
10:00
11:40
13:20
15:00
16:40
18:20
20:00
21:40
23:20
Snapshot ID - 20 minutes interval
SingleBlockReadTimein
Milliseconds
Inst 1
Inst 2
Inst3

32. The Facts About The I/O Bottleneck Problem
• Facts:
– 3 node RAC
– Same storage
– Single block read time for Instance 1 was substantially different from
read times for other instances during off hours
– 5 batch jobs during off hours
• The timings for Instance 1 are obviously not correct
• The sandwich syndrome (output from strace)
gettimeofday({1159440978, 931945}, NULL) = 0
pread(14, "6242003752302+254.000160054001050"..., 8192, 455
057408) = 8192
gettimeofday({1159440978, 944159}, NULL) = 0
• Waiting in runque for CPU exaggerates all wait times of the
process.

33. © 2013 Pythian33
Warm-up 4: Performance!
Database Connection Problems
Jože Senegačnik

34. Too many sessions
• 35000 sessions per instance
• database level 70000 (2-node RAC)

37. http://www.youtube.com/watch?v=xNDnVOCdvQ0