Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
How	to	find	and	fix	your
Java		APEX		ADF		OBIEE		.NET		SQL		PL/SQL
application	performance	
problem
Cary	Millsap
Method	R	Co...
@CaryMillsap 2
Cary	Millsap
@CaryMillsap
Q What	is	the	most	common	
Oracle	performance	
problem	you	see?”
3
“
@CaryMillsap
What	is	the	most	common	
Oracle	performance	
problem	you	see?”
4
“
Assuming	that	other	
people’s	common	probl...
@CaryMillsap 5
Java		APEX		ADF		OBIEE		.NET		SQL		PL/SQL
@CaryMillsap
What	is	a	performance problem?
6
@CaryMillsap 7@CaryMillsap
@CaryMillsap
Performance is not
an attribute of a system.
8
@CaryMillsap 9
ID USERNAME OPERATION R SLR
-- -------- --------- ------ ---
1 FCHANG OE BOOK 12.019 2.0
2 RSMITH OE SHIP 3...
@CaryMillsap 10
ID USERNAME OPERATION R SLR SPEED
-- -------- --------- ------ --- -----
1 FCHANG OE BOOK 12.019 2.0 SLOW
...
@CaryMillsap
Performance is an attribute of
each individual experience
with a system.
11
@CaryMillsap 12
TASK
• id
• name
• ...
EXPERIENCE
• id
• task-id
• user-id
• ip-address
• start-time
• end-time
• ERROR-co...
@CaryMillsap 13
<experience
	 id						=	"b3196c98-906d-4394-bc55-0339518a63b2"
	 task-id	=	"7"
	 uid					=	"238"
	 ip					...
@CaryMillsap
has to finish quickly.”
click
button
link
row
query
report
job
}{“My
14
This	is	what	performance	is.
@CaryMillsap
has to finish quickly.”
click
button
link
row
query
report
job
}{“My
15
A	performance	problem	is	when	it	doesn...
@CaryMillsap 16
“How long does it take?”
Response	time	(R)
Duration	from	service	request	to	
service	fulfillment.
Sanjay Na...
@CaryMillsap 17
Two	big	questions...
1. How	long	did	it	take?
2. Why?
“How long does it take?”
Response	time	(R)
Duration	...
@CaryMillsap
Method	R
18
@CaryMillsap
1. Select	the	experience	you	need	to	improve.
2. Measure	its	response	time	(R)	in	detail.
3. Execute	the	best...
@CaryMillsap
1. Select	the	experience	you	need	to	improve.
2. Measure	its	response	time	(R)	in	detail.
3. Execute	the	best...
@CaryMillsap
1. Select	the	experience	you	need	to	improve.
2. Measure	its	response	time	(R)	in	detail.
3. Execute	the	best...
@CaryMillsap 22
Method	R
OP
TIMIZE ANY
THING
M
e
TH O D R
@CaryMillsap
1. Select	the	experience	you	need	to	improve.
2. Measure	its	response	time	(R)	in	detail.
3. Execute	the	best...
@CaryMillsap
1. Select	the	experience	you	need	to	improve.
2. Measure	its	response	time	(R)	in	detail.
3. Execute	the	best...
@CaryMillsap 25@CaryMillsap
@CaryMillsap
EXADATAD A T A B A S E
ENTERPRISE EDITION
D A T A B A S E
STANDARD EDITION
D A T A B A S E
EXPRESS EDITION
Or...
@CaryMillsap
Measuring	Oracle	response	times
27
@CaryMillsap 28
❶
Activate	
tracing
❷
Get	the	
trace	file
❸
Understand	
its	story
@CaryMillsap
❶
Activate	
tracing
❷
Get	the	
trace	file
❸
Understand	
its	story
29
@CaryMillsap 30
This	is	the	hardest	part.	
...But	only	the	first	time.
After	that,	you	just	lather,	rinse,	repeat.
@CaryMillsap
https://app.com/apex/f?p=150:1:5547991082303::NO:::&P_TRACE=YES
31
Well,	it’s	easy	in	Oracle	APEX.
To	decide	...
@CaryMillsap
Other	technologies	require	a	little	more	work.
First,	the	basics.
32
@CaryMillsap 33
To	decide	at	compile	time	to	trace	all	your	code...
	 dbms_session.session_trace_enable(
	 	 waits	 =>	tru...
@CaryMillsap
if	(should_trace('OE	BOOK',	dbms_random.value(0,1))	{
	 dbms_session.session_trace_enable(
	 	 waits	 =>	true...
@CaryMillsap
sub	should_trace(t,	r)	{
	 select	trace_proportion	from	trace_control	where	task_name	=	:t;
	 return	(r	<=	tr...
@CaryMillsap
Oracle	Database	helps	you	implement
run	time	tracing	decisions...
...without	having	to	make	your	developers	d...
@CaryMillsap
dbms_monitor.serv_mod_act_trace_enable(
	 service_name	 =>	'SYS$USERS',
	 module_name	 =>	'OE	BOOK',
	 action...
@CaryMillsap
How	you	set	your	module	name	varies	by	technology.
SQL			PL/SQL			Java			ADF			.NET			OBIEE
38
@CaryMillsap
--	PL/SQL	example
dbms_application_info.set_module(
	 module_name	 =>	'OE	BOOK',
	 action_name	 =>	sys_guid()...
@CaryMillsap
// JDBC 11g example
String	metrics[]	=	new	String[OraCxn.END_TO_END_STATE_INDEX_MAX];
metrics[END_TO_END_MODU...
@CaryMillsap
// JDBC 12c example
Properties p = new Properties();
p.put("OCSID.MODULE", "OE	BOOK");
p.put("OCSID.ACTION", ...
@CaryMillsap
//	C#	example
conn.ModuleName	=	"OE	BOOK";
conn.ActionName	=	Guid.NewGuid().toString();
//	Your	OE	BOOK	code
...
@CaryMillsap 43
OBIEE
To	set	your	code’s	module	and	action	names...
@CaryMillsap
Here’s	the	goal.
44
@CaryMillsap
User’s	R	
experience
Oracle	
trace	file
45
AppUser Oracle DB
time
You	want	this	to	be	small
You	want	this	to	b...
@CaryMillsap
Another	
experience
An	
experience
Not	the	
trace	file	
you	want
46
AppUser Oracle DB
time
@CaryMillsap
Another	
experience
An	
experience
You	want	one	
trace	file	per	
experience
47
AppUser Oracle DB
time
@CaryMillsap
The	goal:
Trace	exactly	each	user	experience	you	care	about.
...So	that	you	can	see	how	your	code	consumes	ti...
@CaryMillsap 49@CaryMillsap
@CaryMillsap
This	is	what	
you’re	
looking	at	
when	you	use	
systemwide	
aggregations.
50
AppUser Oracle DB
time
@CaryMillsap 51
❶
Activate	
tracing
❷
Get	the	
trace	file
❸
Understand	
its	story
@CaryMillsap 52
This	is	the	boring	part.	
...But	it’s	an	inexpensive	problem	to	solve.
@CaryMillsap
Some	things	to	know...
Your	trace	file	is	on	the	Oracle	Database	server(s),	called:
select tracefile from v$pr...
@CaryMillsap 54
Please, will you help me find my trace file?
@CaryMillsap
There	are	lots	of	ways	to	fetch	the	trace	data.
FTP
Samba
NFS	mount
portable	disk
USB	thumb	drive
Oracle	Data...
@CaryMillsap
Fetching	trace	files	can	be	easy.
You	can	build	tools,	or	you	can	buy	them.
56
Fn’m [ mifp_^ jlif_g.
@CaryMillsap 57
❶
Activate	
tracing
❷
Get	the	
trace	file
❸
Understand	
its	story
@CaryMillsap 58
This	is	the	FUN	part.
@CaryMillsap 59
What’s	in	there?!
@CaryMillsap 60
An	Oracle	trace	file	is	a	log	that	shows
what	your	code	did	inside	the	Oracle	Database.
@CaryMillsap
Some	things	to	know...
Oracle	writes	a	trace	line	when	a	call	(db|os)	finishes.
There	are	two	primary	line	for...
@CaryMillsap 62
method-r.com
For	more	details...
@CaryMillsap
Let’s	look	at	some	
trace	lines...
63
@CaryMillsap 64
begin prepare
CPU
latch-related syscall
CPU
end prepare
begin exec
CPU
write(SQLNET_OUT, result_to_client)...
@CaryMillsap 65
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…
EXEC ...
@CaryMillsap 66
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…
EXEC ...
@CaryMillsap 67
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…
EXEC ...
@CaryMillsap 68
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…
EXEC ...
@CaryMillsap
WAIT #42: nam='latch: library cache'…
PARSE #42:c=10000,…
WAIT #42: nam='SQL*Net message to client'…
EXEC #42...
@CaryMillsap
There	are	lots	of	ways	to	summarize	a	trace	file.
tkprof
SQL	Developer	[Trace]	Viewer
Trace	Analyzer
tvdxstat
...
@CaryMillsap
Profiling	trace	files	can	be	easy.
You	can	build	tools,	or	you	can	buy	them.
71
Fn’m [ mifp_^ jlif_g.
@CaryMillsap
What	you	can	do	with	trace	files
72
@CaryMillsap
Example	1
73
@CaryMillsap 74
mrskew	"r1-fixed.trc"	
CALL-NAME																								DURATION							%		CALLS						MEAN							MIN							...
@CaryMillsap 75
mrskew	--group='($sqlid=~/^#/?"":"[".$sqlid."]")'	--gl=SQLID	--name='message	from	client'	"r1-fixed.trc"
S...
@CaryMillsap 76
mrskew	--rc=p10.rc	--name='SQL*Net	message	from	client'	"r1-fixed.trc"	
								RANGE	{min	≤	e	<	max}					...
@CaryMillsap
App Oracle DB
time
77
~.001	s
~.001	s
~.650	s~.648	s
Each	SQL*Net	message	from	client	call	(~.650	s)	looks	li...
@CaryMillsap 78
mrskew	--name=':dbcall'	--select='$row'	--slabel=ROWS	--precision=0	"r1-fixed.trc"	
CALL-NAME					ROWS				...
@CaryMillsap 79
mrskew	"r1-fixed.trc"	
CALL-NAME																								DURATION							%		CALLS						MEAN							MIN							...
@CaryMillsap
Example	2
80
@CaryMillsap 81
mrskew	"prd1_ora_9031.trc"	
CALL-NAME																								DURATION							%		CALLS						MEAN							MIN		...
@CaryMillsap 82
mrskew	--rc=p10.rc	--name=parse	"prd1_ora_9031.trc"	
								RANGE	{min	≤	e	<	max}				DURATION							%		CA...
@CaryMillsap 83
mrskew	--name=parse	--group='$sqlid'	--gl=SQLID	--sort=4nd	"prd1_ora_9031.trc"	
SQLID												DURATION	...
@CaryMillsap 84
mrskew	--rc=distinct-texts.rc	"prd1_ora_9031.trc"	
				SSQLID		DISTINCT-TEXTS							%		CALLS		MEAN		MIN		M...
@CaryMillsap 85
mrskew	"prd1_ora_9031.trc"	
CALL-NAME																								DURATION							%		CALLS						MEAN							MIN		...
@CaryMillsap
OPTIM
IZE ANYT
HING
M
eTH O D
R
You	might	have	known	that	you	should	
“use	bind	variables,”	but	you	couldn’t	...
@CaryMillsap
BASELINE:
for	each	invoice	number	{
	 cursor	=	parse(“select	...where	invoice_number	=	”	+	number);
	 exec(cu...
@CaryMillsap
BASELINE:	BAD
for	each	invoice	number	{
	 cursor	=	parse(“select	...where	invoice_number	=	”	+	number);
	 exe...
@CaryMillsap
FIX	1	“Hey,	let’s	use	bind	variables”:	STILL	BAD
for	each	invoice	number	{
	 cursor	=	parse(“select	...where	...
@CaryMillsap
FIX	2:	BETTER
cursor	=	parse(“select	...where	invoice_number	=	:a1”);
for	each	invoice	number	{
	 exec(cursor...
@CaryMillsap
And	so	on...
91
@CaryMillsap
Bad	SQL
Bad	PL/SQL
Slow	network
Missing	indexes
Parsing	in	a	loop
Hot	block	problems
Not	enough	memory
Disk	l...
@CaryMillsap
There	are	only	two	possible	root	causes
for	any	response	time	problem:
❶	Call	count	is	too	big.
❷	Latency	is	...
@CaryMillsap 94
mrskew	--top=10	"prd1_ora_9031.trc"	
CALL-NAME																								DURATION							%		CALLS						MEAN			...
@CaryMillsap
With	a	good	trace	file,	you	can	predict	the	
response	time	impact	of	a	proposed	change.*
*This	is	nearly	impos...
@CaryMillsap 96
It	just	takes	practice.
@CaryMillsap
Conclusion
97
@CaryMillsap
Your	code	does	stuff.
Including	some	stuff	inside	Oracle.
The	time	this	stuff	takes	is	your	user’s	response	time...
@CaryMillsap
References
99
@CaryMillsap
Robyn	Sands,	et	al.	2010.
Expert	Oracle	Practices.
Apress
Detailed	information	about	
instrumenting	your	Orac...
@CaryMillsap 101
method-r.com	 www.enkitec.com
method-r.com/facebook	 facebook.com/enkitec
@MethodR			 @Enkitec
cary.mills...
Upcoming SlideShare
Loading in …5
×

How to find and fix your Oracle-based application performance problem

2,830 views

Published on

This session was presented 2016-01-28 at Dallas Oracle User's Group January meeting.
How long does your code take to run? Is it changing? When it is slow, WHY is it slow? Is it your fault, or somebody else's? Can you prove it? How much faster could your code be? Do you know how to measure the performance of your code as user workloads and data volumes increase? These are fundamental questions about performance, but the vast majority of Oracle application developers can't answer them. The most popular performance tools available to them—and to the database administrators that run their code in production—are incapable of answering any of these questions. But the Oracle Database can give you exactly what you need to answer these questions and many more. You can know exactly where YOUR CODE is spending YOUR TIME. This session explains how.

Published in: Data & Analytics
  • Be the first to comment

How to find and fix your Oracle-based application performance problem

  1. 1. How to find and fix your Java APEX ADF OBIEE .NET SQL PL/SQL application performance problem Cary Millsap Method R Corporation and Accenture Enkitec Group @CaryMillsap DOUG January Meeting · Richardson, Texas 5:00p–7:15p Thursday 28 January 2016 © 2006, 2016 Cary Millsap 1 TM MeTHOD R TM
  2. 2. @CaryMillsap 2 Cary Millsap
  3. 3. @CaryMillsap Q What is the most common Oracle performance problem you see?” 3 “
  4. 4. @CaryMillsap What is the most common Oracle performance problem you see?” 4 “ Assuming that other people’s common problems must be your problem. ... Q A
  5. 5. @CaryMillsap 5 Java APEX ADF OBIEE .NET SQL PL/SQL
  6. 6. @CaryMillsap What is a performance problem? 6
  7. 7. @CaryMillsap 7@CaryMillsap
  8. 8. @CaryMillsap Performance is not an attribute of a system. 8
  9. 9. @CaryMillsap 9 ID USERNAME OPERATION R SLR -- -------- --------- ------ --- 1 FCHANG OE BOOK 12.019 2.0 2 RSMITH OE SHIP 3.528 5.0 3 DJOHNSON OE PICK 1.211 5.0 4 FFORBES OE BOOK 0.716 2.0 5 FCHANG OE BOOK 1.917 2.0 6 LBUMONT PA MTCH 1.305 2.0 #define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)
  10. 10. @CaryMillsap 10 ID USERNAME OPERATION R SLR SPEED -- -------- --------- ------ --- ----- 1 FCHANG OE BOOK 12.019 2.0 SLOW 2 RSMITH OE SHIP 3.528 5.0 FAST 3 DJOHNSON OE PICK 1.211 5.0 FAST 4 FFORBES OE BOOK 0.716 2.0 FAST 5 FCHANG OE BOOK 1.917 2.0 FAST 6 LBUMONT PA MTCH 1.305 2.0 FAST #define FASTid ((Rid ≤ SLRid)?”FAST”:”SLOW”)
  11. 11. @CaryMillsap Performance is an attribute of each individual experience with a system. 11
  12. 12. @CaryMillsap 12 TASK • id • name • ... EXPERIENCE • id • task-id • user-id • ip-address • start-time • end-time • ERROR-code • WORK-done SQL • ID • Task-id • ... N 1 N 1
  13. 13. @CaryMillsap 13 <experience id = "b3196c98-906d-4394-bc55-0339518a63b2" task-id = "7" uid = "238" ip = "142.128.130.186" t0 = "2014-04-10T08:32:14.137886" t1 = "2014-04-10T08:32:17.891173" err = "" work = "3" />
  14. 14. @CaryMillsap has to finish quickly.” click button link row query report job }{“My 14 This is what performance is.
  15. 15. @CaryMillsap has to finish quickly.” click button link row query report job }{“My 15 A performance problem is when it doesn’t.
  16. 16. @CaryMillsap 16 “How long does it take?” Response time (R) Duration from service request to service fulfillment. Sanjay Nancy Ken Jorge R t0 t1 R = t1 – t0 Two big questions... 1. How long did it take? 2. Why?
  17. 17. @CaryMillsap 17 Two big questions... 1. How long did it take? 2. Why? “How long does it take?” Response time (R) Duration from service request to service fulfillment. Sanjay Nancy Ken Jorge R t0 t1 R = t1 – t0
  18. 18. @CaryMillsap Method R 18
  19. 19. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-payoff remedy. 4. Repeat until economically optimal. 19 Method R
  20. 20. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-payoff remedy. 4. Repeat until economically optimal. 20 Method R
  21. 21. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-payoff remedy. 4. Repeat until economically optimal. 21 Method R
  22. 22. @CaryMillsap 22 Method R OP TIMIZE ANY THING M e TH O D R
  23. 23. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-payoff remedy. 4. Repeat until economically optimal. 23 Method R
  24. 24. @CaryMillsap 1. Select the experience you need to improve. 2. Measure its response time (R) in detail. 3. Execute the best net-payoff remedy. 4. Repeat until economically optimal. 24 Method R How do you do this, when the it is your code?
  25. 25. @CaryMillsap 25@CaryMillsap
  26. 26. @CaryMillsap EXADATAD A T A B A S E ENTERPRISE EDITION D A T A B A S E STANDARD EDITION D A T A B A S E EXPRESS EDITION Oracle extended SQL tracing is a feature of every Oracle Database. 26 Oracle7 1992 Oracle8 1997 Oracle8i 2000 Oracle9i 2001 Oracle 10g 2004 Oracle 11g 2007 Oracle 12c 2013
  27. 27. @CaryMillsap Measuring Oracle response times 27
  28. 28. @CaryMillsap 28 ❶ Activate tracing ❷ Get the trace file ❸ Understand its story
  29. 29. @CaryMillsap ❶ Activate tracing ❷ Get the trace file ❸ Understand its story 29
  30. 30. @CaryMillsap 30 This is the hardest part. ...But only the first time. After that, you just lather, rinse, repeat.
  31. 31. @CaryMillsap https://app.com/apex/f?p=150:1:5547991082303::NO:::&P_TRACE=YES 31 Well, it’s easy in Oracle APEX. To decide at run time whether to trace your code...
  32. 32. @CaryMillsap Other technologies require a little more work. First, the basics. 32
  33. 33. @CaryMillsap 33 To decide at compile time to trace all your code... dbms_session.session_trace_enable( waits => true, binds => true, plan_stat => 'ALL_EXECUTIONS' ); -- Your OE BOOK code dbms_session.session_trace_disable();
  34. 34. @CaryMillsap if (should_trace('OE BOOK', dbms_random.value(0,1)) { dbms_session.session_trace_enable( waits => true, binds => true, plan_stat => 'ALL_EXECUTIONS' ); } -- Your OE BOOK code dbms_session.session_trace_disable(); 34 To decide at run time whether to trace your code...
  35. 35. @CaryMillsap sub should_trace(t, r) { select trace_proportion from trace_control where task_name = :t; return (r <= trace_proportion); } 35 ...where should_trace looks like this. task_name trace_proportion OE BOOK 0.05 OE PICK 0.02 OE SHIP 1.00 OE INVOICE 0.01 should_trace("OE BOOK", 0.00) → true should_trace("OE BOOK", 0.01) → true should_trace("OE BOOK", 0.02) → true ... should_trace("OE BOOK", 0.05) → true should_trace("OE BOOK", 0.06) → false should_trace("OE BOOK", 0.07) → false should_trace("OE BOOK", 0.08) → false ... should_trace("OE BOOK", 1.00) → false 5% 95% trace_control
  36. 36. @CaryMillsap Oracle Database helps you implement run time tracing decisions... ...without having to make your developers do the if block stuff. 36
  37. 37. @CaryMillsap dbms_monitor.serv_mod_act_trace_enable( service_name => 'SYS$USERS', module_name => 'OE BOOK', action_name => dbms_monitor.all_actions, waits => true, binds => true, plan_stat => 'ALL_EXECUTIONS' ); 37 The DBA does this, at run time. But this works only if your code sets its module name to “OE BOOK”.
  38. 38. @CaryMillsap How you set your module name varies by technology. SQL PL/SQL Java ADF .NET OBIEE 38
  39. 39. @CaryMillsap -- PL/SQL example dbms_application_info.set_module( module_name => 'OE BOOK', action_name => sys_guid() ); -- Your OE BOOK code dbms_application_info.set_module( module_name => null, action_name => null ); 39 SQL PL/SQL To set your code’s module and action names...
  40. 40. @CaryMillsap // JDBC 11g example String metrics[] = new String[OraCxn.END_TO_END_STATE_INDEX_MAX]; metrics[END_TO_END_MODULE_INDEX] = "OE BOOK"; metrics[END_TO_END_ACTION_INDEX] = UUID.randomUUID().toString(); conn.setEndToEndMetrics(metrics, (short) 0); // Your OE BOOK code metrics[END_TO_END_MODULE_INDEX] = ""; metrics[END_TO_END_ACTION_INDEX] = ""; conn.setEndToEndMetrics(metrics, (short) 0); 40 Java ADF To set your code’s module and action names...
  41. 41. @CaryMillsap // JDBC 12c example Properties p = new Properties(); p.put("OCSID.MODULE", "OE BOOK"); p.put("OCSID.ACTION", UUID.randomUUID().toString()); connection.setClientInfo(p); // Your OE BOOK code p.put("OCSID.MODULE", ""); p.put("OCSID.ACTION", ""); connection.setClientInfo(p); 41 Java ADF To set your code’s module and action names...
  42. 42. @CaryMillsap // C# example conn.ModuleName = "OE BOOK"; conn.ActionName = Guid.NewGuid().toString(); // Your OE BOOK code conn.ModuleName = ""; conn.ActionName = ""; 42 ODP.NET To set your code’s module and action names...
  43. 43. @CaryMillsap 43 OBIEE To set your code’s module and action names...
  44. 44. @CaryMillsap Here’s the goal. 44
  45. 45. @CaryMillsap User’s R experience Oracle trace file 45 AppUser Oracle DB time You want this to be small You want this to be small
  46. 46. @CaryMillsap Another experience An experience Not the trace file you want 46 AppUser Oracle DB time
  47. 47. @CaryMillsap Another experience An experience You want one trace file per experience 47 AppUser Oracle DB time
  48. 48. @CaryMillsap The goal: Trace exactly each user experience you care about. ...So that you can see how your code consumes time when it behaves properly, and when it misbehaves. 48
  49. 49. @CaryMillsap 49@CaryMillsap
  50. 50. @CaryMillsap This is what you’re looking at when you use systemwide aggregations. 50 AppUser Oracle DB time
  51. 51. @CaryMillsap 51 ❶ Activate tracing ❷ Get the trace file ❸ Understand its story
  52. 52. @CaryMillsap 52 This is the boring part. ...But it’s an inexpensive problem to solve.
  53. 53. @CaryMillsap Some things to know... Your trace file is on the Oracle Database server(s), called: select tracefile from v$process where addr=( select paddr from v$session where sid=( select sid from v$mystat where rownum=1 ) ) Sessions with DOP = k can create more than 2k + 1 trace files. 53
  54. 54. @CaryMillsap 54 Please, will you help me find my trace file?
  55. 55. @CaryMillsap There are lots of ways to fetch the trace data. FTP Samba NFS mount portable disk USB thumb drive Oracle Database directory objects Method R Trace extension for Oracle SQL Developer 55
  56. 56. @CaryMillsap Fetching trace files can be easy. You can build tools, or you can buy them. 56 Fn’m [ mifp_^ jlif_g.
  57. 57. @CaryMillsap 57 ❶ Activate tracing ❷ Get the trace file ❸ Understand its story
  58. 58. @CaryMillsap 58 This is the FUN part.
  59. 59. @CaryMillsap 59 What’s in there?!
  60. 60. @CaryMillsap 60 An Oracle trace file is a log that shows what your code did inside the Oracle Database.
  61. 61. @CaryMillsap Some things to know... Oracle writes a trace line when a call (db|os) finishes. There are two primary line formats: one for db calls, one for os calls. Each call is associated with a SQL or PL/SQL statement through a cursor id. Each line contains a time stamp (tim) and a duration (e|ela). R ≠ ∑(e|ela) because parent call durations include child call durations. 61
  62. 62. @CaryMillsap 62 method-r.com For more details...
  63. 63. @CaryMillsap Let’s look at some trace lines... 63
  64. 64. @CaryMillsap 64 begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); Oracle kernel code path This is the kind of stuff your code causes the Oracle kernel to do.
  65. 65. @CaryMillsap 65 WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Oracle extended SQL trace data begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); Oracle kernel code path This is the kind of trace data the kernel produces.
  66. 66. @CaryMillsap 66 WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Oracle extended SQL trace data Of course, you don’t directly get to see the kernel code path.
  67. 67. @CaryMillsap 67 WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Oracle extended SQL trace data ...Or that helpful grid that I drew for you.
  68. 68. @CaryMillsap 68 WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… Oracle extended SQL trace data All you get to see is this.
  69. 69. @CaryMillsap WAIT #42: nam='latch: library cache'… PARSE #42:c=10000,… WAIT #42: nam='SQL*Net message to client'… EXEC #42:c=10000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='latch: cache buffers chains'… WAIT #42: nam='SQL*Net message to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… WAIT #42: nam='SQL*Net message to client'… WAIT #42: nam='SQL*Net more data to client'… WAIT #42: nam='SQL*Net more data to client'… FETCH #42:c=20000,… WAIT #42: nam='SQL*Net message from client'… 69 Oracle extended SQL trace dataOracle kernel code path begin prepare CPU latch-related syscall CPU end prepare begin exec CPU write(SQLNET_OUT, result_to_client); end exec read(SQLNET_IN, next_request_from_client); begin fetch CPU latch-related syscall CPU write(SQLNET_OUT, result_to_client); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); begin fetch CPU write(SQLNET_OUT, result_to_client); write(SQLNET_OUT, more_results); write(SQLNET_OUT, more_results); end fetch read(SQLNET_IN, next_request_from_client); You can learn to envision the kernel’s code path that motivated your trace file.
  70. 70. @CaryMillsap There are lots of ways to summarize a trace file. tkprof SQL Developer [Trace] Viewer Trace Analyzer tvdxstat xtrace OraSRP Method R Profiler 70
  71. 71. @CaryMillsap Profiling trace files can be easy. You can build tools, or you can buy them. 71 Fn’m [ mifp_^ jlif_g.
  72. 72. @CaryMillsap What you can do with trace files 72
  73. 73. @CaryMillsap Example 1 73
  74. 74. @CaryMillsap 74 mrskew "r1-fixed.trc" CALL-NAME DURATION % CALLS MEAN MIN MAX --------------------------- ------------ ------ ----- -------- -------- -------- SQL*Net message from client 1,403.927942 99.7% 2,161 0.649666 0.000000 0.927028 FETCH 3.013549 0.2% 2,161 0.001395 0.000000 0.005000 direct path read temp 1.259022 0.1% 83 0.015169 0.003287 0.046968 SQL*Net more data to client 0.141213 0.0% 2,460 0.000057 0.000005 0.001269 SQL*Net message to client 0.007964 0.0% 2,161 0.000004 0.000001 0.000376 --------------------------- ------------ ------ ----- -------- -------- -------- TOTAL (5) 1,408.349690 100.0% 9,026 0.156033 0.000000 0.927028 99.7% of the time is 2,161 network round-trips. What SQL statements cause the round-trips?
  75. 75. @CaryMillsap 75 mrskew --group='($sqlid=~/^#/?"":"[".$sqlid."]")' --gl=SQLID --name='message from client' "r1-fixed.trc" SQLID DURATION % CALLS MEAN MIN MAX --------------- ------------ ------ ----- -------- -------- -------- [7d0bv6ds85q1f] 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 --------------- ------------ ------ ----- -------- -------- -------- TOTAL (1) 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 Just one. All 2,161 round-trips are executed on behalf of just one SQL statement.
  76. 76. @CaryMillsap 76 mrskew --rc=p10.rc --name='SQL*Net message from client' "r1-fixed.trc" RANGE {min ≤ e < max} DURATION % CALLS MEAN MIN MAX ----------------------------- ------------ ------ ----- -------- -------- -------- 1. 0.000000 0.000001 0.000000 0.0% 1 0.000000 0.000000 0.000000 2. 0.000001 0.000010 3. 0.000010 0.000100 4. 0.000100 0.001000 5. 0.001000 0.010000 6. 0.010000 0.100000 7. 0.100000 1.000000 1,403.927942 100.0% 2,160 0.649967 0.547110 0.927028 8. 1.000000 10.000000 9. 10.000000 100.000000 10. 100.000000 1,000.000000 11. 1,000.000000 +∞ ----------------------------- ------------ ------ ----- -------- -------- -------- TOTAL (11) 1,403.927942 100.0% 2,161 0.649666 0.000000 0.927028 Each round-trip consumes an average of .649967 ≈ .650 s. Why?
  77. 77. @CaryMillsap App Oracle DB time 77 ~.001 s ~.001 s ~.650 s~.648 s Each SQL*Net message from client call (~.650 s) looks like this. If round-trip network latency is ~.002 s, then this experience is spending ~.648 s in the Java code executed between database calls.
  78. 78. @CaryMillsap 78 mrskew --name=':dbcall' --select='$row' --slabel=ROWS --precision=0 "r1-fixed.trc" CALL-NAME ROWS % CALLS MEAN MIN MAX --------- ------- ------ ----- ---- --- --- FETCH 216,017 100.0% 2,161 100 17 100 --------- ------- ------ ----- ---- --- --- TOTAL (1) 216,017 100.0% 2,161 100 17 100 One final check... The trace file shows that the application, at least, is fetching an average of 100 rows per fetch call (per round-trip). This helps explain the Java-side latency, but still, .648 s to process just 100 rows needs some explaining.
  79. 79. @CaryMillsap 79 mrskew "r1-fixed.trc" CALL-NAME DURATION % CALLS MEAN MIN MAX --------------------------- ------------ ------ ----- -------- -------- -------- SQL*Net message from client 1,403.927942 99.7% 2,161 0.649666 0.000000 0.927028 FETCH 3.013549 0.2% 2,161 0.001395 0.000000 0.005000 direct path read temp 1.259022 0.1% 83 0.015169 0.003287 0.046968 SQL*Net more data to client 0.141213 0.0% 2,460 0.000057 0.000005 0.001269 SQL*Net message to client 0.007964 0.0% 2,161 0.000004 0.000001 0.000376 --------------------------- ------------ ------ ----- -------- -------- -------- TOTAL (5) 1,408.349690 100.0% 9,026 0.156033 0.000000 0.927028 No matter how long you try to “fix the database” here, you’re going to see at most only a .3% difference in response time. The problem here is in the Java.
  80. 80. @CaryMillsap Example 2 80
  81. 81. @CaryMillsap 81 mrskew "prd1_ora_9031.trc" CALL-NAME DURATION % CALLS MEAN MIN MAX ----------------------------- ---------- ------ ----- -------- -------- -------- PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 ----------------------------- ---------- ------ ----- -------- -------- -------- TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 PARSE calls account for 78.9% of the experience duration. That is never appropriate.
  82. 82. @CaryMillsap 82 mrskew --rc=p10.rc --name=parse "prd1_ora_9031.trc" RANGE {min ≤ e < max} DURATION % CALLS MEAN MIN MAX ----------------------------- ---------- ------ ----- -------- -------- -------- 1. 0.000000 0.000001 0.000000 0.0% 307 0.000000 0.000000 0.000000 2. 0.000001 0.000010 3. 0.000010 0.000100 4. 0.000100 0.001000 0.007992 0.0% 8 0.000999 0.000999 0.000999 5. 0.001000 0.010000 0.033000 0.0% 33 0.001000 0.001000 0.001000 6. 0.010000 0.100000 7. 0.100000 1.000000 8. 1.000000 10.000000 735.385205 100.0% 350 2.101101 1.333797 4.498316 9. 10.000000 100.000000 10. 100.000000 1,000.000000 11. 1,000.000000 +∞ ----------------------------- ---------- ------ ----- -------- -------- -------- TOTAL (11) 735.426197 100.0% 698 1.053619 0.000000 4.498316 That’s a lot of time spent parsing, and these PARSE calls are really expensive.
  83. 83. @CaryMillsap 83 mrskew --name=parse --group='$sqlid' --gl=SQLID --sort=4nd "prd1_ora_9031.trc" SQLID DURATION % CALLS MEAN MIN MAX ------------- ---------- ------ ----- -------- -------- -------- gkbss8w49204k 4.176363 0.6% 349 0.011967 0.000000 4.135371 66kf30526wrgy 3.153521 0.4% 1 3.153521 3.153521 3.153521 3r3dhkb0z824v 2.911558 0.4% 1 2.911558 2.911558 2.911558 3tzra8a2a7pny 1.605757 0.2% 1 1.605757 1.605757 1.605757 2hycpfzdzsu98 3.155520 0.4% 1 3.155520 3.155520 3.155520 6ppu3s1jszy3a 2.208665 0.3% 1 2.208665 2.208665 2.208665 66vkb784j9rcu 1.901711 0.3% 1 1.901711 1.901711 1.901711 5wamvs45j6nh4 1.492773 0.2% 1 1.492773 1.492773 1.492773 dj1buvhxg7h19 1.499772 0.2% 1 1.499772 1.499772 1.499772 41yrts4g94ghn 1.628753 0.2% 1 1.628753 1.628753 1.628753 340 others 711.691804 96.8% 340 2.093211 1.333797 4.498316 ------------- ---------- ------ ----- -------- -------- -------- TOTAL (350) 735.426197 100.0% 698 1.053619 0.000000 4.498316 One statement was parsed 349 times; at least 348 of those are unnecessary.* There are 350 distinct SQL statements executed by this report. ...Which is funny, because you know this report, and you don’t remember there being that many. *Actually all 349 are unnecessary, because the trace file shows that there’s never an EXEC call associated with any of these PARSE calls, but that’s a story for another day.
  84. 84. @CaryMillsap 84 mrskew --rc=distinct-texts.rc "prd1_ora_9031.trc" SSQLID DISTINCT-TEXTS % CALLS MEAN MIN MAX ---------- -------------- ------ ----- ---- --- --- 4151812497 70 20.0% 70 1 1 1 3642320257 70 20.0% 70 1 1 1 2047770123 70 20.0% 70 1 1 1 1928547239 70 20.0% 70 1 1 1 1138917066 69 19.7% 69 1 1 1 3957414185 1 0.3% 349 0 0 1 ---------- -------------- ------ ----- ---- --- --- TOTAL (6) 350 100.0% 698 1 0 1 For the first 5 “shared SQL id” values shown here, there are ~70 distinct statements that could have been sharable. You should be able to reduce the parse call count from 698 to 6, by writing sharable SQL statements, and pulling PARSE calls out of loops.
  85. 85. @CaryMillsap 85 mrskew "prd1_ora_9031.trc" CALL-NAME DURATION % CALLS MEAN MIN MAX ----------------------------- ---------- ------ ----- -------- -------- -------- PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 ----------------------------- ---------- ------ ----- -------- -------- -------- TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 Before your boss will let you “fix” this code, you have to predict the benefit. Reducing the parse count from 698 to 6 should reduce parsing duration from ~735 to ~7, a savings of about 730 s. Response time should improve from ~932 s to ~200 s, just from eliminating the PARSE calls only.
  86. 86. @CaryMillsap OPTIM IZE ANYT HING M eTH O D R You might have known that you should “use bind variables,” but you couldn’t have quantified the R impact on the experience without this trace file. 86
  87. 87. @CaryMillsap BASELINE: for each invoice number { cursor = parse(“select ...where invoice_number = ” + number); exec(cursor); loop over the result set to fetch all the rows; } 87 BAD This is horrific: • Uses too much CPU for PARSE calls • Serialization on library cache and shared pool latches • Consumes too much memory in the library cache • May execute too many network round-trips
  88. 88. @CaryMillsap BASELINE: BAD for each invoice number { cursor = parse(“select ...where invoice_number = ” + number); exec(cursor); loop over the result set to fetch all the rows; } FIX 1 “Hey, let’s use bind variables”: for each invoice number { cursor = parse(“select ...where invoice_number = :a1”); exec(cursor, number); loop over the result set to fetch all the rows; } 88 STILL BAD A little better, but still really awful: • Uses too much CPU for PARSE calls • Serialization on library cache latches • Maybe, too many network round-trips
  89. 89. @CaryMillsap FIX 1 “Hey, let’s use bind variables”: STILL BAD for each invoice number { cursor = parse(“select ...where invoice_number = :a1”); exec(cursor, number); loop over the result set to fetch all the rows; } FIX 2: cursor = parse(“select ...where invoice_number = :a1”); for each invoice number { exec(cursor, number); loop over the result set to fetch all the rows; } 89 BETTER Better (only 1 PARSE call now!), but still lots of network round-trips.
  90. 90. @CaryMillsap FIX 2: BETTER cursor = parse(“select ...where invoice_number = :a1”); for each invoice number { exec(cursor, number); loop over the result set to fetch all the rows; } FIX 3: cursor = parse(“ select ...where invoice_number in (select invoice number from wherever your for each was getting them) ”); exec(cursor); loop over the result set to fetch all the rows; 90 Now, only 1 PARSE call, only 1 EXEC call, and the minimum possible number of network round-trips.* *Unless there’s a way to return fewer rows. BETTER YET
  91. 91. @CaryMillsap And so on... 91
  92. 92. @CaryMillsap Bad SQL Bad PL/SQL Slow network Missing indexes Parsing in a loop Hot block problems Not enough memory Disk latency problems Row locking problems Row-at-a-time processing Bad data structure choice Hardware misconfigurations Too much load on the system OS parameters set inadequately Oracle parameters set inadequately SQL returns more rows than it should Database buffer cache hot/cold problems Oracle query optimizer choosing bad plans Reports run with poorly limiting parameter values Inefficient code between database calls in the application 92 A trace file shows you where your time has gone. Performance problems cannot hide from that.
  93. 93. @CaryMillsap There are only two possible root causes for any response time problem: ❶ Call count is too big. ❷ Latency is too big.* *Probably because someone else’s call counts are too big. 93 #ProTip
  94. 94. @CaryMillsap 94 mrskew --top=10 "prd1_ora_9031.trc" CALL-NAME DURATION % CALLS MEAN MIN MAX ----------------------------- ---------- ------ ----- -------- -------- -------- PARSE 735.426197 78.9% 698 1.053619 0.000000 4.498316 SQL*Net message from client 104.762229 11.2% 1,378 0.076025 0.000391 3.554818 FETCH 91.800028 9.8% 680 0.135000 0.000000 0.506923 db file sequential read 0.104670 0.0% 14 0.007476 0.001067 0.016408 EXEC 0.083988 0.0% 349 0.000241 0.000000 0.002000 gc cr block 2-way 0.073233 0.0% 96 0.000763 0.000280 0.001968 gc current block 2-way 0.031298 0.0% 47 0.000666 0.000361 0.001640 gc current grant busy 0.028037 0.0% 47 0.000597 0.000156 0.001508 SQL*Net more data from client 0.025819 0.0% 837 0.000031 0.000000 0.002564 CLOSE 0.018999 0.0% 698 0.000027 0.000000 0.001000 12 others 0.061576 0.0% 1,633 0.000038 0.000000 0.001687 ----------------------------- ---------- ------ ----- -------- -------- -------- TOTAL (22) 932.416074 100.0% 6,477 0.143958 0.000000 4.498316 See how there are only two ways to reduce a DURATION? You have the CALLS column, and the MEAN column. Profiles like this make it easy to see how anything you do to make something go faster must translate to a manipulation of either CALLS or MEAN.
  95. 95. @CaryMillsap With a good trace file, you can predict the response time impact of a proposed change.* *This is nearly impossible to do with systemwide aggregated statistics. 95 #ProTip
  96. 96. @CaryMillsap 96 It just takes practice.
  97. 97. @CaryMillsap Conclusion 97
  98. 98. @CaryMillsap Your code does stuff. Including some stuff inside Oracle. The time this stuff takes is your user’s response time. You can see exactly what it is. It’s not that hard. 98
  99. 99. @CaryMillsap References 99
  100. 100. @CaryMillsap Robyn Sands, et al. 2010. Expert Oracle Practices. Apress Detailed information about instrumenting your Oracle application code. Cary Millsap. 2011. Mastering Oracle Trace Data. Method R Corporation Textbook for 1-day course that teaches you how to master Oracle trace data. Ron Crisco, et al. 2011. Expert PL/SQL Practices. Apress Detailed information about instrumenting your Oracle application code. Cary Millsap, Jeff Holt. 2003. Optimizing Oracle Performance. O’Reilly Detailed information about Oracle trace data and what to do with it. 100
  101. 101. @CaryMillsap 101 method-r.com www.enkitec.com method-r.com/facebook facebook.com/enkitec @MethodR @Enkitec cary.millsap@method-r.com cary.millsap@accenture.com Q&A

×