Feb14 successful development

successful
application
development...

a little digression on servers

memory access direct from CPU

22

"Simply put, x86 systems are a
lot more powerful than you think"
- EMC presentation

50,000 SQL statements
per second !

read 5GB (bytes) I/O
per second
at 30% cpu load !

33

probably all you ever need

37

you can't blame the server

39

if you can't get that performance ...

"The sooner your company admits that you are not
Google, the sooner you can get down to some real work"
- Ted Dziuba

50

delete from MY_TABLE;

drop table MY_TABLE;
select *
from
EMPLOYEE
where EMPNO > 1234;

cursors
declare
cursor C(p number) is
select * from DEPT
where DEPTNO = p;
begin
for rec in C loop
…
end loop;
end;

begin
MY_PROCEDURE(1,2,3);
end;

open
gimme some memory

process
close

64

open
gimme some memory

process
do some stuff using that memory,
probably access other memory

close

65

open
gimme some memory

process

close

here's your memory back

66

memory access = controlled access

serialised access

aka, someone waits
67

74

1) get latch

protected by
SGA

2) access memory

protected by
SGA

3) release latch

protected by
SGA

80

3) release latch

protected by
SGA

81

someone must wait ...

protected by
SGA

can I have the latch ?
can I have the latch

spinning




86

2000 times !
(_spin_count)

90

yield

mutex
nowadays...
posting

CAS

Good reading: http://andreynikolaev.wordpress.com/

you ... get ... nothing ... done

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

98

Syntax
Validity
Optimization
SQL>
Rowsourcing select *
2 frmo emp;
Executionfrmo emp
*
ERROR at line 2:
(Fetch)
ORA-00923: FROM keyword

not found where expected

99

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

SQL> select empnoo
2 from
emp;
select empnoo
*
ERROR at line 1:
ORA-00904: invalid column name

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

PLAN
------------------------------------SELECT STATEMENT
TABLE ACCESS BY INDEX ROWID EMP
INDEX RANGE SCAN EMP_PK

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)
EMP_PK

EMP

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

103

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

SQL> select name, gets
2 from v$latch
3 where name like 'library%' or name like 'shared%';
NAME
GETS
------------------------------ ---------shared pool
333937
library cache load lock
0
shared pool simulator
7191
shared pool sim alloc
133
shared server configuration
6
shared server info
1

2 from v$latch
NAME
GETS
------------------------------ ---------shared pool
333937
0
7191
133
6
shared server info
1

SQL> select 0 from dual;

0
---------0

2 from v$latch
NAME
GETS
------------------------------ ---------shared pool
333937
0
7191
133
6
shared server info
1

GETS
---------333991
0
7196
135
6
1

= 54
= 5
= 2

SQL> select 0 from dual;

0
---------0

total = 945 !

previously
executed statements

117

parse statement
already in library cache ?
reuse optimizer info
reuse row source info

select surname, firstname from emp where empno = 123

probability of reuse low ?
select * from dept where dname = ‘SALES’


some queries are “nearly the same”

120

parse this...
select surname, firstname from emp where empno = ?

binding

parse this...

binding

now run it with ? = 7362
now run it with ? = 123

library
cache
select * from dept where deptno = ?
select * from customer where locality = ?
select prno from property where reference = ?
select * from organisation where officer = ?
select offender from crimes where crno = ?

library
cache
select prno from property where reference = ?
select * from organisation where officer = ?
select offender from crimes where crno = ?

Five full parses avoided

SQL> declare
2
c number := dbms_sql.open_cursor;
3 begin
4
for i in 1 .. 10000 loop
5
dbms_sql.parse(c,'
6
select * from dba_objects o, dba_segments s
7
where object_id = '||i||'
8
and o.owner = s.owner
9
and o.object_name = s.segment_name' ,
10
dbms_sql.native);
12
end loop;
12
dbms_sql.close_cursor(c);
13 end;

more complex = more parsing
Elapsed: 00:36:07.71

we aren’t even running the thing !

so binding really matters

for high frequency SQL

SQL>
2
3
4
5
6
7
8
9
10
11
12

declare
c number := dbms_sql.open_cursor;
begin
dbms_sql.parse(c,'select * from dba_objects
where object_id = :x' ,
dbms_sql.native);
dbms_sql.bind_variable(c,':x',i);
end loop;
end;
/

PL/SQL procedure successfully completed.
Elapsed: 00:00:00.57

Syntax
Validity
Optimization
Rowsourcing
Execution
(Fetch)

Hard
Parse

not found
in lib cache

Soft
Parse

found

scott@mydb> select *
2
from
EMP
3
where EMPNO = :b1

mike@mydb> select *
2
from
EMP
3
where EMPNO = :b1

scott@mydb> select *
2
from
EMP
3
where EMPNO = :b1

mike@mydb> select *
2
from
EMP
3
where EMPNO = :b1
SQL> desc mike.emp
Name
Null?
----------------- -------EMPNO
NOT NULL
ENAME
JOB

Type
-----------NUMBER(4)
VARCHAR2(10)
VARCHAR2(9)

parsing =
memory access =
latching

SQL> declare
2
c number;
3 begin
4
c := dbms_sql.open_cursor;
5
6
dbms_sql.parse(c,'select * from dba_objects
7
where object_id = :x' ,
8
dbms_sql.native);
9
dbms_sql.bind_variable(c,':x',i);
10
end loop;
12
12 end;

latching hurts concurrency
NAME
COUNT
------------------------------ ---------library cache
10675
library cache pin
8430
library cache pin allocation
4145
shared pool
4390
137

lib cache

select *
from emp

your SQL is always shared

lib cache

select *
from emp

cursor

your cursor points to it

141

lib cache

select *
from emp

close cursor = lose pointer

close "ignored"

3rd execution

open
gimme some memory

process

close

here's your memory back

remember what cursors we’ve used

don’t allow invalid cursor reuse

create procedure MY_PROC is

begin
select *
into

…

from

dba_objects

where

object_id = my_plsql_variable

...
end;

automatically uses
bind variables
static SQL in stored PLSQL

hold’s cursors open
(even if you close)

procedure MY_PROC is
begin
select *
into
…
from
dept
where …
end;

select * from dept …

begin
select *
into
…
from
dept
where …
end;

cursor c_emp is
select * from emp;
begin
open c_emp;
fetch c_emp
into …;
close c_emp;
end;


begin
select *
into
…
from
dept
where …
end;

cursor c_emp is
select * from emp;
begin
open c_emp;
fetch c_emp
into …;
close c_emp;
end;

select * from emp

the future looks bleak
Statement s = "select * from emp where empno = " + nEmpno

avoid parsing costs
for high frequency SQL

164

break the invalidation chain

178

proc A

proc B

proc C

proc D

pack A

pack B

pack C

pack D

body A

body B

body C

body D

pack A

pack B

pack C

body A

body B

pack D

body C
body D

edition based redefinition

12c
184

client bind non-SQL datatypes

12.1

boolean
records
JPublisher / odp.net

arrays

189

only if you're IT shop is ignorant

multitenant snapshot

(not) free :-)

there are no more excuses

201

free dependency analysis
DBA_DEPENDENCIES

definers / invokers

Whitelists (12c)

secure your data
role enabled procedure (12c)

encapsulation

to read data, you read blocks...

from disk
to memory

to you

209

SQL> select *
2 from
EMP
3 /

Buffer Cache

/d01/ts/my_data1.dat

but memory access is complex

213

SQL> select *
2 from
EMP
3 where …

SQL> select max(EMP)
2 from
DEPT

SQL> delete
2 from
DEPT
3 /

SQL> update EMP
2 set …
3 where …

SQL> select *
2 from
EMP
3 where …

SQL> insert into
2 EMP
3 values (…)

SQL> select *
2 from
CUSTOMER
3 /

"give me a row"

• Get latch
• Search along list of blocks in
memory (or "chain")
• Read block
• Extract row of interest
• Release latch
• Give row back to client

218

OPEN c_emp_list
LOOP
FETCH c_emp_list INTO ...
insert into EMP_COPY
( . . . )
values
( . . . );
END LOOP;

222

•
•
•
•
•
•

Get latch
Walk along chain
Get block
Extract single row
Release latch
Give row back to client

Row #1

OPEN c_emp_list
LOOP
( . . . )
values
( . . . );
END LOOP;

223

•
•
•
•
•
•

OPEN c_emp_list
LOOP
( . . . )
values
( . . . );

Get latch
Walk along chain
Get block
Extract single row
Release latch

Row #1

•
•
•
•
•
•

Get latch (again)
Walk along chain (again)
Get block (again)
Extract single row
Release latch (again)
Give row back to client (again)

Row #2

END LOOP;

224

•
•
•
•
•
•

OPEN c_emp_list
LOOP
( . . . )
values
( . . . );
END LOOP;

Get latch
Walk along chain
Get block
Extract single row
Release latch

Row #1

•
•
•
•
•
•

Get latch (again)
Get block (again)
Extract single row

Row #2

•
•
•
•
•
•

Get latch (again)
Get block (again)
Extract single row

Row #3

etc
etc
etc
225

a better strategy.... “pinning”

"give me a row"

“by the way, I’ll
be wanting
several rows
from this block”

"give me a row"

“by the way, I’ll
be wanting
several rows
from this block”

•
•
•
•
•
•
•
•
•
•
•
•

Get latch
Walk along chain
Get block
Pin the block
Release latch
Give row 1 back to client
Give row 2 back to client
…
Give row n to client
Get latch
Walk along chain
Remove my pin on the block
Release latch

how to convey
“I may want many rows”

OCIAttrSet(…
(ub4)OCI_ATTR_PREFETCH_ROWS,
…)

“relational”

not
“rowlational”

235

SQL>
2
3
4
5
6
7
8
9
10
11
12
13
14
15

declare
n number;
d date;
cursor c is
select empno, hiredate
from
big_emp;
begin
open c;
loop
fetch c into n,d;
exit when c%notfound;
end loop;
close c;
end;
/

1,000,000 rows

Elapsed: 00:01:15.06

SQL>
2
3
4
5
6
7
8
9
10
11
12
13

declare
cursor c is
select empno, hiredate
from
big_emp;
type c_list is table of c%rowtype;
r c_list;
begin
open c;
fetch c
bulk collect into r;
close c;
end;
/

Elapsed: 00:00:10.02

all that data goes somewhere

244

SQL> select * from v$mystats
2 where name like '%pga%'
3 /
NAME
VALUE
------------------------------ ---------session pga memory
101534672
session pga memory max
101534672

SQL> declare
. . .
9 begin
10
open c;
11
loop
12
fetch c
13
bulk collect
14
into r limit 100;
15
exit when c%notfound;
. . .
19 /

Elapsed: 00:00:08.03

NAME
VALUE
------------------------------ ---------session pga memory
225912
session pga memory max
225912

SQL> begin
2
3
insert into t1 values ( i );
4
end loop;
5 end;
6 /
Elapsed: 00:00:11.03

252

SQL>
2
3
4
5
6
7
8
9
10
11
12

declare
type num_list is table of number;
n num_list := num_list();
begin
n.extend(50000);
n(i) := i;
end loop;
forall i in 1 .. 50000
insert into t1 values ( n(i) );
end;
/


Elapsed: 00:00:00.76

remember our auditing trigger ?

SQL>
2
3
4

insert
select
from
where

into T
owner, object_name
all_objects
rownum <= 10000;

10000 rows created.

213

SQL> insert into T
2 select owner, object_name
insert into T
3 from
all_objects
select owner, object_name
4 all_objects rownum <= 10000;
where
from
where

rownum <= 10000

call
cpu
elapsed
10000 count ------- ---------rows created.
------- -----Parse
1
Execute
1
Fetch
0
------- -----total
2

disk
query
current
-------- --------- ---------0
0
0
88
123
10642
0
0
0
-------- --------- ---------88
123
10642

rows
---------0
10000
0
---------10000

cpu
elapsed
disk
query
current
------- ---------- -------- --------- ---------0.00
0.00
0
0
0
0.79
0.97
2
109
10845
0.00
0.00
0
0
0
------- ---------- -------- --------- ---------0.79
0.97
2
109
10845

rows
---------0
10000
0
---------10000

0.01
0.00
3.10
3.05
0.00
0.00
------- ---------3.12
3.06

INSERT INTO T_AUDIT
VALUES (SYSDATE ,:B3 ,:B1 ,:B2 )

call
count
------- -----Parse
1
Execute 10000
Fetch
0
------- -----total
10001

213

create or replace
package T_PKG is
type each_row is record ( action varchar2(1),
owner varchar2(30),
object_name varchar2(30)
);

type row_list is table of each_row
index by pls_integer;
g

row_list;

end;
/

214

257

create or replace
trigger AUDIT_TRG1
before insert or update or
delete on T

215

258

create or replace
trigger AUDIT_TRG1
before insert or update or
delete on T
begin
t_pkg.g.delete;
end;
/

215

259

create or replace
trigger AUDIT_TRG2
after insert or update or delete on T
for each row

216

create or replace
trigger AUDIT_TRG2
for each row
begin
if updating or inserting then
t_pkg.g(t_pkg.g.count+1).owner
t_pkg.g(t_pkg.g.count).object_name
else
t_pkg.g(t_pkg.g.count).owner
t_pkg.g(t_pkg.g.count).object_name
end if;
end;
/

:= :new.owner;
:= :new.object_name;
:= :old.owner;
:= :old.object_name;

216

create or replace
trigger AUDIT_TRG3

217

create or replace
trigger AUDIT_TRG3
declare
v_action varchar2(1) :=
case when updating then 'U'
when deleting then 'D'
else 'I' end;
begin
forall i in 1 .. t_pkg.g.count
insert into T_AUDIT
values (
sysdate,
v_action,
t_pkg.g(i).owner,
t_pkg.g(i).object_name);
t_pkg.g.delete;
end;
/

217

SQL>
2
3
4

insert
select
from
where

into T
owner, object_name
all_objects
rownum <= 10000;

10000 rows created.

218

SQL>
2
insert
3
select
4
from
where

insert into T
select owner, object_name
into T
from
all_objects
owner, object_name
where
all_objects rownum <= 10000;
rownum <= 10000

call
count
cpu
elapsed
10000 ------ ------- --------rows created.
------Parse
1
Execute
1
Fetch
0
------- -----total
2

disk
query
current
-------- ---------- ---------0
33
0
0
91
10653
0
0
0
-------- ---------- ---------0
124
10653

rows
---------0
10000
0
---------10000

cpu
elapsed
disk
query
current
------- --------- -------- ---------- ---------0.00
0.00
0
0
0
0.04
0.03
0
90
478
0.00
0.00
0
0
0
------- --------- -------- ---------- ---------0.04
0.03
0
90
478

rows
---------0
10000
0
---------10000
218

0.00
0.00
0.56
0.58
0.00
0.00
------- --------0.56
0.59

INSERT INTO T_AUDIT
VALUES
( SYSDATE, :B1 , :B2 , :B3 )

call
count
------- -----Parse
1
Execute
1
Fetch
0
------- -----total
2

use bulk collect / bulk bind

268

a little known fetch 'feature'

SQL> set autotrace traceonly stat
SQL> set arraysize 100
SQL> select object_type
2 from
T;
89405 rows selected.
Statistics
--------------------------------------------4 recursive calls
0 db block gets
2452 consistent gets
0 physical reads
0 redo size
1198478 bytes sent via SQL*Net to client

271

SQL> select object_type
2 from
T
3 order by object_type;
89405 rows selected.
Statistics
-------------------------------------------14 recursive calls
0 db block gets
1507 consistent gets
0 physical reads
0 redo size
618820 bytes sent via SQL*Net to client

“Readers don’t block writers"
“Writers don't block readers"

Maximum concurrency…
… very few locking worries

“I’m a developer…
…who cares about all that?”

EMP

DWEMP
“every hour,
populate data warehouse,
copy of OLTP data”

demo1a/b

DIY data integrity

... true story
287

alter table add primary key
Manufacturer

Product

alter table add foreign key

Manufacturer

“ensure that every
product corresponds to
a valid manufacturer”
Product

create or replace
trigger PROD_MAN_CHECK
before insert or update on PRODUCT
for each row
declare
l_man_id number;
begin
select man_id
into
l_man_id
from
manufacturer
where man_id = :new.man_id;
exception
when no_data_found then
raise_application_error(-20000,
'Manufacturer is not valid');
end;

create or replace
trigger MAN_PROD_CHECK
before delete on MANUFACTURERS
for each row
declare
l_prd_cnt number;
begin
select count(*)
into
l_prd_cnt
from
product
where prd_id = :old.prd_id;
if l_prd_cnt > 0 then
raise_application_error(-20001,
'Manufacturer in use by product records');
end if;
end;

SQL> insert into PRODUCT (...,MAN_ID,...)
2 values (...,100, ...);

1 row created.
SQL> insert into PRODUCT (...,MAN_ID,...)
2 values (...,101,...);

ERROR at line 1:
ORA-20000: Manufacturer is not valid

SQL> delete from MANUFACTURER
2 where man_id = 5768;
ERROR at line 1:
ORA-20001: Manufacturer in use by product records

An unexpected error has
occurred. Manufacturer
record not found

SQL> select count(*)
2 from
3
( select man_id from PRODUCT
4
minus
5
select man_id from MANUFACTURER )
6 /
COUNT(*)
---------86

297

delete MAN=7
Trigger fires
- any PRD with MAN=7
- "No" (not yet)
- OK

commit

Time
create PRD with MAN=7
Trigger fires
- does MAN=7 exist?
- "Yes"
- OK

commit

referential integrity
seems simple…

“look up X before allowing Y”
300

referential integrity
is complex

read consistency,
blocking / locking

“what percentage data integrity
do we have?”
- project manager

302

beware the module spec
"When creating a product...
validate registration date – ensure not in the future,
validate …
validate …
validate manufacturer ID – must exist in Manufacturer table"

concurrency testing vital

304

Oracle consistency model is …

(DIY) inter-table checking is
hard to do right

(DIY) intra-table checking is
hard to do right

310

"commits dont seem so bad"

314

SQL> begin
2
3
insert into T1
4
values (i,i);
5
commit;
6
end loop;
7 end;
8 /
Elapsed: 00:00:05.40

315

another PLSQL trick

no sync
316

SQL> begin
2
3
insert into T1
4
values (i,i);
5
commit write immediate wait;
6
end loop;
7 end;
8 /
Elapsed: 00:00:24.73

317

SQL> begin
2
3
insert into T1
4
values (i,i);
6
7
8

end loop;
end;
/

Elapsed: 00:00:01.58

319

SQL> create table T ( processed date );
Table created.
SQL>
2
3
4
5
6
7
8
9

insert into T
select
case when mod(rownum,20) != 0 then
to_date('01-JAN-11')+rownum/10
else
null
end
from dual
connect by level <= 10000;

10000 rows created.
321

SQL> select min(processed), max(processed), count(*)
2 from t;
MIN(PROCE MAX(PROCE
COUNT(*)
--------- --------- ---------01-JAN-11 26-SEP-13
10000

10 rows per day
across ~3 years

322

SQL> select *
2 from
T
3 where processed between
4
'01-JUN-13' and '01-JUL-13';

-------------------------------------------| Id | Operation
| Name | Rows |
-------------------------------------------|
0 | SELECT STATEMENT
|
|
287 |
|* 1 | FILTER
|
|
|
|* 2 |
TABLE ACCESS FULL| T
|
287 |
--------------------------------------------

323

SQL> select *
2 from
T
4
'01-JUN-13' and '01-JUL-13';

-------------------------------------------| Id | Operation
| Name | Rows |
-------------------------------------------|
|
|
287 |
|* 1 | FILTER
|
|
|
|* 2 |
|
287 |
--------------------------------------------

reality = 285
324

SQL> truncate table T;
Table truncated.
SQL>
2
3
4
5
6
7
8
9

insert into T
select
case when mod(rownum,20) != 0 then
to_date('01-JAN-11')+rownum/10
else
to_date('31-DEC-3000')
end
from dual
connect by level <= 10000;

10000 rows created.
326

SQL> select *
2 from
T
4
'01-JUN-13' and '01-JUL-13';

------------------------------------------| Id | Operation
| Name | Rows |
------------------------------------------|
|
|
3 |
|* 1 | FILTER
|
|
|
|* 2 |
|
3 |
-------------------------------------------

uh oh...

SQL> begin
2
dbms_stats.gather_table_stats('','T',
3
method_opt=>'for all columns size auto');
4 end;
5 /
SQL>
2
3
4

select
from
where
and

HISTOGRAM
user_tab_col_statistics
table_name = 'T'
column_name = 'PROCESSED';

HISTOGRAM
--------------HYBRID

SQL> select *
2 from
T
4
'01-JUN-13' and '01-JUL-13';

------------------------------------------| Id | Operation
| Name | Rows |
------------------------------------------|
|
|
291 |
|* 1 | FILTER
|
|
|
|* 2 |
|
291 |
-------------------------------------------

330

lots of stuff we didn’t get to
cover

331

minimise parsing
intelligent commit

as long as you are too
set processing
plsql
335

ORA-00041
“active time limit exceeded - session terminated”

connormcdonald.wordpress.com

319

Feb14 successful development

More Related Content

What's hot

Viewers also liked

Similar to Feb14 successful development

More from Connor McDonald

Recently uploaded

Feb14 successful development