APEX Connect 2019 - array/bulk processing in PLSQL

Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Connor McDonald

2

3

4

Getting in touch is easy...
connor-mcdonald.com
https://linktr.ee/connor

6https://asktom.oracle.com

https://asktom.oracle.com/officehours

200 hours free access so far
8

before we begin

let's look at real life ...

"We'd never do that .... that's stupid"

but everyone does ... :-(

the most common PLSQL program

SQL> declare
2 cursor c_products is
3 select aisle, item, descr
4 from hardware;
5 l_each c_products%rowtype;
6 begin
7 open c_products;
8 loop
9 fetch c_products
10 into l_each;
11 exit when c_products%notfound;
12 end loop;
13 close c_products;
14 end;
15 /
PL/SQL procedure successfully completed.

our setup for this session

SQL> create table HARDWARE
2 ( aisle NUMBER(5),
3 item NUMBER(12),
4 descr CHAR(50),
5 stocked DATE
6 );
Table created.
SQL> insert /*+ APPEND */ into HARDWARE
2 select trunc(rownum/1000)+1 aisle,
3 rownum item,
4 'Description '||rownum descr,
5 to_date('01-JAN-2011','dd-mon-yyyy')+rownum/86400
6 from
7 ( select 1 from dual connect by level <= 1000),
8 ( select 1 from dual connect by level <= 1000);
1000000 rows created.
SQL> create index HARDWARE_IX on HARDWARE ( aisle, item );
Index created.
1,000,000 items,
across 1000 aisles
quick access to
items in an aisle

no matter what platform

no matter what direction

row by row :-(

"The database is slow"

data is stored in blocks | pages

to read data, you read blocks...

SQL> select *
2 from EMP
3 where ...
/u01/my_data1.dat
memory

memory access = controlled access

SQL> select *
2 from EMP
3 where …
SQL> update EMP
2 set …
3 where …
SQL> select *
2 from EMP
3 where …
SQL> insert into
2 EMP
3 values (…)
SQL> select *
2 from CUSTOMER
3 /
SQL> select max(EMP)
2 from DEPT
SQL> delete
2 from DEPT
3 /

a quick primer on (database) memory

35
limited resource
lots of people want it
concurrent access causes problems
it's a complex system

SGA
protected by
2) access memory

SGA
protected by
3) release latch

SGA
protected by
someone must wait ...

spinning
can I have the latch ?
46

• Get latch
• Search along list of blocks in memory
• Read block
• Extract row of interest
• Release latch
• Give row back to client
"fetch a row"
58
SQL>
...
9 fetch c_products
10 into l_each;
...

• Get latch
• Walk along list
• Get block
• Extract single row
• Release latch
• Give row back to client
• Get latch (again)
• Walk along list (again)
• Get block (again)
• Release latch (again)
• Give row back to client (again)
Row #1
Row #2
• Get latch (again)
• Walk along list (again)
• Get block (again)
• Release latch (again)
• Give row back to client (again)
Row #3
62
loop
fetch c_products
into l_each;
exit when c_products%notfound;
end loop;

a better strategy.... “pinning”

"fetch a row"
• Get latch
• Walk along list
• Get block
• Pin the block
• Release latch
• Give row 1 back to client
• Give row 2 back to client
…
• Give row n to client
• Get latch
• Walk along list
• Remove my pin on the block
• Release latch
“and .. I’ll will need
several rows from
this block”
Connor is
using this
block

how do you say
“I may want many rows” ?

the client is in charge
SQL> set arraysize 50 stmt.setFetchSize(50)
proc prefetch=50 reader.FetchSize = 50

plsql = bulk collect | bulk bind

quick revision
68

implicit cursor via select
69
SQL> declare
2 l_descr hardware.descr%type;
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 1 -- single
8 and item = 1; -- row
9 end;
bulk
SQL> declare
2 type t_descr_list is
3 table of hardware.descr%type;
4 l_descr_list t_descr_list;
5 begin
6 select descr
7 bulk collect
8 into l_descr_list
9 from hardware
10 where aisle = 1 -- multiple
11 and item between 1 and 100; -- rows
12 end;

explicit cursor
70
SQL> declare
2 cursor c_tool_list is
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500;
7
9 begin
10 open c_tool_list;
11 loop
12 fetch c_tool_list into l_descr;
13 exit when c_tool_list%notfound;
14 end loop;
15 close c_tool_list;
16 end;
bulk
SQL> declare
3 select descr
4 from hardware
5 where aisle = 1
7
9 table of c_tool_list%rowtype;
11
12 begin
14 fetch c_tool_list
15 bulk collect into l_descr_list;
17 end;

implicit cursor
71
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
6 and item between 1 and 500 )
7 loop
8 [processing]
9 end loop;
10 end;
bulk
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
7 loop
8 [processing]
9 end loop;
10 end;
SQL> select value
2 from v$parameter
3 where name = 'plsql_optimize_level';
VALUE
-----
2

"But why bother?"
bulk1.sql

wow !

it's more than just elapsed time

SQL> exec dbms_monitor.session_trace_enable;
[repeat the demo]
SQL> exec dbms_monitor.session_trace_disable;
SQL ID: frmas27snufjy Plan Hash: 652399433
SELECT DESCR
FROM HARDWARE
call count cpu elapsed disk query current rows
------- ------- -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1000001 4.56 3.70 0 1000011 0 1000000
------- ------- -------- ---------- ---------- ---------- ---------- ----------
total 1000003 4.56 3.70 0 1000011 0 1000000
SQL ID: frmas27snufjy Plan Hash: 652399433
SELECT DESCR
FROM HARDWARE
call count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 1 0.00 0.34 0 10100 0 1000000
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 0.00 0.34 0 10100 0 1000000

but ...

all that data goes somewhere
78
15 bulk collect into l_descr_list; -- 1,000,000 rows into an array

SQL> select * from v$mystats
2 where name like '%pga%'
3 /
NAME VALUE
------------------------------ ----------
session pga memory 201534672
session pga memory max 201534672

200m of memory !
per session

law of diminishing returns

this was about pinning
82

typical block is 8KB
83

NAME VALUE
------------------------------ ----------
session pga memory 2127112
session pga memory max 2155912
SQL> declare
3 select descr
4 from hardware;
5
6 type t_descr_list is table of c_tool_list%rowtype;
8
9 begin
11 loop
13 bulk collect
14 into l_descr_list limit 500;
16 end loop;
18 end;
19 /
Elapsed: 00:00:00.10

important !
85

SQL> declare
3 select descr
4 from hardware;
5
6 type t_descr_list is table of c_tool_list%rowtype;
8
9 begin
11 loop
13 bulk collect
14 into l_descr_list limit 500;
1) array is replaced
16 end loop;
18 end;
19 /
2) < 500 = "not found"
[processing here!]
for i in 1 .. l_descr_list.count

even easier ...
87

implicit cursor
88
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
7 loop
8 [processing]
9 end loop;
10 end;
bulk
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
7 loop
8 [processing]
9 end loop;
10 end;

guidelines
89

1) single row
no change

91
SQL> declare
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 1 -- single
8 and item = 1; -- row
9 end;

2) finite and "reasonable"

93
SQL> declare
3 table of hardware.descr%type;
5 begin
6 select descr
7 bulk collect
8 into l_descr_list
9 from hardware
10 where aisle = 1
12 end;

note: subtle differences
94

SQL> declare
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 0
8 and item = 0;
9 end;
10 /
declare
*
ERROR at line 1:
ORA-01403: no data found
ORA-06512: at line 4
SQL> declare
3 begin
4 select descr
5 into l_descr
6 from hardware
7 where aisle = 0
8 and item = 0;
9 dbms_output.put_line('YES!');
10 exception
11 when no_data_found then
12 dbms_output.put_line('NO!');
13 end;
14 /
NO!

SQL> declare
2 type t_descr_list is table of hardware.descr%type;
4 begin
5 select descr
6 bulk collect
7 into l_descr_list
8 from hardware
9 where aisle = 0
10 and item = 0;
11 dbms_output.put_line('YES!');
12 exception
13 when no_data_found then
14 dbms_output.put_line('NO!');
15 end;
16 /
YES!
no exception
if l_descr_list.count = 0 then raise no_data_found; end if;

3) unknown number of rows
97

98
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
7 loop
8 [processing]
9 end loop;
10 end;

special case #1
99

100
SQL> begin
2 for i in (
3 select descr
4 from hardware
5 where aisle = 1
7 loop
8
...
...
exit [early] when [some condition];
...
..
19 end loop;
20 end;

special case #2
101

dont forget this bit

reduce network roundtrips
103
larger array sizes

sidebar: SQL as well
104

a little known feature

SQL Net de-duplicate

SQL> set autotrace traceonly stat
SQL> set arraysize 100
SQL> select aisle, item, descr
2 from hardware
3 where ...;
89405 rows selected.
Statistics
---------------------------------------------
4 recursive calls
0 db block gets
2452 consistent gets
0 physical reads
0 redo size
1198478 bytes sent via SQL*Net to client
107

SQL> select aisle, item, descr
2 from hardware
3 where ...
4 order by 1,2;
89405 rows selected.
Statistics
--------------------------------------------
14 recursive calls
0 db block gets
1507 consistent gets
0 physical reads
0 redo size
618820 bytes sent via SQL*Net to client

focus on blocks
109

focus on roundtrips
110

Part 2
writing data
111

the same rules apply
112

conventional DML in PLSQL
113

114
SQL> declare
2 l_row hardware%rowtype;
3 begin
4 for i in 1 .. 100 loop
5 l_row.aisle := 1;
6 l_row.item := i;
7 insert into hardware values
8 ( l_row.aisle, l_row.item, l_row.descr) ;
9 end loop;
10 end;
11 /

or even nicer (PLSQL only)
115

116
SQL> declare
2 l_row hardware%rowtype;
3 begin
5 l_row.aisle := 1;
6 l_row.item := i;
7 insert into hardware values l_row;
8 end loop;
9 end;
10 /

DML bulk = "bulk bind / FORALL"
117

118
SQL> declare
2 type t_row_list is table of hardware%rowtype;
3 l_row t_row_list := t_row_list();
4 begin
6 l_row.extend;
7 l_row(i).aisle := 1;
8 l_row(i).item := i;
9 end loop;
10
11 forall i in 1 .. 100
12 insert into hardware values l_row(i);
13 end;
14 /

"Is it worth the extra code?"
bulk2.sql

it's better than that
bulk3.sql

... the same rules apply

all the data is somewhere
PGA memory

law of diminishing returns
bulk4.sql

no auto option :-(

100-1000 typically fine

one special case

SQL> insert /*+ APPEND_VALUES */
2 into HARDWARE ( item ) values (1);
1 row created.
SQL> select item from HARDWARE;
select item from HARDWARE
*
ERROR at line 1:
ORA-12838: cannot read/modify an object after modifying it in parallel
SQL> commit;
Commit complete.
SQL> select item from HARDWARE;
ITEM
----------
1

direct load

high speed, massive data loads
bulk7.sql

challenges with multi-row DML

SQL> alter table hardware
2 add constraint
3 hardware_chk check ( item > 0 );
SQL> begin
2 insert into hardware ( item ) values (1);
3 insert into hardware ( item ) values (-1);
7 insert into hardware ( item ) values (-2);
8 end;
9 /
ERROR at line 1:
ORA-02290: check constraint (MCDONAC.HARDWARE_CHK) violated
SQL> select * from hardware;
no rows selected

SQL> declare
2 type t_list is table of hardware.item%type;
3 l_rows t_list := t_list(1,-1,2,3,4,-2);
4 begin
5 forall i in 1 .. l_rows.count
6 insert into hardware ( item ) values (l_rows(i));
7 end;
8 /
declare
*
ERROR at line 1:
ORA-02290: check constraint (MCDONAC.HARDWARE_CHK) violated

issue #1: which row cause the problem

issue #2: none of the rows were stored

SAVE EXCEPTIONS clause
bulk5.sql

also consider error logging
bulk6.sql

issue #3: sparseness

SQL> declare
2 type t_num_list is table of hardware.item%type index by pls_integer;
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 val(11) := 20;
9 val(12) := 20;
10
11 forall i in 1 .. val.count
12 insert into hardware ( item ) values (val(i));
13
14 end;
15 /
declare
*
ERROR at line 1:
ORA-22160: element at index [1] does not exist

SQL> declare
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 val(11) := 20;
9 val(12) := 20;
10
11 forall i in val.first .. val.last
13
14 end;
15 /

but ... contiguous is assumed
141

SQL> declare
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 -- val(11) := 20;
9 val(12) := 20;
10
11 forall i in val.first .. val.last
13
14 end;
15 /
declare
*
ERROR at line 1:
ORA-22160: element at index [11] does not exist

INDICES OF clause
143

SQL> declare
3
4 val t_num_list;
5 begin
6
7 val(10) := 10;
8 -- val(11) := 20;
9 val(12) := 20;
10
11 forall i in indices of val
13
14 end;
15 /

homework :-)
145

VALUES OF clause
146

"How to get array benefits with SQL?"

SQL does this automatically...

full table scans
index range scans
prefetch

use SQL in preference to PLSQL where possible
150
for another day

wrap up
151

“relational”
not
“rowlational”

process data in sets not rows

use bulk collect / bulk bind where appropriate

Enjoy the rest of the conference!
https://linktr.ee/connor

APEX Connect 2019 - array/bulk processing in PLSQL

More Related Content

What's hot

Similar to APEX Connect 2019 - array/bulk processing in PLSQL

More from Connor McDonald

Recently uploaded

APEX Connect 2019 - array/bulk processing in PLSQL