Embedded R Execution using SQL

www.oraly)cs.com
t
:
@brendan)erney
e
:
brendan.)erney@oraly)cs.com

Embedded

Execu)on
in
the
Oracle
Database

using

.

Brendan Tierney

www.oraly)cs.com
t
:
@brendan)erney
e
:

§  Data
Warehousing
since
1997

§  Data
Mining
since
1998

§  Analy)cs
since
1993

11th
May
18:00-‐20:00

Bank
of
Ireland,

Grand
Canal
Dock

www.oraly)cs.com
t
:
@brendan)erney
e
:

www.oraly)cs.com
t
:
@brendan)erney
e
:

Oracle
R
Technologies

R
Distribu)on
Oracle's
supported
redistribu)on
of
open
source
R,
provided
as
a
free

download
from
Oracle,
enhanced
with
dynamic
loading
of
high

performance
linear
algebra
libraries.

Oracle
R
Enterprise
Integra)on
of
R
with
Oracle
Database.
A
component
of
the
Oracle

Advanced
Analy)cs
Op)on.
Oracle
R
Enterprise
makes
the
open
source
R

sta)s)cal
programming
language
and
environment
ready
for
the

enterprise
with
scalability,
performance,
and
ease
of
produc)on

deployment.

Oracle
R
Advanced
Analy)cs
for

Hadoop

High
performance
na)ve
access
to
the
Hadoop
Distributed
File
System

(HDFS)
and
MapReduce
programming
framework
for
R
users.
Oracle
R

Advanced
Analy)cs
for
Hadoop
is
a
component
of
Oracle
Big
Data

Connectors
soXware
suite.

ROracle
An
open
source
R
package,
maintained
by
Oracle
and
enhanced
to
use
the

Oracle
Call
Interface
(OCI)
libraries
to
handle
database
connec)ons
-‐

providing
a
high-‐performance,
na)ve
C-‐language
interface
to
Oracle

Database.

www.oraly)cs.com
t
:
@brendan)erney
e
:

But
!

www.oraly)cs.com
t
:
@brendan)erney
e
:

But
!

Our
data
no
longer
ﬁts
on
our
laptop.

a
Big
Data
issue?

www.oraly)cs.com
t
:
@brendan)erney
e
:

R
-‐
The
Challenges

§  Scalability

§  Regardless
of
the
number
of
cores
on
your
CPU,
R
will
only
use
1
on
a
default

build

§  Performance

§  R
reads
data
into
memory
by
default.
Easy
to
exhaust
RAM
by
storing
unnecessary

data.
Typically
R
will
throw
an
excep)on
at
2GB.

§  Paralleliza)on
can
be
challenge.
Is
not
Default.
Packages
available

§  Produc)on
Deployment

§  Diﬃcul)es
deploying
R
in
produc)on

§  Typically
need
to
re-‐code
in
…..

www.oraly)cs.com
t
:
@brendan)erney
e
:

What
has
Oracle
done?

www.oraly)cs.com
t
:
@brendan)erney
e
:

> AggData <- aggregate(CUSTOMER_V$CUST_ID,
by = list(CUST_GENDER = CUSTOMER_V$CUST_GENDER), 
FUN = length)  
> # Display the results  
> AggData  
CUST_GENDER x  
F F 18325  
M M 37175
select
cust_gender,
count(*)
X

from

customer_v

group
by
cust_gender;

CUST_GENDER X
----------- ----------
F 18325
M 37175
Oracle
R
Enterprise

Transparency
Layer

1
2
3
4
5
6

www.oraly)cs.com
t
:
@brendan)erney
e
:

ore.scriptDrop(CustomerAge3)
ore.scriptCreate(CustomerAge3, function (dat) {
aggdata - aggregate(dat$AFFINITY_CARD,
by = list(Age = dat$AGE),
FUN = length) } )
ageProfile - ore.tableApply(MINING_DATA_BUILD_V,
FUN.NAME=CustomerAge3,
FUN.VALUE=data.frame(Age=1, x=1) )
ageProfile
AGE AGE_NUM
1 17 18
2 18 21
3 19 32
4 20 32
5 21 26
6 22 42
…

www.oraly)cs.com
t
:
@brendan)erney
e
:

SELECT *
FROM table(rqTableEval(cursor(select * from MINING_DATA_BUILD_V),
NULL,
'select 1 AGE, 1 AGE_NUM from dual',
’CustomerAge3') );
AGE AGE_NUM
---------- ----------
17 18
18 21
19 32
20 32
21 26
22 42
23 41
…
ore.scriptDrop(CustomerAge3)
ore.scriptCreate(CustomerAge3, function (dat) {
aggdata - aggregate(dat$AFFINITY_CARD,
by = list(Age = dat$AGE),
FUN = length) } )

www.oraly)cs.com
t
:
@brendan)erney
e
:

Embedded
R
Execu)on
using
SQL

www.oraly)cs.com
t
:
@brendan)erney
e
:

rqEval

-‐
General
Purpose
func)on

rqEval ( PAR_CUR REF CURSOR IN
OUT_QRY VARCHAR2 IN
EXP_NAM VARCHAR2 IN)
Can
be
a
data
set
(contain
one
row)
or
some
parameters

You
need
to
define
the
format
of
the
returned
data,
rows
or
images

This
is
the
actual
R
code
or
Script

BEGIN
--sys.rqScriptDrop('HelloBrendan');
sys.rqScriptCreate('HelloBrendan',
'function() {
res-data.frame(paste(Hello Brendan, the time is, format(Sys.time(),%X)))
res }
');
END;
-- Call the HelloBrendan ORE script
SELECT *
FROM table(rqEval(NULL,
'select cast(''a'' as varchar2(35)) Ans from dual',
'HelloBrendan') );
Ans
-----------------------------------
Hello Brendan the time is 15:19:32
The
last
R
statement

defines
what
is
returned

Defining
the
format
of
the

output
:
Can
be
challenging

at
first

If
image
then
replace

SELECT
with
‘PNG’

www.oraly)cs.com
t
:
@brendan)erney
e
:

rqTableEval

-‐
main
func)on
for
processing
data
set

rqTableEval ( INP_CUR REF CURSOR IN
PAR_CUR REF CURSOR IN
OUT_QRY VARCHAR2 IN
EXP_NAM VARCHAR2 IN)
Cursor
deﬁned
by
a
SELECT
statement

Any
parameters
required
by
the
ORE
Script,
in
addi)on
to
the
data

BEGIN
--sys.rqScriptDrop('AgeProfile1');
sys.rqScriptCreate('AgeProfile1',
'function(dat) {
aggdata - aggregate(dat$AFFINITY_CARD, by = list(Age =
dat$AGE), FUN = length) }
');
END;
SELECT *
NULL,
'select 1 AGE, 1 AGE_NUM from dual',
'AgeProfile1') );
AGE AGE_NUM
---------- ----------
17 18
18 21
19 32
20 32
21 26
22 42
23 41
…
BEGIN
sys.rqScriptDrop('AgeProfile1');
sys.rqScriptCreate('AgeProfile1',
'function(dat) {
aggdata - aggregate(dat$AFFINITY_CARD, by = list(Age =
dat$AGE), FUN = length)
res - plot(aggdata$Age, aggdata$x, type = l) }
');
END;
SELECT *
NULL,
PNG,
'AgeProfile1') );
NAME---------------------------------------------------------------
----------------- ID----------
IMAGE--------------------------------------------------------------
------------------
189504E470D0A1A0A0000000D49484452000001E0000001E008060000007DD4BE95
0000200049444154789CEDDD797494F5BD3FF0F79305421632210B181609C90448D
15A542C138522B46CB5B66CAD5A

www.oraly)cs.com
t
:
@brendan)erney
e
:

rqTableEval

-‐
main
func)on
for
processing
data
set

--Phase 1: Creating the Data Mining model
-- Create a Linear Regression model and store in an ORE data
store
BEGIN
--sys.rqScriptDrop('DEMO_LM');
sys.rqScriptCreate('DEMO_LM',
'function(dat, ds_name) {
mod - lm(alcohol ~., data=dat)
ore.save(mod, name=ds_name, overwrite=TRUE)
} ');
END;
-- Now you need to run the DEMO_LM ORE script to create the
model
select *
from table(rqTableEval(cursor(select * from white_wine),
cursor(select 1 as ore.connect, 'DEMO_LM_DS' as ds_name
from dual),
'XML',
'DEMO_LM') );
-- Phase 2: Applying the Data Mining model
-- Create the script that applies the stored model to new data
-- Return the actual value and the predicted value
BEGIN
--sys.rqScriptDrop('DEMO_LM_APPLY');
sys.rqScriptCreate('DEMO_LM_APPLY’,
'function(dat, ds_name) {
ore.load(ds_name)
pre - predict(mod, newdata=dat,
supplemental.cols=alcohol”)
res - cbind(dat, PRED=pre)
res - res[,c(alcohol, PRED)]
} ');
END;
-- Run the apply script on the new data
select *
from table(rqTableEval(cursor(select * from white_wine),
cursor(select 1 as ore.connect, 'DEMO_LM_DS' as ds_name
from dual),
'select 1 as alcohol, 1 as PRED from dual’,
'DEMO_LM_APPLY') );
alcohol PRED
---------- ----------
10 9.58434071
10.6 8.83113972
10.7 10.378701
10 9.58434071
12.5 12.146045
10.6 8.83113972
12.8 12.668966
…

www.oraly)cs.com
t
:
@brendan)erney
e
:

Embedded
R
Execu)on
using
SQL

These
func)ons

allow
Parallel

Execu)on

www.oraly)cs.com
t
:
@brendan)erney
e
:

Database
Server

Machine

Oracle
Database

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

Tables

In-‐Database

SQL
Func)ons

R
Language
Installa)on

ORE
Installed
Packages

R

Scripts

Oracle
SQL
Client
Interfaces

SQL
Developer

SQLcl

SQL*Plus

etc

ORE
Installed
Packages

ORE
Transparency
Layer

R
Client
Installa)on

R
processes
are
spawned.

One
for
each
Parallel
division

Memory
Management

becomes
an
issue
!

You
don’t
want
to
run

out
of
memory
!

www.oraly)cs.com
t
:
@brendan)erney
e
:

§  Vcells
are
memory
allocated
for
vectors
and
arrays

§  Ncells
are
memory
allocated
for
objects
such
as
lists

sys.rqconfigset('MIN_VSIZE', '10M') -- min heap 10MB, default 32MB
sys.rqconfigset('MAX_VSIZE', '100M') -- max heap 100MB, default 4GB
sys.rqconfigset('MIN_NSIZE', '500K') -- min number cons cells 500x1024, default 1M
sys.rqconfigset('MAX_NSIZE', '2M') -- max number cons cells 2M, default 20M
§  Some
guidelines

–  Set
parallel_degree_policy
to
MANUAL

–  Set
parallel_min_servers
to
the
number
of
parallel
slave
processes
to
be
started
when
the

database
instances
start,

•  This
avoids
start
up
)me
for
the
R
processes.

This
is
not
a
problem
for
long
running

processes.
But
can
save
)me
with
processes
running
for
10s
seconds

–  To
avoid
overloading
the
CPUs
if
the
parallel_max_servers
limit
is
reached,
set
the
hidden

parameter
_parallel_statement_queuing
to
TRUE.

Avoids
overloading
and
lets
processes
wait.

–  set
applica)on
tables
and
their
indexes
to
DOP
1
to
reinforce
the
ability
of
ORE
to
determine

when
to
use
parallelism

Use
garbage
collec)on

informa)on
in
R
using
gc()

to
determine
memory
usage

for
a
process.

Get
before
and
aXer

memory
usage
values.

www.oraly)cs.com
t
:
@brendan)erney
e
:

rqGroupEval

-‐
Doesn’t
really
exist
!!!

§  A
concept
and
Framework
for
deﬁning
it
!

BEGIN
--sys.rqScriptDrop('DEMO_GROUP_EVAL');
sys.rqScriptCreate('DEMO_GROUP_EVAL’,
'function(dat) {
dat$AVG_SUGAR - mean(dat$residual.sugar)
res - dat[,c(alcohol, residual.sugar, AVG_SUGAR)]
} ');
END;
CREATE OR REPLACE PACKAGE WhiteWinePkg AS
TYPE cur IS REF CURSOR RETURN WHITE_WINE%ROWTYPE;
END WhiteWinePkg;
CREATE OR REPLACE FUNCTION My_GroupEval(
inp_cur WhiteWinePkg.cur,
par_cur SYS_REFCURSOR,
out_qry VARCHAR2,
grp_col VARCHAR2,
exp_txt CLOB)
RETURN SYS.AnyDataSet
PIPELINED PARALLEL_ENABLE (PARTITION inp_cur BY HASH (alcohol))
CLUSTER inp_cur BY (alcohol”)
USING rqGroupEvalImpl;
SELECT *
FROM table(MY_GroupEval(cursor(SELECT * FROM WHITE_WINE),
NULL,
'select 1 as alcohol, 1 as residual_sugar, 1 as Avg_Sugar
from dual’,
'alcohol',
'DEMO_GROUP_EVAL'));

www.oraly)cs.com
t
:
@brendan)erney
e
:

From
the
DB
server

rqRowEval

Carefully
consider
the
‘number
of
rows’

selected
as
that
determines
the
degree
of

parallelism.

Be
mindful
of
server
resources
in
calcula)ng

the
degree
and
then
the
‘number
or
rows’

SELECT *
FROM table(rqRowEval(cursor(select * from white_wine),
cursor(select 1 as ore.connect, 'DEMO_LM_DS' as ds_name from dual),
'select 1 as alcohol, 1 as PRED from dual',
500,
'DEMO_LM_APPLY') );
500

records

500

records

500

records

Results

Results

Results

Integrate

Results

Back
into
the
DB
server

www.oraly)cs.com
t
:
@brendan)erney
e
:

Integra9ng
ORE
with
other
products

www.oraly)cs.com
t
:
@brendan)erney
e
:

Integra)ng
with
OBIEE,
BI
Publisher

Any
other
Language
or
Tool

Database
Server

Machine

Oracle
Database

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

-‐-‐-‐-‐-‐

Tables

In-‐Database

SQL
Func)ons

R
Language
Installa)on

ORE
Installed
Packages

R

Scripts

www.oraly)cs.com
t
:
@brendan)erney
e
:

--
-- Create an embedded R script
-- Called using the ORE SQL API
-- - performs an aggregration of the data
-- - creates a Graphic Plot
--
begin
sys.rqScriptDrop('AgeProfile');
sys.rqScriptCreate('AgeProfile',
'function(dat) {
mdbv - dat
aggdata - aggregate(mdbv$AFFINITY_CARD,
by = list(Age = mdbv$AGE),
FUN = length)
res - plot(aggdata$Age, aggdata$x, type = l) } ');
end;
/
--
-- Execute the embedded R Script
-- - Graphic created in PNG format for import into OBIEE
-- - change PNG to XML for BI Publisher
--
select * from table(rqTableEval( cursor(select * from MINING_DATA_BUILD_V),
cursor(select 1 ore.connect from dual),
'PNG','AgeProfile'));
ORE
Demo

ORE
Demo
Data
Chart

www.oraly)cs.com
t
:
@brendan)erney
e
:

§  Check
for
OBIEE
examples
in
Book

www.oraly)cs.com
t
:
@brendan)erney
e
:

R
Text
Mining
using
SQL

ore.scriptCreate(prepare_tm_data_2, function (local_data) {
library(tm)
library(SnowballC)
library(wordcloud)
tm_data -
for(i in 1:nrow(local_data)) {
tm_data - paste(tm_data, local_data[i,]$DOC_TEXT, sep= )
}
txt_corpus - Corpus (VectorSource (tm_data))
# data clean up
tm_map - tm_map (txt_corpus, stripWhitespace) # remove white space
tm_map - tm_map (tm_map, removePunctuation) # remove punctuations
tm_map - tm_map (tm_map, removeNumbers) # to remove numbers
tm_map - tm_map (tm_map, removeWords, stopwords(english)) # to remove stop words
tm_map - tm_map (tm_map, removeWords, c(work, use, java, new, support))
# prepare matrix of words and frequency counts
Matrix - TermDocumentMatrix(tm_map) # terms in rows
matrix_c - as.matrix (Matrix)
freq - sort (rowSums (matrix_c)) # frequency data
res - data.frame(words=names(freq), freq)
wordcloud (res$words, res$freq, max.words=100, min.freq=3, scale=c(7,.5),
random.order=FALSE, colors=brewer.pal(8, Dark2))
} )
select *
from table(rqTableEval( cursor(select * from MY_DOCUMENTS),
null,
'PNG',
'prepare_tm_data_2'));
hrp://www.oraly)cs.com/2016/08/oracle-‐text-‐oracle-‐r-‐enterprise-‐and_17.html

www.oraly)cs.com
t
:
@brendan)erney
e
:

hrp://www.oraly)cs.com/2016/09/how-‐to-‐display-‐blob-‐image-‐in-‐apex-‐report.html

www.oraly)cs.com
t
:
@brendan)erney
e
:

hrp://www.oraly)cs.com/2016/07/crea)ng-‐ggplot2-‐graphics-‐using-‐sql.html

www.oraly)cs.com
t
:
@brendan)erney
e
:

install.packages(googleVis)
There
are
a
few
restric)ons
with
using
these
in
your
applica)on.

Some
charts
open
in
web
browser
and
some
require
ﬂash.

hrp://www.oraly)cs.com/2016/06/googlevis-‐r-‐package-‐for-‐crea)ng-‐google.html

www.oraly)cs.com
t
:
@brendan)erney
e
:


@brendan)erney

www.oraly)cs.com

ie.linkedin.com/in/brendan)erney

www.oraly)cs.com
t
:
@brendan)erney
e
:

Word
Cloud
of
the
Oracle
Advanced

Analy)cs
web-‐pages

hrp://www.oraly)cs.com/2015/01/crea)ng-‐word-‐cloud-‐of-‐oracle-‐oaa.html

Embedded R Execution using SQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Embedded R Execution using SQL

Similar to Embedded R Execution using SQL (20)

Recently uploaded

Recently uploaded (20)

Embedded R Execution using SQL