SQL: The one language to rule all your data

www.oraly)cs.com
t
:
@brendan)erney
e
:
brendan.)erney@oraly)cs.com

The
one
language
to
rule
all
your
data

Brendan Tierney

www.oraly)cs.com
t
:
@brendan)erney
e
:

www.oraly)cs.com
t
:
@brendan)erney
e
:

§  Data
Warehousing
since
1997

§  Data
Mining
since
1998

§  Analy)cs
since
1993

Store

Access

Analyze

Protect

Analyze

SELECT product, SUM(sale) AS "Total Sales"
FROM order_details
GROUP BY product;

Analyze

SELECT product, SUM(sale) AS "Total Sales"
FROM order_details
GROUP BY product
HAVING SUM(sale) >= 10000;

www.oraly)cs.com
t
:
@brendan)erney
e
:

Let
us
start
with
some
Basics

www.oraly)cs.com
t
:
@brendan)erney
e
:

SUM(x)

AVG(x)

STDDEV(x)

CORR(x,
y)

www.oraly)cs.com
t
:
@brendan)erney
e
:

Crea)ng
a
story
about
our
data.

www.oraly)cs.com
t
:
@brendan)erney
e
:

How
we
do
Analy)cs?

Some)mes
how
we
are
told
how
to
do
Analy)cs

www.oraly)cs.com
t
:
@brendan)erney
e
:

Do
we
really
need
to
use
other
tools
&
languages?

www.oraly)cs.com
t
:
@brendan)erney
e
:

But
!

www.oraly)cs.com
t
:
@brendan)erney
e
:

But
!

Our
data
no
longer
ﬁts
on
our
laptop.

a
Big
Data
issue?

www.oraly)cs.com
t
:
@brendan)erney
e
:

R
-‐
The
Challenges

§  Scalability

§  Regardless
of
the
number
of
cores
on
your
CPU,
R
will
only
use
1
on
a
default

build

§  Performance

§  R
reads
data
into
memory
by
default.
Easy
to
exhaust
RAM
by
storing
unnecessary

data.
Typically
R
will
throw
an
excep)on
at
2GB.

§  Paralleliza)on
can
be
challenge.
Is
not
Default.
Packages
available

§  Produc)on
Deployment

§  Diﬃcul)es
deploying
R
in
produc)on

§  Typically
need
to
re-‐code
in
…..

www.oraly)cs.com
t
:
@brendan)erney
e
:

I’m
ge`ng
too
old
for
this
new
stuﬀ
!

Can
you
teach
an
old
dog
new
tricks?

www.oraly)cs.com
t
:
@brendan)erney
e
:

What
if
you
could
use
the
language
and
skills
you

already
have?

www.oraly)cs.com
t
:
@brendan)erney
e
:

Did you know?

www.oraly)cs.com
t
:
@brendan)erney
e
:

Sta)s)cal
Func)ons
in
Oracle

All
of
these
are

FREE

with
the
Database

These
are
ocen

forgoden
about

www.oraly)cs.com
t
:
@brendan)erney
e
:

DBMS_STAT_FUNC

www.oraly)cs.com
t
:
@brendan)erney
e
:

set serveroutput on
declare  
   s         DBMS_STAT_FUNCS.SummaryType;  
begin  
 
   DBMS_STAT_FUNCS.SUMMARY('DMUSER', 'MINING_DATA_BUILD_V', 'AGE', 3, s);
   dbms_output.put_line('SUMMARY STATISTICS');  
   dbms_output.put_line('Count : '||s.count);  
   dbms_output.put_line('Min    : '||s.min);  
   dbms_output.put_line('Max    : '||s.max);  
   dbms_output.put_line('Range : '||s.range);  
   dbms_output.put_line('Mean   : '||round(s.mean));  
   dbms_output.put_line('Mode Count : '||s.cmode.count);  
   dbms_output.put_line('Mode        : '||s.cmode(1));  
   dbms_output.put_line('Variance    : '||round(s.variance));  
   dbms_output.put_line('Stddev      : '||round(s.stddev));  
   dbms_output.put_line('Quantile 5 : '||s.quantile_5);  
   dbms_output.put_line('Median      : '||s.median);  
   dbms_output.put_line('Extreme Count : '||s.extreme_values.count);  
   dbms_output.put_line('Extremes      : '||s.extreme_values(1));  
   dbms_output.put_line('Top 5 : '||s.top_5_values(1)||','||  
                                                s.top_5_values(2)||','||  
                                                s.top_5_values(3)||','||  
                                                s.top_5_values(4)||','||  
                                                s.top_5_values(5));  
   dbms_output.put_line('Bottom 5 : '||s.bottom_5_values(5)||','||  
                                                     s.bottom_5_values(4)||','||  
                                                     s.bottom_5_values(1));  
end;
hdp://www.oraly)cs.com/2013/04/part-‐1ge`ng-‐started-‐with-‐sta)s)cs.html

www.oraly)cs.com
t
:
@brendan)erney
e
:

Stats
and
More
Stats

§  Correla)ons

(Spearman’s,
Kendall)

§  Linear
Regression

§  T-‐Test

§  F-‐Test

§  Hypothesis
tes)ng

§  Anova

§  Ranking

§  Window
Aggregate
func)ons

§  Lead
/
Lag

§  Cross
Tabula)on

§  PIVOT
/
UNPIVOT

§  …

Scalable

Highly
Secure

No
Data

Movement

Real
Time

Produc)on

Deployment

Faster

www.oraly)cs.com
t
:
@brendan)erney
e
:

Comprehensive
Advanced
Analy)cs
Plalorm

www.oraly)cs.com
t
:
@brendan)erney
e
:

Oracle
Data
Mining

§  PL/SQL
Package

§  DBMS_DATA_MINING

§  DBMS_DATA_MINING_TRANSFORM

§  DBMS_PREDICTIVE_ANALYTICS

§  SQL
Func)ons

–  PREDICTION

–  PREDICTION_PROBABILITY

–  PREDICTION_BOUNDS

–  PREDICTION_COST

–  PREDICTION_DETAILS

–  PREDICTION_SET

–  CLUSTER_ID

–  CLUSTER_DETAILS

–  CLUSTER_DISTANCE

–  CLUSTER_PROBABILITY

–  CLUSTER_SET

–  FEATURE_ID

–  FEATURE_DETAILS

–  FEATURE_SET

–  FEATURE_VALUE

§  12c
–
Predic)ve
Queries

§  aka

Dynamic
Queries

§  Transi)ve
dynamic
Data
Mining
models

§  Can
scale
to
many
100+
models
all
in
one

statement

www.oraly)cs.com
t
:
@brendan)erney
e
:

select cust_id, affinity_card,
PREDICTION( FOR to_char(affinity_card) USING *) OVER () pred_affinity_card
from mining_data_build_v;
PQ
to
predict
the

AFFINITY_CARD
value.

Using
all
the
data

USING
*

www.oraly)cs.com
t
:
@brendan)erney
e
:

PREDICTION( FOR to_char(affinity_card) USING *) OVER () pred_affinity_card
With
PQs
we
can

dynamically
create

new
DM
models
based

on
an
Adribute(s)

www.oraly)cs.com
t
:
@brendan)erney
e
:

PREDICTION( FOR to_char(affinity_card) USING *) OVER
(PARTITION BY "COUNTRY_NAME") pred_affinity_card
A
new
DM
Model
will

be
created
for
each

Country
(19)

With
PQs
we
can

dynamically
create

new
DM
models
based

on
an
Adribute(s)

www.oraly)cs.com
t
:
@brendan)erney
e
:

Analy)c
Func)ons

(in
12c)

>46
Analy)cs
Func)ons
in
12c

www.oraly)cs.com
t
:
@brendan)erney
e
:

What
about
R
?

www.oraly)cs.com
t
:
@brendan)erney
e
:

--
-- There are 2 ways to use the GLM model : in Batch and in Real-Time mode
--
-- First Step : Build the in-database R script to score you new data
--
Begin
sys.rqScriptDrop('Demo_GLM_Batch');
sys.rqScriptCreate('Demo_GLM_Batch',
'function(dat, datastore_name) {
ore.load(datastore_name)
prd <- predict(mod, newdata=dat)
prd[as.integer(rownames(prd))] <- prd
res <- cbind(dat, PRED = prd)
res}');
end;
/
--
-- Now you can run the script to score the new data in Batch model
-- The data is located in the table MINING_DATA_APPLY
--
select * from table(rqTableEval(
cursor(select CUST_GENDER, AGE, CUST_MARITAL_STATUS, COUNTRY_NAME, CUST_INCOME_LEVEL, EDUCATION,
HOUSEHOLD_SIZE, YRS_RESIDENCE
from MINING_DATA_APPLY_V
where rownum <= 10),
cursor(select 1 as "ore.connect", 'myDatastore' as "datastore_name" from dual),
'select CUST_GENDER, AGE, CUST_MARITAL_STATUS, COUNTRY_NAME, CUST_INCOME_LEVEL, EDUCATION,
HOUSEHOLD_SIZE, YRS_RESIDENCE, 1 PRED from MINING_DATA_APPLY_V','Demo_GLM_Batch'))
order by 1, 2, 3;

www.oraly)cs.com
t
:
@brendan)erney
e
:

External

View

External

View

Conceptual
Schema

Physical
Schema

www.oraly)cs.com
t
:
@brendan)erney
e
:

CREATE TABLE countries_ext (
country_code VARCHAR2(5),
country_name VARCHAR2(50),
country_language VARCHAR2(50)
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY ext_tab_data
ACCESS PARAMETERS (
RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY ','
MISSING FIELD VALUES ARE NULL
(
country_code CHAR(5),
country_name CHAR(50),
country_language CHAR(50)
)
)
LOCATION ('Countries1.txt','Countries2.txt')
)
PARALLEL 5
REJECT LIMIT UNLIMITED;
SELECT * FROM countries_ext ORDER BY country_name;
COUNT COUNTRY_NAME COUNTRY_LANGUAGE
----- ---------------------------- -----------------------------
ENG England English
FRA France French
GER Germany German
IRE Ireland English
External

View

External

View

Conceptual
Schema

Physical
Schema

www.oraly)cs.com
t
:
@brendan)erney
e
:

CREATE TABLE json_dump_file_contents (
json_document CLOB
)
ORGANIZATION EXTERNAL (
TYPE ORACLE_LOADER
DEFAULT DIRECTORY order_entry_dir
ACCESS PARAMETERS (
RECORDS DELIMITED BY 0x'0A'
DISABLE_DIRECTORY_LINK_CHECK
BADFILE loader_output_dir: 'JSONDumpFile.bad'
LOGFILE order_entry_dir: 'JSONDumpFile.log'
FIELDS (
json_document CHAR(5000)
)
)
LOCATION (order_entry_dir:'PurchaseOrders.dmp')
)
PARALLEL
SELECT count(*)
FROM json_dump_file_contents po
WHERE to_number(json_value(json_document, '$.PONumber')) > 1500;
External

View

External

View

Conceptual
Schema

Physical
Schema

www.oraly)cs.com
t
:
@brendan)erney
e
:

CREATE TABLE json_documents (
id RAW(16) NOT NULL,
data CLOB,
CONSTRAINT json_documents_pk PRIMARY KEY (id),
CONSTRAINT json_documents_json_chk CHECK (data IS JSON (STRICT) )
);
INSERT INTO json_documents (id, data)
VALUES (SYS_GUID(),
'{ "FirstName" : ”Brendan",
"LastName" : ”Tierney",
"Job" : "Clerk",
"Address" : { "Street" : ”1 Main Street",
"City" : ”Dublin",
"Country" : ”Ireland”},
"ContactDetails" : { "Email" : ”xyz@oralytics.com",
"Phone" : ”353 123 1234567",
"Twitter" : "@brendantierney" },
"DateOfBirth" : "01-JAN-2000",
"Active" : unknown }');
SELECT a.data.FirstName,
a.data.LastName,
a.data.ContactDetails.Email AS Email
FROM json_documents a
ORDER BY a.data.FirstName, a.data.LastName;
FIRSTNAME LASTNAME EMAIL
--------------- --------------- -------------------------
Brendan Tierney xyz@oralytics.com
External

View

External

View

Conceptual
Schema

Physical
Schema

www.oraly)cs.com
t
:
@brendan)erney
e
:

CREATE TABLE customer (
id NUMBER(38),
name VARCHAR2(100),
address VARCHAR2(100),
city VARCHAR2(40),
country VARCHAR2(50),
location MDSYS.SDO_GEOMETRY
);
INSERT INTO customer VALUES (
cust_seq.nextval,‘Brendan Tierney’, ‘1 Main Street’, ‘Dublin’, ‘Ireland’,
SDO_GEOMETRY
(2001, -- Geometry Type: 2-D Point
8307, -- SRID, Datum: WGS84
SDO_POINT_TYPE
(53.3498, -- Longitude for Dublin
6.2603, -- Latitude for Dublin
NULL),
NULL,
NULL
)
)
SELECT sdo_geom.sdo_distance(c1.locationm c2.location, 0.5, ‘unit=kilometer’)
FROM customer c1,
customer c2
WHERE c1.id = 1
AND c2.id = 2;
External

View

External

View

Conceptual
Schema

Physical
Schema

Spa)al

&

Graph

www.oraly)cs.com
t
:
@brendan)erney
e
:

Using
Oracle
Big
Data
SQL,
organiza)ons
can:

•  Combine
data
from
Oracle
Database,
Apache
Hadoop
and
NoSQL
in
a
single
SQL
query

•  Query
and
analyze
data
In
Apache
Hadoop
and
NoSQL

•  Maximize
query
performance
on
all
data
using
advanced
techniques
like
Smart
Scan,

Par))on
Pruning,
Storage
Indexes,
Bloom
Filters
and
Predicate
Push-‐Down
in
a

distributed
architecture

•  Integrate
big
data
analyses
into
exis)ng
applica)ons
and
architectures

External

View

External

View

Conceptual
Schema

Physical
Schema

Spa)al

&

Graph

Oracle

NoSQL

www.oraly)cs.com
t
:
@brendan)erney
e
:

External

View

External

View

Conceptual
Schema

Physical
Schema

Spa)al

&

Graph

Oracle

NoSQL

Accessing
data
on
Hadoop
or
an
Oracle
NoSQL
Database
requires
access
via
Hive/
HCatalog.

To
use
this
Hadoop
or
NoSQL
data

•  Crea)ng
a
NoSQL
Store
and
a
Table
(or
Hadoop
data)

•  Conﬁguring
Hive/HCatalog
to
access
NoSQL
Table
or
other
data

•  Conﬁguring
Oracle
Database
to
talk
to
HCatalog
via
an
external
table

www.oraly)cs.com
t
:
@brendan)erney
e
:

External

View

External

View

Conceptual
Schema

Physical
Schema

Spa)al

&

Graph

Oracle

NoSQL

CREATE TABLE movieapp_log_json (
custid INTEGER ,
movieid INTEGER ,
genreid INTEGER ,
time VARCHAR2 (20) ,
recommended VARCHAR2 (4) ,
activity NUMBER,
rating INTEGER,
price NUMBER
)
ORGANIZATION EXTERNAL
(
TYPE ORACLE_HIVE
DEFAULT DIRECTORY DEFAULT_DIR
)
SELECT f.custid, m.title, m.year, m.gross, f.rating
FROM movieapp_log_json f, movie m
WHERE f.movieId = m.movie_id
AND f.rating > 4
Selects Hadoop data
and in-database data

www.oraly)cs.com
t
:
@brendan)erney
e
:

Data
Security

§  Can
apply
all
the
typical
data
security
that
comes
with
Oracle
on
all
our
data

–  Masking/Redac)on

–  Virtual
Private
Databases

–  Fine-‐grained
access
control

www.oraly)cs.com
t
:
@brendan)erney
e
:

Store

Access

Analyze

Protect

External

View

External

View

Conceptual
Schema

Physical
Schema

Spa)al

&

Graph

Oracle

NoSQL

www.oraly)cs.com
t
:
@brendan)erney
e
:

SQL

One
Ring
to
rule
them
all,
One
Ring
to
ﬁnd
them,

One
Ring
to
bring
them
all
and
in
the
darkness
bind
them

Sauron
–
Lord
of
the
Rings

www.oraly)cs.com
t
:
@brendan)erney
e
:

SQL

One
SQL

to
rule
them
all,
One

SQL
to
ﬁnd
them,

One

SQL
to
bring
them
all
and
in
the
Database
bind
them

Sauron
–
Lord
of
the
Rings

Brendan
Tierney

www.oraly)cs.com
t
:
@brendan)erney
e
:

SQL

One
Language
to
rule
them
all,
One
Language
to
ﬁnd
them,

One
Language
to
bring
them
all
and
in
the
Database
bind
them

Sauron
–
Lord
of
the
Rings

Brendan
Tierney

www.oraly)cs.com
t
:
@brendan)erney
e
:


@brendan)erney

www.oraly)cs.com

ie.linkedin.com/in/brendan)erney

www.oraly)cs.com
t
:
@brendan)erney
e
:

Word
Cloud
of
the
Oracle
Advanced

Analy)cs
web-‐pages

hdp://www.oraly)cs.com/2015/01/crea)ng-‐word-‐cloud-‐of-‐oracle-‐oaa.html

SQL: The one language to rule all your data

More Related Content

What's hot

Similar to SQL: The one language to rule all your data

Recently uploaded

SQL: The one language to rule all your data