Austin Data Geeks - Why relationships are cool but join sucks

Switching from
Relational to Graph
Luca
Garulli
–
Founder
and
CEO
@Orient
Model
Technologies
Ltd
Author
of
OrientDB
(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported Licewnwswe. o rien1technologies.com
!
www.twitter.com/lgarulli

1979
First Relational DBMS available as product
!
!
!
!
!
!
2009
NoSQL movement
!
(c) Luca Garulli Licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License 2

1979
First Relational DBMS available as product
!
!
!
!
!
!
2009
30
yrs
is
a
long
time!
NoSQL movement
!

Before 2009, teams of developers
always wanted a say in selecting:
!
Operative System
Programming Language
Middleware (App-Servers)
!
What about the Database?

One of the main reasons
RDBMS users resist passing to NoSQL is
related to the
complexity of the RDBMS model:
!
NoSQL products are great for
BigData and BigScale
but...

…can it handle complexity?

What is the NoSQL answer
for managing complex domains?
!
!
Key-Value stores ?
Column-Based ?
Document database ?
Graph database !
No Relationship
support

Why don’t most NoSQL
products support
Relationships
between entities?

To understand why,
let’s see how
Relational DBMS
manages them

Domain: the super minimal “Selling App”
Customer Address
Order Stock
Registry
system
Order
system

Domain: the super minimal “Selling App”
Customer Address
Stock
Registry
system
Order
Order
system
How
does
Relational
DBMS
manage
this
relationship?

Relational World: 1-1 Relationships
!
!
!
!
!
!
!
!
!
Primary
key Primary
key
Customer
Id Name Address
10 Luca 34
11 Jill 44
34 John 54
56 Mark 66
88 Steve 68
Address
Id Location
34 Rome
44 London
54 Moscow
66 New
Mexico
68 Palo
Alto
Foreign
key
JOIN Customer.Address -> Address.Id

Relational World: 1-N Relationships
!
!
!
!
!
!
!
!
!
Customer
Id Name
10 Luca
11 Jill
34 John
56 Mark
88 Steve
Address
Id Customer Location
24 10 Rome
33 10 London
44 34 Moscow
66 56 Cologne
68 88 Palo
Alto
Inverse JOIN Address.Customer -> Customer.Id

Relational World: N-M Relationships
!
!
!
!
!
!
!
!
CustomerAddress
Id Address
10 24
10 33
34 44
Additional table with 2 JOINs
Id Location
24 Rome
33 London
44 Moscow
66 Cologne
68 Palo
Alto
(1) CustomerAddress.Id -> Customer.Id and
(2) CustomerAddress.Address -> Address.Id
Customer
Id Name
10 Luca
11 Jill
34 John
56 Mark
88 Steve
Address

What’s wrong with the
Relational Model?

The JOIN is evil!
Customer
Id Name
10 Luca
11 Jill
34 John
56 Mark
88 Steve
Address
Id Location
24 Rome
33 London
44 Moscow
66 Cologne
68 Palo
Alto
CustomerAddress
Id Address
10 24
10 33
34 24
These
JOINs
are
all
executed
every
time
you
traverse
a
relationship

A JOIN means searching for a key in
another table
!
The usual way to improve performance
is to index all the keys
!
Indexing speeds up searches, but slows
down inserts, updates and deletes

So a JOIN is essentially a lookup
into an index
!
This is done for every single join!
!
If you traverse hundreds of relationships,
you’re executing hundreds of JOINs

Index Lookup
is it really that fast?

Index Lookup: how does it work?
A-‐Z
A-‐L M-‐Z
Imagine
an
Address
Book
where
we
want
to
find
Luca’s
phone
number

A-‐Z
A-‐L M-‐Z
A-‐L
A-‐D E-‐L
M-‐Z
M-‐R S-‐Z
Index
algorithms
are
all
similar
and
based
on
balanced
trees

A-‐Z
A-‐L M-‐Z
A-‐L
A-‐D E-‐L
M-‐Z
M-‐R S-‐Z
A-‐D
A-‐B C-‐D
E-‐L
E-‐G H-‐L

A-‐Z
A-‐L M-‐Z
A-‐L
A-‐D E-‐L
M-‐Z
M-‐R S-‐Z
A-‐D
A-‐B C-‐D
E-‐L
E-‐G H-‐L
E-‐G
E-‐F G
H-‐L
H-‐J K-‐L

A-‐Z
A-‐L M-‐Z
A-‐L
A-‐D E-‐L
M-‐Z
This
lookup
took
M-‐R S-‐Z
A-‐D
A-‐B C-‐D
E-‐L
E-‐G H-‐L
E-‐G
E-‐F G
H-‐L
H-‐J K-‐L
Luca
Found!
5
steps.
With
millions
of
indexed
records,
the
tree
depth
could
be
1000’s
of
levels!

Can you imagine
how many steps a
Lookup operation takes for an
Index with Billions
of records?

And this JOIN is executed
for every involved table,
multiplied
for all scanned records!

Querying more tables can easily
produce millions of JOINs/Lookups!
!
Here’s the rule:
more entries
= more lookup steps
= slower queries

AAhThis is why the
query performance of your
database suffers
as the database
becomes bigger,
and bigger,
and bigger!

What about
Document Databases
like MongoDB?

How MongoDB manages relationships:
!
{
“_id” : “292846512”,
“type” : “Order”,
“number” : 1223,
“customer” : “123456789”
}

MongoDB uses the
same RDBMS approach:
!
it stores the _id of the connected
documents. At run-time, it looks up
the _id by using an index.

Is there a better way to
manage relationships?

“A graph database is any
storage system
that provides
index-free adjacency”
!
- Marko Rodriguez
(author of TinkerPop Blueprints)

How does a GraphDB manage
index-free relationships?

Every developer knows
the Relational Model,
but who knows the
Graph one?

Back to school:
Graph Theory crash course
!

Basic Graph
Likes
Luca Austin

Property Graph Model*
Luca
!
name:
Luca
surname:
Garulli
company:
Orient
Technologies
!
Austin
!
population:
1,900,000
Vertices
are
directed
Likes
since:
2014
Vertices
and
Edges
can
have
properties
*
https://github.com/tinkerpop/blueprints/wiki/Property-‐Graph-‐Model

1-N relationships
Luca
!
Austin
Likes
!
since:
2014
when:
An
Edge
connects
only
2
vertices
!
Use
multiple
edges
to
represent
1-‐N
and
N-‐M
relationships
Visited
!
[2013,
2014]

Graph Example
IsMemberOf
Likes
Luca
Austin
Data
Geeks
Hosts
Austin
Visited

Congrats! This is your diploma in
«Graph Theory»

The Graph theory
is so simple,
yet so
powerful

Let’s go back
to the Graph Stuff
!
How does OrientDB
manage relationships?

OrientDB: traverse a relationship
The Record ID (RID)
is the physical position
RID
=
#13:35 RID
=
#13:100
Luca
(vertex)
label
:
‘Customer’
name
:
‘Luca’
Rome
(vertex)
label
=
‘City’
name
=
‘Rome’

OrientDB: traverse a relationship
The Edge’s RID is saved
inside both vertices, as «out»
and «in»
RID
=
#13:35 RID
=
#13:100
RID
=
#14:54
Lives
Luca
(vertex)
out
:
[#14:54]
label
:
‘Customer’
name
:
‘Luca’
out:
[#13:35]
in:
[#13:100]
Label
:
‘Lives’
Rome
(vertex)
in:
[#14:54]
label
=
‘City’
name
=
‘Rome’

OrientDB: traverse -> outgoing
RID
=
#13:35
RID
=
#14:54
Luca Lives
out
:
[#14:54]
label
:
‘Customer’
name
:
‘Luca’
out:
[#13:35]
in:
[#13:100]
Label
:
‘Lives’
RID
=
#13:100
Rome
in:
[#14:54]
label
=
‘City’
name
=
‘Rome’

OrientDB: traverse <- incoming
RID
=
#13:35
RID
=
#14:54
Luca Lives
out
:
[#14:54]
label
:
‘Customer’
name
:
‘Luca’
out:
[#13:35]
in:
[#13:100]
Label
:
‘Lives’
RID
=
#13:100
Rome
in:
[#14:54]
label
=
‘City’
name
=
‘Rome’

A GraphDB handles relationships as a
physical LINK to the record,
assigned only when the edge is created
!
VS
!
RDBMS computes the
relationship every time you query a database
!
Isn’t that crazy?!

This means jumping from a
O(log N) algorithm to a near O(1)
!
With OrientDB, the traversing time is
not affected by database size!
!
This is huge in the BigData age

an Open Source (Apache licensed)
document-graph NoSQL dbms

OrientDB in the Blueprints micro-benchmark,
on common hw, with a hot cache,
traverses 29,6 Million
records in less than 5 seconds
!
about 6 Million nodes traversed per sec!
Do not try this at home
!
!
!
!
with a RDBMS*!
*unless you live in Google’s server farm

Create the graph in SQL
!
$luca> cd bin
$luca> ./console.sh
OrientDB console v.1.6.1 (www.orientdb.org)
Type 'help' to display all the commands supported.
!
orientdb> create vertex Customer set name = ‘Luca’
Created vertex #13:35 in 0.03 secs
!
orientdb> create vertex Address set name = ‘Rome’
Created vertex #13:100 in 0.02 secs
!
orientdb> create edge Lives from #13:35 to #13:100
Created edge #14:54 in 0.02 secs

Create the graph in Java
!!
Graph graph = new OrientGraph("local:/tmp/db/graph”);
!
Vertex luca = graph.addVertex( “class:Customer” );
luca.setProperty( “name", “Luca” );
!
Vertex rome = graph.addVertex ( “class:Address” );
rome.setProperty( “name", “Rome” );
!
Edge edge = luca.addEdge( “Lives”, rome );
!
graph.shutdown();

Query the graph in SQL
!!!
orientdb> select in(‘Lives’) from Address where name = ‘Rome’ !!
---+------+---------|--------------------+--------------------+--------+
#| RID |@class |label |out_Lives |in |
---+------+---------+--------------------+--------------------+--------+
0| 13:35|Customer |Luca |[#14:54] | |
---+------+---------+--------------------+--------------------+--------+
1 item(s) found. Query executed in 0.007 sec(s).
Incoming vertices

More on query power
!!
orientdb> select sum( out(‘Order’).total ) from Customer
where name = ‘Luca’
!
orientdb> traverse both(‘Friend’)
from Customer while $depth <= 7
!
orientdb> select from (
traverse both(‘Friend’)
from Customer while $depth <= 7
) where @class=‘Customer’ and city.name = ‘Austin’

Query vs traversal
!
With a well-connected database
in the form of a Super Graph, you can
cross records instead of query them!
!
All you need is a few “Root Vertices”
to start traversing

Query vs traversal
Customers
Luca Mark Jill
Order
2332
Order
8834
Stocks
White
Soap
Special
Customers
This is a
root vertex

Root Vertices can be enriched by
Meta Graphs
to decorate Graphs with
additional information
and make the retrieval
easier/faster

Temporal based Meta Graph
Day
9/4/2013
Order
2333
Order
2334
Calendar
Hour
9/4/2013
10:00
Month
April
2013
Hour
9/4/2013
09:00
Order
2332
Year
2013

Location based Meta Graph
State
RM
Order
2333
Order
2334
Location
City
Fiumicino
Region
Lazio
City
Rome
Order
2332
Country
Italy

Mix & Merge graphs
State
RM
Order
2333
Order
2334
Location
City
Fiumicino
City
Rome
Order
2332
Region
Lazio
Country
Italy
Calendar
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013

State
RM
Get all the orders
sold in “Rome”
on 9/4/2013 at 10:Order
2333
00
Order
2334
Location
City
Rome
!
!
City
Fiumicino
Order
2332
Region
Lazio
Country
Italy
Calendar
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013

Start from Calendar, look for Hour 10:00
State
RM
Order
2333
Order
2334
Location
City
Fiumicino
City
Rome
Order
2332
Region
Lazio
Country
Italy
Calendar
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013

Found Found 2 2 Orders, Orders, now filter filter
by
State
RM
by incoming incoming edges<
edges
Order
2333
Order
2334
Location
City
Rome
City
Rome
Order
2332
Region
Lazio
Country
Italy
Calendar
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013

State
RM
Order
2333
Location
City
Rome
Order
2332
Region
Lazio
Country
Italy
Only “Order 2333” has
incoming connections with
Calendar
City
Fiumicino
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013
Order
2334
“Rome”

Or start from Location, look for Rome
State
RM
Order
2333
Location
City
Rome
Order
2332
Region
Lazio
Country
Italy
Calendar
City
Fiumicino
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Month
April
2013
Year
2013
Order
2334

Start from Location, look for Rome
State
RM
Order
2333
City
Rome
Order
2332
Location
Calendar
City
Fiumicino
Hour
9/4/2013
10:00
Hour
9/4/2013
09:00
Day
9/4/2013
Region
Lazio
Month
April
2013
Year
2013
Order
2334
Country
Italy

Recommendation system
Luca
Jill
Enrico
Friend
Friend

Salt
Lick
Eats
Luca
Pappasito's
Jill
Enrico
Hut’s
Burgers
Friend
Friend
Eats
Eats
Eats
Franklin’s
BBQ
Eats

select
both(‘Friend’)
Salt
Lick
from
Person
where
name
=
‘Luca’
Eats
Luca
Pappasito's
Jill
Enrico
Hut’s
Burgers
Friend
Friend
Eats
Eats
Eats
Franklin’s
BBQ

select
both(‘Friend’).out(‘Eats’)
from
Person
where
name
=
‘Luca’
Da
Carlone
Eats
Luca
Meridionale
Pappasito's
Jill
Enrico
Hut’s
La
Burgers
Mediterranea
Friend
Salt
Lick
Friend
Eats
Eats
Eats
Franklin’s
Eaitaly
BBQ

select
both(‘Friend’).out(‘Eats’)
Salt
Lick
from
Person
where
name
=
‘Luca’
Eats
Luca
Pappasito's
Jill
Enrico
Hut’s
Burgers
Friend
Friend
Eats
Eats
Eats
Franklin’s
BBQ

Let’s move like a
Spider
on the web

OrientDB is a Graph - Document DBMS
Document Graph
It provides the flexibility and speed of a document
store combined with the advanced relationship
features of a graph database

!
Download
Installation
in 5 minutes
!
Unzip
!
Run
You only need a JVM installed to run the server

200,000
documents per second
(no index, multi-threads, on commodity hw)

Schema-less
!
schema is not mandatory, relaxed model,
collect heterogeneous documents all together

Schema-full
!
schema with constraints on fields and validation rules
!
Customer.age > 17
Customer.address not null
Customer.surname is mandatory
Customer.email matches 'b[A-Z0-9._%+-]+@[A-Z0-9.-]+.[A-Z]{2,4}b'

Schema-mixed
!
schema with mandatory and optional fields
+ constraints
= the best of schema-less and schema-full modes

ACID Transactions
!!
db.begin();
try {
// your code
...
db.commit();
!
} catch( Exception e ) {
db.rollback();
}

SQL
!
select * from employee where name like '%Jay%'

Why reinvent
yet another language when
almost all developers already
know SQL?
!
OrientDB uses SQL
but extends it adding new
operators for graph manipulation

For most of the queries
a programmer uses everyday
SQL is simpler,
more readable and
compact then
Scripting (Map/Reduce)

Native
JSON
!!
ODocument = new ODocument().fromJSON( "
{
'@rid' = '26:10',
'@class' = 'Developer',
'name' : 'Luca',
'surname' : 'Garulli',
'out' : [ #10:33, #10:232 ]
}“ );

Feature
matrix
FEATURES ORIENTDB)) MONGODB NEO4J MYSQL)
(RDBMS)
Operational Database X X X
Graph Database X X
Document Database X X
Object-Oriented Concepts X
Schema-full, Schema-less, Schema mix X
User and Role & Record Level Security X
Record Level Locking X X X
SQL X X
ACID Transaction X X X
Relationships (Linked Documents) X X X
Custom Data Types X X X
Embedded Documents X X
Multi-Master Zero Configuration Replication X
Sharding X X
Server Side Functions X X X
Native HTTP Rest/ JSON X X
Embeddable with No Restrictions X

Free
!
Open Source Apache 2 license
free for any purpose,
even commercial

Show time!

Ready to try OrientDB?
!
Start from the ETL

OrientDB
ETL
{
"extractor" : {
"jdbc": { "driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://localhost/mysqlcrm",
"userName": "root",
"userPassword": "",
"query": "select * from Client" }
},
"transformers" : [
{ "vertex": { "class": "Client"} }
],
"loader" : {
"orientdb": {
"dbURL": "plocal:/temp/databases/orientdbcrm",
"dbAutoCreate": true
}
}
}

./oetl.sh mydb.json

Drivers
Contribute to build or improve a library!

Professional Services by
!
!
LTD
LTD
(London,
UK)
Development Support
Production Support
Training
Consultancy

We’re
looking
for
worldwide
partners
Write
to
info@orientechnologies.com

http://www.orientechnologies.com/why-‐orientdb

www.orientechnologies.com
Thanks!
Luca
Garulli
–
Founder
and
CEO
!
www.twitter.com/lgarulli

Austin Data Geeks - Why relationships are cool but join sucks

More Related Content

Similar to Austin Data Geeks - Why relationships are cool but join sucks

Recently uploaded

Austin Data Geeks - Why relationships are cool but join sucks