The Pathology of Graph Databases

The Pathology of Graph Databases Marko A. Rodriguez Graph Systems Architect http://markorodriguez.com http://twitter.com/twarko http://tinkerpop.com TinkerPop WindyCityDB - Chicago, Illinois – June 25, 2011

AbstractThere is nothing more fascinating and utterly mind-bending than traversing a graph. Those whosuccumb to this data processing pattern euphoricallysuffer from graph pathology.This is a case study of the Graph Addict. Gremlin G = (V, E) What is presented is as of Gremlin 1.1 (Released July 15, 2011)

What you are about to see may disturb you.It has infected Neo4j, OrientDB, DEX, RDF Sail, TinkerGraph, and ReXster. reXster ...it will infect others.

~$ cd /tmp/graphdb

graphdb$ gremlin

,,,/ (o o)-----o00o(_)-o00o-----gremlin>

The Basics

Gremlin is a Groovy DSL and as such, has native JVM access.Gremlin is a Groovy DSL and as such, has native JVM access.

gremlingremlin

emlingremlin.substring(2)

[1,2,3,4,5]list = [1,2,3,4,5]

key1=value key2=123map = [key1:value,key2:123]

<modem sound/>socket = new Socket(127.0.0.1,8182)

trueTextAnalysis.isMean(You are that which is not good.)

Graph Traversing

/tmp/graphdbg = new Neo4jGraph(/tmp/graphdb)

/tmp/graphdbg

1g.v(1)

name=marko 1g.v(1).name

1g.v(1)

city=santafe 1g.v(1).city

1g.v(1)

name=marko city=santafe 1g.v(1).map()

1g.v(1)

created 1 knows knowsg.v(1).outE

null created since=2010 1 knows since=2005 knowsg.v(1).outE.since

2 created 1 knows 3 knows 4g.v(1).outE.inV

2 name=rexster created 1 knows 3 name=stephen knows 4 name=peterg.v(1).outE.inV.name

/tmp/graphdbg

1g.v(1)

created 1 knows knowsg.v(1).outE[[label:knows]]

1g.v(1)

created 1 knows knowsg.v(1).outE.filter{it.label==knows}

2g.v(1).outE.filter{it.label==knows}.count()

created 1 knows knowsg.v(1).outE.filter{it.label==knows & since > 2006}

[OutEdgesPipe, LabelFilterPipe(NOT_EQUAL,knows), InVertexPipe]g.v(1).outE[[label:knows]].inV.toString()

1g.v(1)

1 knows knowsg.v(1).outE(knows)

1 knows 3 knows 4g.v(1).outE(knows).inV

[OutEdgesPipe(knows), InVertexPipe]g.v(1).outE(knows).inV.toString()

1g.v(1)

1 knows 3 knows 4g.v(1).out(knows)

1 knows 3 name=stephen knows 4 name=peterg.v(1).out(knows).name

1 knows 3 name=stephen knows 4 name=peterg.v(1).out(knows).name.filter{it.startsWith(st)}

1 knows 3 name=stephen knows 4 name=peterg.v(1).out(knows).name.filter{it.matches(p.{4})}

1 knows 3 name=stephen knows 4 name=peterg.v(1).out(knows).name.filter{it.matches(p.{4}) & false}

[OutPipe(knows)]g.v(1).out(knows).toString()

1g.v(1)

o o! o oH W 1g.v(1).filter{new Random().nextBoolean()}

1g.v(1)

t! A lrigh 1g.v(1).filter{new Random().nextBoolean()}

1g.v(1)

Doh! 1g.v(1).filter{new Random().nextBoolean()}

Graph Inference

1g.v(1)

2 created 1g.v(1).out(created)

2 name=rexster created 1g.v(1).out(created).name

1g.v(1)

2 created created 1 3g.v(1).out(created).in(created)

2 created created 1 3g.v(1).out(created).in(created).except([g.v(1)])

2 created created 1 3 name=stepheng.v(1).out(created).in(created).except([g.v(1)]).name

2 created created 1 3 name=stepheng.v(1).out(created).in(created).except([g.v(1)]).name.back(1)

2 created created 1 collaborator 3 name=stepheng.v(1).out(created).in(created).except([g.v(1)]).name.back(1). sideEffect{g.addEdge(g.v(1),it,collaborator)}

2 created created 1 collaborator 3 name=stepheng.v(1).out(created).in(created).except([g.v(1)]).name.back(1). sideEffect{g.addEdge(g.v(1),it,collaborator)}.filter{false}

Graph Backtracking and Looping

1g.v(1)

5 imports 2 imports 6 imports created 7 1g.v(1).out(created).out(imports)

5 name=blueprints imports 2 imports 6 name=pipes imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name

5 name=blueprints imports 2 imports 6 name=pipes imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1)

8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports)

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports).name

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports).name .back(1)

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports).name .back(1).out(imports)

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports).name .back(1).out(imports).out(imports)

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).name.back(1).out(imports).name .back(1).out(imports).out(imports).out(imports)

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin1

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).loop(1){it.loops < 5}

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1 2g.v(1).out(created).out(imports).loop(1){it.loops < 5}

name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(imports).loop(1){it.loops < 5}

mm = [:]

m name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1

m name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).groupCount(m).loop(2){true}

v[5]=1 m v[6]=1 v[7]=1 name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).groupCount(m).loop(2){true}

v[5]=3 v[6]=2 m v[7]=1 v[8]=1 name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).groupCount(m).loop(2){true}

mm = [:]

m name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1

m name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).name.groupCount(m).back(2).loop(2){true}

blueprints=1 m pipes=1 gremlin=1 name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).name.groupCount(m).back(2).loop(2){true}

blueprints=3 pipes=2 m gremlin=1 neo4j=1 name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).out(import).name.groupCount(m).back(2).loop(2){true}

Graph Paths

1g.v(1)

1g.v(1).as(x)

9 10 child child 1g.v(1).as(x).in(child)

9 10 child child 1g.v(1).as(x).in(child).loop(x)

child child 9 10 child child 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}

child child child child 9 10 child child 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}

child child child child child child 9 10 child child 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}

100,000,000 100,000,001 child child ... ... child child child child child child 9 10 child child 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}

100,000,000 100,000,001 child child ... ... child child child child child child 9 10 child child [v[1], v[9], ..., v[100,000,000]] [v[1], v[10], ..., v[100,000,001]] ... 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}.paths

name=angelina name=brad 100,000,000 100,000,001 child child ... ... child child child child child child 9 10 child child 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}.name

name=angelina name=brad 100,000,000 100,000,001 child child ... ... child child child child child child 9 10 child child [v[1], v[9], ..., v[100,000,000], angelina] [v[1], v[10], ..., v[100,000,001], brad] ... 1g.v(1).as(x).in(child).loop(x){it.object.in.count() != 0}.name.paths

1g.v(1)

child child 1g.v(1).inE(child)

thinks="You suck." thinks="Youre great." child child 1g.v(1).inE(child).filter{TextAnalysis.isMean(it.thinks)}

thinks="You suck." child 1g.v(1).inE(child).filter{TextAnalysis.isMean(it.thinks)}

9 thinks="You suck." child 1g.v(1).inE(child).filter{TextAnalysis.isMean(it.thinks)}.outV

irl." ag w like thro ou ks="Y thin thinks="No pony!" thi n ks ="Y ou re fat ." thinks="Get a real job." 9 thinks="You suck." child 1g.v(1).inE(child).filter{TextAnalysis.isMean(it.thinks)}.outV.loop(3){true}

Graph Pattern Matching

SELECT ?x ?y WHERE { 1 created ?x . ?x imports ?y} name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1

SELECT ?x ?y WHERE { ?x ?y 1 created ?x . v[2] v[5] v[2] v[6] ?x imports ?y v[2] v[7]} name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1t = new Table()

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1)

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created)

t name=neo4j 8 imports 5 name=blueprints imports imports x imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).as(x)

t name=neo4j 8 imports 5 name=blueprints imports imports x imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports)

t name=neo4j 8 imports y 5 name=blueprints imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports).as(y)

x y v[2] v[5] t v[2] v[6] v[2] v[7] name=neo4j 8 imports y 5 name=blueprints imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports).as(y).table(t)

x y v[2] v[5] t v[2] v[6] v[2] v[7] name=neo4j 8 imports y 5 name=blueprints v[5] imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1t.get(0,1)

x y v[2] v[5] t v[2] v[6] v[2] v[7] name=neo4j 8 imports y 5 name=blueprints v[2] imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1t.get(0,x)

x y v[2] v[5] t v[2] v[6] v[2] v[7] name=neo4j 8 imports y 5 name=blueprints v[5] v[6] imports v[7] y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1t.getColumn(y)

x y v[2] v[5] t v[2] v[6] v[2] v[7] name=neo4j 8 imports y 5 name=blueprints [x, y] imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1t.getColumnNames()

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1t = new Table()

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1)

t name=neo4j 8 imports 5 name=blueprints imports imports imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created)

t name=neo4j 8 imports 5 name=blueprints imports imports x imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).as(x)

t name=neo4j 8 imports 5 name=blueprints imports imports x imports 2 imports 6 name=pipes imports imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports)

t name=neo4j 8 imports y 5 name=blueprints imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports).as(y)

x y 2 blueprints t 2 pipes 2 gremlin name=neo4j 8 imports y 5 name=blueprints imports y imports x imports 2 imports 6 name=pipes imports y imports created 7 name=gremlin 1g.v(1).out(created).as(x).out(imports).as(y).table(t){it.id}{it.name}

g.clear()

Graph Construction

1a = g.addVertex()

name=pierre 1a.name = pierre

name=pierre city=brussels 1a.city = brussels

name=stephen city=dc 2 name=pierre city=brussels 1b = g.addVertex([name:stephen,city:dc])

name=rexster logo= reXster . 3 name=stephen city=dc 2 name=pierre city=brussels 1c = g.addVertex([name:rexster,logo:[a12f04b312bc...]])

name=rexster logo= reXster . 3 since=2011 created name=stephen city=dc 2 name=pierre city=brussels 1g.addEdge(b,c,created,[since:2011])

name=rexster logo= reXster . 3 since=2011 created name=stephen since=2011 city=dc reviews 2 name=pierre city=brussels 1g.addEdge(a,c,reviews,[since:2011])

name=gremlin logo= Gremlin G = (V, E) . 4 name=rexster imports logo= reXster . 3 since=2011 created name=stephen since=2011 city=dc reviews 2 name=pierre city=brussels 1g.addEdge(c,g.addVertex([name:gremlin,logo:[ab1346f47...]),imports)

name=pipes logo= Pipes. 5 name=gremlin logo= Gremlin G = (V, E) . imports 4 name=rexster imports logo= reXster . 3 since=2011 created name=stephen since=2011 city=dc reviews 2 name=pierre city=brussels 1g.addEdge(c,g.addVertex([name:pipes,logo:[12cbef46...]]),imports)

name=blueprints name=pipes logo= Blueprints . logo= Pipes. 6 5 name=gremlin logo= Gremlin imports . G = (V, E) imports 4 name=rexster imports logo= reXster . 3 since=2011 created name=stephen since=2011 city=dc reviews 2 name=pierre city=brussels 1g.addEdge(c,g.addVertex([name:blueprints,logo:[f13cd1467...]]),imports)

6 5 4 3 2 1g.V

6g.V.count()

blueprints pipes gremlin rexster stephen pierreg.V.name

6 5 4 3 2 1g.V

Blueprints Pipes Gremlin G = (V, E) reXsterg.V.logo

4g.V.logo.count()

name=blueprints name=pipes logo= Blueprints . logo= Pipes . name=gremlin logo= Gremlin G = (V, E) . name=rexster logo= reXster . name=stephen city=dc name=pierre city=brusselsg.V.transform{it.map()}

6g.V.transform{it.map()}.count()

7g.V.transform{it.map()}.count() + 1

8g.V.transform{it.map()}.count() + 1 + 1

4(g.V.transform{it.map()}.count() + 1 + 1) / 2

4 is a number.((g.V.transform{it.map()}.count() + 1 + 1) / 2) + is a number.

14(((g.V.transform{it.map()}.count() + 1 + 1) / 2) + is a number.).length()

imports imports imports created reviewsg.E

since=2011 since=2011g.E.transform{it.map()}

g.clear()

Graph Ranking

7 friend 6 friend friend 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.loadGraphML(loopy-lou-and-the-loops.xml)

m 7 friend 6 friend friend 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3m = [:]

m 7 friend 6 friend friendc 0 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3c = 0

m 7 friend 6 friend friendc 0 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[2]=1m v[6]=1 7 friend 6 friend friendc 0 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[2]=1m v[6]=1 7 friend 6 friend friendc 2 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[2]=1 v[3]=1m v[6]=2 v[7]=1 7 friend 6 friend friendc 2 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[2]=1 v[3]=1m v[6]=2 v[7]=1 7 friend 6 friend friendc 5 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[1]=1 v[2]=1 v[3]=1m v[4]=1 v[6]=2 v[7]=2 v[8]=1 7 friend 6 friend friendc 5 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[1]=1 v[2]=1 v[3]=1m v[4]=1 v[6]=2 v[7]=2 v[8]=1 7 friend 6 friend friendc 9 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[1]=3 v[2]=2 v[3]=1 v[4]=1m v[5]=1 v[6]=3 v[7]=2 v[8]=2 7 friend 6 friend friendc 9 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3g.v(1).out.groupCount(m).loop(2){c++ < 35}.filter{false}

v[1]=5 v[2]=5 v[3]=5 v[4]=3m v[5]=1 v[6]=10 v[7]=7 v[8]=3 7 friend 6 friend friendc 39 8 5 friend friend friend friend friend friend 1 4 friend friend 2 friend 3println do you understand the concept of centrality?

g.clear()

IDE Development reXster Command Line REPL REST Server

Via any JVM language (JSR 223) Neo4j Server

http://gremlin.tinkerpop.com

http://www.graph-database.org	4511
http://nosql.mypopescu.com	2668
url_unknown	53
http://neo4j.tw	41
http://translate.googleusercontent.com	12
http://localhost:8080	6
https://twitter.com	6
http://www.slideshare.net	6
http://feeds.feedburner.com	4
http://webcache.googleusercontent.com	4
http://graph-databases.org	3
http://www.graph-databases.org	3
http://twitter.com	3
http://www.hanrss.com	2
http://paper.li	1
http://www.linkedin.com	1

The Pathology of Graph Databases

by Marko Rodriguez

Accessibility

Categories

Upload Details

Usage Rights

16 Embeds 7,324

Statistics