「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜

Graph DB GraphDB doryokujin+WEB ( Tokyo.Webmining #9-2)

[Me] doryokujin 2 2 33[Company] 1

[ ] MongoDB JP TokyoWebMining MongoDB[ ] MongoDB MongoDB GraphDB

#1[MongoTokyo] Mongo DB Congerence in Japan 2011 03 01 10gen 3 … http://www.10gen.com/conferences/ mongotokyo2011

#2[gihyo ] gihyo.jp 2 DocumentDB GraphDB NoSQL

Graph GraphGraph DB Graph Traversal Graph DB Neo4j, Sones, InfoGrid, OrientDB, InﬁniteGraphTinker Pop Gremlin, Blueprints, Pipes, Rexster, Mutant

GraphGraph GraphGraph DB Graph Traversal Graph DB Neo4j, Sones, InfoGrid, OrientDB, InﬁniteGraphTinker Pop Gremlin, Blueprints, Pipes, Rexster, Mutant

Graph: GraphGraph DB Graph

Graph[Graph] Dots Lines vertices edges 1 (relationship) Dots Lines Graph

Undirected Graph[ (Undirected)Graph] Vertices: Edges: (relationship) (symmetric)

Directed Graph[ (Directed) Graph] Vertices: Edges: (relationship) (asymmetric)

Directed / Underected Graph friend follow friend follow followfriend follow [Facebook] [Twitter] ”Undirected Graph” Follow ”Directed Graph” ” ” ” ” ”friends” ”follow”

Single-Relational GraphSingle-Relational Structures → Undirected / Directed Graph Single-Relatinal 1 Graph

Single-Relational Graph friend follow friend follow followfriend follow [Facebook] [Twitter] ”Undirected Graph” Follow ”Directed Graph” ”Facebook ” ”Twitter ” ”friends” ”follow”

Single-Relational Reply num:5 Reply Block num:5 Reply DM num:5 num:1 RT RT Reply DM num:2 num:2 num:2 num:1 [Twitter] Graph ”Directed Graph” ”Twitter ” ”Reply”,”RT”,”DM”,”Block”

*Facebook Flickr lives_in is is is follow lives_in friend is share * friend share follow follow is[ ] lives_in Undirected Directed is is is lives_in

Multi-Relational GraphMulti-Relational Structures lives_in: User → Country Share: Facebook → Flikcr

Multi-Relational Reply Reply Block DM Reply RT RTReply DM [Twitter] ”Twitter ” ”Reply”,”RT”,”DM”,”Block”

Multi-Relational *Facebook Flickr lives_in has has has follow lives_in friend has share * has friend share follow lives_in[Multi-Relatinal Graph] has has has lives_in

Property Graph Property Graph Multi-Relational Graph (Property) Graph DB Graph 1 key/value id id_A follow id id_Bfollow 100 follow 500follower 200 date 2011/01/23 follower 1000

Property Graph Reply num:5 Reply Block num:5 Reply DM num:5 num:1 RT RTReply DM num:2 num:2num:2 num:1 Graph ”Property Graph” ”Twitter ” ”Reply”,”RT”,”DM”,”Block” ”num”

Property Graph name doryokujin sex man lives_in birth 1985/05/14 has has id id_B follow follow 1000 follower 2000lives_in date 2011/01/23 friend has friend date 2011/01/23 has friend follow follow date 2010/03/23 date 2011/01/23 name full name mail xxx@yyy address zzz lives_in id id_A follow 100 follower 200 has has date 2010/03/23 lives_in

Graph The Graph Traversal Pattern

Property Graph Property Graph Graph Property Graph Graph DB Tinker Pop Hyper Graph

Graph DBGragh GraphGraph DB Graph Traversal Graph DB Neo4j, Sones, InfoGrid, OrientDB, InﬁniteGraphTinker Pop Gremlin, Blueprints, Pipes, Rexster, Mutant

Graph DB: Property Graph DB“Graph DB”

Graph DB[ DB ≠ Graph DB] Graph DB DB Graph DB

RDB Graph[Relatinal Database] AoutV inV A B B C A C C D D D A

Document DB Graph [Document Database] A{ A : { out : [B, C], in : [D] } B : { in : [A] B C } C : { out : [D], in : [A] } D : { out : [A], in : [C] D }}

XML DB Graph [XML Database] A<graphml><graph><node id=A /><node id=B /> B C<node id=C /><edge source=A target=B /><edge source=A target=C /><edge source=C target=D /><edge source=D target=A /></graph> D</graphml>

Graph DB[ ]“A graph database is any storage system that provides index-free adjacency” The Graph Traversal Programming Pattern (“adjacent”) ( “index-free” )

Non-Graph DB and Index-Based Adjacency B E 1. A 3. (B,C) A A B CB, C E D, E D E 2. C D log_2(n) (B,C) time cost

Graph DB and Index-Free Adjacency‣ ”Mini - Index” B E‣ 1. 1 A (B,C)‣ C D id id_B follow 1000 follower 2000

Property (key/value) The Graph Traversal Programming Pattern

GraphDB: Graph Traversal Graph DBGraph DB Query

Graph DB QueryGraph Query = Graph Traversal Traversal = Root Graph Graph Traversal (Root) Index-Free Adjacency

private void printFriends( Node person ){ Traverser traverser = person.traverse( Order.BREADTH_FIRST, // StopEvaluator.END_OF_GRAPH, // Graph ReturnableEvaluator.ALL_BUT_START_NODE, // Root Node MyRelationshipTypes.KNOWS, // ”KNOWS” Direction.OUTGOING ); // for ( Node friend : traverser ) { // Node ”name” System.out.println( friend.getProperty( "name" ) ); } Neo4j Wiki}

1 31 2 Trinity Morpheus Cypher Agent Smith Neo4j Wiki

private void findHackers( Node startNode ) Neo4j Wiki{ Traverser traverser = startNode.traverse( Order.BREADTH_FIRST, // StopEvaluator.END_OF_GRAPH, // Graph new ReturnableEvaluator() // { public boolean isReturnableNode( TraversalPosition currentPosition ) { Relationship rel = currentPosition.lastRelationshipTraversed(); if ( rel != null && rel.isType( MyRelationshipTypes.CODED_BY ) ) { return true; // “CODED_BY” } return false; // } }, // 2 MyRelationshipTypes.CODED_BY, Direction.OUTGOING, // MyRelationshipTypes.KNOWS, Direction.OUTGOING ); // for ( Node hacker : traverser ) { TraversalPosition position = traverser.currentPosition(); System.out.println( "At depth " + position.depth() + " => " + hacker.getProperty( "name" ) ); } ∴ At depth 4 => The Architect

Graph DB[Data Locality] [Local Search, Social Network] 2 [Transition] Web [Recommendation]

[Graph Problems] [Shortest Path] 2GraphDB Traversal Neo4jrb

Graph DB ” ” 10 ”Knows” Tables, Documents, Key/Value Model GraphDB Union, Intersection, Join

Graph DB[ ] Property Graph Index-Free Adjacency Graph Query = Graph Traversal Data Locality

Graph DBGraph GraphGraph DB Graph Traversal Graph DB Neo4j, Sones, InfoGrid, OrientDB, InﬁniteGraphTinker Pop Gremlin, Blueprints, Pipes, Rexster, Mutant

Neo4j[ ] HP Java AGPLv3 2003 24 8 2009 VC ACID Propety Graph Model / Gremlin Lucene

Neo4j[Language Binding - Framework] Python - Django Ruby - Ruby on Rails Clojure Scala Groovy - Grifﬁn / Grails Java - Spring Framework Ruby Ruby Java

Neo4j[Tools] Shell Shell Graph Traverse Indexing neo4j-server Neo4j REST API Admin tools Online BackUp Neoclipse Neo4j ↑ Batch Insert

Neo4j[ver. 1.2] 1.2 Neo4j Server REST API Admin Interface High Availability Kernel

sones[ ] HP C# AGPLv3 2011 VC ACID REST Interface Property Graph Model / Gremlin : Property Hyper Graph Graph Query Language(GQL)

sones [GQL] SQL Traversal Cheat Sheet Query• FROM User SELECT User.Friends.Friends.Name// aggregation• SELECT COUNT(User.Friends)• SELECT User.Friends.Random(2)• SELECT User.Friends.Name.Substring(2,5)

Orient DB

Orient DB[ ] HP Java Apache2.0 1997 C++ → Java Document-Graph DB ACID Shell / REST Interface Propety Graph Model / Gremlin

Orient DB [Document-Graph DB] [ ] Orient DB Object DB Key/Value Server Document DB// DATABASE OPENODatabaseDocumentTx db = new ODatabaseDocumentTx("remote:localhost/petshop").open("admin", "admin");// DocumentODocument doc = new ODocument(db, "Person");doc.field( "name", "Luke" );doc.field( "surname", "Skywalker" );doc.field( "city", new ODocument(db, "City").field("name","Rome").field("country","Italy") ); // Transactiondoc.save();db.close();

Orient DB [Document-Graph DB] OGraphVertex OGraphEdge OGraphElement ODocumentWrapper Document SQLSELECT FROM OGraphVertex WHERE outEdges CONTAINS ( label = knows )//7 ”knows”SELECT FROM OGraphVertex WHERE outEdges TRAVERSE(0,7,out,outEdges)( @class = OGraphEdge and label = knows )

Orient DB[Language Binding Using Binary Protocol] Java C PHP JRuby (Ruby: soon)[Language Binding Using REST Protocol] Python Java Script

InfoGrid

InfoGrid [ ] HP JAVA AGPLv3 ACID REST Interface MeshObject GraphMeshBase _GDB = StoreMeshBase.create(_MySQLStore);MeshObject _xkcd = _GDB.getMeshObjectLifecycleManager().createMeshObject();_xkcd.setProperty("Name", "xkcd");_xkcd.setProperty("Url", "http://www.xkcd.com");_xkcd.relate(_good)

Inﬁnite Graph

Inﬁnite Graph[ ] HP C++ Academic and Start Up 2010 6 Distributed Graph DB ↑Objectivity/DB: distributed database server

Graph DB: Data SQL LikeGraphDB License Language Protocol Gremlin Binding Model Query REST/ Property Ruby, Python, Neo4j AGPLv3 Java Yes Scala,... - JSON Graph REST/ Property sones AGPLv3 C# JSON Graph Yes - Yes (XML) (+Extend) REST/ Property PHP, Jruby,OrientDB Apache2.0 Java Yes Python, JS,... Yes JSON Graph Property REST/Info Grid AGPLv3 Java Graph? - - - JSON (MeshObject) Inﬁnite Property Product C++ - - - - Graph Graph

Graph DB[ ] Graph DB Neo4j Open Source Social Graph Software Not Ready Yet Graph DB Hypergtaph: PropertyGraph HyperGraph Pregel: bulk synchronous parallel model Distributed DB Google FlockDB: Distributed DB for storing adjancency lists Twitter

Tinker PopGraph GraphGraph DB Graph Traversal Graph DB Neo4j, Sones, InfoGrid, OrientDB, InﬁniteGraphTinker Pop Gremlin, Blueprints, Pipes, Rexster, Mutant

Tinker Pop

Tinker Pop[Tinker Pop] HP Property Graph Model GraphDB Blueprints: A Property Graph Model Interface Gremlin: A Graph Traversal Language Pipes: A Data Flow Framework using Process Graphs Rexster: A RESTful Graph Shell Mutant: A Poly-ScriptEngine ScriptEngine

Tinker Pop

Tinker Pop: BluePrints

BluePrints[ ] HP GraphDB ”JDBC”Property Graph Model GraphDB[Now] Tinker Graph: in-memory property graph model Sail: Open RDF Neo4j, Orient DB, sones, ...[Future] Redis Inﬁnite Graph, Dex

BluePrints GraphDBGraph graph = new Neo4jGraph("/tmp/graph/neo4j");// Graph graph = new OrientGraph("/tmp/graph/orientdb");Vertex a = graph.addVertex(null);Vertex b = graph.addVertex(null);a.setProperty("name","marko");b.setProperty("name","aaron");Edge e = graph.addEdge(null,a,b,"knows");e.setProperty("since",2010);graph.shutdown();

BluePrints Transactiongraph.startTransaction();try{ Vertex luca = graph.addVertex(null); luca.setProperty( "name", "Luca" ); Vertex marko = graph.addVertex(null); marko.setProperty( "name", "Marko" ); Edge lucaKnowsMarko = graph.addEdge(null, luca, marko,"knows"); graph.stopTransaction(Conclusion.SUCCESS);} catch( Exception e ) { graph.stopTransaction(Conclusion.FAILURE);}

Tinker Pop: Gremlin

Gremlin[ ] HPGremlin = Graph Programing LanguageBlueprints GraphDBShell GraphDB QueryJava + Groovy

GremlinProperty Graph Basic Graph Traversals

doryokujin$ ./gremlin.sh ,,,/ (o o)-----oOOo-(_)-oOOo-----gremlin> g = TinkerGraphFactory.createTinkerGraph()==>tinkergraph[vertices:6 edges:6] // 6 6gremlin> g.class==>class com.tinkerpop.blueprints.pgm.impls.tg.TinkerGraphgremlin> // gremlin> g.V==>v[3]==>v[2]...gremlin> // gremlin> g.E==>e[10][4-‐created-‐>5]==>e[7][1-‐knows-‐>2]==>e[9][1-‐created-‐>3]... Getting Srarted

gremlin> v = g.v(1) // id=1 ==>v[1]gremlin> v.keys() // ==>age==>namegremlin> v.values() // ==>29==>markogremlin> v.name + is + v.age + years old.==>marko is 29 years old.gremlin> // id=1, name=marko gremlin> v.outE==>e[7][1-‐knows-‐>2]==>e[9][1-‐created-‐>3]==>e[8][1-‐knows-‐>4]gremlin> // gremlin> v.outE.weight==>0.5==>0.4==>1.0 Getting Srarted

gremlin> // id=1 1.0gremlin> v.outE{it.weight < 1.0}.inV==>v[2]==>v[3]gremlin> // gremlin> list = [] gremlin> v.outE{it.weight < 1.0}.inV >> list==>v[2]==>v[3]gremlin> // list property mapsgremlin> list.collect{ it.map() }==>{name=vadas, age=27}==>{name=lop, lang=java}gremlin> // listgremlin> list.inE() ==>e[7][1-‐knows-‐>2]==>e[9][1-‐created-‐>3]... Getting Srarted

gremlin> list.inE{it.label==knows} // knows ==>e[7][1-‐knows-‐>2] gremlin> list.inE()[[label:knows]] // ==>e[7][1-‐knows-‐>2] gremlin> list.inE()[[label:knows]].outV.name // :name ==>marko Getting Srarted~20000ms: g.V.outE{it[label]==followed_by}.inV.outE{it[label]==followed_by}.inV.outE {it[label]==followed_by}.inV >>-‐1~9000ms: g.V.outE{it.label==followed_by}.inV.outE{it.label==followed_by}.inV.outE {it.label==followed_by}.inV >>-‐1~8500ms: g.V.outE{it.getLabel()==followed_by}.inV.outE{it.getLabel()==followed_by}.inV.outE {it.getLabel()==followed_by}.inV >>-‐1~6000ms: g.V.outE[[label:followed_by]].inV.outE[[label:followed_by]].inV.outE [[label:followed_by]].inV >>-‐1 ClosureFilterPipe vs. PropertyFIlterPipe

Tinker Pop: Pipes

Pipes[ ] HPPipes = Data Flow FrameworkPipes Graph Traversal 1 1Pipes ﬁltering, splitting, merging, traversing,...

Gremling:id-v(a)/outE[@label=knows]/inV/outE[@label=develops]/inV/@name Pipe pipe1 = new VertexEdgePipe(Step.OUT_EDGES); Pipe pipe2 = new LabelFilterPipe("knows", Filter.NOT_EQUALS); Pipe pipe3 = new EdgeVertexPipe(Step.IN_VERTEX); Pipe pipe4 = new VertexEdgePipe(Step.OUT_EDGES); Pipe pipe5 = new LabelFilterPipe("develops", Filter.NOT_EQUALS); Pipe pipe6 = new EdgeVertexPipe(Step.IN_VERTEX); Pipe pipe7 = new PropertyPipe("name"); Pipe pipeline = new Pipeline (pipe1,pipe2,pipe3,pipe4,pipe5,pipe6,pipe7); pipeline.setStarts(new SingleIterator(graph.getVertex("a")); for(String name : pipeline) { System.out.println(name); } A Graph Processing Stack

Pipes Pipespublic class NumCharsPipe extends AbstractPipe<String,Integer> { public Integer processNextStart() { String word = this.starts.next(); return word.length(); }} A Graph Processing Stack

Tinker Pop: Rexster

Rexster[ ] HPRexster = A RESTful Graph ShellBlueprints GraphDB RESTfulAPI (JSON)Gremlin

> http://localhost:8182/examplegraph/vertices/b{ "version":"0.1", "results": { "_type":"vertex", "_id":"b", "name":"aaron", "type":"person" }, "query_time":0.1537} A Graph Processing Stack// g:key-v(name,DARK STAR)[0]: Usin gGremlin Code> http://localhost:8182/gratefulgraph/traversals/gremlin?script=g:key-v%28%27name%27,%27DARK%20STAR%27%29[0]{ "results": [{ "_type":"vertex", "_id":"89", "name":"DARK STAR", "song_type":"original", "performances":219, "type":"song"} ], "query_time":6.753024, "success":true, "version"} Using Gremilin

Tinker Pop: Mutant

Mutant[ ] HPMutant = A Poly-ScriptEngine ScriptEngineJVM Script Engine

Mutant Consolemarko:~/software/mutant$ ./mutant.sh // oO ~~-‐____m(___m___~.___ MuTanT 0.1-‐SNAPSHOT_|__|__|__|__|__| [ ?h = help ][gremlin] gremlin 0.6-‐SNAPSHOT[Groovy] Groovy Scripting Engine 2.0[ruby] JSR 223 JRuby Engine 1.5.5[ECMAScript] Mozilla Rhino 1.6 release 2[AppleScript] AppleScriptEngine 1.0mutant[gremlin]> $x := 12[12]mutant[gremlin]> ?xmutant[AppleScript]> ?xmutant[Groovy]> $x12mutant[Groovy]> ?xmutant[ruby]> $x12mutant[ruby]> ?xmutant[ECMAScript]> $x12 Basic Examples

[ ] Graph DB Graph DB Graph Partitioning Pregel Neo4j

…※ Graph DB http://snap.stanford.edu/data/index.html

http://www.nogutetu.com	1796
http://graphdb.jp	1787
http://www.graph-database.org	1741
http://d.hatena.ne.jp	1641
http://fungoing.jp	561
http://paper.li	174
http://ts-technotes.blogspot.jp	154
http://www.techgig.com	106
http://webcache.googleusercontent.com	46
https://twitter.com	44
http://ts-technotes.blogspot.com	24
http://magwolic.tumblr.com	23
http://a-hum.unoke.pfu.co.jp	21
http://localhost:4000	19
http://us-w1.rockmelt.com	11
http://www.onlydoo.com	9
http://a0.twimg.com	8
http://a0.twimg.com	8
http://twitter.com	6
http://translate.googleusercontent.com	6
https://si0.twimg.com	5
http://askmt.tumblr.com	4
http://safe.tumblr.com	3
http://www.google.co.jp	2
url_unknown	1
http://tweetedtimes.com	1
http://www.iweb34.com	1
http://b.hatena.ne.jp	1
https://twimg0-a.akamaihd.net	1
http://www.bing.com	1

「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜

by Takahiro Inoue on Jan 23, 2011

Accessibility

Categories

Upload Details

Usage Rights

30 Embeds 8,205

Statistics

「GraphDB徹底入門」〜構造や仕組み理解から使いどころ・種々のGraphDBの比較まで幅広く〜 Presentation Transcript