NoSQL with Graphs

NoSQL with Graphs mining graphs for fun & proﬁt claudio martella NoSQLDay 2011Saturday, March 26, 2011

Outline Semantic Web Documents Tinkerpop GraphDBs Tools NoSQL Table Graphs O(1) Why Apps Query RDBMS Recommendation 2Saturday, March 26, 2011

Who am I? • PhD in Distributed Graphs @ UniBZ • Analyst @ TIS Innovation Park • Topics: Data / Text Mining with Graphs • Technology: Hadoop, NoSQL, GraphDBs • Writing Grafﬁti 3Saturday, March 26, 2011

Surrounded by graphs • the Web Graph • Semantic Web • Social Networks • Natural Sciences • GIS 4Saturday, March 26, 2011

Property Graph • A Graph is composed by Vertices and Edges • Vertices are connected by Edges • An Edge has a Label and Direction • Edges and Vertices have Properties 5Saturday, March 26, 2011

GraphDB belongs to NoSQL belongs to belongs to likes Hadoop Graffiti works with name: claudio author Me surname: martella email: claudio.martella@gmail.com works at studies at TIS UniBZ Who am I? 6Saturday, March 26, 2011

A graph in RDBMS ID Name Follower Followee 1 Claudio 1 2 2 Cirpo 1 3 3 Okram 1 4 4 Spinoza 2 5 ... ... ... ... 7Saturday, March 26, 2011

BTree Index 101 • Lookup costs Log(N) • Where N is the global size of the data structure • Updating the index is Cirpo Claudio Okram Spinoza also not for free 8Saturday, March 26, 2011

A lookup (RDBMS) I Name Fr Fe • Look for Claudio’s ID 1 Claudio 1 2 [ Log(N) ] 2 Cirpo 1 3 • Look for K Followees [ Log(N) ] 3 Okram 1 4 • Get their names 4 Spinoza 2 5 [ K*Log(N) ] ... ... ... ... 9Saturday, March 26, 2011

A graph in NoSQL ID F1 F2 F3 ... Cirpo ... ... ... ... Claudio Cirpo Okram Spinoza ... Okram ... ... ... ... Spinoza ... ... ... ... ... ... ... ... ... 10Saturday, March 26, 2011

A lookup (NoSQL) ID F1 F2 F3 ... Cirpo ... ... ... ... • Look for Claudio’s ID [ Log(N) ] Claudio ... ... ... ... • Look for Followees Okram ... ... ... ... [ O(K) ] Spinoza ... ... ... ... ... ... ... ... ... 11Saturday, March 26, 2011

A graph in GraphDB name: Cirpo 2 follows follows 1 3 name: Claudio name: Okram follows 4 name: Spinoza 12Saturday, March 26, 2011

A lookup (Graph) name: Cirpo 2 • Look for Claudio’s ID follows follows [ Log(N) ] 1 3 name: Claudio • Look for Followees name: Okram [ O(K) ] follows 4 name: Spinoza 13Saturday, March 26, 2011

What about Friends (of Friends)*? 14Saturday, March 26, 2011

A benchmark Depth RDBMS Graph • 1 Million Vertices 1 100ms 30ms • 4 Million Edges 2 1000ms 500ms • Scale-Free Topology 3 10000ms 3000ms • Postgres VS Neo4J 4 100000ms 50000ms • Both Hash and BTree 5 N/A 100000ms Ref: http://markorodriguez.com/2011/02/18/mysql-vs-neo4j-on-a-large-scale-graph-traversal/ 15Saturday, March 26, 2011

A benchmark DB # Time RDBMS 1K 2000ms • 50 friends on average Graph 1K 2ms • Look if there’s a path connecting two people Graph 1M 2ms RDBMS 1M N/A Ref: http://www.slideshare.net/thobe/nosqleu-graph-databases-and-neo4j 16Saturday, March 26, 2011

A Graph Database allows O(1) access to adjacent Vertices Ref: The Graph Traversal Pattern: Marko A. Rodriguez and Peter Neubauer 17Saturday, March 26, 2011

Example: Queries genre Action director Ocean 11 genre Steven Soderbergh director genre Thriller actor Ocean 12 genre director genre actor Ocean 13 genre Crime genre actor Brad Pitt producer The Departed genre actor genre Drama Se7en genre 18Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi likes Graph Runner tagged Claudio likes tagged Trilogy The Lord of the Graphs tagged likes tagged Adventure Caprazzi likes Javatar likes tagged Geeky likes tagged Cirpo likes PHP I love You tagged Boring 22Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi likes Graph Runner tagged Claudio likes tagged Adventure The Lord of the Graphs tagged likes Trilogy tagged Caprazzi likes Javatar likes tagged Geeky likes tagged Cirpo likes PHP I love You tagged Boring 24Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi likes Graph Runner tagged Claudio likes tagged Adventure The Lord of the Graphs tagged likes Trilogy tagged Caprazzi likes Javatar likes tagged Geeky likes tagged Cirpo likes PHP I love You tagged Boring 25Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi Graph Runner likes Claudio tagged likes tagged Adventure likes The Lord of the Graphs tagged Cirpo likes tagged Trilogy likes Caprazzi Javatar likes tagged Geeky tagged likes PHP I love You tagged Boring 28Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi likes Graph Runner Claudio likes tagged Trilogy tagged likes The Lord of the Graphs Cirpo tagged likes Adventure tagged likes likes Javatar tagged Caprazzi likes tagged Geeky PHP I love You tagged Boring 29Saturday, March 26, 2011

Example: Recommendations tagged Sci-Fi likes Graph Runner Claudio likes tagged Trilogy tagged likes The Lord of the Graphs Cirpo tagged likes Adventure tagged likes likes Javatar tagged Caprazzi likes tagged Geeky PHP I love You tagged Boring 30Saturday, March 26, 2011

Graph Mining How are they connected? Ref: Programming the Semantic Web - O’Reilly 31Saturday, March 26, 2011

Graph Mining Ref: Programming the Semantic Web - O’Reilly 32Saturday, March 26, 2011

Graph Mining 33Saturday, March 26, 2011

Other Applications • Community Analysis • Fraud Detection • Planning • Text Processing • Reasoning 34Saturday, March 26, 2011

as you can’t get rid of logicians 35Saturday, March 26, 2011

there’s an SQL also for Graphs 36Saturday, March 26, 2011

Triplestores Scientology advocate Katie Holmes married Hollywood lives Tom Cruise born actor July 3, 1962 Top Gun 37Saturday, March 26, 2011

Triplestores Subject Predicate Object Tom Cruise actor Top Gun Tom Cruise married Katie Holmes Tom Cruise advocate Scientology Tom Cruise lives Hollywood Tom Cruise born July 3, 1962 38Saturday, March 26, 2011

SPARQL PREFIX ged: <http://www.daml.org/2001/01/gedcom/gedcom#> SELECT ?name ?marriedOn FROM <http://www.daml.org/2001/01/gedcom/royal92.daml> WHERE { ?royal ged:title "Princess". ?royal ged:name ?name. ?royal ged:spouseIn ?family. ?family ged:marriage ?marriage. ?marriage ged:date ?marriedOn. } ORDER BY ASC [?name] 39Saturday, March 26, 2011

what if Internet was your GraphDB? 40Saturday, March 26, 2011

41Saturday, March 26, 2011

what about a NoSPARQL? 42Saturday, March 26, 2011

Tinkerpop 43Saturday, March 26, 2011

• Blueprints is the like the JDBC of the graph database community. • Provides a Java-based interface API for the property graph data model. Graph,Vertex, Edge, Index. • Provides implementations of the interfaces for TinkerGraph, Neo4j, OrientDB, Sails (e.g. AllegroSail, Neo4jSail), and soon (hopefully) others such as InﬁniteGraph, InfoGrid, Sones, and HyperGraphDB 44Saturday, March 26, 2011

• A dataﬂow framework with support for Blueprints-based graph processing. • Provides a collection of “pipes” (implement Iterable and Iterator) ✴ Filters: ComparisonFilterPipe, RandomFilterPipe, etc. ✴ Traversal:VertexEdgePipe, EdgeVertexPipe, PropertyPipe, etc. ✴ Splitting/Merging: CopySplitPipe, RobinMergePipe, etc. ✴ Logic: OrPipe, AndPipe, etc. 45Saturday, March 26, 2011

• A Turing-complete, graph-based programming language that compiles Gremlin syntax down to Pipes (implements JSR 223). • Builds on top of Groovy • Support various language constructs: :=, foreach, while, repeat, if/else, function and path deﬁnitions, etc. An example of “Amazon’s” recommender: m = [:] g.v(1).outE(purchased).inV.inE(purchased).outV.groupCount(m); m.sort{ a,b -> a.value <=> b.value } 46Saturday, March 26, 2011

• Allows Blueprints graphs to be exposed through a RESTful API (HTTP) • Supports stored traversals written in raw Pipes or Gremlin. • Supports adhoc traversals represented in Gremlin. • Provides “helper classes” for performing search-, score-, and rank-based traversal algorithms—in concert, support for recommendation. 47Saturday, March 26, 2011

Sample Stack • HTTP Request arrives • Converts REST to Gremlin • Gremlin “compiles” to Pipes • Pipes makes Blueprints calls • Store provides the data 48Saturday, March 26, 2011

Neo4J • Engine: Graph • License: AGPLv3 • Language: Java • Transactions: ACID • Distributed: HA, Master-Slave Cache Sharding, Domain-Speciﬁc • Features: Embeddable, REST, many plugins 49Saturday, March 26, 2011

OrientDB • Engine: Document-Graph • License: Apache 2.0 • Language: Java • Transactions: ACID • Distributed: HA through Replication • Features: Embeddable, REST, SQL-like 50Saturday, March 26, 2011

HypergraphDB • Engine: HyperGraph • License: LGPL • Language: Java • Transactions: ACID • Distributed: P2P distribution and replication • Features: Hyperedges, Java OODB, storage on BerkeleyDB 51Saturday, March 26, 2011

InﬁniteGraph • Engine: Graph • License: Commercial • Language: Java • Transactions: ACID • Distributed: Graph Partitioning, Federation on Objectivity • Features: Distributed lock management, scales to Exabytes 52Saturday, March 26, 2011

Where do I go now? Tinkerpop: http://www.tinkerpop.com Neo4J: http://neo4j.org OrientDB: http://www.orientechnologies.com/orient-db.htm InfoGrid: http://infogrid.org InﬁniteGraph: http://www.inﬁnitegraph.com Sones: http://developers.sones.de AllegroGraph: http://www.franz.com/agraph/allegrograph HypergraphDB: http://www.kobrix.com/hgdb.jsp 53Saturday, March 26, 2011

Questions? claudio.martella@gmail.com http://blog.acaro.org http://github.com/claudiomartella/ @claudiomartella http://joind.in/2946Saturday, March 26, 2011

http://blog.acaro.org	1975
http://blog.nosqlfan.com	796
http://www.graph-database.org	229
http://feed.feedsky.com	46
http://www.kuqin.com	30
http://paper.li	14
http://www.zhuaxia.com	13
http://xianguo.com	10
http://reader.youdao.com	9
http://twitter.com	6
http://allenjoe.net	6
http://www.slideshare.net	4
http://www.chfstudio.com	4
url_unknown	2
http://trunk.ly	2
http://blog.acaro.org.	1
http://translate.googleusercontent.com	1
https://twitter.com	1
http://tweetedtimes.com	1
http://graph-databases.org	1
http://abtasty.com	1

NoSQL with Graphs

by Claudio Martella

Accessibility

Categories

Upload Details

Usage Rights

21 Embeds 3,152

Statistics

NoSQL with Graphs Presentation Transcript