Big Data & Cloud | An Introduction to Neo4j (and Doctor Who) | Ian Robinson
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Big Data & Cloud | An Introduction to Neo4j (and Doctor Who) | Ian Robinson

on

  • 992 views

2011-11-01 | 10:40 AM - 11:40 AM...

2011-11-01 | 10:40 AM - 11:40 AM
Doctor Who is the world’s longest running science-fiction TV series. Battling daleks, cybermen and sontarans, and always accompanied by his trusted human companions, the last Timelord has saved earth from destruction more times than you’ve cursed Maven. Neo4j is the world’s leading open source graph database. Designed to interrogate densely connected data with lightning speed, it lets you traverse millions of nodes in a fraction of the time it takes to run a multi-join SQLquery. When these two meet, the result is an entertaining introduction to the complex history of a complex hero, and a rapid survey of the elegant APIs of a delightfully simple graph database. Armed only with a data store packed full of geeky Doctor Who facts, by the end of this session we’ll have you answering questions of the Doctor Who universe like a die-hard fan.

Statistics

Views

Total Views
992
Views on SlideShare
981
Embed Views
11

Actions

Likes
0
Downloads
14
Comments
0

2 Embeds 11

http://lanyrd.com 10
http://tubemote.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Big Data & Cloud | An Introduction to Neo4j (and Doctor Who) | Ian Robinson Presentation Transcript

  • 1. An  Introduc+on  to  Doctor  Who   (and  Neo4j)     @iansrobinson   #neo4j  
  • 2. Neo4j  is  a  Graph  Database   #neo4j  
  • 3. Property  Graph   #neo4j  
  • 4. Property  Graph   #neo4j  
  • 5. Neo4j   #neo4j  
  • 6. #neo4j  
  • 7. 32  billion  nodes  32  billion  rela+onships  64  billion  proper+es  
  • 8. #neo4j  
  • 9. Community   Advanced   Enterprise  
  • 10. stole   companion   from   loves   loves   appeared     enemy   in   companion   appeared     in   appeared     in   enemy   enemy  appeared     appeared     A  Good  Man   in   in   Goes  to  War   Victory  of     the  Daleks   appeared     in  
  • 11. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho");
  • 12. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho"); Node theDoctor = db.createNode(); theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen");
  • 13. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho"); Node theDoctor = db.createNode(); enemy   theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen"); enemy   theDoctor.createRelationshipTo(daleks, DynamicRelationshipType.withName("enemy")); theDoctor.createRelationshipTo(cybermen, DynamicRelationshipType.withName("enemy"));
  • 14. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho");Transaction tx = db.beginTx();try { Node theDoctor = db.createNode(); enemy   theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen"); enemy   theDoctor.createRelationshipTo(daleks, DynamicRelationshipType.withName("enemy")); theDoctor.createRelationshipTo(cybermen, DynamicRelationshipType.withName("enemy")); tx.success();} finally { tx.finish();}
  • 15. How  do  I  query  the  data?  hOp://opfm.jpl.nasa.gov/   hOp://news.xinhuanet.com   #neo4j  
  • 16. Dalek  Props   hOp://www.dalek6388.co.uk/   #neo4j  
  • 17. species:Dalek   APPEARED_IN   !tle:Power  of     the  Daleks   USED_IN   props:Daleks  MEMBER_OF   MEMBER_OF   name:Dalek  Six-­‐5   name:Dalek  1   name:Dalek  2   name:Dalek  7   type:shoulders   type:skirt  
  • 18. name:Dalek  1   name:Dalek  2   name:Dalek  7   name:Dalek  Six-­‐5  ORIGINAL_PROP   ORIGINAL_PROP   ORIGINAL_PROP   ORIGINAL_PROP   name:Dalek  6   name:Dalek  5  
  • 19. !tle:The   !tle:Power  of     !tle:The   Dalek   the  Daleks   Daleks   Invasion  of   Earth   name:   name:Dalek  1   name:Dalek  2   name:Dalek  7   Dalek  Six-­‐5  name:Dalek  Two-­‐1   name:Dalek  One-­‐5   name:Dalek  Six-­‐7  
  • 20. Supply  Chain  Traceability   #neo4j  
  • 21. Traversal  Framework  •  Visits  (and  returns)  nodes  based  on  traversal   descrip+on  •  Powerful;  can  customize:   –  Rela+onships  followed   –  Branch  selec+on  policy   –  Node/rela+onship  uniqueness  constraints   –  Path  evaluators   #neo4j  
  • 22. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Description
  • 23. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Index Lookup
  • 24. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Branching
  • 25. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Relationships
  • 26. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Evaluator
  • 27. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Uniqueness
  • 28. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Traverse
  • 29. Iterable<Path>  path.startNode() path.endNode() +tle:Power  of     species:Dalek   the  Daleks   props:Daleks   name:Dalek  7   type:skirt   name:Dalek  7   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   +tle:Power  of     species:Daleks   the  Daleks   props:Daleks   name:Dalek  7   type:shoulder   name:Dalek  7   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   +tle:Power  of     species:Daleks   the  Daleks   props:Daleks   name:Dalek  Six-­‐5   type:skirt   name:Dalek  5   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP  
  • 30. Cypher  •  Declara+ve  graph  pa8ern  matching  language   –  Humane  regular  expressions  for  graphs  •  New  in  1.4   –  Con+nuing,  rapid  feature  development  •  Supports  queries   –  Including  aggrega+on,  ordering  and  limits   –  Muta+ng  opera+ons  coming  in  1.6   #neo4j  
  • 31. Cypher Querystart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  • 32. Index Lookupstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  • 33. Match Nodes & Relationshipsstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  • 34. match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop     APPEARED_IN   daleks   episode   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   part   originalprop  
  • 35. Return Valuesstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  • 36. In CodeCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  • 37. SetupCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  • 38. Parse & ExecuteCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  • 39. ResultsCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  • 40. In Webadmin
  • 41. The  Hardest  Working  Prop  Part   Dalek  One’s  shoulders   hOp://www.dalek6388.co.uk/   #neo4j  
  • 42. Download  hOp://neo4j.org/download/        hOp://www.springsource.org/spring-­‐data/neo4j   #neo4j  
  • 43. Tutorial  hOps://github.com/jimwebber/neo4j-­‐tutorial   #neo4j  
  • 44. Dematerialize…   @iansrobinson  ian.robinson@neotechnology.com     #neo4j  
  • 45. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  • 46. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  • 47. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  • 48. match  (daleks)-­‐[:APPEARED_IN]-­‐>(episode)<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            (part)-­‐[:ORIGINAL_PROP]-­‐>(originalprop)     APPEARED_IN   daleks   episode   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   part   originalprop