Big Data & Cloud | An Introduction to Neo4j (and Doctor Who) | Ian Robinson

1,062 views
959 views

Published on

2011-11-01 | 10:40 AM - 11:40 AM
Doctor Who is the world’s longest running science-fiction TV series. Battling daleks, cybermen and sontarans, and always accompanied by his trusted human companions, the last Timelord has saved earth from destruction more times than you’ve cursed Maven. Neo4j is the world’s leading open source graph database. Designed to interrogate densely connected data with lightning speed, it lets you traverse millions of nodes in a fraction of the time it takes to run a multi-join SQLquery. When these two meet, the result is an entertaining introduction to the complex history of a complex hero, and a rapid survey of the elegant APIs of a delightfully simple graph database. Armed only with a data store packed full of geeky Doctor Who facts, by the end of this session we’ll have you answering questions of the Doctor Who universe like a die-hard fan.

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,062
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
17
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Big Data & Cloud | An Introduction to Neo4j (and Doctor Who) | Ian Robinson

  1. 1. An  Introduc+on  to  Doctor  Who   (and  Neo4j)     @iansrobinson   #neo4j  
  2. 2. Neo4j  is  a  Graph  Database   #neo4j  
  3. 3. Property  Graph   #neo4j  
  4. 4. Property  Graph   #neo4j  
  5. 5. Neo4j   #neo4j  
  6. 6. #neo4j  
  7. 7. 32  billion  nodes  32  billion  rela+onships  64  billion  proper+es  
  8. 8. #neo4j  
  9. 9. Community   Advanced   Enterprise  
  10. 10. stole   companion   from   loves   loves   appeared     enemy   in   companion   appeared     in   appeared     in   enemy   enemy  appeared     appeared     A  Good  Man   in   in   Goes  to  War   Victory  of     the  Daleks   appeared     in  
  11. 11. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho");
  12. 12. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho"); Node theDoctor = db.createNode(); theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen");
  13. 13. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho"); Node theDoctor = db.createNode(); enemy   theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen"); enemy   theDoctor.createRelationshipTo(daleks, DynamicRelationshipType.withName("enemy")); theDoctor.createRelationshipTo(cybermen, DynamicRelationshipType.withName("enemy"));
  14. 14. GraphDatabaseService db = new EmbeddedGraphDatabase("/data/drwho");Transaction tx = db.beginTx();try { Node theDoctor = db.createNode(); enemy   theDoctor.setProperty("name", "The Doctor"); Node daleks = db.createNode(); daleks.setProperty("name", "Daleks"); Node cybermen = db.createNode(); cybermen.setProperty("name", "Cybermen"); enemy   theDoctor.createRelationshipTo(daleks, DynamicRelationshipType.withName("enemy")); theDoctor.createRelationshipTo(cybermen, DynamicRelationshipType.withName("enemy")); tx.success();} finally { tx.finish();}
  15. 15. How  do  I  query  the  data?  hOp://opfm.jpl.nasa.gov/   hOp://news.xinhuanet.com   #neo4j  
  16. 16. Dalek  Props   hOp://www.dalek6388.co.uk/   #neo4j  
  17. 17. species:Dalek   APPEARED_IN   !tle:Power  of     the  Daleks   USED_IN   props:Daleks  MEMBER_OF   MEMBER_OF   name:Dalek  Six-­‐5   name:Dalek  1   name:Dalek  2   name:Dalek  7   type:shoulders   type:skirt  
  18. 18. name:Dalek  1   name:Dalek  2   name:Dalek  7   name:Dalek  Six-­‐5  ORIGINAL_PROP   ORIGINAL_PROP   ORIGINAL_PROP   ORIGINAL_PROP   name:Dalek  6   name:Dalek  5  
  19. 19. !tle:The   !tle:Power  of     !tle:The   Dalek   the  Daleks   Daleks   Invasion  of   Earth   name:   name:Dalek  1   name:Dalek  2   name:Dalek  7   Dalek  Six-­‐5  name:Dalek  Two-­‐1   name:Dalek  One-­‐5   name:Dalek  Six-­‐7  
  20. 20. Supply  Chain  Traceability   #neo4j  
  21. 21. Traversal  Framework  •  Visits  (and  returns)  nodes  based  on  traversal   descrip+on  •  Powerful;  can  customize:   –  Rela+onships  followed   –  Branch  selec+on  policy   –  Node/rela+onship  uniqueness  constraints   –  Path  evaluators   #neo4j  
  22. 22. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Description
  23. 23. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Index Lookup
  24. 24. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Branching
  25. 25. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Relationships
  26. 26. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Evaluator
  27. 27. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Uniqueness
  28. 28. Node theDaleks = database.index().forNodes("species").get("name", "Dalek").getSingle();Traverser traverser = Traversal.description() .depthFirst() .relationships(Rels.APPEARED_IN, Direction.OUTGOING) .relationships(Rels.USED_IN, Direction.INCOMING) .relationships(Rels.MEMBER_OF, Direction.INCOMING) .relationships(Rels.COMPOSED_OF, Direction.OUTGOING) .relationships(Rels.ORIGINAL_PROP, Direction.OUTGOING) .evaluator(new Evaluator() { @Override public Evaluation evaluate(Path path) { if (path.lastRelationship() != null && path.lastRelationship().isType(Rels.ORIGINAL_PROP)){ return Evaluation.INCLUDE_AND_PRUNE; } return Evaluation.EXCLUDE_AND_CONTINUE; } }) .uniqueness(Uniqueness.NONE) .traverse(theDaleks); Traverse
  29. 29. Iterable<Path>  path.startNode() path.endNode() +tle:Power  of     species:Dalek   the  Daleks   props:Daleks   name:Dalek  7   type:skirt   name:Dalek  7   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   +tle:Power  of     species:Daleks   the  Daleks   props:Daleks   name:Dalek  7   type:shoulder   name:Dalek  7   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   +tle:Power  of     species:Daleks   the  Daleks   props:Daleks   name:Dalek  Six-­‐5   type:skirt   name:Dalek  5   APPEARED_IN   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP  
  30. 30. Cypher  •  Declara+ve  graph  pa8ern  matching  language   –  Humane  regular  expressions  for  graphs  •  New  in  1.4   –  Con+nuing,  rapid  feature  development  •  Supports  queries   –  Including  aggrega+on,  ordering  and  limits   –  Muta+ng  opera+ons  coming  in  1.6   #neo4j  
  31. 31. Cypher Querystart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  32. 32. Index Lookupstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  33. 33. Match Nodes & Relationshipsstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  34. 34. match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop     APPEARED_IN   daleks   episode   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   part   originalprop  
  35. 35. Return Valuesstart  daleks=node:species(species=Dalek)    match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            part-­‐[:ORIGINAL_PROP]-­‐>originalprop  return  originalprop.name,  part.type,  count(episode.Qtle)    order  by  count(episode.Qtle)  desc    limit  1
  36. 36. In CodeCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  37. 37. SetupCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  38. 38. Parse & ExecuteCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  39. 39. ResultsCypherParser  parser  =  new  CypherParser();  ExecuQonEngine  engine  =  new  ExecuQonEngine(universe.getDatabase());        String  cql  =  "start  daleks=node:species(species=Dalek)"          +  "  match  daleks-­‐[:APPEARED_IN]-­‐>episode<-­‐[:USED_IN]-­‐()<-­‐[:MEMBER_OF]-­‐()"          +  "-­‐[:COMPOSED_OF]-­‐>part-­‐[:ORIGINAL_PROP]-­‐>originalprop"          +  "  return  originalprop.name,  part.type,  count(episode.Qtle)"          +  "  order  by  count(episode.Qtle)  desc  limit  1";    Query  query  =  parser.parse(cql);  ExecuQonResult  result  =  engine.execute(query);    String  originalProp  =  result.javaColumnAs("originalprop.name").next().toString();  String  part  =  result.javaColumnAs("part.type").next().toString();  int  episodeCount  =  (Integer)  result.javaColumnAs("count(episode.Qtle)").next();
  40. 40. In Webadmin
  41. 41. The  Hardest  Working  Prop  Part   Dalek  One’s  shoulders   hOp://www.dalek6388.co.uk/   #neo4j  
  42. 42. Download  hOp://neo4j.org/download/        hOp://www.springsource.org/spring-­‐data/neo4j   #neo4j  
  43. 43. Tutorial  hOps://github.com/jimwebber/neo4j-­‐tutorial   #neo4j  
  44. 44. Dematerialize…   @iansrobinson  ian.robinson@neotechnology.com     #neo4j  
  45. 45. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  46. 46. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  47. 47. 1.  Subtract:  Select  rectangle  then  3  black   triangles  2.  Union:  All    
  48. 48. match  (daleks)-­‐[:APPEARED_IN]-­‐>(episode)<-­‐[:USED_IN]-­‐                          ()<-­‐[:MEMBER_OF]-­‐()-­‐[:COMPOSED_OF]-­‐>                            (part)-­‐[:ORIGINAL_PROP]-­‐>(originalprop)     APPEARED_IN   daleks   episode   USED_IN   MEMBER_OF   COMPOSED_OF   ORIGINAL_PROP   part   originalprop  

×