Designing and Building a Graph Database Application - Ian Robinson (Neo Technology)

567 views
478 views

Published on

Presented at JAX London

In this session we'll look at some of the design and implementation strategies you can employ when building a Neo4j-based graph database solution, including architectural choices, data modelling, and testing.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
567
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
23
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Designing and Building a Graph Database Application - Ian Robinson (Neo Technology)

  1. 1. Twi$er:  @ianSrobinson  #neo4j   Designing  and  Building  a  Graph   Database  Applica5on  
  2. 2. Roadmap   •  Complex  data   •  Designing  a  graph  data  model  and  queries   •  Tes@ng  
  3. 3. Data  Complexity   complexity = f(size , semi-structure, connectedness)
  4. 4. Social  Network  
  5. 5. Network  Impact  Analysis  
  6. 6. Route  Finding  
  7. 7. Recommenda@ons  
  8. 8. Logis@cs  
  9. 9. Access  Control  
  10. 10. Fraud  Analysis  
  11. 11. Securi@es  and  Debt   Image:  orgnet.com  
  12. 12. Graphs  Are  Everywhere  
  13. 13. Graph  Databases   •  Store   •  Manage   •  Query   data  
  14. 14. Neo4j  is  a  Graph  Database  
  15. 15. Labeled  Property  Graph  
  16. 16. Designing  a  Graph  Model  
  17. 17. Models   Purposeful  abstrac@on  of  a  domain  designed  to   sa@sfy  par@cular  applica@on/end-­‐user  goals   Images:  en.wikipedia.org  
  18. 18. Applica@on/End-­‐User  Goals   As  an  emp   loyee   I  want  to  k now  who  i n  the  comp has  similar any    skills  to  m   e   So  that  we  can  excha nge  knowl edge  
  19. 19. Ques@ons  To  Ask  of  the  Domain   As  an  emp loyee     I  want  to  k now  who  i n  the  co has  similar   skills  to  me mpany       So  that  we  can  excha nge  knowle dge   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?  
  20. 20. Iden@fy  En@@es   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?     Person   Company   Skill  
  21. 21. Iden@fy  Rela@onships  Between  En@@es   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?     Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill  
  22. 22. Convert  to  Cypher  Paths   Rela@onship   Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill   Label   (:Person)-[:WORKS_FOR]->(:Company), (:Person)-[:HAS_SKILL]->(:Skill)
  23. 23. Consolidate  Paths   (:Person)-[:WORKS_FOR]->(:Company), (:Person)-[:HAS_SKILL]->(:Skill) (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  24. 24. Candidate  Data  Model   (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  25. 25. Express  Ques@on  as  Graph  Pa$ern   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?  
  26. 26. Cypher  Query   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  27. 27. Graph  Pa$ern   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  28. 28. Anchor  Pa$ern  in  Graph   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC Search  nodes  labeled   ‘Person’,  matching  on   ‘name’  property  
  29. 29. Create  Projec@on  of  Results   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  30. 30. First  Match  
  31. 31. Second  Match  
  32. 32. Third  Match  
  33. 33. Running  the  Query   +-----------------------------------+ | name | score | skills | +-----------------------------------+ | "Lucy" | 2 | ["Java","Neo4j"] | | "Bill" | 1 | ["Neo4j"] | +-----------------------------------+ 2 rows
  34. 34. From  User  Story  to  Model  and  Query   As  an  emp loyee     I  want  to  k now  who  i n  the  co has  similar   skills  to  me mpany       So  that  we  can  excha nge  knowle dge   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC ? Which  people,  who  work  for  the  same   company  as  me,  have  similar  skills  to  me? Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  35. 35. Tes@ng  
  36. 36. Why  Test?   •  Ensure  model  is  fit  for  queries   –  Rapid  feedback   •  Ensure  correctness  of  queries   •  Document  your  understanding  of  your  domain   –  Including  corner  cases  and  excep@ons   •  Provide  a  regression  test  suite   –  Allows  you  to  change  and  evolve  model  and   queries  
  37. 37. Method   •  Develop  queries,  or  classes  that  encapsulate   queries,  using  unit  tests   •  Use  small,  well-­‐understood  datasets  in  each  test   –  Create  data  in  test  setup   –  Test  dataset  expresses  your  understanding  of  (part  of)   the  domain   •  Inject  in-­‐memory  graph  database  (or  Cypher   engine)  into  object  under  test   •  The  exact  strategy  you  use  depends  on  your   applica@on  architecture…  
  38. 38. Applica@on  Architectures   •  Embedded   •  Server   •  Server  with  Extensions  
  39. 39. Applica@on  Architectures   •  Embedded   –  Host  in  Java  process   –  Access  to  Java  APIs   •  Server   •  Server  with  Extensions   Applica@on   Java  APIs  
  40. 40. Applica@on  Architectures   •  Embedded   •  Server   –  HTTP/JSON  interface   –  Server  wraps  embedded   instance   •  Server  with  Extensions   Applica@on   REST  Client   Write  LB   REST  API   REST  API   Read  LB   REST  API  
  41. 41. Applica@on  Architectures   •  Embedded   •  Server   •  Server  with  Extensions   REST  API   –  Execute  complex  logic  on  server   –  Control  HTTP  request/response  format   Extensions  
  42. 42. Embedded  Example   •  Company  social  network   •  Find  colleagues  with  similar  skills   •  Encapsulate  query  in  a  ColleagueFinder
  43. 43. Unit  Test  Fixture   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  44. 44. Create  Database   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  45. 45. Populate  Graph   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  46. 46. Create  Object  Under  Test   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } } Inject     Execu@onEngine  
  47. 47. ImpermanentGraphDatabase   •  In-­‐memory   •  For  tes@ng  only,  not  produc@on!   <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j-kernel</artifactId> <version>${project.version}</version> <type>test-jar</type> <scope>test</scope> </dependency>
  48. 48. Create  Sample  Data   public static void populate( GraphDatabaseService db ) { ExecutionEngine engine = new ExecutionEngine( db ); String cypher = "CREATE ian:Person VALUES {name:'Ian'},n" + " bill:Person VALUES {name:'Bill'},n" + " lucy:Person VALUES {name:'Lucy'},n" + " acme:Company VALUES {name:'Acme'},n" + // Cypher continues... " " " " (bill)-[:HAS_SKILL]->(neo4j),n" + (bill)-[:HAS_SKILL]->(ruby),n" + (lucy)-[:HAS_SKILL]->(java),n" + (lucy)-[:HAS_SKILL]->(neo4j)"; engine.execute( cypher ); }
  49. 49. Unit  Test   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  50. 50. Execute  Object  Under  Test   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  51. 51. Assert  Results   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  52. 52. Ensure  No  More  Results   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  53. 53. ColleagueFinder   public class ColleagueFinder { private final ExecutionEngine executionEngine; public ColleagueFinder( ExecutionEngine executionEngine ) { this.executionEngine = executionEngine; } public Iterator<Map<String, Object>> findColleaguesFor( String name ) { ... } }
  54. 54. Inject  Execu@onEngine   public class ColleagueFinder { private final ExecutionEngine executionEngine; public ColleagueFinder( ExecutionEngine executionEngine ) { this.executionEngine = executionEngine; } public Iterator<Map<String, Object>> findColleaguesFor( String name ) { ... } }
  55. 55. findColleaguesFor()  Method   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  56. 56. Cypher  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  57. 57. Parameterized  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  58. 58. Execute  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  59. 59. Server  Extension  Example   •  Same  data  mode  and  query  as  before   •  This  @me,  we’ll  host  ColleagueFinder  in  a   server  extension  
  60. 60. Server  Extension   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  61. 61. JAX-­‐RS  Annota@ons   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  62. 62. Map  HTTP  Request  to  Object  +  Method   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } } GET   /similar-­‐skills   /Ian  
  63. 63. CypherExecutor  Injected  by  Server   Ensures   Execu@onEngine   reused  across   resource  instances   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  64. 64. Generate  and  Format  Response   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  65. 65. Extension  Test  Fixture   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  66. 66. Build  and  Configure  Server   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  67. 67. Start  Server   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  68. 68. Populate  Database   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  69. 69. CommunityServerBuilder   •  Programma@c  configura@on   <dependency> <groupId>org.neo4j.app</groupId> <artifactId>neo4j-server</artifactId> <version>${project.version}</version> <type>test-jar</type> </dependency>
  70. 70. Tes@ng  Extension  Using  HTTP  Client   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  71. 71. Create  HTTP  Client   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  72. 72. Issue  Request   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  73. 73. Parse  Response   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  74. 74. Assert  Results   ... assertEquals( 200, response.getStatus() ); assertEquals( MediaType.APPLICATION_JSON, response.getHeaders().get( "Content-Type" ).get( 0 ) ); assertEquals( "Lucy", results.get( 0 ).get( "name" ) ); assertThat( (Iterable<String>) results.get( 0 ).get( "skills" ), hasItems( "Java", "Neo4j" ) ); }
  75. 75. ts gy en lo i m no pl h m ec Co eo T N of Graph h Databases Thank  you   Twi$er:  @ianSrobinson   #neo4j     Neo4j  User  Group   30th  Oct   Skillsma$er                       Ian Robinson, Jim Webber & Emil Eifrem github.com/iansrobinson/neo4j-­‐good-­‐prac@ces    

×