• Like

Designing and Building a Graph Database Application – Architectural Choices, Data Modeling, and Testing - Ian Robinson @ GraphConnect NY 2013

  • 4,055 views
Uploaded on

Ian closely looks at design and implementation strategies you can employ when building a Neo4j-based graph database solution, including architectural choices, data modelling, and testing.g

Ian closely looks at design and implementation strategies you can employ when building a Neo4j-based graph database solution, including architectural choices, data modelling, and testing.g

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
4,055
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
161
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Designing  and  Building  a  Graph   Database  Applica5on   Twi9er:  @ianSrobinson   #neo4j    
  • 2. Outline   •  Data  modeling  with  graphs   •  Neo4j  applica5on  architecture  op5ons   •  Tes5ng  your  data  model    
  • 3. Graph  data  modeling  
  • 4. Labeled  Property  Graph  
  • 5. Models   Purposeful  abstrac5on  of  a  domain  designed  to   sa5sfy  par5cular  applica5on/end-­‐user  goals   Images:  en.wikipedia.org  
  • 6. Example  Applica5on   •  Knowledge  management   –  People,  companies,  skills   –  Cross  organiza5onal   •  Find  my  professional  social  network   –  Exchange  knowledge   –  Interest  groups   –  Help   –  Staff  projects  
  • 7. Applica5on/End-­‐User  Goals   As  an  emp   loyee   I  want  to  k now  who  i n  the  comp has  similar any    skills  to  m   e   So  that  we  can  excha nge  knowl edge  
  • 8. Ques5ons  To  Ask  of  the  Domain   As  an  emp loyee     I  want  to  k now  who  i n  the  co has  similar   skills  to  me mpany       So  that  we  can  excha nge  knowle dge   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?  
  • 9. Iden5fy  En55es   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?     Person   Company   Skill  
  • 10. Iden5fy  Rela5onships  Between  En55es   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?     Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill  
  • 11. Convert  to  Cypher  Paths   Rela5onship   Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill   Label   (:Person)-[:WORKS_FOR]->(:Company), (:Person)-[:HAS_SKILL]->(:Skill)
  • 12. Consolidate  Paths   (:Person)-[:WORKS_FOR]->(:Company), (:Person)-[:HAS_SKILL]->(:Skill) (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  • 13. Candidate  Data  Model   (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  • 14. Express  Ques5on  as  Graph  Pa9ern   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?  
  • 15. Cypher  Query   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  • 16. Graph  Pa9ern   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  • 17. Anchor  Pa9ern  in  Graph   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC Search  nodes  labeled   ‘Person’,  matching  on   ‘name’  property  
  • 18. Create  Projec5on  of  Results   Which  people,  who  work  for  the  same  company   as  me,  have  similar  skills  to  me?   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC
  • 19. First  Match  
  • 20. Second  Match  
  • 21. Third  Match  
  • 22. Running  the  Query   +-----------------------------------+ | name | score | skills | +-----------------------------------+ | "Lucy" | 2 | ["Java","Neo4j"] | | "Bill" | 1 | ["Neo4j"] | +-----------------------------------+ 2 rows
  • 23. From  User  Story  to  Model  and  Query   As  an  emp loyee     I  want  to  k now  who  i n  the  co has  similar   skills  to  me mpany       So  that  we  can  excha nge  knowle dge   MATCH (company)<-[:WORKS_FOR]-(me:Person)-[:HAS_SKILL]->(skill), (company)<-[:WORKS_FOR]-(colleague)-[:HAS_SKILL]->(skill) WHERE me.name = {name} RETURN colleague.name AS name, count(skill) AS score, collect(skill.name) AS skills ORDER BY score DESC ? Which  people,  who  work  for  the  same   company  as  me,  have  similar  skills  to  me? Person  WORKS_FOR  Company   Person  HAS_SKILL  Skill (:Company)<-[:WORKS_FOR]-(:Person)-[:HAS_SKILL]->(:Skill)
  • 24. Tes5ng  
  • 25. Why  Test?   •  Ensure  model  is  fit  for  queries   –  Rapid  feedback   •  Ensure  correctness  of  queries   •  Document  your  understanding  of  your  domain   –  Including  corner  cases  and  excep5ons   •  Provide  a  regression  test  suite   –  Allows  you  to  change  and  evolve  model  and   queries  
  • 26. Method   •  Develop  queries,  or  classes  that  encapsulate   queries,  using  unit  tests   •  Use  small,  well-­‐understood  datasets  in  each  test   –  Create  data  in  test  setup   –  Test  dataset  expresses  your  understanding  of  (part  of)   the  domain   •  Inject  in-­‐memory  graph  database  (or  Cypher   engine)  into  object  under  test   •  The  exact  strategy  you  use  depends  on  your   applica5on  architecture…  
  • 27. Applica5on  Architectures   •  Embedded   •  Server   •  Server  with  Extensions  
  • 28. Applica5on  Architectures   •  Embedded   –  Host  in  Java  process   –  Access  to  Java  APIs   •  Server   •  Server  with  Extensions   Applica5on   Java  APIs  
  • 29. Applica5on  Architectures   •  Embedded   •  Server   –  HTTP/JSON  interface   –  Server  wraps  embedded   instance   •  Server  with  Extensions   Applica5on   REST  Client   Write  LB   REST  API   REST  API   Read  LB   REST  API  
  • 30. Applica5on  Architectures   •  Embedded   •  Server   •  Server  with  Extensions   REST  API   –  Execute  complex  logic  on  server   –  Control  HTTP  request/response  format   Extensions  
  • 31. Embedded  Example   •  Company  social  network   •  Find  colleagues  with  similar  skills   •  Encapsulate  query  in  a  ColleagueFinder
  • 32. Unit  Test  Fixture   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  • 33. Create  Database   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  • 34. Populate  Graph   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } }
  • 35. Create  Object  Under  Test   public class ColleagueFinderTest { private GraphDatabaseService db; private ColleagueFinder finder; @Before public void init() { db = new TestGraphDatabaseFactory().newImpermanentDatabase(); ExampleGraph.populate( db ); finder = new ColleagueFinder( new ExecutionEngine( db ) ); } @After public void shutdown() { db.shutdown(); } } Inject     Execu5onEngine  
  • 36. ImpermanentGraphDatabase   •  In-­‐memory   •  For  tes5ng  only,  not  produc5on!   <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j-kernel</artifactId> <version>${project.version}</version> <type>test-jar</type> <scope>test</scope> </dependency>
  • 37. Create  Sample  Data   public static void populate( GraphDatabaseService db ) { ExecutionEngine engine = new ExecutionEngine( db ); String cypher = "CREATE ian:Person VALUES {name:'Ian'},n" + " bill:Person VALUES {name:'Bill'},n" + " lucy:Person VALUES {name:'Lucy'},n" + " acme:Company VALUES {name:'Acme'},n" + // Cypher continues... " " " " (bill)-[:HAS_SKILL]->(neo4j),n" + (bill)-[:HAS_SKILL]->(ruby),n" + (lucy)-[:HAS_SKILL]->(java),n" + (lucy)-[:HAS_SKILL]->(neo4j)"; engine.execute( cypher ); }
  • 38. Unit  Test   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  • 39. Execute  Object  Under  Test   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  • 40. Assert  Results   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  • 41. Ensure  No  More  Results   @Test public void shouldFindColleaguesWithSimilarSkills() throws Exception { // when Iterator<Map<String, Object>> results = finder.findColleaguesFor( "Ian" ); // then assertEquals( "Lucy", results.next().get( "name" ) ); assertEquals( "Bill", results.next().get( "name" ) ); assertFalse( results.hasNext() ); }
  • 42. ColleagueFinder   public class ColleagueFinder { private final ExecutionEngine executionEngine; public ColleagueFinder( ExecutionEngine executionEngine ) { this.executionEngine = executionEngine; } public Iterator<Map<String, Object>> findColleaguesFor( String name ) { ... } }
  • 43. Inject  Execu5onEngine   public class ColleagueFinder { private final ExecutionEngine executionEngine; public ColleagueFinder( ExecutionEngine executionEngine ) { this.executionEngine = executionEngine; } public Iterator<Map<String, Object>> findColleaguesFor( String name ) { ... } }
  • 44. findColleaguesFor()  Method   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  • 45. Cypher  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  • 46. Parameterized  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  • 47. Execute  Query   public Iterator<Map<String, Object>> findColleaguesFor( String name ) { String cypher = "MATCH (me:Person)-[:WORKS_FOR]->(company),n" + " (me)-[:HAS_SKILL]->(skill),n" + " (colleague)-[:WORKS_FOR]->(company),n" + " (colleague)-[:HAS_SKILL]->(skill)n" + "WHERE me.name = {name}n" + "RETURN colleague.name AS name,n" + " count(skill) AS score,n" + " collect(skill.name) AS skillsn" + "ORDER BY score DESC"; Map<String, Object> params = new HashMap<String, Object>(); params.put( "name", name ); return executionEngine.execute( cypher, params ).iterator(); }
  • 48. Server  Extension  Example   •  Same  data  mode  and  query  as  before   •  This  5me,  we’ll  host  ColleagueFinder  in  a   server  extension  
  • 49. Server  Extension   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  • 50. JAX-­‐RS  Annota5ons   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  • 51. Map  HTTP  Request  to  Object  +  Method   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } } GET   /similar-­‐skills   /Ian  
  • 52. CypherExecutor  Injected  by  Server   Ensures   Execu5onEngine   reused  across   resource  instances   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  • 53. Generate  and  Format  Response   @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  • 54. Extension  Test  Fixture   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  • 55. Build  and  Configure  Server   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  • 56. Start  Server   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  • 57. Populate  Database   public class ColleagueFinderExtensionTest { private CommunityNeoServer server; @Before public void startServer() throws IOException { server = CommunityServerBuilder.server() .withThirdPartyJaxRsPackage( "org.neo4j.good_practices", "/colleagues" ) .build(); server.start(); ExampleGraph.populate( server.getDatabase().getGraph() ); } @After public void stopServer() { server.stop(); } }
  • 58. CommunityServerBuilder   •  Programma5c  configura5on   <dependency> <groupId>org.neo4j.app</groupId> <artifactId>neo4j-server</artifactId> <version>${project.version}</version> <type>test-jar</type> </dependency>
  • 59. Tes5ng  Extension  Using  HTTP  Client   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  • 60. Create  HTTP  Client   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  • 61. Issue  Request   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  • 62. Parse  Response   @Test public void shouldReturnColleaguesWithSimilarSkills() throws Exception { Client client = Client.create( new DefaultClientConfig() ); WebResource resource = client .resource( "http://localhost:7474/colleagues/similar-skills/Ian" ); ClientResponse response = resource .accept( MediaType.APPLICATION_JSON ) .get( ClientResponse.class ); List<Map<String, Object>> results = new ObjectMapper() .readValue(response.getEntity( String.class ), List.class ); // Assertions ...
  • 63. Assert  Results   ... assertEquals( 200, response.getStatus() ); assertEquals( MediaType.APPLICATION_JSON, response.getHeaders().get( "Content-Type" ).get( 0 ) ); assertEquals( "Lucy", results.get( 0 ).get( "name" ) ); assertThat( (Iterable<String>) results.get( 0 ).get( "skills" ), hasItems( "Java", "Neo4j" ) ); }
  • 64. ts gy en lo i m no pl h m ec Co eo T N of Graph h Databases Thank  you   Twi9er:  @ianSrobinson   #neo4j         Ian Robinson, Jim Webber & Emil Eifrem github.com/iansrobinson/neo4j-­‐good-­‐prac5ces