Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018

265 views

Published on

Learn how to build applications to process highly connected data by working backwards from your questions to your graph data model.

  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL. BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Work Backwards to Your Graph Data Model & Queries with Amazon Neptune (DAT330) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Work Backwards to Your Graph Data Model & Queries with Amazon Neptune Ian Robinson Data Architect AWS D A T 3 3 0 Mike Personick Sr. Software Dev Engineer AWS
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Introduction to Amazon Neptune Overview of the data modelling process How to apply the property graph building blocks ”Slow motion” example of developing a model and queries
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related breakouts Monday, November 26 On-Ramp to Graph Databases and Amazon Neptune - DAT335 4:45 p.m. | Mirage, St. Thomas B Tuesday, November 27 Migrating to Amazon Neptune - DAT338 11:30 a.m. | Venetian, Level 4, Lando 4305 Wednesday, November 28 Getting Started with Amazon Neptune and Amazon SageMaker Jupyter Notebooks - DAT359 2:30 p.m. | Aria West, Level 3, Starvine 10, Table 7
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Neptune Fast, reliable graph database Optimized for storing and querying highly connected data
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fully managed graph database Fast Reliable Open Query billions of relationships with millisecond latency Six replicas of your data across three AZs with full backup and restore Build powerful queries easily with Gremlin and SPARQL Supports Apache TinkerPop & W3C RDF graph models Easy
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Do I have a graph workload? Complex domain model Variable schema Connected queries Large dataset Many different entities Similar entities may have different properties Highly connected Entities connected in many different ways Navigate connected structure Take account of strength, weight, or quality of relationships Variable structure Social networking Recommendations Knowledge graphs Fraud detection Life sciences Network & IT operations
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working backwards—Design for queryability
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Use property graph primitives to build your model Vertices Entities (things with identity, complex value types) Properties Entity attributes + metadata Labels Roles, grouping, tagging Edges Relationships between entities, variable structure Properties Relationship strength, weight, or property + metadata Labels Relationship semantics
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Vertex and edge IDs Neptune will create UUID IDs for new vertices and edges Alternatively, you can supply your own string-based IDs All vertex IDs must be unique All edge IDs must be unique You must supply your own IDs when bulk loading data You can also create id properties Avoid doing this: it creates confusion
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Labels Tags, keyless properties Typically used to “type” a vertex, or name a relationship Vertices must have one or more labels Neptune will create a “vertex” label if you don’t supply one Edges must have exactly one label
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Edges—The routes to success Performance depends on how much of the graph a query must “touch” Choose domain-meaningful edge labels Discover only what is absolutely necessary “Grey out” unnecessary portions of the graph
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How fine-grained should my edge labels be? Is it an open or extensible set of label values? Do you need to query across values in the set? (for example, all addresses)
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bi-directional relationships Ignore edge direction in queries g.V().hasLabel('Person'). both('FRIENDS')
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Uni-directional relationships Be explicit about edge direction in queries g.V().hasLabel('Person'). out('FOLLOWS’) g.V().hasLabel('Person'). in('FOLLOWS')
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multiple edges between vertices
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. You can even have self-edges
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example scenario Employment history application People, companies, roles Use cases 1. Find the companies where X has worked, and their roles at those companies 2. Find the people who have worked for a company at a specific location during a particular time period 3. Find the people in more senior roles at the companies where X worked
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the companies where X has worked, and their roles at those companies Which companies has X worked for, and in what roles?
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify entities, relationships, and attributes Find the companies where X has worked, and their roles at those companies Which companies has X worked for, and in what roles? Companies Company Entity X Person Entity Worked for Worked for Relationship Roles Role Attribute?
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify candidate vertices, labels, and properties Company Entity Vertex Company Person Entity Vertex Person Worked for Relationship Edge WORKED_FOR Role Attribute? Property? role name CompanyPerson
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is role an entity or an attribute? Does it have identity (or is it a value type)? X Is it a complex type (with multiple fields)? X Are there any structural relations between values? ? Keep it simple Prefer properties to vertices/edges until the need arises
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 1
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 1 Which companies has Li worked for, and in what roles?
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create dataset (g. addV('Person').property(id,'p-1').property('firstName','Martha').property('lastName','Rivera'). addV('Person').property(id,'p-2').property('firstName','Richard').property('lastName','Roe'). addV('Person').property(id,'p-3').property('firstName','Li').property('lastName','Juan'). addV('Person').property(id,'p-4').property('firstName','John').property('lastName','Stiles'). addV('Person').property(id,'p-5').property('firstName','Saanvi').property('lastName','Sarkar'). addV('Company').property(id,'c-1').property('name','Example Corp'). addV('Company').property(id,'c-2').property('name','AnyCompany'). V('p-1').addE('WORKED_FOR').to(V('c-1')).property('role','Principal Analyst'). V('p-2').addE('WORKED_FOR').to(V('c-1')).property('role','Senior Analyst'). V('p-3').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-4').addE('WORKED_FOR').to(V('c-1')).property('role','Analyst'). V('p-5').addE('WORKED_FOR').to(V('c-2')).property('role','Manager'). V('p-3').addE('WORKED_FOR').to(V('c-2')).property('role','Associate Analyst'). toList())
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Write a test Sample data and code Amazon SageMaker Jupyter notebooks AWS Cloud9 cloud IDE Gremlin console Unit tests Bedrock of asserted behavior Fast feedback as you evolve your model
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unit test %%unittest results = None # Which companies has Li worked for, and in what roles? assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail AssertionError: None != [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unit test %%unittest results = (g.V('p-3'). outE('WORKED_FOR').as_('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the people who have worked for a company at a specific location during a particular time period Who worked for company X, and at which locations, between Y1-Y2?
  49. 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entities, relationships, and attributes Find the people who have worked for a company at a specific location during a particular time period Who worked for company X, and at which locations, between Y1-Y2? Locations Location Entity? Y1-Y2 Date range or Attributes dates (from and to)?
  50. 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is location an entity or an attribute? Does it have identity (or is it a value type)? ✓ Is it a complex type (with multiple fields)? ✓ Are there any structural relations between values? ? Model location as a vertex
  51. 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We can’t ”attribute” an edge with a vertex name CompanyPerson Location
  52. 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introduce an intermediate vertex name CompanyPerson Location ???
  53. 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised model Represent facts/events as vertices Connect to multiple dimensions name CompanyPerson Location Job
  54. 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 2
  55. 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 2 Who worked for Example Corp., and at which locations, between 2015- 2017?
  56. 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  57. 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  58. 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  59. 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  60. 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  61. 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  62. 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 2 g.V('c-1'). in('COMPANY'). or( (has('from', between(datetime(2015,1,1), datetime(2018,1,1)))), (has('to', between(datetime(2015,1,1), datetime(2018,1,1)))) ). project('name', 'location'). by(in('JOB').values('firstName', 'lastName').fold()). by(out('LOCATION').values('name', 'address').fold()). toList()
  63. 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We’ve broken query 1! %%unittest results = (g.V('p-3'). outE('WORKED_FOR').as_('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail AssertionError: Lists differ: [] != [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst'}]
  64. 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  65. 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  66. 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  67. 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). outE('WORKED_FOR').as('e'). otherV(). project('company', 'role'). by('name'). by(select('e').values('role')). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()
  68. 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  69. 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What questions would we have to ask of our data? Find the people in more senior roles at the companies where X worked Who were in senior roles at the companies where X worked?
  70. 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entities, relationships, and attributes Find the people in more senior roles at the companies where X worked Who were in more senior roles at the companies where X worked? Senior Roles Role Entity?
  71. 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Entity or attribute? Vertex or property? Is role an entity or an attribute? Does it have identity (or is it a value type)? X Is it a complex type (with multiple fields)? X Are there any structural relations between values? ✓ Model structural relations with edges Promote role to being a vertex
  72. 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Role hierarchy Role Role Role
  73. 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised model name CompanyPerson Location Job Role
  74. 74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model 3
  75. 75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Traversal 3 Who were in more senior roles at the companies where Li worked?
  76. 76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create data to test predicates
  77. 77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Create data to test predicates assert results == [{'role': 'Principal Analyst', 'name': ['Martha', 'Rivera']}, {'role': 'Manager', 'name': ['Saanvi', 'Sarkar']}]
  78. 78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  79. 79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  80. 80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  81. 81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  82. 82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  83. 83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  84. 84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  85. 85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  86. 86. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gremlin query 3 g.V('p-3'). out('JOB').as('j1'). out('ROLE'). repeat(out('PARENT_ROLE')).until(outE().count().is(0)). emit().in('ROLE').as('j2'). or( (where('j1', between('j2', 'j2')).by('from').by('from').by('to')), (where('j1', between('j2', 'j2')).by('to').by('from').by('to')), (where('j1', lte('j2').and(gt('j2'))).by('from').by('from').by('to').by('from')) ). project('role', 'name'). by(out('ROLE').values('name')). by(in('JOB').values('firstName', 'lastName').fold()). toList())
  87. 87. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. We’ve broken query 1 (again)! %%unittest results = (g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList()) assert results == [{'company': 'Example Corp', 'role': 'Analyst'}, {'company': 'AnyCompany', 'role': 'Associate Analyst’}] Fail The property does not exist as the key has no associated value for the provided element: v[j-3]:role
  88. 88. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by(out('ROLE').values('name')). toList()
  89. 89. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Revised query 1 g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by('role'). toList() g.V('p-3'). out('JOB'). project('company', 'role'). by(out('COMPANY').values('name')). by(out('ROLE').values('name')). toList()
  90. 90. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  91. 91. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What have we learned? Work backwards from your use case to make a useful model Start simple Evolve as necessary Your model should offer up the most efficient routes for your queries The model is the superset of paths traversed by your queries Use interactive development environments to develop your model and queries Write unit tests that give you fast feedback as you evolve your model
  92. 92. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What have we learned? • Work backwards from your use case to make a useful model • Start simple and evolve as necessary • Your model should offer up the most efficient routes for your queries • The model is the superset of paths traversed by your queries • Use an interactive environment to develop your model and queries • Write unit tests that give you fast feedback as you evolve your model
  93. 93. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Ian Robinson ianrob@amazon.com Mike Personick mprsnck@amazon.com
  94. 94. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×