Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ingraph: Live Queries on Graphs

842 views

Published on

GraphConnect Europe 2017
Gábor Szárnyas, Budapest University of Technology & Economics

Published in: Technology
  • Be the first to comment

ingraph: Live Queries on Graphs

  1. 1. ingraph: Live Queries on Graphs Gábor Szárnyas GraphConnect 2017
  2. 2. Live Graph Queries
  3. 3. Railway operation model
  4. 4. Railway operation model
  5. 5. Railway operation model
  6. 6. Railway operation model
  7. 7. Railway operation model Proximity detection
  8. 8. Railway operation model Proximity detection
  9. 9. Railway operation model Proximity detection
  10. 10. Railway operation model Trailing the switch Proximity detection
  11. 11. Railway operation model
  12. 12. Railway operation model
  13. 13. Railway operation model c d e g fdiv 2 a b 1
  14. 14. Railway operation model c d e g fdiv 2 NEXT NEXT STRAIGHT TOP ON a b 1 NEXT ON NEXT
  15. 15. Proximity detection Proximity detection ≤ 𝟏 segment
  16. 16. Proximity detection seg1 NEXT: 1..2 t1 ON Proximity detection seg2 t2 ON ≤ 𝟏 segment
  17. 17. Proximity detection seg1 NEXT: 1..2 t1 ON MATCH (t1:Train)-[:ON]->(seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) <-[:ON]-(t2:Train) RETURN t1, t2, seg1, seg2 Proximity detection seg2 t2 ON ≤ 𝟏 segment
  18. 18. Proximity detection seg1 NEXT: 1..2 t1 ON MATCH (t1:Train)-[:ON]->(seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) <-[:ON]-(t2:Train) RETURN t1, t2, seg1, seg2 Proximity detection seg2 t2 ON ≤ 𝟏 segment
  19. 19. Trailing the switch
  20. 20. Trailing the switch seg div t STRAIGHT ON
  21. 21. Trailing the switch seg div t STRAIGHT ON MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw
  22. 22. Trailing the switch seg div t STRAIGHT ON MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw
  23. 23. Trailing the switch seg div t STRAIGHT ON MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Evaluate continuously
  24. 24. Batch vs. live queries
  25. 25. Batch vs. live queries 1. Client selects a query 2. Results are calculated Batch queries Results obtained on demand
  26. 26. Batch vs. live queries 1. Client selects a query 2. Results are calculated 1. Client registers queries 2. Graph is changed 3. Results are maintained 4. Goto 2 Batch queries Live queries Results always available Clients receive notifications Results obtained on demand
  27. 27. Batch vs. live queries 1. Client selects a query 2. Results are calculated 1. Client registers queries 2. Graph is changed 3. Results are maintained 4. Goto 2 Batch queries Live queries Results always available Clients receive notifications Results obtained on demand Incremental query evaluation
  28. 28. Incremental query engines CLIPS C NASA Drools Java Red Hat VIATRA Java/EMF BME & IncQuery Labs
  29. 29. Incremental query engines CLIPS C NASA Drools Java Red Hat VIATRA Java/EMF BME & IncQuery Labs INSTANS LISP/RDF Aalto University i3QL Scala TU Darmstadt IncQuery-D Scala/RDF BME
  30. 30. Incremental query engines CLIPS C NASA Drools Java Red Hat VIATRA Java/EMF BME & IncQuery Labs INSTANS LISP/RDF Aalto University i3QL Scala TU Darmstadt IncQuery-D Scala/RDF BME No implementations for property graphs yet
  31. 31. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability
  32. 32. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability ingraphClient
  33. 33. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability ingraphClient register queries
  34. 34. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability ingraphClient register queries query results
  35. 35. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability ingraphClient register queries query results update graph
  36. 36. ingraph  PoC query engine  Goals: o Provide incremental query evaluation for graphs o Parallel & distributed operation to allow scalability ingraphClient register queries query results change notifications update graph
  37. 37. Under the Hood
  38. 38. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON div STRAIGHT Trailing the switch ON
  39. 39. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON
  40. 40. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON
  41. 41. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON a 1 ON e 2 ON
  42. 42. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON a 1 ON e 2 ON
  43. 43. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON
  44. 44. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e div STRAIGHT
  45. 45. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e div STRAIGHT e div STRAIGHT
  46. 46. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON
  47. 47. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e div STRAIGHT e2 ON
  48. 48. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e div STRAIGHT e2 ON e div 2 STRAIGHT ON
  49. 49. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e div STRAIGHT e2 ON e div 2 STRAIGHT ON
  50. 50. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON div STRAIGHTON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e2 div STRAIGHTON e2
  51. 51. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON div STRAIGHTON e div STRAIGHT 2 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON e2 div STRAIGHTON e2
  52. 52. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON div 2
  53. 53. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON div 2
  54. 54. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c d e g fdiv 2 NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON ON div STRAIGHT Trailing the switch ON div 2
  55. 55. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  56. 56. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT e2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  57. 57. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  58. 58. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  59. 59. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON e div STRAIGHT 2 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d e div2
  60. 60. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  61. 61. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON e div STRAIGHT 2 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d e div2
  62. 62. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  63. 63. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div2 c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d div2
  64. 64. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON c e g fdiv NEXT NEXT STRAIGHT TOP a b 1 NEXT NEXT ON div STRAIGHT Trailing the switch ON div ON 2 d
  65. 65. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div STRAIGHT Trailing the switch ON
  66. 66. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div STRAIGHT Trailing the switch ON Actors
  67. 67. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div STRAIGHT Trailing the switch ON Actors Async messages
  68. 68. πt.number, sw σsw.position = ′diverging′ ⋈ STRAIGHTON e div STRAIGHT d2 ON a1 ON div STRAIGHT Trailing the switch ON Actors Async messages Szárnyas, G. et al. IncQuery-D: A distributed incremental model query framework in the cloud. MODELS 2014
  69. 69. Architecture
  70. 70. ingraph’s query language openCypher: an open specification of the Cypher language  Cypher Reference Documentation  Grammar specification  TCK (Technology Compatibility Kit)  Cypher language specification
  71. 71. ingraph’s query language openCypher: an open specification of the Cypher language  Cypher Reference Documentation  Grammar specification  TCK (Technology Compatibility Kit)  Cypher language specification Work-in-progress, no formal semantics
  72. 72. Formalisation of openCypher  First work to formalise of openCypher  Covers most standard constructs Marton, J., Szárnyas, G. and Varró, D.: Formalising openCypher Graph Queries in Relational Algebra, Preprint on arXiv
  73. 73. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw openCypher query
  74. 74. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw openCypher query Query syntax tree
  75. 75. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Query parser openCypher query Query syntax tree
  76. 76. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Query parser openCypher query Relational Graph Algebra Query syntax tree
  77. 77. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Relational algebra builder Query parser openCypher query Relational Graph Algebra Query syntax tree
  78. 78. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Relational algebra builder Query parser openCypher query Relational Graph Algebra Query syntax tree
  79. 79. MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw Relational algebra builder Query parser openCypher query Relational Graph Algebra Query syntax tree
  80. 80. Relational Graph Algebra Rete network Rete network model
  81. 81. Relational Graph Algebra Rete network Rete network model Transformer and optimizer
  82. 82. Relational Graph Algebra Rete network Rete network model Transformer and optimizer VIATRA
  83. 83. Relational Graph Algebra Rete network Rete network model Transformer and optimizer Query deployer VIATRA
  84. 84. Use Cases
  85. 85. Constraint validation  Design-time constraints in a railway network  Scalable graph generator  EMF (Eclipse modelling)  Property graph  RDF (semantic web)  SQL  Validation queries and model transformations  Implemented for 12+ tools Szárnyas, G. et al.: The Train Benchmark: cross-technology performance evaluation of continuous model queries, SOSYM 2017 ftsrg/trainbenchmark
  86. 86. Static analysis of JavaScript var foo = 1 / 0; ftsrg/codemodel-rifle
  87. 87. Static analysis of JavaScript VariableDeclarator BindingIdentifier name = `foo` BinaryExpression operator = `Div` LNExpression value = 1.0 LNExpression value = 0.0 var foo = 1 / 0; ftsrg/codemodel-rifle
  88. 88. Static analysis of JavaScript VariableDeclarator BindingIdentifier name = `foo` BinaryExpression operator = `Div` LNExpression value = 1.0 LNExpression value = 0.0 var foo = 1 / 0; MATCH (binding:BindingIdentifier)<-[:binding]-()--> (be:BinaryExpression)-[:right]-> (right:LNExpression) WHERE be.operator = 'Div' AND right.value = 0.0 RETURN binding ftsrg/codemodel-rifle
  89. 89. Static analysis of JavaScript VariableDeclarator BindingIdentifier name = `foo` BinaryExpression operator = `Div` LNExpression value = 1.0 LNExpression value = 0.0 binding be right var foo = 1 / 0; MATCH (binding:BindingIdentifier)<-[:binding]-()--> (be:BinaryExpression)-[:right]-> (right:LNExpression) WHERE be.operator = 'Div' AND right.value = 0.0 RETURN binding ftsrg/codemodel-rifle
  90. 90. Static analysis of JavaScript VariableDeclarator BindingIdentifier name = `foo` BinaryExpression operator = `Div` LNExpression value = 1.0 LNExpression value = 0.0 binding be right var foo = 1 / 0; MATCH (binding:BindingIdentifier)<-[:binding]-()--> (be:BinaryExpression)-[:right]-> (right:LNExpression) WHERE be.operator = 'Div' AND right.value = 0.0 RETURN binding ECMAScript 6 • Dead code detection • Type inferencing ftsrg/codemodel-rifle
  91. 91. Summary ftsrg/ingraph
  92. 92. Summary  Consider live graph queries for o Large graph o Complex queries o Continuous changes ftsrg/ingraph
  93. 93. Summary  Consider live graph queries for o Large graph o Complex queries o Continuous changes  Current goals o Test with the openCypher Technology Compatibility Kit o Performance evaluation using the LDBC Social Network Benchmark ftsrg/ingraph
  94. 94. Summary  Consider live graph queries for o Large graph o Complex queries o Continuous changes  Current goals o Test with the openCypher Technology Compatibility Kit o Performance evaluation using the LDBC Social Network Benchmark  Use cases wanted o Example datasets o Fraud detection queries ftsrg/ingraph
  95. 95. Open-source projects Incremental Graph Engine: github.com/ftsrg/ingraph Train Benchmark: github.com/ftsrg/trainbenchmark Codemodel-Rifle: github.com/ftsrg/codemodel-rifle openCypher TCK: github.com/bme-db-lab/opencypher-tck-tests This project was supported by the MTA-BME Lendület Research Group on Cyber-Physical Systems. Thanks to the openCypher team and the contributors of ingraph.

×