The document discusses incremental query evaluation on graphs. It describes a proof-of-concept query engine called ingraph that aims to provide incremental query evaluation and parallel/distributed operation for property graphs. The document outlines how ingraph works including registering queries with the engine, handling updates to the graph, and maintaining query results incrementally. It also provides examples of representing graph queries as relational algebra operations that can be evaluated incrementally as the graph changes.

1. ingraph: Live Queries on Graphs
Gábor Szárnyas
GraphConnect 2017

2. Live Graph Queries

3. Railway operation model

4. Railway operation model

5. Railway operation model

6. Railway operation model

7. Railway operation model
Proximity detection

8. Railway operation model
Proximity detection

9. Railway operation model
Proximity detection

10. Railway operation model
Trailing the switch
Proximity detection

11. Railway operation model

12. Railway operation model

13. Railway operation model
c d e
g
fdiv
2
a b
1

14. Railway operation model
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
ON
a b
1
NEXT
ON
NEXT

15. Proximity detection
Proximity detection
≤ 𝟏 segment

16. Proximity detection
seg1
NEXT: 1..2
t1
ON
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

17. Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

18. Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

19. Trailing the switch

20. Trailing the switch
seg div
t
STRAIGHT
ON

21. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

22. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

23. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Evaluate
continuously

24. Batch vs. live queries

25. Batch vs. live queries
1. Client selects a query
2. Results are calculated
Batch queries
Results obtained on demand

26. Batch vs. live queries
1. Client selects a query
2. Results are calculated
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
Batch queries Live queries
Results always available
Clients receive notifications
Results obtained on demand

27. Batch vs. live queries
1. Client selects a query
2. Results are calculated
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
Batch queries Live queries
Results always available
Clients receive notifications
Results obtained on demand
Incremental query evaluation

28. Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs

29. Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs
INSTANS LISP/RDF Aalto University
i3QL Scala TU Darmstadt
IncQuery-D Scala/RDF BME

30. Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs
INSTANS LISP/RDF Aalto University
i3QL Scala TU Darmstadt
IncQuery-D Scala/RDF BME
No implementations for property graphs yet

31. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability

32. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient

33. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries

34. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results

35. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results
update graph

36. ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results
change notifications
update graph

37. Under the Hood

38. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
div
STRAIGHT
Trailing the switch
ON

39. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

40. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

41. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

42. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

43. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

44. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT

45. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e div
STRAIGHT

46. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

47. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON

48. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

49. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

50. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

51. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

52. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

53. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

54. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

55. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

56. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

57. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

58. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

59. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
e div2

60. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

61. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
e div2

62. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

63. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
div2

64. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

65. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON

66. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors

67. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors
Async messages

68. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors
Async messages
Szárnyas, G. et al.
IncQuery-D: A distributed incremental model query framework in the cloud.
MODELS 2014

69. Architecture

70. ingraph’s query language
openCypher: an open specification of the Cypher language
 Cypher Reference Documentation
 Grammar specification
 TCK (Technology Compatibility Kit)
 Cypher language specification

71. ingraph’s query language
openCypher: an open specification of the Cypher language
 Cypher Reference Documentation
 Grammar specification
 TCK (Technology Compatibility Kit)
 Cypher language specification
Work-in-progress,
no formal semantics

72. Formalisation of openCypher
 First work to formalise of openCypher
 Covers most standard constructs
Marton, J., Szárnyas, G. and Varró, D.:
Formalising openCypher Graph Queries in Relational Algebra,
Preprint on arXiv

73. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query

74. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query
Query
syntax tree

75. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Query
syntax tree

76. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

77. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

78. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

79. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

80. Relational
Graph
Algebra
Rete
network
Rete
network
model

81. Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer

82. Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer
VIATRA

83. Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer
Query
deployer
VIATRA

84. Use Cases

85. Constraint validation
 Design-time constraints
in a railway network
 Scalable graph generator
 EMF (Eclipse modelling)
 Property graph
 RDF (semantic web)
 SQL
 Validation queries and
model transformations
 Implemented for 12+ tools
Szárnyas, G. et al.: The Train Benchmark:
cross-technology performance evaluation
of continuous model queries, SOSYM 2017
ftsrg/trainbenchmark

86. Static analysis of JavaScript
var foo = 1 / 0;
ftsrg/codemodel-rifle

87. Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
var foo = 1 / 0;
ftsrg/codemodel-rifle

88. Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ftsrg/codemodel-rifle

89. Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
binding be
right
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ftsrg/codemodel-rifle

90. Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
binding be
right
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ECMAScript 6
• Dead code detection
• Type inferencing
ftsrg/codemodel-rifle

91. Summary
ftsrg/ingraph

92. Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
ftsrg/ingraph

93. Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
 Current goals
o Test with the openCypher Technology Compatibility Kit
o Performance evaluation using the LDBC Social Network Benchmark
ftsrg/ingraph

94. Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
 Current goals
o Test with the openCypher Technology Compatibility Kit
o Performance evaluation using the LDBC Social Network Benchmark
 Use cases wanted
o Example datasets
o Fraud detection queries
ftsrg/ingraph

95. Open-source projects
Incremental Graph Engine: github.com/ftsrg/ingraph
Train Benchmark: github.com/ftsrg/trainbenchmark
Codemodel-Rifle: github.com/ftsrg/codemodel-rifle
openCypher TCK: github.com/bme-db-lab/opencypher-tck-tests
This project was supported by the
MTA-BME Lendület Research Group on Cyber-Physical Systems.
Thanks to the openCypher team and the contributors of ingraph.