Presented at the First openCypher Implementers Meeting in Walldorf, Germany, February 2017 @ http://www.opencypher.org/blog/2017/03/31/first-ocim-blog/

1. Budapest University of Technology and Economics
Department of Measurement and Information Systems
Department of Telecommunications and Media Informatics
Budapest University of Technology and Economics
McGill University, Montréal
Incremental Graph Queries for Cypher
Gábor Szárnyas, József Marton

2. Live railway model

3. Live railway model

4. Live railway model

5. Live railway model

6. Live railway model
Proximity detection

7. Live railway model
Proximity detection

8. Live railway model
Proximity detection

9. Live railway model
Trailing the switch
Proximity detection

10. Live railway model

11. Live railway model

12. Live railway model
c d e
g
fdiv
2
a b
1

13. Live railway model
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
ON
a b
1
NEXT
ON
NEXT

14. Proximity detection
Proximity detection
≤ 𝟐 segments

15. Proximity detection
seg1
NEXT: 1..2
t1
ON
Proximity detection
seg2
t2
ON
≤ 𝟐 segments

16. Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟐 segments

17. Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟐 segments

18. Trailing the switch

19. Trailing the switch
seg div
t
STRAIGHT
ON

20. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

21. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

22. Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Evaluate
continuously

23. Incremental queries

24. Incremental queries
 Register a set of standing queries
 Continuously evaluate queries on changes

25. Incremental queries
 Register a set of standing queries
 Continuously evaluate queries on changes
 The Rete algorithm (1974)
o Originally for rule-based expert systems
o Indexes the graph and caches interim query results

26. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
div
STRAIGHT
Trailing the switch
ON

27. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON

28. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON

29. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

30. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

31. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON

32. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT

33. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e div
STRAIGHT

34. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON

35. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON

36. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

37. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

38. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

39. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

40. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

41. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

42. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

43. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

44. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

45. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

46. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

47. πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

48. Batch vs. incremental queries
 Batch queries
(pull / request-driven):
1. Client selects a query
2. Results are calculated
 Query results
obtained on demand

49. Batch vs. incremental queries
 Batch queries
(pull / request-driven):
1. Client selects a query
2. Results are calculated
 Query results
obtained on demand
 Incremental queries
(push / event-driven):
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
 Query results are
always available

50. Incremental query engines
 CLIPS C structures NASA
 Drools POJO Red Hat
 VIATRA EMF BME / IncQuery Labs.

51. Incremental query engines
 CLIPS C structures NASA
 Drools POJO Red Hat
 VIATRA EMF BME / IncQuery Labs.
 INSTANS RDF Aalto University
 i3QL POJO TU Darmstadt
 IncQuery-D RDF BME

52. Incremental query engines
 CLIPS C structures NASA
 Drools POJO Red Hat
 VIATRA EMF BME / IncQuery Labs.
 INSTANS RDF Aalto University
 i3QL POJO TU Darmstadt
 IncQuery-D RDF BME
 No implementations for property graphs yet

53. ingraph
 An incremental, in-memory graph query engine

54. ingraph
 An incremental, in-memory graph query engine
ingraphclient

55. ingraph
 An incremental, in-memory graph query engine
ingraphclient
register queries

56. ingraph
 An incremental, in-memory graph query engine
ingraphclient
register queries
update graph

57. ingraph
 An incremental, in-memory graph query engine
ingraphclient
register queries
query results
update graph

58. ingraph
 An incremental, in-memory graph query engine
ingraphclient
register queries
query results
change notifications
update graph

59. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query

60. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query
Query
syntax tree

61. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Query
syntax tree

62. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Relational
algebra
model
Query
syntax tree

63. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
algebra
model
Query
syntax tree

64. MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
algebra
model
Query
syntax tree

65. Relational
algebra
model
Rete
network
Rete
network
model

66. Relational
algebra
model
Rete
network
Rete
network
model
Transformer
and optimizer
VIATRA

67. Relational
algebra
model
Rete
network
Rete
network
model
Transformer
and optimizer
VIATRA

68. Relational
algebra
model
Rete
network
Rete
network
model
Transformer
and optimizer
Query
deployer
VIATRA

69. FORMALIZATION OF OPENCYPHER

70. Relational algebra
 Standard relational algebra
o 𝜋, 𝜎
o ∪,∩,∖
o ×, ⋈
 Common extensions
o 𝛿 – duplicate elimination
o 𝛾 – grouping
o 𝜏 – sorting

71. Graph-specific operators
Jürgen Hölsch, Michael Grossniklaus:
An Algebra and Equivalences to Transform
Graph Patterns in Neo4j, GraphQ 2016, EDBT,
http://ceur-ws.org/Vol-1558/paper24.pdf
GetVertices: returns a graph relation containing all
vertices of the underlying graph G
Expand: return the neighbors of a given node

72. Additional extensions
GetEdges: returns a graph relation containing all
edges of the underlying graph G
Gábor Szárnyas, József Marton:
openCypher specification, Technical report
http://docs.inf.mit.bme.hu/ingraph/pub/
opencypher-report.pdf

73. While pattern matching, Neo4j makes
sure to not include matches where the
same graph relationship is found
multiple times in a single pattern.
Uniqueness of edges
All-different operator

74. EXAMPLE QUERIES

75. Uniqueness of edges

76. Get neighbours

77. Filter out based on node prop name

78. Use multiple MATCH clauses to do a Cartesian product

79. Two subgraphs

80. SwitchMonitored

82. INCREMENTAL GRAPH QUERIES
WITH OPENCYPHER

83. openCypher constructs
 Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths

84. openCypher constructs
 Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths
 Legacy constructs
o indexing, constraints
o regular expressions
o some list functions,
including reduce
o most predicate functions
o shortest path functions
o CASE expressions
o id()

85. openCypher constructs
 Standard constructs
o pattern matching
o filtering
o lists, maps
o data manipulation
o variable length paths
 Legacy constructs
o indexing, constraints
o regular expressions
o some list functions,
including reduce
o most predicate functions
o shortest path functions
o CASE expressions
o id()
Difficult to handle
incrementally

86. Challenges for incremental openCypher
 Lists
o ['a', 1, 2, true]
o ['a', [1, [2]], true]
o UNWIND
 Efficient aggregation
o min(), max()
o collect()
 Bag semantics, ORDER BY, SKIP and LIMIT
o Idea: collect(x ORDER BY x.name)

87. Incremental queries – use cases
Standing queries on large & quickly changing graph
 Runtime monitoring (train example)
 Model validation: The Train Benchmark
 Static analysis of JavaScript source code
 Fraud detection
 IT infrastructure monitoring

88. Future work
Path operator
Giacomo Bergami, Matteo Magnani, and Danilo Montesi,
A join operator for property graphs, GraphQ 2016, EDBT,
http://jackbergus.alwaysdata.net/paper_graph_graphq2
017.pdf
Unwind operator: 𝜔
Elena Botoeva et al.:
A Formal Presentation of MongoDB,
https://arxiv.org/abs/1603.09291

89. Open-Source Projects
Incremental Graph Engine:
https://github.com/ftsrg/ingraph
Train Benchmark:
https://github.com/ftsrg/trainbenchmark
BME-MODES3:
https://github.com/ftsrg/bme-modes3
Available under EPL v1.0.