ingraph: Live Queries on Graphs
Gábor Szárnyas
GraphConnect 2017

Live Graph Queries

Railway operation model

Railway operation model

Railway operation model

Railway operation model

Railway operation model
Proximity detection

Railway operation model
Proximity detection

Railway operation model
Proximity detection

Railway operation model
Trailing the switch
Proximity detection

Railway operation model

Railway operation model

Railway operation model
c d e
g
fdiv
2
a b
1

Railway operation model
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
ON
a b
1
NEXT
ON
NEXT

Proximity detection
Proximity detection
≤ 𝟏 segment

Proximity detection
seg1
NEXT: 1..2
t1
ON
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

Proximity detection
seg1
NEXT: 1..2
t1
ON
MATCH
(t1:Train)-[:ON]->(seg1:Segment)
-[:NEXT*1..2]->(seg2:Segment)
<-[:ON]-(t2:Train)
RETURN t1, t2, seg1, seg2
Proximity detection
seg2
t2
ON
≤ 𝟏 segment

Trailing the switch

Trailing the switch
seg div
t
STRAIGHT
ON

Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw

Trailing the switch
seg div
t
STRAIGHT
ON
MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Evaluate
continuously

Batch vs. live queries

Batch vs. live queries
1. Client selects a query
2. Results are calculated
Batch queries
Results obtained on demand

Batch vs. live queries
1. Client selects a query
2. Results are calculated
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
Batch queries Live queries
Results always available
Clients receive notifications
Results obtained on demand

Batch vs. live queries
1. Client selects a query
2. Results are calculated
1. Client registers queries
2. Graph is changed
3. Results are maintained
4. Goto 2
Batch queries Live queries
Results always available
Clients receive notifications
Results obtained on demand
Incremental query evaluation

Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs

Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs
INSTANS LISP/RDF Aalto University
i3QL Scala TU Darmstadt
IncQuery-D Scala/RDF BME

Incremental query engines
CLIPS C NASA
Drools Java Red Hat
VIATRA Java/EMF BME & IncQuery Labs
INSTANS LISP/RDF Aalto University
i3QL Scala TU Darmstadt
IncQuery-D Scala/RDF BME
No implementations for property graphs yet

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results
update graph

ingraph
 PoC query engine
 Goals:
o Provide incremental query evaluation for graphs
o Parallel & distributed operation to allow scalability
ingraphClient
register queries
query results
change notifications
update graph

Under the Hood

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
a
1
ON
e
2
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e div
STRAIGHT

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e div
STRAIGHT
e2
ON
e div
2
STRAIGHT
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
div
STRAIGHTON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
e2 div
STRAIGHTON
e2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c d e
g
fdiv
2
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
ON
div
STRAIGHT
Trailing the switch
ON
div
2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
e2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
e div2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
e div
STRAIGHT
2
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
e div2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div2
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d
div2

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
c e
g
fdiv
NEXT NEXT
STRAIGHT TOP
a b
1
NEXT NEXT
ON
div
STRAIGHT
Trailing the switch
ON
div
ON
2
d

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors
Async messages

πt.number, sw
σsw.position = ′diverging′
⋈
STRAIGHTON e div
STRAIGHT
d2
ON
a1
ON
div
STRAIGHT
Trailing the switch
ON
Actors
Async messages
Szárnyas, G. et al.
IncQuery-D: A distributed incremental model query framework in the cloud.
MODELS 2014

Architecture

ingraph’s query language
openCypher: an open specification of the Cypher language
 Cypher Reference Documentation
 Grammar specification
 TCK (Technology Compatibility Kit)
 Cypher language specification

ingraph’s query language
openCypher: an open specification of the Cypher language
 Cypher Reference Documentation
 Grammar specification
 TCK (Technology Compatibility Kit)
 Cypher language specification
Work-in-progress,
no formal semantics

Formalisation of openCypher
 First work to formalise of openCypher
 Covers most standard constructs
Marton, J., Szárnyas, G. and Varró, D.:
Formalising openCypher Graph Queries in Relational Algebra,
Preprint on arXiv

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
openCypher
query
Query
syntax tree

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Query
syntax tree

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

MATCH (t:Train)-[:ON]->(seg:Segment)
<-[:STRAIGHT]-(sw:Switch)
WHERE sw.position = 'diverging'
RETURN t.number, sw
Relational
algebra
builder
Query
parser
openCypher
query
Relational
Graph
Algebra
Query
syntax tree

Relational
Graph
Algebra
Rete
network
Rete
network
model

Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer

Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer
VIATRA

Relational
Graph
Algebra
Rete
network
Rete
network
model
Transformer
and optimizer
Query
deployer
VIATRA

Use Cases

Constraint validation
 Design-time constraints
in a railway network
 Scalable graph generator
 EMF (Eclipse modelling)
 Property graph
 RDF (semantic web)
 SQL
 Validation queries and
model transformations
 Implemented for 12+ tools
Szárnyas, G. et al.: The Train Benchmark:
cross-technology performance evaluation
of continuous model queries, SOSYM 2017
ftsrg/trainbenchmark

Static analysis of JavaScript
var foo = 1 / 0;
ftsrg/codemodel-rifle

Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
var foo = 1 / 0;
ftsrg/codemodel-rifle

Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ftsrg/codemodel-rifle

Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
binding be
right
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ftsrg/codemodel-rifle

Static analysis of JavaScript
VariableDeclarator
BindingIdentifier
name = `foo`
BinaryExpression
operator = `Div`
LNExpression
value = 1.0
LNExpression
value = 0.0
binding be
right
var foo = 1 / 0;
MATCH
(binding:BindingIdentifier)<-[:binding]-()-->
(be:BinaryExpression)-[:right]->
(right:LNExpression)
WHERE be.operator = 'Div'
AND right.value = 0.0
RETURN binding
ECMAScript 6
• Dead code detection
• Type inferencing
ftsrg/codemodel-rifle

Summary
ftsrg/ingraph

Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
ftsrg/ingraph

Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
 Current goals
o Test with the openCypher Technology Compatibility Kit
o Performance evaluation using the LDBC Social Network Benchmark
ftsrg/ingraph

Summary
 Consider live graph queries for
o Large graph
o Complex queries
o Continuous changes
 Current goals
o Test with the openCypher Technology Compatibility Kit
o Performance evaluation using the LDBC Social Network Benchmark
 Use cases wanted
o Example datasets
o Fraud detection queries
ftsrg/ingraph

Open-source projects
Incremental Graph Engine: github.com/ftsrg/ingraph
Train Benchmark: github.com/ftsrg/trainbenchmark
Codemodel-Rifle: github.com/ftsrg/codemodel-rifle
openCypher TCK: github.com/bme-db-lab/opencypher-tck-tests
This project was supported by the
MTA-BME Lendület Research Group on Cyber-Physical Systems.
Thanks to the openCypher team and the contributors of ingraph.