i3QL: Language-Integrated Live Data Views. Ralf Mitschke, Sebastian Erdweg, Mirko Köhler, Mira Mezini, and Guido Salvaneschi. In Proceedings of Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2014. Abstract: An incremental computation updates its result based on a change to its input, which is often an order of magnitude faster than a recomputation from scratch. In particular, incrementalization can make expensive computations feasible for settings that require short feedback cycles, such as interactive systems, IDEs, or (soft) real-time systems. This paper presents i3QL, a general-purpose programming language for specifying incremental computations. i3QL provides a declarative SQL-like syntax and is based on incremental versions of operators from relational algebra, enriched with support for general recursion. We integrated i3QL into Scala as a library, which enables programmers to use regular Scala code for non-incremental subcomputations of an i3QL query and to easily integrate incremental computations into larger software projects. To improve performance, i3QL optimizes user-defined queries by applying algebraic laws and partial evaluation. We describe the design and implementation of i3QL and its optimizations, demonstrate its applicability, and evaluate its performance.

1. i3QL: Language-Integrated Live Data Views
Ralf Mitschke
Sebastian Erdweg
Mirko Köhler
Mira Mezini
Guido Salvaneschi

2. 2
World airports
Flight data

3. 2
World airports
Flight data
From each city, how many
flights to Portland in 2014?

4. 3
World airports
Flight data
From each city, how many
flights to Portland in 2014?
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
!
!
!

5. 3
World airports
Flight data
From each city, how many
flights to Portland in 2014?
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
!
AND f.takeoff = '2014-01-01'
!
AND f.takeoff = '2014-12-31'
!

6. 3
World airports
Flight data
From each city, how many
flights to Portland in 2014?
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
!
!
!
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city

7. 3
World airports
Flight data
From each city, how many
flights to Portland in 2014?
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
!
!
!
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city

8. 4
World airports
Flight data
From each city, how many
flights to Portland in 2014?
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city

9. 5
World airports
Flight data
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland

10. 5
World airports
Flight data
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm

11. 5
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm

12. 6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm

13. 6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm
3 5 2014-09-14 8:15pm Additional flight

14. 6
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm
3 5 2014-09-14 8:15pm Additional flight
+1

15. From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm
3 5 2014-09-14 8:15pm
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland

16. From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm
3 5 2014-09-14 8:15pm
2 5 2015-01-01 11:05am
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
Rescheduled

17. From To Take off
1 5 2014-01-01 10:00am
1 5 2014-01-03 10:00am
1 2 2014-01-01 1:20pm
4 5 2013-12-31 4:00pm
4 5 2014-01-01 4:00pm
2 5 2014-12-31 5:05pm
3 5 2014-09-14 8:15pm
2 5 2015-01-01 11:05am
7
World airports
Flight data
From Flights
Amsterdam 360
Boston 350
San Francisco 3468
Tokyo 349
SELECT a1.city AS From
COUNT(*) AS Flights
FROM Airport a1, Airport a2, Flight f
WHERE f.from = a1.id
AND f.to = a2.id
AND a2.code = ‘PDX’
AND f.takeoff = '2014-01-01'
AND f.takeoff = '2014-12-31'
GROUP BY a1.city
ID Code City
1 AMS Amsterdam
2 BOS Boston
3 SFO San Francisco
4 NRT Tokyo
5 PDX Portland
-1
Rescheduled

18. 8

19. 8

20. 8
Change
Update

21. 8
Change
Update

22. 9
Maintain live data views

23. 9
Maintain live data views
=
Incremental computations

24. 10
Incremental computations
at our fingertips

25. 10
Incremental computations
at our fingertips
(inside PL)

26. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !

27. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL

28. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API

29. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API
based on relational algebra

30. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API
based on relational algebra
case class Airport(id: Int, code: String, city: String)
case class Flight(from: Int, to: Int, takeoff: Date)

31. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API
based on relational algebra
case class Airport(id: Int, code: String, city: String)
case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]
val flight: Table[Flight]

32. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API
based on relational algebra
case class Airport(id: Int, code: String, city: String)
case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]
val flight: Table[Flight]
airport += Airport(1, AMS, Amsterdam)
airport += Airport(2, BOS, Boston)
airport += Airport(5, PDX, “Portland)

33. 11
i3QL!
integrated! ! ! ! !
in-memory! ! ! ! !
incremental! ! !
Scala-embedded DSL
collections API
based on relational algebra
case class Airport(id: Int, code: String, city: String)
case class Flight(from: Int, to: Int, takeoff: Date)
val airport: Table[Airport]
val flight: Table[Flight]
airport += Airport(1, AMS, Amsterdam)
airport += Airport(2, BOS, Boston)
airport += Airport(5, PDX, “Portland)
flight += Flight(1, 5, new Date(2014, 1, 1, 10, 0))
flight += Flight(1, 5, new Date(2014, 1, 3, 10, 0))
flight += Flight(2, 5, new Date(2014, 12, 31, 17, 5))

34. 12
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
val q = (
!
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)
WHERE ((a1, a2, f) =
f.from == a1.id AND
f.to == a2.id AND
a2.code == PDX
!
)

35. 12
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
val q = (
!
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)
WHERE ((a1, a2, f) =
f.from == a1.id AND
f.to == a2.id AND
a2.code == PDX
!
)
AND
f.takeoff = new Date(2014, 01, 01) AND
f.takeoff new Date(2015, 01, 01)

36. 12
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
val q = (
!
!
!
!
!
!
!
!
!
!
!
)
FROM (airport, airport, flight)
WHERE ((a1, a2, f) =
f.from == a1.id AND
f.to == a2.id AND
a2.code == PDX
!
)
AND
f.takeoff = new Date(2014, 01, 01) AND
f.takeoff new Date(2015, 01, 01)
GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight])
= a1.city)

37. 12
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
val q = (
!
!
!
!
!
!
!
!
!
!
!
)
SELECT ((city: Rep[String]) = city,
COUNT(*))
FROM (airport, airport, flight)
WHERE ((a1, a2, f) =
f.from == a1.id AND
f.to == a2.id AND
a2.code == PDX
!
)
AND
f.takeoff = new Date(2014, 01, 01) AND
f.takeoff new Date(2015, 01, 01)
GROUP BY ((a1: Rep[Airport], a2: Rep[Airport], f: Rep[Flight])
= a1.city)

38. 13
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
val q = SELECT … FROM … WHERE …

39. 13
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
val q = SELECT … FROM … WHERE …
val result = q.asMaterialized

40. 13
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
val q = SELECT … FROM … WHERE …
val result = q.asMaterialized
From Flights
Amsterdam 360
Boston 350
San Francisco 3467
Tokyo 349

41. 14
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
From Flights
val q = SELECT … FROM … WHERE …
Amsterdam 360
!
Boston 350
San Francisco 3467
val result = q.asMaterialized
Tokyo 349
!
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))

42. 14
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
From Flights
val q = SELECT … FROM … WHERE …
Amsterdam 360
!
Boston 350
San Francisco 3467
val result = q.asMaterialized
Tokyo 349
!
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))
+1

43. 15
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
From Flights
val q = SELECT … FROM … WHERE …
Amsterdam 360
!
Boston 350
San Francisco 3468
val result = q.asMaterialized
Tokyo 349
!
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))
flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5))
- Flight(2, 5, new Date(2015, 1, 1, 11, 5))

44. 15
i3QL queries
val airport: Table[Airport]
val flight: Table[Flight]
!
From Flights
val q = SELECT … FROM … WHERE …
Amsterdam 360
!
Boston 350
San Francisco 3468
val result = q.asMaterialized
Tokyo 349
!
flight += Flight(3, 5, new Date(2014, 9, 14, 20, 15))
flight ~= Flight(2, 5, new Date(2014, 12, 31, 17, 5))
- Flight(2, 5, new Date(2015, 1, 1, 11, 5))
-1

45. 16
i3QL query execution

46. 16
i3QL query execution
1.Translate i3QL query to relational-algebra tree

47. 16
i3QL query execution
1.Translate i3QL query to relational-algebra tree
2.Use incremental relational-algebra operators
that propagate change events

48. 17
×
airport flight

49. 17
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
airport flight

50. 17
γ GROUP a1.city
COUNT(*)
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
airport flight

51. 17
γ GROUP a1.city
COUNT(*)
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
airport flight
add((3, 5, new Date(2014, 9, 14, 20, 15)))

52. 17
γ GROUP a1.city
COUNT(*)
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
for (a1, a2 - airport)
add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
airport flight
add((3, 5, new Date(2014, 9, 14, 20, 15)))

53. add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))
17
γ GROUP a1.city
COUNT(*)
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
for (a1, a2 - airport)
add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
airport flight
add((3, 5, new Date(2014, 9, 14, 20, 15)))

54. update((“San Francisco”, 3467)
- (“San Francisco”, 3468))
add((3, “SFO”), (5, “PDX”), (3, 5, new Date(2014, 9, 14, 20, 15)))
17
γ GROUP a1.city
COUNT(*)
σ f.from == a1.id
×
f.to == a2.id
a2.code == “PDX”
f.takeoff = new Date(2014, 01, 01)
f.takeoff new Date(2015, 01, 01)
for (a1, a2 - airport)
add(a1, a2, (3, 5, new Date(2014, 9, 14, 20, 15)))
airport flight
add((3, 5, new Date(2014, 9, 14, 20, 15)))

55. 18
i3QL query execution
1.Translate i3QL query to relational-algebra tree
!
3.Use incremental relational-algebra operators
that propagate change events

56. 18
i3QL query execution
1.
Translate i3QL query to relational-algebra tree
2.Optimize relational-algebra tree
!
!! - algebraic laws
!! - automatic indexing
!! - partial evaluation
!! - common subquery elimination
} based on LMS
by Rompf et al.
3.Use incremental relational-algebra operators
that propagate change events

57. 19
GROUP a1.city γ
COUNT(*)
f.from == a1.id σ
f.to == a2.id
a2.code == “PDX”
Date(2014, 01, 01)
Date(2015, 01, 01)
×
optimize
airport flight

58. 19
GROUP a1.city γ
COUNT(*)
f.from == a1.id σ
f.to == a2.id
a2.code == “PDX”
Date(2014, 01, 01)
Date(2015, 01, 01)
×
airport flight
airport flight
optimize
σcode==“PDX” σ takeoff =
new Date(2014, 01, 01)
takeoff
new Date(2015, 01, 01)

59. 19
GROUP a1.city γ
COUNT(*)
f.from == a1.id σ
f.to == a2.id
a2.code == “PDX”
Date(2014, 01, 01)
Date(2015, 01, 01)
×
airport flight
automatic indexing
⋈f.from == a1.id
f.to == a2.id
airport flight
optimize
σcode==“PDX” σ takeoff =
new Date(2014, 01, 01)
takeoff
new Date(2015, 01, 01)

60. 19
GROUP a1.city γ
COUNT(*)
f.from == a1.id σ
f.to == a2.id
a2.code == “PDX”
Date(2014, 01, 01)
Date(2015, 01, 01)
×
airport flight
γ GROUP a1.city
COUNT(*)
automatic indexing
⋈f.from == a1.id
f.to == a2.id
airport flight
optimize
σcode==“PDX”
σ takeoff =
new Date(2014, 01, 01)
takeoff
new Date(2015, 01, 01)

61. 20
i3QL operators
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.

62. 20
i3QL operators
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
Support recursion
WITH RECURSIVE (init UNION ALL query)
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.

63. 20
i3QL operators
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
Support recursion
WITH RECURSIVE (init UNION ALL query)
Extensible
! Add new operators
! Add new optimizations
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.

64. 20
i3QL operators
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
Support recursion
WITH RECURSIVE (init UNION ALL query)
Extensible
! Add new operators
! Add new optimizations
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Use regular Scala code
for nonincremental subcomputations
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.
Operator Use
Selection SELECT ... FROM t1 WHERE pred(t1.f1)
Projection SELECT f1 FROM t1
Unnesting SELECT ... FROM UNNEST (t1, .f1)
Aggregation SELECT ... FROM t1 GROUP BY t1.f1
Dupl. elimination SELECT DISTINCT f1 FROM t1
Recursion WITH RECURSIVE(...)
Cartesian product SELECT ... FROM (t1, t2)
Join SELECT ... FROM (t1, t2)
WHERE t1.f1 === t2.f2
Semi-join SELECT ... FROM t1 WHERE EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Anti semi-join SELECT ... FROM t1 WHERE
NOT EXISTS
(SELECT ... FROM t2
WHERE t1.f1 === t2.f2)
Difference SELECT ... FROM t1
EXCEPT SELECT ... FROM t2
Union SELECT ... FROM t1
UNION SELECT ... FROM t2
for resulting elements, defined by the type argument passed
to Observable. In case of selections,V is the input and output
type of the operator. The change propagation of selection is
comparatively simple: Selection only propagates a change
to its observers if the involved value satisfies the predicate.
Finally, selection registers itself as an observer of the under-lying
relation.
1 class Selection[V](rel: Relation[V], predicate: V = Boolean)
2 extends Observer[V] with Observable[V] { 3 def added(v: V) { if(predicate(v)) notify added(v) } 4 def removed(v: V) { if(predicate(v)) notify removed(v) } 5 def updated(oldV: V, newV: V) { 6 if(predicate(oldV)) { 7 if(predicate(newV)) notify updated(oldV, newV)
8 else notify removed(oldV) } 9 else if(predicate(newV)) notify added(newV) } 10 rel addObserver this
11 }
Figure 4: The incremental selection operator.
Join Operator The join operator is a binary operator that
correlates elements of a left-hand and right-hand side rela-tion.
The results are pairs of elements from the two rela-tions.

65. 21
Performance evaluation

66. 22
Implemented 15 FindBugs
JVM bytecode analyses in i3QL
Update
Change

67. 22
Implemented 15 FindBugs
JVM bytecode analyses in i3QL
Test data: Vespucci revision history
! 381 class files initially
! 242 changes (add, remove, update)
! 1 change affects 1 to 312 class files Update
Change

68. 22
Implemented 15 FindBugs
JVM bytecode analyses in i3QL
Test data: Vespucci revision history
! 381 class files initially
! 242 changes (add, remove, update)
! 1 change affects 1 to 312 class files
For each analysis
! Analyze initial 381 class files
! Replay changes in order
Update
Change

69. 23
Mean update time for all 15 analyses
0 50 100 150 200 250 300
0 50 100 150 200 250
Change size (in number of affected classes)
Time (ms)
(a) Update time per change.
same time for each revision, checking to about 130 seconds, whereas i3QL only than 2 seconds. This means that on our a speedup of 65x compared to FindBugs analyses.
Figure 13b shows the variation change. Like in the graphic above, size of one change. The y-axis shows less) memory is needed after the change We run the garbage collector after the of each change to get accurate measurements. shows that the memory impact of a most Update time per change
120 KiB for a change affecting more which means that auxiliary data stored does not have a significant the class files of the initial revision account

70. 23
0 50 100 150 200 250 300
0 50 100 150 200 250
Change size (in number of affected classes)
Time (ms)
(a) Update time per change.
same time for each revision, checking to about 130 seconds, whereas i3QL only than 2 seconds. This means that on our a speedup of 65x compared to FindBugs analyses.
Figure 13b shows the variation change. Like in the graphic above, size of one change. The y-axis shows less) memory is needed after the change We run the garbage collector after the of each change to get accurate measurements. shows that the memory impact of a most Update time per change
120 KiB for a change affecting more which means that auxiliary data stored does not have a significant the class files of the initial revision account Update time ~ linear in change size!
! ! Small change = small effect
! ! Larger change = larger effect
Mean update time for all 15 analyses

71. 23
0 50 100 150 200 250 300
0 50 100 150 200 250
Change size (in number of affected classes)
Time (ms)
(a) Update time per change.
same time for each revision, checking to about 130 seconds, whereas i3QL only than 2 seconds. This means that on our a speedup of 65x compared to FindBugs analyses.
Figure 13b shows the variation change. Like in the graphic above, size of one change. The y-axis shows less) memory is needed after the change We run the garbage collector after the of each change to get accurate measurements. shows that the memory impact of a most Update time per change
120 KiB for a change affecting more which means that auxiliary data stored does not have a significant the class files of the initial revision account Update time ~ linear in change size!
! ! Small change = small effect
! ! Larger change = larger effect
Mean update time for all 15 analyses
Comparison incremental vs nonincremental
FindBugs
i3QL sequential
i3QL incremental
130s
31s
2s
1 10 100
time (seconds), log-scale

72. 24
Effect of optimizations
time (seconds), log-scale
1000
100
10
1
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations

73. 24
Effect of optimizations
time (seconds), log-scale
1000
100
10
1
Speedup up to 575x
Slowdown of 4% in one case
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations

74. 24
Effect of optimizations
time (seconds), log-scale
1000
100
10
1
Speedup up to 575x
Slowdown of 4% in one case
When optimizations apply, huge effect
A B C D E F G H I J K L M N O
i3QL without optimizations i3QL with optimizations

75. 25
In the paper
• Detailed description of incremental operators

76. 25
In the paper
• Detailed description of incremental operators
• Case study:
Complete definition of incremental parser

77. 25
In the paper
• Detailed description of incremental operators
• Case study:
Complete definition of incremental parser
• Evaluation of memory consumption

78. 25
In the paper
• Detailed description of incremental operators
• Case study:
Complete definition of incremental parser
• Evaluation of memory consumption
• Detailed discussion of related work!
- embedded query languages!
- in-memory collections!
- constraints and functional dependencies!
- view maintenance in DB systems!
- optimization techniques

79. i3QL: Language-Integrated Live Data Views

80. integrated!! ! ! !
in-memory! ! ! ! !
incremental!! !
high-performance
applicable
Scala-embedded DSL!
collections API!
based on relational algebra
using algebraic laws and partial evaluation
used for FindBugs analyses on JVM byte code
https://github.com/seba--/i3QL/