1. Event Stream
Event Stream
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
Processing
Processing
with
with
CRSNG
NSERC

2. PART 1
PART 1
INTRODUCTION
INTRODUCTION

3. What is an event?
$
A unit of data produced by the execu�on or the
observa�on of a "system".
$$$$$
$$$$$
$$$$$
update of a stock price
sensor reading
link
state of a video game
user visi�ng a web page
parcel being delivered
/
move in a chess game

4. ...and an event stream?
An ordered sequence of events of a given type
...

5. ...and an event stream?
An ordered sequence of events of a given type
...
...

6. ...and an event stream?
An ordered sequence of events of a given type
...
...
...

7. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...

8. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...
called a log
when stored
for later use

9. ...and an event stream?
An ordered sequence of events of a given type
...
...
...
...
called a log
when stored
for later use
Events can be
composite: ...

10. Event stream processing
A calcula�on done on an event stream

11. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?

12. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?

13. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?

14. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?

15. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?

16. Event stream processing
A calcula�on done on an event stream
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?
what is the move
taken most frequently
a�er e2 e4?

17. Event stream processing
A calcula�on done on an event stream
Similar to a database query, but:
o�en concerned with the order of occurrence
result is produced incrementally
what is the maximum
temperature of the last
100 readings?
who is the employee
who delivered the most
parcels?
is there a user who
accessed page A from
page B?
$ has the stock decreased
three days in a row?
has Mario ever collided
with an enemy during
a jump?
what is the move
taken most frequently
a�er e2 e4?

18. Stream vs. batch mode
BATCH STREAM
higher performance
one-shot opera�on
offline results
incremental calcula�on
stateful
online results

19. Storing streams
Streams can be stored...
in a text file (JSON,
XML, CSV, etc.)
A B C
3.14
in a spreadsheet
in a database you don't store
them!
+

20. The right tool...
...for the right task

21. The right tool...
...for the right task

22. The right tool...
...for the right task
A B C
3.14

23. The right tool...
...for the right task
A B C
3.14

24. The right tool...
...for the right task
A B C
3.14

25. The right tool...
...for the right task
A B C
3.14

26. The right tool...
...for the right task
A B C
3.14

27. The right tool...
...for the right task
A B C
3.14
$ bash

28. The right tool...
...for the right task
A B C
3.14
$ bash

29. The right tool...
...for the right task
A B C
3.14
$ bash

30. The right tool...
...for the right task
A B C
3.14
(One example of this later)
$ bash

31. PART 2
PART 2
FUNDAMENTALS OF
BEEPBEEP
FUNDAMENTALS OF
BEEPBEEP

32. Introducing
BeepBeep is an open source library wri�en in Java
and developed at since 2015.
It can be used to perform stream processing in
any language that uses the JVM:
this presenta�on
h�ps://liflab.github.io/beepbeep-3

33. Introducing
A complete textbook is available under an open
access license
research papers
and case studies
downloads
Best Paper
Award
1000
1000
20
20
1
1
h�ps://bit.ly/beepbeep-book
Presses de l'Université du Québec

34. Design principles
Simple elementary building blocks
zero-configura�on

35. Design principles
Simple elementary building blocks
zero-configura�on
Intui�ve simple mental model
few rules of opera�on

36. Design principles
Simple elementary building blocks
zero-configura�on
Intui�ve simple mental model
few rules of opera�on
Flexible composi�on
dozens of extensions

37. Design principles
Simple elementary building blocks
zero-configura�on
Intui�ve simple mental model
few rules of opera�on
Flexible composi�on
dozens of extensions
Lightweight low system requirements
core < 300 kb

38. Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)

39. Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)
+
⊇?
...but also...
sets func�ons plots colors
chess
moves
. . .

40. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P

41. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor

42. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe

43. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe
event

44. Processor
A processor is a computa�on unit that ingests
input events and produces output events
P
processor
input pipe
event
output pipe

45. Arity
Some processors have more than one input pipe...
"2:1 processor"

46. Arity
Some processors have more than one input pipe...
"2:1 processor"
"1:2 processor"
...others can have more than one output pipe

47. A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard

48. A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard
A sink is a processor with no output pipe
queue sink print black hole

49. A processor can have pipes with different types:
Pipe types
Each pipe receives / produces events of a specific
type. Color conven�ons are used for common types:
Booleans strings tuples
tables blobs
maps/sets
numbers

50. Basic processors
A set of "core" processors that perform elementary
opera�ons on streams.
n
P
{
f
f n n
Σ
f
n
k
n
fork filter window slice turn into
apply decimate trim cumulate prefix
f
P

51. ApplyFunction
Applies a func�on to events
of the input
f
f
f

52. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5

53. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
square root

54. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5

55. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5

56. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

57. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
get a�ribute "A" from a tuple
tuple
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

58. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

59. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

60. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0

61. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe

62. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe

63. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the first input pipe

64. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10

65. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10

66. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10

67. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10

68. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E

69. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
if-then-else

70. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E

71. ApplyFunction
Applies a func�on to events
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E

72. TurnInto
Turns every event into
k
k

73. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

74. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

75. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

76. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

77. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

78. TurnInto
Turns every event into
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0
a a a a a a
a a
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥

79. CountDecimate
Keeps one event every
n
n

80. CountDecimate
Keeps one event every
n
n
a b d f g
c e a c g
e
2

81. CountDecimate
Keeps one event every
n
n
a b d f g
c e a c g
e
2

82. CountDecimate
Keeps one event every
n
n
a b d f g
c e a c g
e
2

83. CountDecimate
Keeps one event