Event Stream Processing with BeepBeep 3

Event Stream
Event Stream
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
SYLVAIN HALLÉ
Université du Québec
à Chicoutimi
Processing
Processing
with
with
CRSNG
NSERC

PART 1
PART 1
INTRODUCTION
INTRODUCTION

What is an event?
$
A unit of data produced by the execu�on or the
observa�on of a "system".
$$$$$
$$$$$
$$$$$
update of a stock price
sensor reading
link
state of a video game
user visi�ng a web page
parcel being delivered
/
move in a chess game

...and an event stream?
An ordered sequence of events of a given type
...

...
...

...
...
...

...
...
...
...

...
...
...
...
called a log
when stored
for later use

...
...
...
...
called a log
when stored
for later use
Events can be
composite: ...

Event stream processing
A calcula�on done on an event stream

what is the maximum
temperature of the last
100 readings?

what is the maximum
100 readings?
who is the employee
who delivered the most
parcels?

what is the maximum
100 readings?
who is the employee
parcels?
is there a user who
accessed page A from
page B?

what is the maximum
100 readings?
who is the employee
parcels?
is there a user who
page B?
$ has the stock decreased
three days in a row?

what is the maximum
100 readings?
who is the employee
parcels?
is there a user who
page B?
has Mario ever collided
with an enemy during
a jump?

what is the maximum
100 readings?
who is the employee
parcels?
is there a user who
page B?
a jump?
what is the move
taken most frequently
a�er e2 e4?

Similar to a database query, but:
o�en concerned with the order of occurrence
result is produced incrementally
what is the maximum
100 readings?
who is the employee
parcels?
is there a user who
page B?
a jump?
what is the move
taken most frequently
a�er e2 e4?

Stream vs. batch mode
BATCH STREAM
higher performance
one-shot opera�on
oﬄine results
incremental calcula�on
stateful
online results

Storing streams
Streams can be stored...
in a text ﬁle (JSON,
XML, CSV, etc.)
A B C
3.14
in a spreadsheet
in a database you don't store
them!
+

The right tool...
...for the right task

The right tool...
A B C
3.14

The right tool...
A B C
3.14
$ bash

The right tool...
A B C
3.14
(One example of this later)
$ bash

PART 2
PART 2
FUNDAMENTALS OF
BEEPBEEP
FUNDAMENTALS OF
BEEPBEEP

Introducing
BeepBeep is an open source library wri�en in Java
and developed at since 2015.
It can be used to perform stream processing in
any language that uses the JVM:
this presenta�on
h�ps://liﬂab.github.io/beepbeep-3

Introducing
A complete textbook is available under an open
access license
research papers
and case studies
downloads
Best Paper
Award
1000
1000
20
20
1
1
h�ps://bit.ly/beepbeep-book
Presses de l'Université du Québec

Design principles
Simple elementary building blocks
zero-conﬁgura�on

Design principles
Intui�ve simple mental model
few rules of opera�on

Design principles
Flexible composi�on
dozens of extensions

Design principles
Flexible composi�on
dozens of extensions
Lightweight low system requirements
core < 300 kb

Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)

Events
2
3
4
π abc
3 8 a
3 8 a
2 6 c
An event can be any (Java) object:
numbers Booleans strings tuples (key-value)
+
⊇?
...but also...
sets func�ons plots colors
chess
moves
. . .

Processor
A processor is a computa�on unit that ingests
input events and produces output events
P

Processor
P
processor

Processor
P
processor
input pipe

Processor
P
processor
input pipe
event

Processor
P
processor
input pipe
event
output pipe

Arity
Some processors have more than one input pipe...
"2:1 processor"

Arity
Some processors have more than one input pipe...
"2:1 processor"
"1:2 processor"
...others can have more than one output pipe

A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard

A source is a processor with no input pipe
Sources and sinks
3 1 4
4 1 5 9 2
queue source keyboard
A sink is a processor with no output pipe
queue sink print black hole

A processor can have pipes with diﬀerent types:
Pipe types
Each pipe receives / produces events of a speciﬁc
type. Color conven�ons are used for common types:
Booleans strings tuples
tables blobs
maps/sets
numbers

Basic processors
A set of "core" processors that perform elementary
opera�ons on streams.
n
P
{
f
f n n
Σ
f
n
k
n
fork ﬁlter window slice turn into
apply decimate trim cumulate preﬁx
f
P

ApplyFunction
Applies a func�on to events
of the input
f
f
f

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
square root

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
get a�ribute "A" from a tuple
tuple
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0

ApplyFunction
of the input
f
f
f
f
√
4 9 16 1 9 1
25 2 3 1 3 1
4 5
f a b z
A
3
8
a
A B C
0
5
b
A B C
1
2
z
A B C
f
4 0 21 3 1 0
-2 ⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
>
1 0
constant 0
the ﬁrst input pipe

ApplyFunction
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10

ApplyFunction
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E

ApplyFunction
of the input
f
f
f
f
+
4 9 16 1 9 1
25
4 0 21 3 1 0
-2 8 9 37 1
23
4 10
f
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
A B D F G
C E
a B d f G
c E
if-then-else

TurnInto
Turns every event into
k
k

TurnInto
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0

TurnInto
k
k
3 1 1 9 2
4 5 0 0 0 0 0 0
0 0
a a a a a a
a a
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥

CountDecimate
Keeps one event every
n
n

CountDecimate
Keeps one event every
n
n
a b d f g
c e a c g
e
2

Trim
Discards the ﬁrst events
n
n

Trim
Discards the ﬁrst events
n
n
3 1 1 9 2
4 5 1 5 2
9
3

Fork
Replicates the input stream into
each output pipe (>1)

Fork
Replicates the input stream into
each output pipe (>1)
3 1 1 9 2
4 5
3 1 1 9 2
4 5
3 1 1 9 2
4 5

Filter
Discards events from a stream
based on a control signal

Filter
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
a c f
d

Filter
⊤ ⊥ ⊤ ⊤ ⊥
⊤ ⊥
a b d f g
c e
a c f
d
The Boolean stream does not need to be related
to the ﬁltered stream

Cumulate
Applies func�on to current
input and last output, star�ng with
value
f
n
Σ
f
n

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
+

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events
⊥ ⊥ ⊥ ⊥ ⊤
⊥ ⊤ ⊥ ⊥ ⊥ ⊤ ⊤
⊥ ⊤
Σ
⊥
∧

Cumulate
value
f
n
Σ
f
n
3 1 1 9 2
4 5 3 4 9
8 14
Σ
0
+
23 25
sum of all
events
Σ
-∞
max
3 1 1 9 2
4 5 3 3 4
4 5 9 9
max of all
events
⊥ ⊥ ⊥ ⊥ ⊤
⊥ ⊤ ⊥ ⊥ ⊥ ⊤ ⊤
⊥ ⊤
Σ
⊥
∧
have we seen
⊤ so far?

Window
Runs processor on each
window of successive inputs
and returns the last event it produces
n
n
n
P
{
P

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
3 1 4 3 4 8

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
1 4 1 1 5 6

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
4 1 5 4 5 10

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
1 5 9 1 6 15

Window
n
n
n
P
{
P
n
3
{
Σ
0
+
3 1 1 9 2
4 5 8 6 10 15 16
sum of 3 successive events

Window
Note that can be any
processor (not limited to
aggrega�ons or numerical inputs)
n
n
P
{
P

Window
n
n
P
{
P
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Window
n
n
P
{
P
⊥ ⊤
⊥ ⊤ ⊤ ⊤
⊥ ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Window
n
n
P
{
P
⊥ ⊥
⊤ ⊥ ⊤ ⊤
⊤ ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Window
n
n
P
{
P
⊥ ⊥
⊥ ⊥ ⊥ ⊥
⊥ ⊥
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Window
n
n
P
{
P
⊥ ⊤
⊥ ⊥ ⊥ ⊤
⊥ ⊥
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Window
n
n
P
{
P
true if a window of 4 events
contains at least one ⊤
n
3
{
Σ
+
n
4
{
Σ
0
⊥
⊥ ⊤ ⊥ ⊥ ⊤
⊥ ⊥
∧
⊤ ⊤ ⊤
⊥

Pipelines
Complex calcula�ons can be achieved through
composi�on

Pipelines
composi�on
passing the output of a processor
to the input of another

Pipelines
composi�on
Doing so creates processor chains or "pipelines".
Example:
2
f
x2

Pipelines
composi�on
Example:
2
f
x2
2
1 3 4 5 9
1 25

Pipelines
composi�on
Example:
2
f
x2
2
1 3 4 5 9
1 25
the square of every
other number

Pipelines
Pipelines are one-way: events ﬂow from outputs
to inputs

Pipelines
Any output can be connected to any input, as long
as they have the same type

Pipelines
Any output can be connected to any input, as long
as they have the same type
many types can occur in the same pipeline

A few examples
BeepBeep's textbook comes with 150+ illustrated
code examples (in Java) available online
h�ps://liﬂab.github.io/beepbeep-3-examples

f
Σ
0
+
Σ
0
+
÷
1
What does this pipeline calculate?
A few examples

f
Σ
0
+
Σ
0
+
÷
1
A few examples
1
3 2 6 3

f
Σ
0
+
Σ
0
+
÷
1
A few examples
1
3 2 6 3
4
3 6 12 15

f
Σ
0
+
Σ
0
+
÷
1
A few examples
1
3 2 6 3
4
3 6 12 15
2
1 3 4 5

f
Σ
0
+
Σ
0
+
÷
1
A few examples
1
3 2 6 3
4
3 6 12 15
2
1 3 4 5
2
3 2 3 3

f
Σ
0
+
Σ
0
+
÷
1
A few examples
1
3 2 6 3
4
3 6 12 15
2
1 3 4 5
2
3 2 3 3
the running average of the input stream

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
2
3 2
1
3 2

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
1
1 3
2
1 6
1
2

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
2
6
2 3 4 32
3

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3
6
3
6 2 32
3
41
2

3
{
f
Σ
0
+
Σ
0
+
÷
1
A few examples
the running average
of the input stream
on each window of
3 successive events
1
3 2 6 3 2 4 3 3
2 3
32
3 32
3 32
3

A few examples
f
foo
1
Σ
0
+

A few examples
f
foo
1
Σ
0
+
read lines
from a ﬁle

A few examples
f
foo
1
Σ
0
+
split line into
word list

A few examples
f
foo
1
Σ
0
+
output each
word as an event

A few examples
f
foo
1
Σ
0
+
create a slice
for each word

A few examples
f
foo
1
Σ
0
+
count

A few examples
f
foo
1
Σ
0
+
keep last
event

A few examples
f
foo
1
Σ
0
+
the number of occurrences
of each word in a ﬁle
brown
fox
jumped
lazy
dog
quick
the
over
1
1
5
2
3
2
12
4

Slice
n
P
{
f
Splits an input stream into
sub-stream according to , and
runs on each sub-stream
P
f

Slice
n
P
{
f
Splits an input stream into
sub-stream according to , and
runs on each sub-stream
P
f
Example: mul�ple players rolling a dice

Slice
n
P
{
f
Suppose we have two func�ons
and to extract the contents
of a "dice event"
? ?

Slice
n
P
{
f
get the player get the dice value
? ?
Suppose we have two func�ons
and to extract the contents
of a "dice event"
? ?

Slice
n
P
{
f
n
{
?
f
?
Σ
0
+

Slice
n
P
{
f
n
{
?
f
?
Σ
0
+
for each player...

Slice
n
P
{
f
n
{
?
f
?
Σ
0
+
for each player...
get dice value

Slice
n
P
{
f
n
{
?
f
?
Σ
0
+
for each player...
get dice value sum

Slice
n
P
{
f
The output is a map that associates
each slice and the last event
produced by P

Slice
n
P
{
f
produced by P
1

Slice
n
P
{
f
produced by P
1
5
1

Slice
n
P
{
f
produced by P
5
5
1
5
1

Slice
n
P
{
f
produced by P
5
5
3
5
5
1
5
1

Slice
n
P
{
f
produced by P
5
5
5
5
5
3
5
5
1
5
1

Slice
n
P
{
f
produced by P
5
6
5
5
5
5
5
5
3
5
5
1
5
1

Slice
n
P
{
f
n
{
f
?
?
}
{
for each dice value...

Slice
n
P
{
f
n
{
f
?
?
}
{
get player

Slice
n
P
{
f
n
{
f
?
?
}
{
get player put into set

Slice
n
P
{
f
Slices correspond to dice values

Push vs. pull mode
The default state of a processor
is to sleep.
ZZ
Z

Push vs. pull mode
is to sleep.
ZZ
Z
It springs into ac�on when either:

Push vs. pull mode
is to sleep.
ZZ
Z
we give it a new input event to ingest
!
1

Push vs. pull mode
is to sleep.
ZZ
Z
! may in turn give events
to downstream
processors: push mode
1

Push vs. pull mode
is to sleep.
ZZ
Z
to downstream
1
we ask it for new output events to produce
2
?

Push vs. pull mode
is to sleep.
ZZ
Z
to downstream
1
we ask it for new output events to produce
2
may in turn ask events
to upstream
processors: pull mode
?

Pipelines
A pipeline is closed if either:

Pipelines
!
!
- all its downstream endpoints are sinks

Pipelines
- all its downstream endpoints are sinks
?
- all its upstream endpoints are sources

QueueSource
Outputs events from a queue
when pulled ( )
?

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25 ?

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25
4

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25
9

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25
16

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25
1

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25
25

QueueSource
when pulled ( )
?
4 9 16 1 9 1
25 ?
×

QueueSink
!
Accumulates pushed ( ) events
into a queue

QueueSink
!
into a queue
!
4

QueueSink
!
into a queue
4
!
4

QueueSink
!
into a queue
4

QueueSink
!
into a queue
4
!
9

QueueSink
!
into a queue
9
4
!
9

QueueSink
!
into a queue
9
4

QueueSink
!
into a queue
9
4
!
16

QueueSink
!
into a queue
9 16
4
!
16

QueueSink
!
into a queue
9 16
4

QueueSink
!
into a queue
9 16
4
!
1

QueueSink
!
into a queue
9 16 1
4
!
1

Synchronous operation
Processors operate in a "lockstep" fashion
consume input events one at a �me
produce output events one at a �me

3 1 1 9 2
f
+
4 5
2 7 8
1
Example in pull mode

3 1 1 9 2
f
+
4 5
2 7 8
1
?

3 1 1 9 2
f
+
4 5
2 7 8
1
?
?

3 1 1 9 2
f
+
4 5
2 7 8
1
?
3

3 1 1 9 2
f
+
4 5
2 7 8
1
?
?
3

3 1 1 9 2
f
+
4 5
2 7 8
1
?
2
3

3 1 1 9 2
f
+
4 5
2 7 8
1
?
3
2

3 1 1 9 2
f
+
4 5
2 7 8
1
5

3 1 1 9 2
f
+
4 5
2 7 8
1
?
1

3 1 1 9 2
f
+
4 5
2 7 8
1
?
?
1

3 1 1 9 2
f
+
4 5
2 7 8
1
?
7
1

3 1 1 9 2
f
+
4 5
2 7 8
1
?
1
7

3 1 1 9 2
f
+
4 5
2 7 8
1
8

3 1 1 9 2
f
+
4 5
2 7 8
1
?
4

3 1 1 9 2
f
+
4 5
2 7 8
1
?
?
4

3 1 1 9 2
f
+
4 5
2 7 8
1
?
1
4

3 1 1 9 2
f
+
4 5
2 7 8
1
?
4
1

3 1 1 9 2
f
+
4 5
2 7 8
1
?
8
1

3 1 1 9 2
f
+
4 5
2 7 8
1
?
1
8

3 1 1 9 2
f
+
4 5
2 7 8
1
9

3 1 1 9 2
f
+
4 5
2 7 8
1
?
5

3 1 1 9 2
f
+
4 5
2 7 8
1
?
?
5

3 1 1 9 2
f
+
4 5
2 7 8
1
?
×
5

3 1 1 9 2
f
+
4 5
2 7 8
1
5
×

ZZ
Z
A processor needs an event in each input pipe
to execute a computa�on step.

When this is the case, it can produce an
output event...

output event... or more than one output event...

output event... or more than one output event...
or nothing at all

ZZ
Z
Otherwise, incoming events are stored into internal
buﬀers un�l they can be consumed.

Otherwise, incoming events are stored into internal
buﬀers un�l they can be consumed.

f
+
Example in push mode

f
+
!
3

f
+
3

f
+
!
2
3

f
+
3
2

f
+
!
5

f
+
5

f
+
5
!
7

f
+
5
7

f
+
5
!
7
1

f
+
5
7
1

f
+
5
!
7
1
1

f
+
5
1
1
7

f
+
5
!
1
8

f
+
5 8
1

f
+
5 8
!
1
8

f
+
5 8
1
8

f
+
5 8
!
1
4
8

f
+
5 8
8
4
1

f
+
5 8
!
8
5

f
+
5 8 5
8

f
+
5 8 5
!
8
1

f
+
5 8 5
1
8

f
+
5 8 5
!
9

f
+
5 8 5 9

f
+
5 8 5 9
!
5

f
+
5 8 5 9
5

f
+
5 8 5 9
!
×
5

f
+
5 8 5 9
!
5
×

f
+
5 8 5 9 ×
5

BeepBeep programs
Crea�ng a BeepBeep program is made of 3 steps:
instan�a�ng the processors
1
connec�ng them
2
pushing/pulling events
3
Prerequisites
Java 8 (or higher)
Groovy 4.0 (or higher)
to run CLI scripts

"Hello world"
!
Hello world
@Grab("ca.uqac.lif.cep:beepbeep-3:0.11")
import ca.uqac.lif.cep.io.*
new Print() << "Hello world"

A simple program
@Grab("io.github.liflab:beepbeep-3:0.11")

A simple program
Automa�cally downloads the library
from the Central Repository
and installs it locally
(zero-conﬁgura�on)

A simple program
import ca.uqac.lif.cep.Connector
import ca.uqac.lif.cep.functions.*
import ca.uqac.lif.cep.tmf.*
import ca.uqac.lif.cep.util.*

A simple program
load BeepBeep
0

A simple program
dc = new CountDecimate(2)
load BeepBeep
0
2
dc

A simple program
sq = new ApplyFunction(Numbers.square)
load BeepBeep
0
2
f
x2
dc sq

A simple program
pr = new Print()
load BeepBeep
0
2
f
x2
dc sq pr

A simple program
pr = new Print()
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr

A simple program
pr = new Print()
Connector.connect(dc, sq)
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr

A simple program
pr = new Print()
Connector.connect(sq, pr)
load BeepBeep
instan�ate
processors
1
0
2
f
x2
dc sq pr

A simple program
pr = new Print()
load BeepBeep
instan�ate
processors
connect
1
2
0
2
f
x2
dc sq pr

A simple program
squares.gvy
pr = new Print()
load BeepBeep
instan�ate
processors
connect
1
2
0
2
f
x2
dc sq pr

Pipelines can be created using few lines of code
(1 per box / pipe)
A simple program
No installa�on required (@Grab)
No compila�on required: programs can be saved
to a text ﬁle and run immediately at the
command line*
*When using Groovy

This program does nothing ( ). It is missing
part : pushing / pulling events.
A simple program
3
ZZ
Z

A simple program
3
ZZ
Z
Two objects can be used:

A simple program
3
ZZ
Z
Pushable
!
reference to an
input pipe
used to push events
Pushable
!

A simple program
3
ZZ
Z
Pushable
!
reference to an
input pipe
used to push events
Pushable
! vs.
Pullable
?
reference to an
output pipe
used to pull events
vs.
Pullable
?

A simple program (cont'd.)
2
f
x2
dc sq pr

2
f
x2
dc sq pr
...
p = dc.getPushableInput()
p

2
f
x2
dc sq pr
...
p
2
1 3 4 5
p.push(1)
p.push(2)
p.push(3)
p.push(4)
p.push(5)

2
f
x2
dc sq pr
...
p
2
1 3 4 5
p.push(1)
p.push(2)
p.push(3)
p.push(4)
p.push(5)
push events
3

At the command prompt:

$ groovy squares.gvy

$ groovy squares.gvy
1,9,25

Another example
addition.gvy
A = new Fork()
B = new CountDecimate(3)
C = new ApplyFunction(Numbers.addition)
Connector.connect(A, 0, B, 0)
Connector.connect(B, 0, C, 0)
Connector.connect(A, 1, C, 1)
3
f
+
A
B
C

0
When a processor has
more than one input
or output pipe, the index
must be speciﬁed
1
0
1
0 0
A = new Fork()
Another example
addition.gvy
3
f
+
A
B
C

Syntactical shortcuts
Pipelines wri�en in Groovy can be shortened.

Connector.connect(A, B) A | B

Connector.connect(A, x, B, y) A[x] | B[y]

The | connec�ve returns the last processor of the
chain

chain
Connector.connect(A, B)
Connector.connect(B, C)
p = C
p = A | B | C

chain
p = C
p = A | B | C
Pushable objects do not need to be explicitly
obtained:

chain
p = C
p = A | B | C
Pushable objects do not need to be explicitly
obtained:
p = proc.getPushableInput()
p.push(x)
proc << x

3
f
+
A
B
C
A = new Fork()
p = A.getPushableInput()
p.push(0)
p.push(1)

3
f
+
A
B
C
A = new Fork()
C = A[0] | new CountDecimate(3) |
new ApplyFunction(Numbers.addition)
A[1] | C[1]
A << 0 << 1

Other processors
BeepBeep provides many other processors and
func�ons. Some examples:
SELECT * FROM foo
http://...
↑
freeze an
event
k
repeat each
event
k
insert an
event
read from an
HTTP request
read from a
DB query
serialize an
event

Other processors
BeepBeep provides many other processors and
func�ons. Some examples:
P
1 P
2 P
3
, ,
TANK
k
Moore machine detect peaks
pack events
tank pump splice sources

PART 3
PART 3
USE CASES
USE CASES

Chess games
Data ﬁle of 2 million chess game
summaries in PGN format [Event "Masters"]
[Site "Hampstead"]
[Date "1998.03.28"]
[Round "1"]
[White "Aagaard, J. (wh)"]
[Black "Panczyk, K. (bl)"]
[Result "1/2-1/2"]
[WhiteElo "2395"]
[BlackElo "2390"]
[ECO "A32"]
1. d4 Nf6 2. c4 e6
3. Nf3 c5 4. Nc3 cxd4
5. Nxd4 Bb4 6. Nb5 a6
7. Nd6+ Ke7 8. Nxc8+ Qxc8
9. Qb3 Nc6 ...

Chess games
[Site "Hampstead"]
[Date "1998.03.28"]
[Round "1"]
[Result "1/2-1/2"]
[WhiteElo "2395"]
[BlackElo "2390"]
[ECO "A32"]
1. d4 Nf6 2. c4 e6
3. Nf3 c5 4. Nc3 cxd4
5. Nxd4 Bb4 6. Nb5 a6
9. Qb3 Nc6 ...
Goal: measure ﬁrst-move
advantage (more wins for
white)

Count games according to
each possible result:
1-0
0-1
½-½
(White wins)
(Black wins)
(draw)
Chess games
[Site "Hampstead"]
[Date "1998.03.28"]
[Round "1"]
[Result "1/2-1/2"]
[WhiteElo "2395"]
[BlackElo "2390"]
[ECO "A32"]
1. d4 Nf6 2. c4 e6
3. Nf3 c5 4. Nc3 cxd4
5. Nxd4 Bb4 6. Nb5 a6
9. Qb3 Nc6 ...
Goal: measure ﬁrst-move
advantage (more wins for
white)

Chess games
f
f
?
1
Σ
0
+
[Result
1
A possible BeepBeep pipeline:

Chess games
f
f
?
1
Σ
0
+
[Result
1
read the ﬁle
line by line

Chess games
f
f
?
1
Σ
0
+
[Result
1
retain only those
containing the result

Chess games
f
f
?
1
Σ
0
+
[Result
1
slice according
to line content

Chess games
f
f
?
1
Σ
0
+
[Result
1
turn each line
into 1

Chess games
f
f
?
1
Σ
0
+
[Result
1
sum

Chess games
f
f
?
1
Σ
0
+
[Result
1
line count
for each score

Chess games
f
f
?
1
Σ
0
+
[Result
1
keep only
last event

Chess games
f
f
?
1
Σ
0
+
[Result
1
print to
console

Chess games
f
f
?
1
Σ
0
+
[Result
1
Output:
{
[Result "*"]=221.0,
[Result "1-0"]=852305.0,
[Result "1/2-1/2"]=690934.0,
[Result "0-1"]=653728.0
}

A program running on a Hadoop cluster processes
the 1.7 GB ﬁle in 26 minutes.*
The right tool... (cont'd)
h�ps://tomhayden3.com/2013/12/27/chess-mr-job
*

*
The BeepBeep pipeline on a desktop computer
takes...

*
The BeepBeep pipeline on a desktop computer
takes... 35 seconds.

Robots
Industrial robot moving on a shop ﬂoor, producing
an event stream of its current state
x y θ x y θ
, x y θ
, ...
θ
(x,y)

Robots
Industrial robot moving on a shop ﬂoor, producing
an event stream of its current state
x y θ x y θ
, x y θ
, ...
Goal: detect skidding
situa�ons
"large" diﬀerence between
orienta�on and mo�on

Robots
A possible pipeline:
1
f
−
f
x

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°
½
0

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°
½
0
½
0

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°
½
0
½
0
½

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°
½
0
½
0
½
½

Robots
1
f
−
f
x
0 0 30° ½ ¼ 32°
½
0
½
0
½
½
the x-distance traveled between
two successive events

1
f
−
f
x
1
f
−
f
y
f
atan2
Robots

1
f
−
f
x
1
f
−
f
y
f
atan2
Robots
Δx

1
f
−
f
x
1
f
−
f
y
f
atan2
Robots
Δx
Δy

1
f
−
f
x
1
f
−
f
y
f
atan2
Robots
Δx
Δy
arctan( )
Δx
Δy
-
Δy
Δx
θ'

Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |

Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
Δθ = |θ−θ'|

f
≥
1 k
Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
Δθ = |θ−θ'|

f
≥
1 k
if Δθ ≥ k
outputs ⊤
Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
Δθ = |θ−θ'|

Robots
An isolated skidding
event is considered
a false alarm

Robots
An isolated skidding
event is considered
a false alarm
Report only if skidding
lasts for at least n
successive events
×1 ×2 ×3

Robots
A possible pipeline (ﬁnal version):
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
f
≥
1 k

n
{
Σ
∧
⊤
Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
f
≥
1 k

n �mes in a row
outputs if Δθ ≥ k
⊤
n
{
Σ
∧
⊤
Robots
1
f
−
f
x
1
f
−
f
y
f
atan2
f
θ
f
−
| |
f
≥
1 k

Conclusion
Event streams can be found in a variety of
situa�ons
Processing them in real �me does not (always)
require a complex/distributed setup
The BeepBeep stream processing engine can
perform a wide range of calcula�ons on streams

WARP ZONE
WARP ZONE
To be continued...
There are many more things to explore!

Pale�e
Set of processors and func�ons,
centered around a par�cular use case
There are dozens of pale�es that
extend the set of core processors for
various scenarios

→
→
?
?
T
/a/b
//status/text()
=? Walker
*
/a/b
//status/text()
=? Blocker
*
*
Guards on transi�ons
are arbitrary func�ons
on events
States output values
of any type (Moore machine)
Finite-state (i.e. Moore) machines

Linear Temporal Logic
X
G F
U
1
<
Σ f
⊥
<
Σ f
⊥
F
G
=
= =
= Σ f
?
♣

Event Stream Processing with BeepBeep 3

Recommended

Recommended

More Related Content

Similar to Event Stream Processing with BeepBeep 3

Similar to Event Stream Processing with BeepBeep 3 (20)

More from Sylvain Hallé

More from Sylvain Hallé (20)

Recently uploaded

Recently uploaded (20)

Event Stream Processing with BeepBeep 3