When RV Meets CEP (RV 2016 Tutorial)

When RV Meets CEP...When RV Meets CEP...
A NEW TUTORIAL BY SYLVAIN HALLÉ
RUNTIME
VERIFICATION
COMPLEX EVENT
PROCESSING
CRSNG
NSERC
(C) 2016 LABORATOIRE D'INFORMATIQUE FORMELLE, UNIVERSITÉ DU QUÉBEC À CHICOUTIMI, CANADA
PRESENTED AT THE INTERNATIONAL CONFERENCE ON RUNTIME VERIFICATION, MADRID, SPAIN, SEPTEMBER 27TH, 2016
WORK FUNDED BY THE CANADA RESEARCH CHAIR IN SPECIFICATION, TESTING AND VERIFICATION OF SOFTWARE SYSTEMS
Can
these
two fields
benefit
from
each
other?

TODAY'S
PROGRAM
Part I
RV vs. CEP
Part II
A Fistful of Processors
Part III
The Monitor Strikes Back

public static void
main(String[] args) {
Map m = new HashMap();
...
Iterator i = m.iterator();
while (i.hasNext()) {
...
}
}

public static void
main(String[] args) {
Map m = new HashMap();
...
Iterator i = m.iterator();
while (i.hasNext()) {
...
}
}
HashMap.new = HashMap$1263
HashMap$1263.iterator() = Iterator$32045
HashMap$1263.contains(MyObject$26) = false
Iterator$32045
HashMap$1263.hasNext = true
...
Method
calls

Any call to next() must be
immediately preceded by a call to
hasNext() that returns true.
Two calls to remove() must be
separated by at least one call to next
().
Calls to an iterator's methods are
forbidden once its parent collection has
been modiﬁed by calls like put() or
remove().

start(vase,3,15).
bid(vase,15).
start(ring,5,30).
endOfDay.
bid(ring,32).
bid(ring,33).
bid(vase,18).
sell(vase).

start(vase,3,15).
bid(vase,15).
start(ring,5,30).
endOfDay.
bid(ring,32).
bid(ring,33).
bid(vase,18).
sell(vase).
Event name

start(vase,3,15).
bid(vase,15).
start(ring,5,30).
endOfDay.
bid(ring,32).
bid(ring,33).
bid(vase,18).
sell(vase).
Parameters

start(vase,3,15).
bid(vase,15).
start(ring,5,30).
endOfDay.
bid(ring,32).
bid(ring,33).
bid(vase,18).
sell(vase).
For a given item, any bid
must be higher than the
previous one.
Any bid must be higher
than the minimum price.
No bid can be placed
before a start event.

Server
Ajax
web
app
<message>
<name>CartAdd</name>
<CartId>123</CartId>
<Items>
<ItemId>42</ItemId>
<ItemId>23</ItemId>
</Items>
</message>
<message>
<name>CartAddResponse</name>
<CartId>123</CartId>
</message>

The
Beep
Store
GO
Sign inor regi ster
What is this?
Login
Ask for account
Contact us
Fault parameters
Search: Your Cart
Fault parameters
Don’t check Results’s type
In the detailed search form, sends an ItemSearch message without
checking that the Results element is an integer.
"Add to cart" enabled if item present in cart
Makes the "Add to cart" button available for items that are already in the
user's cart.
Message schemas
Cart manipulations
Highlights
documentation
Disables the
verification

1. The element must be an integer between 1 and 20.
2. The element is mandatory only if is present,
otherwise it is forbidden.
3. The request cannot be resent if its response is
successful.
4. must follow a successful Logi nResponse.
5. There can be at most one active cart ID per session key.
6. You cannot add the same item twice to the shopping cart.
Page
Page Resul t s
Logi n
Car t Cr eat e

The lock graph of a set of tasks
should not contain cycles
A communication window has three
phases: prep, active and cleanup
Every command must be responded
by either a success or a failure

System
Instrumentation
Trace
Events

System
Instrumentation
Trace
Events
Trace
validation

System
Runtime monitoring
Instrumentation

System
Runtime monitoring
Overhead
Instrumentation

hasNext
next
hasNext
Option #1 Finite-State Machines

hasNext
next
hasNext
Option #1 Finite-State Machines
...and extensions
1 2 3
groupStart(t)
init(p)∀p
QEA: Reger et al., TACAS 2015
groupEnd(t2)
phaseStart
4
run(p)
finish(p)
t2−t < 480

A call to next must be followed by a call
to hasNext
No CartCreate request can occur
before a LoginResponse message
A received order must eventually
be shipped
Three successive login attempts should
trigger an alarm
G (next → X hasNext)
¬ CartCreate U hasNext
G (receive → F ship)
G ¬(fail ∧ (X (fail ∧ X fail)))
Option #2 Linear Temporal Logic

G (∃ retAddrVal ∈ ./return-address : (
(./instruction = call) ∧ (¬ ((F( ((./instruction = mov)
∧ (./output/type = general-register)) →
(∃ regA ∈ ./output/name : (F (( ((./instruction = mov)
∧ (./output/type = general-register))
∧ (./input/type = litteral) ) →
(∃ regB ∈ ./output/name :
( ∃ constAddr ∈ ./input/value :
(F (((./instruction = cmp ) ∧
(./output/type = regA)) →
(∃ loc ∈ ./location : (F((
((./instruction = mov ) ∧
(./output/type = general-register)) ∧
(./output/name = regA ) ) ∧
((./input/name = regB ) ∧ (./input/type = ptr)))))) ))))) )))
U ((./instruction = return)
∧ (./fonction-returned = retAddrVal)) )) ))
...and extensionsLTL-FO+, Hallé et al.
IEEE Trans. Serv. Comput. 2012

νZ.(
∨a ∈ A
a ⊤ ∧
∧a ∈ A
[a] Z )
Option #4 mu-calculus
(hasNext+ next)*
Option #3 Regex

class SeqMonitor extends EvrMonitor {
val SeqStart, SeqDone = fact
def seq name(s:String) = words(s)(2)
"start_seq" −− EVR(’id -> "EVR_SEQ_START", ’sclk -> ’S, ’msg -> ’M) 7−-> {
val w = words(’M.s)
val seq name = w(15).slice(1, w(15).length−2)
insert (SeqStart(seq name, ’S.d))
}
"end_seq_ok" −− EVR(’id -> "EVR_SEQ_SUCCESS", ’sclk -> ’E, ’msg -> ’M)
& ’SeqStart(’name, ’S) --> {
if (seq name(’M.s) == ’sname.s) {
replace (SeqStart)(SeqDone(’name.s, ’S.d, ’E.d, "OK"))
}
}
"end_seq_fail" −− EVR(’id -> "EVR_SEQ_FAILURE", ’sclk -> ’F, ’msg -> ’M)
& ’SeqStart(’name, ’S) --> {
if (seq name(’M.s) == ’name.s) {
replace (SeqStart)(SeqDone(’name.s, ’S.d, ’F.d, "FAIL"))
}
}
"print" −− SeqDone(’name, ’S, ’E, ’stat) --> {
updateCSV(’name.s, ’S.d, ’E.d, ’ stat .s)
remove(SeqDone)
}
}
Option #5 LogFire

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
10/2/29 APPL 1212.2
10/2/29 MSFT 1031.6
10/2/29 GOGL 341.1
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/4/29 APPL 1164.3
10/4/29 MSFT 1025.1
10/4/29 GOGL 345.4
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
10/2/29 APPL 1212.2
10/2/29 MSFT 1031.6
10/2/29 GOGL 341.1
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/4/29 APPL 1164.3
10/4/29 MSFT 1025.1
10/4/29 GOGL 345.4
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
Timestamp

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
10/2/29 APPL 1212.2
10/2/29 MSFT 1031.6
10/2/29 GOGL 341.1
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/4/29 APPL 1164.3
10/4/29 MSFT 1025.1
10/4/29 GOGL 345.4
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
Stock symbol

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
10/2/29 APPL 1212.2
10/2/29 MSFT 1031.6
10/2/29 GOGL 341.1
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/4/29 APPL 1164.3
10/4/29 MSFT 1025.1
10/4/29 GOGL 345.4
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
10/3/29 GOGL 362.2
10/3/29 MSFT 1024.4
10/3/29 APPL 1179.0
Price

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
What is the closing price of MSFT
for the ﬁrst ﬁve trading days?
Select all days after the 100th
where MSFT closed over $50.
On every 5th day, calculate the
average closing price for MSFT for
the last 5 days.
What are the stocks that closed
higher than MSFT for a given day?

$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
$$$$$$$$$$$
$$$$$$$$$$
What is the closing price of MSFT
for the ﬁrst ﬁve trading days?
Select all days after the 100th
where MSFT closed over $50.
On every 5th day, calculate the
average closing price for MSFT for
the last 5 days.
What are the stocks that closed
higher than MSFT for a given day?
Snapshot query
Landmark query
Sliding query
Join query

4948
4949
6.3132
Drug
prescriptions

4948
4949
6.3132
MSH|^~&|SOURCE|383018129|PRIORITY
HEALTH|382715520|
2007100914484648||ORU^R01|
0129938170710091448|P|2.3|PID|1|
1034157|012993817||
LASTNAME^FIRSTNAME||19520101|M|||
1234 MAIN^^DEARBORN
HEIGHT^MI^48127||||||||PID|1||
94000000000^^^Priority Health||
LASTNAME^FIRSTNAME||19400101|F|
PD1|1|||
1234567890^DOCLAST^DOCFIRST^M^
^^^^NPI|OBR|1|||80061^LIPID
PROFILE^CPT-4||20070911||||||||||OBX|1|
NM|13457-7^LDL (CALCULATED)^LOINC|
49.000|MG/DL| 0.000 - 100.000|N|||F|
OBX|2|NM|
2093-3^CHOLESTEROL^LOINC|138.000|
MG/DL|100.000 - 200.000|N|||F|OBX|3|
NM|2086-7^HDL^LOINC|24.000|MG/DL|
45.000 - 150.000|L|||F|OBX|4|NM|
2571-8^TRIGLYCERIDES^LOINC|324.000|
HL7HL7

Notify me when two out of three
successive data points lie more than
1 standard deviation from the mean
on the same side of the mean line.

Notify me when two out of three
successive data points lie more than
1 standard deviation from the mean
on the same side of the mean line.
Trend query

TOLL
TollId
Toll
License
State
Model
48392
$3.50
QF5010
KW
T

TOLL
Compute the toll of each booth
over 3-minute intervals

Select AVG(closingPrice)
From ClosingStockPrices
Where stockSymbol = ‘MSFT’
for (t = ST; t < ST+50, t+= 5) {
WindowIs(ClosingStockPrices, t - 4, t);
}
Select c2.*
FROM
ClosingStockPrices as c1,
ClosingStockPrices as c2
WHERE c1.stockSymbol = ‘MSFT’ and
c2.stockSymbol!= ‘MSFT’ and
c2.closingPrice > c1.closingPrice and
c2.timestamp = c1.timestamp
for (t = ST; t < ST +20 ; t++ ){
WindowIs(c1, t - 4, t);
WindowIs(c2, t - 4, t);
}
Two queries in
the language of
TelegraphCQ

SELECT * FROM
FILTER {cnt >= 10}(
(SELECT *, 1 AS cnt FROM
FILTER {contains(summary,’iPod’) = 1}(webfeeds))
FOLD {, $.cnt < 10 AND DUR < 1 DAY, $.cnt + 1 AS cnt}
(SELECT * FROM
FILTER {contains(summary,’iPod’) = 1}(webfeeds))
)
PUBLISH ipod_popularity
A query in the language of
Cayuga

from inputtrace#window.length(10)
select stockSymbol, timestamp, avg(closingPrice) as avg10
insert into avg10trace;
from inputtrace#window.length(20)
select stockSymbol, timestamp, avg(closingPrice) as avg20
insert into avg20trace;
@info(name = 'query1')
from avg10trace#window.length(10) join
avg20trace#window.length(20)
on avg10trace.timestamp == avg20trace.timestamp
select avg10trace.stockSymbol as stockSymbol,
avg10trace.timestamp, avg20trace.timestamp as ts2,
avg10trace.avg10 as avg10, avg20trace.avg20 as avg20
having avg10 > 50 or avg20 > 50
insert into output;
SiddhiQL...

CEPRV
event traces
"properties"
tuple-ish
overhead
event streams
"queries"
tuples*
throughput
=When

CEPRV
outputs T/F
result cannot be reused
in another property
mostly variants of FSM
and logic
good for sequential
patterns
not quite
When
outputs event streams
result can be reused as
input of another query
mostly "extensions" of
SQL
not quite
good for data
manipulation

CEPRVBeyond &
Both RV and CEP assume events to be "tuples":
associative maps from strings to single scalar values
start(vase,3,15).
bid(vase,15).
start(ring,5,30).
3/10/29 MSFT 1059.2
3/10/29 APPL 365.4
3/10/29 GOGL 1120.1
HashMap.new =
HashMap$1263
HashMap$1263.hasNext =
true
TollId
Toll
License
State
Model
48392
$3.50
QF5010
KW
T

CEPRVBeyond &
start(vase,3,15).
bid(vase,15).
start(ring,5,30).
3/10/29 MSFT 1059.2
3/10/29 APPL 365.4
3/10/29 GOGL 1120.1
HashMap.new =
HashMap$1263
true
TollId
Toll
License
State
Model
48392
$3.50
QF5010
KW
T
And what if events
are not tuples?

CEPRVBeyond &
start(vase,3,15).
bid(vase,15).
start(ring,5,30).
3/10/29 MSFT 1059.2
3/10/29 APPL 365.4
3/10/29 GOGL 1120.1
HashMap.new =
HashMap$1263
true
TollId
Toll
License
State
Model
48392
$3.50
QF5010
KW
T
And what if events
are not tuples?
And what if events
are not tuples?
We make them
ﬁt!

CEPRVBeyond &
Suppose each event is a set:
{6,5,3,9}
{5,4,10}
...
We can "ﬁt" it into a tuple:
a
6
a=4 ∨ b=4 ∨ c=4 ∨ d=4
c
3
d
9
Imposes upper
bound on set size
But then 4 ∈ S becomes:
b
5

CEPRVBeyond &
Both RV and CEP rely on monolithic
speciﬁcation/query languages
Everything must be written
in that one language (FO-LTL,
QEA, SiddhiQL, etc.)
The language must
accommodate every possible
use case
In general, no support for
extensions

Part 2:
A Fistful of
Processors

Event stream query engine developed
based on the previous observations
Aims at borrowing strengths from both
RV and CEP (and beyond)
Key concepts: composability, modularity,
extensibility
Open source, developed in Java
http://liflab.github.io/beepbeep-3
peeB peeB 3

EventsEvents
An event is an element e taken from
some set E, called the event type.
No restriction on the type! In BeepBeep,
events can be any Object.
Booleans Numbers
2
3
4
π
Strings
abc
Functions Sets PlotsTuples
3 8 a
3 8 a
2 6 c
+
⊇?
XML
documents
<a><a><a>
. . .
?

TracesTraces
An event trace (or event stream) is a potentially
infinite sequence of events of a given type:
2 0 6 3
4 9 . . .
Traces are symbolically denoted by:
e = e0 e1 e2 e3 ...
The set of all traces of type T is denoted as:
T*

FunctionsFunctions
A function takes 0 or more events as its
input, and returns 1 or more events.
Functions are first-class objects; they descend
from the class Function
1 : 1
function
2 : 1
function
1 : 2
function
⊇?
3
4
2+5i
₹
2 5
6
0 : 1
function
6

ProcessorsProcessors
A processor takes 0 or more event traces as
its input, and returns 0 or more event traces as
its output
1 : 1 processor
2 : 1 processor
. . . . . .

ProcessorsProcessors
When a processor takes more than one input
trace, the set of events at matching positions
in each trace is called a front.
bacd
3601
b 3
a 6
c 0
d 1
1st event
2nd
3rd
4th
. . .
. . . . . .

Synchronous processingSynchronous processing
Events are processed one front at a time.
+
Buffers collect events until a complete
front can be processed.
⇒

+
⇒
5
3

+
⇒
5
3
+

+
⇒
8

+
⇒
1

+
⇒
6
1

+
⇒
6
1
4

+
⇒
6
1
4
+

+
⇒
6
5

+
⇒
6

Makes a couple of things simpler
Don't care about what event arrived
first or upstream computation time
"Pen and paper" calculation is identical
to the real one
Otherwise, can do a lot with simple
timeouts ⇒ contained asynchrony
Motto: Don't use asychronous processing...

Makes a couple of things simpler
Don't care about what event arrived
first or upstream computation time
"Pen and paper" calculation is identical
to the real one
Otherwise, can do a lot with simple
timeouts ⇒ contained asynchrony
Motto: Don't use asychronous processing...
...unless you really have to

In BeepBeep, all synchronous processors
are descendents of the SingleProcessor
class
Takes care of handling input/output
buffers
Calls (abstract) method process() when
an input front is ready to be consumed
Processor only needs to produce an
output front from this input
Makes it easy to create your own
(more on that later)

A high-level event trace can be produced by
composing ("piping") together one or more
processors from lower-level traces
CompositionComposition

Each processor has its own input/output
buffers

Any output can be connected to any input, as
long as they have the same type

Any output can be connected to any input, as
long as they have the same type
Many types can occur in the same chain

Trace Manipulation
Functions/Processors
ArchitectureArchitecture
TMF/P EMF/P
Event Manipulation
Type-agnostic
Stateful
Type-dependent
Stateless

Trace Manipulation
TMF/P EMF/P
Event Manipulation
Type-agnostic
Stateful
Type-dependent
Stateless
Not clear-cut, but general design goal

BeepBeep provides only a few built-in
processors and functions
Palette
Set of processors and functions,
centered around a particular
use case
Concretely, a JAR library
defining new Processor and
Function objects

<?
+
<? =?
−
÷ ×
f Σ f
n
{
n
n
Function Cumulative Trim
ForkDecimate Group
WindowSliceFilter
Built-in processors
Built-in
functions
.n
<
<

SemanticsSemantics
Let P be a processor and a
b
c
= a1,a2,...
= b1,b2,...
= c1,c2,... be traces
a,b,c : P[[
n
= e1,e2,...
denotes the n-th output trace of P, given
traces a, b, c as input.

f
FunctionFunction
Applies an n-ary function f to
every front of size n
"Lifts" any function into a
processor
a,b : f[[
n
= f n
(ai,bi)

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output
i
The i-th event

f
FunctionFunction
processor
a,b : f[[
n
= f n
(ai,bi)
The n-th output
i
The i-th event
f
+
f
<0?
Pairwise sum of
events
Is each event
negative?

CumulativeCumulative
Applies a 2 : 1 function f to
its previous value and the
current event
a : Σf[[ = f(x,a1), f(f(x,a1),a2), ...
+
<Sum of all
events
Have we seen
⊤ so far?
Σ f
x
Σ f Σ f
0 ⊥

TrimTrim
Returns the input trace,
trimmed of its first n events
a : [[ = an+1, an+2, ...
n
n

TrimTrim
Returns the input trace,
trimmed of its first n events
a : [[ = an+1, an+2, ...
n
n
1
f
=?=?
<
Σ f
⊥
Does the same
number
repeat twice in
a row?

GroupGroup
Makes a set of connected
processors behave as a single
processor

GroupGroup
Makes a set of connected
processors behave as a single
processor
1
f
=?=?
<
Σ f
⊥

ForkFork
Replicates its input on each
of its n outputs
a : Ψ[[ = a1, a2, ...
n

DecimateDecimate
Outputs every n-th input
event
a : [[ = a1, an+1, a2n+1, ...n
Ψ
n

FilterFilter
A n : n-1 processor; outputs
the n-1 first components of a
front if its last component is ⊤;
otherwise, discards it
a , a , ..., a : =[[ 2
k1 n
F
i
{a if a =⊤
ε otherwise
k
i
n
i
Powerful mechanism: "anything can
be filtered" (don't care about condition)
Boolean trace does not need to come
from the same source as the inputs being
filtered

FilterFilter
A n : n-1 processor; outputs
the n-1 first components of a
front if its last component is ⊤;
otherwise, discards it
f
<0?
Get only the negative
events of the input

WindowWindow
Returns the output of a
processor P on a sliding
window of width n
a : Υ =[[ P i
be windowed" (don't care about function)
n
{
n
ai, ... ai+n : P[[ *

WindowWindow
window of width n
a : Υ =[[ P i
be windowed" (don't care about function)
n
{
n
ai, ... ai+n : P[[ *
The last event

WindowWindow
window of width n
n
{
3
{+
Σ f
0

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 3 4 2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 3 24

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 243

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
2
243

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
2
243
4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
6
243
2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
6
243
23

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2
3 2 1

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1
9
243
6 2
9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 243 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
1 43 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
43

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
7 4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
7 41

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 91
8 7 4

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 81
8 7 4
9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
43 81 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
31 8 9

WindowWindow
window of width n
n
{
3
{+
Σ f
0
31 8 9
The sum of all 3
successive events

SliceSlice
Dispatches an event e to a
distinct instance of processor
P according to the value of
some function f
{a if f(ai)=k
ε otherwise
i
πf
k[a : =[i
P
f[a : =[i U πf
k[a1,...ai : [i[ [: P
k *
+

SliceSlice
Dispatches an event e to a
distinct instance of processor
P according to the value of
some function f
{a if f(ai)=k
ε otherwise
i
πf
k[a : =[i
P
f[a : =[i U πf
k[a1,...ai : [i[ [: P
k *
+
Multiset union

f(x) = x mod 2
πf
0[a : =[subtrace of even numbers
πf
1[a : =[subtrace of odd numbers

f(x) = x mod 2
πf
πf
mod 2

f(x) = x mod 2
πf
πf
mod 2
5

f(x) = x mod 2
πf
πf
mod 2
5
U+

f(x) = x mod 2
πf
πf
mod 2
5
U+
{9}{5}

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}6

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}
6

f(x) = x mod 2
πf
πf
mod 2
5
{9}{5}
6
U+

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
U+
{9}{5}

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
{9}{5}1

f(x) = x mod 2
πf
πf
mod 2
5
{9}{6,5}
6
{9}{5}
1

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,5}
6
{9}{5}
5
U+

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,1}
6
{9}{6,5}
5
U+
{5}

f(x) = x mod 2
πf
πf
mod 2
1
{9}{6,1}
6
{9}{6,5}
5
U+
{5}
The last odd and even
numbers seen so far

f(x) = x mod 2
πf
πf
mod 2
+
Σ f
0

f(x) = x mod 2
πf
πf
mod 2
+
Σ f
0
The sum of all odd
numbers and all even
numbers seen so far
{9}{6,6} {9}{6,5} {5}

Input/outputInput/output
0 : 1 processors can be used to produce an
event trace out of an external source (i.e.
standard input, a file, etc.)
Ditto for 1 : 0 processors
a . . .b
a . . .b

PalettesPalettes
BeepBeep provides only a few built-in
processors and functions
Palette
Set of processors and functions,
centered around a particular
use case
Concretely, a JAR library
defining new (reusable!)
Processor and Function
objects

XMLXML
Provides two new Functions
Ditto for the JSON library (using json-lif)
XML parser: converts String events
into XMLElement events (from
xml-lif library)
XPath: evaluates an XPath expression
on an XMLElement (result is a set of
XMLElements)

SetsSets
New Functions:
New processor:
Multiset operations: add/remove
elements, membership, union/intersection
→ →
Dropper
Outputs each element of a
multiset as an event

GnuplotGnuplot
New Functions:
New processor:
Scatterplot: converts a set of (x,y) pairs
into a Gnuplot string producing a plot
(ditto for Histogram)
→ →
Gnuplot Caller
Calls Gnuplot on a string and
returns its output (i.e. a binary
string)

TuplesTuples
New event type: tuple
New function: SELECT
Creates an output tuple by combining
attributes from input tuples
Grammar extension: allow eSQL to use a
SELECT statement, backward-compatible with
SQL
3 8 a
3 8 a
2 6 c

TuplesTuples
SELECT <tuple> FROM <processor-list>
* | <tuple-element> | <tuple-element> , <tuple>
<tuple-name> <rename> | <function-call>
<rename>
<word>.<word> | <word>
<processor> <rename> | <processor>
<rename> , <processor-list>
AS <eml-attribute>
<select> ::=
<tuple> ::=
<tuple-element> ::=
<tuple-name> ::=
<processor-list> ::=
<rename> ::=

FSMFSM
New processor: finite-state machine
→
→
?
?
T
/a/b
//status/text()
=? Walker
*
/a/b
//status/text()
=? Blocker
*
*
Guards on transitions
are arbitrary functions
on events
States output values
of any type (Moore machine)

Linear Temporal LogicLinear Temporal Logic
New processors for LTL temporal operators
and first-order quantification (LTL-FO+)
Two semantics for a formula φ:
Boolean: the i-th output is the (2-valued)
verdict of evaluating φ, starting at the i-th
event
⇒ filter
Troolean: the i-th output is the (3-valued)
verdict of evaluating φ up to the i-th event
⇒ monitor

Consider the LTL formula F b on the
trace
In a monitoring context, we want the
output of processor F b to be:
ac c b
F b
"Tell me whether
eventually b"

trace
ac c b
F b
"Tell me whether
eventually b"
c

trace
ac c b
F b
"Tell me whether
eventually b"
?

trace
ac c b
F b
"Tell me whether
eventually b"
?c

trace
ac c b
F b
"Tell me whether
eventually b"
? ?

trace
ac c b
F b
"Tell me whether
eventually b"
? ?b

trace
ac c b
F b
"Tell me whether
eventually b"
⊤ ? ?

trace
ac c b
F b
"Tell me whether
eventually b"
⊤ ? ?a

trace
ac c b
F b
"Tell me whether
eventually b"
⊤ ⊤ ? ?

trace
In a filtering context, we want the
ac c b
F b
"Get me all the events
where eventually b"

trace
ac c b
F b
where eventually b"
c

trace
ac c b
F b
where eventually b"
b

trace
ac c b
F b
where eventually b"
⊤ ⊤ ⊤

trace
ac c b
F b
where eventually b"
a ⊤ ⊤ ⊤

Boolean operators are easy
X
G F
U
1
<
Σ f
⊥
<
Σ f
⊥
F
G
=
= =
= Σ f
?
♣

First-order quantifiers
Processor to run
on each slice
→
A
→ x →
E
→ x
Quantified
variable
(Arbitrary) domain
function
Variable is added to
the processor's context
Boolean and Troolean
versions

Let's put it all
toghether!
i.e. a few examples of
queries from past publications

Use BeepBeep as
a library in your
program

→
Fork f = new Fork(2);
Use BeepBeep as
a library in your
program

→
→→
→
FunctionProcessor sum =
new FunctionProcessor( );
f
Use BeepBeep as
a library in your
program

→
→→
+
→
new FunctionProcessor(Addition.instance);
f
Use BeepBeep as
a library in your
program

→
→→
+
→
CountDecimate decimate = new CountDecimate(n);
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→→
Connector.connect(fork, LEFT, sum, LEFT)
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(fork, RIGHT, decimate, INPUT)
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(decimate, OUTPUT, sum, RIGHT);
f
n
Use BeepBeep as
a library in your
program

→
→→
+
→ →→
.connect(decimate, OUTPUT, sum, RIGHT);
Pullable p = sum.getOutputPullable(OUTPUT);
while (p.hasNext() != NextStatus.NO) {
Object o = p.next();
...
}
f
n
Use BeepBeep as
a library in your
program

→
→ →
^
n
→
→ →
+
→
→
→
→
→
÷
→ →1 → →
+
Σ
Σ
0
0
The statistical moment of order n

→
→ →
^
n
→
→ →
+
→
→
→
→
→
÷
→ →1 → →
+
Σ
Σ
0
0
The statistical moment of order n
→ →E(x)
n
As a group
processor

→
→
→
→
1
→ →
>
Trigger an alarm when two successive
events

→
→
→
1
→
→
→
→
→
<?
→
→
→
1
→ →
>
events are more than 1

→
→
→
→
1
→ →
→
÷
→
→
→
→
→
<?
→
→
→
1
→
→ →σ
→
>
events are more than 1 standard
deviation

→
→
→
→
→
1
→ →
→
-
→ →
→
÷
→
→
→
→
→
<?
→
→
→
1
→
→ →E(x)
1
→ →σ
→
>
events are more than 1 standard
deviation from the mean

→
⊇?
→ → → A →→
T
??...<a> <a> ...
/a/b
//character/id/text()
→
→

→
→
?
?
T
/a/b
//status/text()
=? Walker
*
/a/b
//status/text()
=? Blocker
*
*
→→ →
/a/b
//character
→
→→
A
/a/b
//id/text()
→
T
??...<a> <a> ...

→
A
/a/b
//character[status=Walker]/id/text()
→ p1
→
A
→ p2
→ →
→
→
/a/b
//character[status=Blocker]/id/text()
→ →
→
3
→
→
→
<?
→
→
→
→
→
→
→
→
f1
f2
→
→ →
→
→
→
→
/a/b
//character[id=p1]/position/x/text()
/a/b
-
|...|
<?
6
>
...
f1
/a/b
/a/b
-
|...|
f2

→
→
*
*
*
*
Create Auction
=?
0
@
Last Price
Days
0:=
3@Max. Days :=Min. Price 2@:=
Days :=
Days 1
+
End of Day
=?
0
@
>
<?
Days
Last Price
2
@:=
Bid
=?
0
@
>
Min. Price
<?
2
@
Last Price
>?
2
@
Bid
=?
0
@
>
Last Price
>?
2
@
Days :=
Days 1
+
End of Day
=?
0
@
Max. Days
→
End of Day
*
1@
*
→
→→
→
*
Sold
=?
0
@
Days
Days
+ | |.÷

→
→
1@
+=
Sanitise
=?
0
@
Clean :=
Derive
=?
0
@
>
∊ ?
Clean
T
??...Derive(a,b)
Clean
2@
+=Clean :=
Clean
1
@
Use
=?
0
@
>
∊ ?
Clean
1
@
T *
*
T
...

Fastest car in the
world
Slow on every
type of road

Fastest car in the
world
Slow on every
type of road
Only runs on a
closed circuit

Fastest car in the
world
Slow on every
type of road
Only runs on a
closed circuit
Works on every
type of road

Fastest car in the
world
Slow on every
type of road
Only runs on a
closed circuit
Works on every
type of road
Heavy setup required

Fastest car in the
world
Slow on every
type of road
Only runs on a
closed circuit
Works on every
type of road
Heavy setup required Turn the key

PART THREE
The Monitor
Strikes Back

import ca.uqac.lif.cep.*;
public class MyProcessor extends SingleProcessor {
public Queue<Object[]> compute(Object[] inputs)
{
}
public void build(Stack<Object> s)
{
}
}
. . . Create output events from input . . .
. . . Instantiate processor from parse stack . . .
<processor> := . . .
<number> := . . .
<string> := . . .
Add new rules to any symbol from
the basic grammar

Example: let us create a new 1 : 1 Processor
that increments its input by a fixed value
We would like to use it in eSQL with this
syntax:
INCREMENT ( P ) BY value
Any expression
deﬁning a processor
Number

<processor> := <my_processor> ;
<my_processor> := INCREMENT ( <processor> ) BY <number> ;
Symbols already defined in basic grammar
Adds a new case to an existing rule

public class MyProcessor extends SingleProcessor {
private int increment;
public Queue<Object[]> compute(Object[] inputs)
{
Queue<Vector<Object>> out = new Queue<Vector<Object>>();
Object[] v = new Object[1];
Integer i = (Integer) inputs[0] + increment;
v[0] = i;
out.put(v);
return out;
}
. . .

. . .
{
Number n = (Number) s.pop();
s.pop();
s.pop();
Processor p = (Processor) s.pop();
s.pop();
s.pop();
increment = n.intValue();
Connector.connect(p, this);
s.push(this);
}
}
Read contents of
parse stack
<number>
BY
(
<processor>
)
INCREMENT
Set processor's state
Pipe it to its input
Put on parse stack

Total: 6+9=15 lines of code
. . .
{
Number n = (Number) s.pop();
s.pop();
s.pop();
Processor p = (Processor) s.pop();
s.pop();
s.pop();
increment = n.intValue();
Connector.connect(p, this);
s.push(this);
}
}
Read contents of
parse stack
<number>
BY
(
<processor>
)
INCREMENT
Set processor's state
Pipe it to its input
Put on parse stack

Some pre-packaged grammar extensions:
Manipulation of name-value tuples
Set theory
Formatted input (CSV, XML, JSON)
Graphing (histograms, scatterplots, ...)
Basic signal processing (smoothing,
peak detection, ...)
Create your own!

import ca.uqac.lif.cep.eml.tuples.*;
public class MyExample {
public static void main(String[] args)
{
Interpreter my_int = new Interpreter();
my_int.extendGrammar(TupleGrammar.class);
Pullable p = my_int.executeQuery(
""HELLO WORLD"");
for (int i = 0; i < 10; i++)
{
EmlString s = (EmlString) p.pull();
System.out.println(s);
}
}
}
Create query interpreter
Load a grammar extension
Execute a query
Pull an output event

Example: let us create a new 1 : 1 Processor
that parses XML events (using an existing
XML library)
This can be done by first defining a Function
that converts Strings into XML objects

import ca.uqac.lif.xml.*;
public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> {
public static XmlParsingFunction instance = new XmlParsingFunction();
private XmlParsingFunction() {
super(String.class, XmlElement.class);
}
public XmlElement getValue(String x) {
try {
return XmlElement.parse(x);
}
catch (XmlParseException e) {
}
return null;
}
}
}

import ca.uqac.lif.xml.*;
public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> {
public static XmlParsingFunction instance = new XmlParsingFunction();
private XmlParsingFunction() {
super(String.class, XmlElement.class);
}
public XmlElement getValue(String x) {
try {
return XmlElement.parse(x);
}
catch (XmlParseException e) {
}
return null;
}
}
}
Total: 10 lines of code

Query Editor
+
?
>>
<
>
=
+
*
n
A /

A /
Playback controls
Can execute a query
one event at a time
(play/pause), or at a
custom input rate

A /
Toggle between graph
and grammar view

>>
<
>
=
+
*
n
Palettes: drag/drop on playground
to compose query. Conventions:
Light pipes = input
Dark pipes = output
Colours = event types
Correct ﬂow = dark → light
Type discipline = matching colours
Logic
I/O
Tuples
Plumbing
Numbers
Booleans
Blobs
Name-value tuples
Math

Query Editor
+
?
>>
<
>
=
+
*
n
A /
3 5 5
3 2
"Magniﬁer" reveals
realtime content of message
queues...

Query Editor
+
?
>>
<
>
=
+
*
n
A /
Threshold
Some processors have parameters;
can modify them with contextual menu

Query Editor
+
?
>>
<
>
=
+
*
n
A /
Site C
Site BSite A
Separation of query parts on
multiple sites

BeepBeep needs you!BeepBeep needs you!
Licensed under LGPL: download and
use it
Create you own domain-specific palettes
Take your own monitor/script and wrap
it into a new Processor object
For free: benefit from all existing processors
and functions already available
https://liflab.github.io/beepbeep-3

When RV Meets CEP (RV 2016 Tutorial)

More Related Content

What's hot

Similar to When RV Meets CEP (RV 2016 Tutorial)

More from Sylvain Hallé

Recently uploaded

When RV Meets CEP (RV 2016 Tutorial)