The document formalizes complex event stream processing and introduces a framework for processing event traces symbolically represented as sequences of events. It discusses the roles of processors in transforming input traces, the composition of processors to create complex operations, and various processor functionalities like filtering, spawning, and windowing. It emphasizes the advantages of a non-procedural approach to event processing, allowing for efficient and dynamic computations over streaming data.

1. A Formalization of
Complex Event
Stream Processing
SYLVAIN HALLÉ
SIMON VARVARESSOS

2. Context
The execution of information systems
produces events

3. Events
An event is an element e taken from some
set E, called the event type
B
Booleans
3
4π
R
2
Numbers
abc
S
Strings
X→ Y
Functions
2X
Sets
Primitive
types
Composite
types

4. A sample log
[10:24:31] INFO Game starts
[10:24:33] WARN Lemming into Blocker...[
[10:25:01] DEBG Lemming into Floater, id: 32,
x: 320, y: 67 ; id: 31, x: 450, y: 43 ;
id: 23, x: 229, y: 40 ; ... ...
A file (or stream) of events
Each event has one or more
data elements
Actual (physical) format not relevant
for us

5. Searching the log
Select AVG(closingPrice)
From ClosingStockPrices
Where stockSymbol = `MSFT'
for (t = ST; t < ST+50, t+= 5) {
WindowIs(ClosingStockPrices, t - 4, t);
}

6. Problems
Formal languages (e.g. logic, automata)
focus on event ordering; not so good at
performing computations over events
Complex Event Processing often reduces
to a thin layer over custom procedural
code
Goal: provide a formal and
non-procedural framework for
the processing of event streams

7. Traces
An event trace (or event stream) is a potentially
infinite sequence of events of a given type:
4 9 . . .
2 0 6 3
Traces are symbolically denoted by:
e = e0 e1 e2 e3 ...
The set of all traces of type T is denoted as:
T*

8. Processors
A processor is a function that takes 0 or more
event traces as input, and returns 0 or 1
event trace as output
1 : 1 processor
. . . . . .
2 : 1 processor

9. Composition
A high-level event trace can be produced by
composing ("piping") together one or more
processors from lower-level traces

10. Processor algebra
Goal: come up with a "toolbox" of basic
processors sufficient to perform various
computations over traces
?

11. A few useful functions
ιt(x) = {t if x = ε
x otherwise
Identity function: returns an event if given one,
or t if passed the empty event ε
+(x) = {x}
Wrap function
-({x}) = x
Peel function
/π
Path function: returns subtree at end
of path π

12. Semantics
Processors can be defined formally by
describing how their output trace is created
from their input trace(s)
Input trace(s)
e0, ..., en : φ(x0 , ..., xn)
Symbolic variables:
xi refers to the i-th trace
on the left

13. Constants as processors
Any element t of type T can be lifted as a
0 : 1 processor producing the infinite trace
t t t t ...
t t t . . .
The constant
processor t e : t = t t t ...

14. Input/output
0 : 1 processors can be used to produce an
event trace out of an external source (i.e.
standard input, a file, etc.)
a b . . .
Ditto for 1 : 0 processors
a b . . .

15. Mutator
Returns t, but only as many times as the
number of events received so far
e t t
i.e. "mutates" input events into t

16. Functions as processors
Any n-ary function f defined on individual
events can be lifted to an n:1 processor on
traces, by applying it successively to n-uples
. . . 2 0 6
+ 7 8 5
3 8 1
. . .
. . .

17. Functions as processors
Any n-ary function f defined on individual
events can be lifted to an n:1 processor on
traces, by applying it successively to n-uples
e0, e1 : x0+x1
=
e00+e10 e01+e11 , e02+e12 , , . . .

18. Freeze
Returns the first event received, upon every
event received
. . . b b a a a a . . .
e : x = e0 e0 e0 ...

19. Delay
Returns every the input trace, starting from its
n-th event
. . . c b a b . . .
2
e : n
x = en en+1 en+2 ...
= e n : x
c

20. Decimate
Returns every n-th event of the input trace
. . . c b a a . . .
2
e : n
x = e0 en e2n ...
Ψ c
Ψ
e : Ψ n x i = e : x ni

21. COMPLEX PROCESSORS

22. Window
Simulates the application of a "sliding
window" to a trace
Υn φ
Takes as arguments: another processor φ
and a window width n
Returns the result of φ after processing
events 0 to n-1...
Then the result of (a new instance of) φ
that processes events 1 to n...
...an so on

23. Window
Example: execution of the processor
on the trace
2 1 5 0
Υ2++
Υ2
2 1 5 0 3 6 5
2 1 2 13
12 15 12 16
25 01 25 15

24. Window
The window processor can take any
processor as an argument...
...i.e. the sliding window can be applied to
anything.
Formally:
e : Υ n φ i = e i
: φ n-1

25. Filter
Discards events from an input trace based
on a selection criterion
Φ φ
Takes as argument another processor φ
Evaluates φ on the trace that starts at event
0; returns that event if the first event
returned by φ is T
Same process on the trace that starts at
event 1...
...an so on

26. Filter
Example: execution of the processor
on the trace
2 1 5 0
Φ∈2IN
∈2IN
2 1 5 0 Φ 2 0
2 1 5 0 ∈2IN

27. Filter
The filter can take any processor as an
argument...
...including a processor that requires multiple
input events before outputting something
Formally:
e : Φ φ = Φ(e, φ) , e 1 : Φ φ
Φ(e, φ) = { e0 if
e : φ = T
0
no event otherwise

28. Spawn
Cumulative combination of a processor's
output for every suffix of a trace
Σf φ
Creates one new instance of processor
φ upon every new input event
Feeds each input event to all existing
instances of φ
Combines the value returned by each
instance using function f
...and outputs it

29. Spawne
Example: execution of the processor
on the trace
2 1 5 0
Σ+
x
x
2 1 5 0 Σ+ 2 3 8
x
8
2 1 5 0 2 1 5 0
+
1 5 0 x 1 5 0
+
5 0 x 5 0

30. Spawn
Formally:
e :
Σf φ
=
e : φφ 0 , f ( e : φφ 0 , e 1
:
Σf φ )
Turns out to be a powerful device; depending
on φ and f, can provide many useful
processors...

31. Spawn
Count events Σ+1
Cumulative sum Σ+
Set of all events Σ∪ +
= #
= ++
∪ =

32. Composition
These processors can be freely composed
Compute the statistical moment of order n
n
Σ+
1
Σ+
÷

33. Composition
These processors can be freely composed
Compute the statistical moment of order n
n
Σ+
1
Σ+
÷
= #

34. Composition
These processors can be freely composed
Return sum of two successive events,
only if it is greater than 5
++
Υ2
> 5
Φ

35. Linear Temporal Logic
Operators for Linear Temporal Logic can
also be defined
Eventually Σ∨ (φ )
Next φ
...etc (see the paper)
= F φ
= X φ
3

36. All together now

37. All together now
Count pairs of successive events that are
more than one standard deviation from
the mean
E(X)
-

38. All together now
Count pairs of successive events that are
more than one standard deviation from
the mean
σ
E(X)
-
÷

39. All together now
Count pairs of successive events that are
more than one standard deviation from
the mean
σ
E(X)
-
> 1
÷ Φ

40. All together now
Count pairs of successive events that are
more than one standard deviation from
the mean
σ
E(X)
-
÷
X
> 1
Φ
∧ Φ

41. All together now
Count pairs of successive events that are
more than one standard deviation from
the mean
> 1
σ #
E(X)
-
÷
X
Φ
∧ Φ

42. Advantages
No imperative constructs
No restrictions on what can be piped to
what (modulo type compatibility)
Streaming operation: outputs produced
as inputs are being consumed
Implicit handling of buffering, duplication,
etc.

43. Demo!
Prototype implementation in Java
In this example, handles 100 events/sec.
Go see it on YouTube: http://goo.gl/QoS8Dy

44. GAME OVER

45. GAME OVER
QUESTIONS ?
YES NO

46. GAME OVER
QUESTIONS ?
YES NO