Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

When RV Meets CEP (RV 2016 Tutorial)

712 views

Published on

This presentation is an introduction to Complex Event Processing (CEP) intended for an practicioners of Runtime Verification. It first describes typical CEP problems, popular tools and their query languages. It then presents BeepBeep 3, an event stream processor that attempts to bridge the gap between RV and CEP. Thanks to BeepBeep’s generic architecture and flexible input language, queries and properties from both fields can be efficiently processed.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

When RV Meets CEP (RV 2016 Tutorial)

  1. 1. When RV Meets CEP...When RV Meets CEP... A NEW TUTORIAL BY SYLVAIN HALLÉ RUNTIME VERIFICATION COMPLEX EVENT PROCESSING CRSNG NSERC (C) 2016 LABORATOIRE D'INFORMATIQUE FORMELLE, UNIVERSITÉ DU QUÉBEC À CHICOUTIMI, CANADA PRESENTED AT THE INTERNATIONAL CONFERENCE ON RUNTIME VERIFICATION, MADRID, SPAIN, SEPTEMBER 27TH, 2016 WORK FUNDED BY THE CANADA RESEARCH CHAIR IN SPECIFICATION, TESTING AND VERIFICATION OF SOFTWARE SYSTEMS Can these two fields benefit from each other?
  2. 2. TODAY'S PROGRAM Part I RV vs. CEP Part II A Fistful of Processors Part III The Monitor Strikes Back
  3. 3. PART ONE RV vs. CEP
  4. 4. public static void main(String[] args) { Map m = new HashMap(); ... Iterator i = m.iterator(); while (i.hasNext()) { ... } }
  5. 5. public static void main(String[] args) { Map m = new HashMap(); ... Iterator i = m.iterator(); while (i.hasNext()) { ... } } HashMap.new = HashMap$1263 HashMap$1263.iterator() = Iterator$32045 HashMap$1263.contains(MyObject$26) = false Iterator$32045 HashMap$1263.hasNext = true ... Method calls
  6. 6. Any call to next() must be immediately preceded by a call to hasNext() that returns true. Two calls to remove() must be separated by at least one call to next (). Calls to an iterator's methods are forbidden once its parent collection has been modified by calls like put() or remove().
  7. 7. 06bb5c mov esp, ebp | EBP=001bfbf4 | ESP=001bfbf4 06bb5d pop ebp | ESP=001bfbf4 [001bfbf4]=001bfc24 | EBP=001bfc24 ESP=001bfbf8 06bb5e push ecx | ECX=71f1a8b9 ESP=001bfbf8 | ESP=001bfbf4 [001bfbf4]=71f1a8b9 06bb5f ret | ESP=001bfbf4 [001bfbf4]=71f1a8b9 | ESP=001bfbf8 06bb60 ret | ESP=001bfbf8 [001bfbf8]=01391036 | ESP=001bfbfc 06bb61 add esp, 0x20 | ESP=001bfbfc | ESP=001bfc1c EFLAGS= 06bb62 cmp [ebp-0x4], 0x3e8 | [001bfc20]=000003e8 EBP=001bfc24 | EFLAGS=ZP 06bb63 jnz 0x1391057 | EFLAGS=ZP | 06bb64 push 0x1392144 | ESP=001bfc1c | ESP=001bfc18 [001bfc18]=01392144 Return address protection Integer overflow detection Call sequence profiling Pointer subterfuge detection Malicious pattern detection Etc.
  8. 8. 06bb5c mov esp, ebp | EBP=001bfbf4 | ESP=001bfbf4 06bb5d pop ebp | ESP=001bfbf4 [001bfbf4]=001bfc24 | EBP=001bfc24 ESP=001bfbf8 06bb5e push ecx | ECX=71f1a8b9 ESP=001bfbf8 | ESP=001bfbf4 [001bfbf4]=71f1a8b9 06bb5f ret | ESP=001bfbf4 [001bfbf4]=71f1a8b9 | ESP=001bfbf8 06bb60 ret | ESP=001bfbf8 [001bfbf8]=01391036 | ESP=001bfbfc 06bb61 add esp, 0x20 | ESP=001bfbfc | ESP=001bfc1c EFLAGS= 06bb62 cmp [ebp-0x4], 0x3e8 | [001bfc20]=000003e8 EBP=001bfc24 | EFLAGS=ZP 06bb63 jnz 0x1391057 | EFLAGS=ZP | 06bb64 push 0x1392144 | ESP=001bfc1c | ESP=001bfc18 [001bfc18]=01392144 Return address protection "After a call instruction, the current location on the stack cannot be the target of a MOV instruction until the call returns."
  9. 9. start(vase,3,15). bid(vase,15). start(ring,5,30). endOfDay. bid(ring,32). bid(ring,33). bid(vase,18). sell(vase).
  10. 10. start(vase,3,15). bid(vase,15). start(ring,5,30). endOfDay. bid(ring,32). bid(ring,33). bid(vase,18). sell(vase). Event name
  11. 11. start(vase,3,15). bid(vase,15). start(ring,5,30). endOfDay. bid(ring,32). bid(ring,33). bid(vase,18). sell(vase). Parameters
  12. 12. start(vase,3,15). bid(vase,15). start(ring,5,30). endOfDay. bid(ring,32). bid(ring,33). bid(vase,18). sell(vase). For a given item, any bid must be higher than the previous one. Any bid must be higher than the minimum price. No bid can be placed before a start event.
  13. 13. Server Ajax web app
  14. 14. Server Ajax web app <message> <name>CartAdd</name> <CartId>123</CartId> <Items> <ItemId>42</ItemId> <ItemId>23</ItemId> </Items> </message> <message> <name>CartAddResponse</name> <CartId>123</CartId> </message>
  15. 15. The Beep Store GO Sign inor regi ster What is this? Login Ask for account Contact us Fault parameters Search: Your Cart Fault parameters Don’t check Results’s type In the detailed search form, sends an ItemSearch message without checking that the Results element is an integer. "Add to cart" enabled if item present in cart Makes the "Add to cart" button available for items that are already in the user's cart. Message schemas Cart manipulations Highlights documentation Disables the verification
  16. 16. 1. The element must be an integer between 1 and 20. 2. The element is mandatory only if is present, otherwise it is forbidden. 3. The request cannot be resent if its response is successful. 4. must follow a successful Logi nResponse. 5. There can be at most one active cart ID per session key. 6. You cannot add the same item twice to the shopping cart. Page Page Resul t s Logi n Car t Cr eat e
  17. 17. ???
  18. 18. CAM 1
  19. 19. CAM 1
  20. 20. The lock graph of a set of tasks should not contain cycles A communication window has three phases: prep, active and cleanup Every command must be responded by either a success or a failure
  21. 21. R V untime erification
  22. 22. System
  23. 23. System
  24. 24. System Instrumentation
  25. 25. System Instrumentation
  26. 26. System Instrumentation Trace
  27. 27. System Instrumentation Trace Events
  28. 28. System Instrumentation Trace Events
  29. 29. System Instrumentation Trace Events Trace validation
  30. 30. System Instrumentation
  31. 31. System Runtime monitoring Instrumentation
  32. 32. System Runtime monitoring Instrumentation
  33. 33. System Runtime monitoring Overhead Instrumentation
  34. 34. hasNext next hasNext Option #1 Finite-State Machines
  35. 35. hasNext next hasNext Option #1 Finite-State Machines ...and extensions 1 2 3 groupStart(t) init(p)∀p QEA: Reger et al., TACAS 2015 groupEnd(t2) phaseStart 4 run(p) finish(p) t2−t < 480
  36. 36. A call to next must be followed by a call to hasNext No CartCreate request can occur before a LoginResponse message A received order must eventually be shipped Three successive login attempts should trigger an alarm G (next → X hasNext) ¬ CartCreate U hasNext G (receive → F ship) G ¬(fail ∧ (X (fail ∧ X fail))) Option #2 Linear Temporal Logic
  37. 37. G (∃ retAddrVal ∈ ./return-address : ( (./instruction = call) ∧ (¬ ((F( ((./instruction = mov) ∧ (./output/type = general-register)) → (∃ regA ∈ ./output/name : (F (( ((./instruction = mov) ∧ (./output/type = general-register)) ∧ (./input/type = litteral) ) → (∃ regB ∈ ./output/name : ( ∃ constAddr ∈ ./input/value : (F (((./instruction = cmp ) ∧ (./output/type = regA)) → (∃ loc ∈ ./location : (F(( ((./instruction = mov ) ∧ (./output/type = general-register)) ∧ (./output/name = regA ) ) ∧ ((./input/name = regB ) ∧ (./input/type = ptr)))))) ))))) ))) U ((./instruction = return) ∧ (./fonction-returned = retAddrVal)) )) )) ...and extensionsLTL-FO+, Hallé et al. IEEE Trans. Serv. Comput. 2012
  38. 38. νZ.( ∨a ∈ A a ⊤ ∧ ∧a ∈ A [a] Z ) Option #4 mu-calculus (hasNext+ next)* Option #3 Regex
  39. 39. class SeqMonitor extends EvrMonitor { val SeqStart, SeqDone = fact def seq name(s:String) = words(s)(2) "start_seq" −− EVR(’id -> "EVR_SEQ_START", ’sclk -> ’S, ’msg -> ’M) 7−-> { val w = words(’M.s) val seq name = w(15).slice(1, w(15).length−2) insert (SeqStart(seq name, ’S.d)) } "end_seq_ok" −− EVR(’id -> "EVR_SEQ_SUCCESS", ’sclk -> ’E, ’msg -> ’M) & ’SeqStart(’name, ’S) --> { if (seq name(’M.s) == ’sname.s) { replace (SeqStart)(SeqDone(’name.s, ’S.d, ’E.d, "OK")) } } "end_seq_fail" −− EVR(’id -> "EVR_SEQ_FAILURE", ’sclk -> ’F, ’msg -> ’M) & ’SeqStart(’name, ’S) --> { if (seq name(’M.s) == ’name.s) { replace (SeqStart)(SeqDone(’name.s, ’S.d, ’F.d, "FAIL")) } } "print" −− SeqDone(’name, ’S, ’E, ’stat) --> { updateCSV(’name.s, ’S.d, ’E.d, ’ stat .s) remove(SeqDone) } } Option #5 LogFire
  40. 40. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$
  41. 41. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ 10/2/29 APPL 1212.2 10/2/29 MSFT 1031.6 10/2/29 GOGL 341.1 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/4/29 APPL 1164.3 10/4/29 MSFT 1025.1 10/4/29 GOGL 345.4 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0
  42. 42. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ 10/2/29 APPL 1212.2 10/2/29 MSFT 1031.6 10/2/29 GOGL 341.1 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/4/29 APPL 1164.3 10/4/29 MSFT 1025.1 10/4/29 GOGL 345.4 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 Timestamp
  43. 43. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ 10/2/29 APPL 1212.2 10/2/29 MSFT 1031.6 10/2/29 GOGL 341.1 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/4/29 APPL 1164.3 10/4/29 MSFT 1025.1 10/4/29 GOGL 345.4 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 Stock symbol
  44. 44. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ 10/2/29 APPL 1212.2 10/2/29 MSFT 1031.6 10/2/29 GOGL 341.1 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/4/29 APPL 1164.3 10/4/29 MSFT 1025.1 10/4/29 GOGL 345.4 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 10/3/29 GOGL 362.2 10/3/29 MSFT 1024.4 10/3/29 APPL 1179.0 Price
  45. 45. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ What is the closing price of MSFT for the first five trading days? Select all days after the 100th where MSFT closed over $50. On every 5th day, calculate the average closing price for MSFT for the last 5 days. What are the stocks that closed higher than MSFT for a given day?
  46. 46. $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ $$$$$$$$$$$ $$$$$$$$$$ What is the closing price of MSFT for the first five trading days? Select all days after the 100th where MSFT closed over $50. On every 5th day, calculate the average closing price for MSFT for the last 5 days. What are the stocks that closed higher than MSFT for a given day? Snapshot query Landmark query Sliding query Join query
  47. 47. 4948 4949 6.3132
  48. 48. 4948 4949 6.3132 Test result
  49. 49. 4948 4949 6.3132 Drug prescriptions
  50. 50. 4948 4949 6.3132 Meeting
  51. 51. 4948 4949 6.3132 MSH|^~&|SOURCE|383018129|PRIORITY HEALTH|382715520| 2007100914484648||ORU^R01| 0129938170710091448|P|2.3|PID|1| 1034157|012993817|| LASTNAME^FIRSTNAME||19520101|M||| 1234 MAIN^^DEARBORN HEIGHT^MI^48127||||||||PID|1|| 94000000000^^^Priority Health|| LASTNAME^FIRSTNAME||19400101|F| PD1|1||| 1234567890^DOCLAST^DOCFIRST^M^ ^^^^NPI|OBR|1|||80061^LIPID PROFILE^CPT-4||20070911||||||||||OBX|1| NM|13457-7^LDL (CALCULATED)^LOINC| 49.000|MG/DL| 0.000 - 100.000|N|||F| OBX|2|NM| 2093-3^CHOLESTEROL^LOINC|138.000| MG/DL|100.000 - 200.000|N|||F|OBX|3| NM|2086-7^HDL^LOINC|24.000|MG/DL| 45.000 - 150.000|L|||F|OBX|4|NM| 2571-8^TRIGLYCERIDES^LOINC|324.000| HL7HL7
  52. 52. Notify me when two out of three successive data points lie more than 1 standard deviation from the mean on the same side of the mean line.
  53. 53. Notify me when two out of three successive data points lie more than 1 standard deviation from the mean on the same side of the mean line. Trend query
  54. 54. TOLL
  55. 55. TOLL TollId Toll License State Model 48392 $3.50 QF5010 KW T
  56. 56. TOLL TollId Toll License State Model 48392 $3.50 QF5010 KW T
  57. 57. TOLL Compute the toll of each booth over 3-minute intervals
  58. 58. C E omplex vent Processing
  59. 59. Select AVG(closingPrice) From ClosingStockPrices Where stockSymbol = ‘MSFT’ for (t = ST; t < ST+50, t+= 5) { WindowIs(ClosingStockPrices, t - 4, t); } Select c2.* FROM ClosingStockPrices as c1, ClosingStockPrices as c2 WHERE c1.stockSymbol = ‘MSFT’ and c2.stockSymbol!= ‘MSFT’ and c2.closingPrice > c1.closingPrice and c2.timestamp = c1.timestamp for (t = ST; t < ST +20 ; t++ ){ WindowIs(c1, t - 4, t); WindowIs(c2, t - 4, t); } Two queries in the language of TelegraphCQ
  60. 60. SELECT * FROM FILTER {cnt >= 10}( (SELECT *, 1 AS cnt FROM FILTER {contains(summary,’iPod’) = 1}(webfeeds)) FOLD {, $.cnt < 10 AND DUR < 1 DAY, $.cnt + 1 AS cnt} (SELECT * FROM FILTER {contains(summary,’iPod’) = 1}(webfeeds)) ) PUBLISH ipod_popularity A query in the language of Cayuga
  61. 61. from inputtrace#window.length(10) select stockSymbol, timestamp, avg(closingPrice) as avg10 insert into avg10trace; from inputtrace#window.length(20) select stockSymbol, timestamp, avg(closingPrice) as avg20 insert into avg20trace; @info(name = 'query1') from avg10trace#window.length(10) join avg20trace#window.length(20) on avg10trace.timestamp == avg20trace.timestamp select avg10trace.stockSymbol as stockSymbol, avg10trace.timestamp, avg20trace.timestamp as ts2, avg10trace.avg10 as avg10, avg20trace.avg20 as avg20 having avg10 > 50 or avg20 > 50 insert into output; SiddhiQL...
  62. 62. CEPRV event traces "properties" tuple-ish overhead event streams "queries" tuples* throughput =When
  63. 63. CEPRV outputs T/F result cannot be reused in another property mostly variants of FSM and logic good for sequential patterns not quite When outputs event streams result can be reused as input of another query mostly "extensions" of SQL not quite good for data manipulation
  64. 64. CEPRVBeyond &
  65. 65. CEPRVBeyond & Both RV and CEP assume events to be "tuples": associative maps from strings to single scalar values start(vase,3,15). bid(vase,15). start(ring,5,30). 3/10/29 MSFT 1059.2 3/10/29 APPL 365.4 3/10/29 GOGL 1120.1 HashMap.new = HashMap$1263 HashMap$1263.hasNext = true TollId Toll License State Model 48392 $3.50 QF5010 KW T
  66. 66. CEPRVBeyond & Both RV and CEP assume events to be "tuples": associative maps from strings to single scalar values start(vase,3,15). bid(vase,15). start(ring,5,30). 3/10/29 MSFT 1059.2 3/10/29 APPL 365.4 3/10/29 GOGL 1120.1 HashMap.new = HashMap$1263 HashMap$1263.hasNext = true TollId Toll License State Model 48392 $3.50 QF5010 KW T And what if events are not tuples?
  67. 67. CEPRVBeyond & Both RV and CEP assume events to be "tuples": associative maps from strings to single scalar values start(vase,3,15). bid(vase,15). start(ring,5,30). 3/10/29 MSFT 1059.2 3/10/29 APPL 365.4 3/10/29 GOGL 1120.1 HashMap.new = HashMap$1263 HashMap$1263.hasNext = true TollId Toll License State Model 48392 $3.50 QF5010 KW T And what if events are not tuples? And what if events are not tuples? We make them fit!
  68. 68. CEPRVBeyond & Suppose each event is a set: {6,5,3,9} {5,4,10} ... We can "fit" it into a tuple: a 6 a=4 ∨ b=4 ∨ c=4 ∨ d=4 c 3 d 9 Imposes upper bound on set size But then 4 ∈ S becomes: b 5
  69. 69. CEPRVBeyond &
  70. 70. CEPRVBeyond &
  71. 71. CEPRVBeyond &
  72. 72. CEPRVBeyond & Both RV and CEP rely on monolithic specification/query languages Everything must be written in that one language (FO-LTL, QEA, SiddhiQL, etc.) The language must accommodate every possible use case In general, no support for extensions
  73. 73. Part 2: A Fistful of Processors
  74. 74. Event stream query engine developed based on the previous observations Aims at borrowing strengths from both RV and CEP (and beyond) Key concepts: composability, modularity, extensibility Open source, developed in Java http://liflab.github.io/beepbeep-3 peeB peeB 3
  75. 75. EventsEvents An event is an element e taken from some set E, called the event type. No restriction on the type! In BeepBeep, events can be any Object. Booleans Numbers 2 3 4 π Strings abc Functions Sets PlotsTuples 3 8 a 3 8 a 2 6 c + ⊇? XML documents <a><a><a> . . . ?
  76. 76. TracesTraces An event trace (or event stream) is a potentially infinite sequence of events of a given type: 2 0 6 3 4 9 . . . Traces are symbolically denoted by: e = e0 e1 e2 e3 ... The set of all traces of type T is denoted as: T*
  77. 77. FunctionsFunctions A function takes 0 or more events as its input, and returns 1 or more events. Functions are first-class objects; they descend from the class Function 1 : 1 function 2 : 1 function 1 : 2 function ⊇? 3 4 2+5i ₹ 2 5 6 0 : 1 function 6
  78. 78. ProcessorsProcessors A processor takes 0 or more event traces as its input, and returns 0 or more event traces as its output 1 : 1 processor 2 : 1 processor . . . . . .
  79. 79. ProcessorsProcessors When a processor takes more than one input trace, the set of events at matching positions in each trace is called a front. bacd 3601 b 3 a 6 c 0 d 1 1st event 2nd 3rd 4th . . . . . . . . .
  80. 80. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒
  81. 81. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 5 3
  82. 82. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 5 3
  83. 83. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 5 3 +
  84. 84. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 8
  85. 85. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 1
  86. 86. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 1
  87. 87. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 1
  88. 88. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 1
  89. 89. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 1 4
  90. 90. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 1 4
  91. 91. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 1 4 +
  92. 92. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6 5
  93. 93. Synchronous processingSynchronous processing Events are processed one front at a time. + Buffers collect events until a complete front can be processed. ⇒ 6
  94. 94. Synchronous processingSynchronous processing Makes a couple of things simpler Don't care about what event arrived first or upstream computation time "Pen and paper" calculation is identical to the real one Otherwise, can do a lot with simple timeouts ⇒ contained asynchrony Motto: Don't use asychronous processing...
  95. 95. Synchronous processingSynchronous processing Makes a couple of things simpler Don't care about what event arrived first or upstream computation time "Pen and paper" calculation is identical to the real one Otherwise, can do a lot with simple timeouts ⇒ contained asynchrony Motto: Don't use asychronous processing... ...unless you really have to
  96. 96. Synchronous processingSynchronous processing In BeepBeep, all synchronous processors are descendents of the SingleProcessor class Takes care of handling input/output buffers Calls (abstract) method process() when an input front is ready to be consumed Processor only needs to produce an output front from this input Makes it easy to create your own (more on that later)
  97. 97. A high-level event trace can be produced by composing ("piping") together one or more processors from lower-level traces CompositionComposition
  98. 98. Each processor has its own input/output buffers CompositionComposition
  99. 99. Any output can be connected to any input, as long as they have the same type CompositionComposition
  100. 100. Any output can be connected to any input, as long as they have the same type CompositionComposition Many types can occur in the same chain
  101. 101. Trace Manipulation Functions/Processors ArchitectureArchitecture TMF/P EMF/P Event Manipulation Functions/Processors Type-agnostic Stateful Type-dependent Stateless
  102. 102. Trace Manipulation Functions/Processors ArchitectureArchitecture TMF/P EMF/P Event Manipulation Functions/Processors Type-agnostic Stateful Type-dependent Stateless Not clear-cut, but general design goal
  103. 103. ArchitectureArchitecture BeepBeep provides only a few built-in processors and functions Palette Set of processors and functions, centered around a particular use case Concretely, a JAR library defining new Processor and Function objects
  104. 104. <? + <? =? − ÷ × f Σ f n { n n Function Cumulative Trim ForkDecimate Group WindowSliceFilter Built-in processors Built-in functions .n < <
  105. 105. SemanticsSemantics Let P be a processor and a b c = a1,a2,... = b1,b2,... = c1,c2,... be traces a,b,c : P[[ n = e1,e2,... denotes the n-th output trace of P, given traces a, b, c as input.
  106. 106. f FunctionFunction Applies an n-ary function f to every front of size n "Lifts" any function into a processor a,b : f[[ n = f n (ai,bi)
  107. 107. f FunctionFunction Applies an n-ary function f to every front of size n "Lifts" any function into a processor a,b : f[[ n = f n (ai,bi) The n-th output
  108. 108. f FunctionFunction Applies an n-ary function f to every front of size n "Lifts" any function into a processor a,b : f[[ n = f n (ai,bi) The n-th output i The i-th event
  109. 109. f FunctionFunction Applies an n-ary function f to every front of size n "Lifts" any function into a processor a,b : f[[ n = f n (ai,bi) The n-th output i The i-th event f + f <0? Pairwise sum of events Is each event negative?
  110. 110. CumulativeCumulative Applies a 2 : 1 function f to its previous value and the current event a : Σf[[ = f(x,a1), f(f(x,a1),a2), ... + <Sum of all events Have we seen ⊤ so far? Σ f x Σ f Σ f 0 ⊥
  111. 111. TrimTrim Returns the input trace, trimmed of its first n events a : [[ = an+1, an+2, ... n n
  112. 112. TrimTrim Returns the input trace, trimmed of its first n events a : [[ = an+1, an+2, ... n n 1 f =?=? < Σ f ⊥ Does the same number repeat twice in a row?
  113. 113. GroupGroup Makes a set of connected processors behave as a single processor
  114. 114. GroupGroup Makes a set of connected processors behave as a single processor 1 f =?=? < Σ f ⊥
  115. 115. ForkFork Replicates its input on each of its n outputs a : Ψ[[ = a1, a2, ... n
  116. 116. DecimateDecimate Outputs every n-th input event a : [[ = a1, an+1, a2n+1, ...n Ψ n
  117. 117. FilterFilter A n : n-1 processor; outputs the n-1 first components of a front if its last component is ⊤; otherwise, discards it a , a , ..., a : =[[ 2 k1 n F i {a if a =⊤ ε otherwise k i n i Powerful mechanism: "anything can be filtered" (don't care about condition) Boolean trace does not need to come from the same source as the inputs being filtered
  118. 118. FilterFilter A n : n-1 processor; outputs the n-1 first components of a front if its last component is ⊤; otherwise, discards it f <0? Get only the negative events of the input
  119. 119. WindowWindow Returns the output of a processor P on a sliding window of width n a : Υ =[[ P i Powerful mechanism: "anything can be windowed" (don't care about function) n { n ai, ... ai+n : P[[ *
  120. 120. WindowWindow Returns the output of a processor P on a sliding window of width n a : Υ =[[ P i Powerful mechanism: "anything can be windowed" (don't care about function) n { n ai, ... ai+n : P[[ * The last event
  121. 121. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0
  122. 122. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 3 4 2
  123. 123. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 3 4 2
  124. 124. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 3 24
  125. 125. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 243
  126. 126. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 2 243
  127. 127. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 2 243
  128. 128. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 2 243 4
  129. 129. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 6 243 2
  130. 130. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 6 243 23
  131. 131. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 9 243 6 2
  132. 132. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 9 243 6 2 3 2 1
  133. 133. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 9 243 6 2 9
  134. 134. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 243 9
  135. 135. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 1 43 9
  136. 136. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91
  137. 137. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 4
  138. 138. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 4
  139. 139. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 43
  140. 140. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 7 4
  141. 141. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 7 41
  142. 142. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 91 8 7 4
  143. 143. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 81 8 7 4 9
  144. 144. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 43 81 9
  145. 145. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 31 8 9
  146. 146. WindowWindow Returns the output of a processor P on a sliding window of width n n { 3 {+ Σ f 0 31 8 9 The sum of all 3 successive events
  147. 147. SliceSlice Dispatches an event e to a distinct instance of processor P according to the value of some function f {a if f(ai)=k ε otherwise i πf k[a : =[i P f[a : =[i U πf k[a1,...ai : [i[ [: P k * +
  148. 148. SliceSlice Dispatches an event e to a distinct instance of processor P according to the value of some function f {a if f(ai)=k ε otherwise i πf k[a : =[i P f[a : =[i U πf k[a1,...ai : [i[ [: P k * + Multiset union
  149. 149. f(x) = x mod 2
  150. 150. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers
  151. 151. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2
  152. 152. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5
  153. 153. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5
  154. 154. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 U+
  155. 155. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 U+ {9}{5}
  156. 156. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 U+ {9}{5}
  157. 157. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{5}6
  158. 158. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{5} 6
  159. 159. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{5} 6 U+
  160. 160. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{6,5} 6 U+ {9}{5}
  161. 161. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{6,5} 6 {9}{5}1
  162. 162. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 5 {9}{6,5} 6 {9}{5} 1
  163. 163. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 1 {9}{6,5} 6 {9}{5} 5 U+
  164. 164. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 1 {9}{6,1} 6 {9}{6,5} 5 U+ {5}
  165. 165. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 1 {9}{6,1} 6 {9}{6,5} 5 U+ {5} The last odd and even numbers seen so far
  166. 166. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 + Σ f 0
  167. 167. f(x) = x mod 2 πf 0[a : =[subtrace of even numbers πf 1[a : =[subtrace of odd numbers mod 2 + Σ f 0 The sum of all odd numbers and all even numbers seen so far {9}{6,6} {9}{6,5} {5}
  168. 168. Input/outputInput/output 0 : 1 processors can be used to produce an event trace out of an external source (i.e. standard input, a file, etc.) Ditto for 1 : 0 processors a . . .b a . . .b
  169. 169. WARP ZONE
  170. 170. PalettesPalettes BeepBeep provides only a few built-in processors and functions Palette Set of processors and functions, centered around a particular use case Concretely, a JAR library defining new (reusable!) Processor and Function objects
  171. 171. XMLXML Provides two new Functions Ditto for the JSON library (using json-lif) XML parser: converts String events into XMLElement events (from xml-lif library) XPath: evaluates an XPath expression on an XMLElement (result is a set of XMLElements)
  172. 172. SetsSets New Functions: New processor: Multiset operations: add/remove elements, membership, union/intersection → → Dropper Outputs each element of a multiset as an event
  173. 173. GnuplotGnuplot New Functions: New processor: Scatterplot: converts a set of (x,y) pairs into a Gnuplot string producing a plot (ditto for Histogram) → → Gnuplot Caller Calls Gnuplot on a string and returns its output (i.e. a binary string)
  174. 174. TuplesTuples New event type: tuple New function: SELECT Creates an output tuple by combining attributes from input tuples Grammar extension: allow eSQL to use a SELECT statement, backward-compatible with SQL 3 8 a 3 8 a 2 6 c
  175. 175. TuplesTuples SELECT <tuple> FROM <processor-list> * | <tuple-element> | <tuple-element> , <tuple> <tuple-name> <rename> | <function-call> <rename> <word>.<word> | <word> <processor> <rename> | <processor> <rename> , <processor-list> AS <eml-attribute> <select> ::= <tuple> ::= <tuple-element> ::= <tuple-name> ::= <processor-list> ::= <rename> ::=
  176. 176. FSMFSM New processor: finite-state machine → → ? ? T /a/b //status/text() =? Walker * /a/b //status/text() =? Blocker * * Guards on transitions are arbitrary functions on events States output values of any type (Moore machine)
  177. 177. Linear Temporal LogicLinear Temporal Logic New processors for LTL temporal operators and first-order quantification (LTL-FO+) Two semantics for a formula φ: Boolean: the i-th output is the (2-valued) verdict of evaluating φ, starting at the i-th event ⇒ filter Troolean: the i-th output is the (3-valued) verdict of evaluating φ up to the i-th event ⇒ monitor
  178. 178. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b"
  179. 179. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" c
  180. 180. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ?
  181. 181. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ?c
  182. 182. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ? ?
  183. 183. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ? ?b
  184. 184. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ⊤ ? ?
  185. 185. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ⊤ ? ?a
  186. 186. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a monitoring context, we want the output of processor F b to be: ac c b F b "Tell me whether eventually b" ⊤ ⊤ ? ?
  187. 187. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b"
  188. 188. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" c
  189. 189. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b"
  190. 190. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" c
  191. 191. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b"
  192. 192. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" b
  193. 193. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" ⊤ ⊤ ⊤
  194. 194. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" a ⊤ ⊤ ⊤
  195. 195. Linear Temporal LogicLinear Temporal Logic Consider the LTL formula F b on the trace In a filtering context, we want the output of processor F b to be: ac c b F b "Get me all the events where eventually b" ⊤ ⊤ ⊤
  196. 196. Linear Temporal LogicLinear Temporal Logic Boolean operators are easy X G F U 1 < Σ f ⊥ < Σ f ⊥ F G = = = = Σ f ? ♣
  197. 197. Linear Temporal LogicLinear Temporal Logic First-order quantifiers Processor to run on each slice → A → x → E → x Quantified variable (Arbitrary) domain function Variable is added to the processor's context Boolean and Troolean versions
  198. 198. Let's put it all toghether! i.e. a few examples of queries from past publications
  199. 199. Use BeepBeep as a library in your program
  200. 200. → Fork f = new Fork(2); Use BeepBeep as a library in your program
  201. 201. → →→ → Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor( ); f Use BeepBeep as a library in your program
  202. 202. → →→ + → Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); f Use BeepBeep as a library in your program
  203. 203. → →→ + → Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); CountDecimate decimate = new CountDecimate(n); f n Use BeepBeep as a library in your program
  204. 204. → →→ + →→ Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); CountDecimate decimate = new CountDecimate(n); Connector.connect(fork, LEFT, sum, LEFT) f n Use BeepBeep as a library in your program
  205. 205. → →→ + → →→ Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); CountDecimate decimate = new CountDecimate(n); Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT) f n Use BeepBeep as a library in your program
  206. 206. → →→ + → →→ Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); CountDecimate decimate = new CountDecimate(n); Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT) .connect(decimate, OUTPUT, sum, RIGHT); f n Use BeepBeep as a library in your program
  207. 207. → →→ + → →→ Fork f = new Fork(2); FunctionProcessor sum = new FunctionProcessor(Addition.instance); CountDecimate decimate = new CountDecimate(n); Connector.connect(fork, LEFT, sum, LEFT) .connect(fork, RIGHT, decimate, INPUT) .connect(decimate, OUTPUT, sum, RIGHT); Pullable p = sum.getOutputPullable(OUTPUT); while (p.hasNext() != NextStatus.NO) { Object o = p.next(); ... } f n Use BeepBeep as a library in your program
  208. 208. → → → ^ n → → → + → → → → → ÷ → →1 → → + Σ Σ 0 0 The statistical moment of order n
  209. 209. → → → ^ n → → → + → → → → → ÷ → →1 → → + Σ Σ 0 0 The statistical moment of order n → →E(x) n As a group processor
  210. 210. Trigger an alarm
  211. 211. → → → → 1 → → > Trigger an alarm when two successive events
  212. 212. → → → 1 → → → → → <? → → → 1 → → > Trigger an alarm when two successive events are more than 1
  213. 213. → → → → 1 → → → ÷ → → → → → <? → → → 1 → → →σ → > Trigger an alarm when two successive events are more than 1 standard deviation
  214. 214. → → → → → 1 → → → - → → → ÷ → → → → → <? → → → 1 → → →E(x) 1 → →σ → > Trigger an alarm when two successive events are more than 1 standard deviation from the mean
  215. 215. → ⊇? → → → A →→ T ??...<a> <a> ... /a/b //character/id/text() → →
  216. 216. → → ? ? T /a/b //status/text() =? Walker * /a/b //status/text() =? Blocker * * →→ → /a/b //character → →→ A /a/b //id/text() → T ??...<a> <a> ...
  217. 217. → A /a/b //character[status=Walker]/id/text() → p1 → A → p2 → → → → /a/b //character[status=Blocker]/id/text() → → → 3 → → → <? → → → → → → → → f1 f2 → → → → → → → /a/b //character[id=p1]/position/x/text() /a/b //character[id=p2]/position/x/text() - |...| <? 6 > ... f1 /a/b //character[id=p1]/position/x/text() /a/b //character[id=p2]/position/x/text() - |...| f2
  218. 218. → → * * * * Create Auction =? 0 @ Last Price Days 0:= 3@Max. Days :=Min. Price 2@:= Days := Days 1 + End of Day =? 0 @ > <? Days Last Price 2 @:= Bid =? 0 @ > Min. Price <? 2 @ Last Price >? 2 @ Bid =? 0 @ > Last Price >? 2 @ Days := Days 1 + End of Day =? 0 @ Max. Days → End of Day * 1@ * → →→ → * Sold =? 0 @ Days Days + | |.÷
  219. 219. → → 1@ += Sanitise =? 0 @ Clean := Derive =? 0 @ > ∊ ? Clean T ??...Derive(a,b) Clean 2@ +=Clean := Clean 1 @ Use =? 0 @ > ∊ ? Clean 1 @ T * * T ...
  220. 220. Fastest car in the world
  221. 221. Fastest car in the world Slow on every type of road
  222. 222. Fastest car in the world Slow on every type of road Only runs on a closed circuit
  223. 223. Fastest car in the world Slow on every type of road Only runs on a closed circuit Works on every type of road
  224. 224. Fastest car in the world Slow on every type of road Only runs on a closed circuit Works on every type of road Heavy setup required
  225. 225. Fastest car in the world Slow on every type of road Only runs on a closed circuit Works on every type of road Heavy setup required Turn the key
  226. 226. PART THREE The Monitor Strikes Back
  227. 227. import ca.uqac.lif.cep.*; public class MyProcessor extends SingleProcessor { public Queue<Object[]> compute(Object[] inputs) { } public void build(Stack<Object> s) { } } . . . Create output events from input . . . . . . Instantiate processor from parse stack . . . <processor> := . . . <number> := . . . <string> := . . . Add new rules to any symbol from the basic grammar
  228. 228. Example: let us create a new 1 : 1 Processor that increments its input by a fixed value We would like to use it in eSQL with this syntax: INCREMENT ( P ) BY value Any expression defining a processor Number
  229. 229. <processor> := <my_processor> ; <my_processor> := INCREMENT ( <processor> ) BY <number> ; Symbols already defined in basic grammar Adds a new case to an existing rule
  230. 230. import ca.uqac.lif.cep.*; public class MyProcessor extends SingleProcessor { private int increment; public Queue<Object[]> compute(Object[] inputs) { Queue<Vector<Object>> out = new Queue<Vector<Object>>(); Object[] v = new Object[1]; Integer i = (Integer) inputs[0] + increment; v[0] = i; out.put(v); return out; } . . .
  231. 231. . . . public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop(); increment = n.intValue(); Connector.connect(p, this); s.push(this); } } Read contents of parse stack <number> BY ( <processor> ) INCREMENT Set processor's state Pipe it to its input Put on parse stack
  232. 232. Total: 6+9=15 lines of code . . . public void build(Stack<Object> s) { Number n = (Number) s.pop(); s.pop(); s.pop(); Processor p = (Processor) s.pop(); s.pop(); s.pop(); increment = n.intValue(); Connector.connect(p, this); s.push(this); } } Read contents of parse stack <number> BY ( <processor> ) INCREMENT Set processor's state Pipe it to its input Put on parse stack
  233. 233. Some pre-packaged grammar extensions: Manipulation of name-value tuples Set theory Formatted input (CSV, XML, JSON) Graphing (histograms, scatterplots, ...) Basic signal processing (smoothing, peak detection, ...) Create your own!
  234. 234. import ca.uqac.lif.cep.*; import ca.uqac.lif.cep.eml.tuples.*; public class MyExample { public static void main(String[] args) { Interpreter my_int = new Interpreter(); my_int.extendGrammar(TupleGrammar.class); Pullable p = my_int.executeQuery( ""HELLO WORLD""); for (int i = 0; i < 10; i++) { EmlString s = (EmlString) p.pull(); System.out.println(s); } } } Create query interpreter Load a grammar extension Execute a query Pull an output event
  235. 235. Example: let us create a new 1 : 1 Processor that parses XML events (using an existing XML library) This can be done by first defining a Function that converts Strings into XML objects
  236. 236. import ca.uqac.lif.cep.*; import ca.uqac.lif.xml.*; public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> { public static XmlParsingFunction instance = new XmlParsingFunction(); private XmlParsingFunction() { super(String.class, XmlElement.class); } public XmlElement getValue(String x) { try { return XmlElement.parse(x); } catch (XmlParseException e) { } return null; } } }
  237. 237. import ca.uqac.lif.cep.*; import ca.uqac.lif.xml.*; public static class XmlParsingFunction extends UnaryFunction<String,XmlElement> { public static XmlParsingFunction instance = new XmlParsingFunction(); private XmlParsingFunction() { super(String.class, XmlElement.class); } public XmlElement getValue(String x) { try { return XmlElement.parse(x); } catch (XmlParseException e) { } return null; } } } Total: 10 lines of code
  238. 238. Query Editor + ? >> < > = + * n A /
  239. 239. A / Playback controls Can execute a query one event at a time (play/pause), or at a custom input rate
  240. 240. A / Toggle between graph and grammar view
  241. 241. >> < > = + * n Palettes: drag/drop on playground to compose query. Conventions: Light pipes = input Dark pipes = output Colours = event types Correct flow = dark → light Type discipline = matching colours Logic I/O Tuples Plumbing Numbers Booleans Blobs Name-value tuples Math
  242. 242. Query Editor + ? >> < > = + * n A / 3 5 5 3 2 "Magnifier" reveals realtime content of message queues...
  243. 243. Query Editor + ? >> < > = + * n A / Threshold Some processors have parameters; can modify them with contextual menu
  244. 244. Query Editor + ? >> < > = + * n A / Site C Site BSite A Separation of query parts on multiple sites
  245. 245. BeepBeep needs you!BeepBeep needs you! Licensed under LGPL: download and use it Create you own domain-specific palettes Take your own monitor/script and wrap it into a new Processor object For free: benefit from all existing processors and functions already available https://liflab.github.io/beepbeep-3

×