Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors
Upcoming SlideShare
Loading in...5
×
 

Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors

on

  • 1,088 views

Traffic monitoring or crowd management systems produce large amounts of data in the form of events that need to be processed to detect relevant incidents. ...

Traffic monitoring or crowd management systems produce large amounts of data in the form of events that need to be processed to detect relevant incidents.

Rule-based pattern recognition is a promising approach for these applications, however, increasing amounts of data as well as large and complex rule sets demand for more and more processing power and memory. In order to scale such applications, a rule-based pattern detection system needs to be distributable over multiple machines. Today’s approaches are however focused on static distribution of rules or do not support reasoning over the full set of events.

We propose Cloud PARTE, a complex event detection system that implements the Rete algorithm on top of mobile actors. These actors can migrate between machines to respond to changes in the work load distribution. Cloud PARTE is an extension of PARTE and offers the first rule engine specifically tailored for continuous complex event detection that is able to benefit from elastic systems as provided by cloud computing platforms. It supports fully automatic load balancing and supports online rules with access to the entire event pool.

Statistics

Views

Total Views
1,088
Views on SlideShare
583
Embed Views
505

Actions

Likes
0
Downloads
2
Comments
0

9 Embeds 505

http://soft.vub.ac.be 228
http://www.stefan-marr.de 195
http://stefan-marr.de 52
http://toolslave.net 16
http://instantsvc.toolslave.net 9
http://prog.vub.ac.be 2
http://die-wixxer.toolslave.net 1
http://vps-1012701-4512.united-hoster.de 1
http://som.stefan-marr.de 1
More...

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • First: all the good ideas are the ones of my colleagues, and the bad ones are all mine.Second: what I am going to talk about is a distributed Rete engineA forward chaining inference engine, for complex event detection, pattern matching
  • NYC area 8mio cars, and 18mio peopleBeijing 5mio cars, 20mio peopleImagine we want to manage the traffic based on realtime dataThe amount of data is not the real problem:Singapore: 5mio people.
  • State the conclusions, requirements, base to work fromMany to many matchingMany Rules and Many events
  • Declarative rules!Online processing of real-time eventsLoad-balancing (dynamic)Elastic (add more machines)Transparent distribution (single system image)
  • So, why would you care? Well, we believe that we do not only have a contribution in the field of Rete engines, but at the same time a relevant application of the actor modelInitial prototype with:Transparent DistributionMobile actorsCentral managing interface (managing actor)Simple load balancing heuristic
  • Make explicit: all nodes are actors, communicate via message passingPoint out the data intensive nodes, cross-product of cars, and cams and cars.Point out that tests can be all kind of things, complex expressions, but also compute intensive external functions, e.g., to calculate the spatial relationships between points
  • Mobility built on top of theronclassic code mobilitythe actor contains it’s code, for instance the test expressionGather load info per machineFind load imbalance node overloaded if > 50 messages machine overloaded if ≥ 5 overloaded nodesmachine underloaded if < 3 overloaded nodes 3. Select node to move and destination
  • Single machine comparison:PARTE vs. CloudPARTE
  • Goal: chose realistic scenario
  • Starts with static distribution over 2 machines

Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors Presentation Transcript

  • Cloud PARTE Elastic Complex Event Processing based on Mobile Actors J. Swalens, T. Renaux, L. Hoste, S. Marr, W. De Meuter Software Languages Lab AGERE Workshop, 2013-10-27
  • Use Case Traffic Management ©Filip Bogdan http://photo.net/photodb/photo?photo_id=5953570 2
  • Traffic Management at City Scale? ©Jupiter Systems, Beijing Traffic Management Bureau Singapore: GPS coord. of 16000 taxis = 200 coord./sec = 12KB/sec Source: City-Scale Traffic Estimation from a Roving Sensor Network, Javed Aslam, Sejoon Lim, Xinghao Pan, and Daniela Rus. Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems. ACM, (2012)
  • Challenges: Large Variety of Input Sources Personal Onboard Units Pedestrian Sensing Vehicle Sensing ©Jupiter Systems, Beijing Traffic Management Bureau 4
  • Challenges: Combining the Data Imperative int gru_tapping(struct grail *ge, const struct utouch_frame *frame) { struct gesture_recognizer *gru = ge->gru; struct tapping_model *state = &gru->tapping; struct move_model *move = &gru->move; state->tap = 0; if (frame->num_active && !frame->prev->num_active) { state->mintouch = 0; state->maxtouch = 0; } if (move->ntouch > state->maxtouch) { if (state->active) { gin_gid_discard(ge, state->gid); state->active = 0; } state->start = move->time; state->maxtouch = move->ntouch; set_props(ge->gin, state, move, frame); if (state->maxtouch <= 5) { int type = GRAIL_TYPE_TAP1 + state->maxtouch - 1; state->gid = gin_gid_begin(ge, type, PRIO_TAP, frame); state->active = 1; } return 0; } if (!state->active) { state->mintouch = move->ntouch; state->maxtouch = move->ntouch; return 0; } if (move->ntouch <= state->mintouch) { int x = state->prop[GRAIL_PROP_TAP_X]; int y = state->prop[GRAIL_PROP_TAP_Y]; int t = move->time - state->start; if (t > move->fm[FM_X].bar_ms) { gin_gid_discard(ge, state->gid); state->mintouch = move->ntouch; state->maxtouch = move->ntouch; state->active = 0; return 0; } state->tap = state->maxtouch; state->prop[GRAIL_PROP_TAP_DT] = t; gin_gid_event(ge, state->gid, x, y, state->maxtouch, state->prop, state->nprop, 1); state->mintouch = move->ntouch; Declarative a simple tap (defrule detectTap (Press (x ?x) (y ?y) (finger "index") (timestamp ?t1)) (Release (x ?x) (y ?y) (finger "index") (timestamp ?t2)) (test (< ?t1 ?t2)) => (tapDetected ?x ?y))) 5
  • Challenges: Combining the Data (defrule StolenCarInTraffic (defrule CarTooFast (StolenCar (plate ?pl)) (Car (camera ?c1) (plate ?pl) (Car (plate ?pl) (camera ?c)) (time ?t1)) (Camera (id ?c) (highway ?hw) (Car (camera ?c2) (plate ?pl) (direction ?d) (position ?p)) (time ?t2)) (test (> ?t2 ?t1)) => (Camera (id ?c1) (highway ?hw) (printout "Sloten car " ?pl (direction ?d) (position ?p1)) " seen on highway " ?hw (Camera (id ?c2) (highway ?hw) " at " ?p (direction ?d) (position ?p2)) " in direction " ?d)) (test (> (speed ?t1 ?p1 ?t2 ?p2) *speed-limit*)) => (printout "Car too fast: " ?pl " at " (speed ?t1 ?p1 ?t2 ?p2) " km/h.")) 6
  • Challenges: Combining the Data (defrule StationaryCar (Car (camera ?c) (plate ?pl) (time ?t1)) (Car (camera ?c) (plate ?pl) (time ?t2)) (test (> (- ?t2 ?t1) 30)) (Camera (id ?c) (highway ?hw) (direction ?d) (position ?p)) => (assert (StationaryCar (camera ?c) (plate ?pl) (time ?t1)))) (defrule StationaryCars (Camera (id ?c) (highway ?hw) (direction ?d) (position ?p)) (StationaryCar (camera ?c) (plate ?pl1) (time ?t1)) (StationaryCar (camera ?c) (plate ?pl2) (time ?t2)) (StationaryCar (camera ?c) (plate ?pl3) (time ?t3)) (test (!= ?pl1 ?pl2 ?pl3)) (test (< 0 (- ?t2 ?t1) 1)) (test (< 0 (- ?t3 ?t2) 1)) => (assert (StationaryCars (camera ?c) (time ?t1)))) 7
  • How to approach such scenarios? • Declarative rules! • Online processing of real-time events! 1 set of rules 1 set of facts Load balancing Elasticity (transparent distribution) 8
  • Our Approach Parallel + Soft Real-Time • • • • Distributed for “Big Data” Declarative rules Mobile actors Central managing interface Simple heuristics for load balancing 9
  • PARTE: A Parallel, Actor-based Rete Engine (defrule StationaryCar (Car (cam ?c) (plate ?plt) (time ?t1)) (Car (cam ?c) (plate ?plt) (time ?t2)) (test (> (- ?t2 ?t1) 30)) (Cam (id ?c) (position ?p)) Events e=Car • Each node an actor • Unification memory intensive • Tests computationally intensive e=Cam Unification 1.cam=2.cam 1.plate=2.plate => (assert (StationaryCar (pos ?p))) Type Tests 1 2 Test time-diff > 30 Unification cam=id 3 4 assert [1] Rete: A Fast Algorithm for the Many Patterns/Many Objects Match Problem, Charles L. Forgy. Artif. Intell. 19(1):17–37 (1982) [2] Parallel Gesture Recognition with Soft Real-Time Guarantees, T. Renaux, L. Hoste, S. Marr, and W. De Meuter. AGERE’12, 35–46. 10
  • Cloud PARTE From Parallel to Distributed 1 Events 2 3 (defrule StationaryCar (Car (cam ?c) (plate ?plt) (time ?t1)) (Car (cam ?c) (plate ?plt) (time ?t2)) (test (> (- ?t2 ?t1) 30)) (Cam (id ?c) (position ?p)) => (assert (StationaryCar (pos ?p))) 4 master App • Transparent Distribution worker Cloud PARTE builds on Theron (C++ actor library, http://www.theron-library.com/) 11
  • Cloud PARTE From Parallel to Distributed Events 1 Load Balancer 2 (defrule StationaryCar (Car (cam ?c) (plate ?plt) (time ?t1)) (Car (cam ?c) (plate ?plt) (time ?t2)) (test (> (- ?t2 ?t1) 30)) (Cam (id ?c) (position ?p)) => (assert (StationaryCar (pos ?p))) 4 master App 3 worker • Automatic Load Balancing • Mobile actors • Heuristic – Length of message queue Cloud PARTE builds on Theron (C++ actor library, http://www.theron-library.com/) 12
  • Cloud PARTE From Parallel to Distributed 1 Load Balancer 3 4 master App Events 2 worker • Elasticity – Adding/removing of workers • Coordination required (currently) 13
  • Cloud PARTE From Parallel to Distributed • Elasticity 2 Load Balancer 3 4 master App Events 1 worker – Adding/removing of workers • Coordination required (currently) 14
  • Performance EVALUATION 10/27/20 15
  • Microbenchmarks 16 heavy tests Cloud PARTE Scalability PARTE vs. CloudPARTE Slow down 101 simple tests 501 simple tests 101 complex tests … with variables 10x101 simple tests 10x8 heavy tests 16 heavy tests 32 heavy tests 64 heavy tests 128 heavy tests Joining tree Search 18x 29x 5x 5x 60x 2x 4x 4x 4x 3x 6x 2x Single machine Performance Evaluation: Comparison with non-distributed PARTE 16
  • Performance: Traffic Scenario Simulating events from variety of sources 17
  • 11% speedup 2 machines Static Actor Distribution 2 machines Static Actor Distribution + Dynamic Load Balancing 17% speedup 3 machines Static Actor Distribution + Dynamic Load Balancing + Empty Machine 18
  • CONCLUSION 19
  • 1 set of rules 1 set of facts Load balancing Elasticity (transparent distribution) Traffic Management Cloud PARTE: Elastic Complex Event Processing based on Mobile Actors 20
  • BACKUP 21
  • Mobile Actors: Movement Protocol 22