Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Debs 2011 tutorial on non functional properties of event processing


Published on

DEBS 2011 tutorial:
Non Functional properties of event processing

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • One of the best presentation slides I've ever read. Highly recommended to CEP researchers and engine developers.
    Millions of Thanks to IBM Haifa
    Are you sure you want to  Yes  No
    Your message goes here

Debs 2011 tutorial on non functional properties of event processing

  1. 1. Non Functional Properties of Event Processing Presenters: Opher Etzion and Tali Yatzkar-Haham Participated in the preparation: Ella Rabinovich and Inna Skarbovsky
  2. 2. Introduction to non functional properties of event processing
  3. 3. The variety There are variety of cheesecakes There are many systems that conceptually look like EPN, but they are different in non functional properties
  4. 4. Two examples Very large network management: Millions of events every minute; Very few are significant, same event is repeated. Time windows are very short. Patient monitoring according to medical Treatment protocol : Sporadic events, but each is meaningful, time windows can span for weeks. Both of them can be implemented by event Processing – but very differently.
  5. 5. Agenda Introduction to Non functional properties of event processing Performance and scalability considerations Availability considerations Usability considerations Security and privacy considerations Summary I II III IV V VI
  6. 6. Performance and Scalability Considerations
  7. 7. Performance benchmarks There is a large variance among applications, thus a collection of benchmarks should be devised, and each application should be classified to a benchmark Some classification criteria: Application complexity Filtering rate Required Performance metrics
  8. 8. Performance benchmarks – cont. Adi A., Etzion O. Amit - the situation manager. The VLDB Journal – The International Journal on Very Large Databases. Volume 13 Issue 2, 2004. Mendes M., Bizarro P., Marques P. Benchmarking event processing systems: current state and future directions. WOSP/SIPEW 2010: 259-260 . Previous studies ‎indicate that there is a major performance degradation as application complexity increases.
  9. 9. <ul><li>Previous studies ‎indicate that there is a major performance degradation as </li></ul><ul><li>application complexity increases  a single performance measure (e.g., event/s) is </li></ul><ul><li>not good enough. </li></ul><ul><li>Example for event processing system benchmark: </li></ul><ul><li>Scenario 1: an empty scenario (upper bound on the performance) </li></ul><ul><li>Scenario 2: low percentage of event instances is filtered in, agents are simple </li></ul><ul><li>Scenario 3: low percentage of event instances is filtered in, agents are complex </li></ul><ul><li>Scenario 4: high percentage of event instances is filtered in, agents are complex </li></ul>Some benchmarks scenarios Adi A., Etzion O. Amit - the situation manager. The VLDB Journal – The International Journal on Very Large Databases. Volume 13 Issue 2, 2004. 100000 100000 100000 100000 total external events 16503 7903 scenario 3 124319 1742 1372 accumulated latency (ms) 1923 57470 72887 throughput (event/s) scenario 4 scenario 2 scenario 1
  10. 10. Performance indicators One of the sources of variety Observations: The same system provides extremely different behavior based on type of functions employed Different application may require different metrics
  11. 11. Throughput Input throughput output throughput Processing throughput Measures: number of input events that the system can digest within a given time interval Measures: Total processing times / # of event processed within a given time interval Measures: # of events that were emitted to consumers within a given time interval
  12. 12. Latency latency In the E2E level it is defined as the elapsed time FROM the time-point when the producer emits an input event TO the time-point when the consumer receives an output event The latency definition But – input event may not result in output event: It may be filtered out, participate in a pattern but does not result in pattern detection, or participates in deferred operation (e.g. aggregation) Similar definitions for the EPA level, or path level
  13. 13. Latency definition – two variations: Producer 1 Producer 2 Producer 3 EPA Detecting Sequence (E1,E2,E3) within Sliding window of 1 hour E1 E2 E3 Consumer 11:00 12:00 11:10 11:15 11:30 E1 E2 E3 11:40 E2 Variation I: We measure the latency of E3 only Variation II: We measure the Latency of each event; for events that don’t create derived events directly, we measure the time until the system finishes processing them
  14. 14. Performance goals and metrics <ul><li>Multi-objective optimization function: </li></ul><ul><ul><li>min(  *avg latency + (1-  )*(1/thoughput)) </li></ul></ul>Max throughput All/ 80% have max/avg latency < δ All/ 90% of time units have throughput > Ω minmax latency minavg latency latency leveling
  15. 15. Optimization tools Blackbox optimizations: Distribution Parallelism Scheduling Load balancing Load shedding Whitebox optimizations: Implementation selection Implementation optimization Pattern rewriting
  16. 16. Scalability Scalability is the ability of a system to handle growing amounts of work in a graceful manner, or its ability to be enlarged effortlessly and transparently to accommodate this growth Scale up Vertical scalability Adding resources within the same logical unit to increase capacity Scale out Horizontal scalability Adding additional logical units to increase processing power
  17. 17. Vertical Scalability- Scaling up <ul><ul><li>Parallel concurrent execution support, such as multi-threading </li></ul></ul>Qualifications of application designed for scale-up <ul><ul><li>Common design patterns: </li></ul></ul><ul><ul><li>the Actor model </li></ul></ul><ul><ul><li>Utilizes the in-process memory for message passing </li></ul></ul>Adding resources to a single logical unit to increase it’s processing abilities <ul><li>Adding CPUs, memory </li></ul><ul><li>Expanding storage by adding hard-drives </li></ul>
  18. 18. Horizontal Scalability - Scaling out Qualifications of application designed for scale-out For stateful applications <ul><ul><li>Master/Worker </li></ul></ul><ul><ul><li>Shared Nothing approach </li></ul></ul><ul><ul><li>Spaced Based Architecture </li></ul></ul><ul><ul><li>Map Reduce </li></ul></ul>Different patterns associated <ul><ul><li>Distributed services -do not assume locality </li></ul></ul><ul><ul><li>Load balancing </li></ul></ul>Adding multiple logical units and making them work as a single unit <ul><li>Computer cluster </li></ul><ul><li>Load balancing </li></ul><ul><ul><li>Distributed caching </li></ul></ul><ul><ul><li>Partitioning of state (sharding) </li></ul></ul>
  19. 19. Scale-out and scale-up tradeoffs Scale up Scale out <ul><li> </li></ul><ul><li>Simpler programming model </li></ul><ul><li>Simpler management layer </li></ul><ul><li>No network overhead due to in-memory communication </li></ul><ul><li>Finite growth limit </li></ul><ul><li>Single point of failure </li></ul><ul><li> </li></ul><ul><li>Redundancy </li></ul><ul><li>Flexibility </li></ul><ul><li>Fault tolerance </li></ul><ul><li>Increased management complexity </li></ul><ul><li>More complex programming model </li></ul><ul><li>Issues as throughput and latency between nodes </li></ul>
  20. 20. General approach to scalability Usually applications combine the two approaches… Scaling out by… <ul><li> </li></ul><ul><li>Spreading application modules </li></ul><ul><li>Load partitioning and load balancing </li></ul><ul><li>Distributed cache </li></ul>Scaling up by… <ul><li> </li></ul><ul><li>Running multiple threads in each module </li></ul>
  21. 21. Scalability in event processing: various dimensions # of producers # of input events # of EPA types # of concurrent runtime instances # of concurrent runtime contexts Internal state size # of consumers # of derived events Processing complexity # of geographical Locations # of geographical Locations
  22. 22. Event-processing techniques for scalability Load shedding Load partitioning according to EPAs topology and Runtime Contexts
  23. 23. Scalability in event volume <ul><li>Scalability in event volume is the ability to handle variable event loads effectively as the quantity of events may go up and down over time </li></ul>Scale out techniques <ul><ul><li>Load partitioning </li></ul></ul><ul><ul><li>Parallel processing </li></ul></ul>Scale up techniques <ul><ul><li>Load shedding </li></ul></ul>Applicable scale-up and scale-out techniques <ul><ul><li>Load balancing </li></ul></ul>Scale out techniques Some applications requiring high event throughput financial weather phone-call tracking
  24. 24. Scalability in quantity of event processing agents <ul><li>Scalability in the quantity of EPAs is the ability of the system to adapt to substantial growth of event processing network and a high quantity of event processing agents </li></ul><ul><li>Some applications allow users to create their own custom EPAs </li></ul>Applicable scale-up and scale-out techniques <ul><ul><li>Partitioning </li></ul></ul>Optimization in agent assignment (mapping between logical and physical artifacts) <ul><ul><li>Parallelism and distribution </li></ul></ul>
  25. 25. Scalability in quantity of event processing agents – partitioning and parallelism <ul><li>Parallelism : Running all artifacts in a single powerful unit </li></ul><ul><ul><li>Saves network communication overhead </li></ul></ul><ul><li>Distribution : Running all artifacts in multiple units </li></ul><ul><ul><li>When event load is also an issue </li></ul></ul>Parallelism/Distribution Partitioning <ul><ul><li>Dependency analysis </li></ul></ul><ul><ul><li>Number of core processors </li></ul></ul><ul><ul><li>Level of distribution </li></ul></ul><ul><ul><li>Communication overhead </li></ul></ul><ul><ul><li>Performance objective fun. </li></ul></ul><ul><ul><li>EPA complexity analysis </li></ul></ul>
  26. 26. Scalability in a number of producers/consumers Growth in a number of producers usually results in growth in event load even if number of events produced by each one is small Growth in a number of consumers Requires optimization at routing level, such as multicasting
  27. 27. Scalability in a number of context partitions and context-state size Hash (customer id) Nodes events Each context partition is represented by internal state of a certain size <ul><ul><li>Use partitioning on context </li></ul></ul>Growth in a number of context partitions <ul><ul><li>Affects EPA performance since iterating on large states </li></ul></ul>Significant growth of internal state for a single context partition <ul><ul><li>Use EPA optimization techniques </li></ul></ul>
  28. 28. Availability Considerations
  29. 29. Availability Availability is ratio of time the system is perceived as functioning by its users to the time it is required or expected to function Can be expressed as <ul><li>Direct proportion : 9/10 or 0.9 </li></ul><ul><li>Percentage: 99% </li></ul><ul><li>Can be expressed in terms of average or total downtime </li></ul>
  30. 30. Availability expectations and solutions Continuous availability provides the ability to keep the business application running without any noticeable downtime Major outages… <ul><li> </li></ul><ul><li>Disaster recovery techniques </li></ul><ul><ul><li>Replicas on site </li></ul></ul><ul><ul><li>Additional sites </li></ul></ul>Continuous operation is the ability to avoid planned outages Minor outages… <ul><li> </li></ul><ul><li>High availability </li></ul><ul><ul><li>System design and implementation approach </li></ul></ul><ul><ul><li>Ensures pre-arranged level of availability during measuring period (SLA) </li></ul></ul><ul><ul><li>Represents ability to avoid minor unplanned outages by eliminating single points of failure </li></ul></ul>
  31. 31. Components of high availability Fault avoidance – redundancy and duplication <ul><li>Distributed application </li></ul><ul><li>Clustering </li></ul><ul><li>Duplication of storage systems </li></ul><ul><li>Failover for systems and data </li></ul>Fault tolerance -recoverability <ul><li>Failure recovery </li></ul>
  32. 32. Redundancy and duplication Redundancy <ul><li>Using multiple components with a method to detect failure and perform failover of the failed component </li></ul>Scale out techniques <ul><ul><li>Continuous monitoring of components (“heart-bit”) </li></ul></ul>Failover – automatic reconfiguration Load balancing is one of the players <ul><ul><li>When one fails – load balancer no longer sends traffic </li></ul></ul><ul><ul><li>When initial component recovers the load balancer routes traffic back </li></ul></ul>Duplication <ul><li>A single live component is paired with a single backup which takes over in event of failure </li></ul><ul><ul><li>Example : Storage – RAID 0 </li></ul></ul>
  33. 33. Recoverability in stateful applications – state management tradeoffs Data grid – replication of state between multiple machines <ul><li>Recoverability achieved by duplication of state </li></ul><ul><li>Better performance than pure db </li></ul><ul><li>Complexity in persistency layer implementation </li></ul><ul><li>Performance costs on cache misses and cache outs </li></ul><ul><li>Network overhead on replication of state </li></ul><ul><li>Complexity in synchronization of replicas </li></ul>Memory based state Better performance than pure db <ul><ul><li>Complexity in recoverability implementation </li></ul></ul>In-memory db with caching capabilities <ul><li>Better performance than pure db </li></ul><ul><li>Guaranteed recoverability </li></ul>
  34. 34. High availability costs Implementing some of HA practices can be very expensive… <ul><li> </li></ul><ul><li>Performance costs </li></ul><ul><ul><li>State changes need to be logged </li></ul></ul><ul><ul><li>Entire state has to be persisted at least periodically </li></ul></ul><ul><ul><li>Toll on processing latency and overall event throughput </li></ul></ul><ul><li>Actual costs </li></ul><ul><ul><li>Duplication of hardware for redundancy and duplication </li></ul></ul><ul><li>Application complexity </li></ul><ul><ul><li>For implementing failover , recovery </li></ul></ul>
  35. 35. Availability in event processing <ul><li> </li></ul><ul><li>Fault avoidance </li></ul><ul><ul><li>Duplication and redundancy of processing components </li></ul></ul><ul><ul><li>Failover mechanisms for processing components </li></ul></ul><ul><li>Fault tolerance </li></ul><ul><ul><li>Recoverability of state for all processing components </li></ul></ul><ul><ul><ul><li>EPAs state </li></ul></ul></ul><ul><ul><ul><li>Context state </li></ul></ul></ul><ul><ul><ul><li>Channels state </li></ul></ul></ul>Using the general availability techniques…
  36. 36. Cost-effectiveness of recoverability techniques in EP Have to consider if implementing recoverability is cost-effective? <ul><li> </li></ul><ul><li>Applications not requiring recoverability solution </li></ul><ul><ul><li>Applications where events are symptom of some underlying problem and will occur again </li></ul></ul><ul><ul><li>Systems looking for statistical trends, which might be based on sampling </li></ul></ul><ul><li> </li></ul><ul><li>Mission critical applications </li></ul><ul><ul><li>Lost state might result in incorrect decisions </li></ul></ul><ul><ul><li>Recoverability is a must </li></ul></ul>
  37. 37. Usability Considerations
  38. 38. Usability 101 Definition by Jakob Nilsen * * http :// www . useit . com / alertbox / 20030825 . html Learnability: How easy it is for Users to accomplish basic tasks the first time they encounter the system? Efficiency: Once users have Learned the system, How quickly can they perform tasks? Memorability: When users return after period of not using the system, How easily can they reestablish proficiency ? Errors: How many errors do users make, how severe are these errors, and how easily they can recover from the errors? Satisfaction: How pleasant is it to use the system? Utility: Does the system do what the user intended?
  39. 39. In this part of the tutorial we’ll talk about Build time IDE Runtime control and audit tools Correctness – internal Consistency Debug and validation Consistency with the environment - Transactional behavior
  40. 40. Build time interfaces Text based programming languages Visual languages Form based languages Natural languages interfaces
  41. 41. Text-based IDE (Sybase/CCL)
  42. 42. Another Text-based IDE (Apama)
  43. 43. Visual language – StreamSQL EventFlow (Streambase)
  44. 44. Visual language – StreamSQL EventFlow (Streambase) – cont.
  45. 45. Form based language – Websphere Business Events (IBM) <ul><li>Whenever transfer occurs more than once in a month, then the Account Manager </li></ul><ul><li>should be notified and Sales should contact the customer. </li></ul>
  46. 46. <ul><li>Business-oriented tool that intended to define business concepts that involve events and rules without consideration of the implementation details </li></ul><ul><li>The tool uses an adaptation of the OMG's SBVR standard </li></ul>Natural language for event processing Based on work done by Mark H Linehan (IBM T.J.Watson Research Center) free text Frequent big cash deposit pattern is defined as “at least 4 big cash deposits to the same account”, where big deposit decision depends on customer’s profile. structured English A derived event that is derived from a big cash deposit using the frequent deposits in same account applying threshold the count of the participant event set of frequent big cash deposits is greater than or equal to 4.
  47. 47. Run time tools Performance monitoring Dashboards Audit and provenance Two types of run time tools: Monitoring the application Monitoring the event processing systems
  48. 48. Performance Monitoring (Aleri/Sybase)
  49. 49. Dashboard (Apama)
  50. 50. Dashboard Construction (Apama)
  51. 51. Dashboard (IBM WBE)
  52. 52. Provenance and audit Tracking all consequences of an event Tracking the reasons that something happens Within the event processing system: Derivation of events, routing of events, Actions triggered by the events
  53. 53. Example: Pharmaceutical pedigree
  54. 54. Validation and debugging Debugger Testing and simulation Validation
  55. 55. Breakpoints and Debugging
  56. 56. Breakpoints and Debugging (StreamBase)
  57. 57. Testing & simulation – IBM WBE
  58. 58. <ul><li>Changing a certain event, what are the application artifacts affected? </li></ul><ul><li>What are all possible ways to produce a certain action (derived event)? </li></ul><ul><li>There was an event that should have resulted in a certain action, but that never happened! </li></ul><ul><li>“ Wrong” action was taken, how did that happen? </li></ul>Application validation
  59. 59. Validation techniques Static Analysis <ul><li>Navigate through mass of information wisely </li></ul><ul><li>Discover event processing application artifacts dependencies and change rules with confidence </li></ul>Dynamic Analysis <ul><li>Compare the actual output against the expected results </li></ul><ul><li>Explore rule coverage with multiple scenario invocation </li></ul><ul><li>System consistency tests </li></ul>Build-time Development phase Run-time Development and production phases Analysis with Formal Methods <ul><li>Advanced correctness and logical integrity observations </li></ul>Build-time Development phase
  60. 60. <ul><li>Disconnected agents </li></ul><ul><li>Event possible consequences </li></ul><ul><li>Event possible provenance </li></ul><ul><li>Potential infinite cycles </li></ul>Static analysis
  61. 61. <ul><li>Event instance forward trace </li></ul><ul><li>Event instance backward trace </li></ul><ul><li>Application coverage by scenario execution </li></ul><ul><li>Agent evaluation in context </li></ul>Dynamic Analysis Runtime Scenario Dynamic Analysis Component EP Application Definition History Data Store Observations for dynamic analysis EP system invocation on runtime scenario Results analysis for correctness and coverage Analysis results
  62. 62. <ul><li>Static analysis methods enable to derive a set of “shallow” observations on top of the </li></ul><ul><li>application graph  an agent can be physically connected to the graph, but not reachable </li></ul><ul><li>during the application runtime (e.g., due to a self-contradicting condition) </li></ul>Advanced verification with formal methods <ul><li>Agent/derived event unreachability </li></ul><ul><li>Automatic generation of scenario for application coverage </li></ul><ul><li>Logical equivalence of several agents </li></ul><ul><li>Mutual exclusion of several agents </li></ul>
  63. 63. Correctness The ability of a developer to create correct implementation for all cases (including the boundaries) Observation: A substantial amount of effort is invested today in many of the tools to workaround the inability of the language to easily create correct solutions
  64. 64. Some correctness topics The right interpretation of language constructs The right order of events The right classification of events to windows
  65. 65. The right interpretation of language constructs – example All (E1, E2) – what do we mean? A customer both sells and buys the same security in value of more than $1M within a single day Deal fulfillment: Package arrival and payment arrival 6/3 10:00 7/3 11:00 8/3 11:00 8/3 14:00
  66. 66. Fine tuning of the semantics (I) When should the derived event be emitted? When the Pattern is matched ? At the window end?
  67. 67. Fine tuning of the semantics (II) How many instances of derived events should be emitted? Only once? Every time there is a match ?
  68. 68. Fine tuning of the semantics (III) What happens if the same event happens several times? Only one – first, last, higher/lower value on some predicate? All of them participate in a match?
  69. 69. Fine tuning of the semantics (IV) Can we consume or reuse events that participate in a match?
  70. 70. Fine tuning of semantics – conclusion <ul><li>Some languages have explicit policies: </li></ul><ul><li>Example: CCL Keep policies </li></ul><ul><ul><li>KEEP LAST PER Id </li></ul></ul><ul><ul><li>KEEP 3 MINUTES </li></ul></ul><ul><ul><li>KEEP EVERY 3 MINUTES </li></ul></ul><ul><ul><li>KEEP UNTIL (”MON 17:00:00”) </li></ul></ul><ul><ul><li>KEEP 10 ROWS </li></ul></ul><ul><ul><li>KEEP LAST ROW </li></ul></ul><ul><ul><li>KEEP 10 ROWS PER Symbol </li></ul></ul>In other cases – explicit programming and workarounds are used if semantics intended is different than the default semantics
  71. 71. The right order of events - scenario <ul><li>Bid scenario- ground rules: </li></ul><ul><li>All bidders that issued a bid within the validity interval participate in the bid. </li></ul><ul><li>The highest bid wins. In the case of tie between bids, the first accepted bid wins the auction </li></ul>===Input Bids=== Bid Start 12:55:00 credit bid id=2,occurrence time=12:55:32,price=4 cash bid id=29,occurrence time=12:55:33,price=4 cash bid id=33,occurrence time=12:55:34,price=3 credit bid id=66,occurrence time=12:55:36,price=4 credit bid id=56,occurrence time=12:55:59,price=5 Bid End 12:56:00 ===Winning Bid=== cash bid id=29,occurrence time=12:55:33,price=4 Trace: Race conditions: Between events; Between events and Window start/end
  72. 72. Ordering in a distributed environment - possible issues Even if the occurrence time of an event is accurate, it might arrive after some processing has already been done If we used occurrence time of an event as reported by the sources it might not be accurate, due to clock accuracy in the source Most systems order event by detection time – but events may switch their order on the way
  73. 73. Clock accuracy in the source Clock synchronization Time server, example:
  74. 74. Buffering technique <ul><li>Assumptions: </li></ul><ul><ul><li>Events are reported by the producers as soon as they occur; </li></ul></ul><ul><ul><li>The delay in reporting events to the system is relatively small, and can be bounded by a time-out offset ; </li></ul></ul><ul><ul><li>Events arriving after this time-out can be ignored. </li></ul></ul><ul><li>Principles: </li></ul><ul><ul><li>Let  be the time-out offset, according to the assumption it is safe to assume that at any time-point t , all events whose occurrence time is earlier than t -  have already arrived. </li></ul></ul><ul><ul><li>Each event whose occurrence time is To is then kept in the buffer until To+  , at which time the buffer can be sorted by occurrence time, and then events can be processed in this sorted order. </li></ul></ul>Sorted Buffer (by occurrence time) To t > To +  Producers Event Processing
  75. 75. Retrospective compensation <ul><li>Find out all EPAs that have already sent derived events which would have been affected by the &quot;out-of-order&quot; event if it had arrived at the right time. </li></ul><ul><li>Retract all the derived events that should not have been emitted in their current form. </li></ul><ul><li>Replay the original events with the late one inserted in its correct place in the sequence so that the correct derived events are generated. </li></ul>
  76. 76. Classification to windows - scenario Calculate Statistics for each Player (aggregate per quarter) Calculate Statistics for each Team (aggregate per quarter) Window classification: Player statistics are calculated at the end of each quarter Team statistics are calculated at the end of each quarter based on the players events arrived within the same quarter All instances of player statistics that occur within a quarter window must be classified to the same window, even if they are derived after the window termination.
  77. 77. Transactional Behavior <ul><li>In a complete transactional system: </li></ul><ul><li>In event processing system this implies: </li></ul>Nothing gets out of the system until the transaction is committed <ul><li>The ability to track the effects of event (forward and backwards) </li></ul><ul><li>The system knows to withdraw events from the EPAs’ internal state </li></ul>
  78. 78. Transactional behavior in event processing? Typically, event processing systems have decoupled architecture, and does not exhibit transactional behavior However, in several cases event processing is embedded within a transactional environment
  79. 79. CASE I: Transactional ECA at the consumer side When a derived event is emitted to a consumer, there is an ECA rule, with several actions, that is required to run as atomic unit. If failed, the Derived event should be withdrawn
  80. 80. CASE II: An event processing system monitors transactional system In this case, the producer may emit events that are not confirmed and may be rolled back.
  81. 81. Case III: Event processing is part of a chain There is some transactional relationship between the producer and consumer The event processing system should transfer rollback notice from the consumer to the producer <ul><li>Need to be able to track the effects/causes of event (forward and backwards) </li></ul><ul><li>This implies rollback of other events </li></ul>
  82. 82. Case IV: A path in the event processing network should act as “unit of work” Example: the “determine winner” fails, and the bid is cancelled, all bid events are not kept in the event stores, and are withdrawn for other processing purposes
  83. 83. Transactions in event processing systems <ul><li>Usually in transactional systems there is assumption that a transaction time is short </li></ul><ul><li>This is not necessarily the case in event processing systems </li></ul>All (E1, E2) - E2 arrived 5 days after E1 - The processing of the pattern failed – What do we mean? Withdraw only E2? Withdraw also E1 after 5 days?
  84. 84. Security and Privacy Considerations
  85. 85. Security, privacy and trust Security requirements ensure that operations are only performed by authorized parties, and that privacy considerations are met. Based on Enhancing the Development Life Cycle to Produce Secure Software [DHS/DACS 08] Characteristics of secure application: Containing no malicious logic that causes it to behave in a malicious manner. Trustworthiness Recovering as quickly as possible with as little damage as possible from attacks. Survivability Executing predictably and operating correctly under all conditions, including hostile conditions. Dependability
  86. 86. Towards security assurance Identify and categorize the information the software is going to contain Low sensitivity – The impact of security violation is minimal High sensitivity – Violation may pose a threat to human life Develop security requirements <ul><li>Access control (Authentication) </li></ul><ul><li>Data management and data access (Authorization) </li></ul><ul><li>Human resource security (Privacy) </li></ul><ul><li>Audit trails </li></ul>
  87. 87. Security in event processing systems <ul><li>Only authorized parties are allowed to be event producers or consumers </li></ul><ul><li>Incoming events are filtered to avoid events that producers are not entitled to publish </li></ul><ul><li>Consumers only receive derived events to which they are entitled (in some cases only some attributes of an event) </li></ul><ul><ul><li>Extensive work on secure subscription was done in pub/sub systems </li></ul></ul>authorized authorized
  88. 88. Security in event processing systems – cont. <ul><li>Unauthorized parties can not make modifications in the application </li></ul><ul><ul><li>Off-line definition modifications or hot updates </li></ul></ul><ul><li>All database and data communications links used by the system are secure, including data transfer in distributed environments </li></ul><ul><li>Keeping auditable logs of events received and processed </li></ul><ul><li>Preventing spam events </li></ul><ul><ul><li>Can all twitter events be trusted? </li></ul></ul>
  89. 89. Security patterns in event processing <ul><li>Application definitions access patterns </li></ul><ul><ul><li>Access type control – view/edit/manage </li></ul></ul><ul><ul><li>Access destination control – application parts access restrictions per user/group </li></ul></ul><ul><ul><li>Both above should be enforced in development and runtime phases (hot updates) </li></ul></ul><ul><li>Event data access patterns </li></ul><ul><ul><li>Access to events satisfying a certain condition (selection) </li></ul></ul><ul><ul><li>Access to a subset of event attributes (projection) </li></ul></ul>
  90. 90. Summary
  91. 91. Summary Non Functional properties determine the nature of event processing applications – distribution, availability, optimization, correctness and security are some of the dimensions There are often the main decision factor in selecting whether to use an event processing system, and in the selection among various alternatives.