Complex Event Processors are capable of handling large volumes of events - by filtering, aggregating or detecting patterns. Java Applications use a CEP to pre-process incoming signals. These applications can also generate the events - for example the user click and navigation behavior in the web application - and report them to the CEP. The web application can subsequently utilize the outcomes from the CEP to for example intelligently guide the user or present relevant details. This session will show various ways in which a CQL based CEP can be integrated into a Java application to enhance the web application's behavior.
The intended audience for this presentation consists of experience Web Java and Enterprise Java developers.
- introduction to Complex Event Processing
- demonstration of CQL event processing on events arriving on JMS
- discussion of how the web application can absorb the CEP results
- demonstration of a simple Web Shop application that published events to the CEP and utilizes the CEP results
- discussion on when and how CEP can add value to Java applications
Why and how to engage a Complex Event Processor from a Java Web Application
1. Why and How to Engage a Complex Event Processor from a Java Web Application Lucas Jellema CTO, AMIS (The Netherlands) Oracle ACE Director
2. An Event A fact When it becomes a fact Timestamp! Of potential interest to anyone With some information associated with it When the fact is changed, that too may constitute an event Note that even when no change occurs, that could be considered an event (absence of expected event)
3. The Data Avalanche The league of real time events Continuous stream of a multitude of tiny events with hardly any payload, most of which are not interesting at all Sent from physical sensors (temperature, pressure, RFID, security gates), process sensors, Twitter, manufacturing equipment, database triggers, web servers, ESBs, stock trade tickers, sport statistics, RSS, network switches, …
5. Making Sense of Non-senseComplex Event Processing Useful information could be hidden in those streams of events – and CEP extracts it By detecting patterns, deviations, aggregations in these event streams CEP publishes ‘meaningful’ (business) events to report its findings – by executing some Event Processing Langugae events Complex Event Processor
6. Continuous Query Language (CQL) Standard extension/complement to SQL For handling large volumes of continuously arriving events Executing the queries continuously Operates on streams of continually arriving events instead of static relational data sets Events can be enriched with context Results keep being produced as event stream Is used for comparing events of same type, to find Trends and Patterns Exceptions Aggregates Complex Event Processor CQL
7. Example CQL Query Report the average temperature Aggregated over last 60 seconds Reported every 30 seconds Determined per Cluster Temperaturereadings Complex Event Processor Average temperature values (over 60s, per cluster, every 30s)
8. Event Processing Network Adapters – consuming or producing events Channels – carrying specific event types Processors – running CQL
9. Complex Event Processing Alert JMS Dashboard SOA Application DB Data Warehouse WebService File events Complex Event Processor JMS HTTP JMX File DB Event Consumer
11. Various CEP offerings available Progress Software (Apama) Aleri (Aleri and Coral8) IBM (WebSphere Business Events, System S/InfoSphere Streams, ILog rule engine) StreamBase Systems, TIBCO Software (Tibco Business Events) Oracle (CEP - fka BEA WLEvs - and Glassfish OpenESB IEP) UC4 (Senactive) EsperTech JBoss Drools Fusion
12. Oracle CEP 11g Architecturefka BEA WLEVS(erver) Design Time Run Time CEP Visualizer(web console) Other WebApp Eclipse http CEP plugin Web (Bayeux)Server CEP Server DB JMS
17. Case of Temperature Hospital Temperature sensors Cooling units Maternity Ward and Intensive Care units Fire detection Frequent readings (subsecond) Monitoring for: trends of threshold crossing, fire!, fault detection
18. Demo: Temperature Sensors Application should display a dashboard for temperature sensor readings The user is interested in cluster aggregates , deviations from pre-set values and broken sensors How would a Java Application process thousands of events per second extract and report aggregates detect and report missing values Note: user wants real time updates – but not more frequently than once per minute
19. Technical Layout Temperature Sensors are all over the place Clustered per room or location Temperature Readings arrive over JMS Results are sent through JMS to dashboard JMS queue for temperature aggregates Per cluster, once every 30 seconds JMS queue for failed sensor detection
21. Event Processor for aggregation Group by Cluster Report once every 30 seconds Over all events in the last 60 seconds
22.
23. Sensor Failure When a sensor has not reported for 30 seconds, we consider it failed And potentially broken down CEP needs to trap the missing events and report those
26. Feeding Temperature Aggregate Findings into real time BAM CEP publishes result events on JMS CEP is not aware of Business Activity Monitoring BAM creates an enterprise message source Based on JMS and MapMessages Feeding into a Data Object in the Active Data Cache BAM listens on the JMS Queue and maps the messages to the ADC Data Object
29. Java Application producing the events to process The user interaction with an application produces many events Navigation, Selection, Submission (scroll, expand/collapse, drill down) Most events are not recorded persistently The events can reveal interesting information about the application, the data and the users For example in a web shop The information could be used to improve the effectiveness of the application
36. Derive intelligence from WebShop events Actual Metrics: #searches in last 5 minutes, #buyers in the last 24 hours, #search-to-views drilldowns Top 10 search terms (in last [24] hour) Top 10 useless search terms After search on term there is no view event Most Effective search terms Search is followed by (one or more) view events and subsequently a buy event Items frequently viewed but not often bought #buy/#views is ranks low
37. WebShop & CEP WebShop sends JMS events to single queue Whenever a user in a session searches, drills down or buys Type of event, search term or item identifier, session id CEP reads from JMS channel Routes based on event type Processes events using CQL processor Emits derived events to JMS queue WebShop listens to JMS queue Updates application scope ‘top10’ beans Provides dashboard based on ‘top10’ beans Potentially using Server Push (or periodic poll)
38. Webshop feeding into CEP and receiving refined feedback WebShop Top 10 Popular Search Terms Top 10 Useless Search Terms Most Effective Search Terms search view buy
41. Summary Complex Event Processing For analysis of real time, high volume, continuous streams of fine grained, non-persistent events Used for detecting patterns, spotting exceptions, calculating aggregates and tracking trends CEP uses CQL queries to derive information from event streams – continuously, as events continue to arrive Results are published as (higher level) events – often to JMS queues
42. Summary [Java] Applications can have CEP pre-process events to do complex analysis and filtering producing business events that applications can understand Java Applications produce events that CEP can process And possibly feed back into the applications JMS and http are primary communication channels
43. Resources Presentation and demos are on our blog http://technology.amis.nl/blog Contact me at:lucas.jellema@amis.nl Note: Chapter 19 in Oracle SOA Suite 11gHandbook describesOracle CEP and BAM
Editor's Notes
Demo in plain SQL: http://technology.amis.nl/blog/3569/simple-event-processing-using-sql-with-analytical-functions-to-detect-traffic-patterns
Wat zijn de populairste zoektermen van dit moment (bijvoorbeeld de top 10 van de afgelopen 24 uur) – dit is een vrij eenvoudige aggregatie – gegroepeerd op zoekterm en met om de zoveel uur een rapportage in de vorm van tien events voor de top zoek-termenWat zijn de populairste waardeloze zoektermen? Dat zjin zoektermen die de gebruikers kennelijk niet de door hen gewenste resultaten opleveren omdat na het zoek-event met die term niet een bekijk-event volgt, maar een nieuw zoek-event of helemaal geen event meer.Wat zijn kennelijk de meest succevolle zoektermen? Hier zoeken we naar het patroon van zoek-event gevolgd door een of meerdere bekijk–events en leidend tot een koop-event zonder dat er nog een ander zoek-event tussen zitWat zijn items die wel vaak bekeken maar niet vaak gekocht worden? Dat zijn producten waarvoor het percentage kijk+koop van het totaal aantal kijk-events laag is. Ook deze lijst kan bijvoorbeeld iedere 12 of 24 uur worden opgehoest door CEP