Real Time Event Processing and Decision Making


Published on

Speaker: Lyndon Adams
As Information Technology evolves and improves, the demands of industry and the internet continue to challenge its capabilities. Today's 'Internet of Things', with millions of devices reporting their status on the network every second, produces massive streams of events. Getting the information in these events consumed and correlated so that it can be used to make routing and processing decisions on related events is particularly challenging. As the data demands of these processes increases, traditional client-server architectures begin to fail. Combining event processing with patterns and knowledge gleamed from Big Data models enables whole new kinds of real-time applications. Spring Integration and other Spring projects play a big role in this new paradigm, adding developer productivity and ease of integration into the mix. Lyndon will discuss and demonstrate the architectural decisions and use cases that occur when Fast Data meets Big Data.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Real Time Event Processing and Decision Making

  1. 1. Real Time Event Processing and Decision Making By Lyndon Adams, Sr. Architect Pivotal Real Time © 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.
  2. 2. Introduction What does ‘Real Time’ mean? Applications are getting ‘realer’ and ‘timelier’ Where do we see real time use cases? Banking, Telco, Sensors, Smart Grids, Intelligence and others How can I make real time decisions in response to events? Route events to servers based on data affinity
  3. 3. What is Real Time? re·al Adjective /ˈrē(əә)l/ Actually existing as a thing or occurring in fact; not imagined or supposed Julius Caesar was a real person Used to emphasize the significance or seriousness of a situation or circumstance there is a real danger of civil war the competitive threat from overseas is very real Relating to something as it is, not merely as it may be described or distinguished (of a substance or thing) Not imitation or artificial; genuine the earring was presumably real gold True or actual his real name is James this isn't my real reason for coming (of a person or thing) Rightly so called; proper he's my idea of a real man Jamie is my only real friend Complete; utter (used for emphasis) the tour turned out to be a real disaster Adjusted for changes in the value of money; assessed by purchasing power real incomes had fallen by 30 percent an increase in real terms of 11.6 percent Of fixed property (i.e., land and buildings), as distinct from personal property he lost nearly all of his real holdings (of a number or quantity) Having no imaginary part (of an image) Of a kind in which the light that forms it actually passes through it; not virtual Applications with consequence need accurate and up-to-date information All related information needs to be brought together in one place Copies of data, caches, eventual consistency and batch windows can all lead to errors “How would you know the difference between the dream world and the real world?” Morpheus, The Matrix
  4. 4. What is Real Time? time Noun /tīm/ The indefinite continued progress of existence and events in the past, present, and future regarded as a whole one of the greatest wits of all time The progress of this as affecting people and things things were getting better as time passed Time or an amount of time as reckoned by a conventional standard it's eight o'clock Eastern Standard Time A point of time as measured in hours and minutes past midnight or noon the time is 9:30 A moment or definite portion of time allotted, used, or suitable for a purpose the scheduled departure time should we set a time for the meeting? An indefinite period traveling always distorts one's feelings for a time A more or less definite portion of time in history or characterized by particular events or circumstances Victorian times the park is beautiful at this time of year The conditions of life during a particular period times have changed The appropriate or expected time for something, in particular childbirth or death he seemed old before his time … Appropriateness is situational, defined by the user Past, current and predictions of the future may all be relevant Concurrent events and circumstances can affect the relevant ‘portion’ of time Each use may have its own definition of relevant time “Time is an illusion. Lunchtime, doubly so” Ford Prefect, The Hitchhikers Guide to the Galaxy
  5. 5. Pivotal Real Time Intelligence Pivotal RTI includes fault tolerant and distributed data management for the volatile data it uses. Kafka Rabbit MQ SpringXD Pivotal RTI ETL Throughput Pivotal RTI is designed to provide low latency, high throughput, elastic stream processing for applications with a very strong dependence on highly volatile reference data. Storm Map-Reduce Traditional Client-Server Rules/CEP Engines CPU Optimized Data Optimized
  6. 6. Use Cases
  7. 7. Real Time Pricing and Risk 43.7B records Threshold Detected Monte Carlo Simulation Market Data Trades Bids Offers Market Volatility isn’t a problem … it’s an opportunity. Position Price Fast Active Data Correction + Big Historical Data
  8. 8. Customer Behavior ‘Customer Options’ Targeted Marketing Location Event Fast Location Record Data Fabric Big Data Archive ‘Customer Habits’ MicroCube Big Data Analytics Periodic Refresh
  9. 9. Sensor Data Processing Continuous Real-time Analysis 000’s data points/sec Fast Data Monitor & Control Tuning & Optimization, Emergency Shutdown Maintenance Big Data Billing
  10. 10. Electricity must be consumed in the instant that it is produced. Smart Grids, Smart Cites Demand kVA Tower Occupancy Holidays/Events Weather Data Weather Data Supply kVA Insulator Temp Coolant Temp Conductor Temp Phase Variance Conductor Tension Stadium/Convention Schedule Emergency Operations Intrusion Detection1 Public Transport Demand Footing Torque Residential Consumption Vehicle Charging Output Freq. Local Sunshine kVA Output Bearing Temp Fast Data Big Data Windings Temp Shaft Torque Input Pressure Exhaust Pressure
  11. 11. Asset Management and Intelligence Events Details Fast Data Big Data Analysis
  12. 12. Complex Decisions Require More & Real-time Data Send to Voicemail Incoming Call What time is it (locally)? Am I travelling? Have I interacted w/ phone recently? Am I in a meeting? Am I driving? Is it family calling? Am I on vacation? Is call marked ‘urgent’? Do I know the caller? What phone am I using? Who am I talking to? Ring My Current Phone This information is individual and changing all the time. There isn’t time to go to disk for this information.
  13. 13. Solution
  14. 14. Data Optimized Applications ‘Pivot’ the Architecture App C Scale Stateless App w/ more instances App B App A App C App A Stateless Client App Additional Stateless Apps App B App A Partition the Data into memory Caches provide limited value if: •  The data sets are big •  The data changes Database and network contention limit scale/performance Deploy Apps to all nodes Events jump nodes only when necessary Database w/ all data Customers A-D Customers E-J Customers K-O Customers P-Z
  15. 15. Pivotal RTI with Pivotal One App Dev + + Fast Data Widgets Big Data PaaS
  16. 16. Spring Integration Adapters REST & RSS Twitter FTP & SFTP TCP & UDP Web Services JMS & AMQP Byte Streams Mail File Tuple, POJO, Message
  17. 17. Real Time Processing Merge Business Event Related Streams Sliding Window Threshold Detection Acknowledgement Raw Stream Event Eviction Validate Enrich Filter Widget Archive
  18. 18. Widgets Accumulators Set Arithmetic Geo-Spatial Threshold Analytic
  19. 19. Real Time Analysis Ingestion Linear Regression Example Operations Complex Events take action event data Predict y Is this normal? Is this fraud? periodic update parameters & noise Analyitics Big Data
  20. 20. Geo-Spatial Widgets Friends in Area Geo Heat Maps Distance to POI Enter/Leave Zone Debouncing
  21. 21. / Composition of Applications Root /myCompany Business Unit /Network Business Unit Shared Channel /CDR Ingestion Network Events SpringXD App /DroppedCalls Reference: /myCompany/Network/CDRIngestion/DroppedCalls /Marketing Business Unit /DroppedCallThreshold Spring Integration App Shared Container /ExcessiveDroppedCalls /CustomerRetention Issue Rebate Spring Integration App /Sales Business Unit /ContractLIfecycle Spring Integration App Shared Container /ExpringContracts
  22. 22. Demo
  23. 23. Real time dash board •  Streaming UI for call monitoring. •  Stack •  Spring Integration use case flows. •  GeoSpatial service and event set processing. •  OpenLayers, HighCharts and jqwidgets components. •  Tomcat and GemFire embedded client. •  Websockets.
  24. 24. Dropped Calls •  •  •  Geographic dropped call locations. Uses streaming dropped calls feed. OpenLayers functionality utilized for proximity dropped call count. –  Rules used for color coding. –  Radius set to 35 miles. •  System uses EPSG:4326 encoding standard then we project to map coordinate system.
  25. 25. Client contract alerts •  Alerts generated for a contract coming up for renewal that has experienced dropped calls in real time. •  Intersect of client Id from contracts and dropped calls sets. •  Uses declarative predicates to filter data. •  •  DroppedCall à dropped ? contractId : null; Contract à underreview and expired ? Id : null; •  Grid displays all dropped calls for filtered client. •  Grid provided grouping function etc.
  26. 26. Aggregated dropped calls by region •  •  •  •  Call records are tagged with region location using a geospatial location service. UK regions are specified upfront via a set of uk region polygons. Geospatial processes occurs for each call to determine geographic region. Geo region count managed by cluster.
  27. 27. High level architecture Feed Client Processing Storage GF PR Results Set Widget filter(contract) GF PR (colo) Contracts reviewClients filter(dropped calls) Calls droppedCalls Geo Widget geoZonedDroppedCalls W e b
  28. 28. Conclusion
  29. 29. Pivotal Real Time Intelligence Some stream processing cases are CPU intensive. Other cases benefit from colocation with volatile data. Spring Integration and SpringXD provide support for elastic Staged Event Driven Architectures for processing high volume event streams. runs Spring process flows inside of a managed data environment with new cluster-aware widgets
  30. 30. ? ? ? ?
  31. 31. Learn More. Stay Connected. •  Email •  Talk to us on Twitter: @springcentral •  Find Session replays on YouTube: