Your SlideShare is downloading. ×
Summer Tech Day 2014 - Fast Data
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Summer Tech Day 2014 - Fast Data

166

Published on

Fast Data – przetwarzanie …

Fast Data – przetwarzanie
zdarzeniowe w kontekście Big Data

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
166
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Confidential – Oracle Internal1 Fast Data – przetwarzanie zdarzeniowe w kontekście Big Data Jarosław Stępień Waldemar Thiel
  • 2. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Event Processing 12c • Modern Computing Challenges • Solution Product Overview • Oracle Event Processing Applications • Continuous Query Language (CQL) • Internet of Things (IoT) • Fast Data • 12c Features
  • 3. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | The application evaluates if the roaming asset (device) is within a certain distance of the business. If it is, the system will generate an event to send the customer a promotional message. This powerful example encompasses the integration of JMS, Real Time Spatial Analysis and Big Data integration Mobile Marketing - OEP in Action: DEMO
  • 4. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Modern Computing Challenges Does the problem you want to solve have any one or more of the following conditions?  Requires high-throughput and low-latency processing  Continuously streaming data (Log Files, converting Batch to Real Time)  Real-time correlation between multiple incoming data sources  Time-sensitive alerts, aggregations and calculations  Needs to look for patterns in the data stream (+Spatial Analysis)  Identify when some event(s) should have happened and did not.  Data does not need to be stored, if there is nothing of interest in it  Problem is more easily solved by analyzing before storing in DB Conditions for Event Processing
  • 5. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Solution Overview
  • 6. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Solution Product Overview Solution involves one or more of the following: • High Volume • Continuous Streaming • Sub-Millisecond Latency • Disparate Sources • Time-Window Processing • Pattern Matching • Business Event Visualization Oracle Event Processing OEP Streaming Event Data Alerts, Actions Filtering, Pattern Matching, Missing Events, Aggregations, Correlations, Calculations, Geo-Spatial
  • 7. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Solution Product Overview Inverted Database  Data is ‘static’  Queries are ‘dynamic’ • Data (event) is ‘dynamic’ • Queries are ‘static’
  • 8. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Event Processing POJO App Frameworks Spring Services Stream Management Core Event Infrastructure Complex Event Processor Real-Time Kernel Data Cartridges Extended Event Infrastructure Cluster Management HTTP Pub/Sub Engine Event Repository Coherence Foundation Services Security Logging  Processes Streaming Data In-Memory & in Real-Time  Event Analysis: Pattern Matching, Missing Events, Aggregations, Correlations, Calculations, Coherence Integration, Spatial Functions, Big Data Integration  Performs Aggregations and Correlates Multiple Streams of Data  Integrates easily with other Oracle products: Big Data (Hadoop & NoSQL) – FAST DATA, SOA (EDN), Coherence, Spatial, RTD, etc. StreamingEventData Alerts, Actions Solution Overview: Oracle Event Processing (OEP)
  • 9. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Solution Overview Filtering, Correlation & Aggregation • New stream filtered for specific criteria, e.g. stock price > $22 • Process events from different streams together • Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour Detect Absence of Events and Missing Events • Event “A” NOT followed by Event “B” within 10 minutes • Event “A”, Event “B” should occur next, but Event “C” occurs instead Oracle Event Processing
  • 10. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Perform Calculations in Real-Time Perform Calculations on Data From Sensors: • Use last X seconds or minutes of data OR • Use last X data points • Smooth out “noise” in the data
  • 11. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Programming Model
  • 12. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Input Adapter Channel Input Adapter Channel Business Logic (CQL) Channel Channel Channel Output Adapter Output Adapter Oracle Event Processing Application DB Input adapters connect to data sourcesChannels help control the flow of data and can be tuned for optimal performanceDatabases and Coherence caches can be referenced directly in CQL processorsCQL processors contain filtering, correlation, aggregation and pattern matching business logicOutput adapters send data and alerts to downstream systems and business processes Coherence Business Logic (CQL) Business Logic (CQL)
  • 13. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | High-Performance In-Memory Data Processing Input Adapter Data Input Adapter Data Channel Business Logic (CQL) Channel Data Data Analytics Channel Business Logic (CQL) Enrich Output Adapter Data Data Data Data Analytics: Continuously Sliding Time Windows of Streaming Data, Filtering, Correlations, Calculations, Aggregations, Pattern Matching, Missing Event Detection, Spatial Analysis, etc. Enrichment: Integrate with data from DB, Coherence, NoSQL, Hadoop etc. Oracle Event Processing
  • 14. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Administration & Monitoring View the data flow of the application “Event Processing Network” (EPN)
  • 15. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Administration & Monitoring Monitor throughput and latency between any two nodes in the application
  • 16. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Event Processing & Coherence Challenges Benefits Handle and correlate events in real-time, including support for multiple patterns: • Pre-processing (buffer OEP) • Within OEP (to cache reference data) • Post OEP (to expose processed events to consuming apps) • High throughput for storing data • Aggregation and event querying • Pattern implementation flexibility combining two complementary technologies Data Grid OEP Consolidated & in-context Data
  • 17. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Simple Filtering SELECT * FROM inputChannel [NOW] WHERE eventValue > 10 Continuously calculate the last hour sales by store SELECT SUM(amount) as salesTotal, storeID FROM inputChannel [range 60 minutes] GROUP BY storeID Calculate the average of the last 2 stock ticks by stock symbol SELECT AVG(stockPrice) as avgPrice, stockSymbol FROM inputChannel [PARTITION BY stockSymbol ROWS 2] GROUP BY stockSymbol Sample CQL Queries Filter for events meeting specific threshold values Running total of up- to-the- moment sales by store Average of the last 2 stock ticks by symbol
  • 18. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | CQL view that joins a cache containing reference data to an individual streaming event to provide additional context for further processing. Cache Join Query Sample CQL Queries
  • 19. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Pattern Matching  Powerful concept that allows identification of complex event patterns  Defined as regular expressions PATTERN (X+ Y+) 1 or more X events … … followed by 1 or more Y events
  • 20. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Find passengers stuck in security when their flight reaches “final boarding”. SELECT stuck.reservationLocator, 'STUCK' as state FROM PassengerStateEventChannel MATCH_RECOGNIZE ( PARTITION BY reservationLocator MEASURES Entered.reservationLocator AS reservationLocator PATTERN (CheckIn Entered NotExited*? Final) DEFINE CheckIn AS state = 'CHECKIN', Entered AS state = 'ENTERED', NotExited AS state != 'EXITED', Final AS state = 'FINAL' ) AS stuck Find passengers who are stuck in security when their flight is in the “FINAL BOARDING” process. Sample CQL Queries
  • 21. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Flight is in final boarding! Find passengers stuck in security. Pattern Matching
  • 22. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Bus Stop Alerts Oracle Spatial • Integrated combination of Oracle Database Spatial Analysis and The Oracle Event Processing Technologies for the Oracle SOA Suite – Oracle Spatial • Location-enable business applications • Manage all geospatial data including vector and raster data, topology, and network models. – Oracle Event Processing • Handles the real time geographical streaming data (GPS) • Provides the geofencing analysis with the assigned geographical assets • Close, Entering and Exiting relationship determination • Transparently integrates with SOA Suite technologies to initiate Business Processes or Enterprise Services based on the geofencing triggersSpatial Data Cartridge Real-Time Geostreaming and Geofencing using Integrated Event Processing & Geographical Analysis
  • 23. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Real Time Tracking via GPS and resource evaluation in relation to virtual geographical areas Real Time Dynamic definition of virtual geographical complex (polygon) areas Real-Time Spatial Geo-Fencing Solutions
  • 24. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Point-in-Polygon Matching OEP detects entry into a pre-defined area
  • 25. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Dynamic Query Generation Business Users Modify Processing in Real-Time Simple forms can be used to allow users to add, modify or delete CQL queries at run-time without stopping the application or the server allowing existing processing to continue running while business users dynamically make modifications.
  • 26. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Internet of Things (IoT)
  • 27. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Solution Overview: Oracle Event Processing (OEP)  Architecture designed and Optimized for Java Enabled Devices/Gateways  High-speed real-time device data capture and analysis helps mitigate the risk of down-time and helps with continuous process improvement  Enables local real-time process monitoring to detect degradation of performance, thus helping with initiation of alerts or requests for operations and maintenance actions  Significantly reduces noise to signal ratio of data reaching the server, enabling savings in bandwidth and scalability costs  Faster time to market for devices and easy over-the- air updates post-deployment.
  • 28. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Event Processing for Java Embedded • Runs on devices capable of running Java SE Embedded • Key component of Internet of Things (IoT) • Smaller, lighter-weight version of the same OEP product • No need to learn new skills to program application for devices OEP
  • 29. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | OEP OEP OEP OEP OEP OEP in Device to Data Center Deployments Real-time local data analysis for real-world event data • Upstream nodes perform basic filtering and aggregation • Larger servers downstream perform complex combining and correlation across multiple streams • OEP in embedded devices allows initial processing to be handled by less powerful devices near the origin/edge. Event Flow Edge devices Servers Gateways Enterprise Apps OEP OEP OEP OEP
  • 30. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Fast Data
  • 31. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Fast Data for Big Data Fast Data Big Data Infrastructure Oracle BI Foundation Suite Oracle Real-Time Decisions Endeca Information Discovery Business Analytics Oracle Event Processing Oracle Big Data Connectors Oracle Data Integrator Oracle Advanced Analytics Oracle Database Oracle Spatial & Graph Apache Flume Oracle GoldenGate Oracle NoSQL Database Cloudera Hadoop Oracle R Distribution
  • 32. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 1. Big data ≠ Infinite storage Yes, storage is cheap but it helps to have clean data, with context and less redundancy 2. Hadoop is batch-oriented and there is inherent latency "With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes […] it will never be true real-time. " * Raymie Stata, Yahoo! CTO (June 2011) Some Challenges Working with Big Data : http://www.theregister.co.uk/2011/06/30/yahoo_hadoop_and_realtime/ minutes
  • 33. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 1. Filter out noise (ex: data ticks with no change), add context (by correlating multiple sources), increase relevance 1. Identify certain critical conditions as you insert data into the warehouse Move time-critical analysis to front of process Filter & Correlate Filter out, correlate Use Event Processing Techniques
  • 34. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Fast Data Getting Ahead of the Curve Big Data minutesms Fast Data Historicaldepth:deep Historicaldepth: shallow Example: analysis of traffic patterns and congestion times for urban planning Example: monitoring of traffic cameras to ensure given license plates are not in use on multiple vehicles Add “depth” to your fast data by merging output of MapReduce to stream processing
  • 35. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | COMPANY OVERVIEW • Motorola Solutions connects people through technology • Motorola Solutions serves both enterprise and government customers with core markets in public safety government agencies and commercial enterprises • Global presence with 23,000 employees worldwide in 65 countries with sales in over 100 countries CHALLENGES/OPPORTUNITIES • Provide a Fast Data intelligence layer for Big data streaming video feeds • Event Processing analyzes Real Time Video meta-data for distinct patterns of interest to Government agencies and law enforcement • Centralized Exalogic implementation PROJECT OBJECTIVES • Projects worldwide using a reusable scalable architecture supporting 2800 or more cameras • Streaming video meta data interfaces with IOmniscient processing for Face recognition and License plate monitoring • Event patterns for duplicate plates within temporal period beyond distance capabilities • Speeding analysis with driver recognition RESULTS • Ongoing partner relationship delivering planned projects in Mexico, Oman • Prototyping successfully completed with live video feed data • Foundational technology for Smart Cities platform solutions Motorola Smart IPVS : Vehicle Capacity & Flow Control WARNING: Suspect License plate – CABO . MEXICO CITY WARNING: Driver Criminal Record
  • 36. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Event Processing 12c new features
  • 37. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | JDeveloper IDE for OEP 12c
  • 38. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | JDeveloper IDE for OEP 12c • Create Event Processing Network • Graphically develop an OEP application • Drag EPN components to canvas and configure • Deploy to Configured OEP Servers • Define OEP Servers and Deployment Profiles
  • 39. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Improved Installer • Installer and Domain Creation Wizard Improvements • Mac OS X is supported for development.
  • 40. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | CQL Pattern Templates Starts the Query Logic for the Developer
  • 41. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | List of 12c Enhancements • CQL enhancements • Spatial Improvements • Application compiler • Application and CQL testing • New and Improved adapters • Public Cartridge API • Improved Installer • Improved Coherence Cartridge • Support for JDeveloper • EDN integration • Business Rules integration • Performance optimizations
  • 42. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Request Request Event Data Event Data SOA Composite Instances OEP Application SOA EPN EDN Integration SOA Composite & Event Processing Network
  • 43. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Customer1 $600 Customer1 $600 Customer1 $700 > $1000 per customer in 24 hours Customer1 $700 EDN Integration
  • 44. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |  High-Volume Low-Latency Event Processing Infrastructure  Event Processing Network (EPN)  Light-weight Java Application Server (embeddable)  Easily Customizable  Integrate with existing infrastructure and other Oracle Products (e.g. Coherence, Business Activity Monitoring, Database, Big Data Appliance, Data Mining, Spatial, NoSQL Database etc.)  Time Management & Pattern Matching  Continuously Perform Calculations Over Time Windows  Partition Event Streams By Key Values  Perform Complex Pattern Matching, Detect Missing Events  Adjust Core Business Logic in Real-time without Redeploying Oracle Event Processing (OEP) Summary
  • 45. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Summary: When To Use OEP 1 Business Logic Layer for Event-Driven/Coherence Applications 2 High-Volume Business Activity Monitoring Applications 3 Real-Time Spatial Applications 4 Fast Data: Real-Time Requirements with Big Data Infrastructure 5 High Volume Batch to Real-Time Conversion Projects 6 Internet of Things (IoT): Processing Data On and From Devices 7 Pattern Matching / Missing Events / Alerting / Fraud Detection
  • 46. Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

×