SlideShare a Scribd company logo
Event Processing for better
(Big) Data
Vinod Vydier
Middleware Specialist @ Oracle
Agenda
§  Why use event processing?
§  Event Processing Applications
§  Technical Architecture
§  Use of In-Memory data-grid
§  Use cases
Challenges Working with Big Data
• Storing Data has becoming cheap, however the
storage is not infinite and has to be managed to make
use of the data effectively.
• Hadoop has inherent latency for responding to real
time events (which can produce high volume data at
high velocity) and typically involves real responses.
• Event Processing helps in getting clean data with
context and less redundancy into HDFS, so the Hadoop
jobs can be more effective.
• Event Processing helps in responding back in real
time, and storing the data in HDFS for better historical
analysis.
Why use Event Processing Infrastructure
Application has any one or more of the following
conditions:
§  Requires high throughput and low latency
processing.
§  Has continuously streaming data.
§  Real-time correlation between multiple incoming data
sources.
§  Time-sensitive alerts, aggregations and calculations.
§  Needs to look for patterns in the data stream.
§  Data does not need to be stored, if there is nothing
of interest in it.
§  Problem is more easily solved by analyzing before
storing in HDFS.
Filtering, Real-time Intelligence for Big Data
VOLUME VELOCITY VARIETY VALUE
SOCIAL
BLOG
SMART
METER
101100101001
001001101010
101011100101
010100100101
FAST DATA
Event Processing Intelligence
GREATER
Stay ahead of Big Data
Filter out,
correlate
Move time-critical analysis to front of process
• Filter out noise (example: data ticks with no
changes), add context (by correlating multiple
sources), increase relevance.
• Identify certain critical conditions as you insert data
into the warehouse.
Getting ahead of the curve: Fast Data
Big Data
minutesms
Fast Data
Historicaldepth:deep
Historicaldepth:shallow
Example:
analysis of traffic
patterns and
congestion times for
urban planning
Example:
monitoring of traffic
cameras to ensure given
license plates are not in
use on multiple vehicles
Add “depth” to your fast data by
merging output of MapReduce to
stream processing
Adapter
Adapter
Processor
Adapter HDFS
Data Source
Queries
<<Source>>
<<Source>>
<<Sink>>
Service1 Service2
Export Import
Event Processing Network (EPN)
Event Processing Application
Queries
Channel
Channel
Channel
Channel
What is an Event Processing application
Data Source
Event Processing inputs
Ø  Streams
Ø  Continuous input, often in high-
volume
Ø  Time ordered
Ø  Does not end
Ø  Impossible to process / analyze in
real-time with traditional relational
database systems
Example: Raw Sensor Event
streams, GPS, Market Data Feeds
BA BOEING D 77.575 800 20080305 10:03:02:78
DO DUPOD
NT
D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
Event Processing provides a new data
management infrastructure to support and
analyze Streams in real-time
BA BOEING D 77.575 41.575
800
20080305 10:03:02:78
DO DUPONT D 41.575 3000 20080305 10:03:04:12
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
BA BOEING D 77.575 800 20080305 10:03:02:78
Filtering
Ø  New stream filtered for specific criteria,
e.g. stock price > $22
Ø  Correlation & Aggregation
Ø  Scrolling, time-based window metrics,
e.g. average # of stock trades in the
last hour
Ø  Pattern Matching
Ø  Notification of detected event patterns,
e.g. price changes A, B and C occurred
within 15 minute window
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
……
• Event Processing done in-Memory (not in Database)
• Logic is defined through Continuous Queries on the data
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
BA BOEING D 77.575 41.575
800
20080305 10:03:02:78
DO DUPONT D 41.575 3000 20080305 10:03:04:12
COMPLEX QUERIES
Event Processing outputs
Data crunching for Event Processing done in a
in-memory data grid
•  High throughput for storing data
•  Aggregation and event querying
•  Pattern implementation flexibility combining complementary
technologies
•  Handle and correlate events in real time, including support for
multiple patterns:
•  Pre processing (buffer inputs)
•  In Event Processing (to cache reference data)
•  Post Processing (to expose processed events to consuming
apps)
Data Grid
Event Processing
Consolidat
ed & in-
context
Data
Filtered/
Aggregat
ed Data
HDFS and
traditional storage
In-memory events on the data stream
n  Threshold Management
n  Detecting threshold conditions across multiple
event streams
n  Using cache to:
n  Allow dynamic configuration of thresholds
n  Add (via join) contextual data to support
aggregation
n  Using pattern matching to find sustained
conditions
n  Alert Generation
n  Using relations to represent state and state
transitions
n  Using “missing event” patterns to monitor
expected response(s)
n  Alarm Management
n  Using pattern matching to remove extraneous
alarm events
n  e.g. power off alarm preceded by tamper alarm
within (n) minutes
X
Alarm Filtering Scenario
Discard Power Off Alarm if there was a Tamper
Alarm for the same meter within the previous 5
seconds
Visualizing events on the data stream
JMS
Resource Locations
Matches and Alerts
SQL
Event Processing Application
JMS
Geo-Fencing Definitions
SQL
MapViewer
Manager
JMS Protocol Integration
n Common integration touch point with Service Bus
n Business Activity Monitoring integration
HTTP Publish/Subscribe
n Support pub/sub events between Event server and
web clients.
n Clients don’t need to poll for updates (unlike
traditional HTTP).
n Clients subscribe to and publish to event channels.
n Bayeux protocol
n Light weight and the payload is JSON
Visual/SOA integration with Event Processing
Event Processing High Level Architecture
JSON
Adapter CacheProcessor POJO
EPN (Event Processing Network) Elements
HTTP Pub/S
Query Plan and Real Time Monitoring
Event Driven SOA: Simplify Business
Complexity
•  Real-time business insight
•  Preempt and react instantaneously to Enterprise, Environmental and Global
Business conditions
•  Gain business insight using previously untapped, raw event sources
•  Hot-pluggable integration
•  Transparent SOA infrastructure interoperability
•  Distributed, deployment ready, pre-integrated, in-memory Data Grid,
and Java low latency determinism.
•  Lightweight high performance Java Event Server platform
•  Real-time business friendly analyst oriented
visualization layers
•  Powerful, extensible Event Processing Analysis abstraction
•  Business user dashboards
•  Business user domain specific natural language layers
•  Real-time predictive analytics
Event Processing use cases in different
industries
1.  Customer Experience
2.  Transportation, Logistics & Fleet Management
3.  Utilities: Demand & Response, Smart Meter
4.  Public Sector: Emergency Response,
Intelligence
5.  Telcos: Real Time billing & WiFi offloading,
Mobile billboard
Customer Experience
n  Industry focus on new buzzword: Customer
Experience
n  Desire to harness potential of social networks for
better targeted marketing
Event Processing can help with:
n  Monitoring in real-time customer activity (social
networks, location (e.g. proximity to stores, etc) and
identifying opportunities in real-time
n  Correlating with existing information (customer/
shopping profiles, etc.)
n  Generating real-time alerts
Transportation, Logistics and Fleet Management
n  Constant industry pressure for greater
efficiency
n  Need to differentiate through premium
services and greater reliability and visibility
n  Availability of cheap wireless sensors
(temperature, GPS, etc.) that can be included
in packages/containers/trucks
Event Processing can help with:
n  Real-time monitoring of inflow of data from
sensors
n  Trends detection / prediction (to rise, etc.)
n  Leveraging spatial/geo-location capabilities.
Utilities
n  Adoption of Smart Meters: concerns about bandwidth/ processing
power required to handle the information they generate, desire to offer
value-add services
n  Ever increasing electricity demand
n  Demand for real-time billing & analytics
n  Greater customer expectations re: outage & response times
n  Regulations
Event Processing can help with:
n  Alerting of consumption trends in real-time, enabling “Demand/
Response”
n  Real-time detection of problems (abnormal spikes in consumption
indicative of leaks, etc.)
n  Filtering out redundant or nested (ex: tree fell on the line) outage
errors and problems
n  Tracking of resources and personnel
Telco
n  Overloaded data networks and new strategies to offload traffic:
real-time billing vs. unlimited, offloading to WiFi, degradation of
service from 4G to 3G, etc.
n  GPS-enabled phones offer new location-based marketing
opportunities: “mobile billboards”
How can Event Processing help:
n  Event Processing infrastructure can handle massive amounts
of data generated by mobile devices, filter out, correlate and
aggregate in real-time to only retain valuable information
n  Event Processing can plug into all types of feeds, from devices
to social networks
n  Event Processing can be integrated with spatial and geo-
location technology to send location specific data to the user.
Public Sector
n  Heightened security requirements
n  Ever increasing population in urban areas drives optimization
requirements
n  Increasing number of real-time data: video feeds, GPS data,
traffic data, etc.
n  Applications: Security Intelligence, geo-fencing, “Smart
Cities”, traffic control, gateless tolls
How Event Processing can help:
n  Event Processing can be integrated with spatial and geo-
location technology to track location specific data with a user.
n  Event Processing can plug in any data feed such as video /
face recognition
n  Event Processing meets performance & availability
requirements in this space
Thanks for attending!!

More Related Content

What's hot

Five ways database modernization simplifies your data life
Five ways database modernization simplifies your data lifeFive ways database modernization simplifies your data life
Five ways database modernization simplifies your data life
SingleStore
 
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler AnswersLambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
VoltDB
 
HP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big DataHP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big Data
Rob Winters
 
DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)
DataStax
 
Intuit Analytics Cloud 101
Intuit Analytics Cloud 101Intuit Analytics Cloud 101
Intuit Analytics Cloud 101
DataWorks Summit/Hadoop Summit
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
SingleStore
 
Transforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case ExamplesTransforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case Examples
VoltDB
 
Our journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scaleOur journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scale
Itai Yaffe
 
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 Migration and Coexistence between Relational and NoSQL Databases by Manuel H... Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
Big Data Spain
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan Chhabra
Big Data Spain
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplow
yalisassoon
 
Data Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and GasData Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and Gas
Denodo
 
How to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top ContendersHow to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top Contenders
VoltDB
 
Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...
Big Data Spain
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business Innovation
VoltDB
 
Netflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering MeetupNetflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering Meetup
Blake Irvine
 
Converging Database Transactions and Analytics
Converging Database Transactions and Analytics Converging Database Transactions and Analytics
Converging Database Transactions and Analytics
SingleStore
 
Building the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free LifeBuilding the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free Life
SingleStore
 
Architecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data AnalyticsArchitecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data Analytics
Rob Winters
 
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo
 

What's hot (20)

Five ways database modernization simplifies your data life
Five ways database modernization simplifies your data lifeFive ways database modernization simplifies your data life
Five ways database modernization simplifies your data life
 
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler AnswersLambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
 
HP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big DataHP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big Data
 
DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)
 
Intuit Analytics Cloud 101
Intuit Analytics Cloud 101Intuit Analytics Cloud 101
Intuit Analytics Cloud 101
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
 
Transforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case ExamplesTransforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case Examples
 
Our journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scaleOur journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scale
 
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 Migration and Coexistence between Relational and NoSQL Databases by Manuel H... Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan Chhabra
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplow
 
Data Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and GasData Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and Gas
 
How to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top ContendersHow to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top Contenders
 
Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business Innovation
 
Netflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering MeetupNetflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering Meetup
 
Converging Database Transactions and Analytics
Converging Database Transactions and Analytics Converging Database Transactions and Analytics
Converging Database Transactions and Analytics
 
Building the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free LifeBuilding the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free Life
 
Architecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data AnalyticsArchitecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data Analytics
 
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
 

Viewers also liked

Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our Lives
Rukshan Batuwita
 
Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016
Mark Proctor
 
The Future of Work
The Future of WorkThe Future of Work
The Future of Work
Sandy Kemsley
 
Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016
Mark Proctor
 
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Mia Yuan Cao
 
Intelligent Business Processes
Intelligent Business ProcessesIntelligent Business Processes
Intelligent Business Processes
Sandy Kemsley
 
Installing Complex Event Processing On Linux
Installing Complex Event Processing On LinuxInstalling Complex Event Processing On Linux
Installing Complex Event Processing On Linux
Osama Mustafa
 
Reactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingReactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream Processing
Andy Piper
 
Comparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsComparative Analysis of Personal Firewalls
Comparative Analysis of Personal Firewalls
Andrej Šimko
 
Access control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanAccess control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azman
Hafiza Abas
 
Session hijacking
Session hijackingSession hijacking
Session hijacking
Vishal Punjabi
 
Debs 2011 tutorial on non functional properties of event processing
Debs 2011 tutorial  on non functional properties of event processingDebs 2011 tutorial  on non functional properties of event processing
Debs 2011 tutorial on non functional properties of event processing
Opher Etzion
 
Tutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsTutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing Patterns
Opher Etzion
 
Ceh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksCeh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networks
Asep Sopyan
 
Complex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBComplex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESB
Prabath Siriwardena
 
Chapter 12
Chapter 12Chapter 12
Chapter 12
cclay3
 
CyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning Networks
CyberLab
 
Nmap scripting engine
Nmap scripting engineNmap scripting engine
Nmap scripting engine
n|u - The Open Security Community
 
Debs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages TutorialDebs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages Tutorial
Opher Etzion
 
Analizadores de Protocolos
Analizadores de ProtocolosAnalizadores de Protocolos
Analizadores de Protocolos
Milton Muñoz
 

Viewers also liked (20)

Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our Lives
 
Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016
 
The Future of Work
The Future of WorkThe Future of Work
The Future of Work
 
Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016
 
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
 
Intelligent Business Processes
Intelligent Business ProcessesIntelligent Business Processes
Intelligent Business Processes
 
Installing Complex Event Processing On Linux
Installing Complex Event Processing On LinuxInstalling Complex Event Processing On Linux
Installing Complex Event Processing On Linux
 
Reactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingReactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream Processing
 
Comparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsComparative Analysis of Personal Firewalls
Comparative Analysis of Personal Firewalls
 
Access control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanAccess control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azman
 
Session hijacking
Session hijackingSession hijacking
Session hijacking
 
Debs 2011 tutorial on non functional properties of event processing
Debs 2011 tutorial  on non functional properties of event processingDebs 2011 tutorial  on non functional properties of event processing
Debs 2011 tutorial on non functional properties of event processing
 
Tutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsTutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing Patterns
 
Ceh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksCeh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networks
 
Complex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBComplex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESB
 
Chapter 12
Chapter 12Chapter 12
Chapter 12
 
CyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning Networks
 
Nmap scripting engine
Nmap scripting engineNmap scripting engine
Nmap scripting engine
 
Debs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages TutorialDebs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages Tutorial
 
Analizadores de Protocolos
Analizadores de ProtocolosAnalizadores de Protocolos
Analizadores de Protocolos
 

Similar to Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft Private Cloud
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike Freedman
Spark Summit
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex
Apache Apex
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Dataconomy Media
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
Guido Schmutz
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
Databricks
 
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
DataScienceConferenc1
 
SQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsightSQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsight
Eduardo Castro
 
Observability at Spotify
Observability at SpotifyObservability at Spotify
Observability at Spotify
Aleksandr Kuboskin, CFA
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of data
confluent
 
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBBest Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
InfluxData
 
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
confluent
 
WebAction-Sami Abkay
WebAction-Sami AbkayWebAction-Sami Abkay
WebAction-Sami Abkay
Inside Analysis
 
AI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer ExperienceAI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer Experience
Databricks
 
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
DataWorks Summit
 
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel TalkWinter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
Sudhendu Rai
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your Enterprise
WSO2
 
High Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time ControlHigh Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time Control
Marcus Quigley
 
Self-Tuning Data Centers
Self-Tuning Data CentersSelf-Tuning Data Centers
Self-Tuning Data Centers
Reza Rahimi
 

Similar to Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013 (20)

Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike Freedman
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
 
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scale
 
SQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsightSQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsight
 
Observability at Spotify
Observability at SpotifyObservability at Spotify
Observability at Spotify
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of data
 
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBBest Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
 
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
 
WebAction-Sami Abkay
WebAction-Sami AbkayWebAction-Sami Abkay
WebAction-Sami Abkay
 
AI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer ExperienceAI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer Experience
 
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
 
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel TalkWinter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your Enterprise
 
High Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time ControlHigh Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time Control
 
Self-Tuning Data Centers
Self-Tuning Data CentersSelf-Tuning Data Centers
Self-Tuning Data Centers
 

More from StampedeCon

Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
StampedeCon
 
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
StampedeCon
 
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
StampedeCon
 
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
StampedeCon
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
StampedeCon
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
StampedeCon
 
Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017
StampedeCon
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
StampedeCon
 
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
StampedeCon
 
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
StampedeCon
 
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
StampedeCon
 
A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017
StampedeCon
 
Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017
StampedeCon
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
StampedeCon
 
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
StampedeCon
 
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
StampedeCon
 
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
StampedeCon
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016
StampedeCon
 
Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016
StampedeCon
 
Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016
StampedeCon
 

More from StampedeCon (20)

Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
 
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
 
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
 
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
 
Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
 
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
 
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
 
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
 
A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017
 
Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
 
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
 
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
 
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016
 
Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016
 
Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016
 

Recently uploaded

Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
Pravash Chandra Das
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
fredae14
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Jeffrey Haguewood
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
SitimaJohn
 
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStrDeep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
saastr
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
innovationoecd
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
Intelisync
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 

Recently uploaded (20)

Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Operating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptxOperating System Used by Users in day-to-day life.pptx
Operating System Used by Users in day-to-day life.pptx
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
 
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
Letter and Document Automation for Bonterra Impact Management (fka Social Sol...
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxOcean lotus Threat actors project by John Sitima 2024 (1).pptx
Ocean lotus Threat actors project by John Sitima 2024 (1).pptx
 
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStrDeep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
Deep Dive: Getting Funded with Jason Jason Lemkin Founder & CEO @ SaaStr
 
Presentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of GermanyPresentation of the OECD Artificial Intelligence Review of Germany
Presentation of the OECD Artificial Intelligence Review of Germany
 
A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024A Comprehensive Guide to DeFi Development Services in 2024
A Comprehensive Guide to DeFi Development Services in 2024
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 

Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

  • 1. Event Processing for better (Big) Data Vinod Vydier Middleware Specialist @ Oracle
  • 2. Agenda §  Why use event processing? §  Event Processing Applications §  Technical Architecture §  Use of In-Memory data-grid §  Use cases
  • 3. Challenges Working with Big Data • Storing Data has becoming cheap, however the storage is not infinite and has to be managed to make use of the data effectively. • Hadoop has inherent latency for responding to real time events (which can produce high volume data at high velocity) and typically involves real responses. • Event Processing helps in getting clean data with context and less redundancy into HDFS, so the Hadoop jobs can be more effective. • Event Processing helps in responding back in real time, and storing the data in HDFS for better historical analysis.
  • 4. Why use Event Processing Infrastructure Application has any one or more of the following conditions: §  Requires high throughput and low latency processing. §  Has continuously streaming data. §  Real-time correlation between multiple incoming data sources. §  Time-sensitive alerts, aggregations and calculations. §  Needs to look for patterns in the data stream. §  Data does not need to be stored, if there is nothing of interest in it. §  Problem is more easily solved by analyzing before storing in HDFS.
  • 5. Filtering, Real-time Intelligence for Big Data VOLUME VELOCITY VARIETY VALUE SOCIAL BLOG SMART METER 101100101001 001001101010 101011100101 010100100101 FAST DATA Event Processing Intelligence GREATER
  • 6. Stay ahead of Big Data Filter out, correlate Move time-critical analysis to front of process • Filter out noise (example: data ticks with no changes), add context (by correlating multiple sources), increase relevance. • Identify certain critical conditions as you insert data into the warehouse.
  • 7. Getting ahead of the curve: Fast Data Big Data minutesms Fast Data Historicaldepth:deep Historicaldepth:shallow Example: analysis of traffic patterns and congestion times for urban planning Example: monitoring of traffic cameras to ensure given license plates are not in use on multiple vehicles Add “depth” to your fast data by merging output of MapReduce to stream processing
  • 8. Adapter Adapter Processor Adapter HDFS Data Source Queries <<Source>> <<Source>> <<Sink>> Service1 Service2 Export Import Event Processing Network (EPN) Event Processing Application Queries Channel Channel Channel Channel What is an Event Processing application Data Source
  • 9. Event Processing inputs Ø  Streams Ø  Continuous input, often in high- volume Ø  Time ordered Ø  Does not end Ø  Impossible to process / analyze in real-time with traditional relational database systems Example: Raw Sensor Event streams, GPS, Market Data Feeds BA BOEING D 77.575 800 20080305 10:03:02:78 DO DUPOD NT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 Event Processing provides a new data management infrastructure to support and analyze Streams in real-time BA BOEING D 77.575 41.575 800 20080305 10:03:02:78 DO DUPONT D 41.575 3000 20080305 10:03:04:12 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 BA BOEING D 77.575 800 20080305 10:03:02:78
  • 10. Filtering Ø  New stream filtered for specific criteria, e.g. stock price > $22 Ø  Correlation & Aggregation Ø  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour Ø  Pattern Matching Ø  Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 …… • Event Processing done in-Memory (not in Database) • Logic is defined through Continuous Queries on the data CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 BA BOEING D 77.575 41.575 800 20080305 10:03:02:78 DO DUPONT D 41.575 3000 20080305 10:03:04:12 COMPLEX QUERIES Event Processing outputs
  • 11. Data crunching for Event Processing done in a in-memory data grid •  High throughput for storing data •  Aggregation and event querying •  Pattern implementation flexibility combining complementary technologies •  Handle and correlate events in real time, including support for multiple patterns: •  Pre processing (buffer inputs) •  In Event Processing (to cache reference data) •  Post Processing (to expose processed events to consuming apps) Data Grid Event Processing Consolidat ed & in- context Data Filtered/ Aggregat ed Data HDFS and traditional storage
  • 12. In-memory events on the data stream n  Threshold Management n  Detecting threshold conditions across multiple event streams n  Using cache to: n  Allow dynamic configuration of thresholds n  Add (via join) contextual data to support aggregation n  Using pattern matching to find sustained conditions n  Alert Generation n  Using relations to represent state and state transitions n  Using “missing event” patterns to monitor expected response(s) n  Alarm Management n  Using pattern matching to remove extraneous alarm events n  e.g. power off alarm preceded by tamper alarm within (n) minutes X
  • 13. Alarm Filtering Scenario Discard Power Off Alarm if there was a Tamper Alarm for the same meter within the previous 5 seconds
  • 14. Visualizing events on the data stream JMS Resource Locations Matches and Alerts SQL Event Processing Application JMS Geo-Fencing Definitions SQL MapViewer Manager
  • 15. JMS Protocol Integration n Common integration touch point with Service Bus n Business Activity Monitoring integration HTTP Publish/Subscribe n Support pub/sub events between Event server and web clients. n Clients don’t need to poll for updates (unlike traditional HTTP). n Clients subscribe to and publish to event channels. n Bayeux protocol n Light weight and the payload is JSON Visual/SOA integration with Event Processing
  • 16. Event Processing High Level Architecture JSON Adapter CacheProcessor POJO EPN (Event Processing Network) Elements HTTP Pub/S
  • 17. Query Plan and Real Time Monitoring
  • 18. Event Driven SOA: Simplify Business Complexity •  Real-time business insight •  Preempt and react instantaneously to Enterprise, Environmental and Global Business conditions •  Gain business insight using previously untapped, raw event sources •  Hot-pluggable integration •  Transparent SOA infrastructure interoperability •  Distributed, deployment ready, pre-integrated, in-memory Data Grid, and Java low latency determinism. •  Lightweight high performance Java Event Server platform •  Real-time business friendly analyst oriented visualization layers •  Powerful, extensible Event Processing Analysis abstraction •  Business user dashboards •  Business user domain specific natural language layers •  Real-time predictive analytics
  • 19. Event Processing use cases in different industries 1.  Customer Experience 2.  Transportation, Logistics & Fleet Management 3.  Utilities: Demand & Response, Smart Meter 4.  Public Sector: Emergency Response, Intelligence 5.  Telcos: Real Time billing & WiFi offloading, Mobile billboard
  • 20. Customer Experience n  Industry focus on new buzzword: Customer Experience n  Desire to harness potential of social networks for better targeted marketing Event Processing can help with: n  Monitoring in real-time customer activity (social networks, location (e.g. proximity to stores, etc) and identifying opportunities in real-time n  Correlating with existing information (customer/ shopping profiles, etc.) n  Generating real-time alerts
  • 21. Transportation, Logistics and Fleet Management n  Constant industry pressure for greater efficiency n  Need to differentiate through premium services and greater reliability and visibility n  Availability of cheap wireless sensors (temperature, GPS, etc.) that can be included in packages/containers/trucks Event Processing can help with: n  Real-time monitoring of inflow of data from sensors n  Trends detection / prediction (to rise, etc.) n  Leveraging spatial/geo-location capabilities.
  • 22. Utilities n  Adoption of Smart Meters: concerns about bandwidth/ processing power required to handle the information they generate, desire to offer value-add services n  Ever increasing electricity demand n  Demand for real-time billing & analytics n  Greater customer expectations re: outage & response times n  Regulations Event Processing can help with: n  Alerting of consumption trends in real-time, enabling “Demand/ Response” n  Real-time detection of problems (abnormal spikes in consumption indicative of leaks, etc.) n  Filtering out redundant or nested (ex: tree fell on the line) outage errors and problems n  Tracking of resources and personnel
  • 23. Telco n  Overloaded data networks and new strategies to offload traffic: real-time billing vs. unlimited, offloading to WiFi, degradation of service from 4G to 3G, etc. n  GPS-enabled phones offer new location-based marketing opportunities: “mobile billboards” How can Event Processing help: n  Event Processing infrastructure can handle massive amounts of data generated by mobile devices, filter out, correlate and aggregate in real-time to only retain valuable information n  Event Processing can plug into all types of feeds, from devices to social networks n  Event Processing can be integrated with spatial and geo- location technology to send location specific data to the user.
  • 24. Public Sector n  Heightened security requirements n  Ever increasing population in urban areas drives optimization requirements n  Increasing number of real-time data: video feeds, GPS data, traffic data, etc. n  Applications: Security Intelligence, geo-fencing, “Smart Cities”, traffic control, gateless tolls How Event Processing can help: n  Event Processing can be integrated with spatial and geo- location technology to track location specific data with a user. n  Event Processing can plug in any data feed such as video / face recognition n  Event Processing meets performance & availability requirements in this space