SlideShare a Scribd company logo

Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

At the StampedeCon 2013 Big Data conference in St. Louis, Vinod Vydier, Middleware Specialist at Oracle, discussed Real Time Event Processing and In-­memory analysis of Big Data. There are multiple projects (for example Cloudera’s Impala) that do real­time or near real time analysis of Big Data. However, if there are events that need to be looked at and responded to in real time (for example credit card fraud or a vehicle metrics to alert a driver) this can have a significant impact on data collection and analysis using the traditional Big Data techniques. In this session, I will introduce strategies on how you can use an Event Processing Engine to respond to events in real time, and then filter and categorize data in memory before storing data in HDFS. This will make it easier to run Hadoop jobs on the data collected, and also have end clients respond to the critical events in real time.

Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

1 of 25
Event Processing for better
(Big) Data
Vinod Vydier
Middleware Specialist @ Oracle
Agenda
§  Why use event processing?
§  Event Processing Applications
§  Technical Architecture
§  Use of In-Memory data-grid
§  Use cases
Challenges Working with Big Data
• Storing Data has becoming cheap, however the
storage is not infinite and has to be managed to make
use of the data effectively.
• Hadoop has inherent latency for responding to real
time events (which can produce high volume data at
high velocity) and typically involves real responses.
• Event Processing helps in getting clean data with
context and less redundancy into HDFS, so the Hadoop
jobs can be more effective.
• Event Processing helps in responding back in real
time, and storing the data in HDFS for better historical
analysis.
Why use Event Processing Infrastructure
Application has any one or more of the following
conditions:
§  Requires high throughput and low latency
processing.
§  Has continuously streaming data.
§  Real-time correlation between multiple incoming data
sources.
§  Time-sensitive alerts, aggregations and calculations.
§  Needs to look for patterns in the data stream.
§  Data does not need to be stored, if there is nothing
of interest in it.
§  Problem is more easily solved by analyzing before
storing in HDFS.
Filtering, Real-time Intelligence for Big Data
VOLUME VELOCITY VARIETY VALUE
SOCIAL
BLOG
SMART
METER
101100101001
001001101010
101011100101
010100100101
FAST DATA
Event Processing Intelligence
GREATER
Stay ahead of Big Data
Filter out,
correlate
Move time-critical analysis to front of process
• Filter out noise (example: data ticks with no
changes), add context (by correlating multiple
sources), increase relevance.
• Identify certain critical conditions as you insert data
into the warehouse.
Ad

Recommended

Big Data, Analytics and Real Time Event Processing
Big Data, Analytics and Real Time Event Processing Big Data, Analytics and Real Time Event Processing
Big Data, Analytics and Real Time Event Processing WSO2
 
WSO2 Big Data Analytics Platform
WSO2 Big Data Analytics PlatformWSO2 Big Data Analytics Platform
WSO2 Big Data Analytics PlatformSamisa Abeysinghe
 
TripleLift: Preparing for a New Programmatic Ad-Tech World
TripleLift: Preparing for a New Programmatic Ad-Tech WorldTripleLift: Preparing for a New Programmatic Ad-Tech World
TripleLift: Preparing for a New Programmatic Ad-Tech WorldVoltDB
 
VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big Data
VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big DataVoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big Data
VoltDB and HPE Vertica Present: Building an IoT Architecture for Fast + Big DataVoltDB
 
ironSource Atom BigData Berlin
ironSource Atom BigData BerlinironSource Atom BigData Berlin
ironSource Atom BigData BerlinShimon Tolts
 
Building the Next-gen Digital Meter Platform for Fluvius
Building the Next-gen Digital Meter Platform for FluviusBuilding the Next-gen Digital Meter Platform for Fluvius
Building the Next-gen Digital Meter Platform for FluviusDatabricks
 
ChakraView – A 360° Approach to Data Quality
ChakraView – A 360° Approach to Data QualityChakraView – A 360° Approach to Data Quality
ChakraView – A 360° Approach to Data QualityDatabricks
 
Billions of Rows, Millions of Insights, Right Now
Billions of Rows, Millions of Insights, Right NowBillions of Rows, Millions of Insights, Right Now
Billions of Rows, Millions of Insights, Right NowRob Winters
 

More Related Content

What's hot

Five ways database modernization simplifies your data life
Five ways database modernization simplifies your data lifeFive ways database modernization simplifies your data life
Five ways database modernization simplifies your data lifeSingleStore
 
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler AnswersLambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers VoltDB
 
HP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big DataHP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big DataRob Winters
 
DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)DataStax
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017SingleStore
 
Transforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case ExamplesTransforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case ExamplesVoltDB
 
Our journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scaleOur journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scaleItai Yaffe
 
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 Migration and Coexistence between Relational and NoSQL Databases by Manuel H... Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...Big Data Spain
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraBig Data Spain
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplowyalisassoon
 
Data Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and GasData Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and GasDenodo
 
How to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top ContendersHow to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top ContendersVoltDB
 
Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Big Data Spain
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationVoltDB
 
Netflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering MeetupNetflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering MeetupBlake Irvine
 
Converging Database Transactions and Analytics
Converging Database Transactions and Analytics Converging Database Transactions and Analytics
Converging Database Transactions and Analytics SingleStore
 
Building the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free LifeBuilding the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free LifeSingleStore
 
Architecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data AnalyticsArchitecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data AnalyticsRob Winters
 
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo
 

What's hot (20)

Five ways database modernization simplifies your data life
Five ways database modernization simplifies your data lifeFive ways database modernization simplifies your data life
Five ways database modernization simplifies your data life
 
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler AnswersLambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
Lambda-B-Gone: In-memory Case Study for Faster, Smarter and Simpler Answers
 
HP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big DataHP Discover: Real Time Insights from Big Data
HP Discover: Real Time Insights from Big Data
 
DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)DataStax Enterprise in Practice (Field Notes)
DataStax Enterprise in Practice (Field Notes)
 
Intuit Analytics Cloud 101
Intuit Analytics Cloud 101Intuit Analytics Cloud 101
Intuit Analytics Cloud 101
 
In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017In-Memory Computing Webcast. Market Predictions 2017
In-Memory Computing Webcast. Market Predictions 2017
 
Transforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case ExamplesTransforming Your Business with Fast Data – Five Use Case Examples
Transforming Your Business with Fast Data – Five Use Case Examples
 
Our journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scaleOur journey with druid - from initial research to full production scale
Our journey with druid - from initial research to full production scale
 
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 Migration and Coexistence between Relational and NoSQL Databases by Manuel H... Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
Migration and Coexistence between Relational and NoSQL Databases by Manuel H...
 
VP of WW Partners by Alan Chhabra
VP of WW Partners by Alan ChhabraVP of WW Partners by Alan Chhabra
VP of WW Partners by Alan Chhabra
 
Big data meetup budapest adding data schemas to snowplow
Big data meetup budapest   adding data schemas to snowplowBig data meetup budapest   adding data schemas to snowplow
Big data meetup budapest adding data schemas to snowplow
 
Data Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and GasData Strategies for Managing the Cycles in Oil and Gas
Data Strategies for Managing the Cycles in Oil and Gas
 
How to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top ContendersHow to Build Fast Data Applications: Evaluating the Top Contenders
How to Build Fast Data Applications: Evaluating the Top Contenders
 
Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...Advanced data science algorithms applied to scalable stream processing by Dav...
Advanced data science algorithms applied to scalable stream processing by Dav...
 
Memory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business InnovationMemory Database Technology is Driving a New Cycle of Business Innovation
Memory Database Technology is Driving a New Cycle of Business Innovation
 
Netflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering MeetupNetflix Data Engineering @ Uber Engineering Meetup
Netflix Data Engineering @ Uber Engineering Meetup
 
Converging Database Transactions and Analytics
Converging Database Transactions and Analytics Converging Database Transactions and Analytics
Converging Database Transactions and Analytics
 
Building the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free LifeBuilding the Foundation for a Latency-Free Life
Building the Foundation for a Latency-Free Life
 
Architecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data AnalyticsArchitecting for Real-Time Big Data Analytics
Architecting for Real-Time Big Data Analytics
 
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time ResponsesDenodo DataFest 2017: Outpace Your Competition with Real-Time Responses
Denodo DataFest 2017: Outpace Your Competition with Real-Time Responses
 

Viewers also liked

Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesRukshan Batuwita
 
Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016Mark Proctor
 
Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016Mark Proctor
 
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...Mia Yuan Cao
 
Intelligent Business Processes
Intelligent Business ProcessesIntelligent Business Processes
Intelligent Business ProcessesSandy Kemsley
 
Installing Complex Event Processing On Linux
Installing Complex Event Processing On LinuxInstalling Complex Event Processing On Linux
Installing Complex Event Processing On LinuxOsama Mustafa
 
Reactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingReactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingAndy Piper
 
Comparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsComparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsAndrej Šimko
 
Access control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanAccess control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanHafiza Abas
 
Debs 2011 tutorial on non functional properties of event processing
Debs 2011 tutorial  on non functional properties of event processingDebs 2011 tutorial  on non functional properties of event processing
Debs 2011 tutorial on non functional properties of event processingOpher Etzion
 
Tutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsTutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsOpher Etzion
 
Ceh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksCeh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksAsep Sopyan
 
Complex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBComplex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBPrabath Siriwardena
 
Chapter 12
Chapter 12Chapter 12
Chapter 12cclay3
 
CyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab
 
Debs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages TutorialDebs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages TutorialOpher Etzion
 
Analizadores de Protocolos
Analizadores de ProtocolosAnalizadores de Protocolos
Analizadores de ProtocolosMilton Muñoz
 

Viewers also liked (20)

Big Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our LivesBig Data and Data Science: The Technologies Shaping Our Lives
Big Data and Data Science: The Technologies Shaping Our Lives
 
Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016Learning Rule Based Programming using Games @DecisionCamp 2016
Learning Rule Based Programming using Games @DecisionCamp 2016
 
The Future of Work
The Future of WorkThe Future of Work
The Future of Work
 
Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016Drools Happenings 7.0 - Devnation 2016
Drools Happenings 7.0 - Devnation 2016
 
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
Real-Time Analytics and Visualization of Streaming Big Data with JReport & Sc...
 
Intelligent Business Processes
Intelligent Business ProcessesIntelligent Business Processes
Intelligent Business Processes
 
Installing Complex Event Processing On Linux
Installing Complex Event Processing On LinuxInstalling Complex Event Processing On Linux
Installing Complex Event Processing On Linux
 
Reactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream ProcessingReactconf 2014 - Event Stream Processing
Reactconf 2014 - Event Stream Processing
 
Comparative Analysis of Personal Firewalls
Comparative Analysis of Personal FirewallsComparative Analysis of Personal Firewalls
Comparative Analysis of Personal Firewalls
 
Access control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azmanAccess control attacks by nor liyana binti azman
Access control attacks by nor liyana binti azman
 
Session hijacking
Session hijackingSession hijacking
Session hijacking
 
Debs 2011 tutorial on non functional properties of event processing
Debs 2011 tutorial  on non functional properties of event processingDebs 2011 tutorial  on non functional properties of event processing
Debs 2011 tutorial on non functional properties of event processing
 
Tutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing PatternsTutorial in DEBS 2008 - Event Processing Patterns
Tutorial in DEBS 2008 - Event Processing Patterns
 
Ceh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networksCeh v8 labs module 03 scanning networks
Ceh v8 labs module 03 scanning networks
 
Complex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESBComplex Event Processing with Esper and WSO2 ESB
Complex Event Processing with Esper and WSO2 ESB
 
Chapter 12
Chapter 12Chapter 12
Chapter 12
 
CyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning NetworksCyberLab CCEH Session - 3 Scanning Networks
CyberLab CCEH Session - 3 Scanning Networks
 
Nmap scripting engine
Nmap scripting engineNmap scripting engine
Nmap scripting engine
 
Debs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages TutorialDebs2009 Event Processing Languages Tutorial
Debs2009 Event Processing Languages Tutorial
 
Analizadores de Protocolos
Analizadores de ProtocolosAnalizadores de Protocolos
Analizadores de Protocolos
 

Similar to Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft Private Cloud
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Summit
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex Apache Apex
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...Dataconomy Media
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream ProcessingGuido Schmutz
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastDatabricks
 
SQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsightSQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsightEduardo Castro
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of dataconfluent
 
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBBest Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBInfluxData
 
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...Flink Forward
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Servicesconfluent
 
AI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer ExperienceAI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer ExperienceDatabricks
 
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...DataWorks Summit
 
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel TalkWinter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel TalkSudhendu Rai
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your EnterpriseWSO2
 
High Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time ControlHigh Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time ControlMarcus Quigley
 
Self-Tuning Data Centers
Self-Tuning Data CentersSelf-Tuning Data Centers
Self-Tuning Data CentersReza Rahimi
 
Let's get to know the Data Streaming
Let's get to know the Data StreamingLet's get to know the Data Streaming
Let's get to know the Data StreamingKnoldus Inc.
 

Similar to Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013 (20)

Microsoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview PresentationMicrosoft SQL Server - StreamInsight Overview Presentation
Microsoft SQL Server - StreamInsight Overview Presentation
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike Freedman
 
Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex Big Data Berlin v8.0 Stream Processing with Apache Apex
Big Data Berlin v8.0 Stream Processing with Apache Apex
 
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
Thomas Weise, Apache Apex PMC Member and Architect/Co-Founder, DataTorrent - ...
 
Introduction to Stream Processing
Introduction to Stream ProcessingIntroduction to Stream Processing
Introduction to Stream Processing
 
Cloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and FastCloud Experience: Data-driven Applications Made Simple and Fast
Cloud Experience: Data-driven Applications Made Simple and Fast
 
SQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsightSQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsight
 
Observability at Spotify
Observability at SpotifyObservability at Spotify
Observability at Spotify
 
Real-time processing of large amounts of data
Real-time processing of large amounts of dataReal-time processing of large amounts of data
Real-time processing of large amounts of data
 
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBBest Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
 
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...Flink Forward San Francisco 2018:  David Reniz & Dahyr Vergara - "Real-time m...
Flink Forward San Francisco 2018: David Reniz & Dahyr Vergara - "Real-time m...
 
Apache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial ServicesApache Kafka® Use Cases for Financial Services
Apache Kafka® Use Cases for Financial Services
 
WebAction-Sami Abkay
WebAction-Sami AbkayWebAction-Sami Abkay
WebAction-Sami Abkay
 
AI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer ExperienceAI-Powered Streaming Analytics for Real-Time Customer Experience
AI-Powered Streaming Analytics for Real-Time Customer Experience
 
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
Disaster Recovery Experience at CACIB: Hardening Hadoop for Critical Financia...
 
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel TalkWinter Simulation Conference 2021 - Process Wind Tunnel Talk
Winter Simulation Conference 2021 - Process Wind Tunnel Talk
 
Analytics in Your Enterprise
Analytics in Your EnterpriseAnalytics in Your Enterprise
Analytics in Your Enterprise
 
High Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time ControlHigh Performance Green Infrastructure, New Directions in Real-Time Control
High Performance Green Infrastructure, New Directions in Real-Time Control
 
Self-Tuning Data Centers
Self-Tuning Data CentersSelf-Tuning Data Centers
Self-Tuning Data Centers
 
Let's get to know the Data Streaming
Let's get to know the Data StreamingLet's get to know the Data Streaming
Let's get to know the Data Streaming
 

More from StampedeCon

Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...StampedeCon
 
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017StampedeCon
 
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017StampedeCon
 
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...StampedeCon
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017StampedeCon
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017StampedeCon
 
Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017StampedeCon
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...StampedeCon
 
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...StampedeCon
 
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017StampedeCon
 
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017StampedeCon
 
A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017StampedeCon
 
Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017StampedeCon
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017StampedeCon
 
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017StampedeCon
 
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...StampedeCon
 
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...StampedeCon
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016StampedeCon
 
Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016StampedeCon
 
Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016StampedeCon
 

More from StampedeCon (20)

Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
Why Should We Trust You-Interpretability of Deep Neural Networks - StampedeCo...
 
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
The Search for a New Visual Search Beyond Language - StampedeCon AI Summit 2017
 
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
Predicting Outcomes When Your Outcomes are Graphs - StampedeCon AI Summit 2017
 
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
Novel Semi-supervised Probabilistic ML Approach to SNP Variant Calling - Stam...
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
 
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
Getting Started with Keras and TensorFlow - StampedeCon AI Summit 2017
 
Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017Foundations of Machine Learning - StampedeCon AI Summit 2017
Foundations of Machine Learning - StampedeCon AI Summit 2017
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
 
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
Bringing the Whole Elephant Into View Can Cognitive Systems Bring Real Soluti...
 
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
Automated AI The Next Frontier in Analytics - StampedeCon AI Summit 2017
 
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017AI in the Enterprise: Past,  Present &  Future - StampedeCon AI Summit 2017
AI in the Enterprise: Past, Present & Future - StampedeCon AI Summit 2017
 
A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017A Different Data Science Approach - StampedeCon AI Summit 2017
A Different Data Science Approach - StampedeCon AI Summit 2017
 
Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017Graph in Customer 360 - StampedeCon Big Data Conference 2017
Graph in Customer 360 - StampedeCon Big Data Conference 2017
 
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
End-to-end Big Data Projects with Python - StampedeCon Big Data Conference 2017
 
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
Doing Big Data Using Amazon's Analogs - StampedeCon Big Data Conference 2017
 
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
Enabling New Business Capabilities with Cloud-based Streaming Data Architectu...
 
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
Big Data Meets IoT: Lessons From the Cloud on Polling, Collecting, and Analyz...
 
Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016Innovation in the Data Warehouse - StampedeCon 2016
Innovation in the Data Warehouse - StampedeCon 2016
 
Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016Creating a Data Driven Organization - StampedeCon 2016
Creating a Data Driven Organization - StampedeCon 2016
 
Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016Using The Internet of Things for Population Health Management - StampedeCon 2016
Using The Internet of Things for Population Health Management - StampedeCon 2016
 

Recently uploaded

Navigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersNavigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersOnePlan Solutions
 
Tete thermostatique Zigbee MOES BRT-100 V2.pdf
Tete thermostatique Zigbee MOES BRT-100 V2.pdfTete thermostatique Zigbee MOES BRT-100 V2.pdf
Tete thermostatique Zigbee MOES BRT-100 V2.pdfDomotica daVinci
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Daniel Toomey
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingshrey Ansh
 
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-CManual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-CDomotica daVinci
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringMassimo Talia
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERNRonnelBaroc
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfkeyaramicrochipusa
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfIsidro Navarro
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellencePrecisely
 
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...Adrian Sanabria
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIMemory Fabric Forum
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveDomotica daVinci
 
Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Mateusz Kwasniewski
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?GleecusTechlabs1
 

Recently uploaded (20)

Navigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio LeadersNavigating the Never Normal Strategies for Portfolio Leaders
Navigating the Never Normal Strategies for Portfolio Leaders
 
Tete thermostatique Zigbee MOES BRT-100 V2.pdf
Tete thermostatique Zigbee MOES BRT-100 V2.pdfTete thermostatique Zigbee MOES BRT-100 V2.pdf
Tete thermostatique Zigbee MOES BRT-100 V2.pdf
 
Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024Microsoft Azure News - Feb 2024
Microsoft Azure News - Feb 2024
 
5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion5 Tech Trend to Notice in ESG Landscape- 47Billion
5 Tech Trend to Notice in ESG Landscape- 47Billion
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
Bluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons workingBluetooth Low Energy(BLE) and beacons working
Bluetooth Low Energy(BLE) and beacons working
 
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-CManual  sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
Manual sensor Zigbee 3.0 MOES ZSS-X-PIRL-C
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineering
 
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
21ST CENTURY LITERACY FROM TRADITIONAL TO MODERN
 
Semiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdfSemiconductor Review Magazine Feature.pdf
Semiconductor Review Magazine Feature.pdf
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptx
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdf
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center Excellence
 
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...
Avoiding Bad Stats and the Benefits of Playing Trivia with Friends: PancakesC...
 
Q1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AIQ1 Memory Fabric Forum: XConn CXL Switches for AI
Q1 Memory Fabric Forum: XConn CXL Switches for AI
 
COE AI Lab Universities
COE AI Lab UniversitiesCOE AI Lab Universities
COE AI Lab Universities
 
Manual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-WaveManual Eurotronic Thermostatic Valve Comry Z-Wave
Manual Eurotronic Thermostatic Valve Comry Z-Wave
 
Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.Put a flag on it. A busy developer's guide to feature toggles.
Put a flag on it. A busy developer's guide to feature toggles.
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
 

Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon 2013

  • 1. Event Processing for better (Big) Data Vinod Vydier Middleware Specialist @ Oracle
  • 2. Agenda §  Why use event processing? §  Event Processing Applications §  Technical Architecture §  Use of In-Memory data-grid §  Use cases
  • 3. Challenges Working with Big Data • Storing Data has becoming cheap, however the storage is not infinite and has to be managed to make use of the data effectively. • Hadoop has inherent latency for responding to real time events (which can produce high volume data at high velocity) and typically involves real responses. • Event Processing helps in getting clean data with context and less redundancy into HDFS, so the Hadoop jobs can be more effective. • Event Processing helps in responding back in real time, and storing the data in HDFS for better historical analysis.
  • 4. Why use Event Processing Infrastructure Application has any one or more of the following conditions: §  Requires high throughput and low latency processing. §  Has continuously streaming data. §  Real-time correlation between multiple incoming data sources. §  Time-sensitive alerts, aggregations and calculations. §  Needs to look for patterns in the data stream. §  Data does not need to be stored, if there is nothing of interest in it. §  Problem is more easily solved by analyzing before storing in HDFS.
  • 5. Filtering, Real-time Intelligence for Big Data VOLUME VELOCITY VARIETY VALUE SOCIAL BLOG SMART METER 101100101001 001001101010 101011100101 010100100101 FAST DATA Event Processing Intelligence GREATER
  • 6. Stay ahead of Big Data Filter out, correlate Move time-critical analysis to front of process • Filter out noise (example: data ticks with no changes), add context (by correlating multiple sources), increase relevance. • Identify certain critical conditions as you insert data into the warehouse.
  • 7. Getting ahead of the curve: Fast Data Big Data minutesms Fast Data Historicaldepth:deep Historicaldepth:shallow Example: analysis of traffic patterns and congestion times for urban planning Example: monitoring of traffic cameras to ensure given license plates are not in use on multiple vehicles Add “depth” to your fast data by merging output of MapReduce to stream processing
  • 8. Adapter Adapter Processor Adapter HDFS Data Source Queries <<Source>> <<Source>> <<Sink>> Service1 Service2 Export Import Event Processing Network (EPN) Event Processing Application Queries Channel Channel Channel Channel What is an Event Processing application Data Source
  • 9. Event Processing inputs Ø  Streams Ø  Continuous input, often in high- volume Ø  Time ordered Ø  Does not end Ø  Impossible to process / analyze in real-time with traditional relational database systems Example: Raw Sensor Event streams, GPS, Market Data Feeds BA BOEING D 77.575 800 20080305 10:03:02:78 DO DUPOD NT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 Event Processing provides a new data management infrastructure to support and analyze Streams in real-time BA BOEING D 77.575 41.575 800 20080305 10:03:02:78 DO DUPONT D 41.575 3000 20080305 10:03:04:12 BA BOEING D 77.575 800 20080305 10:03:02:78 C CITIGROUP D 34.125 2000 20080305 10:03:03:05 BA BOEING D 77.575 800 20080305 10:03:02:78
  • 10. Filtering Ø  New stream filtered for specific criteria, e.g. stock price > $22 Ø  Correlation & Aggregation Ø  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour Ø  Pattern Matching Ø  Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 …… • Event Processing done in-Memory (not in Database) • Logic is defined through Continuous Queries on the data CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46 DO DUPONT D 41.575 3000 20080305 10:03:04:12 AA ALCOA INC D 20.125 1000 20080305 10:03:01:55 AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10 BA BOEING D 77.575 800 20080305 10:03:02:78 BA BOEING D 77.575 41.575 800 20080305 10:03:02:78 DO DUPONT D 41.575 3000 20080305 10:03:04:12 COMPLEX QUERIES Event Processing outputs
  • 11. Data crunching for Event Processing done in a in-memory data grid •  High throughput for storing data •  Aggregation and event querying •  Pattern implementation flexibility combining complementary technologies •  Handle and correlate events in real time, including support for multiple patterns: •  Pre processing (buffer inputs) •  In Event Processing (to cache reference data) •  Post Processing (to expose processed events to consuming apps) Data Grid Event Processing Consolidat ed & in- context Data Filtered/ Aggregat ed Data HDFS and traditional storage
  • 12. In-memory events on the data stream n  Threshold Management n  Detecting threshold conditions across multiple event streams n  Using cache to: n  Allow dynamic configuration of thresholds n  Add (via join) contextual data to support aggregation n  Using pattern matching to find sustained conditions n  Alert Generation n  Using relations to represent state and state transitions n  Using “missing event” patterns to monitor expected response(s) n  Alarm Management n  Using pattern matching to remove extraneous alarm events n  e.g. power off alarm preceded by tamper alarm within (n) minutes X
  • 13. Alarm Filtering Scenario Discard Power Off Alarm if there was a Tamper Alarm for the same meter within the previous 5 seconds
  • 14. Visualizing events on the data stream JMS Resource Locations Matches and Alerts SQL Event Processing Application JMS Geo-Fencing Definitions SQL MapViewer Manager
  • 15. JMS Protocol Integration n Common integration touch point with Service Bus n Business Activity Monitoring integration HTTP Publish/Subscribe n Support pub/sub events between Event server and web clients. n Clients don’t need to poll for updates (unlike traditional HTTP). n Clients subscribe to and publish to event channels. n Bayeux protocol n Light weight and the payload is JSON Visual/SOA integration with Event Processing
  • 16. Event Processing High Level Architecture JSON Adapter CacheProcessor POJO EPN (Event Processing Network) Elements HTTP Pub/S
  • 17. Query Plan and Real Time Monitoring
  • 18. Event Driven SOA: Simplify Business Complexity •  Real-time business insight •  Preempt and react instantaneously to Enterprise, Environmental and Global Business conditions •  Gain business insight using previously untapped, raw event sources •  Hot-pluggable integration •  Transparent SOA infrastructure interoperability •  Distributed, deployment ready, pre-integrated, in-memory Data Grid, and Java low latency determinism. •  Lightweight high performance Java Event Server platform •  Real-time business friendly analyst oriented visualization layers •  Powerful, extensible Event Processing Analysis abstraction •  Business user dashboards •  Business user domain specific natural language layers •  Real-time predictive analytics
  • 19. Event Processing use cases in different industries 1.  Customer Experience 2.  Transportation, Logistics & Fleet Management 3.  Utilities: Demand & Response, Smart Meter 4.  Public Sector: Emergency Response, Intelligence 5.  Telcos: Real Time billing & WiFi offloading, Mobile billboard
  • 20. Customer Experience n  Industry focus on new buzzword: Customer Experience n  Desire to harness potential of social networks for better targeted marketing Event Processing can help with: n  Monitoring in real-time customer activity (social networks, location (e.g. proximity to stores, etc) and identifying opportunities in real-time n  Correlating with existing information (customer/ shopping profiles, etc.) n  Generating real-time alerts
  • 21. Transportation, Logistics and Fleet Management n  Constant industry pressure for greater efficiency n  Need to differentiate through premium services and greater reliability and visibility n  Availability of cheap wireless sensors (temperature, GPS, etc.) that can be included in packages/containers/trucks Event Processing can help with: n  Real-time monitoring of inflow of data from sensors n  Trends detection / prediction (to rise, etc.) n  Leveraging spatial/geo-location capabilities.
  • 22. Utilities n  Adoption of Smart Meters: concerns about bandwidth/ processing power required to handle the information they generate, desire to offer value-add services n  Ever increasing electricity demand n  Demand for real-time billing & analytics n  Greater customer expectations re: outage & response times n  Regulations Event Processing can help with: n  Alerting of consumption trends in real-time, enabling “Demand/ Response” n  Real-time detection of problems (abnormal spikes in consumption indicative of leaks, etc.) n  Filtering out redundant or nested (ex: tree fell on the line) outage errors and problems n  Tracking of resources and personnel
  • 23. Telco n  Overloaded data networks and new strategies to offload traffic: real-time billing vs. unlimited, offloading to WiFi, degradation of service from 4G to 3G, etc. n  GPS-enabled phones offer new location-based marketing opportunities: “mobile billboards” How can Event Processing help: n  Event Processing infrastructure can handle massive amounts of data generated by mobile devices, filter out, correlate and aggregate in real-time to only retain valuable information n  Event Processing can plug into all types of feeds, from devices to social networks n  Event Processing can be integrated with spatial and geo- location technology to send location specific data to the user.
  • 24. Public Sector n  Heightened security requirements n  Ever increasing population in urban areas drives optimization requirements n  Increasing number of real-time data: video feeds, GPS data, traffic data, etc. n  Applications: Security Intelligence, geo-fencing, “Smart Cities”, traffic control, gateless tolls How Event Processing can help: n  Event Processing can be integrated with spatial and geo- location technology to track location specific data with a user. n  Event Processing can plug in any data feed such as video / face recognition n  Event Processing meets performance & availability requirements in this space