Trivadis, profile picture
Uploaded byTrivadis
1,066 views

Processing Twitter Stream with Oracle Event Processing (OEP)

The document discusses the processing of Twitter streams using Oracle Event Processing (OEP), outlining the agenda that includes use cases, processing methods, and visualization techniques. It emphasizes the significance of big data, the evolution of data generation and consumption, and the importance of using efficient storage technologies like Apache Cassandra. Additionally, it covers the components of an event processing system and the benefits of visual tools for managing and analyzing data.

Embed presentation

Downloaded 29 times
2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) Guido Schmutz OFM Partner Forum Malta 19.2.2014 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 1
2013 © Trivadis Guido Schmutz •  Working for Trivadis for more than 17 years •  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis •  More than 25 years of software development experience •  Contact: guido.schmutz@trivadis.com •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 2
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 3
2013 © Trivadis Big Data Definition (4 Vs) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4 + Time to action ? – Big Data + Event Processing = Fast Data Characteristics of Big Data: Its Volume, Velocity and Variety in combination
2013 © Trivadis The world is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 5
2013 © Trivadis Who is generating Big Data? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 6 The progress and innovation is no longer hindered by the ability to collect data But by the ability to manage, analyze, summarize, visualize and discover knowledge from the collected data in a timely manner and in a scalable fashion Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Mobile devices (tracking all objects all the time) Sensor technology and networks (measuring all kinds of data)
2013 © Trivadis 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 7
2013 © Trivadis Internet Of Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 8 Source: CiscoSource: The Economist
2013 © Trivadis Data as an Asset - Store Anything? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 9 But then data is
 just too valuable
 to delete!
 We must 
 store anything! Nonsense! Just 
 store the data 
 you know 
 you need today! It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today!
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 10
2013 © Trivadis Retrieve Tweets and Visualize 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 11
2013 © Trivadis Access to Tweets 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 12 Quelle Source Limitations Cost Twitter’s Search API 3200 / user 5000 / keyword 180 requests / 15 minutes free Twitter’s Streaming API 1%-40% of total volume free DataSift none 0.15 -0.20$ / unit Gnip none On request
2013 © Trivadis How to design a stream (event) processing system? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 13 Twitter Stream tweet Sensor tweet Persist (Queue) Twitter Stream tweet Sensor tweet Processing tweet Processing result result Twitter Stream tweet Receiving/ Processing result
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 14
2013 © Trivadis Oracle Event Processing (OEP) - Engine Lightweight Java Application Server •  Full environment for running Java applications •  Module Framework - OSGi High Throughput •  Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment •  Service Framework – Spring DM, POJO Enterprise Web 2.0 & Eclipse-based tooling Multiple-choice VM •  JRockit or WebLogic RealTime 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 15
2013 © Trivadis Oracle Event Processing – Event Processing Network Concept 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 16
2013 © Trivadis Oracle Event Processing – In Memory, Continuous Queries Event Processing Output §  Filtering -  New stream filtered for specific criteria, e.g. stock price > $22 §  Correlation & Aggregation -  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour §  Pattern Matching -  Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 17
2013 © Trivadis Oracle Event Processing - CQL Initiative for a complete “continuous” query language Start with SQL ’99 plus “continuous” query extensions §  Based on Stanford University research Industry standards discussions §  Event Processing Technical Society (EPTS) §  ANSI SQL §  OMG Adoption Today §  ANSI SQL Standards Proposal for CQL Pattern Matching -  Oracle, IBM, Stanford University §  OpenSource Adoption of CQL §  Oracle Complex Event Processor (CEP) Releaseà Available in 11g 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 18
2013 © Trivadis Oracle Event Processing – Visual Development Tools 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 19
2013 © Trivadis Oracle Event Processing – Operation & Management 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 20
2013 © Trivadis Implementation – complete picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 21 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds
2013 © Trivadis 1) Creating a Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 22 Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter
2013 © Trivadis 2) Send Tweets to BAM 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 23 Twitter Adapter BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta JMS Twitter
2013 © Trivadis 3) Extract interesting information from Tweet 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 24 Mention Extractor Twitter Adapter Hashtag
 Extractor Author Extractor BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta JMS Twitter
2013 © Trivadis 4) Count occurrences within period 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 25 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
2013 © Trivadis 5) Adding Cassandra NoSQL for storing results 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 26 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
2013 © Trivadis Implementing in Oracle Event Processing 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 27 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
2013 © Trivadis 1) Creating Twitter Adapter – Connecting to Twitter Stream 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 28
2013 © Trivadis 1) Creating Twitter Adapter – Tweet Event 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 29
2013 © Trivadis 1) Creating Twitter Adapter – Adapter Factory 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 30
2013 © Trivadis 1) Creating Twitter Adapter – Assembly 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 31
2013 © Trivadis 1) Creating Twitter Adapter – Export Adapter to server 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 32
2013 © Trivadis 1) Creating Twitter Adapter – Using Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 33
2013 © Trivadis 2) Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 34
2013 © Trivadis 2) Sending Tweets to BAM – Convert event to JMS MapMessage 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 35
2013 © Trivadis 3) Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 36
2013 © Trivadis 3) Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 37
2013 © Trivadis 4) Count occurrences within period - Using CQL 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 38
2013 © Trivadis Implementation – Complete Picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 39
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 40
2013 © Trivadis Oracle BAM: Architected for Integration and Visualization Processing Twitter Stream with Oracle Event Processing (OEP) Internet BAM Dashboards WebApplications StartPage ActiveViewer ActiveStudio Architect Administrator ReportServer iCommand Oracle Database (Grid) BAM Data & Metadata External Data Objects WebServices Internet Enterprise Integration Framework Application Server BI Web Services JMS Connector BAM Adapter ADF BAM DataControl ADF Pages with DVT BAM ServerEventEngine Actions & Escalations Notification Services ReportCache Snapshots & Change Lists Memory / Disk ActiveDataCache ViewSets API Kernel DataSets DataStorageEngine ODI Databases OLTP & Data Warehouses Mobile Devices Data & Metadata Import & Export BPEL BPM Message Queues CEP OESB 19.02.2014 41
2013 © Trivadis Oracle BAM – Create a Data Object 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 42
2013 © Trivadis Oracle BAM Enterprise Message Source Configuration 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 43
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 44
2013 © Trivadis Implementation – Storing information in NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 45 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
2013 © Trivadis Event Processing Network in OEP 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 46
2013 © Trivadis The world is changing … new data stores Problem of traditional (R)DBMS approach: §  Complex object graph §  Schema evolution §  Semi-structured data §  Scaling Polyglot persistence §  Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + In- Memory) §  Selected based on the way data is being used by individual applications •  Why using an RDBMS if there are better storage alternatives? •  Key/Value, Column Family, Document, Graph-oriented, Relational, … §  Can occur both over the enterprise as well as within a single application 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 47 ORDER ADDRESS CUSTOMER ORDER_LINES Order ID: 1001 Order Date: 15.9.2012 Line Items Customer First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 Name Ipod Touch Monster Beat Apple Mouse Quantity 1 2 1 Price 220.95 190.00 69.90
2013 © Trivadis Apache Cassandra – NoSQL database •  Developed at Facebook •  Open source distributed database management system •  Professional grade support from company called DataStax •  Main Features §  Real-Time §  Highly Distributed §  Support for Multiple Data Center §  Highly Scalable §  No Single Point of Failure §  Fault Tolerant §  Tunable Consistency §  Cassandra Query Language (CQL) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 48
2013 © Trivadis Apache Cassandra - NoSQL Database • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 49 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 Row-key Columns à
2013 © Trivadis Apache Cassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 50
2013 © Trivadis Apache Cassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 51
2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 52
2013 © Trivadis Big Data Reference Architecture – Combine Streaming and Batch 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 53
2013 © Trivadis Summary Cassandra •  No single point of failure •  Forget your data modeling skills •  Model around the queries •  Query Language Oracle Event Processing •  Very light weight server •  Very easy to write adapters •  Very strong CQL language Oracle Business Activity Monitoring §  11g version a bit “old fashioned” §  Easy to integrate through JMS 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 54
2013 © Trivadis Questions and answers ... 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 Guido Schmutz Technology Manager guido.schmutz@trivadis.com 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 55

More Related Content

PDF
Processing Twitter Stream with Oracle Event Processing (OEP)
byGuido Schmutz
 
PDF
Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c
byGuido Schmutz
 
PDF
Unified Log Processing Architecture
byGuido Schmutz
 
PPTX
Fast Data: A Customer’s Journey to Delivering a Compelling Real-Time Solution
byGuido Schmutz
 
PDF
Oracle Stream Explorer - Simplifying Event/Stream Processing
byGuido Schmutz
 
PDF
Blueprints for the analysis of social media
byGuido Schmutz
 
PDF
Introduction to Streaming Analytics
byGuido Schmutz
 
PDF
Oracle Panel: Expert Insights into Faster Oracle SOA Suite Project Delivery
byGuido Schmutz
 
Processing Twitter Stream with Oracle Event Processing (OEP)
byGuido Schmutz
 
Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c
byGuido Schmutz
 
Unified Log Processing Architecture
byGuido Schmutz
 
Fast Data: A Customer’s Journey to Delivering a Compelling Real-Time Solution
byGuido Schmutz
 
Oracle Stream Explorer - Simplifying Event/Stream Processing
byGuido Schmutz
 
Blueprints for the analysis of social media
byGuido Schmutz
 
Introduction to Streaming Analytics
byGuido Schmutz
 
Oracle Panel: Expert Insights into Faster Oracle SOA Suite Project Delivery
byGuido Schmutz
 

What's hot

PDF
Big Data and Fast Data - Lambda Architecture in Action
byGuido Schmutz
 
PDF
Internet of Things - Are traditional architectures good enough?
byGuido Schmutz
 
PDF
Architektur von Big Data Lösungen
byGuido Schmutz
 
PDF
Real Time Analytics with Apache Cassandra - Cassandra Day Munich
byGuido Schmutz
 
PDF
Reliable Data Intestion in BigData / IoT
byGuido Schmutz
 
PDF
SQL vs. NoSQL
byGuido Schmutz
 
PDF
JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchD...
byKai Wähner
 
PPTX
Data Apps with the Lambda Architecture - with Real Work Examples on Merging B...
byAltan Khendup
 
PDF
Real Time Analytics with Apache Cassandra - Cassandra Day Berlin
byGuido Schmutz
 
PPTX
Stream Analytics
byFranco Ucci
 
PDF
Manage the Digital Transformation with Machine Learning in a Reactive Microse...
byDataWorks Summit
 
PDF
Evolution of PaaS
byPaul Fremantle
 
PDF
Introduction to Stream Processing
byGuido Schmutz
 
PDF
Oracle Stream Analytics - Simplifying Stream Processing
byGuido Schmutz
 
PDF
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
byBig Data Spain
 
PDF
Building Reactive Real-time Data Pipeline
byTrieu Nguyen
 
PPTX
Monitoring in Big Data Frameworks @ Big Data Meetup, Timisoara, 2015
byInstitute e-Austria Timisoara
 
PDF
Microsoft Technologies for Data Science 201612
byMark Tabladillo
 
PDF
Big Data Monitoring Cockpit
byStefan Bergstein
 
PDF
Pa getting-started-azure-openshift-ebook-f20686-201911-en
byLisandro Julian Aguero
 
Big Data and Fast Data - Lambda Architecture in Action
byGuido Schmutz
 
Internet of Things - Are traditional architectures good enough?
byGuido Schmutz
 
Architektur von Big Data Lösungen
byGuido Schmutz
 
Real Time Analytics with Apache Cassandra - Cassandra Day Munich
byGuido Schmutz
 
Reliable Data Intestion in BigData / IoT
byGuido Schmutz
 
SQL vs. NoSQL
byGuido Schmutz
 
JBoss OneDayTalk 2013: "NoSQL Integration with Apache Camel - MongoDB, CouchD...
byKai Wähner
 
Data Apps with the Lambda Architecture - with Real Work Examples on Merging B...
byAltan Khendup
 
Real Time Analytics with Apache Cassandra - Cassandra Day Berlin
byGuido Schmutz
 
Stream Analytics
byFranco Ucci
 
Manage the Digital Transformation with Machine Learning in a Reactive Microse...
byDataWorks Summit
 
Evolution of PaaS
byPaul Fremantle
 
Introduction to Stream Processing
byGuido Schmutz
 
Oracle Stream Analytics - Simplifying Stream Processing
byGuido Schmutz
 
The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...
byBig Data Spain
 
Building Reactive Real-time Data Pipeline
byTrieu Nguyen
 
Monitoring in Big Data Frameworks @ Big Data Meetup, Timisoara, 2015
byInstitute e-Austria Timisoara
 
Microsoft Technologies for Data Science 201612
byMark Tabladillo
 
Big Data Monitoring Cockpit
byStefan Bergstein
 
Pa getting-started-azure-openshift-ebook-f20686-201911-en
byLisandro Julian Aguero
 

Similar to Processing Twitter Stream with Oracle Event Processing (OEP)

PDF
Soa12c launch 5 event processing shmakov eng cr
byVasily Demin
 
PDF
Speeding up big data with event processing
byAlexandre de Castro Alves
 
PDF
Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon ...
byStampedeCon
 
PDF
Łukasz Romaszewski on Internet of Things Raspberry Pi and Java Embedded JavaC...
byTomek Borek
 
PDF
Complex Event Processing with Esper
byAntónio Alegria
 
PPT
Oracle Cep Xstreams adapter
byMohit Thatte
 
PDF
Demo intelligent user experience with oracle mobility for publishing
byVasily Demin
 
PPTX
Apache Spark and Oracle Stream Analytics
byPrabhu Thukkaram
 
PDF
Oracle Stream Explorer
byTrivadis
 
PDF
Oracle Stream Explorer Guido Schmutz
byDésirée Pfister
 
PDF
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
byRittman Analytics
 
PPTX
Guido schmutz-jax2011-event-driven soa
byGuido Schmutz
 
PDF
Social Stream Analysis Use Cases
byWSO2
 
ODP
Batch processing in EDA (Event Driven Architectures)
byÁlvaro Santuy Elena
 
PDF
Event-Processing-und-BigData-kombiniert-guido_schmutz
byTrivadis
 
PPTX
How Fast Data Is Turned into Fast Information and Timely Action (OOW 2014)
byLucas Jellema
 
PDF
How fast data is turned into fast information and timely action - Oracle Open...
byGetting value from IoT, Integration and Data Analytics
 
PPTX
Big Data Analytics Strategy and Roadmap
bySrinath Perera
 
PDF
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
byWSO2
 
PDF
Event Stream Processing SAP
byGaurav Ahluwalia
 
Soa12c launch 5 event processing shmakov eng cr
byVasily Demin
 
Speeding up big data with event processing
byAlexandre de Castro Alves
 
Real Time Event Processing and In-­memory analysis of Big Data - StampedeCon ...
byStampedeCon
 
Łukasz Romaszewski on Internet of Things Raspberry Pi and Java Embedded JavaC...
byTomek Borek
 
Complex Event Processing with Esper
byAntónio Alegria
 
Oracle Cep Xstreams adapter
byMohit Thatte
 
Demo intelligent user experience with oracle mobility for publishing
byVasily Demin
 
Apache Spark and Oracle Stream Analytics
byPrabhu Thukkaram
 
Oracle Stream Explorer
byTrivadis
 
Oracle Stream Explorer Guido Schmutz
byDésirée Pfister
 
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
byRittman Analytics
 
Guido schmutz-jax2011-event-driven soa
byGuido Schmutz
 
Social Stream Analysis Use Cases
byWSO2
 
Batch processing in EDA (Event Driven Architectures)
byÁlvaro Santuy Elena
 
Event-Processing-und-BigData-kombiniert-guido_schmutz
byTrivadis
 
How Fast Data Is Turned into Fast Information and Timely Action (OOW 2014)
byLucas Jellema
 
How fast data is turned into fast information and timely action - Oracle Open...
byGetting value from IoT, Integration and Data Analytics
 
Big Data Analytics Strategy and Roadmap
bySrinath Perera
 
WSO2 Product Release Webinar - Introducing the WSO2 Complex Event Processor
byWSO2
 
Event Stream Processing SAP
byGaurav Ahluwalia
 

More from Trivadis

PDF
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
byTrivadis
 
PDF
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
byTrivadis
 
PDF
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
byTrivadis
 
PDF
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
byTrivadis
 
PDF
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
byTrivadis
 
PDF
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
byTrivadis
 
PDF
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
byTrivadis
 
PDF
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
byTrivadis
 
PDF
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...
byTrivadis
 
PDF
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
byTrivadis
 
PDF
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
byTrivadis
 
PDF
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
byTrivadis
 
PDF
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
byTrivadis
 
PDF
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
byTrivadis
 
PDF
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
byTrivadis
 
PDF
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
byTrivadis
 
PDF
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
byTrivadis
 
PDF
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
byTrivadis
 
PDF
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
byTrivadis
 
PDF
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
byTrivadis
 
Azure Days 2019: Azure Chatbot Development for Airline Irregularities (Remco ...
byTrivadis
 
Azure Days 2019: Trivadis Azure Foundation – Das Fundament für den ... (Nisan...
byTrivadis
 
Azure Days 2019: Business Intelligence auf Azure (Marco Amhof & Yves Mauron)
byTrivadis
 
Azure Days 2019: Master the Move to Azure (Konrad Brunner)
byTrivadis
 
Azure Days 2019: Keynote Azure Switzerland – Status Quo und Ausblick (Primo A...
byTrivadis
 
Azure Days 2019: Grösser und Komplexer ist nicht immer besser (Meinrad Weiss)
byTrivadis
 
Azure Days 2019: Get Connected with Azure API Management (Gerry Keune & Stefa...
byTrivadis
 
Azure Days 2019: Infrastructure as Code auf Azure (Jonas Wanninger & Daniel H...
byTrivadis
 
Azure Days 2019: Wie bringt man eine Data Analytics Plattform in die Cloud? (...
byTrivadis
 
Azure Days 2019: Azure@Helsana: Die Erweiterung von Dynamics CRM mit Azure Po...
byTrivadis
 
TechEvent 2019: Kundenstory - Kein Angebot, kein Auftrag – Wie Du ein individ...
byTrivadis
 
TechEvent 2019: Oracle Database Appliance M/L - Erfahrungen und Erfolgsmethod...
byTrivadis
 
TechEvent 2019: Security 101 für Web Entwickler; Roland Krüger - Trivadis
byTrivadis
 
TechEvent 2019: Trivadis & Swisscom Partner Angebote; Konrad Häfeli, Markus O...
byTrivadis
 
TechEvent 2019: DBaaS from Swisscom Cloud powered by Trivadis; Konrad Häfeli ...
byTrivadis
 
TechEvent 2019: Status of the partnership Trivadis and EDB - Comparing Postgr...
byTrivadis
 
TechEvent 2019: More Agile, More AI, More Cloud! Less Work?!; Oliver Dörr - T...
byTrivadis
 
TechEvent 2019: Kundenstory - Vom Hauptmann zu Köpenick zum Polizisten 2020 -...
byTrivadis
 
TechEvent 2019: Vom Rechenzentrum in die Oracle Cloud - Übertragungsmethoden;...
byTrivadis
 
TechEvent 2019: The sleeping Power of Data; Eberhard Lösch - Trivadis
byTrivadis
 

Recently uploaded

PDF
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
PPTX
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
PDF
apidays New York 2025 | AI in Application Security
byapidays
 
PDF
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
PDF
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
PDF
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
PDF
apidays Paris 2025 | Zero Trust By Design
byapidays
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PPTX
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
PPTX
Celonis Optimizing your future with process
bydevjeremyappleyard
 
PDF
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
PDF
final.pdf
byomarbishtawi04
 
PDF
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
PPTX
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
PDF
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PDF
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
PDF
UiPath Automation Developer Associate Training Series 2025 - Session 4
byDianaGray10
 
PDF
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
apidays New York 2025 | AI in Application Security
byapidays
 
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
apidays Paris 2025 | Zero Trust By Design
byapidays
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
Celonis Optimizing your future with process
bydevjeremyappleyard
 
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
final.pdf
byomarbishtawi04
 
Agent to agent service discovery using HashiCorp Consul and Vault
byBram Vogelaar
 
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
GDG Cloud Southlake #49: Pradeep R Kumar: Implications of Agentic AI for Iden...
byJames Anderson
 
UiPath Automation Developer Associate Training Series 2025 - Session 4
byDianaGray10
 
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 

Processing Twitter Stream with Oracle Event Processing (OEP)

  • 1.
    2013 © Trivadis BASELBERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) Guido Schmutz OFM Partner Forum Malta 19.2.2014 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 1
  • 2.
    2013 © Trivadis GuidoSchmutz •  Working for Trivadis for more than 17 years •  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis •  More than 25 years of software development experience •  Contact: guido.schmutz@trivadis.com •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 2
  • 3.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 3
  • 4.
    2013 © Trivadis BigData Definition (4 Vs) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4 + Time to action ? – Big Data + Event Processing = Fast Data Characteristics of Big Data: Its Volume, Velocity and Variety in combination
  • 5.
    2013 © Trivadis Theworld is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 5
  • 6.
    2013 © Trivadis Whois generating Big Data? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 6 The progress and innovation is no longer hindered by the ability to collect data But by the ability to manage, analyze, summarize, visualize and discover knowledge from the collected data in a timely manner and in a scalable fashion Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Mobile devices (tracking all objects all the time) Sensor technology and networks (measuring all kinds of data)
  • 7.
    2013 © Trivadis 19.02.2014 ProcessingTwitter Stream with Oracle Event Processing (OEP) 7
  • 8.
    2013 © Trivadis InternetOf Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 8 Source: CiscoSource: The Economist
  • 9.
    2013 © Trivadis Dataas an Asset - Store Anything? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 9 But then data is
 just too valuable
 to delete!
 We must 
 store anything! Nonsense! Just 
 store the data 
 you know 
 you need today! It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today!
  • 10.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 10
  • 11.
    2013 © Trivadis RetrieveTweets and Visualize 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 11
  • 12.
    2013 © Trivadis Accessto Tweets 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 12 Quelle Source Limitations Cost Twitter’s Search API 3200 / user 5000 / keyword 180 requests / 15 minutes free Twitter’s Streaming API 1%-40% of total volume free DataSift none 0.15 -0.20$ / unit Gnip none On request
  • 13.
    2013 © Trivadis Howto design a stream (event) processing system? 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 13 Twitter Stream tweet Sensor tweet Persist (Queue) Twitter Stream tweet Sensor tweet Processing tweet Processing result result Twitter Stream tweet Receiving/ Processing result
  • 14.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 14
  • 15.
    2013 © Trivadis OracleEvent Processing (OEP) - Engine Lightweight Java Application Server •  Full environment for running Java applications •  Module Framework - OSGi High Throughput •  Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment •  Service Framework – Spring DM, POJO Enterprise Web 2.0 & Eclipse-based tooling Multiple-choice VM •  JRockit or WebLogic RealTime 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 15
  • 16.
    2013 © Trivadis OracleEvent Processing – Event Processing Network Concept 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 16
  • 17.
    2013 © Trivadis OracleEvent Processing – In Memory, Continuous Queries Event Processing Output §  Filtering -  New stream filtered for specific criteria, e.g. stock price > $22 §  Correlation & Aggregation -  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour §  Pattern Matching -  Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 17
  • 18.
    2013 © Trivadis OracleEvent Processing - CQL Initiative for a complete “continuous” query language Start with SQL ’99 plus “continuous” query extensions §  Based on Stanford University research Industry standards discussions §  Event Processing Technical Society (EPTS) §  ANSI SQL §  OMG Adoption Today §  ANSI SQL Standards Proposal for CQL Pattern Matching -  Oracle, IBM, Stanford University §  OpenSource Adoption of CQL §  Oracle Complex Event Processor (CEP) Releaseà Available in 11g 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 18
  • 19.
    2013 © Trivadis OracleEvent Processing – Visual Development Tools 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 19
  • 20.
    2013 © Trivadis OracleEvent Processing – Operation & Management 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 20
  • 21.
    2013 © Trivadis Implementation– complete picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 21 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds
  • 22.
    2013 © Trivadis 1)Creating a Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 22 Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter
  • 23.
    2013 © Trivadis 2)Send Tweets to BAM 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 23 Twitter Adapter BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta JMS Twitter
  • 24.
    2013 © Trivadis 3)Extract interesting information from Tweet 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 24 Mention Extractor Twitter Adapter Hashtag
 Extractor Author Extractor BAM Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity robertvanmolken #ofmforum #malta JMS Twitter
  • 25.
    2013 © Trivadis 4)Count occurrences within period 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 25 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
  • 26.
    2013 © Trivadis 5)Adding Cassandra NoSQL for storing results 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 26 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta #ofmforum,5 #malta,2 Robertvanmolken,1 @SOASimone,1 @SOACommunity,5 JMS JMS Twitter range 30 seconds
 slide 30 seconds @SOASimone @SOACommunity robertvanmolken #ofmforum #malta
  • 27.
    2013 © Trivadis Implementingin Oracle Event Processing 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 27 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
  • 28.
    2013 © Trivadis 1)Creating Twitter Adapter – Connecting to Twitter Stream 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 28
  • 29.
    2013 © Trivadis 1)Creating Twitter Adapter – Tweet Event 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 29
  • 30.
    2013 © Trivadis 1)Creating Twitter Adapter – Adapter Factory 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 30
  • 31.
    2013 © Trivadis 1)Creating Twitter Adapter – Assembly 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 31
  • 32.
    2013 © Trivadis 1)Creating Twitter Adapter – Export Adapter to server 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 32
  • 33.
    2013 © Trivadis 1)Creating Twitter Adapter – Using Twitter Adapter 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 33
  • 34.
    2013 © Trivadis 2)Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 34
  • 35.
    2013 © Trivadis 2)Sending Tweets to BAM – Convert event to JMS MapMessage 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 35
  • 36.
    2013 © Trivadis 3)Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 36
  • 37.
    2013 © Trivadis 3)Extract information from Tweet – Extract Hashtags from TweetEvent 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 37
  • 38.
    2013 © Trivadis 4)Count occurrences within period - Using CQL 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 38
  • 39.
    2013 © Trivadis Implementation– Complete Picture 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 39
  • 40.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 40
  • 41.
    2013 © Trivadis OracleBAM: Architected for Integration and Visualization Processing Twitter Stream with Oracle Event Processing (OEP) Internet BAM Dashboards WebApplications StartPage ActiveViewer ActiveStudio Architect Administrator ReportServer iCommand Oracle Database (Grid) BAM Data & Metadata External Data Objects WebServices Internet Enterprise Integration Framework Application Server BI Web Services JMS Connector BAM Adapter ADF BAM DataControl ADF Pages with DVT BAM ServerEventEngine Actions & Escalations Notification Services ReportCache Snapshots & Change Lists Memory / Disk ActiveDataCache ViewSets API Kernel DataSets DataStorageEngine ODI Databases OLTP & Data Warehouses Mobile Devices Data & Metadata Import & Export BPEL BPM Message Queues CEP OESB 19.02.2014 41
  • 42.
    2013 © Trivadis OracleBAM – Create a Data Object 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 42
  • 43.
    2013 © Trivadis OracleBAM Enterprise Message Source Configuration 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 43
  • 44.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 44
  • 45.
    2013 © Trivadis Implementation– Storing information in NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 45 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor Cassandra Counter BAM Tweet Cassandra Tweet BAM Counter JMS JMS Twitter range 30 seconds
 slide 30 seconds
  • 46.
    2013 © Trivadis EventProcessing Network in OEP 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 46
  • 47.
    2013 © Trivadis Theworld is changing … new data stores Problem of traditional (R)DBMS approach: §  Complex object graph §  Schema evolution §  Semi-structured data §  Scaling Polyglot persistence §  Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + In- Memory) §  Selected based on the way data is being used by individual applications •  Why using an RDBMS if there are better storage alternatives? •  Key/Value, Column Family, Document, Graph-oriented, Relational, … §  Can occur both over the enterprise as well as within a single application 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 47 ORDER ADDRESS CUSTOMER ORDER_LINES Order ID: 1001 Order Date: 15.9.2012 Line Items Customer First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 Name Ipod Touch Monster Beat Apple Mouse Quantity 1 2 1 Price 220.95 190.00 69.90
  • 48.
    2013 © Trivadis ApacheCassandra – NoSQL database •  Developed at Facebook •  Open source distributed database management system •  Professional grade support from company called DataStax •  Main Features §  Real-Time §  Highly Distributed §  Support for Multiple Data Center §  Highly Scalable §  No Single Point of Failure §  Fault Tolerant §  Tunable Consistency §  Cassandra Query Language (CQL) 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 48
  • 49.
    2013 © Trivadis ApacheCassandra - NoSQL Database • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 49 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 Row-key Columns à
  • 50.
    2013 © Trivadis ApacheCassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 50
  • 51.
    2013 © Trivadis ApacheCassandra – NoSQL database 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 51
  • 52.
    2013 © Trivadis AGENDA 1. Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (CEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 52
  • 53.
    2013 © Trivadis BigData Reference Architecture – Combine Streaming and Batch 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 53
  • 54.
    2013 © Trivadis Summary Cassandra • No single point of failure •  Forget your data modeling skills •  Model around the queries •  Query Language Oracle Event Processing •  Very light weight server •  Very easy to write adapters •  Very strong CQL language Oracle Business Activity Monitoring §  11g version a bit “old fashioned” §  Easy to integrate through JMS 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 54
  • 55.
    2013 © Trivadis Questionsand answers ... 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 Guido Schmutz Technology Manager guido.schmutz@trivadis.com 19.02.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 55