Your SlideShare is downloading. ×
0
Processing Twitter Stream with
Oracle Event Processing (OEP)
Guido Schmutz
OFM Partner Forum Malta
19.2.2014

BASEL

1

BE...
Guido Schmutz
• 
• 

Working for Trivadis for more than 17 years
Oracle ACE Director for Fusion Middleware and SOA

• 
• 
...
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Big Data Definition (4 Vs)

Characteristics of Big Data: Its Volume,
Velocity and Variety in combination

+ Time to action...
The world is changing …
The model of Generating/Consuming Data has changed ….
Old Model: few companies are generating data...
Who is generating Big Data?

Mobile devices
(tracking all objects all the time)
Social media and networks
(all of us are g...
7

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02.2014
Internet Of Things – Sensors
are/will be everywhere
There are more devices tapping into the
internet than people on earth
...
Data as an Asset - Store Anything?

But then data is

just too valuable

to delete!

We must 

store anything!

9

It depe...
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Retrieve Tweets and Visualize

11

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02.2014
Access to Tweets
Source
Twitter’s Search API

Twitter’s Streaming API
DataSift

Limitations

Cost

3200 / user
5000 / keyw...
How to design a stream (event) processing system?

Twitter
Stream

Twitter
Stream

tweet

Twitter
Stream

13

tweet

tweet...
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Oracle Event Processing (OEP) - Engine
Lightweight Java Application Server
•  Full environment for running Java applicatio...
Oracle Event Processing – Event Processing Network
Concept

16

2013 © Trivadis
Processing Twitter Stream with Oracle Even...
Oracle Event Processing – In Memory, Continuous
Queries
Event Processing Output
§  Filtering
-  New stream filtered for s...
Oracle Event Processing - CQL
Initiative for a complete “continuous” query language
Start with SQL ’99 plus “continuous” q...
Oracle Event Processing – Visual Development Tools

19

2013 © Trivadis
Processing Twitter Stream with Oracle Event Proces...
Oracle Event Processing – Operation & Management

20

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processi...
Implementation – complete picture
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta
@SOAS...
1) Creating a Twitter Adapter

Twitter

Twitter
Adapter
@SOASimone
@SOACommunity heard you
couldn’t make it. We miss
you! ...
2) Send Tweets to BAM
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta

Twitter

Twitter...
3) Extract interesting information from Tweet
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum ...
4) Count occurrences within period
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum #malta

BAM...
5) Adding Cassandra NoSQL for storing results
@SOASimone @SOACommunity
heard you couldn’t make it. We
miss you! #ofmforum ...
Implementing in Oracle Event Processing

BAM
Tweet

JMS

BAM
Counter

JMS

Twitter
Hashtag

Extractor
Mention
Extractor

T...
1) Creating Twitter Adapter –
Connecting to Twitter Stream

28

2013 © Trivadis
Processing Twitter Stream with Oracle Even...
1) Creating Twitter Adapter –
Tweet Event

29

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP...
1) Creating Twitter Adapter –
Adapter Factory

30

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing ...
1) Creating Twitter Adapter –
Assembly

31

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
1...
1) Creating Twitter Adapter –
Export Adapter to server

32

2013 © Trivadis
Processing Twitter Stream with Oracle Event Pr...
1) Creating Twitter Adapter –
Using Twitter Adapter

33

2013 © Trivadis
Processing Twitter Stream with Oracle Event Proce...
2) Sending Tweets to BAM
Using Oracle BAM Enterprise Message Sources (JMS) interface

34

2013 © Trivadis
Processing Twitt...
2) Sending Tweets to BAM –
Convert event to JMS MapMessage

35

2013 © Trivadis
Processing Twitter Stream with Oracle Even...
3) Extract information from Tweet –
Extract Hashtags from TweetEvent

36

2013 © Trivadis
Processing Twitter Stream with O...
3) Extract information from Tweet –
Extract Hashtags from TweetEvent

37

2013 © Trivadis
Processing Twitter Stream with O...
4) Count occurrences within
period - Using CQL

38

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing...
Implementation – Complete Picture

39

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02....
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Oracle BAM: Architected for Integration and
Visualization
Application Server
Message
Queues

CEP

WebApplications

BAM Ser...
Oracle BAM – Create a
Data Object

42

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02....
Oracle BAM Enterprise Message
Source Configuration

43

2013 © Trivadis
Processing Twitter Stream with Oracle Event Proces...
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Implementation – Storing information in NoSQL
database
BAM
Tweet

JMS

Cassandra
Tweet
Twitter
Hashtag

Extractor
Mention
...
Event Processing Network in OEP

46

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02.20...
The world is changing …
new data stores

ORDER
Order
ID: 1001
Order Date: 15.9.2012

Problem of traditional (R)DBMS approa...
Apache Cassandra – NoSQL database
•  Developed at Facebook
•  Open source distributed database management system
•  Profes...
Apache Cassandra - NoSQL Database
• Don’t think of relational table => more of a sorted map
• Know your application => mod...
Apache Cassandra – NoSQL database

50

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02....
Apache Cassandra – NoSQL database

51

2013 © Trivadis
Processing Twitter Stream with Oracle Event Processing (OEP)
19.02....
AGENDA
1.  Introduction
2.  Twitter Use Case
3.  Processing with Oracle Event Processing (OEP)
4.  Visualization with Orac...
Big Data Reference Architecture – Combine Streaming
and Batch

53

2013 © Trivadis
Processing Twitter Stream with Oracle E...
Summary
Oracle Event Processing

Cassandra

•  Very light weight server

•  No single point of failure

•  Very easy to wr...
Questions and answers ...
Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com

BASEL

55

BERN

BRUGG

LAUSANNE

Z...
Upcoming SlideShare
Loading in...5
×

Processing Twitter Stream with Oracle Event Processing (OEP)

2,071

Published on

This session will present how to connect to the Twitter Streaming API and process and analyse the tweets in real-time by using Oracle Event Processing. We will see how typical analytics on a such social media data can be done directly on the event stream using Java and/or the Continuous Query Language (CQL). How do we best store the results, so that they can be made available continously to a dashboard for presentation will be shown as well. Additionally the so-called Lambda architecture will be presented, where such an event streaming solution plays a critical part, in combination with a Big Data batch processing layer.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,071
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
79
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Processing Twitter Stream with Oracle Event Processing (OEP)"

  1. 1. Processing Twitter Stream with Oracle Event Processing (OEP) Guido Schmutz OFM Partner Forum Malta 19.2.2014 BASEL 1 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 

  2. 2. Guido Schmutz •  •  Working for Trivadis for more than 17 years Oracle ACE Director for Fusion Middleware and SOA •  •  Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data •  •  Member of Trivadis Architecture Board Technology Manager @ Trivadis •  More than 25 years of software development experience •  Contact: guido.schmutz@trivadis.com •  •  Blog: http://guidoschmutz.wordpress.com Twitter: gschmutz 2 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  3. 3. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 3 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  4. 4. Big Data Definition (4 Vs) Characteristics of Big Data: Its Volume, Velocity and Variety in combination + Time to action ? – Big Data + Event Processing = Fast Data 4 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  5. 5. The world is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 5 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  6. 6. Who is generating Big Data? Mobile devices (tracking all objects all the time) Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But by the ability to manage, analyze, summarize, visualize and discover knowledge from the collected data in a timely manner and in a scalable fashion 6 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  7. 7. 7 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  8. 8. Internet Of Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 8 2013 © Trivadis Source: The Economist Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 Source: Cisco
  9. 9. Data as an Asset - Store Anything? But then data is
 just too valuable
 to delete!
 We must 
 store anything! 9 It depends … but Big Data technologies allow to store the raw information from both new data sources as well as existing ones so that you can later use it to create new data-driven products, you would not have thought about today! 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 Nonsense! Just 
 store the data 
 you know 
 you need today!
  10. 10. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 10 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  11. 11. Retrieve Tweets and Visualize 11 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  12. 12. Access to Tweets Source Twitter’s Search API Twitter’s Streaming API DataSift Limitations Cost 3200 / user 5000 / keyword 180 requests / 15 minutes free 1%-40% of total volume free Quelle none Gnip 12 none 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 0.15 -0.20$ / unit On request
  13. 13. How to design a stream (event) processing system? Twitter Stream Twitter Stream tweet Twitter Stream 13 tweet tweet Receiving/ Processing Sensor Sensor result tweet tweet Processing Persist (Queue) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 result tweet Processing result
  14. 14. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 14 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  15. 15. Oracle Event Processing (OEP) - Engine Lightweight Java Application Server •  Full environment for running Java applications •  Module Framework - OSGi High Throughput •  Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment •  Service Framework – Spring DM, POJO Enterprise Web 2.0 & Eclipse-based tooling Multiple-choice VM •  JRockit or WebLogic RealTime 15 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  16. 16. Oracle Event Processing – Event Processing Network Concept 16 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  17. 17. Oracle Event Processing – In Memory, Continuous Queries Event Processing Output §  Filtering -  New stream filtered for specific criteria, e.g. stock price > $22 §  Correlation & Aggregation -  Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour §  Pattern Matching -  17 Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  18. 18. Oracle Event Processing - CQL Initiative for a complete “continuous” query language Start with SQL ’99 plus “continuous” query extensions §  Based on Stanford University research Industry standards discussions §  Event Processing Technical Society (EPTS) §  ANSI SQL §  OMG Adoption Today §  ANSI SQL Standards Proposal for CQL Pattern Matching -  Oracle, IBM, Stanford University §  OpenSource Adoption of CQL §  Oracle Complex Event Processor (CEP) Releaseà Available in 11g 18 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  19. 19. Oracle Event Processing – Visual Development Tools 19 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  20. 20. Oracle Event Processing – Operation & Management 20 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  21. 21. Implementation – complete picture @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS Cassandra Tweet Twitter #ofmforum Hashtag
 #malta Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 21 Cassandra @SOACommunity,5 Counter @SOASimone Counter
 #ofmforum,5 Mention @SOACommunity Processor Extractor #malta,2 Robertvanmolken,1 BAM range 30 seconds
 Counter slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 @SOASimone,1 JMS
  22. 22. 1) Creating a Twitter Adapter Twitter Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 22 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  23. 23. 2) Send Tweets to BAM @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 23 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Tweet JMS
  24. 24. 3) Extract interesting information from Tweet @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta Twitter #ofmforum Hashtag
 #malta Extractor @SOASimone Mention @SOACommunity Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 24 Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Tweet JMS
  25. 25. 4) Count occurrences within period @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS BAM Counter JMS Twitter #ofmforum Hashtag
 #malta Extractor @SOASimone,1 @SOACommunity,5 @SOASimone Mention #ofmforum,5 Counter
 @SOACommunity Processor Extractor #malta,2 Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 25 Robertvanmolken,1 range 30 seconds
 slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  26. 26. 5) Adding Cassandra NoSQL for storing results @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta BAM Tweet JMS Cassandra Tweet Twitter #ofmforum Hashtag
 #malta Extractor Twitter Adapter @SOASimone @SOACommunity heard you couldn’t make it. We miss you! #ofmforum #malta 26 Cassandra @SOACommunity,5 Counter @SOASimone Counter
 #ofmforum,5 Mention @SOACommunity Processor Extractor #malta,2 Robertvanmolken,1 BAM range 30 seconds
 Counter slide 30 seconds Author Extractor robertvanmolken 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 @SOASimone,1 JMS
  27. 27. Implementing in Oracle Event Processing BAM Tweet JMS BAM Counter JMS Twitter Hashtag
 Extractor Mention Extractor Twitter Adapter Author Extractor 27 Counter
 Processor range 30 seconds
 slide 30 seconds 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  28. 28. 1) Creating Twitter Adapter – Connecting to Twitter Stream 28 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  29. 29. 1) Creating Twitter Adapter – Tweet Event 29 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  30. 30. 1) Creating Twitter Adapter – Adapter Factory 30 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  31. 31. 1) Creating Twitter Adapter – Assembly 31 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  32. 32. 1) Creating Twitter Adapter – Export Adapter to server 32 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  33. 33. 1) Creating Twitter Adapter – Using Twitter Adapter 33 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  34. 34. 2) Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 34 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  35. 35. 2) Sending Tweets to BAM – Convert event to JMS MapMessage 35 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  36. 36. 3) Extract information from Tweet – Extract Hashtags from TweetEvent 36 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  37. 37. 3) Extract information from Tweet – Extract Hashtags from TweetEvent 37 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  38. 38. 4) Count occurrences within period - Using CQL 38 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  39. 39. Implementation – Complete Picture 39 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  40. 40. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 40 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  41. 41. Oracle BAM: Architected for Integration and Visualization Application Server Message Queues CEP WebApplications BAM Server Enterprise Integration Framework ActiveDataCache BPM ActiveViewer Actions & Escalations API Architect Administrator Kernel WebServices Web Services Internet Internet ReportCache ReportServer ViewSets ADF Pages with DVT DataSets Snapshots & Change Lists DataStorageEngine Memory / Disk ADF ODI BAM DataControl External Data Objects iCommand BI Data & Metadata Import & Export OLTP & Data Warehouses BAM Data & Metadata Databases 41 BAM Dashboards Notification Services BAM Adapter BPEL StartPage ActiveStudio JMS Connector OESB Mobile Devices EventEngine Oracle Database (Grid) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  42. 42. Oracle BAM – Create a Data Object 42 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  43. 43. Oracle BAM Enterprise Message Source Configuration 43 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  44. 44. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 44 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  45. 45. Implementation – Storing information in NoSQL database BAM Tweet JMS Cassandra Tweet Twitter Hashtag
 Extractor Mention Extractor Twitter Adapter Author Extractor 45 Cassandra Counter Counter
 Processor range 30 seconds
 slide 30 seconds 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 BAM Counter JMS
  46. 46. Event Processing Network in OEP 46 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  47. 47. The world is changing … new data stores ORDER Order ID: 1001 Order Date: 15.9.2012 Problem of traditional (R)DBMS approach: §  §  §  §  Complex object graph Schema evolution Semi-structured data Scaling Customer CUSTOMER First Name: Peter Last Name: Sample Billing Address Street: Somestreet 10 City: Somewhere Postal Code: 55901 ADDRESS Line Items Quantity Price Ipod Touch 1 220.95 Monster Beat 2 190.00 Apple Mouse 1 69.90 Name ORDER_LINES Polyglot persistence §  Using multiple data storage technologies (RDMBS + NoSQL + NewSQL + InMemory) §  Selected based on the way data is being used by individual applications •  Why using an RDBMS if there are better storage alternatives? •  Key/Value, Column Family, Document, Graph-oriented, Relational, … §  Can occur both over the enterprise as well as within a single application 47 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  48. 48. Apache Cassandra – NoSQL database •  Developed at Facebook •  Open source distributed database management system •  Professional grade support from company called DataStax •  Main Features §  §  §  §  §  §  §  §  48 Real-Time Highly Distributed Support for Multiple Data Center Highly Scalable No Single Point of Failure Fault Tolerant Tunable Consistency Cassandra Query Language (CQL) 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  49. 49. Apache Cassandra - NoSQL Database • Don’t think of relational table => more of a sorted map • Know your application => model around the queries • De-normalize and duplicate for read performance • Index is not an afterthought, anymore=> index upfront • Think of physical storage structure Row-key Columns à 2013-08 Day-1, keyword-1=>100 Day-2, keyword-1=>150 Day-3, keyword-1=>170 …. Day-31, keyword-1 =>170 2013-08-31 Hour-1, keyword-1 =>10 Hour-2, keyword-1 =>15 Hour-3, keyword-1 =>17 …. Hour-24, keyword-1 =>17 2013-08-31-10 Minute-1, keyword-1=>2 Minute-2, keyword-1=>3 Minute-3, keyword-1 =>5 …. Minute-60, keyword-1=>2 49 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  50. 50. Apache Cassandra – NoSQL database 50 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  51. 51. Apache Cassandra – NoSQL database 51 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  52. 52. AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Store Information in Apache Cassandra 6.  Summary 52 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  53. 53. Big Data Reference Architecture – Combine Streaming and Batch 53 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  54. 54. Summary Oracle Event Processing Cassandra •  Very light weight server •  No single point of failure •  Very easy to write adapters •  Forget your data modeling skills •  Very strong CQL language •  Model around the queries •  Query Language Oracle Business Activity Monitoring §  11g version a bit “old fashioned” §  Easy to integrate through JMS 54 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014
  55. 55. Questions and answers ... Guido Schmutz Technology Manager guido.schmutz@trivadis.com BASEL 55 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. 2013 © Trivadis Processing Twitter Stream with Oracle Event Processing (OEP) 19.02.2014 FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 

  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×