Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

1,651 views

Published on

This session will present how to connect to the Twitter Streaming API and process and analyse the tweets in real-time by using Oracle Event Processing. We will see how typical analytics on a such social media data can be done directly on the event stream using Java and/or the Continuous Query Language (CQL). How do we best store the results, so that they can be made available continously to a dashboard for presentation will be shown as well. Additionally the so-called Lambda architecture will be presented, where such an event streaming solution plays a critical part, in combination with a Big Data batch processing layer.

Published in: Technology
  • Be the first to comment

Processing Twitter Events in Real-Time with Oracle Event Processing (OEP) 12c

  1. 1. 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 2013 © Trivadis Echtzeitverarbeitung von Twitter Streams mit Oracle Event Processing (OEP) 12c Guido Schmutz DOAG Konferenz 2014 19.11.2014 – 15:00 Raum Kiev 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 1
  2. 2. 2013 © Trivadis Guido Schmutz •  Working for Trivadis for more than 17 years •  Oracle ACE Director for Fusion Middleware and SOA •  Co-Author of different books •  Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data •  Member of Trivadis Architecture Board •  Technology Manager @ Trivadis •  More than 25 years of software development experience •  Contact: guido.schmutz@trivadis.com •  Blog: http://guidoschmutz.wordpress.com •  Twitter: gschmutz 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 2
  3. 3. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Event Processing in Information Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 3
  4. 4. 2013 © Trivadis Big Data Definition (4 Vs) 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4 + Time to action ? – Big Data + Event Processing = Fast Data Characteristics of Big Data: Its Volume, Velocity and Variety in combination
  5. 5. 2013 © Trivadis The world is changing … The model of Generating/Consuming Data has changed …. Old Model: few companies are generating data, all others are consuming data New Model: all of use are generating data, and all of us are consuming data 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 5
  6. 6. 2013 © Trivadis 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 6
  7. 7. 2013 © Trivadis Internet Of Things – Sensors are/will be everywhere There are more devices tapping into the internet than people on earth How do we prepare our systems/architecture for the future? 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 7 Source: CiscoSource: The Economist
  8. 8. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Event Processing in Information Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 8
  9. 9. 2013 © Trivadis Retrieve Tweets and Visualize 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 9
  10. 10. 2013 © Trivadis Access to Tweets 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 10 Quelle Source Limitations Cost Twitter’s Search API 3200 / user 5000 / keyword 180 requests / 15 minutes free Twitter’s Streaming API 1%-40% of total volume free DataSift none 0.15 -0.20$ / unit Gnip none On request
  11. 11. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Event Processing in Information Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 11
  12. 12. 2013 © Trivadis Oracle Event Processing (OEP) - Engine Lightweight Java Application Server •  Full environment for running Java applications •  Module Framework - OSGi High Throughput •  Hundreds of thousands of events/second Event Processing Infrastructure Easy-to-use development environment •  Service Framework – Spring DM, POJO Multiple-choice VM •  JRockit or WebLogic RealTime 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 12
  13. 13. 2013 © Trivadis Oracle Event Processing Application Adapter •  Provide a way for EPN to exchange data with components that are external to network •  Either an Event Source (inbound) or an Event Sink (outbound) Processor •  Contain query code written in CQL and consumes event data from a preceding channel •  May also generate new events to an output channel •  CQL is an extension of SQL 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 13
  14. 14. 2013 © Trivadis Oracle Event Processing Application Bean Bean •  Provide a place for logic written in Java •  Java code can receive events (Event Sink) and/or send events (Event Source) •  Are implemented using the Spring Framework model or the OEP event bean model Channel •  Two types of channel exist: Stream and Relation CQL Processor •  Executes Continous Query Language (CQL) queries •  Used to declaratively describe logic (queries) to be applied on the event stream 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 14
  15. 15. 2013 © Trivadis Oracle Event Processing - CQL CQL is an extension to Structured Query Language (SQL) •  Same keywords and syntax rules, but additional support for streaming data event conceptually corresponds to a row in a database table, but •  one event is always before or after another in time •  stream is potentially infinite and ever-changing 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 15 Data Query Result Tuples Result Tuples Traditional Database CEP Engine Query Data
  16. 16. 2013 © Trivadis Oracle Event Processing – Visual Development Tools 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 16
  17. 17. 2013 © Trivadis Oracle Event Processing – Operation & Management 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 17
  18. 18. 2013 © Trivadis Implementation – complete picture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 18 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http:// bit.ly/1xafUZJ gschmutz #doag2014 #oep12 #bam12c,2 gschmutz,1 @doag2014,5 #oep12c,1 JMS JMSTwitter range 30 seconds
 slide 30 seconds Sentiment
 Enricher Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http://bit.ly/1xafUZJ 1 #bam12c
  19. 19. 2013 © Trivadis 1) Creating a Twitter Adapter 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 19 Twitter Adapter Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http:// bit.ly/1xafUZJ Twitter
  20. 20. 2013 © Trivadis 2) Send Tweets to BAM 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 20 Twitter Adapter BAM Tweet Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http:// bit.ly/1xafUZJ JMSTwitter Sentiment
 Enricher Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http://bit.ly/1xafUZJ 1
  21. 21. 2013 © Trivadis 3) Extract interesting information from Tweet 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 21 Mention Extractor Twitter Adapter Hashtag
 Extractor Author Extractor BAM Tweet Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http:// bit.ly/1xafUZJ gschmutz #doag2014 #oep12 JMSTwitter Sentiment
 Enricher Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http://bit.ly/1xafUZJ 1 #bam12c
  22. 22. 2013 © Trivadis 4) Count occurrences within period 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 22 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http:// bit.ly/1xafUZJ gschmutz #doag2014 #oep12 #bam12c,2 gschmutz,1 @doag2014,5 #oep12c,1 JMS JMSTwitter range 30 seconds
 slide 30 seconds Sentiment
 Enricher Working on my OEP & BAM demo for the session ”Echtzeitverarbeitung von Twitter Feeds …” #doag2014 #oep12c #bam12c http://bit.ly/1xafUZJ 1 #bam12c
  23. 23. 2013 © Trivadis Implementing in Oracle Event Processing 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 23 Mention Extractor Twitter Adapter Counter
 Processor Hashtag
 Extractor Author Extractor BAM Tweet BAM Counter JMS JMSTwitter range 30 seconds
 slide 30 seconds Sentiment
 Enricher
  24. 24. 2013 © Trivadis 1) Creating Twitter Adapter – Connecting to Twitter Stream 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 24
  25. 25. 2013 © Trivadis 1) Creating Twitter Adapter – Tweet Event 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 25
  26. 26. 2013 © Trivadis 1) Creating Twitter Adapter – Adapter Factory 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 26
  27. 27. 2013 © Trivadis 1) Creating Twitter Adapter – Assembly 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 27
  28. 28. 2013 © Trivadis 1) Creating Twitter Adapter – Export Adapter to server 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 28
  29. 29. 2013 © Trivadis 1) Creating Twitter Adapter – Using Twitter Adapter 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 29
  30. 30. 2013 © Trivadis 2) Sentiment Enrichment 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 30
  31. 31. 2013 © Trivadis 3) Sending Tweets to BAM Using Oracle BAM Enterprise Message Sources (JMS) interface 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 31
  32. 32. 2013 © Trivadis 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 32 3) Sending Tweets to BAM – Convert event to JMS MapMessage
  33. 33. 2013 © Trivadis 4) Extract information from Tweet – Extract Hashtags from TweetEvent 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 33
  34. 34. 2013 © Trivadis 4) Extract information from Tweet – Extract Hashtags from TweetEvent 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 34
  35. 35. 2013 © Trivadis 5) Count hashtag occurrences within period - Using CQL 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 35
  36. 36. 2013 © Trivadis Implementation – Complete Picture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 36
  37. 37. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Event Processing in Information Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 37
  38. 38. 2013 © Trivadis Oracle BAM: Architected for Integration and Visualization Processing Twitter Stream with Oracle Event Processing (OEP) 19.11.2014 38
  39. 39. 2013 © Trivadis Oracle BAM – Create a Data Object 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 39
  40. 40. 2013 © Trivadis Oracle BAM Enterprise Message Source Configuration 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 40
  41. 41. 2013 © Trivadis Oracle BAM Composer 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 41
  42. 42. 2013 © Trivadis AGENDA 1.  Introduction 2.  Twitter Use Case 3.  Processing with Oracle Event Processing (OEP) 4.  Visualization with Oracle Business Activity Monitoring (BAM) 5.  Event Processing in Information Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 42
  43. 43. 2013 © Trivadis Architectural Pattern: Standalone Event Stream Processing 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4343 Event Processing (ESP / CEP) State Store / Event Store EnterpriseEventBus (Ingress) Event Cloud Internet of Things Social Media Streams Enterprise EventBus 43 Analytical Applications DB Enterprise ServiceBus Business Rule Management SystemRules Event Processing Result Store
  44. 44. 2013 © Trivadis Hadoop Big Data Infrastructure Architectural Pattern: Event Stream Processing as part of Lambda Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4444 Event Processing (ESP / CEP) State Store / Event Store EnterpriseEventBus (Ingress) Event Cloud Internet of Things Social Media Streams Enterprise EventBus 44 Analytical Applications DB Enterprise ServiceBus Event Processing Map/ Reduce HDFS Result Store Result Store
  45. 45. 2013 © Trivadis Hadoop Big Data Infrastructure Architectural Pattern: Event Stream Processing as part of “Kappa” Architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 4545 Event Processing (ESP / CEP) State Store / Event Store EnterpriseEventBus (Ingress) Event Cloud Internet of Things Social Media Streams 45 Analytical Applications DB Enterprise ServiceBus Event Processing ReplayHDFS Result Store
  46. 46. 2013 © Trivadis Event Cloud Event Processing in modern architecture 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 46 Enterprise Applications WS REST JMS RDMBS LocalESB External Cloud Service Providers EnterpriseServiceBus(ESB) EJB Event Processing (ESP / CEP) State Store / Event Store BPM and SOA Platform Event REST Business Logic/Rules NoSQL Analytical Applications DataAnalytics Internet of Things Event Processing 46 Mobile Apps DB Rich (Web) Client Apps DB Social Media Streams EnterpriseEventBus (Ingress) EnterpriseEventBus Visualization Biz Logic Rules WS Event Business Rule
 Management SystemRules Processes ACM HumanWF Oracle Event Processing for Java Embedded Coherence Oracle NoSQL Business Activity Monitoring (BAM) Oracle RDBMS ServiceBus SOA Suite BPM Suite WebLogicJMS WebLogicJMSEDN Oracle Rules Oracle Event Processing (OEP) Oracle NoSQLOracle RDBMS Business Activity 
 Monitoring (BAM) APIGateway EDN Oracle Real-Time Decisions (RTD)
  47. 47. 2013 © Trivadis Questions and answers ... 2013 © Trivadis BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
 Guido Schmutz Technology Manager guido.schmutz@trivadis.com 19.11.2014 Processing Twitter Stream with Oracle Event Processing (OEP) 47

×