Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Event Driven Streaming Analytics - Demostration on Architecture of IoT

1,196 views

Published on

Demostration on Architecture of IoT

Published in: Data & Analytics

Event Driven Streaming Analytics - Demostration on Architecture of IoT

  1. 1. EVENT DRIVEN STREAMING ANALYTICS Demonstration on Architecture of IoT Lei Xu 2016.09
  2. 2. 1.DEMO SCENARIO
  3. 3. Taxi/Devices Coordinates & Payment Info Devices generate transactions after payment
  4. 4. Taxi/Devices Data Flow Coordinates & Payment Info IoT Gateway Applications & Services Data Management Cloud Infrastructure IoT Connectivity Platform Online Analytics • Dropoff-district Count during the last 30 min • Real-time Taxi Dropoff Events • 10 Most Frequent Routes(Pickup- area Dropoff-area) during the last 30 minutes • 10 Most Profitable areas within the last 15 minutes Offline Analytics • 10 Most Frequent Routes by week day • 10 Most Profitable Areas by week day • Most Profitable Hours at the 10 Most Profitable Areas by week day Transaction being sent to IoT Platform for Analytics Thousands of Browsers Hundreds of BI Clients
  5. 5. Data Management Multi-Channel Data Integration Structured & Unstructured Data Analytics & Machine Learning Cloud Infrastructure Cloud Native Platform Scalability & High Availability Security & Governance Visualize Data Reporting, BI & User Apps History Data & Real-time Data Multiple Devices & Platforms IoT Gateway Device Management Authentication & Authorization Gateways & Interfaces Apps & Services Rule Engine & Process Mgmt. Complex Event Processing Business & Data MicroServices IoT Platform Components
  6. 6. 2.REQUEST DRIVEN FLOW
  7. 7. Storing Enriched Data Time Based Enrichment & Filtering Receiving & Storing Raw Events Time Based Batch Aggregation Storing Aggregated Data Sending Raw Events Polling Requests for Latest Data Taxi/Devices Traditional Request Driven Flow Event-driven Flow Request-driven Flow Time-based Flow Analytics BI Visualization
  8. 8. DRAWBACKS & CHALLENGES • Polling + Request/Response Style => Overhead • Time Based Batch Process => High Latency • Architecture - Monolith or SOA will work well with HA? • How to scale & how to maintain system performance with lots of user requests? • How to scale & how to maintain system performance with huge amount of data? • Portable for both Cloud and On-Premise environments?
  9. 9. REQUEST DRIVEN & EVENT DRIVEN Request Driven Event Driven How is an action being taken? As a response to a specific request Triggered by the fact of a specific situation When is an action being taken? When the request is being processed Determined by the context of the situation What happens when the event/request occurs? A response is produced The event can be ignored, increment the state, trigger an internal derive event, or trigger a solution http://epthinking.blogspot.jp/2012/12/more-on-request-driven-vs-event-driven.html
  10. 10. 3.EVENT DRIVEN FLOW
  11. 11. Publishing/ Subscribing Enriched Events Enrichment & Filtering Receiving Raw Events Real-time Aggregation/ Analytics Publishing/ Subscribing Aggregated Events Sending Raw Events Events Visualization Online Analytics Taxi/Devices Expectations • Event Driven • Low Latency • Scalability • High Availability • Portability • Light Weight Event-driven Flow
  12. 12. Publishing/ Subscribing Enriched Events Enrichment & Filtering Receiving Raw Events Real-time Aggregation/ Analytics Publishing/ Subscribing Aggregated Events Sending Raw Events Events Visualization Online & Offline Analytics Storing Data for Analytics Analytics BI Visualization Taxi/Devices Event-driven Flow Request-driven Flow
  13. 13. Publishing/ Subscribing Enriched Events Enrichment & Filtering Receiving Raw Events Real-time Aggregation/ Analytics Publishing/ Subscribing Aggregated Events Sending Raw Events Events Visualization Machine Learning Loopback to Streaming Storing Data for Analytics Analytics & Machine Learning BI Visualization Taxi/Devices Event-driven Flow Request-driven Flow
  14. 14. 4.EVENT DRIVEN ARCHITECTURE
  15. 15. Binding Abstraction Layer Messaging Options Data Pipeline Data Microservices Data Spring Cloud Stream App Spring Cloud Stream App Spring Cloud Stream App Spring Cloud Stream/Data Flow as Event Driven Architecture Data Filter, Enrich, Transform, Aggregation, Rules, CEP, Analytics
  16. 16. Publishing/Subscribing Geocoded Events Reverse Geocoding Receiving Raw Events Real-time Aggregation/Analytics Publishing/Subscribing Aggregated Events Sending Raw Events Events Visualization Simulator HTTP Source MongoDB Processor GemFire Sink GemFire Source WebSocket Sink WebSocket Sink Reverse GeocodingReal-time Aggregation/Analytics Messaging Apps & Services – Online Analytics
  17. 17. Publishing/Subscribing Geocoded Events Reverse Geocoding Receiving Raw Events Real-time Aggregation/Analytics Publishing/Subscribing Aggregated Events Sending Raw Events Events Visualization Simulator HTTP Source MongoDB Processor GemFire Sink GemFire Source WebSocket Sink WebSocket Sink Reverse GeocodingReal-time Aggregation/Analytics Messaging BI Visualization Analytics & Machine Learning Storing Data for Analytics HDFS Sink SQL Query Apps & Services – Online & Offline Analytics MPP SQL on HDFS
  18. 18. Publishing/Subscribing Geocoded Events Reverse Geocoding Receiving Raw Events Real-time Aggregation/Analytics Publishing/Subscribing Aggregated Events Sending Raw Events Events Visualization Simulator HTTP Source MongoDB Processor GemFire Sink GemFire Source WebSocket Sink WebSocket Sink Reverse GeocodingReal-time Aggregation/Analytics Messaging BI Visualization Analytics & Machine Learning Storing Data for Analytics MPP SQL on HDFS HDFS Sink SQL Query Machine Learning Apps & Services – Machine Learning Loopback to Streaming Machine Learning
  19. 19. 5.ONLINE ANALYTICS LOCAL DEPLOYMENT
  20. 20. MongoDB Processor Real-time Aggregation/Analytics Spring Boot App Spring Cloud Stream App Spring Cloud Stream App HTTP Source GemFire Sink GemFire Source WebSocket Sink Simulator Spring Cloud Stream App Spring Cloud Stream App Spring Cloud Stream App Java CLI App Messaging WebSocket Sink Spring Cloud Stream App MVW App Data Flow Deployment Deployment Reverse Geocoding
  21. 21. MongoDB Processor Real-time Aggregation/Analytics Spring Boot App Spring Cloud Stream App Spring Cloud Stream App HTTP Source GemFire Sink GemFire Source WebSocket Sink Simulator Spring Cloud Stream App Spring Cloud Stream App Spring Cloud Stream App Java CLI App Messaging WebSocket Sink Spring Cloud Stream App MVW App Data Flow Deployment Scale Out When NecessaryReverse Geocoding
  22. 22. MongoDB Processor Real-time Aggregation/Analytics Spring Boot App Spring Cloud Stream App Spring Cloud Stream App HTTP Source GemFire Sink GemFire Source WebSocket Sink Simulator Spring Cloud Stream App Spring Cloud Stream App Spring Cloud Stream App Java CLI App Messaging WebSocket Sink Spring Cloud Stream App MVW App Data Flow Deployment Runtime Reverse Geocoding
  23. 23. 6. PRODUCTION LEVEL DEPLOYMENT
  24. 24. MQTT Source Enrichment Processor IMDG Sink Machine Learning Processor WebSocket Sink MPP/HDFS Sink Real-time IMDG Sink MPP/HDFS Source Data Pipelines IMDG Source Broker by RabbitMQ WebSocket by RabbitMQ Front-end Services Spring Cloud Stream Apps MVW App Spring Cloud Stream Apps Data MicroServicesBusiness MicroServices …… Containerized Applications Taxi/Devices • Event Driven Spring Cloud Stream • Low Latency Messaging by RabbitMQ Real-time by GemFire • Scalability Apps Scale-Out by Containers Governance by Data Flow Services Scale-out • High Availability Platform HA by Cloud Native Platform or PaaS • Portability Apps & Services portable to other Cloud Native Platforms • Light Weight Loosely Coupled MicroServices or or etc. Back-end Services RabbitMQGeode MongoDB HAWQ or Spring Cloud Data Flow
  25. 25. 7.DEPLOYMENT & RUNTIME SCREENSHOTS
  26. 26. • Real-time Taxi Dropoff Events • Dropoff-district Count
  27. 27. • 10 most frequent routes • 10 most frequent routes
  28. 28. 8. LOCAL DEPLOYMENT DEMOVIDEO

×