Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Lego-like building blocks of Storm and Spark Streaming Pipelines

619 views

Published on

Lego-like building blocks of Storm and Spark Streaming Pipelines

Published in: Technology
  • Be the first to comment

Lego-like building blocks of Storm and Spark Streaming Pipelines

  1. 1. Lego-Like Building Blocks of Storm and Spark Streaming Pipelines For Rapid IOT and Streaming Analytics App Development Speakers: Anand Venugopal, Punit Shah
  2. 2. Approach to this presentation • Sharing our learnings and best practices from various Streaming Implementations • Fairly simple concept - certainly not rocket science – but we do hope there may be some interesting ideas for you. • Illustrating using a specific tool but you are free to implement the same concepts anyway you like
  3. 3. IOT and Streaming Analytics is HOT 30-50B Devices USD 661.74 Billion
  4. 4. Use Cases for Streaming Analytics • Store, Warehouse operations – Retail • Predictive Maintenance – Manufacturing, Oil & Gas • Clinical Care and Patient Management – Healthcare - Clinical • Sensor Analytics – IOT, Manufacturing, Others • Fleet Operations – Transportation, Logistics • Fraud and Anomaly Detection – IT Security, Financial Services • Gaming Analytics – Entertainment, Gaming • Churn Analytics – Telecom, Banking, Retail • Network Traffic Analysis and Optimization – Telco • Internet Advertising – Retail, e-commerce V E R T I C AL S
  5. 5. Use Cases for Streaming Analytics H O R I Z O N TAL S • Customer Experience • Clickstream Analytics • Context-sensitive Offers And Recommendations • IT Log Analytics • Security • Business Activity Monitoring
  6. 6. Use Cases for Streaming Analytics C O M B O • Internet of Things • Mobile App Analytics • Call Center Monitoring and Analytics
  7. 7. Adoption Pattern of IOT and Streaming Analytics Department 1 Department 2 Department 3Department 4
  8. 8. Adoption Pattern of IOT and Streaming Analytics Department 1 Department 2 Department 3Department 4
  9. 9. With Scale – we need a centralized efficient approach Department 1 Department 2 Department 3Department 4 CENTRALIZED APPROACH • Unified multi-tenant visual platform • Collaborative re-use of components
  10. 10. Three levels of re-use Functions E.g. ETL functions (Date/ String/ Object/ Integer manipulations) Operators E.g. Kafka Channel, Write_to_HDFS, Time-based aggregation; Pipelines i.e. Highest level of abstraction – lego-like building blocks
  11. 11. Re-usable stream processing patterns as pipelines Ingest – Pre-processing, Cleanup, De-duplication, re-sequencing; Filters, Classification/ Routing - Pass on instantly to different downstream subscribers – Instant anomaly detection – Security breaches / Fraud/ Costly failure scenarios Rules based alerting - Customer setup rules- Notifications and triggers Enrichment – Get key fields from the stream – dip into one or more Master DBs; create aggregate record Time Window calculations - counters, statistics Visualization block of raw and derived data Data storage – a) Batch up and write data into HDFS/ HBASE etc. b) Instantly write data into an indexing store Specific predictive model blocks
  12. 12. Connect pre-built pipelines to build an app Ingest/ Filter/ Classify Anomaly Detection Alerting Action Triggers Index, Visualize Time Window Statistics Persist, Visualize Low Latency Engine Low Latency Engine Low Latency Engine Micro-batch engine Micro-batch engine Rapid Development Best Engine for the task Dynamic Routing
  13. 13. A-B Testing, Champion Challenger, Hot Swap Ingest/ Filter/ Classify MODEL 1 MODEL 2 UI CONFIGURABLE DYNAMIC ROUTING RULES
  14. 14. DEMO
  15. 15. Thank you inquiry@streamanalytix.com

×