Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Interactive Visualization of Streaming Data Powered by Spark

996 views

Published on

Spark Summit 2016 talk by Ruhollah Farchtchi (Zoomdata)

Published in: Data & Analytics
  • Be the first to comment

Interactive Visualization of Streaming Data Powered by Spark

  1. 1. INTERACTIVE VISUALIZATION OF STREAMING DATA POWERED BY SPARK
  2. 2. Streaming @Zoomdata Visualizations react to new data delivered Users start, stop, pause the stream Users select a rolling window or pin a start time to capture cumulative metrics
  3. 3. Drivers for Streaming Data Data Freshness Time to Analytic Business Context
  4. 4. Challenges • Time • Frequency • Retention • Synchronization • Order • Updates
  5. 5. Addressing streaming @Zoomdata Historical Revised Receive Data JMS Kafka Manipulate Stream Single JVM in Memory Spark Streaming Hold Data in Buffer MongoDB Pluggable Interact with Data Custom Code Pluggable
  6. 6. Technology Cast • The Stream - Kafka, Kinesis, JMS • Processing Fabric - Spark Streaming • Landing Area - MemSQL, Solr, Kudu, Others
  7. 7. How it looks
  8. 8. With the rest of the app
  9. 9. Scale Out
  10. 10. Let’s put it all together • Process fruit orders from all over • Buyers make orders via an API • API is SMS – [fruit] [quantity] e.g., apple 10
  11. 11. Demo
  12. 12. Benefits • Contextual Expressiveness with Streaming Data • Independent scalability (scale-up, scale- around) • Expressiveness powered by Spark -- using Windowing (dataframe API with stream) • DR COOP, other Data management concerns
  13. 13. Future Work • Cross stream synchronization & fusion • On-demand scale out and resource management via Mesos • Schema evolution • More extensible landing strategies
  14. 14. Questions For more information contact: ruhollah@zoomdata.com Come visit Zoomdata at our booth H2!
  15. 15. Thank You

×