Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armbrust


Published on

In Spark 2.0, we introduced Structured Streaming, which allows users to continually and incrementally update your view of the world as new data arrives, while still using the same familiar Spark SQL abstractions. I talk about progress we’ve made since then on robustness, latency, expressiveness and observability, using examples of production end-to-end continuous applications.

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

Insights Without Tradeoffs Using Structured Streaming keynote by Michael Armbrust

  1. 1. Insights Without Tradeoffs Using Structured Streaming Michael Armbrust - @michaelarmbrust Spark Summit East 2017
  2. 2. Parallelism Split up the problem to harness many machines for computation Complexity Handle cross-machine communication and failures
  3. 3. Developer Productivity Quickly and concisely express common computations Efficiency Hand-tune code to minimize overheads and process the most data per cycle SQL
  4. 4. Throughput Process large historical repositories quickly Latency Up-to-date answers as new data arrives Structured Streaming
  5. 5. Production Use Cases
  6. 6. Streaming at Collect logs and metrics from a variety of sources to ensure the security, availability and performance of our cloud platform.
  7. 7. Engineer Office Hours MY HOURS • Spark Core • R • Data Science • ML • GraphFrames, Deep Learning • Databricks • Spark SQL • Structured Streaming • Spark SQL • Structured Streaming • Databricks TODAY 4:30 – 5:15 OTHER ENGINEERS TODAY 1:45 – 5:15 THURS 10:30 – 2:30 @ Databricks Booth
  8. 8. Thank you! All code available at and @michaelarmbrust