The document discusses insights from Michael Armbrust's presentation at Spark Summit East 2017 on structured streaming, emphasizing the importance of parallelism, developer productivity, and efficiency in data processing. It highlights structured streaming's ability to handle large datasets while providing up-to-date analytical answers and includes several production use cases. Additional resources and office hours information are shared for further engagement with the engineering community.

1. Insights Without
Tradeoffs Using
Structured Streaming
Michael Armbrust - @michaelarmbrust
Spark Summit East 2017

2. Parallelism
Split up the problem
to harness many
machines for computation
Complexity
Handle cross-machine
communication and failures

3. Developer Productivity
Quickly and concisely express
common computations
Efficiency
Hand-tune code to minimize
overheads and process the
most data per cycle
SQL

4. Throughput
Process large historical
repositories quickly
Latency
Up-to-date answers
as new data arrives
Structured Streaming

5. Production Use Cases

6. Streaming at
Collect logs and metrics from a variety of
sources to ensure the security,
availability and performance of our cloud
platform.

7. Engineer Office Hours
MY HOURS
• Spark Core
• R
• Data Science
• ML
• GraphFrames, Deep Learning
• Databricks
• Spark SQL
• Structured Streaming
• Spark SQL
• Structured Streaming
• Databricks
TODAY 4:30 –
5:15
OTHER ENGINEERS
TODAY 1:45 –
5:15
THURS 10:30 –
2:30
@ Databricks Booth

8. Thank you!
All code available at databricks.com/blog
and @michaelarmbrust