The document discusses structured streaming and its capabilities in handling parallelism and complexity for efficient computation using Apache Spark. It highlights the benefits of low latency and high throughput for processing large datasets and emphasizes production use cases related to security and performance monitoring. Additionally, it provides information on office hours for engineering support and mentions resource availability on Databricks' blog.

1. Insights Without Tradeoffs
Using Structured Streaming
Michael Armbrust - @michaelarmbrust
Spark Summit East 2017

2. Parallelism
Split up the problem
to harness many machines
for computation
Complexity
Handle cross-machine
communication and failures

3. DeveloperProductivity
Quickly and concisely express
common computations
Efficiency
Hand-tune code to minimize
overheads and process the
most data per cycle
SQL

4. Throughput
Process large historical
repositories quickly
Latency
Up-to-date answers
as new data arrives
Structured Streaming

5. Production Use Cases

6. Streaming at
Collect logs and metrics from a variety of
sources to ensure the security, availability
and performance of our cloud platform.

7. Engineer Office Hours
MY HOURS
• Spark Core
• R
• Data Science
• ML
• GraphFrames, Deep Learning
• Databricks
• Spark SQL
• StructuredStreaming
• Spark SQL
• StructuredStreaming
• Databricks
TODAY 4:30 – 5:15
OTHER ENGINEERS
TODAY 1:45 – 5:15 THURS 10:30 – 2:30
@ Databricks Booth

8. Thank you!
All code available at databricks.com/blog
and @michaelarmbrust