This document summarizes Marc Gonzalez's presentation on migrating batch jobs into structured streaming. The presentation covers streams and tables theory, including how bounded and unbounded tables relate to streams. It then demonstrates migrating a batch job that sums scores over windows to a structured streaming job in Spark, covering aspects like event vs processing time and watermarks. Finally, it discusses the Beam model and upcoming Apache Beam meetup.

1. MIGRATING
BATCH JOBS INTO
STRUCTURED STREAMING
Introduction to near real time analytics
MARCGONZALEZ.EU

2. Hi, I’m Marc!
Freelance Data Engineer
Developer, Consultant
(and now) speaker
5+ years of big data experience,
applied to classifieds market.

3. Q? sli.do #5910

4. Q? sli.do #5910

5. Q? sli.do #5910

6. Q? sli.do #5910

7. Q? sli.do #5910
Audience
• Experience with Dataframes.
• Experience with DStreams.
• Streams and tables theory.
• Beam model.

8. Q? sli.do #5910
Notes
• Most material is from Tyler Akidau,
either from his blog, talks or book.

9. Q? sli.do #5910
TALK STRUCTURE
Streams and tables theory.
Structured streaming migration demo.
Working with new developments.3
2
1
Q Q&A sli.do code: 5910

10. Q? sli.do #5910
TALK STRUCTURE
Streams and tables theory.
Structured streaming migration demo.
Working with new developments.3
2
1

11. Q? sli.do #5910
“Every Stream can yield a Table at a certain time,
& every Table can be observed into a Stream.”
1. Streams & Tables theory

12. Q? sli.do #5910
1. Streams & Tables theory:
Demonstration
Formally:
By example:
Stream =
∂Table
∂t
Table =
∫
now
t0
Stream
Underlying structure of
a Database system for
handling updates.
Change Data Capture
(CDC) in microservices.

13. Q? sli.do #5910
Operation
Stream !=> Stream Mapping
Stream !=> Table Grouping
Table !=> Stream Partitioning
This helps a lot with our migration right?
1. Streams & Tables theory:
General approach

14. Q? sli.do #5910
“Semantically batch is really just
a (strict) subset of streaming.”
1. Streams & Tables theory:
Batch & Streaming Engines

15. Q? sli.do #5910
1. Streams & Tables theory:
Bounded & Unbounded Tables
struct
!=>Insights
Unbounded tableData stream

16. Q? sli.do #5910
1. Streams & Tables theory:
Bounded & Unbounded Tables
So can we swap one with the other?
struct
!=> Insights
Data stream

17. YES*
*but you’re going to need:
Tools for reasoning about time
Guarantee Correctness

18. Q? sli.do #5910
1. Streams & Tables theory:
Tools for reasoning about time
• Event vs Processing Time
• Windowing
• Triggers

19. Q? sli.do #5910
1.1. Tools for reasoning about time:
Event vs Processing Times

20. Q? sli.do #5910
1.1. Tools for reasoning about time:
Event vs Processing Times Example

21. Q? sli.do #5910
1.1. Tools for reasoning about time:
Windowing
• Partitioning a data set along temporal boundaries.
Fixed Sliding Session
Event-Time

22. Q? sli.do #5910
1.1. Tools for reasoning about time:
2 Minute Windowing Example

23. Q? sli.do #5910
1.1. Tools for reasoning about time:
Triggers
• Mechanism for declaring when the output for a window should be
materialized (relative to some external signal).
• Per element
• Window completion
• Fixed

24. Q? sli.do #5910
1.1. Tools for reasoning about time:
2 Minute Triggers Example

25. Q? sli.do #5910
1. Streams & Tables theory:
Correctness
• State
• Watermarks
• Late data firing
• Exactly one

26. Q? sli.do #5910
1.2. Correctness:
State
• Amount of context stored between runs.

27. Q? sli.do #5910
1.2. Correctness:
Watermarks
• Watermarks are temporal notions of input completeness in the event-time
domain.

28. Q? sli.do #5910
1.2. Correctness:
Watermarks Example

29. Q? sli.do #5910
1.2. Correctness:
Handling late data
• Firing functions when events are observed outside the state.
Technique Side-effect
Discarding Approximate
Accumulation Duplicates
Accumulation
& Retraction
Late updates

30. Q? sli.do #5910
1.2. Correctness:
Discarding late data Example

31. Q? sli.do #5910
1.2. Correctness:
Exactly one
“Exactly one = At least one + Only one”

32. Q? sli.do #5910
1.2. Correctness:
At least one
• Checkpoints (HDFS compatible)
• Write-ahead log

33. Q? sli.do #5910
1.2. Correctness:
At least one
• Checkpoints relates to State:

34. Q? sli.do #5910
1.2. Correctness:
Only one
Technique Scope
Deduplication Micro-Batch
Deduplication
with Watermark
State
Deduplication
with Left Join
Resources

35. Q? sli.do #5910
Recap Part 1
• Processing of Bounded & Unbounded Tables.
• Event vs Processing time & how it relates to Windowing and Triggering.
• Stateful processing is useful when working to guarantee correctness.
• State is managed with Watermarks, Late Data firings & Fault Tolerant
Exactly One semantics.

36. Q? sli.do #5910
TALK STRUCTURE
Streams and tables theory.
Structured streaming migration demo.
Working with new developments.3
2
1

37. Q? sli.do #5910
import org.apache.spark.sql.functions._
def selectKafkaContent(df: DataFrame): DataFrame =
df.selectExpr("CAST(value AS STRING) as sValue")
def jsonScore(df: DataFrame): DataFrame =
df.selectExpr("CAST(get_json_object(sValue, '$.score') as INT) score")
def parse(df: DataFrame): DataFrame = jsonScore(selectKafkaContent(df))
def sumScores(df: DataFrame): DataFrame =
df.agg(sum("score").as("total"))
it should "sum 48 after consuming everything" in {
publishToMyKafka
kafka.getTopics().size shouldBe 1
val topicsAndOffsets = kafkaUtils.getTopicsAndOffsets("eu.marcgonzalez.demo")
topicsAndOffsets.foreach { topicAndOffset: TopicAndOffsets =>
val df = kafkaUtils
.load(topicAndOffset, kafkaConfiguration)
val jsonDf = df
.transform(parse)
.transform(sumScores)
jsonDf.collect()(0).get(0) shouldBe 48
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
2.Structured streaming migration demo:
Batch Example
• kafkaUtils handles offsets
• parse is a
stream->stream mapping op
• sumScores is a
stream->table grouping op

38. Q? sli.do #5910
def jsonScoreAndDate(df: DataFrame): DataFrame =
df.selectExpr(
"from_json(sValue, 'score INT, eventTime LONG, delayInMin INT') struct",
"timestamp as procTime")
.select(col("struct.*"), 'procTime)
.selectExpr("timestamp(eventTime/1000) as eventTime", "score", "procTime")
def parse(df: DataFrame): DataFrame = {
jsonScoreAndDate(selectKafkaContent(df))
}
def windowedSumScores(df: DataFrame): DataFrame =
df.groupBy(window($"eventTime", "2 minutes")).agg(sum("score").as("total"))
it should "sum 14, 18, 4, 12 after consuming everything in 2 minute windows" in {
val topicsAndOffsets = kafkaUtils.getTopicsAndOffsets("eu.marcgonzalez.demo")
topicsAndOffsets.foreach { topicAndOffset: TopicAndOffsets =>
val df = kafkaUtils
.load(topicAndOffset, kafkaConfiguration)
val jsonDf = df
.transform(parse)
.transform(windowedSumScores)
jsonDf
.sort("window").collect()
.foldLeft(Seq.empty[Int])(
(a, v) => a ++ Seq(v.get(1).asInstanceOf[Long].toInt)
) shouldBe Seq(14, 18, 4, 12)
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
2.Structured streaming migration demo:
Windowed Batch
• Extract eventTime from JSON
• Always try to enforce a
schema (JSON, AVRO)
• Fixed window partitions our
insight

39. Q? sli.do #5910
it should "sum 14,18,4,12 after streaming everything in 2 minute windows" in {
val topicsAndOffsets = kafkaUtils.getTopicsAndOffsets("eu.marcgonzalez.demo")
topicsAndOffsets.foreach { topicAndOffset: TopicAndOffsets =>
val df = spark.readStream
.format(“kafka")
.option(“kafka.bootstrap.servers", "localhost:9092")
.option("startingOffsets", "earliest")
.option("subscribe", topicAndOffset.topic)
.load()
df.isStreaming shouldBe true
val jsonDf = df
.transform(parse)
.transform(windowedSumScores)
val query = jsonDf.writeStream
.outputMode("update") //complete
.format("memory") //console
.queryName(queryName)
.trigger(Trigger.ProcessingTime("5 seconds")) //Once
.start()
query.awaitTermination(10 * 1000)
spark.sql(s"select * from $queryName order by window asc")
.collect()
.foldLeft(Seq.empty[Int])(
(a, v) => a ++ Seq(v.get(1).asInstanceOf[Long].toInt)
) shouldBe Seq(14, 18, 4, 12)
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
2.Structured streaming migration demo:
Windowed Stream Example
• read -> readStream, allowing S.S.S.
to take control over the input
offsets.
• Our dataframe is now a streaming
dataframe, & supports all previous
ops.
• write -> writeStream
• Modes: Complete, Update &
Append
• Sinks: Kafka, File, forEach,
memory & console for debug.
• Queryable state -> Eventual
Consistency

40. Q? sli.do #5910
it should "sum 5,18,4,12 after streaming everything in 2 minute windows" in {
timelyPublishToMyKafka
val topicsAndOffsets = kafkaUtils.getTopicsAndOffsets("eu.marcgonzalez.demo")
topicsAndOffsets.foreach { topicAndOffset: TopicAndOffsets =>
// Same reader as previous
val jsonDf = df
.transform(parse)
.withWatermark("eventTime", "2 minutes")
.transform(windowedSumScores)
val query = jsonDf
.writeStream
.outputMode(“update") //append
.format(“memory")
.queryName(queryName)
.trigger(Trigger.ProcessingTime("5 seconds”))
.start()
query.awaitTermination(15 * SECONDS_MS)
spark.sql(s"select window, max(total) from $queryName
group by window order by window asc")
.collect()
.foldLeft(Seq.empty[Int])(
(a, v) => a ++ Seq(v.get(1).asInstanceOf[Long].toInt)
) shouldBe Seq(5, 18, 4, 12)
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
2.Structured streaming migration demo:
Windowed Stream + Watermark Example
• Introduce a delay when
sending to kafka.
• We add a 2 minute watermark
• Event on a closed window gets
discarded!
• Append mode only sends
closed windows.

41. Q? sli.do #5910
Recap Part 2
• Reuse the same DF transformations
• Change read by readStream & write by writeStream
• Fixed triggers + start()
• Choosing the right Sinks + Output modes are key.
• State is managed through Spark Checkpoints. Do not mess with it!

42. Q? sli.do #5910
TALK STRUCTURE
Streams and tables theory.
Structured streaming migration demo.
Working with new developments.3
2
1

43. Q? sli.do #5910
Beam model
• What results are calculated?
• Where in event time are results calculated?
• When in processing time are results materialized?
• How do refinements of results relate?

44. Q? sli.do #5910
Beam model
• What results are calculated? Insights
• Where in event time are results calculated?
• When in processing time are results materialized?
• How do refinements of results relate?

45. Q? sli.do #5910
Beam model
• What results are calculated? Insights
• Where in event time are results calculated? Windowing
• When in processing time are results materialized?
• How do refinements of results relate?

46. Q? sli.do #5910
Beam model
• What results are calculated? Insights
• Where in event time are results calculated? Windowing
• When in processing time are results materialized? Triggers & Watermarks
• How do refinements of results relate?

47. Q? sli.do #5910
Beam model
• What results are calculated? Insights
• Where in event time are results calculated? Windowing
• When in processing time are results materialized? Triggers & Watermarks
• How do refinements of results relate? Late Data Firings, Exactly Once

48. Q? sli.do #5910
Apache Beam
• Unified model
• Multiples languages
• Multiples runners

49. Q? sli.do #5910
Runners Comparison

50. Q? sli.do #5910
Upcoming Meetup!
January 29th @ 7pm
🍕🍺 & amazing terrace
https:!//!!www.meetup.com/
Barcelona-Apache-Beam-
Meetup

51. Q? sli.do #5910
Thank you!
sli.do