Downloaded 33 times
![What is Spark and Why we Should Care
Streaming APIs in Spark
- Structured Streaming Overview
- Interactive Session 1
- Spark Streaming Overview
- Interactive Session 2
Spark Streaming [AND|OR|XOR] Structured Streaming](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-8-2048.jpg)
![Operations
...
val rawValues = rawData.selectExpr("CAST(value AS STRING)").as[String]
val jsonValues = rawValues.select(from_json($"value", schema) as "record")
val sensorData = jsonValues.select("record.*").as[SensorData]
…](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-17-2048.jpg)
![DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
t0 t1 t2 t3 ti ti+1
RDD[U] RDD[U] RDD[U] RDD[U] RDD[U]
Transformation
T -> U
Actions](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-23-2048.jpg)
![val kafkaParams = Map[String, String](
"metadata.broker.list" -> kafkaBootstrapServer,
"group.id" -> "sensor-tracker-group",
"auto.offset.reset" -> "largest",
"enable.auto.commit" -> (false: java.lang.Boolean).toString
)
val topics = Set(topic)
@transient val stream = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](
streamingContext, kafkaParams, topics)
Source](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-32-2048.jpg)
![import spark.implicits._
val sensorDataStream = stream.transform{rdd =>
val jsonData = rdd.map{case (k,v) => v}
val ds = sparkSession.createDataset(jsonData)
val jsonDF = spark.read.json(ds)
val sensorDataDS = jsonDF.as[SensorData]
sensorDataDS.rdd
}
Transformations](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-33-2048.jpg)
![Usecases
● Stream-stream joins
● Complex state management (local + cluster state)
● Streaming Machine Learning
○ Learn
○ Score
● Join Streams with Updatable Datasets
● [-] Event-time oriented analytics
● [-] Continuous processing](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-36-2048.jpg)
![Spark Streaming + Structured Streaming
38
val parse: Dataset[String] => Dataset[Record] = ???
val process: Dataset[Record] => Dataset[Result] = ???
val serialize: Dataset[Result] => Dataset[String] = ???
val kafkaStream = spark.readStream…
val f = parse andThen process andThen serialize
val result = f(kafkaStream)
result.writeStream
.format("kafka")
.option("kafka.bootstrap.servers",bootstrapServers)
.option("topic", writeTopic)
.option("checkpointLocation", checkpointLocation)
.start()
val dstream = KafkaUtils.createDirectStream(...)
dstream.map{rdd =>
val ds = sparkSession.createDataset(rdd)
val f = parse andThen process andThen serialize
val result = f(ds)
result.write.format("kafka")
.option("kafka.bootstrap.servers", bootstrapServers)
.option("topic", writeTopic)
.option("checkpointLocation", checkpointLocation)
.save()
}
Structured StreamingSpark Streaming](https://image.slidesharecdn.com/reducedversionwebinarstreaming-with-spark1-171201144525/75/A-Tale-of-Two-APIs-Using-Spark-Streaming-In-Production-38-2048.jpg)
Uploaded byLightbend
A Tale of Two APIs: Using Spark Streaming In Production
The document presents a detailed overview of streaming data processing using Apache Spark, focusing on structured streaming and its various components such as Kafka and machine learning. It highlights practical use cases, code examples, and the integration of different data sources and operations. The author, Gerard Maas, is an experienced software engineer and an active contributor to the Scala and Spark communities.
More Related Content
Similar to A Tale of Two APIs: Using Spark Streaming In Production
A Tale of Two APIs: Using Spark Streaming In Production
- 2. Gerard Maas Señor SWEngineer Computer Engineer Scala Developer Early Spark Adopter (v0.9) Cassandra MVP (2015, 2016) Stack Overflow Top Contributor (Spark, Spark Streaming, Scala) Wannabe { IoT Maker Drone crasher/tinkerer } @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 4. Streaming | BigData
- 5.
- 6.
- 7. ∑ Stream =Dataset Dataset = Stream
- 8. What is Sparkand Why we Should Care Streaming APIs in Spark - Structured Streaming Overview - Interactive Session 1 - Spark Streaming Overview - Interactive Session 2 Spark Streaming [AND|OR|XOR] Structured Streaming
- 9.
- 10. Apache Spark Core SparkSQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources
- 11. Apache Spark Core SparkSQL SparkMLLib SparkStreaming Structured Streaming DataFrames DataSets GraphFrames Data Sources
- 12.
- 13.
- 14. Sensor Data Producer Structured Streaming Fast Data Platform SparkNotebook Local Process Demo Scenario
- 15.
- 16. val rawData =sparkSession.readStream .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("subscribe", sourceTopic) .option("startingOffsets", "latest") .load() Sources
- 17. Operations ... val rawValues =rawData.selectExpr("CAST(value AS STRING)").as[String] val jsonValues = rawValues.select(from_json($"value", schema) as "record") val sensorData = jsonValues.select("record.*").as[SensorData] …
- 18. Event Time ... val movingAverage= sensorData.withColumn("timestamp", toSeconds($"ts").cast(TimestampType)) .withWatermark("timestamp", "30 seconds") .groupBy($"id", window($"timestamp", "30 seconds", "10 seconds")) .agg(avg($"temp")) ...
- 19. Sinks... val visualizationQuery =sensorData.writeStream .queryName("visualization") // this query name will be the SQL table name .outputMode("update") .format("memory") .start() ... val kafkaWriterQuery = kafkaFormat.writeStream .queryName("kafkaWriter") // this query name will be the table name .outputMode("append") .format("kafka") .option("kafka.bootstrap.servers", kafkaBootstrapServer) .option("topic", targetTopic) .option("checkpointLocation", "/tmp/spark/checkpoint") .start()
- 20. Use Cases ● StreamingETL ● Stream aggregations, windows ● Event-time oriented analytics ● Join Streams with Fixed Datasets ● Apply Machine Learning Models
- 21.
- 22. Spark Streaming Kafka Flume Kinesis Twitter Sockets HDFS/S3 Custom ApacheSpark SparkSQL SparkML ... Databases HDFS API Server Streams
- 23. DStream[T] RDD[T] RDD[T] RDD[T]RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformation T -> U Actions
- 24.
- 25.
- 26. API: Transformations transform val iotDstream= MQTTUtils.createStream(...) val devicePriority = sparkContext.cassandraTable(...) val prioritizedDStream = iotDstream.transform{rdd => rdd.map(d => (d.id, d)).join(devicePriority) }
- 27.
- 28.
- 29. Sensor Data Producer Structured Streaming Fast Data Platform SparkNotebook Local Process Demo Scenario
- 30.
- 31. import org.apache.spark.streaming.StreamingContext val streamingContext= new StreamingContext(sparkContext, interval) Streaming Context
- 32. val kafkaParams =Map[String, String]( "metadata.broker.list" -> kafkaBootstrapServer, "group.id" -> "sensor-tracker-group", "auto.offset.reset" -> "largest", "enable.auto.commit" -> (false: java.lang.Boolean).toString ) val topics = Set(topic) @transient val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder]( streamingContext, kafkaParams, topics) Source
- 33. import spark.implicits._ val sensorDataStream= stream.transform{rdd => val jsonData = rdd.map{case (k,v) => v} val ds = sparkSession.createDataset(jsonData) val jsonDF = spark.read.json(ds) val sensorDataDS = jsonDF.as[SensorData] sensorDataDS.rdd } Transformations
- 34. val model =new M2Model() … model.trainOn(inputData) … val scoredDStream = model.predictOnValues(inputData) Model
- 35. suspects.foreachRDD{rdd => val sample= rdd.take(20).map(_.toString) val total = s"total found: ${rdd.count}" outputBox(total +: sample) } Output
- 36. Usecases ● Stream-stream joins ●Complex state management (local + cluster state) ● Streaming Machine Learning ○ Learn ○ Score ● Join Streams with Updatable Datasets ● [-] Event-time oriented analytics ● [-] Continuous processing
- 37.
- 38. Spark Streaming +Structured Streaming 38 val parse: Dataset[String] => Dataset[Record] = ??? val process: Dataset[Record] => Dataset[Result] = ??? val serialize: Dataset[Result] => Dataset[String] = ??? val kafkaStream = spark.readStream… val f = parse andThen process andThen serialize val result = f(kafkaStream) result.writeStream .format("kafka") .option("kafka.bootstrap.servers",bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .start() val dstream = KafkaUtils.createDirectStream(...) dstream.map{rdd => val ds = sparkSession.createDataset(rdd) val f = parse andThen process andThen serialize val result = f(ds) result.write.format("kafka") .option("kafka.bootstrap.servers", bootstrapServers) .option("topic", writeTopic) .option("checkpointLocation", checkpointLocation) .save() } Structured StreamingSpark Streaming
- 39. Streaming Pipelines (example) StructuredStreaming Keyword Extraction Keyword Relevance Similarity DB Storage
- 40.
- 41.
- 42. Features 1. One-click componentinstallations 2. Automatic dependency checks 3. One-click access to install logs 4. Real-time cluster visualization 5. Access to consolidated production logs Benefits: 1. Easy to get started 2. Ready access to all components 3. Increased developer velocity Fast Data Platform Manager, for Managing Running Clusters
- 43.
- 44. If you’re seriousabout having end-to-end monitoring for your Fast Data and streaming applications, let’s chat! SET UP A 20-MIN DEMO