Download as PDF, PPTX
![@maasg
Structured Streaming
val ctx= new StreamingContext(conf,Seconds(1))
val lines = ssc.socketTextStream ("localhost" , 9999)
val words = lines.flatMap(_.split(" "))
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)
wordCounts.print()
ctx.start()
https://spark.apache.org/docs/latest/streaming-programming-guide.html
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
val lines = spark.readStream
.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
val words= lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()
val query = wordCounts.writeStream
.outputMode("complete")
.format("console")
.start()](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-10-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-13-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-17-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-18-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-19-320.jpg)
![@maasg
Sensor
Data
Multiplexer
Data Exploration
[Structured Streaming]
Sensor Anomaly Detection Pipeline
Data Preparation
[Structured Streaming]
Online Model
Creation +
Training
[Spark Streaming]
Anomaly Detection
[Structured Streaming]
Kafka Source
Memory Sink
SQL Operations
Event Time
Windows
Watermark
Kafka Sink
RDD Programming
Local vs Distributed
Use Spark SQL
Kafka Source + Sink
Controlled Streams:
Spark Streaming +
Structured Streaming](https://image.slidesharecdn.com/webinar-streaming-with-spark1-181207125041/85/Hands-On-With-Spark-Creating-A-Fast-Data-Pipeline-With-Structured-Streaming-And-Spark-Streaming-20-320.jpg)
Uploaded byLightbend
Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming
The document discusses creating a fast data pipeline using Apache Spark's Structured Streaming and Spark Streaming. It presents a sensor anomaly detection pipeline that uses Structured Streaming for data exploration, preparation, and anomaly detection, and Spark Streaming for online model creation and training. It compares the execution and abstraction models of Structured Streaming and Spark Streaming, and demonstrates how to build the sensor anomaly detection pipeline using Kafka sources and sinks with SQL operations, event time windows, and watermarks.
More Related Content
Similar to Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming
Hands On With Spark: Creating A Fast Data Pipeline With Structured Streaming And Spark Streaming
- 1. Hands-on with ApacheSpark: Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming Gerard Maas Senior SW Engineer, Lightbend, Inc.
- 2. Gerard Maas Señor SWEngineer @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 3. @maasg Agenda Creating a FastData Pipeline with Structured Streaming and Spark Streaming
- 4.
- 5.
- 6. @maasg Data Pipelines ● CreateComposable Streaming Applications ● Using the Best Tool for the Job ● Generating a Network Effect
- 7. @maasg Agenda Creating a FastData Pipeline with Structured Streaming and Spark Streaming
- 8. @maasg Apache Spark Core SparkSQL SparkMLLib SparkStreaming Structured Streaming Datasets/Frames GraphFrames Data Sources
- 9. @maasg Apache Spark Core SparkSQL SparkMLLib SparkStreaming StructuredStreaming Datasets/Frames GraphFrames Data Sources RDD DStream Query
- 10. @maasg Structured Streaming val ctx=new StreamingContext(conf,Seconds(1)) val lines = ssc.socketTextStream ("localhost" , 9999) val words = lines.flatMap(_.split(" ")) val pairs = words.map(word => (word, 1)) val wordCounts = pairs.reduceByKey(_ + _) wordCounts.print() ctx.start() https://spark.apache.org/docs/latest/streaming-programming-guide.html https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html val lines = spark.readStream .format("socket") .option("host", "localhost") .option("port", 9999) .load() val words= lines.as[String].flatMap(_.split(" ")) val wordCounts = words.groupBy("value").count() val query = wordCounts.writeStream .outputMode("complete") .format("console") .start()
- 11. @maasg Time Execution Abstraction Structured Streaming SparkStreaming Abstract (Processing Time, Event Time) Fixed to microbatch Streaming Interval Fixed Micro batch, Best Effort MB, Continuous (NRT) Fixed Micro batch DataFrames/Dataset DStream, RDD Access to the scheduler
- 12. @maasg Agenda Hands On withSpark: Creating a Fast Data Pipeline with Structured Streaming and Spark Streaming
- 13. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] SensorAnomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming]
- 14. @maasg Sensor Data Multiplexer Structured StreamingLocal Process Sensor AnomalyDetection Structured Streaming
- 15. @maasg Sensor Data Multiplexer Structured StreamingLocal Process Sensor AnomalyDetection Structured Streaming
- 16.
- 17. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] SensorAnomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations
- 18. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] SensorAnomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink
- 19. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] SensorAnomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink RDD Programming Local vs Distributed Use Spark SQL Kafka Source + Sink
- 20. @maasg Sensor Data Multiplexer Data Exploration [Structured Streaming] SensorAnomaly Detection Pipeline Data Preparation [Structured Streaming] Online Model Creation + Training [Spark Streaming] Anomaly Detection [Structured Streaming] Kafka Source Memory Sink SQL Operations Event Time Windows Watermark Kafka Sink RDD Programming Local vs Distributed Use Spark SQL Kafka Source + Sink Controlled Streams: Spark Streaming + Structured Streaming
- 21. @maasg Time Execution Abstraction Structured Streaming SparkStreaming Abstract (Processing Time, Event Time) Fixed to microbatch Streaming Interval Fixed Micro batch, Best Effort MB, Continuous (NRT) Fixed Micro batch DataFrames/Dataset DStream, RDD Access to the scheduler
- 22. Resources Notebooks used today: https://github.com/maasg/spark-notebooks/tree/master/streaming-anomaly-detection Pipelines: https://www.reactivesummit.org/2018/schedule/taking-the-pain-out-of-deploying-streaming-applic ations StructuredStreaming + Spark Streaming: https://www.reactivesummit.org/2018/schedule/processing-fast-data-with-apache-spark-the-tale-of -two-streaming-apis Fast Data: https://www.lightbend.com/products/fast-data-platform
- 23. Gerard Maas Señor SWEngineer @maasg https://github.com/maasg https://www.linkedin.com/ in/gerardmaas/ https://stackoverflow.com /users/764040/maasg
- 24.
- 25.