Gerard Maas, profile picture
Uploaded byGerard Maas
1,873 views

Dive into Spark Streaming

The document discusses Spark Streaming and its execution model. It describes how Spark Streaming receives data streams, divides them into micro-batches, and processes the micro-batches using Spark. It also covers transformations and actions that can be performed on DStreams, window operations, and deployment options for Spark Streaming including local, standalone, and cluster modes.

Embed presentation

Downloaded 146 times
@maasg www. .com by Gerard Maas Data Processing Team Lead gerard.maas@virdata.com @maasg Dive into
@maasg www. .com Virdata: A ‘born on the cloud’ IoT Managed Services & Analytics Platform
@maasg www. .com A ‘born on the cloud’ IoT Managed Services & Analytics Platform 2012 2013 2014 affiliate certified
@maasg www. .com Kafka PUB / SUB MQTT / WebSockets RAW Storage >>> Storage` ` ` >>> <<< >>> Query >>> >>> Virdata’s Stack
@maasg www. .com PUB / SUB MQTT / WebSockets RAW Storage Storage ` ` ` Query Notebook Server How Spark is Driving a New Loosely-coupled Stand-alone Service Virdata’s Full StackVirdata’s Spark as a Service
@maasg www. .com 100TB 5MB
@maasg www. .com 100TB 5MB/second
@maasg www. .com Agenda What is Spark Streaming? Programming Model Demo 1 Execution Model Demo 2 Resources Q/A
@maasg www. .com Apache Spark
@maasg www. .com Spark Streaming Scalable, fault-tolerant stream processing system Kafka Flume Kinesis Twitter Sockets HDFS/S3 Databases HDFS Server APPLICATIONS Custom Streams
@maasg www. .com Spark: RDD Operations transformatio n INPUT DATA HDFS TEXT/ Sequence File RDD SparkContext RDD OUTPUT Data HDFS TEXT/ Sequence File Cassandra Lazy evaluation action
@maasg www. .com Spark Streaming: DStreams transformatio n Receiver RDD RDD Lazy evaluation action Stream
@maasg www. .com Spark Streaming
@maasg www. .com DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 Spark Streaming
@maasg www. .com DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformations Spark Streaming
@maasg www. .com DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Actions Transformations Spark Streaming
@maasg www. .com Transformations map, flatmap, filter count, reduce, countByValue, reduceByKey n union, join cogroup
@maasg www. .com Transformations transform val iotDstream = MQTTUtils.createStream(...) val devicePriority = sparkContext.cassandraTable(...) val prioritizedDStream = iotDstream.transform{rdd => rdd.map(d => (d.id, d)).join(devicePriority) }
@maasg www. .com Transformations updateStateByKey … …
@maasg www. .com Transformations trackStateByKey … … SPARK 1.6
@maasg www. .com Actions print ------------------------------------------- Time: 1459875469000 ms ------------------------------------------- data1 data2 saveAsTextFiles, saveAsObjectFiles, saveAsHadoopFiles xxx yyy zzz foreachRDD *
@maasg www. .com Actions - foreachRDD dstream.foreachRDD{rdd => } Dataframes Spark SQL MLLib GraphX Databases ...
@maasg www. .com Actions - foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) val conn = DB.connect(server) asRecords.foreachPartition{partition => partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) }
@maasg www. .com Actions - foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) val conn = DB.connect(server) asRecords.foreachPartition{partition => partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) } Executes on the Driver Executes on the Workers
@maasg www. .com Actions - foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) asRecords.foreachPartition{partition => val conn = DB.connect(server) partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) } Executes on the Driver Executes on the Workers
@maasg www. .com Windows - Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14
@maasg www. .com Windows - Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 4,5,6,7,8,9
@maasg www. .com Windows - Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 4,5,6,7,8,9 7,8,9,10,11,12
@maasg www. .com Windows - Non-Overlapping 1 t dstream.window(windowLength = 6, slideInterval = 6) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 7,8,9,10,11,12
@maasg www. .com Windows - Operations window, countByWindow, reduceByWindow, reduceByKeyAndWindow, countByValueAndWindow
@maasg www. .com Windows - Inverse Function Optimization reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval,[numTasks]) 1 2 3 4 5 6 7 8 9 1,2,3,4,5,6 + + -
@maasg www. .com Windows- Inverse Function Optimization reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval,[numTasks]) 1 2 3 4 5 6 7 8 9 1,2,3,4,5,6 4,5,6,7,8,9 - + - + 7 8 9 1 2 3
@maasg www. .com Top 3 Starter Issues Serialization Where is my closure executing Do I have enough cores?
@maasg www. .com Demo 1 Anatomy of an Spark Streaming Application
@maasg www. .com Tweet few keywords about your interests and experience. Use hashtag “#sparkbe” Scala Spark Streaming DistributedSystems #sparkbe
Ready to dive in?
@maasg www. .com Deployment Options M Local Standalone Cluster WW W Using a Cluster Manager W spark.master=local[*] spark.master=spark://host:port spark.master=mesos://host:port M M D W D W W
@maasg www. .com Deployment Options M Local Standalone Cluster Using a Cluster Manager spark.master=local[*] spark.master=mesos://host:port M M DD Rec Exec Rec Exec Exec Exec Rec Exec Rec Exec ExecExec spark.master=spark://host:port
@maasg www. .com Streaming Spark t0 t1 t2 #0 Consumer Consumer Consumer Scheduling
@maasg www. .com Streaming Spark t0 t1 t2 #1 Consumer Consumer Consumer #0 Process Time < Batch Interval Scheduling
@maasg www. .com Streaming Spark t0 t1 t2 #2 Consumer Consumer Consumer #0 #1 #3 Scheduling Delay Scheduling
@maasg www. .com From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
@maasg www. .com #0 RDD Partitions Spark Spark Executors Spark Streaming From Streams to μbatches
@maasg www. .com #0 RDD Spark Spark Executors Spark Streaming From Streams to μbatches
@maasg www. .com #0 RDD Spark Spark Executors Spark Streaming From Streams to μbatches
@maasg www. .com #0 RDD Spark Spark Executors Spark Streaming From Streams to μbatches
@maasg www. .com From Streams to μbatches
@maasg www. .com From Streams to μbatches
@maasg www. .com Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval From Streams to μbatches
@maasg www. .com Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark spark.streaming.blockInterval = batchInterval x receivers / (partitionFactor x sparkCores) From Streams to μbatches
@maasg www. .com The Importance of Caching dstream.foreachRDD { rdd => rdd.cache() // cache the RDD before iterating! keys.foreach{ key => rdd.filter(elem=> key(elem) == key).saveAsFooBar(...) } rdd.unpersist() }
@maasg www. .com The Receiver model spark.streaming.receiver.maxRate Fault tolerance ? WAL
@maasg www. .com The Receiver Model src: https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
@maasg www. .com Direct Kafka Stream compute(offsets) Driver
@maasg www. .com Kafka:The Receiver-less model Simplified Parallelism Efficiency Exactly-once semantics Less degrees of freedom val directKafkaStream = KafkaUtils.createDirectStream[ [key class], [value class], [key decoder class], [value decoder class] ]( streamingContext, [map of Kafka parameters], [set of topics to consume] ) spark.streaming.kafka.maxRatePerPartition
@maasg www. .com Kafka:The Receiver-less model src: https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
@maasg www. .com Delivery Semantics • Spark Streaming Receiver-based (<v1.2 ) Roughly at least once • Spark Streaming Recv w/ WAL At least once + Zero Data Loss • Spark Streaming Direct At least once + Zero Data Loss • Spark Streaming Direct + Offset management Exactly Once + Idempotent Writes | Transactions
@maasg www. .com Spark Streaming (v1.5) made Reactive proportional-integral-derivative controller (PID controller) Backpressure support
@maasg www. .com Demo 2 Spark Streaming Performance
@maasg www. .com Resources Spark Streaming Official Programming Guide: http://spark.apache.org/docs/latest/streaming-programming-guide.html Backpressure in Spark Streaming: http://blog.garillot.net/post/121183250481/a-quick-update-on-spark-streaming-work-since-i The Virdata’s Spark Streaming tuning guide: http://www.virdata.com/tuning-spark/ Spark Summit Presentations: https://spark-summit.org/ Diving into Spark Streaming Execution Model: https://databricks.com/blog/2015/07/30/diving-into-spark-streamings-execution-model.html Kafka direct approach: https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md
@maasg www. .com Questions?
@maasg www. .com Thanks! Gerard Maas @maasg www. .com - we’re hiring -

More Related Content

PPTX
Spark streaming
byWhiteklay
 
PDF
Strata NYC 2015: What's new in Spark Streaming
byDatabricks
 
PDF
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...
bySpark Summit
 
PPTX
Intro to Spark development
bySpark Summit
 
PPTX
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
byspark-project
 
PDF
Bellevue Big Data meetup: Dive Deep into Spark Streaming
bySantosh Sahoo
 
PPT
Introduction to Spark Streaming
byKnoldus Inc.
 
PDF
Data Analytics with Apache Spark and Cassandra
byGerard Maas
 
Spark streaming
byWhiteklay
 
Strata NYC 2015: What's new in Spark Streaming
byDatabricks
 
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerar...
bySpark Summit
 
Intro to Spark development
bySpark Summit
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
byspark-project
 
Bellevue Big Data meetup: Dive Deep into Spark Streaming
bySantosh Sahoo
 
Introduction to Spark Streaming
byKnoldus Inc.
 
Data Analytics with Apache Spark and Cassandra
byGerard Maas
 

What's hot

PDF
Spark And Cassandra: 2 Fast, 2 Furious
byJen Aman
 
PDF
Taking Spark Streaming to the Next Level with Datasets and DataFrames
byDatabricks
 
POTX
Apache Spark Streaming: Architecture and Fault Tolerance
bySachin Aggarwal
 
PDF
Sqoop on Spark for Data Ingestion-(Veena Basavaraj and Vinoth Chandar, Uber)
bySpark Summit
 
ODP
Meet Up - Spark Stream Processing + Kafka
byKnoldus Inc.
 
PDF
Introduction to Spark Streaming
bydatamantra
 
PDF
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
byHelena Edelson
 
PDF
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
byLegacy Typesafe (now Lightbend)
 
PDF
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
bySpark Summit
 
PPTX
ETL with SPARK - First Spark London meetup
byRafal Kwasny
 
PDF
Lambda Architecture Using SQL
bySATOSHI TAGOMORI
 
PDF
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
byDatabricks
 
PDF
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
byDatabricks
 
PDF
Spark Meetup at Uber
byDatabricks
 
PDF
Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...
byHelena Edelson
 
PPTX
Alpine academy apache spark series #1 introduction to cluster computing wit...
byHolden Karau
 
PPTX
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
PDF
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
byDatabricks
 
PDF
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
byEvan Chan
 
PDF
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 
Spark And Cassandra: 2 Fast, 2 Furious
byJen Aman
 
Taking Spark Streaming to the Next Level with Datasets and DataFrames
byDatabricks
 
Apache Spark Streaming: Architecture and Fault Tolerance
bySachin Aggarwal
 
Sqoop on Spark for Data Ingestion-(Veena Basavaraj and Vinoth Chandar, Uber)
bySpark Summit
 
Meet Up - Spark Stream Processing + Kafka
byKnoldus Inc.
 
Introduction to Spark Streaming
bydatamantra
 
NoLambda: Combining Streaming, Ad-Hoc, Machine Learning and Batch Analysis
byHelena Edelson
 
Four Things to Know About Reliable Spark Streaming with Typesafe and Databricks
byLegacy Typesafe (now Lightbend)
 
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
bySpark Summit
 
ETL with SPARK - First Spark London meetup
byRafal Kwasny
 
Lambda Architecture Using SQL
bySATOSHI TAGOMORI
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
byDatabricks
 
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
byDatabricks
 
Spark Meetup at Uber
byDatabricks
 
Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spa...
byHelena Edelson
 
Alpine academy apache spark series #1 introduction to cluster computing wit...
byHolden Karau
 
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
Structured-Streaming-as-a-Service with Kafka, YARN, and Tooling with Jim Dowling
byDatabricks
 
700 Updatable Queries Per Second: Spark as a Real-Time Web Service
byEvan Chan
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 

Viewers also liked

PPTX
Transformation Processing Smackdown; Spark vs Hive vs Pig
byLester Martin
 
PPTX
Double Your Hadoop Hardware Performance with SmartSense
byHortonworks
 
PDF
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
byDatabricks
 
PDF
Exploring language classification with spark and the spark notebook
byGerard Maas
 
PDF
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...
byLightbend
 
PPTX
Apache Spark and Oracle Stream Analytics
byPrabhu Thukkaram
 
PDF
67 Weeks of TensorFlow
byAltoros
 
PDF
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
PDF
Image Classification Done Simply using Keras and TensorFlow
byRajiv Shah
 
PPTX
Squeezing Deep Learning Into Mobile Phones
byAnirudh Koul
 
PDF
Stream all the things
byDean Wampler
 
PDF
The Power of the Log
byBen Stopford
 
PPTX
Spark Streaming - The simple way
byYogesh Kumar
 
PDF
Architecting a Next Generation Data Platform
byhadooparchbook
 
PDF
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
byHelena Edelson
 
PPTX
Ecossystems near our schools final
byhcpires
 
PPTX
Importante del efectivo
bycinthya vargas
 
PDF
Aquisicion y negociacion de tecnologia
byOLGALUFI
 
PDF
Afa aula 1
byAirton Coelho
 
PPTX
I love maths
byfedacpratesei
 
Transformation Processing Smackdown; Spark vs Hive vs Pig
byLester Martin
 
Double Your Hadoop Hardware Performance with SmartSense
byHortonworks
 
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
byDatabricks
 
Exploring language classification with spark and the spark notebook
byGerard Maas
 
Building Streaming And Fast Data Applications With Spark, Mesos, Akka, Cassan...
byLightbend
 
Apache Spark and Oracle Stream Analytics
byPrabhu Thukkaram
 
67 Weeks of TensorFlow
byAltoros
 
Apache Flink's Table & SQL API - unified APIs for batch and stream processing
byTimo Walther
 
Image Classification Done Simply using Keras and TensorFlow
byRajiv Shah
 
Squeezing Deep Learning Into Mobile Phones
byAnirudh Koul
 
Stream all the things
byDean Wampler
 
The Power of the Log
byBen Stopford
 
Spark Streaming - The simple way
byYogesh Kumar
 
Architecting a Next Generation Data Platform
byhadooparchbook
 
Lambda Architecture with Spark, Spark Streaming, Kafka, Cassandra, Akka and S...
byHelena Edelson
 
Ecossystems near our schools final
byhcpires
 
Importante del efectivo
bycinthya vargas
 
Aquisicion y negociacion de tecnologia
byOLGALUFI
 
Afa aula 1
byAirton Coelho
 
I love maths
byfedacpratesei
 

Similar to Dive into Spark Streaming

PDF
Real-Time Analytics with Apache Cassandra and Apache Spark
byGuido Schmutz
 
PDF
Real-Time Analytics with Apache Cassandra and Apache Spark,
bySwiss Data Forum Swiss Data Forum
 
PDF
Extending Spark Streaming to Support Complex Event Processing
byOh Chan Kwon
 
PDF
Spark cep
byByungjin Kim
 
PPTX
Fraud Detection Architecture
byGwen (Chen) Shapira
 
PPTX
Architecting a Fraud Detection Application with Hadoop
byDataWorks Summit
 
PPTX
07_Spark Streaming introduction all you need to know.pptx
byUttamMittal2
 
PPTX
Spark streaming with apache kafka
bypunesparkmeetup
 
PPTX
Spark Streaming & Kafka-The Future of Stream Processing
byJack Gudenkauf
 
PPTX
Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...
byData Con LA
 
PDF
Fraud Detection using Hadoop
byhadooparchbook
 
PDF
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
PDF
Data processing platforms with SMACK: Spark and Mesos internals
byAnton Kirillov
 
PDF
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
PPTX
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
byMicrosoft Tech Community
 
PDF
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
byCodemotion Dubai
 
PPTX
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
PDF
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
byEvan Chan
 
PPTX
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
byChris Fregly
 
PPTX
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
byTathagata Das
 
Real-Time Analytics with Apache Cassandra and Apache Spark
byGuido Schmutz
 
Real-Time Analytics with Apache Cassandra and Apache Spark,
bySwiss Data Forum Swiss Data Forum
 
Extending Spark Streaming to Support Complex Event Processing
byOh Chan Kwon
 
Spark cep
byByungjin Kim
 
Fraud Detection Architecture
byGwen (Chen) Shapira
 
Architecting a Fraud Detection Application with Hadoop
byDataWorks Summit
 
07_Spark Streaming introduction all you need to know.pptx
byUttamMittal2
 
Spark streaming with apache kafka
bypunesparkmeetup
 
Spark Streaming & Kafka-The Future of Stream Processing
byJack Gudenkauf
 
Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...
byData Con LA
 
Fraud Detection using Hadoop
byhadooparchbook
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
byGuido Schmutz
 
Data processing platforms with SMACK: Spark and Mesos internals
byAnton Kirillov
 
Delivering Meaning In Near-Real Time At High Velocity In Massive Scale with A...
byHelena Edelson
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
byMicrosoft Tech Community
 
SMACK Stack - Fast Data Done Right by Stefan Siprell at Codemotion Dubai
byCodemotion Dubai
 
Real Time Data Processing Using Spark Streaming
byHari Shreedharan
 
Building Scalable Data Pipelines - 2016 DataPalooza Seattle
byEvan Chan
 
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
byChris Fregly
 
Guest Lecture on Spark Streaming in Stanford CME 323: Distributed Algorithms ...
byTathagata Das
 

Recently uploaded

PPTX
Data Skills for Business Analysts: What You Need to Succeed
byScott W. Ambler
 
PPTX
40m_wAIred_DigitalPlatformRabobank_10022026.pptx
bySimonedeGijt
 
PPTX
40m_wAIred_DigitalPlatformRabobank_10022026.pptx
bySimonedeGijt
 
PDF
On Demand Massage Therapist Booking App Development
byV3CUBE TECHNOLABS
 
PDF
The Ultimate Guide to Real Estate Tokenization Platforms
byDaniel Smith
 
PDF
Complete iOS Delivery App Features & Admin Tools
byShiv Technolabs Pvt. Ltd.
 
PDF
Langchain4J – An Introduction for Impatient Developers
byJuarez Junior
 
PDF
Introduction to Modern Artificial Intelligence.pdf
byAI n Dot Net
 
PDF
openstack_operations_slides_version1.3_pp.pdf
byssuser47d511
 
PDF
MySQL Routing Guidelines: Designing Smarter Query Routing Together
byMiguel Araújo
 
PPTX
Best Digital Marketing Company in Lucknow
bydigitallearning9277
 
PPTX
Connecting to MCP Servers in OutSystems Platform
byOzanCali
 
PDF
eresource Xcel ERP for Maunfacturing - Best Manufacturing ERP for SME
byeresource Infotech Pvt.Ltd
 
PDF
JFokus 2026 - Please pass the salt- Serve up passwords with a side of entropy...
byOrtus Solutions, Corp
 
PPTX
‘Analyzing Application Logs Using AI’ Webinar
byTier1 app
 
PDF
Ready, Set, REST! Introducing MySQL's Built-In REST Service
byMiguel Araújo
 
PDF
Metrics, Logs, Traces, and Mayhem_ Introducing an observability adventure gam...
byImma Valls Bernaus
 
PDF
Building Smarter AI: 16 RAG Approaches for Accuracy, Memory, and Reasoning Vi...
byVineet Sharma
 
PDF
Mastering Zero-Allocation Parsers: A Deep Dive into High-Performance C#
byVineet Sharma
 
PDF
ACE Aarhus February 2026: Atlassian news
byDanielEriksen5
 
Data Skills for Business Analysts: What You Need to Succeed
byScott W. Ambler
 
40m_wAIred_DigitalPlatformRabobank_10022026.pptx
bySimonedeGijt
 
40m_wAIred_DigitalPlatformRabobank_10022026.pptx
bySimonedeGijt
 
On Demand Massage Therapist Booking App Development
byV3CUBE TECHNOLABS
 
The Ultimate Guide to Real Estate Tokenization Platforms
byDaniel Smith
 
Complete iOS Delivery App Features & Admin Tools
byShiv Technolabs Pvt. Ltd.
 
Langchain4J – An Introduction for Impatient Developers
byJuarez Junior
 
Introduction to Modern Artificial Intelligence.pdf
byAI n Dot Net
 
openstack_operations_slides_version1.3_pp.pdf
byssuser47d511
 
MySQL Routing Guidelines: Designing Smarter Query Routing Together
byMiguel Araújo
 
Best Digital Marketing Company in Lucknow
bydigitallearning9277
 
Connecting to MCP Servers in OutSystems Platform
byOzanCali
 
eresource Xcel ERP for Maunfacturing - Best Manufacturing ERP for SME
byeresource Infotech Pvt.Ltd
 
JFokus 2026 - Please pass the salt- Serve up passwords with a side of entropy...
byOrtus Solutions, Corp
 
‘Analyzing Application Logs Using AI’ Webinar
byTier1 app
 
Ready, Set, REST! Introducing MySQL's Built-In REST Service
byMiguel Araújo
 
Metrics, Logs, Traces, and Mayhem_ Introducing an observability adventure gam...
byImma Valls Bernaus
 
Building Smarter AI: 16 RAG Approaches for Accuracy, Memory, and Reasoning Vi...
byVineet Sharma
 
Mastering Zero-Allocation Parsers: A Deep Dive into High-Performance C#
byVineet Sharma
 
ACE Aarhus February 2026: Atlassian news
byDanielEriksen5
 

Dive into Spark Streaming

  • 1.
    @maasg www. .com by GerardMaas Data Processing Team Lead gerard.maas@virdata.com @maasg Dive into
  • 2.
    @maasg www. .com Virdata:A ‘born on the cloud’ IoT Managed Services & Analytics Platform
  • 3.
    @maasg www. .com A‘born on the cloud’ IoT Managed Services & Analytics Platform 2012 2013 2014 affiliate certified
  • 4.
    @maasg www. .com KafkaPUB / SUB MQTT / WebSockets RAW Storage >>> Storage` ` ` >>> <<< >>> Query >>> >>> Virdata’s Stack
  • 5.
    @maasg www. .com PUB/ SUB MQTT / WebSockets RAW Storage Storage ` ` ` Query Notebook Server How Spark is Driving a New Loosely-coupled Stand-alone Service Virdata’s Full StackVirdata’s Spark as a Service
  • 7.
  • 8.
  • 9.
    @maasg www. .com Agenda Whatis Spark Streaming? Programming Model Demo 1 Execution Model Demo 2 Resources Q/A
  • 10.
  • 11.
    @maasg www. .com SparkStreaming Scalable, fault-tolerant stream processing system Kafka Flume Kinesis Twitter Sockets HDFS/S3 Databases HDFS Server APPLICATIONS Custom Streams
  • 12.
    @maasg www. .com Spark:RDD Operations transformatio n INPUT DATA HDFS TEXT/ Sequence File RDD SparkContext RDD OUTPUT Data HDFS TEXT/ Sequence File Cassandra Lazy evaluation action
  • 13.
    @maasg www. .com SparkStreaming: DStreams transformatio n Receiver RDD RDD Lazy evaluation action Stream
  • 14.
  • 15.
    @maasg www. .com DStream[T] RDD[T]RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 Spark Streaming
  • 16.
    @maasg www. .com DStream[T] RDD[T]RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformations Spark Streaming
  • 17.
    @maasg www. .com DStream[T] RDD[T]RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Actions Transformations Spark Streaming
  • 18.
  • 19.
    @maasg www. .com Transformations transform valiotDstream = MQTTUtils.createStream(...) val devicePriority = sparkContext.cassandraTable(...) val prioritizedDStream = iotDstream.transform{rdd => rdd.map(d => (d.id, d)).join(devicePriority) }
  • 20.
  • 21.
  • 22.
    @maasg www. .com Actions print ------------------------------------------- Time:1459875469000 ms ------------------------------------------- data1 data2 saveAsTextFiles, saveAsObjectFiles, saveAsHadoopFiles xxx yyy zzz foreachRDD *
  • 23.
    @maasg www. .com Actions- foreachRDD dstream.foreachRDD{rdd => } Dataframes Spark SQL MLLib GraphX Databases ...
  • 24.
    @maasg www. .com Actions- foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) val conn = DB.connect(server) asRecords.foreachPartition{partition => partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) }
  • 25.
    @maasg www. .com Actions- foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) val conn = DB.connect(server) asRecords.foreachPartition{partition => partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) } Executes on the Driver Executes on the Workers
  • 26.
    @maasg www. .com Actions- foreachRDD Usage dstream.foreachRDD{rdd => rdd.cache() val alternatives = restServer.get(“/v1/alternatives”).toSet alternatives.foreach{alternative => val byAlternative = rdd.filter(element => element.kind == alternative) val asRecords = byAlternative.map(element => asRecord(element)) asRecords.foreachPartition{partition => val conn = DB.connect(server) partition.foreach(element => conn.insert(element) } } rdd.unpersist(true) } Executes on the Driver Executes on the Workers
  • 27.
    @maasg www. .com Windows- Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14
  • 28.
    @maasg www. .com Windows- Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 4,5,6,7,8,9
  • 29.
    @maasg www. .com Windows- Sliding 1 t dstream.window(windowLength = 6, slideInterval = 3) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 4,5,6,7,8,9 7,8,9,10,11,12
  • 30.
    @maasg www. .com Windows- Non-Overlapping 1 t dstream.window(windowLength = 6, slideInterval = 6) t 2 3 4 5 6 7 8 9 1,2,3,4,5,6 10 11 12 13 14 7,8,9,10,11,12
  • 31.
    @maasg www. .com Windows- Operations window, countByWindow, reduceByWindow, reduceByKeyAndWindow, countByValueAndWindow
  • 32.
    @maasg www. .com Windows- Inverse Function Optimization reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval,[numTasks]) 1 2 3 4 5 6 7 8 9 1,2,3,4,5,6 + + -
  • 33.
    @maasg www. .com Windows-Inverse Function Optimization reduceByKeyAndWindow(func, invFunc, windowLength, slideInterval,[numTasks]) 1 2 3 4 5 6 7 8 9 1,2,3,4,5,6 4,5,6,7,8,9 - + - + 7 8 9 1 2 3
  • 34.
    @maasg www. .com Top3 Starter Issues Serialization Where is my closure executing Do I have enough cores?
  • 35.
    @maasg www. .com Demo1 Anatomy of an Spark Streaming Application
  • 36.
    @maasg www. .com Tweetfew keywords about your interests and experience. Use hashtag “#sparkbe” Scala Spark Streaming DistributedSystems #sparkbe
  • 37.
  • 38.
    @maasg www. .com DeploymentOptions M Local Standalone Cluster WW W Using a Cluster Manager W spark.master=local[*] spark.master=spark://host:port spark.master=mesos://host:port M M D W D W W
  • 39.
    @maasg www. .com DeploymentOptions M Local Standalone Cluster Using a Cluster Manager spark.master=local[*] spark.master=mesos://host:port M M DD Rec Exec Rec Exec Exec Exec Rec Exec Rec Exec ExecExec spark.master=spark://host:port
  • 40.
    @maasg www. .com Streaming Spark t0t1 t2 #0 Consumer Consumer Consumer Scheduling
  • 41.
    @maasg www. .com Streaming Spark t0t1 t2 #1 Consumer Consumer Consumer #0 Process Time < Batch Interval Scheduling
  • 42.
    @maasg www. .com Streaming Spark t0t1 t2 #2 Consumer Consumer Consumer #0 #1 #3 Scheduling Delay Scheduling
  • 43.
    @maasg www. .com FromStreams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
  • 44.
  • 45.
    @maasg www. .com #0 RDD Spark Spark Executors SparkStreaming From Streams to μbatches
  • 46.
    @maasg www. .com #0 RDD Spark Spark Executors SparkStreaming From Streams to μbatches
  • 47.
    @maasg www. .com #0 RDD Spark Spark Executors SparkStreaming From Streams to μbatches
  • 48.
    @maasg www. .com FromStreams to μbatches
  • 49.
    @maasg www. .com FromStreams to μbatches
  • 50.
    @maasg www. .com Consumer #0#1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval From Streams to μbatches
  • 51.
    @maasg www. .com Consumer #0#1 batchInterval blockInterval Spark Streaming Spark spark.streaming.blockInterval = batchInterval x receivers / (partitionFactor x sparkCores) From Streams to μbatches
  • 52.
    @maasg www. .com TheImportance of Caching dstream.foreachRDD { rdd => rdd.cache() // cache the RDD before iterating! keys.foreach{ key => rdd.filter(elem=> key(elem) == key).saveAsFooBar(...) } rdd.unpersist() }
  • 53.
    @maasg www. .com TheReceiver model spark.streaming.receiver.maxRate Fault tolerance ? WAL
  • 54.
    @maasg www. .com TheReceiver Model src: https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
  • 55.
    @maasg www. .com DirectKafka Stream compute(offsets) Driver
  • 56.
    @maasg www. .com Kafka:TheReceiver-less model Simplified Parallelism Efficiency Exactly-once semantics Less degrees of freedom val directKafkaStream = KafkaUtils.createDirectStream[ [key class], [value class], [key decoder class], [value decoder class] ]( streamingContext, [map of Kafka parameters], [set of topics to consume] ) spark.streaming.kafka.maxRatePerPartition
  • 57.
    @maasg www. .com Kafka:TheReceiver-less model src: https://databricks.com/blog/2015/03/30/improvements-to-kafka-integration-of-spark-streaming.html
  • 58.
    @maasg www. .com DeliverySemantics • Spark Streaming Receiver-based (<v1.2 ) Roughly at least once • Spark Streaming Recv w/ WAL At least once + Zero Data Loss • Spark Streaming Direct At least once + Zero Data Loss • Spark Streaming Direct + Offset management Exactly Once + Idempotent Writes | Transactions
  • 59.
    @maasg www. .com SparkStreaming (v1.5) made Reactive proportional-integral-derivative controller (PID controller) Backpressure support
  • 60.
    @maasg www. .com Demo2 Spark Streaming Performance
  • 62.
    @maasg www. .com Resources SparkStreaming Official Programming Guide: http://spark.apache.org/docs/latest/streaming-programming-guide.html Backpressure in Spark Streaming: http://blog.garillot.net/post/121183250481/a-quick-update-on-spark-streaming-work-since-i The Virdata’s Spark Streaming tuning guide: http://www.virdata.com/tuning-spark/ Spark Summit Presentations: https://spark-summit.org/ Diving into Spark Streaming Execution Model: https://databricks.com/blog/2015/07/30/diving-into-spark-streamings-execution-model.html Kafka direct approach: https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md
  • 63.
  • 64.
    @maasg www. .com Thanks! GerardMaas @maasg www. .com - we’re hiring -