SlideShare a Scribd company logo
1 of 33
Download to read offline
Building Robust, Adaptive Streaming
Apps with Spark Streaming
Tathagata “TD” Das
@tathadas
Spark Summit East 2016
Who am I?
Project ManagementCommittee (PMC) member of Spark
Started Spark Streaming in grad school - AMPLab, UC Berkeley
Currenttechnical lead of Spark Streaming
Software engineerat Databricks
2
Streaming Apps in the Real World
Building a high volume stream processingsystem in
production has many challenges
3
Fast and Scalable Spark Streaming is fast, distributed and
scalableby design!
Running in production at many places
with largeclusters and high data volumes
Streaming Apps in the Real World
Building a high volume stream processingsystem in
production has many challenges
4
Fast and Scalable
Easy to program
Spark Streaming makes it easy to express
complex streaming businesslogic
Interoperateswith Spark RDDs, Spark SQL
DataFrames/Datasets and MLlib
Streaming Apps in the Real World
Building a high volume stream processingsystem in
production has many challenges
5
Fast and Scalable
Easy to program
Fault-tolerant
Spark Streaming is fullyfault-tolerant
and can provide end-to-end semantic
guarantees
See my previous Spark Summit talk for moredetails
Streaming Apps in the Real World
Building a high volume stream processingsystem in
production has many challenges
6
Fast and Scalable
Easy to program
Fault-tolerant
Adaptive Focus of this talk
Adaptive Streaming Apps
Processing conditions can changedynamically
Sudden surges in data rates
Diurnal variationsin processing load
Unexpected slowdownsin downstreamdata stores
Streaming apps should be able to adapt accordingly
7
8
Backpressure
Make apps robust
againstdata surges
Elastic Scaling
Make apps scale with
load variations
9
Backpressure
Make apps robust
againstdata surges
Motivation
Stability condition for any streaming app
Receive data only as fast as the system can processit
Stability condition for Spark Streaming’s “micro-batch” model
Finish processing previous batch before next one arrives
10
Stable micro-batch operation
11
batch interval
1s 1s 1s 1s
batch processing time <=
Previousbatch isprocessed before next one arrives => stable
Spark Streaming runs micro-batchesat fixed batchintervals
time
0s 2s 4s 6s 8s
Unstable micro-batch operation
12
batch processing time > batch interval
2.1s 2.1s 2.1s 2.1s
scheduling delay builds up
Batchescontinuously getsdelayed and backlogged => unstable
time
Spark Streaming runs micro-batchesat fixed batchintervals
0s 2s 4s 6s 8s
Backpressure: Feedback Loop
13
Backpressureintroducesa feedback loop to dynamically
adapt the system and avoid instability
Backpressure: Dynamic rate limiting
14
Batch processingtimes
Scheduling delays
Batch processingtimes and schedulingdelays used to
continuouslyestimate currentprocessing rates
Backpressure: Dynamic rate limiting
15
∑
Batch processingtimes
Scheduling delays
Max stable processingrate estimated with PID Controllertheory
Well known feedbackmechanismused in industrial control systems
PID Controller
Backpressure: Dynamic rate limiting
16
∑
Batch processingtimes
Scheduling delays
Rate limits on
ingestion
Accordingly, the system dynamically adapts
the limits on the data ingestion rates
PID Controller
Data buffered in Kafka, ensures
Spark Streaming stays stable
Backpressure: Dynamic rate limiting
17
If HDFS ingestion slows down,
processingtimes increase
SS lowers rate limits to slow
down receiving from Kafka
Backpressure: Configuration
Available since Spark 1.5
Enabled through SparkConf,set
spark.streaming.backpressure.enabled=true
18
19
Elastic Scaling
Make apps scale with
load variations
Elastic Scaling (aka Dynamic Allocation)
Scaling the number of Spark executorsaccording to the load
Spark alreadysupports Dynamic Allocation for batch jobs
Scale down if executorsare idle
Scale up if tasks are queueing up
Streaming “micro-batch” jobs needdifferent scaling policy
No executoris idle for a long time!
20
Scaling policy with Streaming
21
Load is very low
Lots idle time between batches
Cluster resources wasted
stable but
inefficient
Scaling policy with Streaming
22
scale down
the cluster
scaling down increases batch processing times
stable operation with less resources wasted
stable but
inefficient
stable and
efficient
Scaling policy with Streaming
23
scaling up reduces batch processing times
unstable system becomes stable again
scale up
the cluster
unstable
stable
Elastic Scaling
24
Data buffered in Kafka starts draining,
allowing app adapt to any data rate
If Kafka gets data faster than
what backpressureallows
SS scalesup clusterto
increaseprocessing rate
Elastic Scaling: Configuration
Will be available in Spark 2.0
Enabled through SparkConf,set
spark.streaming.dynamicAllocation.enabled = true
More parameters will be in the online programming guide
25
Elastic Scaling: Configuration
Make surethere is enoughparallelismto take advantage of
max cluster size
# of partitions in reduce,join, etc.
# of Kafka partitions
# of receivers
Gives usualfault-tolerance guaranteeswith files, Kafka Direct,
Kinesis, and receiver-basedsourceswith WAL enabled
26
27
Backpressure Elastic Scaling
28
Awesome
Adaptive
Apps
29
Demo
How app behaveswhen
the data rate suddenly
increases20x
Demo
30
Processing time increases
with data rate, until equal to
batch interval
Backpressurelimits the
ingestion rate lower than 20k
recs/secto keep app stable
Demo
31
Elastic Scaling detects heavy
load and increasescluster size
Processing times reducesas
more resourcesavailable
Demo
32
Backpressurerelaxeslimits to
allow higheringestion rate
But still less than 20x as
cluster is fullyutilized
Takeaways
Backpressure:makes apps robust to sudden changes
Elastic Scaling: makes apps adapt to slower changes
Backpressure+ Elastic Scaling =
AwesomeAdaptive Apps
Follow me @tathadas

More Related Content

What's hot

Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
Databricks
 
Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters
Downscaling: The Achilles heel of Autoscaling Apache Spark ClustersDownscaling: The Achilles heel of Autoscaling Apache Spark Clusters
Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters
Databricks
 
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye ZhouMetrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
Databricks
 

What's hot (20)

Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Spark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod NarasimhaSpark Summit EU talk by Debasish Das and Pramod Narasimha
Spark Summit EU talk by Debasish Das and Pramod Narasimha
 
Spark Under the Hood - Meetup @ Data Science London
Spark Under the Hood - Meetup @ Data Science LondonSpark Under the Hood - Meetup @ Data Science London
Spark Under the Hood - Meetup @ Data Science London
 
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub WozniakNext CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
 
Spark Summit EU talk by John Musser
Spark Summit EU talk by John MusserSpark Summit EU talk by John Musser
Spark Summit EU talk by John Musser
 
Apache Spark Overview
Apache Spark OverviewApache Spark Overview
Apache Spark Overview
 
New Developments in Spark
New Developments in SparkNew Developments in Spark
New Developments in Spark
 
Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)
 
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
Interoperating a Zoo of Data Processing Platforms Using with Rheem Sebastian ...
 
Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters
Downscaling: The Achilles heel of Autoscaling Apache Spark ClustersDownscaling: The Achilles heel of Autoscaling Apache Spark Clusters
Downscaling: The Achilles heel of Autoscaling Apache Spark Clusters
 
Producing Spark on YARN for ETL
Producing Spark on YARN for ETLProducing Spark on YARN for ETL
Producing Spark on YARN for ETL
 
Spark Summit EU talk by Yaroslav Nedashkovsky and Andy Starzhinsky
Spark Summit EU talk by Yaroslav Nedashkovsky and Andy StarzhinskySpark Summit EU talk by Yaroslav Nedashkovsky and Andy Starzhinsky
Spark Summit EU talk by Yaroslav Nedashkovsky and Andy Starzhinsky
 
Lessons from Running Large Scale Spark Workloads
Lessons from Running Large Scale Spark WorkloadsLessons from Running Large Scale Spark Workloads
Lessons from Running Large Scale Spark Workloads
 
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye ZhouMetrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
Metrics-Driven Tuning of Apache Spark at Scale with Edwina Lu and Ye Zhou
 
Apache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper OptimizationApache Spark Core—Deep Dive—Proper Optimization
Apache Spark Core—Deep Dive—Proper Optimization
 
Hyperspace for Delta Lake
Hyperspace for Delta LakeHyperspace for Delta Lake
Hyperspace for Delta Lake
 
Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0Deep Dive into the New Features of Apache Spark 3.0
Deep Dive into the New Features of Apache Spark 3.0
 
New directions for Apache Spark in 2015
New directions for Apache Spark in 2015New directions for Apache Spark in 2015
New directions for Apache Spark in 2015
 
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
Extreme Apache Spark: how in 3 months we created a pipeline that can process ...
 
Operational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache SparkOperational Tips For Deploying Apache Spark
Operational Tips For Deploying Apache Spark
 

Similar to Building Robust, Adaptive Streaming Apps with Spark Streaming

Similar to Building Robust, Adaptive Streaming Apps with Spark Streaming (20)

Headaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsHeadaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous Applications
 
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
 
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
 
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
Learnings From Shipping 1000+ Streaming Data Pipelines To Production with Hak...
 
Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?Big Data Streams Architectures. Why? What? How?
Big Data Streams Architectures. Why? What? How?
 
Solution Brief: Real-Time Pipeline Accelerator
Solution Brief: Real-Time Pipeline AcceleratorSolution Brief: Real-Time Pipeline Accelerator
Solution Brief: Real-Time Pipeline Accelerator
 
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...
 
Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !Apache Kafka - Scalable Message-Processing and more !
Apache Kafka - Scalable Message-Processing and more !
 
EEDC 2010. Scaling SaaS Applications
EEDC 2010. Scaling SaaS ApplicationsEEDC 2010. Scaling SaaS Applications
EEDC 2010. Scaling SaaS Applications
 
Always On - Zero Downtime releases
Always On - Zero Downtime releasesAlways On - Zero Downtime releases
Always On - Zero Downtime releases
 
Wamika Singh, Suman Kumari - Let's decipher the DevOps macedonia - Codemotion...
Wamika Singh, Suman Kumari - Let's decipher the DevOps macedonia - Codemotion...Wamika Singh, Suman Kumari - Let's decipher the DevOps macedonia - Codemotion...
Wamika Singh, Suman Kumari - Let's decipher the DevOps macedonia - Codemotion...
 
Web Speed And Scalability
Web Speed And ScalabilityWeb Speed And Scalability
Web Speed And Scalability
 
Critical Attributes for a High-Performance, Low-Latency Database
Critical Attributes for a High-Performance, Low-Latency DatabaseCritical Attributes for a High-Performance, Low-Latency Database
Critical Attributes for a High-Performance, Low-Latency Database
 
Data Con LA 2022 Keynote
Data Con LA 2022 KeynoteData Con LA 2022 Keynote
Data Con LA 2022 Keynote
 
Confluent kafka meetupseattle jan2017
Confluent kafka meetupseattle jan2017Confluent kafka meetupseattle jan2017
Confluent kafka meetupseattle jan2017
 
Let's decipher the DevOps macedonia
Let's decipher the DevOps macedoniaLet's decipher the DevOps macedonia
Let's decipher the DevOps macedonia
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Become a Performance Diagnostics Hero
Become a Performance Diagnostics HeroBecome a Performance Diagnostics Hero
Become a Performance Diagnostics Hero
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
 

More from Databricks

Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized Platform
Databricks
 
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationStage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI Integration
Databricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchSimplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Databricks
 
Raven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction QueriesRaven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction Queries
Databricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkProcessing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache Spark
Databricks
 

More from Databricks (20)

DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptx
 
Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2
 
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
 
Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDemocratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized Platform
 
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceLearn to Use Databricks for Data Science
Learn to Use Databricks for Data Science
 
Why APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringWhy APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML Monitoring
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixThe Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
 
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationStage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI Integration
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchSimplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorch
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesScaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on Kubernetes
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesScaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
 
Sawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsSawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature Aggregations
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkRedis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
 
Re-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkRe-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and Spark
 
Raven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction QueriesRaven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction Queries
 
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkProcessing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache Spark
 
Massive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeMassive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta Lake
 

Recently uploaded

VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
dharasingh5698
 
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoorTop Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
dharasingh5698
 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ssuser89054b
 

Recently uploaded (20)

Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
Navigating Complexity: The Role of Trusted Partners and VIAS3D in Dassault Sy...
 
Unit 2- Effective stress & Permeability.pdf
Unit 2- Effective stress & Permeability.pdfUnit 2- Effective stress & Permeability.pdf
Unit 2- Effective stress & Permeability.pdf
 
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdfONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
 
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete RecordCCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record
 
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
VIP Model Call Girls Kothrud ( Pune ) Call ON 8005736733 Starting From 5K to ...
 
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 BookingVIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
VIP Call Girls Palanpur 7001035870 Whatsapp Number, 24/07 Booking
 
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar  ≼🔝 Delhi door step de...
Call Now ≽ 9953056974 ≼🔝 Call Girls In New Ashok Nagar ≼🔝 Delhi door step de...
 
Unit 1 - Soil Classification and Compaction.pdf
Unit 1 - Soil Classification and Compaction.pdfUnit 1 - Soil Classification and Compaction.pdf
Unit 1 - Soil Classification and Compaction.pdf
 
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoorTop Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
Top Rated Call Girls In chittoor 📱 {7001035870} VIP Escorts chittoor
 
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7
 
chapter 5.pptx: drainage and irrigation engineering
chapter 5.pptx: drainage and irrigation engineeringchapter 5.pptx: drainage and irrigation engineering
chapter 5.pptx: drainage and irrigation engineering
 
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 
Work-Permit-Receiver-in-Saudi-Aramco.pptx
Work-Permit-Receiver-in-Saudi-Aramco.pptxWork-Permit-Receiver-in-Saudi-Aramco.pptx
Work-Permit-Receiver-in-Saudi-Aramco.pptx
 
Bhosari ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready For ...
Bhosari ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready For ...Bhosari ( Call Girls ) Pune  6297143586  Hot Model With Sexy Bhabi Ready For ...
Bhosari ( Call Girls ) Pune 6297143586 Hot Model With Sexy Bhabi Ready For ...
 
Thermal Engineering -unit - III & IV.ppt
Thermal Engineering -unit - III & IV.pptThermal Engineering -unit - III & IV.ppt
Thermal Engineering -unit - III & IV.ppt
 
Design For Accessibility: Getting it right from the start
Design For Accessibility: Getting it right from the startDesign For Accessibility: Getting it right from the start
Design For Accessibility: Getting it right from the start
 
Intze Overhead Water Tank Design by Working Stress - IS Method.pdf
Intze Overhead Water Tank  Design by Working Stress - IS Method.pdfIntze Overhead Water Tank  Design by Working Stress - IS Method.pdf
Intze Overhead Water Tank Design by Working Stress - IS Method.pdf
 
data_management_and _data_science_cheat_sheet.pdf
data_management_and _data_science_cheat_sheet.pdfdata_management_and _data_science_cheat_sheet.pdf
data_management_and _data_science_cheat_sheet.pdf
 
Block diagram reduction techniques in control systems.ppt
Block diagram reduction techniques in control systems.pptBlock diagram reduction techniques in control systems.ppt
Block diagram reduction techniques in control systems.ppt
 
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated  Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...Top Rated  Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
 

Building Robust, Adaptive Streaming Apps with Spark Streaming

  • 1. Building Robust, Adaptive Streaming Apps with Spark Streaming Tathagata “TD” Das @tathadas Spark Summit East 2016
  • 2. Who am I? Project ManagementCommittee (PMC) member of Spark Started Spark Streaming in grad school - AMPLab, UC Berkeley Currenttechnical lead of Spark Streaming Software engineerat Databricks 2
  • 3. Streaming Apps in the Real World Building a high volume stream processingsystem in production has many challenges 3 Fast and Scalable Spark Streaming is fast, distributed and scalableby design! Running in production at many places with largeclusters and high data volumes
  • 4. Streaming Apps in the Real World Building a high volume stream processingsystem in production has many challenges 4 Fast and Scalable Easy to program Spark Streaming makes it easy to express complex streaming businesslogic Interoperateswith Spark RDDs, Spark SQL DataFrames/Datasets and MLlib
  • 5. Streaming Apps in the Real World Building a high volume stream processingsystem in production has many challenges 5 Fast and Scalable Easy to program Fault-tolerant Spark Streaming is fullyfault-tolerant and can provide end-to-end semantic guarantees See my previous Spark Summit talk for moredetails
  • 6. Streaming Apps in the Real World Building a high volume stream processingsystem in production has many challenges 6 Fast and Scalable Easy to program Fault-tolerant Adaptive Focus of this talk
  • 7. Adaptive Streaming Apps Processing conditions can changedynamically Sudden surges in data rates Diurnal variationsin processing load Unexpected slowdownsin downstreamdata stores Streaming apps should be able to adapt accordingly 7
  • 8. 8 Backpressure Make apps robust againstdata surges Elastic Scaling Make apps scale with load variations
  • 10. Motivation Stability condition for any streaming app Receive data only as fast as the system can processit Stability condition for Spark Streaming’s “micro-batch” model Finish processing previous batch before next one arrives 10
  • 11. Stable micro-batch operation 11 batch interval 1s 1s 1s 1s batch processing time <= Previousbatch isprocessed before next one arrives => stable Spark Streaming runs micro-batchesat fixed batchintervals time 0s 2s 4s 6s 8s
  • 12. Unstable micro-batch operation 12 batch processing time > batch interval 2.1s 2.1s 2.1s 2.1s scheduling delay builds up Batchescontinuously getsdelayed and backlogged => unstable time Spark Streaming runs micro-batchesat fixed batchintervals 0s 2s 4s 6s 8s
  • 13. Backpressure: Feedback Loop 13 Backpressureintroducesa feedback loop to dynamically adapt the system and avoid instability
  • 14. Backpressure: Dynamic rate limiting 14 Batch processingtimes Scheduling delays Batch processingtimes and schedulingdelays used to continuouslyestimate currentprocessing rates
  • 15. Backpressure: Dynamic rate limiting 15 ∑ Batch processingtimes Scheduling delays Max stable processingrate estimated with PID Controllertheory Well known feedbackmechanismused in industrial control systems PID Controller
  • 16. Backpressure: Dynamic rate limiting 16 ∑ Batch processingtimes Scheduling delays Rate limits on ingestion Accordingly, the system dynamically adapts the limits on the data ingestion rates PID Controller
  • 17. Data buffered in Kafka, ensures Spark Streaming stays stable Backpressure: Dynamic rate limiting 17 If HDFS ingestion slows down, processingtimes increase SS lowers rate limits to slow down receiving from Kafka
  • 18. Backpressure: Configuration Available since Spark 1.5 Enabled through SparkConf,set spark.streaming.backpressure.enabled=true 18
  • 19. 19 Elastic Scaling Make apps scale with load variations
  • 20. Elastic Scaling (aka Dynamic Allocation) Scaling the number of Spark executorsaccording to the load Spark alreadysupports Dynamic Allocation for batch jobs Scale down if executorsare idle Scale up if tasks are queueing up Streaming “micro-batch” jobs needdifferent scaling policy No executoris idle for a long time! 20
  • 21. Scaling policy with Streaming 21 Load is very low Lots idle time between batches Cluster resources wasted stable but inefficient
  • 22. Scaling policy with Streaming 22 scale down the cluster scaling down increases batch processing times stable operation with less resources wasted stable but inefficient stable and efficient
  • 23. Scaling policy with Streaming 23 scaling up reduces batch processing times unstable system becomes stable again scale up the cluster unstable stable
  • 24. Elastic Scaling 24 Data buffered in Kafka starts draining, allowing app adapt to any data rate If Kafka gets data faster than what backpressureallows SS scalesup clusterto increaseprocessing rate
  • 25. Elastic Scaling: Configuration Will be available in Spark 2.0 Enabled through SparkConf,set spark.streaming.dynamicAllocation.enabled = true More parameters will be in the online programming guide 25
  • 26. Elastic Scaling: Configuration Make surethere is enoughparallelismto take advantage of max cluster size # of partitions in reduce,join, etc. # of Kafka partitions # of receivers Gives usualfault-tolerance guaranteeswith files, Kafka Direct, Kinesis, and receiver-basedsourceswith WAL enabled 26
  • 29. 29 Demo How app behaveswhen the data rate suddenly increases20x
  • 30. Demo 30 Processing time increases with data rate, until equal to batch interval Backpressurelimits the ingestion rate lower than 20k recs/secto keep app stable
  • 31. Demo 31 Elastic Scaling detects heavy load and increasescluster size Processing times reducesas more resourcesavailable
  • 32. Demo 32 Backpressurerelaxeslimits to allow higheringestion rate But still less than 20x as cluster is fullyutilized
  • 33. Takeaways Backpressure:makes apps robust to sudden changes Elastic Scaling: makes apps adapt to slower changes Backpressure+ Elastic Scaling = AwesomeAdaptive Apps Follow me @tathadas