SlideShare a Scribd company logo
An Experiment-Driven Performance Model of Stream
Processing Operators in Fog Computing Environments
Hamidreza Arkian1
, Guillaume Pierre1
,
Johan Tordsson2
, Erik Elmroth2
1
University of Rennes1/IRISA, France
2
Elastisys AB, Sweden
SAC’20 - March 30-April 3, 2020 - Brno, Czech Republic
2/16
IoT-to-Cloud basic architecture
3/16
Cloud-based stream processing
Apache Flink
4/16
Challenges
Apache Flink
Low Throughput!!
Low Bandwidth!!
Cost!!
Continuously generating
stream of data
with high rate
Latency-sensitive
applications
5/16
Fog-based stream processing
6/16
Operator
2
Stream processing in Fog environment
Source Operator
4
Operator
1
Sink
Operator
2
Operator
3
Source
Operator
4
Operator
1
Sink
Operator
2
Operator
3
Source
Operator
2
Logical graph of DSP
Workflow execution model
7/16
Operator
2
Stream processing in geo-distributed environments
Source Operator
4
Operator
1
Sink
Operator
2
Operator
3
Source
Operator
4
Operator
1
Sink
Operator
2
Operator
3
Source
Operator
2
Logical graph of DSP
Workflow execution model
Deployment in Fog geo-distributed environment
Sink
Operator
1
Op2
Replica2
Operator
3
Operator
4
Source
Op2
Replica1
Op2
Replica3
Source
8/16
Challenges
➢ Understanding the performance of a geo-distributed stream processing application is
difficult.
➢ Any configuration decision can have a significant impact on performance.
9/16
Experimental setup
➢ Emulation of a real fog platform
o 32-core server ≈ 16 fog nodes (2 cores/node)
o Emulated network latencies
o Apache Flink
➢ Test Application
o Input stream of 100,000 Tuple2 records
o The operator calls the Fibonacci function
Fib(24) upon every processed record
➢ Performance metric:
o Processing Time (PT)
10/16
Modeling operator replication
➢ n operator replicas should in principle process data n times faster than a single replica
➢ α represents the computation capacity of a single node.
➢ We can determine the value of α based on one measurement
Experiment Model
11/16
Considering heterogeneous network delays
➢ Network delays between data sources and operator replicas slow down the whole system.
➢ When the network delays are heterogeneous, the dominating one is the greatest one (NDmax
).
➢ γ represents the impact of network delays on overall performance.
➢ We can determine both α and γ based on two measurements
Experiment Model
12/16
Improving the model’s accuracy
➢ Operator replication incurs some amount of parallelization inefficiency
➢ The speedup with n nodes is usually a little less than n
➢ 𝛽 represents Flink’s parallelization inefficiency
➢ We can determine α, 𝛽 and γ based on three or more measurements
Experiment Model
13/16
Prediction accuracy
Accuracy metric: 𝑀𝐴𝑃𝐸
4 measurements,
2.0% accuracy
14/16
What about modeling an entire (simple) workflow?
➢ The throughput of an entire workflow is determined by the slowest operator
𝛱Workflow
= max(𝛱Map+KeyBy
, 𝛱Reduce
)
Experiment ModelWorkflow
15/16
Can we reuse the parameters instead of multiple measurements?
➢ 𝛼 cannot be reused because it is specific to the
computation complexity of one operator.
➢ β and γ capture properties that are independent from
the nature of the computation carried out by the
operator.
➢ β and γ values of one operator’s model might be reused
for other operators’ models.
Calibrated model for Operator 1 Uncalibrated model for Operator 2
𝛼1
β1
γ1
𝛼2
β1
γ1
16/16
Conclusions
➢ Heterogeneous network characteristics make it difficult to understand the
performance of stream processing engines in geo-distributed environments.
➢ A predictive performance model for Apache Flink operators that is backed by
experimental measurements and evaluations was proposed.
➢ The model predictions are accurate within ±2% of the actual values.
Hamidreza Arkian
hamidreza.arkian@irisa.fr
Acknowledgment
This work is part of a project that has received funding from the European Union’s
Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No 765452. The information and views set out
in this publication are those of the author(s) and do not necessarily reflect the
official opinion of the European Union. Neither the European Union institutions
and bodies nor any person acting on their behalf may be held responsible for the
use which may be made of the information contained therein.
Training the next generation of European
Fog computing experts
http://www.fogguru.eu/

More Related Content

What's hot

Architecture and Performance of Runtime Environments for Data Intensive Scala...
Architecture and Performance of Runtime Environments for Data Intensive Scala...Architecture and Performance of Runtime Environments for Data Intensive Scala...
Architecture and Performance of Runtime Environments for Data Intensive Scala...
jaliyae
 
A Guide to Data Versioning with MapR Snapshots
A Guide to Data Versioning with MapR SnapshotsA Guide to Data Versioning with MapR Snapshots
A Guide to Data Versioning with MapR Snapshots
Ian Downard
 
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial NetworksSuperframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
Oka Danil
 
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11aDeadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
Oka Danil
 
Scalable Parallel Computing on Clouds
Scalable Parallel Computing on CloudsScalable Parallel Computing on Clouds
Scalable Parallel Computing on Clouds
Thilina Gunarathne
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
Otávio Carvalho
 
Optimization of graph storage using GoFFish
Optimization of graph storage using GoFFishOptimization of graph storage using GoFFish
Optimization of graph storage using GoFFish
Anushree Prasanna Kumar
 
Super COMPUTING Journal
Super COMPUTING JournalSuper COMPUTING Journal
Super COMPUTING Journal
Pandey_G
 
SERENE 2014 School: Daniel varro serene2014_school
SERENE 2014 School: Daniel varro serene2014_schoolSERENE 2014 School: Daniel varro serene2014_school
SERENE 2014 School: Daniel varro serene2014_school
Henry Muccini
 
C-SAW: A Framework for Graph Sampling and Random Walk on GPUs
C-SAW: A Framework for Graph Sampling and Random Walk on GPUsC-SAW: A Framework for Graph Sampling and Random Walk on GPUs
C-SAW: A Framework for Graph Sampling and Random Walk on GPUs
Pandey_G
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
EUDAT
 
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
deawoo Kim
 
Experiences with High-bandwidth Networks
Experiences with High-bandwidth NetworksExperiences with High-bandwidth Networks
Experiences with High-bandwidth Networks
balmanme
 
Clone cloud
Clone cloudClone cloud
Clone cloud
Bhagavathi Dhass
 
High Performance Computing in the Cloud?
High Performance Computing in the Cloud?High Performance Computing in the Cloud?
High Performance Computing in the Cloud?
Ian Lumb
 
2019 swan-cs3
2019 swan-cs32019 swan-cs3
2019 swan-cs3
Up2Universe
 
Low Power High-Performance Computing on the BeagleBoard Platform
Low Power High-Performance Computing on the BeagleBoard PlatformLow Power High-Performance Computing on the BeagleBoard Platform
Low Power High-Performance Computing on the BeagleBoard Platform
a3labdsp
 
Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.
Phdtopiccom
 
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
NECST Lab @ Politecnico di Milano
 
High performance computing
High performance computingHigh performance computing
High performance computing
Guy Tel-Zur
 

What's hot (20)

Architecture and Performance of Runtime Environments for Data Intensive Scala...
Architecture and Performance of Runtime Environments for Data Intensive Scala...Architecture and Performance of Runtime Environments for Data Intensive Scala...
Architecture and Performance of Runtime Environments for Data Intensive Scala...
 
A Guide to Data Versioning with MapR Snapshots
A Guide to Data Versioning with MapR SnapshotsA Guide to Data Versioning with MapR Snapshots
A Guide to Data Versioning with MapR Snapshots
 
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial NetworksSuperframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial Networks
 
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11aDeadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11a
 
Scalable Parallel Computing on Clouds
Scalable Parallel Computing on CloudsScalable Parallel Computing on Clouds
Scalable Parallel Computing on Clouds
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
 
Optimization of graph storage using GoFFish
Optimization of graph storage using GoFFishOptimization of graph storage using GoFFish
Optimization of graph storage using GoFFish
 
Super COMPUTING Journal
Super COMPUTING JournalSuper COMPUTING Journal
Super COMPUTING Journal
 
SERENE 2014 School: Daniel varro serene2014_school
SERENE 2014 School: Daniel varro serene2014_schoolSERENE 2014 School: Daniel varro serene2014_school
SERENE 2014 School: Daniel varro serene2014_school
 
C-SAW: A Framework for Graph Sampling and Random Walk on GPUs
C-SAW: A Framework for Graph Sampling and Random Walk on GPUsC-SAW: A Framework for Graph Sampling and Random Walk on GPUs
C-SAW: A Framework for Graph Sampling and Random Walk on GPUs
 
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...
 
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
Revisiting Sensor MAC for Periodic Monitoring: Why Should Transmitters Be Ear...
 
Experiences with High-bandwidth Networks
Experiences with High-bandwidth NetworksExperiences with High-bandwidth Networks
Experiences with High-bandwidth Networks
 
Clone cloud
Clone cloudClone cloud
Clone cloud
 
High Performance Computing in the Cloud?
High Performance Computing in the Cloud?High Performance Computing in the Cloud?
High Performance Computing in the Cloud?
 
2019 swan-cs3
2019 swan-cs32019 swan-cs3
2019 swan-cs3
 
Low Power High-Performance Computing on the BeagleBoard Platform
Low Power High-Performance Computing on the BeagleBoard PlatformLow Power High-Performance Computing on the BeagleBoard Platform
Low Power High-Performance Computing on the BeagleBoard Platform
 
Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.Python Master Thesis Projects in UK.
Python Master Thesis Projects in UK.
 
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
DEEP-mon: Dynamic and Energy Efficient Power monitoring for container-based i...
 
High performance computing
High performance computingHigh performance computing
High performance computing
 

Similar to An Experiment-Driven Performance Model of Stream Processing Operators in Fog Computing Environments

Optical Switching in the Datacenter
Optical Switching in the DatacenterOptical Switching in the Datacenter
Optical Switching in the Datacenter
Kostas Katrinis
 
D031201021027
D031201021027D031201021027
D031201021027
inventionjournals
 
An evaluation of manet protocols in terms of tcp variants based on thier perf...
An evaluation of manet protocols in terms of tcp variants based on thier perf...An evaluation of manet protocols in terms of tcp variants based on thier perf...
An evaluation of manet protocols in terms of tcp variants based on thier perf...
eSAT Publishing House
 
Raminder kaur presentation_two
Raminder kaur presentation_twoRaminder kaur presentation_two
Raminder kaur presentation_two
ramikaurraminder
 
Hybrid networking and distribution
Hybrid networking and distribution Hybrid networking and distribution
Hybrid networking and distribution
vivek pratap singh
 
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
 
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd RamlyIPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
MyNOG
 
Parallelization of Coupled Cluster Code with OpenMP
Parallelization of Coupled Cluster Code with OpenMPParallelization of Coupled Cluster Code with OpenMP
Parallelization of Coupled Cluster Code with OpenMP
Anil Bohare
 
FrackingPaper
FrackingPaperFrackingPaper
FrackingPaper
Collin Purcell
 
Directive-based approach to Heterogeneous Computing
Directive-based approach to Heterogeneous ComputingDirective-based approach to Heterogeneous Computing
Directive-based approach to Heterogeneous Computing
Ruymán Reyes
 
Testing tool for an automated ticketing system
Testing tool for an automated ticketing systemTesting tool for an automated ticketing system
Testing tool for an automated ticketing system
VladimirZitoli
 
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
ijceronline
 
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
IRJET Journal
 
Life in the Fast Lane: A Line-Rate Linear Road
Life in the Fast Lane: A Line-Rate Linear RoadLife in the Fast Lane: A Line-Rate Linear Road
Life in the Fast Lane: A Line-Rate Linear Road
AJAY KHARAT
 
IRJET- The RTL Model of a Reconfigurable Pipelined MCM
IRJET- The RTL Model of a Reconfigurable Pipelined MCMIRJET- The RTL Model of a Reconfigurable Pipelined MCM
IRJET- The RTL Model of a Reconfigurable Pipelined MCM
IRJET Journal
 
IRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
IRJET- Performance Improvement of Wireless Network using Modern Simulation ToolsIRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
IRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
IRJET Journal
 
aaa.pptx
aaa.pptxaaa.pptx
Comparing Cpp And Erlang For Motorola Telecoms Software
Comparing Cpp And Erlang For Motorola Telecoms SoftwareComparing Cpp And Erlang For Motorola Telecoms Software
Comparing Cpp And Erlang For Motorola Telecoms Software
l xf
 
Uber mobility - High Performance Networking
Uber mobility - High Performance NetworkingUber mobility - High Performance Networking
Uber mobility - High Performance Networking
Dhaval Patel
 
Operationalizing Machine Learning: Serving ML Models
Operationalizing Machine Learning: Serving ML ModelsOperationalizing Machine Learning: Serving ML Models
Operationalizing Machine Learning: Serving ML Models
Lightbend
 

Similar to An Experiment-Driven Performance Model of Stream Processing Operators in Fog Computing Environments (20)

Optical Switching in the Datacenter
Optical Switching in the DatacenterOptical Switching in the Datacenter
Optical Switching in the Datacenter
 
D031201021027
D031201021027D031201021027
D031201021027
 
An evaluation of manet protocols in terms of tcp variants based on thier perf...
An evaluation of manet protocols in terms of tcp variants based on thier perf...An evaluation of manet protocols in terms of tcp variants based on thier perf...
An evaluation of manet protocols in terms of tcp variants based on thier perf...
 
Raminder kaur presentation_two
Raminder kaur presentation_twoRaminder kaur presentation_two
Raminder kaur presentation_two
 
Hybrid networking and distribution
Hybrid networking and distribution Hybrid networking and distribution
Hybrid networking and distribution
 
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
 
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd RamlyIPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
IPLC Analytic Dashboard - Mohd Rizal bin Mohd Ramly
 
Parallelization of Coupled Cluster Code with OpenMP
Parallelization of Coupled Cluster Code with OpenMPParallelization of Coupled Cluster Code with OpenMP
Parallelization of Coupled Cluster Code with OpenMP
 
FrackingPaper
FrackingPaperFrackingPaper
FrackingPaper
 
Directive-based approach to Heterogeneous Computing
Directive-based approach to Heterogeneous ComputingDirective-based approach to Heterogeneous Computing
Directive-based approach to Heterogeneous Computing
 
Testing tool for an automated ticketing system
Testing tool for an automated ticketing systemTesting tool for an automated ticketing system
Testing tool for an automated ticketing system
 
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
 
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
IRJET- Design and Implementation of Performance Evaluation of Routing Protoco...
 
Life in the Fast Lane: A Line-Rate Linear Road
Life in the Fast Lane: A Line-Rate Linear RoadLife in the Fast Lane: A Line-Rate Linear Road
Life in the Fast Lane: A Line-Rate Linear Road
 
IRJET- The RTL Model of a Reconfigurable Pipelined MCM
IRJET- The RTL Model of a Reconfigurable Pipelined MCMIRJET- The RTL Model of a Reconfigurable Pipelined MCM
IRJET- The RTL Model of a Reconfigurable Pipelined MCM
 
IRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
IRJET- Performance Improvement of Wireless Network using Modern Simulation ToolsIRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
IRJET- Performance Improvement of Wireless Network using Modern Simulation Tools
 
aaa.pptx
aaa.pptxaaa.pptx
aaa.pptx
 
Comparing Cpp And Erlang For Motorola Telecoms Software
Comparing Cpp And Erlang For Motorola Telecoms SoftwareComparing Cpp And Erlang For Motorola Telecoms Software
Comparing Cpp And Erlang For Motorola Telecoms Software
 
Uber mobility - High Performance Networking
Uber mobility - High Performance NetworkingUber mobility - High Performance Networking
Uber mobility - High Performance Networking
 
Operationalizing Machine Learning: Serving ML Models
Operationalizing Machine Learning: Serving ML ModelsOperationalizing Machine Learning: Serving ML Models
Operationalizing Machine Learning: Serving ML Models
 

More from FogGuru MSCA Project

Assignments
AssignmentsAssignments
The magical recipe for speaking in public
The magical recipe for speaking in publicThe magical recipe for speaking in public
The magical recipe for speaking in public
FogGuru MSCA Project
 
Introduction to the economics of innovation
Introduction to the economics of innovationIntroduction to the economics of innovation
Introduction to the economics of innovation
FogGuru MSCA Project
 
Introduction to entrepreneurial finances
Introduction to entrepreneurial financesIntroduction to entrepreneurial finances
Introduction to entrepreneurial finances
FogGuru MSCA Project
 
Financing Innovation and Intellectual property
Financing Innovation and Intellectual property Financing Innovation and Intellectual property
Financing Innovation and Intellectual property
FogGuru MSCA Project
 
Creating Competitive Advantage: Resource and Capabilities
Creating Competitive Advantage: Resource and Capabilities Creating Competitive Advantage: Resource and Capabilities
Creating Competitive Advantage: Resource and Capabilities
FogGuru MSCA Project
 
Business growth: material for exercises
Business growth: material for exercisesBusiness growth: material for exercises
Business growth: material for exercises
FogGuru MSCA Project
 
Business growth: material for discussions
Business growth: material for discussions  Business growth: material for discussions
Business growth: material for discussions
FogGuru MSCA Project
 
Scale-ups and large companies
Scale-ups and large companiesScale-ups and large companies
Scale-ups and large companies
FogGuru MSCA Project
 
Management, organization and leadership
Management, organization and leadershipManagement, organization and leadership
Management, organization and leadership
FogGuru MSCA Project
 
Key strategies for growth
Key strategies for growthKey strategies for growth
Key strategies for growth
FogGuru MSCA Project
 
Financing growth
Financing growthFinancing growth
Financing growth
FogGuru MSCA Project
 
Machine Learning: exercises
Machine Learning: exercises Machine Learning: exercises
Machine Learning: exercises
FogGuru MSCA Project
 
Introduction to Machine Learning
Introduction to Machine Learning Introduction to Machine Learning
Introduction to Machine Learning
FogGuru MSCA Project
 
Writing code well: tools, tips and tricks
Writing code well: tools, tips and tricks Writing code well: tools, tips and tricks
Writing code well: tools, tips and tricks
FogGuru MSCA Project
 
How to make a presentation
How to make a presentationHow to make a presentation
How to make a presentation
FogGuru MSCA Project
 
How to carry out bibliographic research
How to carry out bibliographic research How to carry out bibliographic research
How to carry out bibliographic research
FogGuru MSCA Project
 
Guidelines for empirical evaluations
Guidelines for empirical evaluationsGuidelines for empirical evaluations
Guidelines for empirical evaluations
FogGuru MSCA Project
 
Ethics and Personal Data
Ethics and Personal DataEthics and Personal Data
Ethics and Personal Data
FogGuru MSCA Project
 
Business case 1: Soft mobility in Rennes Metropole
Business case 1: Soft mobility in Rennes Metropole Business case 1: Soft mobility in Rennes Metropole
Business case 1: Soft mobility in Rennes Metropole
FogGuru MSCA Project
 

More from FogGuru MSCA Project (20)

Assignments
AssignmentsAssignments
Assignments
 
The magical recipe for speaking in public
The magical recipe for speaking in publicThe magical recipe for speaking in public
The magical recipe for speaking in public
 
Introduction to the economics of innovation
Introduction to the economics of innovationIntroduction to the economics of innovation
Introduction to the economics of innovation
 
Introduction to entrepreneurial finances
Introduction to entrepreneurial financesIntroduction to entrepreneurial finances
Introduction to entrepreneurial finances
 
Financing Innovation and Intellectual property
Financing Innovation and Intellectual property Financing Innovation and Intellectual property
Financing Innovation and Intellectual property
 
Creating Competitive Advantage: Resource and Capabilities
Creating Competitive Advantage: Resource and Capabilities Creating Competitive Advantage: Resource and Capabilities
Creating Competitive Advantage: Resource and Capabilities
 
Business growth: material for exercises
Business growth: material for exercisesBusiness growth: material for exercises
Business growth: material for exercises
 
Business growth: material for discussions
Business growth: material for discussions  Business growth: material for discussions
Business growth: material for discussions
 
Scale-ups and large companies
Scale-ups and large companiesScale-ups and large companies
Scale-ups and large companies
 
Management, organization and leadership
Management, organization and leadershipManagement, organization and leadership
Management, organization and leadership
 
Key strategies for growth
Key strategies for growthKey strategies for growth
Key strategies for growth
 
Financing growth
Financing growthFinancing growth
Financing growth
 
Machine Learning: exercises
Machine Learning: exercises Machine Learning: exercises
Machine Learning: exercises
 
Introduction to Machine Learning
Introduction to Machine Learning Introduction to Machine Learning
Introduction to Machine Learning
 
Writing code well: tools, tips and tricks
Writing code well: tools, tips and tricks Writing code well: tools, tips and tricks
Writing code well: tools, tips and tricks
 
How to make a presentation
How to make a presentationHow to make a presentation
How to make a presentation
 
How to carry out bibliographic research
How to carry out bibliographic research How to carry out bibliographic research
How to carry out bibliographic research
 
Guidelines for empirical evaluations
Guidelines for empirical evaluationsGuidelines for empirical evaluations
Guidelines for empirical evaluations
 
Ethics and Personal Data
Ethics and Personal DataEthics and Personal Data
Ethics and Personal Data
 
Business case 1: Soft mobility in Rennes Metropole
Business case 1: Soft mobility in Rennes Metropole Business case 1: Soft mobility in Rennes Metropole
Business case 1: Soft mobility in Rennes Metropole
 

Recently uploaded

Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
LINUS PROJECTS (INDIA)
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
CEPTES Software Inc
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
Management Institute of Skills Development
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
ldtexsolbl
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 

Recently uploaded (20)

Pigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending PlantPigging Unit Lubricant Oil Blending Plant
Pigging Unit Lubricant Oil Blending Plant
 
Salesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot WorkshopSalesforce AI & Einstein Copilot Workshop
Salesforce AI & Einstein Copilot Workshop
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Figma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdfFigma AI Design Generator_ In-Depth Review.pdf
Figma AI Design Generator_ In-Depth Review.pdf
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
Types of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technologyTypes of Weaving loom machine & it's technology
Types of Weaving loom machine & it's technology
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 

An Experiment-Driven Performance Model of Stream Processing Operators in Fog Computing Environments

  • 1. An Experiment-Driven Performance Model of Stream Processing Operators in Fog Computing Environments Hamidreza Arkian1 , Guillaume Pierre1 , Johan Tordsson2 , Erik Elmroth2 1 University of Rennes1/IRISA, France 2 Elastisys AB, Sweden SAC’20 - March 30-April 3, 2020 - Brno, Czech Republic
  • 4. 4/16 Challenges Apache Flink Low Throughput!! Low Bandwidth!! Cost!! Continuously generating stream of data with high rate Latency-sensitive applications
  • 6. 6/16 Operator 2 Stream processing in Fog environment Source Operator 4 Operator 1 Sink Operator 2 Operator 3 Source Operator 4 Operator 1 Sink Operator 2 Operator 3 Source Operator 2 Logical graph of DSP Workflow execution model
  • 7. 7/16 Operator 2 Stream processing in geo-distributed environments Source Operator 4 Operator 1 Sink Operator 2 Operator 3 Source Operator 4 Operator 1 Sink Operator 2 Operator 3 Source Operator 2 Logical graph of DSP Workflow execution model Deployment in Fog geo-distributed environment Sink Operator 1 Op2 Replica2 Operator 3 Operator 4 Source Op2 Replica1 Op2 Replica3 Source
  • 8. 8/16 Challenges ➢ Understanding the performance of a geo-distributed stream processing application is difficult. ➢ Any configuration decision can have a significant impact on performance.
  • 9. 9/16 Experimental setup ➢ Emulation of a real fog platform o 32-core server ≈ 16 fog nodes (2 cores/node) o Emulated network latencies o Apache Flink ➢ Test Application o Input stream of 100,000 Tuple2 records o The operator calls the Fibonacci function Fib(24) upon every processed record ➢ Performance metric: o Processing Time (PT)
  • 10. 10/16 Modeling operator replication ➢ n operator replicas should in principle process data n times faster than a single replica ➢ α represents the computation capacity of a single node. ➢ We can determine the value of α based on one measurement Experiment Model
  • 11. 11/16 Considering heterogeneous network delays ➢ Network delays between data sources and operator replicas slow down the whole system. ➢ When the network delays are heterogeneous, the dominating one is the greatest one (NDmax ). ➢ γ represents the impact of network delays on overall performance. ➢ We can determine both α and γ based on two measurements Experiment Model
  • 12. 12/16 Improving the model’s accuracy ➢ Operator replication incurs some amount of parallelization inefficiency ➢ The speedup with n nodes is usually a little less than n ➢ 𝛽 represents Flink’s parallelization inefficiency ➢ We can determine α, 𝛽 and γ based on three or more measurements Experiment Model
  • 13. 13/16 Prediction accuracy Accuracy metric: 𝑀𝐴𝑃𝐸 4 measurements, 2.0% accuracy
  • 14. 14/16 What about modeling an entire (simple) workflow? ➢ The throughput of an entire workflow is determined by the slowest operator 𝛱Workflow = max(𝛱Map+KeyBy , 𝛱Reduce ) Experiment ModelWorkflow
  • 15. 15/16 Can we reuse the parameters instead of multiple measurements? ➢ 𝛼 cannot be reused because it is specific to the computation complexity of one operator. ➢ β and γ capture properties that are independent from the nature of the computation carried out by the operator. ➢ β and γ values of one operator’s model might be reused for other operators’ models. Calibrated model for Operator 1 Uncalibrated model for Operator 2 𝛼1 β1 γ1 𝛼2 β1 γ1
  • 16. 16/16 Conclusions ➢ Heterogeneous network characteristics make it difficult to understand the performance of stream processing engines in geo-distributed environments. ➢ A predictive performance model for Apache Flink operators that is backed by experimental measurements and evaluations was proposed. ➢ The model predictions are accurate within ±2% of the actual values. Hamidreza Arkian hamidreza.arkian@irisa.fr Acknowledgment This work is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 765452. The information and views set out in this publication are those of the author(s) and do not necessarily reflect the official opinion of the European Union. Neither the European Union institutions and bodies nor any person acting on their behalf may be held responsible for the use which may be made of the information contained therein. Training the next generation of European Fog computing experts http://www.fogguru.eu/