SlideShare a Scribd company logo
1 of 26
Download to read offline
GaruaGeo: Global Scale Data Aggregation
in Hybrid Edge and Cloud Computing
Environments
Ot´avio Carvalho, Eduardo Roloff, Philippe O. A. Navaux
Informatics Institute – Federal University of Rio Grande do Sul
9th International Conference on Cloud Computing and Services Science – CLOSER 2019
Table of contents
1. Introduction
2. State of the art
3. Architecture
4. Evaluation
5. Conclusion and Future Works
2
Introduction – Motivation
• Internet of Things social and economic impact [1].
• By 2025, researchers estimate that the IoT will have a potential economic
impact of 11 trillion per year – which would be equivalent to about 11% of
the world economy. They also expect that one trillion IoT devices will be
deployed by 2025.
• Technologies created for IoT are driving computing toward
dispersion [4].
• Multi-Access Edge Computing
• Fog Computing
• Cloudlets
3
Introduction – Main goals
• Explore the potential performance improvements of moving
computation from cloud to edge.
1. Evaluate to what extent is possible to move the workload from cloud to
edge nodes.
2. Explore the limits of the proposed application architecture in terms of
latency and throughput.
3. Create strategies to reduce the amount of data sent to the cloud.
4
State of the art – Distributed Computing
• Since the 60s, computing has alternated between centralization
and decentralization [4]
• 60s & 70s: Centralized approaches like batch processing and
timesharing.
• 80s & 90s: Decentralization through the rise of personal computing.
• Mid-00s: Centralized approaches like cloud computing.
• Nowadays: The rise of fog computing and edge computing.
5
State of the art – Smart Grids
• For 100 years, there has been no change in the basic structure of
the electrical power grid. Experiences have shown that the
hierarchical, centrally controlled grid of the 20th Century is
ill-suited to the needs of the 21st Century. To address the
challenges of the existing power grid, the new concept of smart
grid has emerged [2].
• Smart Grids are the use case scenario for the testbed application
developed in this work.
• Low latency
• Needs to communicate energy consumption quickly to the grid operator
machines in order to balance demand and energy generation.
• High scalability
• Requires communication across large geographic regions to connect large
numbers of households.
6
State of the art – Related Works
Name Cloud Edge Mobility Large Scale Hardware Agnostic
GaruaGeo (this work) • • • •
ENORM • • •
RT-SANE • • •
Tarneberg et al. • • •
HomeCloud • • •
CloudAware • • •
FemtoClouds • •
REPLISOM • •
Cumulus • • •
ParaDrop • •
EdgeIoT • •
7
GaruaGeo – Proposal
• An extension to the Garua architecture.
• Aggregator nodes are placed geographically close to its companion
edge nodes.
• Improve the previous architecture by considering latency issues on
the design.
• Evaluate the platform in a globally distributed scenario.
8
GaruaGeo – Architecture
• Four-layered architecture
• Cloud layer
• Aggregates data from aggregator nodes.
• More performance (VMs on Azure).
• High latency.
• Aggregator layer
• Aggregates data from edge nodes.
• Nodes are placed geographically close to groups of edge nodes.
• Medium latency.
• Intermediate performance (Cloud nodes or physical hardware).
• Edge layer
• Aggregates data from sensors.
• Low latency.
• Less performance (Raspberry Pi Zero W).
• Sensor layer
• Provides events to edge nodes.
• Bluetooth, LTE, WiFi, etc.
9
GaruaGeo – Architecture
VM VM VM
VM
Cloud
Layer
Aggregator Aggregator
Aggregator
Layer
Edge Node Edge Node Edge Node
Edge
Layer
Sensor Sensor Sensor Sensor Sensor
Sensor
Layer
10
GaruaGeo – Aggregators Evaluation – Methodology
• The main goal is to explore the impact on throughput of adding a new
network layer.
• Evaluate the GaruaGeo architecture with a single aggregator in
comparison to Garua architecture.
• Explore distinct sets of edge nodes and messages sizes impact on
throughput.
• Evaluate potential performance gains of using multiple aggregator
nodes in the same geographic region to distribute the load from
multiple edge nodes.
11
GaruaGeo – Aggregators Evaluation
• Evaluate the throughput obtained when using an aggregation layer.
• Varying groups of message windows, from 1 to 1000 messages per window.
baseline aggregator
105
106
107
108
Execution type
Throughput(QPS)
1 10 100 1000
12
GaruaGeo – Aggregators Evaluation
• The main goal of this experiment is to visualize the aggregated impact of message
windowing and number of nodes on throughput.
• Aggregators are limited by the number of messages they can process from edge nodes in
a period of time.
• Message windows combines a larger set of message into a single message and increases
the overall throughput.
13
GaruaGeo – Aggregators Evaluation
• Explore the possibility of adding multiple aggregators in a given geographic region to avoid
overloading of a single aggregator.
• Splitting the load of a given aggregator node appears to produce substantial impact on
throughput.
• In this experiment, the same amount of edge nodes is distributed among distinct sets of
aggregator nodes, from 1 to 8 aggregator nodes.
1 2 4 8
0
50,000
1 ·105
Aggregators (1 to 8)
Throughput(QPS)
14
GaruaGeo – Geo-distributed Analysis – Methodology
• The main goal is to understand the behavior of the architecture in a
global scale deployment scenario.
• Evaluate the throughput on multiple regions to understand potential
performance discrepancies between regions (Microsoft Azure
datacenters).
• Evaluate the achievable throughput in a global scale deployment,
using up to 15 regions and 1366 machines across the globe.
15
GaruaGeo – Geo-distributed Analysis
• In this experiment, it is evaluated the potential performance discrepancies between distinct
Microsoft Azure regions (datacenters).
• It was not found significant discrepancies in performance (in terms of throughput) on the 5
distinct regions analysed.
16
GaruaGeo – Geo-distributed Analysis
• The scale of the global deployment used in the experiment (datacenters).
• 15 geographic regions (datacenters on Microsoft Azure).
• A single global cloud node.
• 15 aggregator nodes (one in each region).
• 90 edge nodes (in each region).
17
GaruaGeo – Geo-distributed Analysis
• Evaluation of the maximum achievable throughput of the platform across geographic
regions.
• 15 geographic regions (datacenters on Microsoft Azure).
• Message windows of 1000 messages.
• A total of 1366 machines on the scenario with 15 regions.
• Aggregation rates above 400 million measurements per second on the scenario with 15
regions.
1000
108.2
108.4
108.6
Groups of regions (5 to 15 regions)
Throughput(QPS)
5 10 15
18
Conclusions
• Evaluated the platform in a geo-distributed environment with
real-world latencies.
• Aggregation rates above 400 million measurements per second.
• Large scale evaluation using virtual machines on 15 geographic regions
across the globe on Microsoft Azure platform.
• A total of 1366 machines in the largest evaluation scenario.
• Improved performance by placing aggregators geographically close to
edge nodes.
• Reduced communication with the cloud by aggregating data at edge
level.
19
Future Works
• Explore distinct techniques for data scheduling, windowing and
aggregation at the edge of the network.
• Evolve the testbed application and its middleware into a generic
framework for distributed data processing.
• Apply other communication protocols to the platform.
20
Thanks! Questions?
21
References I
R. Buyya and A. V. Dastjerdi.
Internet of Things: Principles and paradigms.
Elsevier, 2016.
V. C. G¨ung¨or, D. Sahin, T. Kocak, S. Erg¨ut, C. Buccella, C. Cecati, and
G. P. Hancke.
Smart Grid Technologies: Communication Technologies and
Standards.
Industrial informatics, IEEE transactions on, 7(4):529–539, 2011.
Reuters.
U.S. Smart Grid to Cost Billions, Save Trillions, 2011.
M. Satyanarayanan.
The Emergence of Edge Computing.
Computer, 2017.
22
Appendix: Dataset
1. The dataset used to evaluate the platform originates from the 8th ACM
International Conference on Distributed Event-Based Systems (DEBS 2014)
2. The data file contains over 4055 Millions of measurements for 2125 plugs
distributed across 40 houses, for a total amount of 136 GB
3. Generated measurements cover a period of one month, from Sept. 1st, 2013,
00:00:00, to Sept. 30th, 2013, 23:59:59
23
Appendix: GaruaGeo – Cloud layer
Parameter Description
Instance Type Basic A3 (4 cores, 7 GB RAM)
Operating System Ubuntu 16.04 LTS
Golang version 1.8
GRPC version 1.3.0-dev
Protocol Buffers version 3.2.0
24
Appendix: GaruaGeo – Aggregator layer
Parameter Description
Instance Type Standard DS2 v2 (2 cores, 7 GB RAM)
Operating System Ubuntu 16.04 LTS
Golang version 1.8
GRPC version 1.3.0-dev
Protocol Buffers version 3.2.0
25
Appendix: GaruaGeo – Edge layer
Parameter Description
Instance Type Standard DS1 v2 (1 cores, 3.5 GB RAM)
Operating System Ubuntu 16.04 LTS
Golang version 1.8
GRPC version 1.3.0-dev
Protocol Buffers version 3.2.0
26

More Related Content

What's hot

A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICS
A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICSA RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICS
A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICSIJCNCJournal
 
Towards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingTowards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingRECAP Project
 
RECAP Project Overview
RECAP Project OverviewRECAP Project Overview
RECAP Project OverviewRECAP Project
 
Predictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksPredictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksJustin Brandenburg
 
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scale
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scalePEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scale
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scaleDandong Yin
 
RECAP: The Simulation Approach
RECAP: The Simulation ApproachRECAP: The Simulation Approach
RECAP: The Simulation ApproachRECAP Project
 
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...Monica Vitali
 
Impact of Grid Computing on Network Operators and HW Vendors
Impact of Grid Computing on Network Operators and HW VendorsImpact of Grid Computing on Network Operators and HW Vendors
Impact of Grid Computing on Network Operators and HW VendorsTal Lavian Ph.D.
 
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...Facultad de Informática UCM
 
ACACES 2019: Towards Energy Efficient Deep Learning
ACACES 2019: Towards Energy Efficient Deep LearningACACES 2019: Towards Energy Efficient Deep Learning
ACACES 2019: Towards Energy Efficient Deep LearningLEGATO project
 
Federating Infrastructure as a Service cloud computing systems to create a un...
Federating Infrastructure as a Service cloud computing systems to create a un...Federating Infrastructure as a Service cloud computing systems to create a un...
Federating Infrastructure as a Service cloud computing systems to create a un...David Wallom
 
A modeling approach for cloud infrastructure planning considering dependabili...
A modeling approach for cloud infrastructure planning considering dependabili...A modeling approach for cloud infrastructure planning considering dependabili...
A modeling approach for cloud infrastructure planning considering dependabili...ieeepondy
 
Sensors - The Sparkplug in the Engine of the Internet of Things
Sensors - The Sparkplug in the Engine of the Internet of ThingsSensors - The Sparkplug in the Engine of the Internet of Things
Sensors - The Sparkplug in the Engine of the Internet of ThingsRECAP Project
 
RECAP Project Overview
RECAP Project OverviewRECAP Project Overview
RECAP Project OverviewRECAP Project
 
Challenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing PlatformsChallenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing PlatformsFrederic Desprez
 
Optimizing Monitorability of Multi-cloud Applications
Optimizing Monitorability of Multi-cloud ApplicationsOptimizing Monitorability of Multi-cloud Applications
Optimizing Monitorability of Multi-cloud ApplicationsMonica Vitali
 

What's hot (20)

A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICS
A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICSA RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICS
A RAPID DEPLOYMENT BIG DATA COMPUTING PLATFORM FOR CLOUD ROBOTICS
 
DITAS@CCW2018
DITAS@CCW2018DITAS@CCW2018
DITAS@CCW2018
 
Towards the Intelligent Internet of Everything
Towards the Intelligent Internet of EverythingTowards the Intelligent Internet of Everything
Towards the Intelligent Internet of Everything
 
EnBIS 2016 opening
EnBIS 2016 openingEnBIS 2016 opening
EnBIS 2016 opening
 
RECAP Project Overview
RECAP Project OverviewRECAP Project Overview
RECAP Project Overview
 
Predictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural NetworksPredictive Maintenance Using Recurrent Neural Networks
Predictive Maintenance Using Recurrent Neural Networks
 
Umu seminar 02-2019
Umu seminar 02-2019Umu seminar 02-2019
Umu seminar 02-2019
 
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scale
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scalePEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scale
PEARC17: a CyberGIS-Jupyter framework for geospatial analytics at scale
 
RECAP: The Simulation Approach
RECAP: The Simulation ApproachRECAP: The Simulation Approach
RECAP: The Simulation Approach
 
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...
An Adaptive Monitoring Service exploiting Data Correlations in Fog Computing ...
 
Impact of Grid Computing on Network Operators and HW Vendors
Impact of Grid Computing on Network Operators and HW VendorsImpact of Grid Computing on Network Operators and HW Vendors
Impact of Grid Computing on Network Operators and HW Vendors
 
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...
Fast and energy-efficient eNVM based memory organisation at L3-L1 layers for ...
 
ACACES 2019: Towards Energy Efficient Deep Learning
ACACES 2019: Towards Energy Efficient Deep LearningACACES 2019: Towards Energy Efficient Deep Learning
ACACES 2019: Towards Energy Efficient Deep Learning
 
3. the grid new infrastructure
3. the grid new infrastructure3. the grid new infrastructure
3. the grid new infrastructure
 
Federating Infrastructure as a Service cloud computing systems to create a un...
Federating Infrastructure as a Service cloud computing systems to create a un...Federating Infrastructure as a Service cloud computing systems to create a un...
Federating Infrastructure as a Service cloud computing systems to create a un...
 
A modeling approach for cloud infrastructure planning considering dependabili...
A modeling approach for cloud infrastructure planning considering dependabili...A modeling approach for cloud infrastructure planning considering dependabili...
A modeling approach for cloud infrastructure planning considering dependabili...
 
Sensors - The Sparkplug in the Engine of the Internet of Things
Sensors - The Sparkplug in the Engine of the Internet of ThingsSensors - The Sparkplug in the Engine of the Internet of Things
Sensors - The Sparkplug in the Engine of the Internet of Things
 
RECAP Project Overview
RECAP Project OverviewRECAP Project Overview
RECAP Project Overview
 
Challenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing PlatformsChallenges and Issues of Next Cloud Computing Platforms
Challenges and Issues of Next Cloud Computing Platforms
 
Optimizing Monitorability of Multi-cloud Applications
Optimizing Monitorability of Multi-cloud ApplicationsOptimizing Monitorability of Multi-cloud Applications
Optimizing Monitorability of Multi-cloud Applications
 

Similar to GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

Computation grid as a connected world
Computation grid as a connected worldComputation grid as a connected world
Computation grid as a connected worldijcsa
 
WF-IOT-2014, Seoul, Korea, 06 March 2014
WF-IOT-2014, Seoul, Korea, 06 March 2014WF-IOT-2014, Seoul, Korea, 06 March 2014
WF-IOT-2014, Seoul, Korea, 06 March 2014Charith Perera
 
Show and Tell - Data and Digitalisation, Digital Twins.pdf
Show and Tell - Data and Digitalisation, Digital Twins.pdfShow and Tell - Data and Digitalisation, Digital Twins.pdf
Show and Tell - Data and Digitalisation, Digital Twins.pdfSIFOfgem
 
Grid and Cloud Computing Lecture-2a.pptx
Grid and Cloud Computing Lecture-2a.pptxGrid and Cloud Computing Lecture-2a.pptx
Grid and Cloud Computing Lecture-2a.pptxDrAdeelAkram2
 
Inroduction to grid computing by gargi shankar verma
Inroduction to grid computing by gargi shankar vermaInroduction to grid computing by gargi shankar verma
Inroduction to grid computing by gargi shankar vermagargishankar1981
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...Otávio Carvalho
 
A Case Study On Implementation Of Grid Computing To Academic Institution
A Case Study On Implementation Of Grid Computing To Academic InstitutionA Case Study On Implementation Of Grid Computing To Academic Institution
A Case Study On Implementation Of Grid Computing To Academic InstitutionArlene Smith
 
Fog Computing Platform
Fog Computing PlatformFog Computing Platform
Fog Computing Platform霈萱 蔡
 
Supreet swaran's grid
Supreet swaran's gridSupreet swaran's grid
Supreet swaran's gridSupreet Singh
 
Networking Challenges for the Next Decade
Networking Challenges for the Next DecadeNetworking Challenges for the Next Decade
Networking Challenges for the Next DecadeOpen Networking Summit
 
IRJET- Cost Effective Workflow Scheduling in Bigdata
IRJET-  	  Cost Effective Workflow Scheduling in BigdataIRJET-  	  Cost Effective Workflow Scheduling in Bigdata
IRJET- Cost Effective Workflow Scheduling in BigdataIRJET Journal
 
Gridcomputingppt
GridcomputingpptGridcomputingppt
Gridcomputingpptnavjasser
 
(R)evolution of the computing continuum - A few challenges
(R)evolution of the computing continuum  - A few challenges(R)evolution of the computing continuum  - A few challenges
(R)evolution of the computing continuum - A few challengesFrederic Desprez
 
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degree
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degreeThe UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degree
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degreePradeeban Kathiravelu, Ph.D.
 
Optimization of Fog computing for Industrial IoT applications
Optimization of Fog computing for Industrial IoT applicationsOptimization of Fog computing for Industrial IoT applications
Optimization of Fog computing for Industrial IoT applicationsSabelo Dlamini
 
Panel: NRP Science Impacts​
Panel: NRP Science Impacts​Panel: NRP Science Impacts​
Panel: NRP Science Impacts​Larry Smarr
 

Similar to GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments (20)

Computation grid as a connected world
Computation grid as a connected worldComputation grid as a connected world
Computation grid as a connected world
 
WF-IOT-2014, Seoul, Korea, 06 March 2014
WF-IOT-2014, Seoul, Korea, 06 March 2014WF-IOT-2014, Seoul, Korea, 06 March 2014
WF-IOT-2014, Seoul, Korea, 06 March 2014
 
Show and Tell - Data and Digitalisation, Digital Twins.pdf
Show and Tell - Data and Digitalisation, Digital Twins.pdfShow and Tell - Data and Digitalisation, Digital Twins.pdf
Show and Tell - Data and Digitalisation, Digital Twins.pdf
 
Grid and Cloud Computing Lecture-2a.pptx
Grid and Cloud Computing Lecture-2a.pptxGrid and Cloud Computing Lecture-2a.pptx
Grid and Cloud Computing Lecture-2a.pptx
 
Grid computing
Grid computingGrid computing
Grid computing
 
Inroduction to grid computing by gargi shankar verma
Inroduction to grid computing by gargi shankar vermaInroduction to grid computing by gargi shankar verma
Inroduction to grid computing by gargi shankar verma
 
2. the grid
2. the grid2. the grid
2. the grid
 
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...
 
A Case Study On Implementation Of Grid Computing To Academic Institution
A Case Study On Implementation Of Grid Computing To Academic InstitutionA Case Study On Implementation Of Grid Computing To Academic Institution
A Case Study On Implementation Of Grid Computing To Academic Institution
 
Fog Computing Platform
Fog Computing PlatformFog Computing Platform
Fog Computing Platform
 
Supreet swaran's grid
Supreet swaran's gridSupreet swaran's grid
Supreet swaran's grid
 
Networking Challenges for the Next Decade
Networking Challenges for the Next DecadeNetworking Challenges for the Next Decade
Networking Challenges for the Next Decade
 
IRJET- Cost Effective Workflow Scheduling in Bigdata
IRJET-  	  Cost Effective Workflow Scheduling in BigdataIRJET-  	  Cost Effective Workflow Scheduling in Bigdata
IRJET- Cost Effective Workflow Scheduling in Bigdata
 
Gridcomputingppt
GridcomputingpptGridcomputingppt
Gridcomputingppt
 
Distributed system.pptx
Distributed system.pptxDistributed system.pptx
Distributed system.pptx
 
(R)evolution of the computing continuum - A few challenges
(R)evolution of the computing continuum  - A few challenges(R)evolution of the computing continuum  - A few challenges
(R)evolution of the computing continuum - A few challenges
 
Grid computiing
Grid computiingGrid computiing
Grid computiing
 
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degree
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degreeThe UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degree
The UCLouvain Public Defense of my EMJD-DC Double Doctorate Ph.D. degree
 
Optimization of Fog computing for Industrial IoT applications
Optimization of Fog computing for Industrial IoT applicationsOptimization of Fog computing for Industrial IoT applications
Optimization of Fog computing for Industrial IoT applications
 
Panel: NRP Science Impacts​
Panel: NRP Science Impacts​Panel: NRP Science Impacts​
Panel: NRP Science Impacts​
 

More from Otávio Carvalho

Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Otávio Carvalho
 
Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Otávio Carvalho
 
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Otávio Carvalho
 
Apache Kafka - Free Friday
Apache Kafka - Free FridayApache Kafka - Free Friday
Apache Kafka - Free FridayOtávio Carvalho
 
A Survey of the State-of-the-art in Event Processing
A Survey of the State-of-the-art in Event ProcessingA Survey of the State-of-the-art in Event Processing
A Survey of the State-of-the-art in Event ProcessingOtávio Carvalho
 
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemAnálise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemOtávio Carvalho
 
Utilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemUtilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemOtávio Carvalho
 

More from Otávio Carvalho (7)

Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018Non-Kafkaesque Apache Kafka - Yottabyte 2018
Non-Kafkaesque Apache Kafka - Yottabyte 2018
 
Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017Stream Processing - ThoughtWorks Architecture Group - 2017
Stream Processing - ThoughtWorks Architecture Group - 2017
 
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
Stream Processing: Uma visão geral - TDC Porto Alegre / FISL 17
 
Apache Kafka - Free Friday
Apache Kafka - Free FridayApache Kafka - Free Friday
Apache Kafka - Free Friday
 
A Survey of the State-of-the-art in Event Processing
A Survey of the State-of-the-art in Event ProcessingA Survey of the State-of-the-art in Event Processing
A Survey of the State-of-the-art in Event Processing
 
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em NuvemAnálise e Caracterização das Novas Ferramentas para Computação em Nuvem
Análise e Caracterização das Novas Ferramentas para Computação em Nuvem
 
Utilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvemUtilização de traços de execução para migração de aplicações para a nuvem
Utilização de traços de execução para migração de aplicações para a nuvem
 

Recently uploaded

eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorTier1 app
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...Bert Jan Schrijver
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUsamruddhijedgule2004
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shardsChristopher Curtin
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Piyovi
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...OnePlan Solutions
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsJean Silva
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxRemote DBA Services
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdfAndrey Devyatkin
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dbaRemote DBA Services
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flinkconfluent
 

Recently uploaded (20)

eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
 
2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards2024 DevNexus Patterns for Resiliency: Shuffle shards
2024 DevNexus Patterns for Resiliency: Shuffle shards
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
Transform your Corporate Strategy Office - Harness OnePlan’s Strategic Portfo...
 
Strategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero resultsStrategies for using alternative queries to mitigate zero results
Strategies for using alternative queries to mitigate zero results
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptx
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
2024-04-09 - From Complexity to Clarity - AWS Summit AMS.pdf
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dba
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 

GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments

  • 1. GaruaGeo: Global Scale Data Aggregation in Hybrid Edge and Cloud Computing Environments Ot´avio Carvalho, Eduardo Roloff, Philippe O. A. Navaux Informatics Institute – Federal University of Rio Grande do Sul 9th International Conference on Cloud Computing and Services Science – CLOSER 2019
  • 2. Table of contents 1. Introduction 2. State of the art 3. Architecture 4. Evaluation 5. Conclusion and Future Works 2
  • 3. Introduction – Motivation • Internet of Things social and economic impact [1]. • By 2025, researchers estimate that the IoT will have a potential economic impact of 11 trillion per year – which would be equivalent to about 11% of the world economy. They also expect that one trillion IoT devices will be deployed by 2025. • Technologies created for IoT are driving computing toward dispersion [4]. • Multi-Access Edge Computing • Fog Computing • Cloudlets 3
  • 4. Introduction – Main goals • Explore the potential performance improvements of moving computation from cloud to edge. 1. Evaluate to what extent is possible to move the workload from cloud to edge nodes. 2. Explore the limits of the proposed application architecture in terms of latency and throughput. 3. Create strategies to reduce the amount of data sent to the cloud. 4
  • 5. State of the art – Distributed Computing • Since the 60s, computing has alternated between centralization and decentralization [4] • 60s & 70s: Centralized approaches like batch processing and timesharing. • 80s & 90s: Decentralization through the rise of personal computing. • Mid-00s: Centralized approaches like cloud computing. • Nowadays: The rise of fog computing and edge computing. 5
  • 6. State of the art – Smart Grids • For 100 years, there has been no change in the basic structure of the electrical power grid. Experiences have shown that the hierarchical, centrally controlled grid of the 20th Century is ill-suited to the needs of the 21st Century. To address the challenges of the existing power grid, the new concept of smart grid has emerged [2]. • Smart Grids are the use case scenario for the testbed application developed in this work. • Low latency • Needs to communicate energy consumption quickly to the grid operator machines in order to balance demand and energy generation. • High scalability • Requires communication across large geographic regions to connect large numbers of households. 6
  • 7. State of the art – Related Works Name Cloud Edge Mobility Large Scale Hardware Agnostic GaruaGeo (this work) • • • • ENORM • • • RT-SANE • • • Tarneberg et al. • • • HomeCloud • • • CloudAware • • • FemtoClouds • • REPLISOM • • Cumulus • • • ParaDrop • • EdgeIoT • • 7
  • 8. GaruaGeo – Proposal • An extension to the Garua architecture. • Aggregator nodes are placed geographically close to its companion edge nodes. • Improve the previous architecture by considering latency issues on the design. • Evaluate the platform in a globally distributed scenario. 8
  • 9. GaruaGeo – Architecture • Four-layered architecture • Cloud layer • Aggregates data from aggregator nodes. • More performance (VMs on Azure). • High latency. • Aggregator layer • Aggregates data from edge nodes. • Nodes are placed geographically close to groups of edge nodes. • Medium latency. • Intermediate performance (Cloud nodes or physical hardware). • Edge layer • Aggregates data from sensors. • Low latency. • Less performance (Raspberry Pi Zero W). • Sensor layer • Provides events to edge nodes. • Bluetooth, LTE, WiFi, etc. 9
  • 10. GaruaGeo – Architecture VM VM VM VM Cloud Layer Aggregator Aggregator Aggregator Layer Edge Node Edge Node Edge Node Edge Layer Sensor Sensor Sensor Sensor Sensor Sensor Layer 10
  • 11. GaruaGeo – Aggregators Evaluation – Methodology • The main goal is to explore the impact on throughput of adding a new network layer. • Evaluate the GaruaGeo architecture with a single aggregator in comparison to Garua architecture. • Explore distinct sets of edge nodes and messages sizes impact on throughput. • Evaluate potential performance gains of using multiple aggregator nodes in the same geographic region to distribute the load from multiple edge nodes. 11
  • 12. GaruaGeo – Aggregators Evaluation • Evaluate the throughput obtained when using an aggregation layer. • Varying groups of message windows, from 1 to 1000 messages per window. baseline aggregator 105 106 107 108 Execution type Throughput(QPS) 1 10 100 1000 12
  • 13. GaruaGeo – Aggregators Evaluation • The main goal of this experiment is to visualize the aggregated impact of message windowing and number of nodes on throughput. • Aggregators are limited by the number of messages they can process from edge nodes in a period of time. • Message windows combines a larger set of message into a single message and increases the overall throughput. 13
  • 14. GaruaGeo – Aggregators Evaluation • Explore the possibility of adding multiple aggregators in a given geographic region to avoid overloading of a single aggregator. • Splitting the load of a given aggregator node appears to produce substantial impact on throughput. • In this experiment, the same amount of edge nodes is distributed among distinct sets of aggregator nodes, from 1 to 8 aggregator nodes. 1 2 4 8 0 50,000 1 ·105 Aggregators (1 to 8) Throughput(QPS) 14
  • 15. GaruaGeo – Geo-distributed Analysis – Methodology • The main goal is to understand the behavior of the architecture in a global scale deployment scenario. • Evaluate the throughput on multiple regions to understand potential performance discrepancies between regions (Microsoft Azure datacenters). • Evaluate the achievable throughput in a global scale deployment, using up to 15 regions and 1366 machines across the globe. 15
  • 16. GaruaGeo – Geo-distributed Analysis • In this experiment, it is evaluated the potential performance discrepancies between distinct Microsoft Azure regions (datacenters). • It was not found significant discrepancies in performance (in terms of throughput) on the 5 distinct regions analysed. 16
  • 17. GaruaGeo – Geo-distributed Analysis • The scale of the global deployment used in the experiment (datacenters). • 15 geographic regions (datacenters on Microsoft Azure). • A single global cloud node. • 15 aggregator nodes (one in each region). • 90 edge nodes (in each region). 17
  • 18. GaruaGeo – Geo-distributed Analysis • Evaluation of the maximum achievable throughput of the platform across geographic regions. • 15 geographic regions (datacenters on Microsoft Azure). • Message windows of 1000 messages. • A total of 1366 machines on the scenario with 15 regions. • Aggregation rates above 400 million measurements per second on the scenario with 15 regions. 1000 108.2 108.4 108.6 Groups of regions (5 to 15 regions) Throughput(QPS) 5 10 15 18
  • 19. Conclusions • Evaluated the platform in a geo-distributed environment with real-world latencies. • Aggregation rates above 400 million measurements per second. • Large scale evaluation using virtual machines on 15 geographic regions across the globe on Microsoft Azure platform. • A total of 1366 machines in the largest evaluation scenario. • Improved performance by placing aggregators geographically close to edge nodes. • Reduced communication with the cloud by aggregating data at edge level. 19
  • 20. Future Works • Explore distinct techniques for data scheduling, windowing and aggregation at the edge of the network. • Evolve the testbed application and its middleware into a generic framework for distributed data processing. • Apply other communication protocols to the platform. 20
  • 22. References I R. Buyya and A. V. Dastjerdi. Internet of Things: Principles and paradigms. Elsevier, 2016. V. C. G¨ung¨or, D. Sahin, T. Kocak, S. Erg¨ut, C. Buccella, C. Cecati, and G. P. Hancke. Smart Grid Technologies: Communication Technologies and Standards. Industrial informatics, IEEE transactions on, 7(4):529–539, 2011. Reuters. U.S. Smart Grid to Cost Billions, Save Trillions, 2011. M. Satyanarayanan. The Emergence of Edge Computing. Computer, 2017. 22
  • 23. Appendix: Dataset 1. The dataset used to evaluate the platform originates from the 8th ACM International Conference on Distributed Event-Based Systems (DEBS 2014) 2. The data file contains over 4055 Millions of measurements for 2125 plugs distributed across 40 houses, for a total amount of 136 GB 3. Generated measurements cover a period of one month, from Sept. 1st, 2013, 00:00:00, to Sept. 30th, 2013, 23:59:59 23
  • 24. Appendix: GaruaGeo – Cloud layer Parameter Description Instance Type Basic A3 (4 cores, 7 GB RAM) Operating System Ubuntu 16.04 LTS Golang version 1.8 GRPC version 1.3.0-dev Protocol Buffers version 3.2.0 24
  • 25. Appendix: GaruaGeo – Aggregator layer Parameter Description Instance Type Standard DS2 v2 (2 cores, 7 GB RAM) Operating System Ubuntu 16.04 LTS Golang version 1.8 GRPC version 1.3.0-dev Protocol Buffers version 3.2.0 25
  • 26. Appendix: GaruaGeo – Edge layer Parameter Description Instance Type Standard DS1 v2 (1 cores, 3.5 GB RAM) Operating System Ubuntu 16.04 LTS Golang version 1.8 GRPC version 1.3.0-dev Protocol Buffers version 3.2.0 26