The data streaming paradigm and its use in Fog architectures

The data streaming paradigm
and its use in Fog architectures
Vincenzo Gulisano, Ph.D.
EBSIS Summer School 2016
1

Agenda
• Introduction: Fog architectures
• Data streaming basics
• Trade-offs and challenges in Fog streaming analysis
• Conclusions
• Bibliography
2The data streaming paradigm and its use in Fog architectures

Agenda
• Conclusions
• Bibliography

Introduction: Fog architectures
Advanced Metering Infrastructures (AMIs)
[1,2,3,4]
Vehicular Networks (VNs)
[5,6]

AMIs VNs
• demand-response
• scheduling [7]
• micro-grids
• detection of medium size blackouts [8]
• detection of non technical losses
• ...
5
• autonomous driving
• platooning
• accident detection [9]
• variable tolls [9]
• congestion monitoring [10]
• ...
Access to their data à Increased awareness, security, power-efficiency, ...
The data streaming paradigm and its use in Fog architectures

(some) fundamental questions
• where do we process data?
• how do we process data?
• how do we enforce security and privacy?

the wrong incomplete answer we run the analysis in the cloud!
7
1. does the infrastructure allow for billions of
readings per day to be transferred continuously?
2. the latency incurred while transferring data, does
that undermine the utility of the analysis?
3. is it secure to concentrate all the data in a single
place? [11]
4. is it smart to give away fine-grained data? [12]

A small example of what fine-grained data can reveal...
8
source: Private Memoirs of a Smart Meter [12]

a better answer we leverage the entire infrastructure!
9
Traditional analysis techniques cannot address
all the challenges in these setups [13,14]

Fog architectures
10
source: Fog Computing and Its Role in the Internet of Things [15]
Characteristics [15]:
1. edge location / location
awareness / low latency
2. geographical distribution
3. large-scale
4. support for mobility
5. real-time interactions
6. predominance of wireless
7. heterogeneous
8. interoperability / federation
9. interaction with the cloud

Fog architectures, applications trade-offs
and challenges
11
Sensor/Edge Devices Fog Devices Cloud Devices
Cost + ++ +++1
Computational power + ++ +++
Inter-node heterogeneity +++ ++ +
Intra-node heterogeneity + + +++
Size +++ ++ +
Communication wireless wireless/wired wired
Volume of data millions of small streams2 thousands of medium aggregated streams many large streams
Evolution over time +++ ++ +3
1. depends on who owns the physical hardware
2. with exceptions like e.g. Lidar sensors [16,17]
3. Virtual environments can also evolve ”behind the scenes”

Agenda
• Conclusions
• Bibliography

Motivation
Applications in sensor networks, cyber-physical
systems:
• large and fluctuating volumes of data generated
continuously
demand for:
• Continuous processing of data streams
• In a real-time fashion
Store-then-process is not feasible!!!

Main Memory
Motivation
DBMS vs. DSMS
Disk
1 Data
Query Processing
3 Query
results
2 Query
Main Memory
Query Processing
Continuous
Query
Data
Query
results

Before we start... about data streaming and Stream Processing Engines (SPEs)
15
An incomplete, non-sorted list of SPEs (cf. related work in [18]):
time
Borealis
The Aurora Project
STanfordstREamdatAManager
NiagaraCQ
COUGAR
StreamCloud
Covering all of them / discussing which use cases are best for each one out of scope...
will focus on the ones I work with and on example from vehicular networks

data stream: unbounded sequence of tuples sharing the same schema
16
Example: vehicles’ speed reports
time
Field Field
vehicle id text
time (secs) text
speed (Km/h) double
X coordinate double
Y coordinate double
A 8:00 55.5 X1 Y1
Let’s assume each source
(e.g., vehicle) produces
and delivers a timestamp
sorted stream
A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2

continuous query (or simply query): Directed Acyclic Graph (DAG) of
streams and operators
17
OP
OP
OP
OP OP
OP OP
source op
(1+ out streams)
sink op
(1+ in streams)
stream
op
(1+ in, 1+ out streams)

data streaming operators
Two main types:
• Stateless operators
• do not maintain any state
• one-by-one processing
• if they maintain some state, such state does not evolve depending
on the tuples being processed
• Stateful operators
• maintain a state that evolves depending on the tuples being
processed
• produce output tuples that depend on multiple input tuples
18
OP
OP

stateless operators
19
Filter
...
Map
Union
...
Filter / route tuples based on one (or more) conditions
Transform each tuple
Merge multiple streams (with the same schema) into one

stateless operators
20
Filter
...
Map
Union
...
source: http://storm.apache.org/releases/2.0.0-SNAPSHOT/Trident-tutorial.html

stateless operators
21
Filter
...
Map
Union
...
source: https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/index.html

stateless operators
22
Filter
...
Map
Union
...
source: http://spark.apache.org/docs/latest/streaming-programming-guide.html#transformations-on-dstreams

stateful operators
23
Aggregate information from multiple tuples
(e.g., max, min, sum, ...)
Join tuples coming from 2 streams given a certain predicate
Aggregate
Join

stateful operators
24
source: http://storm.apache.org/releases/2.0.0-SNAPSHOT/Trident-tutorial.html
source: http://spark.apache.org/docs/latest/streaming-programming-
guide.html#transformations-on-dstreams
source: http://spark.apache.org/docs/latest/streaming-programming-
guide.html#transformations-on-dstreams

Wait a moment!
if streams are unbounded, how can we aggregate or join?

windows and stateful analysis [18]
Stateful operations are done over windows:
• Time-based (e.g., tuples in the last 10 minutes)
• Tuple-based (e.g., given the last 50 tuples)
26
time
[8:00,9:00)
[8:20,9:20)
[8:40,9:40)
Usually applications rely on time-based sliding windows

time-based sliding window aggregation (count)
27
Counter: 4
time
[8:00,9:00)
8:05 8:15 8:22 8:45 9:05
Output: 4
Counter: 1
Counter: 2
Counter: 3
Counter: 3
time
8:05 8:15 8:22 8:45 9:05
[8:20,9:20)
we assumed each source
produces and delivers a
timestamp sorted stream!
What happens if this is not
the case?

time-based sliding window joining
28
t1
t2
t3
t4
t1
t2
t3
t4
R S
Sliding
window Window
size WS
WSWR
Predicate P

29
windows and stateful analysis

30
basic operators and user-defined operators
Besides a set of basic operators, SPEs usually allow the user to define
ad-hoc operators (e.g., when existing aggregation are not enough)

sample query
For each vehicle, raise an alert if the speed of the latest report is more
than 2 times higher than its average speed in the last 30 days.
31
time
A 8:00 55.5 X1 Y1 A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2

32
Remove
unused fields
Map
Field
vehicle id
time (secs)
speed (Km/h)
X coordinate
Y coordinate
Field
vehicle id
time (secs)
speed (Km/h)
Compute average
speed for each
vehicle during the
last 30 days
Aggregate
Field
vehicle id
time (secs)
avg speed (Km/h)
Join
Check
condition
Filter
Field
vehicle id
time (secs)
speed (Km/h)
Join on
vehicle id
Field
vehicle id
time (secs)
avg speed (Km/h)
speed (Km/h)
sample query

33
M A J F
sample query
Notice:
• the same semantics can be defined in several ways (using different
operators and composing them in different ways)
• Using many basic building blocks can ease the task of distributing
and parallelizing the analysis (more in the following...)

Agenda
• Conclusions
• Bibliography

1. Distributed deployment
2. Parallel deployment
3. Ordering and determinism
4. Shared-nothing vs shared-memory parallelism
5. Load balancing
6. Elasticity
7. Fault tolerance
8. Data sharing (differential privacy)
35
8 aspects / use cases to discuss challenges (and opportunities)

Before we start...
36
Following examples are from vehicular networks
Road-side unit
RSU
Vehicle
Server

1 - Distributed deployment – where to place a given operator? [19]
37
M
1. What hardware is available at each possible
deployment location?
2. What data is available at each possible
deployment location, depending on
1. the infrastructure?
2. its security regulations?
3. its privacy regulations?
?

1 - Distributed deployment – where to place a given operator?
38
M
1. Low latency (if all the required data is locally
accessible)
2. Access to all local data (parts of which might
not be reported to RSUs / Servers) [4]
3. Embedded devices / limited computational
power [20]

39
M
1. Access to multiple vehicles / high mobility
2. Access to smaller sets of data / partial view of
the data [22]
3. Embedded devices / higher computational
power
4. Wireless / wired communication

40
1. Global view of the data (when possible [22])
2. Access to aggregated data
3. High computational power
M

2 - Parallel deployment – how do we parallelize the analysis?
41
If a parallel operator (e.g., M) feeds multiple downstream
parallel operators (e.g., A), how do we route tuples? [23,24]
M
M A
A

42
M
M A
A
A 8:00 55.5 X1 Y1
A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2
A 8:00 55.5 X1 Y1
A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2
When computing e.g.
per-vehicle statistics,
we need to make sure
that all the tuples
referring to the same
vehicles are sent to
the same aggregate
instance

• Depending on the stateful operator semantic:
• Partition input stream into buckets
• Each bucket is processed by 1 node
• # buckets >> # nodes
43

Keys
domain
Agg1 Agg2 Agg3
A
D
E
B
C F
44

Keys
domain
Agg1 Agg2 Agg3
A
D
E
B
C F
45

46
How can we parallelize the analysis of e.g. an aggregate when
the operations are not performed for disjoint partitions of a
stream (e.g., for all vehicles rather than for each vehicle?)
M
M A
A

47

48
M
M A
A
A 8:00 55.5 X1 Y1
A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2
A 8:00 55.5 X1 Y1
A 8:07 34.3 X3 Y3
A 8:03 70.3 X2 Y2
What if tuple with
timestamp 8:00
arrives after tuple
with timestamp 8:07?
3 – Ordering and determinism

49
3 – Ordering and determinism
if tuples that aggregated together are arbitrarily interleaved,
how to we provide correct results?
time
[8:00,9:00)
8:05 8:15 8:22 8:45 9:05
Output: 4
time
[8:00,9:00)
8:05 8:15 8:22 8:45 9:058:55

50
3 – Ordering and determinism – merge-sorting [25] & buffering [26,27]
addTuple(tuple,sourceID)
allows a tuple from sourceID to be merged by ScaleGate in the
resulting timestamp-sorted stream of ready tuples.
getNextReadyTuple(readerID)
provides to readerID the next earliest ready tuple that has not been
yet consumed by the former.
https://github.com/dcs-chalmers/ScaleGate_Java

4 – shared-nothing vs. shared-memory parallelism [28]
51
M
M
...
A
A
J
...
How to take advantage of multi-core
architectures?
How to boost inter-node parallelism
and intra-node parallelism?

Shared-nothing parallel stream join
Prod
R
Prod
S
PU1
PU2
PUN…
Cons
Add tuple to PUi S
Add tuple to PUi R
Consume results
Pick the next ready tuple t:
1. compare t with all tuples in opposite window given P
2. add t to its window
3. remove stale tuples from t’s window
Chose a PU
Chose a PU
Take the next
ready output tuple
Scalability
High
throughput
Low latency
Disjoint
parallelism
Skew
resilience
Determinism
52
Merge
4 – shared-nothing vs. shared-memory parallelism

Prod
R
Prod
S
PU1
PU2
PUN
…
53
enqueue()
dequeue()
ConsMerge
Shared-nothing parallel stream join

Prod
R
Prod
S
PU1
PU2
PUN
…
Cons
Add tuple SGin
Add tuple SGin
Get next ready
output tuple
from SGout
Get next ready input tuple from SGin
2. add t to its window in a round-robin fashion
54
SGin SGout
Steps for PU
Shared-memory parallel stream join

55
t1
t2
R S
WR
t3
t4
R S
t4
t1
WR
R S
t4
t2
WR
R S
t4
WR
t3
Sequential stream join:
ScaleJoin with 3 PUs:

Prod
R
Prod
S
PU1
PU2
PUN
…
Cons
Add tuple SGin
Add tuple SGin
Get next ready
output tuple
from SGout
56
SGin SGout
Scalability
High
throughput
Low latency
Disjoint
parallelism
Skew
resilience
Determinism
Prod
S
Prod
S
Prod
R
Get next ready input tuple from SGin
2. add t to its window in a round robin fashion
Steps for PUi

5 – load balancing & state transfer [23,29]
57
If we shift the processing of a certain subset of
tuples from node A to node B, how do transfer
its previous state?
M
A
M
A

State transfer challenging for stateful operators
A B
time
58
5 – load balancing & state transfer

Window Recreation Protocol [23]
A B
time
A
A
B
Send to A
Send to B
59
+ No communication
between nodes
- Completion time
proportional to window
size

State Recreation Protocol
A B
time
B
B
B
Copy to B
60
Tuples referring
to caller A
+ Minimizes completion
time
- Communication
between nodes

6 – elasticity [23,30]
61
How / when to provision or
decommission new resources depending
on the analysis’ costs fluctuations?
J
J J
J

Target utilization threshold
Upper utilization threshold
Lower utilization threshold
0%
100%
CPU consumption
Memory
Latency
...
………
OP OP OPOP OP
Elasticity and load balancing actions on per-subcluster basis
62
6 – elasticity [23]

63
6 – elasticity
• How many resources should be provisioned /
decommissioned? [23]
• How to mix load balancing and provisioning /
decommissioning? [23]
• How to chose which resources to provision /
decommission? [30]
• How to predict when more / less resources will
be needed? [30]

Load-aware provisioning
1 2 4 7 11 2617
64
6 – elasticity [23]

7 – fault tolerance [related work in 18, 31, 32]
65
How to replace a failed node minimizing
recovery time (making it transparent to
the end user)?
J
J
J
J
J
J

Active standby
Primary
Replica
AM
A
66
Cost:
Recovery
Time:
7 – fault tolerance

Passive standby
Primary
Replica
AM
A
Periodic
checkpoints
67
Cost:
Recovery
Time:

Passive standby
Primary
AM
Periodic
checkpoints
68
Cost:
Recovery
Time:
Disk

Upstream Backup
AM
Buffer
69
Cost:
Recovery
Time:

Fault tolerance and elasticity can also be integrated
together in solution that externalize the management
of state from the SPE [32]
70

8 – data sharing (differential privacy) [2,33,34,35]
71
How to prevent privacy leaks?
Suppose we are interested in publishing vehicles’
average speed over a window of one hour...
We could aggregate by RSU!

8 – data sharing (differential privacy)
72
Wait a moment!
what if a single vehicle is
connected to a certain RSU?
Whether a certain mechanism preserves or not the privacy
of the underlying data depends on the knowledge of the
adversary
Differential privacy assumes the worst case scenario!

73
time
[8:00,9:00)
22.3 Km/h 10 Km/h ? 15Km/h
avg speed: 30 Km/h
without differential privacy, if knows the speed of , and
and the avg speed, then it can find out the speed of

74
time
[8:00,9:00)
22.3 Km/h 10 Km/h ? 15Km/h
avg speed: 30 Km/h
with differential privacy, if knows the speed of , and , it does
gain extra knowledge from the avg speed in order to find the speed of
+ noise (calibrated from probabilistic distribution) so
that whether participates or not, the results
will be nearly the same

75

Agenda
• Conclusions
• Bibliography

Introduction: Fog architectures
77
• where do we process data?
• how do we process data?
• how do we enforce security and privacy?
... cannot really answer to these questions
independently ...

Fog architectures, applications trade-offs
and challenges
78
Sensor/Edge Devices Fog Devices Cloud Devices
Cost + ++ +++
Computational power + ++ +++
Inter-node heterogeneity +++ ++ +
Intra-node heterogeneity + + +++
Size +++ ++ +
Communication wireless wireless/wired wired
Volume of data millions of small streams thousands of medium aggregated streams many large streams
Evolution over time +++ ++ +

Agenda
• Conclusions
• Bibliography

Bibliography
1. Zhou, Jiazhen, Rose Qingyang Hu, and Yi Qian. "Scalable distributed communication architectures to support advanced
metering infrastructure in smart grid." IEEE Transactions on Parallel and Distributed Systems 23.9 (2012): 1632-1642.
2. Gulisano, Vincenzo, et al. "BES: Differentially Private and Distributed Event Aggregation in Advanced Metering Infrastructures."
Proceedings of the 2nd ACM International Workshop on Cyber-Physical System Security. ACM, 2016.
3. Gulisano, Vincenzo, Magnus Almgren, and Marina Papatriantafilou. "Online and scalable data validation in advanced metering
infrastructures." IEEE PES Innovative Smart Grid Technologies, Europe. IEEE, 2014.
4. Gulisano, Vincenzo, Magnus Almgren, and Marina Papatriantafilou. "METIS: a two-tier intrusion detection system for advanced
metering infrastructures." International Conference on Security and Privacy in Communication Systems. Springer International
Publishing, 2014.
5. Yousefi, Saleh, Mahmoud Siadat Mousavi, and Mahmood Fathy. "Vehicular ad hoc networks (VANETs): challenges and
perspectives." 2006 6th International Conference on ITS Telecommunications. IEEE, 2006.
6. El Zarki, Magda, et al. "Security issues in a future vehicular network." European Wireless. Vol. 2. 2002.
7. Georgiadis, Giorgos, and Marina Papatriantafilou. "Dealing with storage without forecasts in smart grids: Problem
transformation and online scheduling algorithm." Proceedings of the 29th Annual ACM Symposium on Applied Computing.
ACM, 2014.
8. Fu, Zhang, et al. "Online temporal-spatial analysis for detection of critical events in Cyber-Physical Systems." Big Data (Big
Data), 2014 IEEE International Conference on. IEEE, 2014.
The data streaming paradigm and its use in Fog architectures 80

Bibliography
9. Arasu, Arvind, et al. "Linear road: a stream data management benchmark." Proceedings of the Thirtieth
international conference on Very large data bases-Volume 30. VLDB Endowment, 2004.
10. Lv, Yisheng, et al. "Traffic flow prediction with big data: a deep learning approach." IEEE Transactions on
Intelligent Transportation Systems 16.2 (2015): 865-873.
11. Grochocki, David, et al. "AMI threats, intrusion detection requirements and deployment
recommendations." Smart Grid Communications (SmartGridComm), 2012 IEEE Third International
Conference on. IEEE, 2012.
12. Molina-Markham, Andrés, et al. "Private memoirs of a smart meter." Proceedings of the 2nd ACM
workshop on embedded sensing systems for energy-efficiency in building. ACM, 2010.
13. Gulisano, Vincenzo, et al. "Streamcloud: A large scale data streaming system." Distributed Computing
Systems (ICDCS), 2010 IEEE 30th International Conference on. IEEE, 2010.
14. Stonebraker, Michael, Uǧur Çetintemel, and Stan Zdonik. "The 8 requirements of real-time stream
processing." ACM SIGMOD Record 34.4 (2005): 42-47.
15. Bonomi, Flavio, et al. "Fog computing and its role in the internet of things." Proceedings of the first
edition of the MCC workshop on Mobile cloud computing. ACM, 2012.
16. Himmelsbach, Michael, et al. "LIDAR-based 3D object perception." Proceedings of 1st international
workshop on cognition for technical systems. Vol. 1. 2008.

Bibliography
17. Geiger, Andreas, et al. "Vision meets robotics: The KITTI dataset." The International Journal of Robotics Research (2013):
0278364913491297.
18. Gulisano, Vincenzo Massimiliano. StreamCloud: An Elastic Parallel-Distributed Stream Processing Engine. Diss. Informatica,
2012.
19. Cardellini, Valeria, et al. "Optimal operator placement for distributed stream processing applications." Proceedings of the 10th
ACM International Conference on Distributed and Event-based Systems. ACM, 2016.
20. Costache, Stefania, et al. "Understanding the Data-Processing Challenges in Intelligent Vehicular Systems." Proceedings of the
2016 IEEE Intelligent Vehicles Symposium (IV16).
21. Cormode, Graham. "The continuous distributed monitoring model." ACM SIGMOD Record 42.1 (2013): 5-14.
22. Giatrakos, Nikos, AntoniosDeligiannakis, and Minos Garofalakis. "Scalable Approximate Query Tracking over Highly Distributed
Data Streams." Proceedings of the 2016 International Conference on Management of Data. ACM, 2016.
23. Gulisano, Vincenzo, et al. "Streamcloud: An elastic and scalable data streaming system." IEEE Transactions on Parallel and
Distributed Systems 23.12 (2012): 2351-2365.
24. Shah, Mehul A., et al. "Flux: An adaptive partitioning operator for continuous query systems." Data Engineering, 2003.
Proceedings. 19th International Conference on. IEEE, 2003.

Bibliography
25. Cederman, Daniel, et al. "Brief announcement: concurrent data structures for efficient streaming aggregation." Proceedings of
the 26th ACM symposium on Parallelism in algorithms and architectures. ACM, 2014.
26. Ji, Yuanzhen, et al. "Quality-driven processing of sliding window aggregates over out-of-order data streams." Proceedings of the
9th ACM International Conference on Distributed Event-Based Systems. ACM, 2015.
27. Ji, Yuanzhen, et al. "Quality-driven disorder handling for concurrent windowed stream queries with shared operators."
Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems. ACM, 2016.
28. Gulisano, Vincenzo, et al. "Scalejoin: A deterministic, disjoint-parallel and skew-resilient stream join." Big Data (Big Data), 2015
IEEE International Conference on. IEEE, 2015.
29. Ottenwälder, Beate, et al. "MigCEP: operator migration for mobility driven distributed complex event processing." Proceedings
of the 7th ACM international conference on Distributed event-based systems. ACM, 2013.
30. De Matteis, Tiziano, and Gabriele Mencagli. "Keep calm and react with foresight: strategies for low-latency and energy-efficient
elastic data stream processing." Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel
Programming. ACM, 2016.
31. Balazinska, Magdalena, et al. "Fault-tolerance in the Borealis distributed stream processing system." ACM Transactions on
Database Systems (TODS) 33.1 (2008): 3.
32. Castro Fernandez, Raul, et al. "Integrating scale out and fault tolerance in stream processing using operator state
management." Proceedings of the 2013 ACM SIGMOD international conference on Management of data. ACM, 2013.

Bibliography
33. Dwork, Cynthia. "Differential privacy: A survey of results." International Conference on Theory and Applications of
Models of Computation. Springer Berlin Heidelberg, 2008.
34. Dwork, Cynthia, et al. "Differential privacy under continual observation." Proceedings of the forty-second ACM
symposium on Theory of computing. ACM, 2010.
35. Kargl, Frank, Arik Friedman, and Roksana Boreli. "Differential privacy in intelligent transportation systems." Proceedings
of the sixth ACM conference on Security and privacy in wireless and mobile networks. ACM, 2013.

Thank you! Questions?

The data streaming paradigm and its use in Fog architectures

The data streaming paradigm and its use in Fog architectures

Recommended

More Related Content

Viewers also liked

Viewers also liked(20)

Similar to The data streaming paradigm and its use in Fog architectures

Similar to The data streaming paradigm and its use in Fog architectures(20)

More from Vincenzo Gulisano

More from Vincenzo Gulisano(8)

Recently uploaded

Recently uploaded(20)

The data streaming paradigm and its use in Fog architectures