This document proposes an approach for engineering resilient collaborative edge-enabled IoT systems using aggregate computing. It presents aggregate computing as a paradigm for developing collaborative systems in dynamic open environments at scale. The contribution is a process for coordinating IoT/edge devices that uses decentralized coordination, self-* capabilities, neighbor-based communication, and collective behavior to address challenges of scale, dynamicity, failure, and lack of global connectivity. This is demonstrated through simulations of problem solving and resource management scenarios.

1. Engineering Resilient Collaborative Edge-enabled IoT
Roberto Casadei1
, Christos Tsigkanos2
, Schahram Dustdar2
, Mirko Viroli1
1
ALMA MATER STUDIORUM–Università di Bologna, Cesena, Italy
2
TU Wien, Vienna, Austria
IEEE International Conference on Services Computing (SCC)
July 8-13, 2019, Milan, Italy
https://www.slideshare.net/RobertoCasadei/
R. Casadei Introduction and Motivation Background Contribution Wrap-up 1/29

2. Outline
1 Introduction and Motivation
2 Background: Aggregate Computing
3 Contribution
4 Wrap-up
R. Casadei Introduction and Motivation Background Contribution Wrap-up 2/29

3. cyber-physical (eco-)systems
R. Casadei Introduction and Motivation Background Contribution Wrap-up 3/29

4. large scale, heterogeneity
R. Casadei Introduction and Motivation Background Contribution Wrap-up 4/29

5. volatility, fallibility, change
R. Casadei Introduction and Motivation Background Contribution Wrap-up 5/29

6. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
R. Casadei Introduction and Motivation Background Contribution Wrap-up 6/29

7. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale
R. Casadei Introduction and Motivation Background Contribution Wrap-up 6/29

8. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale
dynamicity, failure
R. Casadei Introduction and Motivation Background Contribution Wrap-up 6/29

9. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale
dynamicity, failure
no global connectivity
R. Casadei Introduction and Motivation Background Contribution Wrap-up 6/29

10. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale
dynamicity, failure
no global connectivity
collaboration
R. Casadei Introduction and Motivation Background Contribution Wrap-up 6/29

11. Outline
1 Introduction and Motivation
2 Background: Aggregate Computing
3 Contribution
4 Wrap-up
R. Casadei Introduction and Motivation Background Contribution Wrap-up 7/29

12. Aggregate Computing12
paradigm for Collective Adaptive Systems (CASs)
macro approach (global perspective)
formally founded (computational field calculus)
1Jacob Beal, Danilo Pianini, and Mirko Viroli. “Aggregate Programming for the Internet of Things”. In:
IEEE Computer (2015) [ieeecom]
2Mirko Viroli, Jacob Beal, et al. “From Field-Based Coordination to Aggregate Computing”. In: Int. Conf.
on Coordination Languages and Models. Springer. 2018 [coord18a]
R. Casadei Introduction and Motivation Background Contribution Wrap-up 8/29

13. sensors
local functions
actuators
Application
Code
Developer
APIs
Field Calculus
Constructs
Resilient
Coordination
Operators
Device
Capabilities
functions repnbr
TGCfunctions
communication state
PerceptionPerception
summarize
average
regionMax
…
ActionAction StateState
Collective BehaviorCollective Behavior
distanceTo
broadcast
partition
…
timer
lowpass
recentTrue
…
collectivePerception
collectiveSummary
managementRegions
…
Crowd ManagementCrowd Management
dangerousDensity crowdTracking
crowdWarning safeDispersal
restriction
self­stabilisation
R. Casadei Introduction and Motivation Background Contribution Wrap-up 9/29

14. structure (logical)
devices (individual elements of an aggregate)
neighbouring relationship (by networking or spatial)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 10/29

15. structure (logical)
devices (individual elements of an aggregate)
neighbouring relationship (by networking or spatial)
behaviour (execution protocol)
async rounds of computation
1) full run of “aggregate program” against “local context”
device state
sensor readings
neighbourhood coordination data
2) broadcast coordination data to neighbours
R. Casadei Introduction and Motivation Background Contribution Wrap-up 10/29

16. paradigmatic example (gradient)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 11/29

17. paradigmatic example (gradient)
class MyProgram extends AggregateProgram {
override def main =
rep(Double.PositiveInfinity)(distance =>
mux(source){
0.0
}{
minHoodPlus( nbr{distance} + metric )))
}
)
}
R. Casadei Introduction and Motivation Background Contribution Wrap-up 11/29

18. abstraction & compositionality
trait MyLib extends DistanceLib { self: AggregateProgram =>
def gradient(source: Boolean, metric: => Double): Double =
rep(Double.PositiveInfinity)(distance =>
mux(source){ 0.0 }{ minHoodPlus( nbr{distance} + metric ))) }
)
def channel(src: Boolean, dest: Boolean, w: Double): Boolean =
gradient(src) + gradient(dest) < distanceBetween(src,dest) + w
}
R. Casadei Introduction and Motivation Background Contribution Wrap-up 11/29

19. nice things about aggregate
computing 3
predictable composition of emergent behaviour 4
declarativity 5
flexibility in execution
formal properties: self-stabiliz. 6
, eventual consistency 7
practical (see PLs/tools like ScaFi 8
)
3Mirko Viroli, Jacob Beal, et al. “From Field-Based Coordination to Aggregate Computing”. In: Int. Conf. on
Coordination Languages and Models. Springer. 2018 [coord18a]
4Giorgio Audrito, Mirko Viroli, et al. “A Higher-Order Calculus of Computational Fields”. In: ACM Transactions on
Computational Logic 1 (Jan. 2019) [tocl19Aud]
5Mirko Viroli, Roberto Casadei, and Danilo Pianini. “On execution platforms for large-scale aggregate computing”. In:
UbiComp, Proceedings of. ACM. 2016 [ubicomp16Vir]
6Mirko Viroli, Giorgio Audrito, et al. “Engineering Resilient Collective Adaptive Systems by Self-Stabilisation”. In: ACM
Transactions on Modeling and Computer Simulation 2 (2018) [tomacs18Vir]
7Jacob Beal, Mirko Viroli, et al. “Self-adaptation to device distribution in the Internet of Things”. In: ACM Transactions
on Autonomous and Adaptive Systems (TAAS) 3 (2017) [taas17]
8Roberto Casadei, Danilo Pianini, and Mirko Viroli. “Simulating large-scale aggregate MASs with Alchemist and
Scala”. In: FedCSIS, Proceedings of. IEEE. 2016 [fedcsis16Cas]
R. Casadei Introduction and Motivation Background Contribution Wrap-up 12/29

20. Outline
1 Introduction and Motivation
2 Background: Aggregate Computing
3 Contribution
4 Wrap-up
R. Casadei Introduction and Motivation Background Contribution Wrap-up 13/29

21. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale
dynamicity, failure
no global connectivity
collaboration
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

22. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale decentralised coordination
dynamicity, failure
no global connectivity
collaboration
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

23. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale decentralised coordination
dynamicity, failure self-* capabilities
no global connectivity
collaboration
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

24. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale decentralised coordination
dynamicity, failure self-* capabilities
no global connectivity neighbour-based comm.
collaboration
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

25. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale decentralised coordination
dynamicity, failure self-* capabilities
no global connectivity neighbour-based comm.
collaboration collective behaviour
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

26. problem
development of
collaborative IoT/Edge systems
in dynamic, open environments
scale decentralised coordination
dynamicity, failure self-* capabilities
no global connectivity neighbour-based comm.
collaboration collective behaviour
IoT/Edge system as a CAS
R. Casadei Introduction and Motivation Background Contribution Wrap-up 14/29

27. approach overview
customisable collaboration process
System Concern
Sensing
ResilientCollaborativeEdge-EnabledIoT
EdgeGoalsIoTDevices-Environment
Device
Sensor
Edge
System Concern
Actuation
System Concern
Communication
…
System Concern
Coordination
Aggregate Computing Constructs
Global System Specification
Device
Human
Design Time
Runtime
Robot
Edge
R. Casadei Introduction and Motivation Background Contribution Wrap-up 15/29

28. problem model
R. Casadei Introduction and Motivation Background Contribution Wrap-up 16/29

29. problem model
environment
R. Casadei Introduction and Motivation Background Contribution Wrap-up 16/29

30. problem model
environment
situated devices (coordinators or workers/users)
– static or mobile
– resource-constrained (IoT) or powerful (edge/fog)
– sensors and actuators
R. Casadei Introduction and Motivation Background Contribution Wrap-up 16/29

31. problem model
environment
situated devices (coordinators or workers/users)
– static or mobile
– resource-constrained (IoT) or powerful (edge/fog)
– sensors and actuators
how to coordinate activity and
decision making?
how much locality and globality?
R. Casadei Introduction and Motivation Background Contribution Wrap-up 16/29

32. core coordination process
network
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

33. core coordination process
potential coordinators
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

34. core coordination process
elected coordinators
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

35. core coordination process
regions
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

36. core coordination process
upstreaming
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

37. core coordination process
downstreaming
R. Casadei Introduction and Motivation Background Contribution Wrap-up 17/29

38. combining various SASOp patterns
decentralised, self-healing gradient 9
decentralised, self-healing leader election 10
information flows and feedback loops 11
− information spreading 12
− information collection 13
9Giorgio Audrito, Roberto Casadei, et al. “Compositional blocks for optimal self-healing gradients”. In: Conf. on
Self-Adaptive and Self-Organizing Systems (SASO). IEEE. 2017 [saso17Aud]
10Yuanqiu Mo, Jacob Beal, and Soura Dasgupta. “An Aggregate Computing Approach to Self-Stabilizing Leader
Election”. In: Workshops on Foundations & Applications of Self* Systems (FAS*W). IEEE. 2018 [ecas18Mo]
11Tom De Wolf and Tom Holvoet. “Designing self-organising emergent systems based on information flows and
feedback-loops”. In: Self-Adaptive and Self-Organizing Systems, 2007. SASO’07. 1st Conf. on. IEEE. 2007 [saso07DeW]
12Yuanqiu Mo, Soura Dasgupta, and Jacob Beal. “Robust Stability of Spreading Blocks in Aggregate Computing”. In:
2018 IEEE Conference on Decision and Control (CDC). IEEE. 2018 [cdc18Mo]
13Giorgio Audrito, Sergio Bergamini, et al. “Effective Collective Summarisation of Distributed Data in Mobile Multi-Agent
Systems”. In: 18th International Conference on Autonomous Agents and MultiAgent Systems. ACM. 2019 [aamas19Aud]
R. Casadei Introduction and Motivation Background Contribution Wrap-up 18/29

39. scenario:
situated problem solving
Elected
Coordinator
(Edge)
Detector
Detector/Solver
Solver
…
Coordinator Area
Backup
Coordinator
(Edge)
Downstreaming/
Upstreaming
Upstreaming/
Downstreaming
Profile
Capabilities
Problems
Assignments
Elected
Coordinator
Coordinator Area
Backup
Coordinator
Detector
Solver
…
……
…
Feedbacks
Problems
R. Casadei Introduction and Motivation Background Contribution Wrap-up 19/29

40. toolchain and simulation framework
ScaFi: Aggregate Computing toolkit
https://scafi.github.io
Alchemist: simulator
https://alchemist.github.io
Fully reproducible experiments @ repository:
https://github.com/metaphori/
engineering-collaborative-edge-iot
R. Casadei Introduction and Motivation Background Contribution Wrap-up 20/29

41. implementation schema (summary)
class ProblemSolvingEcosystem extends AggregateProgram with ProblemAPI {
override def main = {
val coordinators = S(grain, priorityField)
val potential = branch(infoPropagationNet){gradient(coordinators)}{+∞}
val problems = collectSets(downTo=potential, problemOccurrences)
val solvers = collectSets(downTo=potential, solverProfile)
val feedbacks = collectSets(downTo=potential, feedbackField).groupBy(_.problem)
val assignments = branch(coordinators){
allocate(coordinators,solvers,problems,feedbacks)
}{ Set() }
val tasks = broadcast(potential, assignments)
branch(workers){ execute(tasks) }{ () }
} }
global specification, collectively executed continuously
composition of aggregate-level building blocks
R. Casadei Introduction and Motivation Background Contribution Wrap-up 21/29

42. implementation schema (detail)
val coordinators = S(grain, priorityField)
boolean field (true for elected leaders)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 21/29

43. implementation schema (detail)
val potential = branch(infoPropagationNet){
gradient(coordinators)
}{+∞}
leaders indirectly define areas by a gradient
(a device belongs to the area of its closest leader)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 21/29

44. implementation schema (detail)
val problems = collectSets(downTo=potential, problemsFound)
val solvers = collectSets(downTo=potential, solverProfile)
val feedbacks = collectSets(downTo=potential, feedbackField)
.groupBy(_.problem)
upstreaming: data is collected into the leaders
R. Casadei Introduction and Motivation Background Contribution Wrap-up 21/29

45. implementation schema (detail)
val assignments = branch(coordinators){
allocate(coordinators,solvers,problems,feedbacks)
}{ Set() }
val tasks = broadcast(potential, assignments)
branch(workers){ execute(tasks) }{ () }
leaders’ control data downstreamed to workers
R. Casadei Introduction and Motivation Background Contribution Wrap-up 21/29

46. simulation setup
smart city environment (map of Vienna)
300 workers (lightweight devices for detection&solving)
10 potential coordinators (edge nodes)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 22/29

47. simulation setup
R. Casadei Introduction and Motivation Background Contribution Wrap-up 22/29

48. simulation setup
problems randomly injected in timeframe [150, 600]
blackout during time [300, 310], triggering re-election
config #1: naive allocation
config #2: smart allocation (worker expertise)
R. Casadei Introduction and Motivation Background Contribution Wrap-up 22/29

49. evaluation: correctness, adaptivity,
resilience
naive coordinators
less reactive
smart coordinators
more reactive
R. Casadei Introduction and Motivation Background Contribution Wrap-up 23/29

50. another scenario:
resource management in edge-clouds
50 fog nodes
200 workers provide resources/services
500 sparse clients make requests
peek of requests in timeframe [150, 250]
R. Casadei Introduction and Motivation Background Contribution Wrap-up 24/29

51. qualitative evaluation
more leaders
lower load on relays/leaders
few leaders
higher load on relays/leaders
R. Casadei Introduction and Motivation Background Contribution Wrap-up 25/29

52. qualitative evaluation
higher utilization of workers
area-wise
lower utilization of workers
area-wise
R. Casadei Introduction and Motivation Background Contribution Wrap-up 25/29

53. qualitative evaluation
higher unavailability (declined
tasks) more retries
more services available in
each area
R. Casadei Introduction and Motivation Background Contribution Wrap-up 25/29

54. Outline
1 Introduction and Motivation
2 Background: Aggregate Computing
3 Contribution
4 Wrap-up
R. Casadei Introduction and Motivation Background Contribution Wrap-up 26/29

55. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

56. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

57. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

58. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

59. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
future work
control theoretical analysis
time guarantees
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

60. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
future work
control theoretical analysis
time guarantees
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

61. conclusion
a decentralised, self-org, spatial, collective approach to
development of collaborative IoT/Edge systems
aggregate computing implementation
evaluation of solution via simulations
future work
control theoretical analysis
time guarantees
R. Casadei Introduction and Motivation Background Contribution Wrap-up 27/29

62. References (1/2)
[aamas19Aud] Giorgio Audrito et al. “Effective Collective Summarisation of Distributed Data in
Mobile Multi-Agent Systems”. In: 18th International Conference on Autonomous
Agents and MultiAgent Systems. ACM. 2019, pp. 1618–1626.
[saso17Aud] Giorgio Audrito et al. “Compositional blocks for optimal self-healing gradients”. In:
Conf. on Self-Adaptive and Self-Organizing Systems (SASO). IEEE. 2017,
pp. 91–100.
[tocl19Aud] Giorgio Audrito et al. “A Higher-Order Calculus of Computational Fields”. In: ACM
Transactions on Computational Logic 20.1 (Jan. 2019), 5:1–5:55. ISSN: 1529-3785.
DOI: 10.1145/3285956.
[ieeecom] Jacob Beal, Danilo Pianini, and Mirko Viroli. “Aggregate Programming for the Internet
of Things”. In: IEEE Computer (2015). ISSN: 1364-503X.
[taas17] Jacob Beal et al. “Self-adaptation to device distribution in the Internet of Things”. In:
ACM Transactions on Autonomous and Adaptive Systems (TAAS) 12.3 (2017), p. 12.
[fedcsis16Cas] Roberto Casadei, Danilo Pianini, and Mirko Viroli. “Simulating large-scale aggregate
MASs with Alchemist and Scala”. In: FedCSIS, Proceedings of. IEEE. 2016,
pp. 1495–1504.
[saso07DeW] Tom De Wolf and Tom Holvoet. “Designing self-organising emergent systems based
on information flows and feedback-loops”. In: Self-Adaptive and Self-Organizing
Systems, 2007. SASO’07. 1st Conf. on. IEEE. 2007, pp. 295–298.
R. Casadei Appendix References 28/29

63. References (2/2)
[ecas18Mo] Yuanqiu Mo, Jacob Beal, and Soura Dasgupta. “An Aggregate Computing Approach
to Self-Stabilizing Leader Election”. In: Workshops on Foundations & Applications of
Self* Systems (FAS*W). IEEE. 2018, pp. 112–117.
[cdc18Mo] Yuanqiu Mo, Soura Dasgupta, and Jacob Beal. “Robust Stability of Spreading Blocks
in Aggregate Computing”. In: 2018 IEEE Conference on Decision and Control (CDC).
IEEE. 2018, pp. 6007–6012.
[tomacs18Vir] Mirko Viroli et al. “Engineering Resilient Collective Adaptive Systems by
Self-Stabilisation”. In: ACM Transactions on Modeling and Computer Simulation 28.2
(2018), 16:1–16:28.
[coord18a] Mirko Viroli et al. “From Field-Based Coordination to Aggregate Computing”. In: Int.
Conf. on Coordination Languages and Models. Springer. 2018, pp. 252–279.
[ubicomp16Vir] Mirko Viroli, Roberto Casadei, and Danilo Pianini. “On execution platforms for
large-scale aggregate computing”. In: UbiComp, Proceedings of. ACM. 2016,
pp. 1321–1326.
R. Casadei Appendix References 29/29