The document discusses integrating machine learning within the aggregate computing paradigm to enhance adaptability in collective self-adaptive systems. Early results indicate that reinforcement learning is particularly suitable for addressing challenges within this framework, offering paths towards more intelligent and robust collective behaviors. Research questions focus on the effective combination of machine learning techniques with varying abstraction levels in aggregate computing.

1. Research directions for Aggregate Computing with
Machine Learning
Gianluca Aguzzi1
Supervisor : Mirko Viroli 1
ACSOS Mentor : Christopher Landauer
1
Alma Mater Studiorum – Università di Bologna, Cesena, Italy
October 1, 2021
Talk @ ACSOS 2021, Doctoral Symposium
2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems

2. Background
Collective (Self-)Adaptive Systems (CSAS) [D’A+19]
Distributed and interconnected systems composed of multiple agents that can perform complex tasks ex-
hibiting robust collective behaviours while achieving system-wide and agent-specific goals.
Background G. Aguzzi 2 / 20

3. Aggregate Computing [BPV15]
A top-down global-to-local approach to express
collective behaviour
Rooted on field-calculus [Aud+18]
Collective behaviour does not depend on system scale
Used in various scenarios ranging from smart cities to
crowd engineering [Cas+19]
Problem i
Building block design is complex
Background G. Aguzzi 3 / 20

4. Machine Learning
Enhance agents with some learning capabilities
Learning improve adaptability, helping agent to act in uncertain environments
Supervised Learning [D’A+19], Reinforcement Learning [HW98; NNN20; MLF07;
HKT19] and Evolutionary Computing [JMK17; PL05] are typically used in CSAS
Problem i
Solutions are application-specific
Background G. Aguzzi 4 / 20

5. Problem statement
Extend Aggregate Computing paradigm to include the Machine Learning capability to improve adapt-
ability and simplify the definition and refinement of collective behaviour.
Problem statement G. Aguzzi 5 / 20

6. Motivation
¢ Aggregate Computing is scale independent by construction
¢ Hybrid collective program description
¢ Try to improve current state-of-the-art Machine Learning applying in CSAS
Problem statement G. Aguzzi 6 / 20

7. Research Questions
? What kind of Machine Learning approach is useful in combination with Aggregate
Computing
? At what level of abstraction can Machine Learning be useful for Aggregate Computing
? What does Aggregate Computing have in common with Machine Learning, applied to
Collective Self-Adaptive System
Problem statement G. Aguzzi 7 / 20

8. Early results
Setting
Focus on simple but well-known
problems in Aggregate Computing
Learning exploited to guide
building-block improvements
Verifying what kind of approach is
well-suited for Aggregate Computing
Constraints
Learning problem framed as
Homogenous Team Learning [PL05]
Learning performed off-line [PMV13]
Early results G. Aguzzi 8 / 20

9. Computational model [HLM15]
Ensemble of nodes with an
identifier
Each node has a local-view (i.e.
neighbours relationship)
Interaction happens with
message passing (executed
continously).
Early results G. Aguzzi 9 / 20

10. Computational model
Round steps
1 Context creation
2 Program evaluation producing an
export
3 Export sharing to neighbourhood
Context
Early results G. Aguzzi 10 / 20

11. Gradient/Hop count example [Aud+17]
Definition
A program that produce a computational field where each node contains the distance from a
source zone.
S
10
15
20
15
0
∞
∞
∞
∞
S
10
15
20
15
0
∞
∞
∞
∞
(∞, ∞, ∞, 0)
out = 0 + 10
S
10
15
20
15
0
10
25
25
30
Early results G. Aguzzi 11 / 20

12. Gradient/Hop count example
Problem i
Naive solutions suffers of the slow-rising problem.
2
1
2
0
2
2
2
2
1
1
2
2
2
2
1
2
1
2
∞
2
2
2
2
1 - 3
1
2
2
2
2
1
4
3
4
∞
4
4
4
4
3
3
4
4
4
4
3
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
∞
Early results G. Aguzzi 12 / 20

13. Aggregate Computing + Reinforcement Learning
Most suitable match
Typically used in CSAS
Designed to maximise long term rewards
Quite easy to express aggregate problems as reinforcement
learning problems
Early results G. Aguzzi 13 / 20

14. Aggregate Computing + Reinforcement Learning: Hop count
Independent Q-Learning
Actions Increase / NoOp
State Temporal Windows of Speed
Reward 0 / Ñ -1
output
Increase
1
NoOp
0
State
[0, 1, 1, 1]
0
-1
Early results G. Aguzzi 14 / 20

15. Conclusion
Endow Aggregate Computing with intelligent behaviour
Reinforcement Learning matches with Aggregate Computing
A path towards robust, adaptive and smarter collective behaviour definition
Conclusion G. Aguzzi 15 / 20

16. References I
[D’A+19] Mirko D’Angelo et al. “On learning in collective self-adaptive systems: State of practice
and a 3D framework”. ICSE Workshop on Software Engineering for Adaptive and
Self-Managing Systems 2019-May (2019), pp. 13–24. ISSN: 21567891. DOI:
10.1109/SEAMS.2019.00012.
[Aud+18] Giorgio Audrito et al. “Space-Time Universality of Field Calculus”. Ed. by Giovanna
Di Marzo Serugendo and Michele Loreti. Vol. 10852. Lecture Notes in Computer Science.
Springer, 2018, pp. 1–20. DOI: 10.1007/978-3-319-92408-3_1.
[Cas+19] Roberto Casadei et al. “Self-organising Coordination Regions: A Pattern for Edge
Computing”. Ed. by Hanne Riis Nielson and Emilio Tuosto. Vol. 11533. Lecture Notes in
Computer Science. Springer, 2019, pp. 182–199. DOI: 10.1007/978-3-030-22397-7_11.
[BPV15] Jacob Beal, Danilo Pianini, and Mirko Viroli. “Aggregate Programming for the Internet of
Things”. Computer 48.9 (2015), pp. 22–30. DOI: 10.1109/MC.2015.261.
[HW98] Junling Hu and Michael P. Wellman. “Multiagent Reinforcement Learning: Theoretical
Framework and an Algorithm”. Ed. by Jude W. Shavlik. Morgan Kaufmann, 1998,
pp. 242–250.
References G. Aguzzi 16 / 20

17. References II
[NNN20] Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. “Deep Reinforcement
Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications”.
IEEE Trans. Cybern. 50.9 (2020), pp. 3826–3839. DOI: 10.1109/TCYB.2020.2977374.
[MLF07] Laëtitia Matignon, Guillaume J. Laurent, and Nadine Le Fort-Piat. “Hysteretic q-learning :
an algorithm for decentralized reinforcement learning in cooperative multi-agent teams”.
IEEE, 2007, pp. 64–69. DOI: 10.1109/IROS.2007.4399095.
[HKT19] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. “A survey and critique of
multiagent deep reinforcement learning”. Auton. Agents Multi Agent Syst. 33.6 (2019),
pp. 750–797. DOI: 10.1007/s10458-019-09421-1.
[JMK17] Junchen Jin, Xiaoliang Ma, and Iisakki Kosonen. “A stochastic optimization framework
for road traffic controls based on evolutionary algorithms and traffic simulation”. Adv.
Eng. Softw. 114 (2017), pp. 348–360. DOI: 10.1016/j.advengsoft.2017.08.005.
[PL05] Liviu Panait and Sean Luke. “Cooperative Multi-Agent Learning: The State of the Art”.
Auton. Agents Multi Agent Syst. 11.3 (2005), pp. 387–434. DOI:
10.1007/s10458-005-2631-2.
References G. Aguzzi 17 / 20

18. References III
[PMV13] Danilo Pianini, Sara Montagna, and Mirko Viroli. “Chemical-oriented Simulation of
Computational Systems with Alchemist”. Journal of Simulation (2013). ISSN: 1747-7778.
DOI: 10.1057/jos.2012.27.
[HLM15] Jianye Hao, Ho-fung Leung, and Zhong Ming. “Multiagent Reinforcement Social Learning
toward Coordination in Cooperative Multiagent Systems”. ACM Trans. Auton. Adapt. Syst.
9.4 (2015), 20:1–20:20. DOI: 10.1145/2644819.
[Aud+17] Giorgio Audrito et al. “Compositional Blocks for Optimal Self-Healing Gradients”. IEEE
Computer Society, 2017, pp. 91–100. DOI: 10.1109/SASO.2017.18.
References G. Aguzzi 18 / 20

19. Aggregate Computing + Supervised Learning
Ground truth generation is done with simulations
A network is trained with a global view and then
Each agent have the same network
S
10
15
20
15
0
10
25
25
30
S
10
15
20
15
0
10
25
25
30
S
0
10
25
25
30
Appendix G. Aguzzi 19 / 20

20. Aggregate Computing + Evolutionary computing
A population of Network is shared within the entire system
During the simulation, a fitness function is evaluated
Already used in Swarm robotics and automatic design
S
?
?
?
?
?
S
0
10
25
25
30
S
0
10
25
25
30
Appendix G. Aguzzi 20 / 20