Awareness sponsored talk by Joerg Denzinger at international workshop on Self-aware Internet of Things in San Jose, California, USA, September 2012.

1. Jörg Denzinger,
Department of Computer Science,
University of Calgary
denzinge@cpsc.ucalgary.ca

2.  Malik Atalla  Torsten Steiner
 Bernhard Bauer  Chris Thornton
 Karel Bergmann  Jan‐Philipp
 Michael Blackadar Steghöfer
 Jeff Boyd
 Tom Flanagan
 Jonathan Hudson
 Holger Kasinger
 Jordan Kidney
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

3.  Awareness (www.aware‐project.eu/), a FET
coordination action funded by the European
Commission under FP7 which provides
support for researchers interested in “Self‐
Awareness in Autonomic Systems”
 Jennifer Willies
 Levent Gürgen, Klaus Moessner, Abdur
Rahim Biswas and Fano Ramparany
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

4. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

5. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

6. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

7. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
 Dangerous adaptations?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

8. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
 Dangerous adaptations?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

9. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
 Dangerous adaptations?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

10. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
 Dangerous adaptations?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

11. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior? SOS!
 Dangerous adaptations?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

12. IoT aims at large number of autonomous
entities working together and manage
themselves to adapt to task and environment
and create emergent properties
 But what about
 Unwanted emergent behavior?
 Dangerous adaptations?
 How can we test for that and/or avoid it?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

13.  General idea:
Use learning to create event sequences for
tested system that reveal examples for
unwanted emergent behavior and dangerous
adaptations.
 Use simulations (if necessary) to provide the
feedback necessary for learning
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

14. Ag tested,1 … Agtested,m
Ag byst,1
.
Env .
.
Ag byst,k
Ag event,1 … Ag event,n
Learner
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

15.  In theory, anything able to learn event
sequences could be used
 In practice, evaluating complete event
sequences is easier than trying to evaluate
potential of partial sequences or sequence
skeletons
 evolutionary methods advised
 But: use as much knowledge as possible
 targeted operators
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

16.  We need to evaluate how near the simulation
of a given event sequence came to creating a
specific (unwanted) behavior:
 We evaluate the simulation state after each
event ( step‐fitness) and sum up these
fitness values
 Step‐fitness depends on the application
(although we see some common patterns in
our applications)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

17.  The main influence on the complexity are
 Number of event generators
 Length of event sequences
( search space of the learner)
 Size of the tested system only plays a role in
the simulations (so, hopefully, linear)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

18.  Testing computer game AIs
 FIFA 99 (CEC‐04, CIG‐05)
 ORTS (CIG‐09)
 Starcraft (AIIDE‐11)
 Finding problems in student written MAS for
rescue simulator ARES (ECAI‐06, IAT‐06)
 Testing surveillance networks
 Harbor surveillance and interdiction (CISDA‐09)
 Patrolling robots with stationary sensor platforms
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

19.  Testing and evaluating self‐organizing, self‐
adapting transportation systems
(ADAPTIVE‐10, SASO‐12)
 Testing agriculture sensor networks (and
watering machinery) (on‐going work)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

20. SASO‐08; Communications of SIWN 4, 2008;
EASe‐09; ADAPTIVE‐10
Scenario:
Group of agents for performing dynamic pickup
and delivery tasks
Objective:
Optimize performance (here: distance traveled)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

21.  Self‐organizing system based on info‐
chemicals:
 tasks announce themselves by infochemicals that
are propagated
 Transport agents follow infochemicals trails while
sending out infochemicals themselves
 Picked‐up tasks announce this via infochem.,
again
 Test goal: How inefficient can the tested
system be?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

22.  Events: pickup and delivery coordinates +
time when event is announced to the tested
system
 Learner: GA learning a single event sequence
 Fitness: One simulation then comparing
emergent solution to optimal solution quality
normalized by optimal solution
(maximizing this difference)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

23.  Test system consistently found event
sequences solved by tested system 4 times
worse than optimum (over varying event
numbers)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

24. EASe 2010, EASe 2011, ADAPTIVE‐10
Scenario:
Group of agents for performing dynamic pickup
and delivery tasks,
quite a number of recurring tasks (every day)
Objective:
Optimize performance (here: distance traveled)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

25.  Self‐organizing system based on info‐chemicals
with an advisor:
 Gets observations from all agents
 Determines recurring events
 Computes optimal (good) solution for these
 Determines what individual agents do wrong and
creates advice exception rules for them (several rule
variants)
 Test goal: how much damage to performance
can adaptations by advisor do (if exploited by
adversary)?
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

26.  Events as before
 Learner: GA learning two event sequences:
 Setup & break sequence
 targeted operators for “twining”
 Fitness:Simulation performs setup sequence often
enough to trigger adaptation then does break
sequence
 Comparing
(1) break sequence emergent solution before and after adaptation plus
(2) achieved adaptation plus
(3) nearness of break solution to optimum,
again normalized by optimal break solution (main focus on maximizing (1))
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

27.  Least intrusive advisor variant only made
things less than twice worse
 Test system allowed to compare the danger
potential of different advisor variants
 Test system also found event sequences
showing off the advantages of the variants
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

28. CISDA‐09, CISDA‐11
Scenario:
Group of mobile and stationary sensor
platforms (with policies guiding movement of
mobile platforms)
 Harbors (simulated)
 Experimental robot setting (simulated and real)
Objective:
Protect a particular location
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

29.  Implementation of patrol and interception
policies (2) for harbor security in a GIS‐based
harbor simulation

Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

30.  Events: high‐level waypoints for attack
agents together with speeds for traveling
between them
Use a standard path planner to navigate
between waypoints (creating low level
waypoints around obstacles)
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

31.  Learner: Particle swarm system
Particle represents an attack (i.e. waypoints
and speed)
 Fitness: Several evaluation functions for a
particle based on a simulation run combined
using goal ordering structures
(<1, {<2, <3, <4},…<n)
 Nearness to target location
 Distance to defense agents
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

32.  A lot of the goal ordering structures are able
to find weaknesses for various policies and
various numbers of defenders and attackers
 Some ordering structures find more time‐
based attacks, others favor sacrifice attacks
 Some found attacks were not very intuitive
 would most probably be overlooked
 Simulation reflects real world well
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

33. Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

34.  Look at advisor concept!
 Before giving advice, test it!
 Using Monte‐Carlo simulations (SASO‐11)
 In adversary situation: use testing described
before
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger

35.  General guidelines for
 Fitness measure
 Targeted operators
 How can we tell approach to find new
problem?
 For self‐awareness:
 Run‐times are an issue
 What if agents need more global view for using
exception rules?
”Distributed” trigger
Testing for unwanted emergent behavior and dangerous self‐adaptations J. Denzinger