Talk presented at the 2017 Context-oriented programming workshop colocated with ECOOP in barcelona. This work presents a reinforcement learning based solution to resolve conflicts between dynamic adaptations.

1. @ncardoz
TICSw
Peace COrP
Nicolás Cardozo Ivana Dusparic Jorge H. Castro
Learning to solve conflicts between
contexts
9th International Workshop on Context-Oriented Programming (COP’17) - Barcelona, Spain
n.cardozo@uniandes.edu.co

2. Context-oriented programming
2
Enable the most
appropriate behavior to
the situations gathered
from the environment
(context)

3. What is the best behavior
3

4. What is the best behavior
4
method executions
observablebehavior

5. What is the best behavior
5
method executions
observablebehavior

6. Where do conflicts come from
6
[Mens et al. Managing the Context Interaction Problem, LASSY’17]

7. What is the best behavior
7
method executions
observablebehavior
This
paper
is
DOES
NOT
deal
with
com
position

8. 8
So…..
gathered contexts are
not always the most
appropriate behavior for
the system

9. Driving assistant
9
Car.prototype.drive=function() {
this.straight();
}
Proximity1 = new cop.Context()
OvertakeBehavior = Trait({
drive: function() {
car.steerLeft();
car.speedUp();
car.speedUp();
}
}
Proximity1.adapt(car, ProximityBehavior)
Speed control
Lane steering
Collision avoidance
[Gonzalez et al. Context Traits, Modularity’13]

10. Driving assistant
10
Car {
drive : function() {
this.straight();
}
}Proximity sensor
Proximity1.activate()
Context discovery
Car {
drive: function() {
car.steerLeft();
car.speedUp();
car.speedUp();
}
}
Adapted behavior
3m

11. Driving assistant
11
Car {
drive : function() {
this.straight();
}
}Speed sensor
Speed40.activate()
Car {
drive: function() {
car.SpeedUp();
}
}
Adapted behavior
40km/h
Context discovery

12. What to do?…
… Let’s learn it!
12

13. Learning the best action
13
Q-learning
Action
State
Q-value
2
3
1
4
10
100
0
-50
0
Q(s, a) = (1 ↵)Q(s, a) + ↵(ri + max
a0
i
Q(s0
i, a0
i))

14. Learning the best action
14
W-learning
Action
State
Q-value
2
3
1
4
10
100
0
-50
0
Action
State
Q-value
2
3
1
4
10
100
0
-50
0
Action
State
Q-value
2
3
1
4
10
100
0
-50
0
Wi(s) = (1 ↵)Wi(s) + ↵(Qi(s, ai) (ri + max
a0
i
Qi(s0
, a0
i)))
W-value

15. Learning the best action to a context
15
Speed sensor 40km/h
function processSpeedSensorData(speed) {
if(speed == 60)
learnerSpeed.reward(speed, 200);
else if(speed > 60)
learnerSpeed.reward(speed, 200);
else if(speed === 0)
learnerSpeed.reward(speed, -100);
else
learnerSpeed.reward(speed, 30);
}
Proximity sensor 3m
function processLaneSensorData(lane) {
if(lane == 2)
learnerLane.reward(lane, -50);
else
learnerLane.reward(lane, 30);
}
function processLaneSensorData(speed) {
if(speed == 2) {
LeftLane.activate();
RightLane.deactivate();
} else
RightLane.activate();
}
function processSpeedSensorData(speed) {
if(speed == 60) {
Speed60.activate();
OverSpeedLimit.deactivate();
UnderSpeedLimit.deactivate();
Speed0.deactivate();
} else if(speed > 60) {
OverSpeedLimit.activate();
} else if(speed === 0) {
Speed0.activate();
} else {
UnderSpeedLimit.activate();
}
}

16. Learning the best action to a context
16
Action
State
Q-value
1 1
0
-
0
Speed40.activate();
Action
State
Q-value
1 1
0
-
0
Action
State
Q-value
1 1
0
-
0
learnerSpeed
learnerCollision
learnerLane
W-value
(learner with the
largest w-value)

17. Validation
17
Static Resolution Dynamic Resolution Learned Resolution
% of crashes Steps at speed limit Steps on the correct lane1 2 3
h"ps://github.com/jcastrov/Peace-COrP

18. Static conflict resolution
18
Speed40.activate();
Proximity1.activate();
Speed40 = new cop.Context({});
…
cop.manager.addObjectPolicy(car,
[OvertakeBehavior,SpeedUpBehavior,LaneBehavior])

19. Dynamic conflict resolution
19
Speed40.activate();
Proximity1.activate();
…
SpeedUpBehavior = Trait({…})
OvertakeBehavior = Trait({…})
…
Activation time Observable Behavior

20. Results
20
0
10
20
30
40
50
60
70
80
90
100
Static Dynamic W-Learning-based
% of vehicle crashes
% of crashes

21. Results
21
0
5
10
15
20
25
30
% of steps on speed limit
Number of execution steps/actions taken
Static Dynamic W-Learning-based
Steps at speed limit

22. Results
22
0
10
20
30
40
50
60
70
80
90
100
1
101
201
301
401
501
601
701
801
901
1001
1101
1201
1301
1401
1501
1601
1701
1801
1901
2001
% of steps in correct lane
Number of execution steps/actions taken
Static Dynamic W-Learning-based
Steps on the correct lane

23. Wrap-up
23
@ncardoz
✓ Adapting to the most appropriate behavior …
….. with respect to the system’s objective
✓ Resolution is dynamic, but converges over time
✓ Reduced development complexity
➡ Composition of adaptations
➡ Use fine-grained contexts to learn behavior
Questions ?