This work is presented at the 2020 Workshop on Testing for Deep Learning and Deep Learning for Testing (DeepTest) co-located with ICSE held virtually. This work analyzes the possibility of using neuron coverage as a test adequacy metric for Deep Reinforcement Learning (DeepRL). This work spawns from the positive results in using neuron coverage to test (deep) neural networks. However, testing DeepRL systems, brings its own challenges. In the presentation, and associated paper, we discuss the characteristics of DeepRL that prevent promoting neuron coverage as an adequacy testing metric. The paper is available at: https://deeptestconf.github.io/pdfs/2020-Trujillo-DeepTest.pdf

1. 1
Does Neuron Coverage Matter for Deep
Reinforcement Learning?
A Preliminary Study
Miller Trujillo, Mario Linares-Vásquez, Camilo Escobar-Velásquez, Ivana Dusparic, Nicolás Cardozo

2. 2
Reinforcement
learning
• Learn optimal actions for specific environment
conditions by trial-and-error.
• Pick an action for a given state to maximize reward.
• Q-learning, state-action pairs.
Policy π(St, at)

3. 3
Deep
Reinforcement
Learning (DRL)
• A neural network can be used to approximate
a policy function.
• Useful when the state space or action space
are too large to be known.

4. 4
Is this the correct/expected behavior?
out
Testing DRL Behavior
In
• Reward-based
• Test case Generation

5. 5
DNN Testing
• Proposes neuron coverage as test
adequacy metric for DNNs.
Neuron Coverage is an adaptation of statement coverage.
• Empirically demonstrates that changes
in neuron coverage are statistically
correlated with changes in the actions
of self-driving cars.

6. Objectives
Does traditional DNN testing techniques apply to DRL?
RQ0: Is there any difference in
the evolution patterns of neuron
coverage for the different layers
in a DeepRL system?
RQ1: Is there any correlation
between neuron coverage and
cumulative reward in a
DeepRL system?
6

7. 7
Mountain Car initial state
The Mountain Car problem
Empirical Study Design

8. 8
The Mountain Car problem
Goal:
Climb the mountain in maximum 200
steps.
Environment state:
Position ( ) and velocity ( ).
Actions:
Accelerate towards left, neutral/no
action, Accelerate towards right.
x v
Bounds:
The position is bounded by
The velocity is bounded by
Reward:
[−1.2,0.6]
[−0.07,0.07]
R(x, v) =
{
10, if x ≥ 0.5
−1, otherwise.
Empirical Study Design

9. We used two different models chosen randomly from GitHub.
Model A Model B
https://github.com/pylSERhttps://github.com/branavg
Empirical Study Design
Models
9

10. 10
Empirical Study Design
Experiment description

11. 11
Empirical Study Design
Models parameters
Batch size:
(learning rate)
(discount factor)
-greedy exploration strategy
starts at and decays after each episode.
32
α = 0.001
γ = 0.99
ϵ
ϵ 1 0.05

12. 12
Empirical Study Design
Analysis Methods
• Neuron Coverage (NC)
• Cumulative Neuron Coverage
(CNC)
• Neuron Layered Coverage (NLC)
• Cumulative Neuron Layered
Coverage (CNLC)

13. Empirical Study Design
Analysis Methods Example
First iteration of first episode Second iteration of first episode Third iteration of first episode
L1L1 L1L2 L2 L2L3 L3 L3
Cumulative
13

14. / This is the place
for subtitle /
Results
14
Results
CNC and CNLC distribution for training and
testing
Model A

15. 15
Results
Model A
NC, NLC, and Reward per episode for training
R(x, v) =
{
10, if x ≥ 0.5
−1, otherwise.

16. / This is the place
for subtitle /
Results
16
CNC and CNLC distribution for training and
testing
Results
Model B

17. 17
Results
NC, NLC, and Reward per episode for training
Model B
R(x, v) =
{
10, if x ≥ 0.5
−1, otherwise.

18. Discussion
Better coverage in training does not
necessarily mean better coverage in
testing.
Neuron coverage is not sufficient to
reach substantial conclusions about
the design or structure of DeepRL
networks.
Training and testing evolution
patterns are not necessarily the
same.

19. 19
Future work
• The evaluation of neuron coverage as
adequacy metric should be extended
to other DeepRL scenarios to
consolidate the validity of our results,
in exploring for a metric that can
correlate coverage and maximize
reward.
• A similar approach could be used to
evaluate other DNN-based learning
models, as means to evaluate the
applicability of coverage testing to
ML.

20. 20
SUMMARY

21. Thank you!
Miller Trujillo
ma.trujillo10@uniandes.edu.co
Mario Linares-Vásquez
m.linaresv@uniandes.edu.co
Camilo Escobar-Velásquez
ca.escobar2434@uniandes.edu.co
Nicolás Cardozo
n.cardozo@uniandes.edu.co
Ivana Dusparic
duspari@scss.tcd.ie