Slides from Bhakti Stephan Onggo's presentation at the Operational Research Society's annual conference (OR58) in Portsmouth (5-8 September 2016)

1. Test-Driven Simulation Modelling (TDSM)
Bhakti Stephan Onggo
Department of Management Science
s.onggo@lancaster.ac.uk

2. • We propose a novel
technique for the verification
and validation of simulation
models
• The technique explicitly
embeds verification and
validation to simulation
modelling and it can be
implemented using standard
unit testing tools
• The proposed technique has
been tested on ABS (Agent-
Based Simulation) and DES
(Discrete-Event Simulation)
Summary
2

3. • Model verification and validation is an important activity in
simulation
• Verification: to check the correctness of a simulation model
against its conceptual model
• Validation: to build confidence that the model is adequate for its
intended use
• In an incremental model development, we need to make sure a new
addition to the model does not break the correctness and validity of
the earlier version of the model
• Iterative -> automate the verification and validation?
• Model documentation is a good practice, so does documentation
about how to verify and validate the model
• Accompany a model with its verification and validation cases
• It is especially important if a significant part of the model is written in
a programming language (e.g. Visual Logic, Java)
Why is our research important?
3

4. • Hence, the contributions of our work are
• Proposing a method to help with simulation model verification
and validation that can be automated using a software tool such
as a unit testing tool
• Our method promotes the principle of parsimony (incremental
model development)
• Our method encourages modellers to think about V&V early
• Our method promotes documenting verification and validation
cases of a simulation model
Why is our research important?
4

5. • Test-Driven Development (TDD) in
software engineering
• TDD is suitable for a software
development process that is
iterative, incremental and
evolutionary (such as simulation)
• Exploratory study on the use of
TDD as a simulation model
verification tool has been done
(Collier & Ozik, Asta et al. and
Onggo et al.)
• Similar approach to TDD has been
explored too (Gurcan et al., Niazzi
et al. and Thiele et al.)
Asta, S., E. Özcan, and P.-O.
Siebers. 2014.
Collier, N., and J. Ozik. 2013
Onggo, B.S.S., C. Indriany, and
M. Gunal. 2014
Gurcan, O., O. Dikenelli, C.
Bernon. 2013.
Niazzi M.A., A. Hussain, and
M. Kolberg. 2009
Thiele, J.C., Kurth, W., and
Grimm V. 2014.
Background
5

6. Method
• We extended TDD to cover model
validation and embed TDD into the
standard simulation modelling
process – we call this Test-Driven
Simulation Modelling (TDSM)
• For each verification or validation
case, we need to specify how the
model (or part of the model)
should behave
• A validation case may require the
model to be run multiple times to
build the confidence intervals
• Sensitivity analysis can be specified
in which a validation case will test
one point in the exploration space
6

7. Case 1: DES
• SharpSim - a general
purpose DES library
written in Visual
CSharp (C#)
• M/M/1 Queue
RUN ARRIVAL
START
SERVICE
END
SERVICE
ID = 1;
S = 2;
Customer c =
new Customer(ID);
edgeR_A.attribute = c;
eStart.queue.Add
(ev.evnt.parameter);
edgeA_S.attribute =
eStart.queue[0];
ID++;
Customer c =
new Customer(ID);
edgeA_A.attribute = c;
if (S > 0)
edgeA_S.condition = true;
else
edgeA_S.condition = false;
(S > 0)
∫
∫
(Q > 0)
eStart.queue
.RemoveAt(0);
S--;
edgeS_E.condition = true;
edgeS_E.attribute = c;
S++;
if (eStart.queue
.Count() == 0)
edgeE_S.condition = false;
else {
edgeE_S.condition = true;
edgeE_S.attribute =
eStart.queue[0];
}
Stats.Collect
ts
ta
ta
7

8. Case 1: DES Verification
Scenario 1: check correctness if
arrivals at 1, 2 and service time 4, 3
8

9. Case 1: DES Validation
Scenario 2: Validate the model by evaluating whether the expected waiting time from the
analytic model (i.e. 0.1922) is within the 95% confidence interval of the expected waiting
time from the simulation model
9

10. Case 2: ABS
• Agent-based maritime
search-operation
simulation (MASSIM) –
Onggo & Karatas
(Wintersim 2014)
• Various search
algorithms: random,
exhaustive, parallel,
creeping, etc.
• Detection functions:
deterministic, various
probabilistic functions
• Single searcher,
multiple searchers
10

11. Case 2: ABS Validation
Exhaustive search
Random search
11

12. Case 2: ABS Validation
Parallel search
12

13. • TDSM is a useful technique in the verification and validation of
simulation models
• TDSM works based on two principles:
• modellers should be guided by the knowledge of relevant
analytic models that are simpler and have known solutions
when they are building a complex model
• Modellers must think about how their model is going to be
verified and validated, even before they begin to develop the
model
• TDSM and other methods (Gurcan et al. 2013, Niazzi et al. 2009
and Thiele et al. 2014) aim to help modellers validate a simulation
model
• TDSM is useful for auto-verification or auto-validation of simulation
models
Conclusion
13

14. • Evaluate TDSM using an experiment with real modellers
• Find an optimum TDSM strategy that balances test coverage and
cost
• Model verification and validation is a combinatorial problem
• We need tool to evaluate the test cases (unit tests)
• A unit test can contain mistakes too!
• We need to embed TDSM components in a Visual Simulation
Modelling tool
• Get some feedback
Next steps/future work
14

15. References
• Onggo BSS and Karatas M. (2016) 'Test-driven
simulation modelling: A case study using agent-based
maritime search-operation simulation', European
Journal of Operational Research, 254 (2):517-531
• Onggo, B. S. S. , Indriany, C. & Gunal, M. (2014). Test-
driven simulation modelling. In A. Bruzzone, E. Çayırcı,
M. Günal, A. R. Kaylan, M. Massei, & C. Zanni-Merk
(Eds.), Proceedings of the 7th international workshop
on applied modeling and sim- ulation (WAMS14) (pp.
43–48). Turkey: Istanbul .
15