Presentation by Lieke Melsen (Wageningen University), Janneke Remmers (Wageningen University) and Carine Wesselius (Deltares), at the The choice - A workshop for modelers, during Delft Software Days - Edition 2022. Wednesday, 19 October 2022.

1. The Choice:
A workshop for
modelers (and others)
Lieke Melsen, assistant professor, Wageningen University
Carine Wesselius, consultant/researcher, Deltares
Janneke Remmers, PhD-candidate, Wageningen University

2. This Workshop
• Who is joining today
• “The Choice”
• Theory on choices in modeling
• Break
• Roles and responsibilities in Modeling
• Discussion
• Closure

4. Different people,
different backgrounds,
different experiences,
different choices!
But….what is the effect on the
end-result?

5. Water
Level
Levee
Transition
Height

6. Water
Level
Levee
Transition
Height

7. Water
Level
Levee
Transition
Height

8. Modelling decisions: example
Model
Platform
Model
Model
Version
Schemati-
sation
Other
Platform
Other
Model
Other
Version
Other
Scemati-
sation

9. Interview study
14 interviews
~ 1400 minutes
~ 350 pages

10. My study: example questions
How do you generally make this decision?
Why option X?
When would you choose a different option?

11. My study: example questions
How do you generally make this decision?
Why option X?
When would you choose a different option?
How much confidence do you have in a model?
How do you estimate your influence as a modeller?

12. Two modellers: introduction

13. Two modellers: introduction
Governmental organisation
Senior
Rainfall-runoff modelling

14. Two modellers: model structure

15. Two modellers: model structure
“It is of course
dependent on
the goal.”

16. Two modellers: model structure
“It is of course
dependent on
the goal.”
“Chosen because
municipality
already worked
with it.”

17. Two modellers: model structure
“It is of course
dependent on
the goal.”
“Chosen because
municipality
already worked
with it.”
“Generally used
within the
governmental
organisations.”

18. Two modellers: model structure
“It is of course
dependent on
the goal.”
“Chosen because
municipality
already worked
with it.”
“Generally used
within the
governmental
organisations.”
“We trust the
model. Why, I
don’t know.”

19. Two modellers: sensitivity analysis

20. Two modellers: sensitivity analysis
We always ask a
consultancy company
to do this.

21. Two modellers: sensitivity analysis
“We think it is
important.”
We always ask a
consultancy company
to do this.

22. Two modellers: sensitivity analysis
“We think it is
important.”
We always ask a
consultancy company
to do this.
“Well... It is
maybe not very
explicitly done.”

23. Two modellers: sensitivity analysis
“We think it is
important.”
We always ask a
consultancy company
to do this.
“Well... It is
maybe not very
explicitly done.”
“I would find it difficult
anyway. How do you do
such a thing?”

24. Two modellers: calibration

25. Two modellers: calibration
“Actually, it depends on
what the consultancy
company suggests.”

26. Two modellers: calibration
“The quality of
the data also
plays a role.”
“Actually, it depends on
what the consultancy
company suggests.”

27. Two modellers: calibration
“The quality of
the data also
plays a role.”
“The data should
be available for
calibration.”
“Actually, it depends on
what the consultancy
company suggests.”

28. Two modellers: calibration
“The quality of
the data also
plays a role.”
“The data should
be available for
calibration.”
“Actually, it depends on
what the consultancy
company suggests.”
“You look at what is there,
if it is trustworthy and if I
can execute my
calibration based on this.”

29. Perceived influence of modellers

30. Perceived influence of modellers
“I think you have
a lot of influence
as a modeller.”

31. Perceived influence of modellers
“I think you have
a lot of influence
as a modeller.”
“Too much”
“Quite some”

32. Perceived influence of modellers
“I think you have
a lot of influence
as a modeller.”
“Too much”
“Quite some”
“I don’t think it
matters that
much.”

33. Perceived influence of modellers
“I think you have
a lot of influence
as a modeller.”
“Too much”
“Quite some”
“I don’t think it
matters that
much.”
“Same result”
“Different method”

34. Investigating the modeller...
14 in-depth interviews
3 different institutes (France, Belgium, Germany)
February – June 2020
835 minutes of material (13.9 hours)
110.767 words (about 220 pages)
Inductive thematic content analysis
3 Group leaders
4 Senior researchers
4 Post-docs
3 PhD-candidates
107 codes, 87 motivations

35. Melsen, 2022. It takes a village to run a
model – the social aspects of
hydrological modelling, WRR

37. Epistemic
presumed to promote the
truth-like character of science
Contextual

38. Epistemic
presumed to promote the
truth-like character of science
Contextual

39. Epistemic
presumed to promote the
truth-like character of science
Contextual

40. This is a self-sustaining system

41. • Model etiquette to move decisions
from contextual to epistemic
Contextual
Epistemic
• Determine which decisions are
most sensitive, and invest in
underpinning those decisions
Data
availability
Time
constraints
projects
Computer
resources

42. BREAK

43. Quality assurance in modelling
Quality
Assurance

44. Quality assurance in modelling – a shared responsibility
CLIENT CONTRACTOR

45. Quality assurance in modelling – a shared responsibility
Problem owner
CONTRACTOR
Contract lead
Requires high quality to justify decisions
to stakeholders and constituency
Actions: Non-biased, open scope.
Requires high quality to justify
results to stakeholders and problem
owner
Actions: Provide resources for
quality assurance. Explicit
requirements in contract on model
quality (assurance).

46. Quality assurance in modelling – a shared responsibility
Problem owner
Employer
Contract lead
Requires high quality to justify decisions
to stakeholders and constituency
Actions: Open scope.
Requires high quality to justify results to
stakeholders and problem owner
Actions: Provide resources for
documentation. Assurance of
documentation. Explicit requirements in
contract on model quality (assurance).
Modelling team
Modeller
A good market position, legal responsibility?
A good market position, internal position
Work performance, appreciation
Actions: Facilitation of peer-review,
internal model guidelines, intervision,
professional development employees
Actions: Peer-review, development
model guidelines, intervision, training of
new employees
Actions: Make choices with best intentions,
provide substantiation, document choices,
sensitivity analysis where necessary

47. Quality assurance in modelling
Quality assurance as part of
Model etiquette
Social norms and practices
that are considered accepted
and good behaviour, in this
case related to modelling and
documentation practices

48. Quality assurance in modelling – Regulatory contexts
Certification
• Requirements and
standards are determined
per project
• No implications if
standards are not met
Voluntarily Legal/juridical
• Can be organized at
different levels (national,
internally)
• Cost of Compliance
involved
• Easy for client to require
certain standards, specified
in the certification
• Model set-up / choices are
prescribed by law
• All contractors have to
obey the same
requirements
• Slow uptake of new
insights
“statutory or legislative tools and frameworks that are used to enforce action toward specific goals”

50. Thank you!
Lieke Melsen
Assistant Professor
Wageningen University
lieke.melsen@wur.nl
Carine Wesselius
Consultant/Researcher
Deltares
carine.wesselius@deltares.nl
Janneke Remmers
PhD-candidate
Wageningen University
janneke.remmers@wur.nl