This document presents a modelling framework for developing models in a regulatory environment. The framework is intended to promote dialogue between analysts, managers, and users. It balances efficiency, effectiveness, cost, and effort. The framework includes four phases (demand, supply, project, use), nine stages (approach, design, establish, etc.), and 34 steps (screening, problem definition, suitability, etc.). It provides 132 considerations to guide decisions at each step, balancing objective and subjective criteria. The goal is to integrate diverse perspectives, increase understanding, and achieve agreed solutions.

1. Analytics in
Context:
Modelling in a
Regulatory Environment
Presented to:
International Conference
on Analytics Solutions
Ottawa, Ontario,
September 29-30, 2014
Albert Simard
Integrated Knowledge Services
albert.simard@outlook.com

2. Outline
2
Modelling Context
Human, Technical
Framework
Structure, Hierarchy

3. 3
Diverse
Perspectives
What Analysts proposed What managers funded
What stakeholders wanted What users needed

4. Manager
Diversity
• Accountable for actions
• Situational pressure
• Broader view than analysis
• Organizational infrastructure
• Depth of understanding
• Confidence in results
• Risk tolerance
• Previous experience
• Belief system

5. Analyst
Diversity
• Objective Criteria
–Problem space
–Type of problem
–Available techniques
• Subjective Criteria
–Awareness
–Diversity
–Skill
–Experience
–Expertise
–Mental model

6. User
Diversity
Compatible Conflicting
Goals
Mutual
Interests
Autonomous
Competition
defence or victory
aggressive approach
no trust
secretive, hostile
Collaboration
joint or peer production
partnership approach
high trust
diverse, synergistic
Negotiation
mutual agreement
adversarial approach
nominal trust
structured, formal
Sharing
leverage knowledge
passive approach
moderate trust
benign, supportive

7. The
Challenge
Integrating the diverse
views to increase
understanding and
achieve a mutually
agreed solution!
7

8. 8
Modelling
Process
• Modelling combines science & computers;
judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge,
transparent, understandable.
• Complexity: Modelling is a dynamic feedback
process with delays and uncertainty.
• Development: techniques are well-understood;
management less understood and practiced.
• Use: Decision making under uncertainty, unknown
elements, outcome probabilities.

9. Spectrum of
Evidence
Quantitative (irrefutable evidence)
• mathematics, logic, proof
• science, engineering, technology
• statistics, data, facts, measurement
• collaboration, validation, management
• expertise, experience, judgement
• opinion, perception, bias
• belief, emotion, values
Qualitative (no evidence)

10. 10
Data
Attributes
A model and its data are inseparable;
they succeed or fail as one.
• Data Needs: Situation may involve nature, the
system, and/or intervention.
• Sampling: Statistics are essential to determine
how much data is needed.
• Source: Ownership? Use rights? Privacy &
security concerns?
• Scale: Time, space, and process scale must
match the situation.
• Quality: What level of accuracy, detail, and
completeness are needed?

11. Knowledge
Domains
Adapted from Kurtz and Snowden (2003)

12. Outline
12
Modelling Context
Human, Technical
Framework
Structure, Hierarchy

13. Framework
Objectives
13
• Support needs-driven and science-driven
analysis.
• Promote dialogue among modelers,
managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.

14. Framework
14
Design
• Reflect modelling, management, and
user perspectives.
• Balance efficiency and effectiveness
with cost and effort.
• Applicable to both demand and supply
approaches to modelling.
• Applicable to both logical and
computational models

15. Framework
15
Hierarchy
• Phase: (4) demand, supply, project, use
• Stage: (9) approach, design, establish,
develop, evaluate, identify, applicability,
implement, conclude
• Step: (34) screening, problem definition,
suitability, knowledge, data
• Consideration (132): classification,
statements, decision

16. Framework
16
Phases
Issue Model
Demand Supply
Project
Implement
End

17. Supply &
Demand
Supply: What can I
do with an
available model?
(Intelligence)
Demand: What
model do I need to
solve a problem?
(Regulation)

18. Supply &
Demand
18
Start
(methods)
Demand-driven
backward chaining,
closed question
Model
Supply-driven
forward chaining,
open question
Start
(model)
Uses

19. Demand
19
Issue
Approach
Design
Development
Acquire
Data
Generate
Knowledge
Phase
Out

20. 20
Framework Stages
Design
Acquire
Data
Generate
Knowledge
Approach
Out
End
Evaluation
Start: (Demand) Issue
Outputs
Implementation
Conclusion
(Manager)
Establishment
Development
(Modeller)
(Manager)
(User)
Applicability
Start (Supply) Model identification

21. Considerations Stage - Steps
21
Issue
Initial
Screening
Problem
Definition
Suitability
Knowledge
Evaluation
Data
Availability
Design
Recurrence
Importance
Problem space
Existence
Business
Function
Intended use
Time available
Situation
Needs
Existing
Gap
Needs
Attributes
Accessibility
Processing
Continue
Continue
Continue
Continue
Continue
Approach
Below threshold
Can’t define
Unsuitable
Excess gap
Inadequate
Generate
?
Yes
Acquire
?
Yes
Out
No
No

22. Considerations
22
• Explain what is being considered
• Classify a situation or write a short
description. (categories)
• Complete a statement. (template)
• Decide where to go next. (decision guide)
• Not a cookbook to be followed without
interpretation.
• Compliments experience & judgement;
doesn’t replace them.

23. Consideration
23
Template
• [Specified data] are needed to [develop, run]
a model of [function, process] because
[explanation].
• The [model name] needs [specified
knowledge] about [specified function,
process] because [explanation].

24. 24
• If a substantial majority of the data needed by the
model exist, go to Data Accessibility.
• If a majority of the data needed by the model exist, go
to Data Accessibility, subject to limits or conditions;
describe the limits.
• If existing data are adequate for a simpler or partial
model, go to problem definition; redefine the problem.
• If existing data are not adequate for a simpler or
partial model, go to Data Acquisition.
Decision
Guide

25. The Modelling
Framework
• Supports business objectives
• Facilitates horizontal integration
• Minimizes waste & inefficiency
• Maximizes likely success
• Documents & justifies decisions
“Using a clear blueprint first prevents
chaos latter.” Carla O’Dell (1998)