Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modeling Framework to Support Evidence-Based Decisions


Published on

Describes a framework for modelling in a regulatory environment founded on sound scientific and knowledge management concepts. It includes 1) demand (isue-driven) and supply (model driven) approaches to modelling, 2) balancing modeler, manager, and user perspectives, 3) documentation to demonstrate due diligence, and a 700-term glossary.

  • Be the first to comment

  • Be the first to like this

Modeling Framework to Support Evidence-Based Decisions

  1. 1. A Mo de lling Frame wo rk: S uppo rting Evide nc e - Bas e d De c is io ns ModSimWorld Montreal, Quebec June 8-9, 2009
  2. 2. Main Messages • Models are needed to understand and predict the behavior of complex systems. • Models are needed to fulfill an agency’s mandate and support its core business. • Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability. Mo de ls s ho uld be used wisely 2
  3. 3. Outline I. Underlying Concepts Scientific underpinning II. Decision Guide Decision making III. Glossary Common understanding 3
  4. 4. Concepts About Models • What are they? – Simplified representations of reality. – Transform data, information, and knowledge into outputs. • Why do we use them? – Reality is too complex – Experiments are infeasible .. – Predict consequences – Increase understanding Nonaka (2000) 4
  5. 5. Concepts What is a Framework? “Structural outline of the components of an organization, system, or process and the relationships among them.” Understanding Knowledge Services NRCan (2006) 5
  6. 6. Concepts Framework Objectives • Support needs-driven and science- driven analysis. • Promote dialogue among modelers, managers, & users. • Reduce wasted time, effort, & money. • Provide a basis for planning and action. • Document and justify decisions. 6
  7. 7. Concepts Framework Design • Reflect modelling, management, and user perspectives. • Balance efficiency and effectiveness with cost and effort. • Applicable to both demand and supply approaches to modelling. • Applicable to both logical and computational models 7
  8. 8. Concepts Different Perspectives What developers proposed What managers funded What stakeholders wanted What users needed 8
  9. 9. Concepts Supply & Demand De mand: I have a problem that needs am odel. S upply: I have a m odel that solves your problem . 9
  10. 10. Concepts Modelling System Manage Use Nature, Internal External Society Models Models Develop Share Preserve Lost Knowledge Models Management 10
  11. 11. Concepts Modelling Process Modelling combines science & computers; judgement & experience; insight & intuition. • Principles: effort, simple, data, knowledge, transparent, understandable. • Complexity: Modelling is a dynamic feedback process with delays and uncertainty. • Development: techniques are well-understood; management less understood and practiced. • Use: Decision making under uncertainty, unknown elements, outcome probabilities. 11
  12. 12. Concepts Systems Hierarchy Ma ge na Mandate led Ma na me ge Business now io n ge nt K at Su me rm nt I nf o isi on pp ort Dec Models Data Policies Processes 12
  13. 13. Concepts Data A model and its data are inseparable; they succeed or fail as one. – Data Needs: Situation may involve nature, the system, and/or intervention. – Sampling: Statistics are essential to determine how much data is needed. – Source: Ownership? Use rights? Privacy & security concerns? – Scale: Time, space, and process scale must match the situation. – Quality: Level of accuracy, detail, and completeness are needed? 13
  14. 14. Concepts Information System Acquisition Storage Processing Outputs Environment Organization Knowledge Interface Access Interface Inputs Audience Channel System Outputs Search s Processing Use Data Retrieval Database Media Model Events Society Economy Availability Interoperability Integration Utility 14
  15. 15. Concepts Models and Knowledge 3. Complex 2. Complicated System: •Predictive feedback (+) •Feedback Behavior: •Non-linear (1:?) •Linear (1:n) Approach: •Simulation •Mathematics Model: •Stochastic •Deterministic Decision: •Uncertainty •Certainty Basis: •Tacit knowledge •Explicit knowledge •Emergent •Flow-through (-) •Disorganized •Fixed (1:1) •Scenario analysis •Planning •Mental •Mechanistic •Reaction •Automated •Intuition •Data, facts 4. Chaotic 1. Common 15
  16. 16. Outline I. Underlying Concepts Scientific underpinning II. Decision Guide Decision making III. Glossary Common understanding 16
  17. 17. Decision Guide - Hierarchy • Phase: (3) demand, supply, project • Stage: (7) approach, design, establish, develop, evaluate, implement, conclude • Step: (34) screening, problem definition, suitability, knowledge, data • Consideration (132): recurrence, importance, problem space, existence 17
  18. 18. Decision Guide - Guide Stages Start: (Demand) Issue Outputs End D1 7 (Manager) Approach Conclusion 6 Acquire Implementation Data Out (User) D2 5 Design Evaluation Generate 3 (Manager) Knowledge Establishment 4 S2 Development Applicability (Modeller) S1 (All) Start (Supply) Model identification 18
  19. 19. Guide Decision Guide Phases Issue Model Demand Supply Project End 19
  20. 20. Guide Supply & Demand Demand-driven Supply-driven backward chaining, closed question forward chaining, open question Start Start (model) Uses (use) Model 20
  21. 21. Guide Demand Phase Issue D1. Approach Generate Acquire Out Knowledge Data D2. Design Development 21
  22. 22. Approach Guide Stage Issue D1.1 Recurrence Below threshold Importance Initial Problem space Screening Existence Continue D1.2 Business Can’t define Function Problem Intended use Definition Continue D1.3 Time available Unsuitable Suitability Out Situation Continue D1.4 Needs Knowledge Excess gap Existing Generate Gap Evaluation ? No Continue Yes D1.5 Needs Attributes Data Inadequate Acquire Accessibility Availability ? No Processing Continue Yes Design 22
  23. 23. Guide Decision Guide Considerations • Explains the question. • Classify a situation or write a short description. • Complete a statement template. • Decide where to go next. • Not a cookbook to be followed without interpretation. • Compliments experience & judgement; doesn’t replace them. 23
  24. 24. Guide Supply Phase Model S1. Identification Development S2. Applicability Conclusion Evaluation 24
  25. 25. Applicability Guide Existing Model Stage S1 Search Description Identification S2.1 Specification criteria (4) Specifications S2.2 Development criteria (3) Development S2.3 Data inaccessible Data criteria (5) Data Availability Acquisition S2.4 Knowledge unjustified Knowledge base Conclusion criteria (3) S2.5 unsuitable Business line Function Suitability Development modify Evaluation 25
  26. 26. Guide Project Phase Design Applicability 3. Project Out Establishment 4. Development 5. Evaluation Situation 6. Implementation 7. Project End Conclusion 26
  27. 27. Developmen Guide Project Establishment Awareness 4.1 t Stage Understanding Consensus Interaction Hierarchy 4.2 Relationships exit Indicators Conceptualization Review continue Logic 4.3 Computation exit Debugging Construction Review continue Inventory Attributes 4.4 Consistency exit Review Verification continue 4.5 Uncertainty exit Representation Validation Review continue Evaluation 27
  28. 28. Outline I. Underlying Concepts Scientific underpinning II. Decision Guide Decision making III. Glossary Common understanding 28
  29. 29. Glossary Glossary • Background (introduction, methods, references). • Taxonomy (organization, nature, risk analysis, content, modelling, concepts). • Definitions - 650 terms from six sources. • Links to taxonomy and related terms. 29
  30. 30. Glossary Sample Definition Mo de l: Abstract and sim plified construct or representation of reality in the form of a pattern, description, or definition that show the essential s structure, relationships, and w orkings of a concept, process, or system . (s e e modelling approach, function, mode lling me thods , proce s s , re lations hip, re pre s e ntation, s y s te m) 30
  31. 31. Mo de lling Frame wo rk: •Supports an agency’ business s •Facilitates horizontal integration •Minimizes waste & inefficiency •Maximizes likely success •Documents & justifies decisions “ Using a clear blueprint first prevents chaos latter.” 31 Carla O’ (1998) Dell