Governance of Risk in Public Policy - Nigel GibbensPresentation Transcript
Governance of risk in public policyNigel Gibbens, UK CVO
When is modelling useful?• Need help to make evidence based decisions in complex systems• Key issues: • Whether & how much taxpayers (& others) money to spend • Counter-factual – what is the cost of not acting? • Choice about disease mitigation – cost benefit (made in normal time, not in disease outbreaks) • Disease control (during outbreak) options, likely outcomes, cost- benefit.• Communication tool• Cross-discipline activity (e.g. disease, epidemiology, resource, economics)
Challenges• Danger of policy makers wanting a definitive answer for Ministers • modelling provides a level of certainty – which may be limited! • cost benefit analysis important, but need to reflect uncertainty • communication (of limitations; of technical outputs) • Beware the central estimate being received and then presented as fact, without caveat• In a high profile area, if you don‟t deploy modelling and make data available, others will model and visit the outcomes on you anyway. • consider friendly modelling vs. aggressive external modelling. • use of unknown assumptions/data e.g. 2001
Policy makers use evidence that comesfrom a variety of sources Economic Industry & Risk other groups assessmentScientific Legal PolicyModelling makers PoliticalVeterinary Public Epidemiological acceptability Anaylses Risk appetite
When are models fit for purpose?• Is scientific uncertainty dealt with properly?• Data quality known/assured• Evidence gaps identified• All assumptions are explained – not a black box• Level of confidence/uncertainties are explained• Outputs feed into options appraisal and IA, integrates with economic analysis
Modelling to inform policy development• Different arenas for modellers – long term strategic questions, with time for „data gathering‟ vs. shorter, specific questions using available data• Dangers of rapid modelling from existing data and importance of obtaining good data in as close to real time as possible• Models are not reality, they are an approximation of possible outcomes• Modelling in haste = more assumptions, less data = less accurate• Models do not remove uncertainty, but may give an illusion of knowledge that is unfounded• A particular problem in disease outbreaks, where decisions have to be made in absence of data (e.g. FMD)
Exotic disease modelling in Defra since2001Modelling accepted as useful for decision-making, now inDefra‟s Contingency Plan, & used for policy development. • Disease outbreak that might occur infrequently • Can define scenarios for more „simple disease‟ but require an outbreak standing capacity for more complex diseases • Standing capacity has established models; a good understanding of the systems modelled; data availability and quality, • thus, minimising problems with modelling at speed.
Concerns arising from exotic diseasemodelling • Conflicting advice/outcomes from different models • Representation and communication of uncertainty • Policy makers wary of using outputs, how can quality & validity be assured? • £2m on FMD modelling since 2002 – do benefits justify costs? • Increasing reliance on models but few understand outputs
Example: FMD vaccination - When tovaccinate?• Must be deployed at the right time: • Generally 4 days after vaccination there are good levels of protection in animals which rises to very good levels after 7 days.• Too soon: • as yet undetected disease could appear beyond the vaccination zone, seriously jeopardising the disease control policy and meaning vaccine would have been wasted. • IPs could appear in an area where vaccination would have a bigger impact than where originally deployed.• Too late: • the spread of disease would have been such that vaccinating would have little or no impact on disease control.Modelling doesnt give the answer, but can illustrate scenarios
Possible change to import controls - Risk of a rabiesoutbreak due to pet travelScenario 1: Single infectedanimal £1mScenario 2: Spread indomestic animals Mean cost of outbreak £10m 90% £2.2mScenario 3: Spread to wildlife x Increase in risk 1 / 220 £40m Expected annual cost £0.01m/year 9% 1%
Working with modelling - Dialogue helps Ministers Managers Policy Interface (via Modeller Intelligent Interface Policy Colleagues Customer) Stakeholders Expert Modellers
Role of an Intelligent Customer• Translation of the requirements from customers/users and the technical outputs of the modellers• Provides institutional memory to policy makers and provides technical understanding to commission and/or interpret the outputs; ensures modelling outputs are not ignored as a result of a lack of time and capacity to make use of them.• Improves the quality of future modelling work• Working with policy colleagues commissioning work to prevent unnecessary modelling work (e.g. spot opportunities to make use of previous modelling work/ existing models).
Constant Dialogue Policy / ICF Modeller Clear Specification This is the problem This is why it‟s important Defined approach This is how we will model it Review of approach These are our assumptions Here are some better assumptions Emerging findings Have you included this effect? The model says this Review of emerging findings We need more data on that Why does the model do that? Does this look sensible? Can you look at this? More findings The model now says this Review of final findings This is what we can infer Can you explain this better? Does this explanation make Can you expand that ? sense?
Broad Engagement Help to identify Keep you honest. parameter ranges Build wider credibility and review model behaviours Experts Peer ReviewStakeholders Help to ensure buy- Other Modellers in, and to force Modeller explanations to be Allow alternative clear and simple approaches to be compared
Good commissioning and delivery• Policy: • Set a good exam question • Try not to change your mind half way through • Explain who the outputs will be seen by• Modellers: • Answer the exam question • Be clear and honest about what you don‟t know • Focus the outputs for the audience(s)
Communication of outputs to policy makers• How much detail do policy makers need to know?• Do „fancy‟ outputs/graphs/charts confuse or mislead policy makers?• How is uncertainty represented?• Managing expectations as to what the model can do
Average annual cost of FMD outbreaksExpert opinions 2009 Median 95%CIAverage interval between outbreaks 25 years 10-59 yearsProbability outbreak is major 5% 2.3%-7.9%Minor outbreak cost £50m £18m-£125mMajor outbreak cost £400m £90m-£3,200mAverage outbreak cost (calculated) £68m £30m-£227mAverage annual cost (calculated) £3m £0.9m-£12m
What we might actually see (based onmedian value)
Economic analysis• Economic modelling provides CBA to inform decision, e.g. impacts of disease in terms of costs generated and benefits for action realised.• Need numbers from modelling outputs but danger that uncertainty can be lost• Currently we use outputs of disease modelling which then has economic analysis bolted on• Multidisciplinary working is key
Eradicating Koi Herpes Virus is notworthwhile• Conflicting stakeholder interests• Disease modelling showed eradication difficult and could cost £220 million over 20 years, but would bring benefits of less than £50 million
Exercise Silver Birch – large outbreak scenario - FMD vaccination decision• Use results of epidemiological model to: • Project the total cost of the outbreak • Compare cost with different control strategies • Trade-off cost against number of animals culled• Assess scale of costs to: • See different impact on different groups • Identify lessons
Use of Models• Epidemiology models (essential) – Planning – how big an outbreak to plan for? – Exercises – realistic scenarios – Outbreaks – geographic spread, size, should we vaccinate?, forward planning etc.• Resource models (nice to have) - Advice to policy on delivery constraints - Informing decisions on ramping up (and down) - Monitoring of actual vs predicted resources• Integrated epidemiology & resource (ideal) – Resource constraints (and time taken) informs the epi modelling – otherwise can be unrealistic• Integrated epidemiology & economic (ideal)
Lessons/What Next ?• Modelling has an important role to play in policy development.• Flexibility and willingness to engage in understanding the wider policy issues by “modellers” really helpful.• Policy customer should be clear at the outset what they want and what they want it for.• Can we put a value on modelling? • In an outbreak how much does modelling save? • what are the benefits of modelling?
Acknowledgments• Francesca Gauntlett – epidemiologist and Defra‟s Intelligent Customer for disease modelling• Simon Scanlon – Defra‟s lead animal health economist