Increasing reliance on models across Government but few people who understand outputs. Be clear about what we can use modelling for, what we can’t and the amount of confidence you attach to its findingsWorking with other specialists ( vets, epidemiologists, social scientists, economists, etc)Impractical to ‘make changes’ in reality to explore scenarios and possible outcomes: Costly, Timely, Practically, Politically!
Challenge of translating modelling outputs into layman’s terms for different audiences – Ministers, stakeholders, public. But the essence of the outputs, assumptions and uncertainty needs to travel to the highest level.
Modelling outputs should be part of wider analysisJudgement: take a decision on basis of: Politics with an implicit ethical dimension, strategy and vision, Legality, Evidence and balance of probability, Timescale, practicality, affordability.
Uncertainty: systems uncertainty (about biology, society, behaviour)Model uncertainty: data input, data processing, conceptualisation of system, Post modelling uncertainty: verification, validation, sensitivity analysis, parameter estimationPeople may be trusting or critical of modelling – which affects their tolerance of uncertainty Or whether the work is high impact or low impact may affect whether people are supportive of modelling on a particular question
Models are not reality, they are an approximation of possible outcomesShorter model = more assumptions, less data = less accurate
Modelling in the face of a disease outbreakDisease outbreak that might benefit from modelling may occur 1 in every 3-5 yearsCan define scenarios for more ‘simple disease’ but require outbreak standing capacity for more ‘complex’ diseasesThe Defra strategy currently sets out a framework, and provides money to support FMD quantitative modelling only (could be generic for quantitative modelling of other exotic diseases in due course -resources permitting).
Benefits are realised in relation to analysis of costs and benefits of government and others investments, for example in disease preparedness or risk mitigation such as bio-security requirements for farmers, and scenario analysis to help inform future decisions or contingency planning exercises. Less easily realised in the teeth of an outbreak where modelling will rarely be sufficiently timely to inform key decisions.
Timing importantNeed to manage expectations of public and ministersDisease will appear to continue increasing until vaccination has time to take effectWill never have all the information needed but if wait too long may miss benefitWhile ideally a vaccination campaign might take place against a backdrop of a clear epidemiological picture, this is not necessarily possible in the early stages of an outbreak when disease information may be incomplete.
It’s a nice example because it includes: expert description of outbreak scenarios and our response, costing of those scenarios and response, expert opinion of what proportion of incursions would lead to each scenario, quantitative VRA of incursion probability with then/relaxed controls (and so the change between the old controls and the proposed relaxed regime), all leading finally to an expected change in outbreak cost per year - for comparison with the benefits of relaxing controls (not shown in the graphic, but mentioned in the text). Note that there are no measures of uncertainty shown here. A central estimate is used to enable an illustration of possible costs.Some policies cause costs every year – in precaution, prevention and preparedness – but potential outbreaks of disease may be rare. These controls could be justified if they make a big enough difference to the cost and/or frequency of outbreaks.Pet travel controls for rabies are now harmonised across the EU. This means more relaxed controls for some countries (including UK) and so a small increase in the risk of a rabies incursion. Using a quantitative veterinary risk assessment showed that the changes might lead to an additional rabies incursion about once every 220 years. Experts described the outbreak scenarios and we estimated the expected costs per year at just £10,000 (Figure 3). The benefits enjoyed by those who enter or return to the UK with their pets vastly outweigh these costs.
Data sharing – an issue for both modellers and policy (who may be providing the data, and may be experts in how it is collected)How much policy interaction can we expect modellers to undertake?Does exposure to policy and understanding of the data result in better outputs or better models? (we think so, and dialogue between these groups helps understanding on both sides)Importance of modeller’s ability to communicate to the customer and for the customer to have someone with appropriate skills at the science: policy interface (intelligent customer)If modelling leads to a policy that changes behaviour, changes whole output. This has to be understood by the modeller and the policy makerPolicy makers may more readily accept outputs from models that they have been involved in developing How simple or complex should models that inform policy be? (data driven = data dependent)
ICF as link between modellers and policy – assisting policy with commissioning new work (framing the questions, regular review of progress)Intelligent customer role to help and steer. If modelling too complicated with many layers = not much useProcurement: esp for non R&D routeLink up with R&D commissioning of new models
Modeller needs to engage with others outside the policy cycleExperts e.g. data providers, epidemiologists, economists, social scientists.Acceptance of new models by the modelling community and others is important – which can be via publication in a peer-reviewed journal or other peer reviewIdentification of best practice, an accepted approach – there are groups of modellers working on frameworks for best practice eg Prof Mark Woolhouse at Edinburgh has published a best practice guide to Quantitative Veterinary Epidemiology.Validation of models, by scientists, by policy makers.Helps to respond to challenge from unfriendly modellers / Anderson review etc
FMD values, and shows how we might use the expert opinions about the scale of risk to predict what we might expect FMD to cost us per year in future. NB the wide margins of uncertainty (CIs, which the risk analysts called credible intervals in this case) that experts attached to their median views. The following slides with the graphs then show what the future might turn out to look like if the MEDIAN values were true and the exotic disease world was a stochastic system (they come from a simple Monte Carlo simulation). The single real world outcome could be completely different (e.g. with few or many outbreaks over 100 years, with or without a massive outbreak).
An example of applying modelling to inform a difficult policy decision
Blue = Infected Premises and Dangerous Ccontact cull onlyRed = as blue, but with vaccination to protect herds not already exposedValues each of the physical aspects of the outbreak to estimate total costVaccination generally increases total outbreak cost except in very large outbreaks In larger outbreaks, vaccination substantially reduces IPs & animals culled
How do we encourage joined up development of epi – economic models (multidisciplinary working)Looking at options to reduce cost to government becoming increasing important, and require a framework to do this.
Defra modelling consortiumWorking to understand each other, and the data, betterSharing respective knowledge and expertise
Governance of Risk in Public Policy - Nigel Gibbens
Governance of risk in public policyNigel Gibbens, UK CVO
When is modelling useful?• Need help to make evidence based decisions in complex systems• Key issues: • Whether & how much taxpayers (& others) money to spend • Counter-factual – what is the cost of not acting? • Choice about disease mitigation – cost benefit (made in normal time, not in disease outbreaks) • Disease control (during outbreak) options, likely outcomes, cost- benefit.• Communication tool• Cross-discipline activity (e.g. disease, epidemiology, resource, economics)
Challenges• Danger of policy makers wanting a definitive answer for Ministers • modelling provides a level of certainty – which may be limited! • cost benefit analysis important, but need to reflect uncertainty • communication (of limitations; of technical outputs) • Beware the central estimate being received and then presented as fact, without caveat• In a high profile area, if you don‟t deploy modelling and make data available, others will model and visit the outcomes on you anyway. • consider friendly modelling vs. aggressive external modelling. • use of unknown assumptions/data e.g. 2001
Policy makers use evidence that comesfrom a variety of sources Economic Industry & Risk other groups assessmentScientific Legal PolicyModelling makers PoliticalVeterinary Public Epidemiological acceptability Anaylses Risk appetite
When are models fit for purpose?• Is scientific uncertainty dealt with properly?• Data quality known/assured• Evidence gaps identified• All assumptions are explained – not a black box• Level of confidence/uncertainties are explained• Outputs feed into options appraisal and IA, integrates with economic analysis
Modelling to inform policy development• Different arenas for modellers – long term strategic questions, with time for „data gathering‟ vs. shorter, specific questions using available data• Dangers of rapid modelling from existing data and importance of obtaining good data in as close to real time as possible• Models are not reality, they are an approximation of possible outcomes• Modelling in haste = more assumptions, less data = less accurate• Models do not remove uncertainty, but may give an illusion of knowledge that is unfounded• A particular problem in disease outbreaks, where decisions have to be made in absence of data (e.g. FMD)
Exotic disease modelling in Defra since2001Modelling accepted as useful for decision-making, now inDefra‟s Contingency Plan, & used for policy development. • Disease outbreak that might occur infrequently • Can define scenarios for more „simple disease‟ but require an outbreak standing capacity for more complex diseases • Standing capacity has established models; a good understanding of the systems modelled; data availability and quality, • thus, minimising problems with modelling at speed.
Concerns arising from exotic diseasemodelling • Conflicting advice/outcomes from different models • Representation and communication of uncertainty • Policy makers wary of using outputs, how can quality & validity be assured? • £2m on FMD modelling since 2002 – do benefits justify costs? • Increasing reliance on models but few understand outputs
Example: FMD vaccination - When tovaccinate?• Must be deployed at the right time: • Generally 4 days after vaccination there are good levels of protection in animals which rises to very good levels after 7 days.• Too soon: • as yet undetected disease could appear beyond the vaccination zone, seriously jeopardising the disease control policy and meaning vaccine would have been wasted. • IPs could appear in an area where vaccination would have a bigger impact than where originally deployed.• Too late: • the spread of disease would have been such that vaccinating would have little or no impact on disease control.Modelling doesnt give the answer, but can illustrate scenarios
Possible change to import controls - Risk of a rabiesoutbreak due to pet travelScenario 1: Single infectedanimal £1mScenario 2: Spread indomestic animals Mean cost of outbreak £10m 90% £2.2mScenario 3: Spread to wildlife x Increase in risk 1 / 220 £40m Expected annual cost £0.01m/year 9% 1%
Role of an Intelligent Customer• Translation of the requirements from customers/users and the technical outputs of the modellers• Provides institutional memory to policy makers and provides technical understanding to commission and/or interpret the outputs; ensures modelling outputs are not ignored as a result of a lack of time and capacity to make use of them.• Improves the quality of future modelling work• Working with policy colleagues commissioning work to prevent unnecessary modelling work (e.g. spot opportunities to make use of previous modelling work/ existing models).
Constant Dialogue Policy / ICF Modeller Clear Specification This is the problem This is why it‟s important Defined approach This is how we will model it Review of approach These are our assumptions Here are some better assumptions Emerging findings Have you included this effect? The model says this Review of emerging findings We need more data on that Why does the model do that? Does this look sensible? Can you look at this? More findings The model now says this Review of final findings This is what we can infer Can you explain this better? Does this explanation make Can you expand that ? sense?
Broad Engagement Help to identify Keep you honest. parameter ranges Build wider credibility and review model behaviours Experts Peer ReviewStakeholders Help to ensure buy- Other Modellers in, and to force Modeller explanations to be Allow alternative clear and simple approaches to be compared
Good commissioning and delivery• Policy: • Set a good exam question • Try not to change your mind half way through • Explain who the outputs will be seen by• Modellers: • Answer the exam question • Be clear and honest about what you don‟t know • Focus the outputs for the audience(s)
Communication of outputs to policy makers• How much detail do policy makers need to know?• Do „fancy‟ outputs/graphs/charts confuse or mislead policy makers?• How is uncertainty represented?• Managing expectations as to what the model can do
Average annual cost of FMD outbreaksExpert opinions 2009 Median 95%CIAverage interval between outbreaks 25 years 10-59 yearsProbability outbreak is major 5% 2.3%-7.9%Minor outbreak cost £50m £18m-£125mMajor outbreak cost £400m £90m-£3,200mAverage outbreak cost (calculated) £68m £30m-£227mAverage annual cost (calculated) £3m £0.9m-£12m
What we might actually see (based onmedian value)
Economic analysis• Economic modelling provides CBA to inform decision, e.g. impacts of disease in terms of costs generated and benefits for action realised.• Need numbers from modelling outputs but danger that uncertainty can be lost• Currently we use outputs of disease modelling which then has economic analysis bolted on• Multidisciplinary working is key
Eradicating Koi Herpes Virus is notworthwhile• Conflicting stakeholder interests• Disease modelling showed eradication difficult and could cost £220 million over 20 years, but would bring benefits of less than £50 million
Exercise Silver Birch – large outbreak scenario - FMD vaccination decision• Use results of epidemiological model to: • Project the total cost of the outbreak • Compare cost with different control strategies • Trade-off cost against number of animals culled• Assess scale of costs to: • See different impact on different groups • Identify lessons
Use of Models• Epidemiology models (essential) – Planning – how big an outbreak to plan for? – Exercises – realistic scenarios – Outbreaks – geographic spread, size, should we vaccinate?, forward planning etc.• Resource models (nice to have) - Advice to policy on delivery constraints - Informing decisions on ramping up (and down) - Monitoring of actual vs predicted resources• Integrated epidemiology & resource (ideal) – Resource constraints (and time taken) informs the epi modelling – otherwise can be unrealistic• Integrated epidemiology & economic (ideal)
Lessons/What Next ?• Modelling has an important role to play in policy development.• Flexibility and willingness to engage in understanding the wider policy issues by “modellers” really helpful.• Policy customer should be clear at the outset what they want and what they want it for.• Can we put a value on modelling? • In an outbreak how much does modelling save? • what are the benefits of modelling?
Acknowledgments• Francesca Gauntlett – epidemiologist and Defra‟s Intelligent Customer for disease modelling• Simon Scanlon – Defra‟s lead animal health economist