This document discusses evidence-based policy and some of the challenges involved. It notes that while randomized controlled trials (RCTs) are considered the gold standard for evidence, politics is also influenced by other factors like public opinion and consensus. Politicians must make decisions quickly without full understanding and evidence can be against their values. The document also discusses how evidence is interpreted through ideological lenses and that context is important for determining what works. It advocates for an experimental approach to governance through limited testing of policies rather than trying to fully implement evidence.
5. A brief survey of the field
• Technocrats: will save the world (somehow)
• Political theorists: technocrats do not understand what
politics is (power! values! interests! disagreements!)
• Politicians: well yeah.. umm.. kinda good idea but…
• Civil servants: We have a lot to do already
• Economists (“randomistas”): let’s do RCTs and all is fine
• Philosophers of science: Umm.. The metaphysics of
causality...
• Scientists in general: we don’t have time to
communicate our results and twitter is so hard
6. They all get something right!
• Politics will get better with the use of evidence
• Politics is mostly about interests and values
• Politicians do their job fast and with limited
information
• Civil servants are busy
• RCTs are superb evidence (if interpreted right)
• Talk about causality is important
• It’s hard to communicate the results of studies to
political actors
7. Utopia
Evidence-based policy: Scientists produce quality
evidence → Politicians use it wisely as a basis for public
policy
Reality?
Policy-based evidence: Scientists produce quality
evidence → Politicians cherry pick results to support
their political beliefs
So we get something like this
8. How policymakers think
• Politics isn’t science: evidence is only one thing that affects political
decisions
– several other principles of good policymaking: consensus, public
opinion
• Cognitive shortcuts: rational (prioritizing certain kinds of information) and
irrational (gut feelings, deeply held beliefs, feelings)
• Have to do decisions fastly; have to try to evaluate things without deep
understanding
• Evidence can be against values → values win
• Evidence is always interpreted through one’s own ideological lens
• Policy process is complex: there is not only one point where to inject
evidence
9. What counts as good evidence?
Hierarchy of evidence
- Comes from evidence-based
medicine: idea is that RCTs
are best to produce causal
claims
- But not that simple: RCTs are
good but most studies in
social sciences observational
studies which are also good
10. Getting evidence: why do we
need experiments (RCTs)?
- World is becoming harder place to understand
without experimenting
- Social problems = Wicked problems
- Civil servants might not know how reality works
- (Even comparative) evidence on policies usually
context dependent
→ it’s good to try them in our own context
12. A methodological word of
caution
- RCTs are a bit overhyped but still really good
- The results from RCTs are context dependent
- contextual differences will shape intervention outcomes
- It’s a big discussion about what we can infer from them RCTs: need to think
about the mechanisms behind results (not a problem)
- Other sources of evidence are also needed because they are black box -
explanations (mechanisms)
- RCTs good at internal validity and generating causal claims, but generalizing is
not easy when we talk about complex issues (as opposed to examples from
medicine)
13. “[In education] ‘what works?’, which is what politicians would love to know about, is the not
the right question, because [in education] everything works somewhere and nothing works
everywhere. The interesting question is ‘under what conditions does this work?’“
14. The nature of evidence
• Evidence is evidence of something for someone: pluralism
• Evidence is uncertain and context-dependent
• There are no common standards for evaluating evidence - what counts as
good evidence? And for whom?
• Evidence-based policy need (mostly) quantative causal knowledge;
unfortunately it’s hard to infer (universal) causal mechanisms even from the
best studies
• Aaann.. Evidence-producing science is also value-based (to some extent) →
plurality of views within disciplines
• If scientists cannot decide what counts as good evidence, how can
politicians?
15. Beyond methods: different disciplines offer
conflicting evidence
- Most policies are still designed based on assumptions
about rational action (economics)
- e.g. taxing, monetary incentives in social policy
- Incentives are good to keep in mind but we are also quite
irrational
- What if we would look empirically how people act and
design policies based on that?
- Problem: many disciplines with good ideas contradicting
with each other → how to do policy recommendations?
16. My (very preliminary) thoughts
• Evidence-based policy is more ex tempore testing of policies (experimental
governance) than trying to get meta-analyses into political processes
– Like this: There is emerging evidence on the effects of policy x (might work, or not)
→ Let’s try with RCT & other methods in our context to see if it works here
– Limited testing is better than not testing (finnish basic income experiment)
– Fostering the culture of experimentation is thus important
– Evidence should be fitted to politicians’ mindsets: workshops, scenario building,
policy briefs
Editor's Notes
A primer on exp. And behav. Sciences in the next era: methodsolgical point of view. Theres a lot of what could be said about these issues but i will skip many important things.
Not going to talk about the process (design for goverment)
Eivät välttämättä mistään muusta kuin valikoitumisesta, eli siitä että esimerkiksi korkeammin koulutetut ja paremmin työelämässä pärjäävät vanhemmat laittavat lapsensa varhaisemmin päivähoitoon.
oinen huolestuttava piirre tutkimuksessa on, että kun varhaisemman päivähoidon ja myöhemmän koulumenestyksen selittävää mallia kontrolloidaan taustamuuttujilla, ei yhteys olekaan enää merkitsevä
EBP touches the basic questions about the relationship between science and politics. It’s good to acknowledge that politics isn’t science. One cannot do politics in the same rational sense what works in science.
→ (Supposing that we have good evidence) Even best evidence does not translate into policy
There are several principles of ‘good’ policymaking and only one is EBPM. Others relate to the value of pragmatism and consensus building, combining science advice with public values, improving policy delivery by generating ‘ownership’ of policy among key stakeholders, and sharing responsibility with elected national and local policymakers.
Willing to consider only a limited range of solutions
Elected politicians are involved primarily in managing government and maximising public and organisational support for policies.
A key argument in policy studies is that it is impossible to separate facts and values when making policy. We often treat our beliefs as facts, or describe certain facts as objective, but perhaps only to simplify our lives or support a political strategy (a ‘self-evident’ fact is very handy for an argument). People make empirical claims infused with their values and often fail to realise just how their values or assumptions underpin their claims.
the use of facts in policy is to underpin evaluations based on values
RCTs because we can learn better when we use science!!
Koska se mahdollistaa toimenpiteen toimivuuden testauksen ennen sen laajaa käyttöönottoa
Koska loppukäyttäjät päättävät, onko palvelu tai tuote hyvä vai ei
Koska kokeileminen nopeuttaa kehittämistä
Koska kokeileminen auttaa näyttämään yhteistyökumppaneille ja sijoittajille, että käsissä on muutakin kuin idea
RCTs are considered as the golden standard because it is based on powerful heuristic: What if -contrafactuals. What if everything else is held constant (point of randomization) and we do intervention to the treatment group, happens? If the intervention seems to work (average treatment effect is the causal parameter) we can infer that intervention has causal power on the individuals.
What would have happened to individual i if they had not been exposed to treatment X=1
John Stuart mills’ method of difference
Random assignment to treatment ensures that units assigned to the treatment and units assigned to the control are identical
What works in one place might not work in other places
In terms of the first: RCTs are the appropriate method for some policy questions - notably ones about effectiveness of interventions - but more specifically intereventions which have measurable, controlable, and direct (non-complex) cause-effect relationships involved.
This approach tends to work well for short term or immediate cause-effect relationships. But is much more difficult to use for complex change processes (e.g. social change or political change, or when we think about addressing the upstream social determinants of health, for example).
Policy decisions are about multiple competing outcomes and considering a range of social values.. Experimental trials cannot tell a decision maker what is 'good' or 'bad' from a value position, only whether an intervention is more or less effective at achieving speific outcomes (it does not tell us whether that outcome is a good idea compared to everything else we might spend our time and money doing). Therefore, trials should be seen as only one type of evidence that may be relevant to the decision process.
The problem is, that human biochemistry and human anatomy are much more universal than most human social and political contexts. Because external validity is so common in medical trials, the difference between internal and external validity has often been lost when the 'medical model' of 'evidence based medicine' has been applied to social policy.
RCTs = black box explanations - how do they work if they work?
→ Need to answer “how” questions
→ we need something else than RCTs too. Why people act like they act?
Qualitative studies, normal cross sectional studies
NPP ei pidä ajatella vain ison mittakaavan asiana. Näyttöä voi tehdä myös politiikan puolella. Bit mahdollisuus? Yhtäältä pieniä kokeiluja esimerkiksi kuntatasolla. Toisaalta meillä on suuria kompleksisia ongelmia kuten ilmastonmuutos. Kompleksisiin ongelmiin ei ole selkeitä (näyttöön perustuvia) ratkaisuja, jotka kerralla toimisivat.