Talk delivered on 25 June 2021 at joint-workshop University of Tilburg and Vanderbilt University

1. Adopting the Gold Standard:
A/B Testing and Randomised
Controlled Trials for Evidence-
based Judicial Policy-making
Arthur Dyevre
Tilburg-Vanderbilt
25 June 2021

2. Courts make policy when:
I Deciding over rules of procedure

3. Courts make policy when:
I Deciding over rules of procedure
I Internal organisation

4. Courts make policy when:
I Deciding over rules of procedure
I Internal organisation
I Spelling out doctrines

5. Courts make policy when:
I Deciding over rules of procedure
I Internal organisation
I Spelling out doctrines
I Administrative and constitutional review

6. Courts make policy when:
I Deciding over rules of procedure
I Internal organisation
I Spelling out doctrines
I Administrative and constitutional review
I Judicial discretion

7. 1 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
3 Adopting the Gold Standard

8. Evidentiary basis for evaluation of judicial policy interventions:
I Intuition

9. Evidentiary basis for evaluation of judicial policy interventions:
I Intuition
I Expert opinion

10. Evidentiary basis for evaluation of judicial policy interventions:
I Intuition
I Expert opinion
I Observational studies

11. Evidentiary basis for evaluation of judicial policy interventions:
I Intuition
I Expert opinion
I Observational studies
I Randomised experiments?

12. 2 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
5 Adopting the Gold Standard

17. 2 A/B Testing

18. 3 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
11 Adopting the Gold Standard

19. I Internal validity

20. I Internal validity
I External validity

21. 3 Vignette. Group 1
Suppose you are presiding over a human rights case. Adam K. was arrested in possession of
explosives as part of a police crackdown on an international terrorist network. Police and intelligence
experts anticipated an imminent attack. In the hope of foiling the terrorist plot, police officers proceeded
to interrogate Adam K., trying to bring him to reveal the names and whereabouts of his accomplices.
Extensive interrogation sessions, only interrupted by short pauses, were held. As a result, Adam K. was
effectively deprived of sleep for 72 hours, time during which he was fed a frugal meal (equivalent of
1200 calories in nutritional value) only every 12 hours. For reasons unknown to police authorities, the
attack did not take place. Yet, according to the police, the interrogation led to the arrest of another
terrorist suspect who was found in possession of large quantities of triacetone triperoxyde, a primary
explosive commonly used in improvised explosive devices. Adam K.’s lawyer argues that police treatment
amounted to torture under the European Convention of Human Rights. The police, however, argue
that the interrogation procedure was not disproportionate and solely motivated by the compelling desire
to protect national security. Proportionality requires that a governmental act be suitable, necessary and
reasonable considering the competing interests at hand. Does the manner in which the police interrogated
Adam K. amount to torture?

22. 3 Vignette: Group 2
Suppose you are presiding over a human rights case. Adam K. was arrested in possession of
explosives as part of a police crackdown on an international terrorist network. Police and intelligence
experts anticipated an imminent attack. In the hope of foiling the terrorist plot, police officers proceeded
to interrogate Adam K., trying to bring him to reveal the names and whereabouts of his accomplices.
Extensive interrogation sessions, only interrupted by short pauses, were held. As a result, Adam K. was
effectively deprived of sleep for 72 hours, time during which he was fed a frugal meal (equivalent of
1200 calories in nutritional value) only every 12 hours. For reasons unknown to police authorities, the
attack did not take place. Yet, according to the police, the interrogation led to the arrest of another
terrorist suspect who was found in possession of large quantities of triacetone triperoxyde, a primary
explosive commonly used in improvised explosive devices. Adam K.’s lawyer argues that police treatment
amounted to torture under the European Convention of Human Rights. The police, however, argue that
the interrogation procedure was not disproportionate and solely motivated by the compelling desire to
protect national security. Article 3 ECHR states: “No one shall be subjected to torture or to inhuman or
degrading treatment or punishment.” Does the manner in which the police interrogated Adam K. amount
to torture?

23. 3 Please rank factors with strong influence on judicial
decision making

24. 3 Reported ideology

25. 3 Treatment Effect

26. 3 Randomisation

27. 3 Causal Inference in Observational Studies

28. 3 In the field

29. 4 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
21 Adopting the Gold Standard

30. 4 Field Randomized Controlled Trials in Law
I First field RCT: Gaudet et al. (1933)

31. 4 Field Randomized Controlled Trials in Law
I First field RCT: Gaudet et al. (1933)
I According to Greiner & Matthews (2016): about 50 field RCTs in
law since 1933

32. 4 Field Randomized Controlled Trials in Law
I First field RCT: Gaudet et al. (1933)
I According to Greiner & Matthews (2016): about 50 field RCTs in
law since 1933
I Resistance to RCTs in legal community

33. 4 Field Randomized Controlled Trials in Law
I First field RCT: Gaudet et al. (1933)
I According to Greiner & Matthews (2016): about 50 field RCTs in
law since 1933
I Resistance to RCTs in legal community
I Hypothesised reasons for differential with medicine:
advocacy-centred character of legal practice

34. 4 Obstacles to RCTs in Law
I "It’s impossible"

35. 4 Obstacles to RCTs in Law
I "It’s impossible"
I "The stakes are too high"

36. 4 Obstacles to RCTs in Law
I "It’s impossible"
I "The stakes are too high"
I "We already know it"

37. 4 Obstacles to RCTs in Law
I "It’s impossible"
I "The stakes are too high"
I "We already know it"
I "It’s unethical"

38. 4 Obstacles to RCTs and A/B Testing in Law

39. 5 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
25 Adopting the Gold Standard

42. I Rules of procedures and judicial resource management

43. I Rules of procedures and judicial resource management
I Online and alternative dispute resolution

44. I Rules of procedures and judicial resource management
I Online and alternative dispute resolution
I Decision support systems

45. I Rules of procedures and judicial resource management
I Online and alternative dispute resolution
I Decision support systems
I Discretionary remedies

46. I Rules of procedures and judicial resource management
I Online and alternative dispute resolution
I Decision support systems
I Discretionary remedies
I Doctrines?

47. 6 Outline
1 Evidence-based Judicial Decision Making
2 The Gold Standard
3 What Makes Field RCTs the Gold Standard?
4 Field RCTs and A/B Testing in Law
5 Potential Scope
6 Conclusion
29 Adopting the Gold Standard

48. 6 Conclusion: Striking Gold?
I Field RCTs and A/B testing rare in law

49. 6 Conclusion: Striking Gold?
I Field RCTs and A/B testing rare in law
I Ethical and cultural challenges

50. 6 Conclusion: Striking Gold?
I Field RCTs and A/B testing rare in law
I Ethical and cultural challenges
I But clearly room for more

51. 6 Conclusion: Striking Gold?
I Field RCTs and A/B testing rare in law
I Ethical and cultural challenges
I But clearly room for more
I Thanks!