1. Keynote address
Legal education research and evidence-
based policy-making: ‘The nightmare and
the noble dream’
Professor Julian Webb
Canadian Association of Law Teachers
Annual Conference
Ottawa
2 June 2015

2. Structure
• Background – the LETR in England and
Wales
• EBPM: ‘the nightmare and the noble dream’
• How we sought to ‘do’ EBPM in an evidence-
lite environment
• Implications of our experience

3. Context: lessons from the LETR
• Driver: Legal
Services Act 2007
(E&W) –
– ‘independent’
regulation and
market liberalisation
agenda
• Push from oversight
regulator (LSB)…

5. David Edmonds – Lord Upjohn Lecture, November
2010
Announced regulators’
review in context of
concerns about quality of
LET
Legal education and
training is central to
encouraging ‘an
independent, strong, diverse
and effective legal
profession’.
“[Workforce development] is
about achieving a constant
interplay between practice and
education, with the two spheres
in constant dialogue, each
driving improvement and
innovation in the other to the
broader public good.”

6. The LETR process
• Stage 1: research-led; ‘cradle to grave’
– Phase 1: A literature review, including an element of
comparative work relating to other jurisdictions and
professions;
– Phase 2: ‘Contextual analysis’ - identifying the knowledge,
skills and attributes currently required in the legal services
sector;
– Phase 3: ‘Workforce development’ - identifying potential
future structural changes and their implications for future
education and training needs
– Phase 4: Report and recommendations
• Stage 2: development and implementation by
the regulators (BSB/IPS/BSB) - ongoing

7. The nightmare and the noble dream…

8. Evidence-based policy making
“What counts is what works”
Powell (1999: 23)

9. EBPM process (1): systematic review (Banks,
2010)
Evidence based
Policy
Transparency
Time
Methodology
Receptive
policy
environment

10. The nightmare? Legal education policy as a
problem case for EBPM
• Lack of clear problem definition
• Paucity of evidence
• Political economy of legal education -
complex, contested terrain
• ….W(h)ither EBPM?

11. Problem definition
• Emphasis on ‘big’ intangibles like
quality, fitness for purpose, competence
• Complex assumptions of causality (and
risks of PBEM!)
• Eg…

12. For example: assuring ‘competence’
Tearing up
assumptions:
in the real
world...
• Complexity of
competence – soft
skills, values,
systems, etc
• Level of
qualification may be
only weakly
correlated to
competence
• Relationship
between cost and
quality of service is
NOT linear
Initial and
continuing
competence
• Inputs vs
outcomes and
outputs
• Passive vs
active
approaches
• Balance
between initial
and continuing
• What would real
assurance look
like?
What about
‘Competence+’
• Common standards +
a step beyond
minimum regulation
• Re-accreditation?
• Required PQ
specialisation ?

13. Quality and range of existing evidence
• We needed to construct the educational past/present
in order to understand the future
• Legal education’s growing engagement with
educational praxis, but
• Virtually no bibliographies
• No meta-reviews
• Very few systematic reviews
• And the quality of empirical research remains highly
variable, with large gaps:
• Mostly small scale ‘one shot’ qualitative studies
• Little proper (especially longitudinal) evaluation
• Very little reliable statistical data
• Little sense of historical emergence and continuity in the
research

14. Cochrane Collaboration

15. It’s the political economy, stupid!
• Law schools operating between the state, the
university and the profession
• Reviews not driven by pedagogic issues: political
engagements triggered by state or profession in
response to perceived ‘problems’ - see further Webb
(2016)
• Persistent demarcation disputes and seemingly
irresolvable issues – have shaped successive
reviews (eg ‘common training’)
• Tendency to create overlapping discursive and policy
spaces between liberal and professional education
and blur distinctions between them (cp LETR
emphasis on “LSET’)

16. EBPM AND SOCIAL
COMPLEXITY
EBPM: beyond the technical-rational paradigm?

17. EBPM assumptions (based on Banks, 2010)
Methodology
Necessary research skills
and capacity
Resources to harvest and
generate data
Note usually underlying cost-
benefit model
Time
Good quality data is
available
What about timeliness?
Transparency
Institutional independence
(of advisors)
Right incentives (to deliver
advice in public interest)
Structures and resources for
public debate
Open data sets

18. ‘New paradigm’: EBPM under conditions of
social complexity
Lessons from
complexity science,
work on bounded
rationality, ‘wicked
problems’, systems
theory, and modern
regulatory theory….
(eg, Majone, 1989; Reed &
Harvey, 1992; Sanderson,
2002, 2006; Baldwin & Black,
2008; Head, 2008, Wegner,
2009)
• Law of unintended
consequences and the limits of
regulatory steering
• Dampening effects on policy
reform of internal system
design/complexity
• Impact of complexity of social
and governance networks
• One size does not fit all:
possibly greater effectiveness of
small-scale, participatory and
localised interventions
• Beyond rational choice: values,
ethical-moral choices, and
desired ends matter

19. LSET as a ‘socially complex’ problem
Characteristics of socially
complex problems
Corresponding features of LSET (eg)
There is no definitive definition
of the problem
Some agreement over a need for reform, but
widespread disagreement over the extent,
priorities and nature of the changes required
They tend to be intractable General lack of effect from a number of recent
education and training reviews
Specific intractable problems:
• Achieving consistency of standards
• Reducing costs of training
• Managing increasing numbers
The information needed to make
sense of the problem is often ill-
defined, changing and may be
difficult to put into use
Currently operating in rapidly changing work
and educational environments
Relative lack of robust, especially longitudinal,
data
Costs of deriving meaningful information are
relatively high

20. From Webb et al (2013), Table 1.1
Characteristics of socially
complex problems
Corresponding features of LSET (eg)
They emerge in fields where
there are multiple stakeholders;
limited consensus as to who
the legitimate stakeholders
and/or problem-solvers are,
and stakeholders are likely to
have different criteria of
success
Large number of stakeholders, with different
understandings of the problem(s), and different
levels of engagement with the process
Legitimacy questions exist, eg, over the extent
of professional and regulatory interest in the
Bachelor of Laws (LLB)
Evidence of different stakeholders having
different ‘objectives’ for the review
Every attempt at a solution
matters significantly
Reform tends to be a ‘one-shot’ operation so
relatively high risk
Exacerbated by uncertainties about the new
regulatory environment, and the tendency of
LSET system to operate as a relatively low trust
environment

21. Problem-solving under conditions of social
complexity:
• Socially complex problems are more likely to require
solutions that:
– recognise that there are few right/wrong solutions as opposed to
better/worse outcomes;
– build shared understanding of the problem amongst a range of
stakeholders;
– build a shared commitment to action;
– recognise that ‘one-shot’ reforms will have unintended as well as intended
consequences;
– recognise that capacity for continuing engagement, and institutional
(re)design needs to be built-in to regulatory solutions.
• If solutions cannot be judged on right/wrong criteria
then ‘social robustness’ / goodness of fit are viable
grounds

22. LETR methodology
• Triangulation – existing literature/data and LETR
original research
 See Table 1.2 LETR Report
 Informed by both educational and regulatory theory
• Transparency
 Independent: regulators and stakeholders also research
subjects
 Open research process (so far as consistent with research
ethics)
• ‘Problem-based approach’
 Iterative research method
 ‘Socially robust’ solutions

23. Problem-based approach (from LETR Report)
• LETR adopted an “an iterative approach that uses the methods
of thematic inquiry to ground a process of collective learning and
collaborative problem-solving. This three-stage process builds
up a picture of the problem, including potential solutions to the
problem, then identifies and addresses critical information gaps,
before developing the actual solution(s) to the problem
collaboratively with stakeholders.” (para 1.18)
• “An iterative approach was adopted which involved returning to
issues as they became better defined. Because much of the
data was to be based on perceptions and experiences it was
necessary to triangulate information from different data sources
so as to increase assurances as to the consistency and
reliability of the findings.” (Appendix D, para 1.2)

24. Qualitative first and foremost
“Evaluation and assessment of LSET required evidence based
upon the experience and judgement of stakeholders and
participants in the provision and consumption of legal education
and legal services. While some of these data were captured
through questionnaire-based attitudinal surveys, qualitative
methods were designed to discover meaning through fine attention
to content, so interviews and focus groups allowed for a wider
range of responses, with richer description and deeper analysis of
the phenomena than would have been achieved by quantitative
research.” (Appendix D, para 1.3)

25. Evidence base: www.letr.org.uk
• Thematic literature review (nearly 300 pages)
• Qualitative research (focus groups/interviews; 306
respondents)
• Quantitative surveys – ca. 1250 respondents
• Re-run of 1991 ‘solicitors and their skills’ use of time
survey
• Secondary (quantitative) analysis of Legal Services
Benchmarking Survey (consumer experience
dataset)
• Stakeholder engagement (written responses to
discussion papers, Symposium outputs, input and
advice from ‘CSP’)

26. Consequences of approach
• Focus on ‘LSET’ system – primarily
structures and processes, less emphasis on
content
• Concern that regulation could act as a barrier
to innovation and collaboration
• Need to create a ‘shared space’ for
collaboration to happen

27. Main recommendations
• Outcomes-led system
– ‘Day one’ outcomes defined by reference to the knowledge, skills and attributes of a
competent practitioner
– Increase consistency in assessment standards
– Greater co-ordination and collaboration between regulators in setting standards
– Redefinition of content will follow; Report highlights concerns in respect of professional ethics,
communication skills, management skills, commercial awareness
• Structures
– Focusing CPD on learning, not just “hours” – move to more outputs-driven model
– Supervised practice (such as the training contract) no longer determined by time-served
– Flexibility – more opportunities to work and study in different configurations
– Access and diversity: Supporting apprenticeships, set standards for internships and work
placements, standards/qualification scheme for paralegals
– Voluntary regulation of paralegals outside regulated organisations
• Better information
– On diversity
– On careers options, opportunities and risks
– Greater coordination across system of information resources, research and development, and
evaluation

28. What sort of regulatory system did we
envisage?
A version of OFR, risk-based, with the
regulator as primus inter pares, part of the
shared space, listening, encouraging,
learning from other disciplines and
professions, from empirical experiments
and theory construction; and shaping the
landscape of regulation to encourage
innovation
(Paul Maharg in Ching et al, Upjohn Lecture, 2014)

29. Early responses…
• A missed opportunity
• Long-winded(!)
• Business as usual
• Better than expected
• A decent compromise
• Sensible recommendations
• An important milestone
• Interesting times ahead

30. Reasons to be cheerful?
Hits?
• Outcomes relatively
conservative
• No strong evidence that the
system was not fit for purpose
• Progress:
– on outcomes
– Significant changes on CPD
– Access and diversity
• Apprenticeships
• Legal exec law firms
• Voluntary regulation of
unregulated paralegals
Misses?
• Outcomes relatively
conservative
• Over-dependence on ‘opinion’
evidence?
• Progress:
– How common a framework?
– Costs of training
– Pace of change
– Rejection of coordinating
‘Legal Education Council’

31. Implications

32. Some lessons to be learned
• POLICY-MAKING IS
EXPERIMENTATION!
• Legal education reform – the
never-ending story?
• Time and timeliness are
critical
• The need for (moderately)
big data – systemic failure so
far
• Need for more sophisticated
research/experimental
design
• History repeats: importance
of regulatory ‘mode’ and
responsiveness

33. EBPM process (2): policy cycle: will it
happen?
Systematic
review
Incorporate into
policy decisions
Oversee
implementation
Monitor
outcomes
Targeted
evaluation

34. Implementation is everything
“ So whilst on the surface everyone (kind of) is saying they are (sort of)
happy, no one is smiling. And no one is smiling because the research
tells them there is not enough research; the research - and a moment's
thought - tells them there is no easy solution; there are many hard
yards; and it is everyone's problem, not yours, not mine, not theirs. It's
ours.
The research stands as an astute reminder that this is the reality:
implementation is everything. This will require resources (knowledge,
skills, bodies, money) and it will need mechanisms for change.
For that reason (I surmise) they propose that the different
constituencies coalesce around things like working out outcomes: what
does a lawyer need to be able to do at day one. And they propose a
Legal Education Council to advise regulators and act as focal point for
sharing best practice. This kind of negotiated solution will require a
great deal of energy and goodwill. I wonder if anyone is really up for it.
A more likely scenario is the regulators who want change will liberalise,
permit change and (if they do not skimp) properly assess it. That would
require both a fleetness of foot and a robustness about testing the links
between competence, education and training which we have yet to see
attempted in the legal sector.”
Prof. Richard Moorhead, Legal Week blog, 27 June 2013

35. Developing a community of practice...
“A resounding sentiment was that
there needs to be a permanent
conversation about the future of the
profession – from broad and
creative imaginings of future
business structures, to assessing
emerging technologies, to
pedagogical reform in the training of
lawyers – to encourage lawyers to
innovate within their own practice
setting. More often than not,
participants affirmed that there
should be space within the CBA for
these issues to be discussed on a
more regular basis.” (Canadian Bar
Association, 2014:24)
...the fundamental tension between education of
lawyers as delivering public value and education
of lawyers as delivering private value is
structural. The tension may manifest itself in
different ways under different conditions, but it
will always be with us and must always be
managed. Other matters likely to continually give
rise to stresses, challenges, and the need for
managing change are: the economics of law
schools; the rapid evolution in the market for
legal services; the function and value of
accreditation standards; the financing of legal
education; the role of parties other than law
schools in legal education; and the role of media
in understanding legal education and
communicating with the public.
Since these forces and factors will always be
with us, it is prudent for the system of legal
education to institutionalize the process of
dealing with them. All parties involved in legal
education should support a framework for the
continual assessment of strengths and
weaknesses and of conditions affecting legal
education, and for fostering continual
improvement. The process should ensure that
not only law schools, but also practicing lawyers,
judges, and other interested actors have a voice
and an opportunity for meaningful contribution.”
(ABA Task Force, 2014:29)

36. Research capacity?
…there is also reason to question whether the thirty-year growth trajectory
of socio legal scholarship can be sustained. The return of legal
fundamentalism, epitomized by recent aggressive moves by the
Federation of Law Societies of Canada to control law school curriculums
and pedagogy, is likely to force them to divert their resources away from
research towards professional training and to shift the focus of their faculty
members towards publishing books and articles more explicitly designed
for consumption by the practicing bar. The declining enthusiasm of
governments for law reform, and social reform more generally, is also likely
to trim socio-legal research budgets and to deprive socio-legal scholarship
of the legitimacy it derived from its contributions to the reform of the legal
system and of society.
• Harry Arthurs, The State of Canadian Socio-legal Research,
CALT blog, http://www.acpd-calt.org/?page_id=2357

37. References
• Baldwin, R. & Black, J. (2008) ‘Really Responsive Regulation’ 71 Modern Law Review 59-
94
• Head, B. (2008) ‘Wicked Problems in Public Policy’ 3(2) Public Policy 101-118
• Majone, G. (1989) Evidence, Argument and Persuasion in the Policy Process (Yale UP)
• Powell, M (ed) (1999) New Labour, New Welfare State? The Third Way in British Social
Policy (Policy Press)
• Reed, M. & Harvey, D. (1992) ‘The New Science and the Old: Complexity and Realism in
the Social Sciences’ 22(4) Journal for the Theory of Social Behaviour 353-380
• Sanderson, I. (2002) ‘Evaluation, Learning and Evidence-Based Policy Making’ 80(1) Public
Administration 1-22
• Sanderson, I. (2006) ‘Complexity, “Practical Rationality” and Evidence-based Policy Making’
34(1) Policy and Politics 115-32
• Webb, J. (2016) ‘A Tale of Two Cities: Reflecting on Lord Neuberger’s “Reforming Legal
Education”’ in Duncan, Ashford & Guth (eds), Perspectives on Legal Education (Routledge,
2016)
• Webb, J, Ching, J, Maharg, P, & Sherr, A (2013) Setting Standards: The Future of Legal
Services Education and Training in England and Wales (LETR Report) www.letr.org.uk
• Wegner, J, (2009) ‘Reframing Legal Education’s “Wicked Problems”’ 61(4) Rutgers Law
Review 867

38. Sources on LETR
• Sherr, A and Thomson, S. (2013) ‘Tesco Law and Tesco Lawyers: Will
Our Needs Change If the Market Develops?’ Oñati Socio-Legal Series,
Vol. 3, No. 3, 2013. Available at SSRN:
http://ssrn.com/abstract=2293660.
• Webb, J. (2013) ‘Regulating Lawyers in a Liberalized Legal Services
Market: The Role of Education and Training’ 24(2) Stanford Law &
Policy Review 533-70.
• Rt Hon Lord Neuberger of Abbotsbury, (2013) ‘Lord Upjohn Lecture
2012: Reforming Legal Education’ 47 Law Teacher 4-17
• Special Issue on LETR: Law Teacher, vol. 48(1) (2014).
• H. Sommerlad et al (eds) (2015),The Futures of Legal Education and
the Legal Profession (Oxford: Hart)
• J. Ching, P. Maharg, A. Sherr and J. Webb, (2015) ‘Lord Upjohn
Lecture 2014 - An Overture for Well-Tempered Regulators: Four
Variations on a LETR Theme’ Law Teacher, vol. 49(1), forthcoming
(eprint available online)