Professor Paul Maharg presented on assessment in legal education at a workshop of the Law Society of Ireland. The presentation included: 1. A taxonomy of task analysis for assessments in simulations, ranging from discrete tasks to assessments involving an entire case file and performative skills. 2. Results from a study on using simulated clients for interviewing assessments, which found it to be more reliable and valid than traditional assessment methods. 3. Ways that problem-based learning can be designed to ensure both breadth and depth of learning, and forms of assessment that align with problem-based learning methods like exams involving analysis of new or previously seen case studies.

1. Professor Paul Maharg
paulmaharg.com/slides
https://www.zotero.org/groups/4722732/lsi_assessment_workshop_27.6.22/library
Law Society of Ireland
Workshop on Assessment
27.6.22
Plenary 2
Zeugma
legal education::technology::
writing::professional education

2. preview of workshop
2
Times Session
1000-1040 1. Introductions (10 mins)
2. Plenary 1: Research, ideas, applications (RIA): Framing research & concepts (20)
3. Brief discussion (10)
1040-1115 Breakout groups: Group discussions (20), plenary discussion (15)
1115-1130 Break
1130-1200 Recap on breakout issues.
Plenary 2: Examples of the architecture of assessment
1200-1230 Breakout groups: Group discussions
1230-1300 Plenary discussion of ways forward & possible projects
Zeugma
legal education::technology::
writing::professional education

3. in this plenary – some framing research &
concepts
Zeugma
legal education::technology::
writing::professional education
ideas
application
research
Some key questions:
1. Is the research out there? Is it
relevant to my subject?
2. Are my ideas feasible re time,
finance, effort?
3. Is the application practical?
4. Will the application improve
professional learning?
5. Is it sustainable?
6. How will it fit programme
learning?
7. How do I think students will
remember my assessments?

4. 1. Task analysis
2. Assessment with Sim Clients
3. Assessment in professional PBL

5. 1. Task analysis of assessments in
simulations

6. task analysis of assessments in simulations
Here’s a taxonomy of task analysis, from simple to complex:
1. Discrete tasks (eg drafting, letter-writing, research)
(Estate Planning)
2. Whole file + performative skill (Personal Injury)
3. Discrete tasks + whole file (Conveyancing)
4. Discrete tasks + file + performative skill (Litigation)
Zeugma
legal education::technology::
writing::professional education

7. • Set context (or not: let student figure that out – the clearing in the
forest…)
• Set task (but in how much detail? Supported with templates?
Guidelines? Commented examples?)
• Design feedforward (but don’t do the task for students)
• Deadline a task (bearing all contextual factors in mind)
• Task completed (and sent to staff in role)
• Feedback on task (by staff in role)
• Debrief (either in role or out of role)
1. discrete tasks
Zeugma
legal education::technology::
writing::professional education

8. • Holistic assessment of document chain
• Bodies of evidence generally, but can embed critical points of
assessment, eg report to client, speech plan, etc
• Preparation for performative skill, including overlap with
other skills – eg relation of legal research to professional
negotiation.
2. whole file + performative skill
Zeugma
legal education::technology::
writing::professional education

9. • Specific tasks are the foreground, eg draft the completion
certificate…
• … but students must also complete entire file process. No
completion, no competence.
• Tasks may shadow tutorial work or precede tutorial work or
neither
• Quaere: How many attempts at each task?
3. tasks + whole file
Zeugma
legal education::technology::
writing::professional education

10. • Most complex, most authentic and most demanding
• Potentially 1-3 plus more – eg performative skill can
be assessed in role.
4. tasks + whole file + performative skill
Zeugma
legal education::technology::
writing::professional education

11. • Example: PI project:
– PI mentor: passes information in real time; takes all fictional roles
including PI senior partner (instructs, praises, warns), e-comm only:
student responses are assessed
– Surgery mentor: gives detailed feedforward on task, f2f, out of role:
responses not assessed
– Discussion forum: gives detailed feedforward on task, e-comm, out of
role: responses not assessed
– Practice Manager: gives coaching on firm experiences, in role: support
& coaching not assessed, but the result is…
use of interleaved learning support
& assessment
Zeugma
legal education::technology::
writing::professional education

12. 2. Assessment with SCs

13. aims of our Simulated Client
Initiative back in 2005
• Could we correlate interviewing assessments to the degree of client
satisfaction & confidence arising from them?
• ie answer the following questions…
1. Was our current system of teaching and assessing interviewing skills
sufficiently reliable and valid?
2. Could the Simulated Patient method be translated successfully from
medical education to the legal domain?
3. Was the method of Simulated Client training and assessment more
reliable, valid and cost-effective than the current system?
Zeugma
legal education::technology::
writing::professional education

14. results from Strathclyde University
pilot
Questions Results
1 Was our current system of teaching and assessing interviewing skills sufficiently
1. reliable?
2. valid?
1. No
2. No
2 Could the Simulated Patient method be translated successfully to the legal
domain? Yes
3 Was the method of Simulated Client training and assessment more
1. reliable,
2. valid
3. cost-effective
than the current system?
1. Yes
2. Yes
3. Yes
Zeugma
legal education::technology::
writing::professional education

15. what changed…?
• We made what the client thought important in the most
salient way for the student: an assessment where most of
the grade was given by the client
• We did not conclude that all aspects of client interviewing
could be assessed by SCs
– We focused the assessment on aspects we believed
could be accurately evaluated by non-lawyers
– We focused the assessment on initial interview (which
was been extended in various ways at other centres to
other contexts)
• This changed the way we enable students, trainees and
lawyers to learn interviewing & client-facing ethical
behaviour

16. current
status
of
SCI
University of Strathclyde Law School
(Glasgow, Scotland)
WS (Writers to the Signet) Society
(Edinburgh, Scotland)
University of New Hampshire Law School
(Concord, NH, USA)
The Australian National University College of Law
(Canberra, Australia)
Northumbria University Law School
(Newcastle, England)
Kwansei Gakuin University Law School
(Osaka, Japan)
Solicitors Regulation Authority -
Qualifying Lawyer Transfer Scheme (QLTS)
(London, England)
Law Society of Ireland -
Continuing Professional Development of Solicitors
(Dublin, Ireland)
Hong Kong University Faculty of Law
(Hong Kong)
National Centre for Skills in Social Care
(London, England)
The Chinese University of Hong Kong Faculty
of Law
(Hong Kong)
Flinders Law School
(Adelaide, South Australia)
Nottingham Trent University Law School,
(Nottingham, England)
Osgoode Hall Law School + OPD
(Toronto, ON, Canada)
Canadian Centre for Professional Legal
Education PREP programme for articling
students (AB, MB, SK + NS)
U of Windsor Faculty of Law
(Windsor, ON, Canada)

17. independent studies…
Gerkman, A., Harman, E., Bond, L., Sullivan, .M. (2015). Ahead of the
Curve: Turning Law Students into Lawyers. A Study of the Daniel
17 Professor Paul Maharg | CC BY-NC-ND 2.5 CANADA
• In focus groups, members of the
profession and alumni said they
believed that students who
graduated from the program were
a step ahead of new law school
graduates;
• When evaluated based on
standardized client interviews,
students in the program
outperformed lawyers who had
been admitted to practice within
the last two years; and
• The only significant predictor of
standardized client interview
performance was whether or not
the interviewer participated in the

18. SCs: people as co-producers, co-
designers
The SC approach challenges:
1. Curriculum methods
2. Ethics of the client encounter
3. The cognitive poverty of conventional law school assessment
4. Law school as a monolithic construct
5. Law school categories of employment
6. The curricular isolation of clinic within many (not all) law schools
7. Hollowed-out skills rhetoric
8. Conventional forms of legal education regulation by regulatory bodies
9. The role of regulator: to be an encourager of innovation & radical
reform…?
10.Disciplinary boundaries – what about a SC Unit that’s interdisciplinary?
11.Local jurisdictional practices: how might such a project work globally?

19. 3. Assessment in professional PBL
‘the key to expert problem-solving lies in how knowledge is organized, not the
quantitative knowledge acquired’ (Lung 2008)

20. PBL design
Zeugma
legal education::technology::
writing::professional education

21. breadth of learning: what we focus on
We:
• Ensured we had breadth of learning as prescribed by
regulatory and other codes (Priestley, AQF, CALD, etc)
• Designed the new context of subject clusters in order to
improve learning and assessment across the entire program
• Used PBL as a heuristic to link courses, eg private and public
law courses; pervasive ethics; linkages of substantive legal
rules with sociolegal research and policy issues arising from
the problem ‘trigger’ and student outcomes in first PBL
sessions and review sessions
Zeugma
legal education::technology::
writing::professional education

22. depth of learning: what we focus on
• deep learning through active learning. Students committed to
being engaged from day 1 – PBL obliges them to do this.
• Experiential process, including awareness of learning &
knowledge, colleagues’ learning, spiral learning, self-
management, ethics.
• We focused students on dealing with sophistication and
complexity and updating knowledge, as well as learning legal
principles, leading cases, statutory knowledge, problem
handling, etc.
Zeugma
legal education::technology::
writing::professional education

23. more depth vs more breadth, or both?
On knowledge acquisition, Schmidt et al (2009) noted what many others observed:
that PBL students better integrate their knowledge, which resulted in more accurate
reasoning; that in the clinical case recall (a measure of expertise) and processing
speed (a sign of better understanding) they were superior to the conventionally-
educated cohorts (227). In skills acquisition, PBL students demonstrated much better
interpersonal skills, and knowledge about skills (a variable closely related to skilled
performance – 236). Student and expert perceptions of the quality of PBL education
were higher than the results for the conventionally-educated cohorts, with students
commenting positively in particular on their practices in independent study and
critical thinking. In passing, Schmidt et al also noted that PBL schools graduate
students faster and in larger numbers and retain students better (237).
(Maharg 2015, 12-13)
Zeugma
legal education::technology::
writing::professional education

24. forms of PBL assessment
• For specialist skills, OSCEs are well-suited
• For knowledge in examinations:
– Use the PBL learning method, eg provide:
• A case worked on during the year, with different questions
• An unseen case and require students to analyse it using the PBL
method
• A student response to a seen case study, and ask students to
analyse its strengths and weaknesses
Zeugma
legal education::technology::
writing::professional education

25. forms of assessment in PBL
• OSCEs for skills
• Forms of exams:
– Align with the learning method. So:
• Use PBL cases seen in workshops, but with different questions
• Use new PBL cases and require learners to analyse it according to
PBL methods
• Present a case + a (fictional) student response, and ask them to
assess its strengths and weaknesses
Zeugma
legal education::technology::
writing::professional education