Workshop given to the Examiners of the College of Family Medicine in Pretoria. Odi District Hospital May 2014

1. OSCE workshop
Dr Ed Parry-Jones RCGP

2. Who am I ?
 edwardparry-jones@nhs.net

3. Overview of workshop
topics
 What is an OSCE
 Blueprinting
 Case writing
 Paperwork
 Mark scheme
 location
 Equipment
 Role players
 Examiners
 The big day
 Standard setting
 Case assurance / metrics and review
 Building and maintaining case bank


4. OSCE
Objective
Structured
Clinical
Examination


5. Objective - all candidates presented with
same test
Structured - the marking scheme for each
station is structured
Clinical Examination - test of skills,
behaviour, attitudes and application of
knowledge

6. Why use OSCEs in clinical
assessment?
• careful specification of content
• observation of wide sample of activities
• increased reliability
• interactions between examiner
and student are structured

7.  Validity – a qualitative issue, testing the RIGHT things in the RIGHT
way

 Reliability- Technical aspects of the test, consistency, accuracy,
ability to predict who is competent or incompetent . Same level
of test wherever or whenever it is taken



9. OSCE stations
3 categories:
Observed
Examiner present
Non-observed
interpretation of clinical data e.g.
video, CT scan - no examiner
Rest or Preparatory station


10. Types of OCSE stations
history taking/ counselling /explanations

11. Types of OCSE stations
Examination skills

12. Types of OCSE stations
Data interpretation

13. Resources
Simulated patients (trained)
Real patients
Volunteers (not trained)
Models
Video/ DVD of patients
Films/ scans
Equipment for tests/ procedures


14. Blueprinting
 Blueprint written and clinical assessments against each
other
 Content of the assessment should align with curriculum
learning objectives
 Use appropriate assessment methods to test
different aspects of curriculum
 Create a matrix - competencies along one axis, system
or specialism along the other
 Sample widely across the whole matrix


15. ●Learning outcome ● ●2.1 Evaluate a patient according to
the bio-psycho social approach
●2.2 Formulate and execute, in consultation
with the patient, a mutually acceptable, cost-
effective management plan
●2.3 Provide comprehensive, continuing care
throughout the life cycle incorporating
preventative, diagnostic, therapeutic,
palliative and rehabilitative interventions
●TOTAL NO OF
QUESTIONS
●Key skills to be examined ● ●
Interpretation of investigations (e.g. x-
rays, ECG, blood tests, etc.) x2
●
●
Diagnostic procedures (e.g. LP, lymph
node biopsy etc.)x2
●
Therapeutic skill or procedures (e.g. injection joint)
x2
●
●
Emergency management and skills (e.g.
resuscitation, primary survey, airway management)
x2
●
●
Anaesthetic skills (e.g. spinal anaesthetic, use of
local anaesthetic, GA) x1
●
●
Brief counseling (e.g. breaking bad news, behavior
change) x2
●
●
Question based on portfolio x1
●
●Weight ●Stations ●4 ●5 ●3 ●
●
Trauma & Orthopedics ●
2 ● ● ● ●
●HIV/AIDS, TB and Malaria ●2 ● ● ● ●
●Women’s and Child Health ●2 ● ● ● ●
●Surgery & Anaesthetics ●2 ● ● ● ●
●ENT, eye, skin, mental health ●2 ● ● ● ●
●General adult medicine ●2 ● ● ● ●
●TOTAL NO OF QUESTIONS ● ● ● ● ●

16. Case writing
n plan the construct that the station is testing
and write it out as a single sentence
e.g. to assess the ability of a candidate to
explain colonscopy clearly, accurately and
succinctly to a patient
e.g. to assess the competence of a candidate
in examining a patient’s thyroid gland
n refer to your construct while you write


17. Paperwork scheme
 Learning Outcome
 Station Title
 Skill tested
 Context and focus for the station
 What the author(s) feel the candidate has to do to pass the station


18. Marking Scheme
n Think about whether to use 2/1/0
2 = done well
1 = adequate
0 = poor/ not done
n …..or 1/0
1 = Performed competently
0 = poorly performed/ not done
OR other schemes
? Check lists and / or Global score


19. Location

20. Equipment

21. Role Players

22. Examiners

23. The Big Day

24. Standard setting
 Standards are based on judgments about examinees’ performances
against a social or educational construct
e.g. competent practitioner or student ready for graduation


25. The Standard Setting Problem
Test
Result
Pass
Fail
Competent Incompetent

26. Classification Scheme
 Relative methods
based on judgments about groups of test takers
 Absolute methods
based on judgments about test questions
based on judgments about the performance of
individual examinees
 Compromise methods
Livingston, S.A. & Zeiky, M.J. (1982) Passing scores: a
manual for setting standards of performance on
educational and occupational tests Educational Testing
Service, Princeton

27. Types of Standards

 Relative standards/norm referenced methods:
 based on a comparison among the performances
of examinees
 a set proportion of candidates fails regardless of
how well they perform e.g. the top 84% pass
Eg Borderline group, regression
 Absolute standards/criterion referenced methods:
 based on how much the examinees know
 candidates pass or fail depending on whether they
meet specified criteria e.g. examinees must
correctly answer 70% of the questions
 Eg Angoff

28. Norm-referenced standard
Test score distribution
30 %
50 %
80 %

29. Criterion referenced standard
50 %
Test score distribution (average group)
Test score distribution (good group)
Test score distribution (poor group)

30. Case assurance / metrics
 Measures of reliability not validity
 Cronbach Alpha co-efficient – Central to producing a reliable
examination is the idea that every question (or OSCE
station)should contribute in the same direction towards the final
mark or result.
 Thus the marks that candidates get on each item / question / case
should broadly correlate positively with each other and with the
total mark

 Standard error of measurement – the confidence interval around a
score


31. Case / question banks
Benefit of saving time writing cases /
questions
Allow ‘development’ of cases / questions
based on metrics
Useful for developing a regional / national
exam
Problems of security

32. Thanks and acknowledgements
 Sandy Mather, Tom Owen and John Howard at the RCGP
 Angela Hall and Richard Wakefield
