Osce presentation RSA May 2014

471 views

Published on

Workshop given to the Examiners of the College of Family Medicine in Pretoria. Odi District Hospital May 2014

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
471
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
37
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • How suitabler
    Not too disruptive to service provision
    Quirt enough
    Enough space
    Not likely to be disturbed
  • Is it available
    Standardisation
    All in correct position
  • Who are the role players
    Patients – pos – real , but neg – sick , ? Ethics , may have to stop the exam , ? Consistancy , cheap
    Actors, standard , not going to be sick , expensive
    Doctors, nurses or students - ? Consistancy
  • Who are your examiners ?
    Ow are examiners selected – how many years qualified, are there further assessmnets needed before selection
    Examiner training
    Maintainance of skills - ? Regualr training schedule
    Demographics of examiners – gender, race
  • Double check arrnagements
  • Osce presentation RSA May 2014

    1. 1. OSCE workshop Dr Ed Parry-Jones RCGP
    2. 2. Who am I ?  edwardparry-jones@nhs.net
    3. 3. Overview of workshop topics  What is an OSCE  Blueprinting  Case writing  Paperwork  Mark scheme  location  Equipment  Role players  Examiners  The big day  Standard setting  Case assurance / metrics and review  Building and maintaining case bank 
    4. 4. OSCE Objective Structured Clinical Examination 
    5. 5. Objective - all candidates presented with same test Structured - the marking scheme for each station is structured Clinical Examination - test of skills, behaviour, attitudes and application of knowledge
    6. 6. Why use OSCEs in clinical assessment? • careful specification of content • observation of wide sample of activities • increased reliability • interactions between examiner and student are structured
    7. 7.  Validity – a qualitative issue, testing the RIGHT things in the RIGHT way   Reliability- Technical aspects of the test, consistency, accuracy, ability to predict who is competent or incompetent . Same level of test wherever or whenever it is taken  
    8. 8. OSCE stations 3 categories: Observed Examiner present Non-observed interpretation of clinical data e.g. video, CT scan - no examiner Rest or Preparatory station 
    9. 9. Types of OCSE stations history taking/ counselling /explanations
    10. 10. Types of OCSE stations Examination skills
    11. 11. Types of OCSE stations Data interpretation
    12. 12. Resources Simulated patients (trained) Real patients Volunteers (not trained) Models Video/ DVD of patients Films/ scans Equipment for tests/ procedures 
    13. 13. Blueprinting  Blueprint written and clinical assessments against each other  Content of the assessment should align with curriculum learning objectives  Use appropriate assessment methods to test different aspects of curriculum  Create a matrix - competencies along one axis, system or specialism along the other  Sample widely across the whole matrix 
    14. 14. ●Learning outcome ● ●2.1 Evaluate a patient according to the bio-psycho social approach ●2.2 Formulate and execute, in consultation with the patient, a mutually acceptable, cost- effective management plan ●2.3 Provide comprehensive, continuing care throughout the life cycle incorporating preventative, diagnostic, therapeutic, palliative and rehabilitative interventions ●TOTAL NO OF QUESTIONS ●Key skills to be examined ● ● Interpretation of investigations (e.g. x- rays, ECG, blood tests, etc.) x2 ● ● Diagnostic procedures (e.g. LP, lymph node biopsy etc.)x2 ● Therapeutic skill or procedures (e.g. injection joint) x2 ● ● Emergency management and skills (e.g. resuscitation, primary survey, airway management) x2 ● ● Anaesthetic skills (e.g. spinal anaesthetic, use of local anaesthetic, GA) x1 ● ● Brief counseling (e.g. breaking bad news, behavior change) x2 ● ● Question based on portfolio x1 ● ●Weight ●Stations ●4 ●5 ●3 ● ● Trauma & Orthopedics ● 2 ● ● ● ● ●HIV/AIDS, TB and Malaria ●2 ● ● ● ● ●Women’s and Child Health ●2 ● ● ● ● ●Surgery & Anaesthetics ●2 ● ● ● ● ●ENT, eye, skin, mental health ●2 ● ● ● ● ●General adult medicine ●2 ● ● ● ● ●TOTAL NO OF QUESTIONS ● ● ● ● ●
    15. 15. Case writing n plan the construct that the station is testing and write it out as a single sentence e.g. to assess the ability of a candidate to explain colonscopy clearly, accurately and succinctly to a patient e.g. to assess the competence of a candidate in examining a patient’s thyroid gland n refer to your construct while you write 
    16. 16. Paperwork scheme  Learning Outcome  Station Title  Skill tested  Context and focus for the station  What the author(s) feel the candidate has to do to pass the station 
    17. 17. Marking Scheme n Think about whether to use 2/1/0 2 = done well 1 = adequate 0 = poor/ not done n …..or 1/0 1 = Performed competently 0 = poorly performed/ not done OR other schemes ? Check lists and / or Global score 
    18. 18. Location
    19. 19. Equipment
    20. 20. Role Players
    21. 21. Examiners
    22. 22. The Big Day
    23. 23. Standard setting  Standards are based on judgments about examinees’ performances against a social or educational construct e.g. competent practitioner or student ready for graduation 
    24. 24. The Standard Setting Problem Test Result Pass Fail Competent Incompetent
    25. 25. Classification Scheme  Relative methods based on judgments about groups of test takers  Absolute methods based on judgments about test questions based on judgments about the performance of individual examinees  Compromise methods Livingston, S.A. & Zeiky, M.J. (1982) Passing scores: a manual for setting standards of performance on educational and occupational tests Educational Testing Service, Princeton
    26. 26. Types of Standards   Relative standards/norm referenced methods:  based on a comparison among the performances of examinees  a set proportion of candidates fails regardless of how well they perform e.g. the top 84% pass Eg Borderline group, regression  Absolute standards/criterion referenced methods:  based on how much the examinees know  candidates pass or fail depending on whether they meet specified criteria e.g. examinees must correctly answer 70% of the questions  Eg Angoff
    27. 27. Norm-referenced standard Test score distribution 30 % 50 % 80 %
    28. 28. Criterion referenced standard 50 % Test score distribution (average group) Test score distribution (good group) Test score distribution (poor group)
    29. 29. Case assurance / metrics  Measures of reliability not validity  Cronbach Alpha co-efficient – Central to producing a reliable examination is the idea that every question (or OSCE station)should contribute in the same direction towards the final mark or result.  Thus the marks that candidates get on each item / question / case should broadly correlate positively with each other and with the total mark   Standard error of measurement – the confidence interval around a score 
    30. 30. Case / question banks Benefit of saving time writing cases / questions Allow ‘development’ of cases / questions based on metrics Useful for developing a regional / national exam Problems of security
    31. 31. Thanks and acknowledgements  Sandy Mather, Tom Owen and John Howard at the RCGP  Angela Hall and Richard Wakefield 

    ×