Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Maggie Cruickshank - Optional work place assessment of training

271 views

Published on

Dr Cruickshank

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Maggie Cruickshank - Optional work place assessment of training

  1. 1. Prof Maggie Cruickshank Professor MB ChB MD FRCOG Division of Medical and Dental Education and Medical Sciences
  2. 2. Work place assessments for training in colposcopy Professor Maggie Cruickshank University of Aberdeen
  3. 3. Objectives • Explain the principles of good assessment • Describe the workplace-based assessment tools • Consider useful feedback • Consider descriptors in workplace assessment tools • Reflect on the challenges related to assessment of performance
  4. 4. Principles of Assessment An effective assessment system should: • Provide feedback for trainees • Clarify action plans • Help assessors to measure trainee performance • Help assessors decide on competency
  5. 5. Features of Good Assessment • Reliability • Validity • Feasibility • Educational impact
  6. 6. What is the best method of assessing Colposcopy practice? …is it valid/reliable?
  7. 7. Reliability • Does the test consistently measure the same thing? • Expressed as a number between 0 and 1  1 = you are making it up  0 = useless
  8. 8. Validity • Does the test measure what it is supposed to measure? …not always easy to define or demonstrate!
  9. 9. Miller’s pyramid Knows Knows how Shows Does
  10. 10. Competence vs Performance • Competence-based assessment measures what doctors can do in controlled representations of professional practice • Performance-based assessment measures what doctors actually do in their professional practice
  11. 11. Knows Knows how Shows Does Written exams OSCE Competence Knowledge Performance Action valid reliable Content validity Concurrent validity Construct validity Predictive validity Test-retest reliability Inter-rater reliability Internal consistency OSCE: Objective Structured Clinical Examination
  12. 12. Knows Knows how Shows Does Competence Knowledge Performance Action
  13. 13. What type of question?
  14. 14. • Critical to learning and has a significant influence on achievement • Feedback is the largest influence on achievement in learning or training • Feedback alone is effective in 71% of education studies • Observation of a clinical performance enhances learning (Hattie 1999; Veloski 2006) Formative Assessment and Feedback
  15. 15. Definition of Workplace-based Assessment (WPA) “The assessment of working practices based on what doctors actually do in the workplace, and is predominantly carried out in the workplace itself”
  16. 16. Workplace-based Assessment (WPAs) Methods • Structured scoring systems to ensure systematic review and feedback on colposcopy performance • Mini-Clinical Evaluation Exercise (mini-CEX) • Directly Observed Procedural Skills (DOPS) • Case-based Discussion (CbD)
  17. 17. Mini-CEX • Focuses on formative assessment of clinical skills • Responds to the assessment problems of the traditional long case • Responds to the educational problems of the workplace
  18. 18. Mini-CEX • Assessor observes a trainee with a patient • Usually a very specific focused task – Take a history – Do a colposcopy examination – Explain results – Form a management plan – Not all components are assessed (i.e., mini) • Assessor scores performance on a structured rating scale • Trainee is given feedback • Takes 5-10 minutes
  19. 19. Complexity of Low Average High case:    Please grade the following areas using the scale below Below expectations Borderline Meets expectations Above expectations U/C* 1 History Taking 1 2   3  4  5 6    2 Examination Skills        3 Communication Skills        4 Clinical Judgement        5 Professionalism        6 Organisation/Efficiency        7 Overall clinical care        *U/C Please mark this if you have not observed the behaviour and therefore feel unable to comment Anything especially good? Suggestions for development Agreed action plan:
  20. 20. Why multiple MiniCEXes ? • Improve the reliability of any single assessment • Completes picture of performance • Must agree in advance of patient interaction and document • Must be relevant to colposcopy training
  21. 21. Direct Observation of Procedure • Assessor observes a trainee undertaking a procedure – Routine colposcopy examination or procedure – Assessor rates performance on a rating scale (check list) – Takes as long as procedure (minutes) – Trainee gets instant feedback – Multiple encounters
  22. 22. Direct Observation of Procedure • Assesses technical competency of a particular procedure • Assesses when trainee is ready to move on to independent practice for a procedure • Combined with other work place assessments to determine competency for independent practice
  23. 23. Case-based Discussion • Structured format for discussing a clinical case • Helps to record conversation about, and presentations of, cases by trainees • Designed to assess clinical decision making and the application or use of medical knowledge • Allows the trainee to discuss why they acted as they did
  24. 24. Strengths of WPA • Sample widely across the training curriculum • Multiple raters • Formative and summative – Used with more reliable assessments to inform high stakes decisions • Reinforce educational culture where feedback for learning is the norm • Identify struggling trainees early
  25. 25. Assessment in training Assessment1 Assessment2 Assessment3 Assessmentx Competency 1 • • Competency 2 • • Competency 3 • Competency x •
  26. 26. Limitations of WPA • Not reliable enough to stand alone • Low scores = “failure” • Early success reduces motivation • Weaker trainees least likely to seek feedback • Time • Availability of trainers
  27. 27. Effective feedback is the key to the success of these assessment methods
  28. 28. Questions ?

×