Your SlideShare is downloading. ×
0
Blueprinting and Choosing                                            Appropriate Tools for Assessment                     ...
Conflict of InterestNone                       2
ObjectivesBy the end of the workshop, participants will be able to:• Define blueprinting in the context of specific    ass...
Agenda•   Assessment and blueprinting•   Exercise #1: Designing a blueprint•   Group Discussion•   Leg stretch•   Assessme...
Microsoft clip-art, 2012                           5
Professional Competence     Developmental       Impermanent     Context dependent                          6
Dimensions of Professional Competence                                                    Cognitive                        ...
The CanMEDS Seven C’s                        Royal College of Physicians and Surgeons of Canada                           ...
Charting the Seven C’s                                                       Continuum                         Curriculum ...
Purposes of Assessment•   Certify the competence of future practitioners     •   Minimal level of competence•   Discrimina...
Purposes of Assessment                  Learning                                • To discover the worth                   ...
Assessment                                             Does  Professional                                                 ...
“If you don’t know where you’regoing, you probably ain’t going to get                there”                      Microsoft...
BlueprintingDefining the “what” should be representedon your assessment• For licensure or certification   •   Practice ana...
Blueprint“Specifies all the elements of performance relevant tothe assessment so that appropriate samples of activityand c...
Domains to be tested     Items tested                       16
BlueprintValid assessment: interpretation of theresults reflect what was meant to be tested• Appropriate blueprint    •   ...
How do I blueprint for my assessment?1. Define the purpose of your assessment   •   Summative or Formative?   •   To decid...
Example of curricular content•   Community acquired pneumonia•   Hospital acquired pneumonia•   Otitis media•   Urinary tr...
How do I blueprint for my assessment?3.   Provide relative weighting of the content                         Importance X f...
How do I blueprint for my assessment?4. Sample opinion on weighting   •   Involve course chairs, teachers, evaluation     ...
Example of weighting  Content presentation     Importance Frequency   IXF   Weight  CAP                      3          3 ...
How do I blueprint for my assessment?5. Decide total number of items on your   assessment   •   Based on your weighting, y...
How do I blueprint for my assessment?6. Decide how you would like to break it   down?• Tasks   •   Diagnosis/Investigation...
How do I blueprint for my assessment?Content                     No. of items   Diagnosis   Investigation   TreatmentCAP  ...
How do I blueprint for my assessment?Final blueprint often looks like a matrix• Patient group X Task• Clinical presentatio...
Exercise #1• Using worksheet #1 start working on a  blueprint for your purpose• Get together with colleagues and choose  o...
Microsoft clip-art, 2012                           28
Assessment Tool BoxesWhat will I use to assess my blueprint?                                          29
How do I assess what I want to?Blueprint• Establishes what you wish to testAssessment tool box• Helps to determine how you...
Assessing the 7 C’s                      31
Assessing the CompetenciesProfessional Competence• More than a demonstration of isolated   competence Eraut, 1994• “…when ...
Assessing the Competencies• Context specific• What may be considered “competent” in  one context may not be in a different...
Assessment tools for CompetenciesRCPSC suggest:•   In-training evaluations (ITERs)•   MCQs•   SAQs•   Essays•   Oral exami...
CanMEDS Evaluations and PGEMed expert    •   ITER>Oral>MCQ>SAQCommunicator    •   ITER>Oral>OSCE>simOther roles    •   ITE...
CanMEDS Evaluation and PGEPG directors levels of satisfaction with CanMEDsevaluations• Medical Expert:     •   satisfied t...
ACGME Competencies•   Patient care•   Medical knowledge•   Practice-based learning and improvement•   Inter-personal and c...
Assessment tools for ACGME CompetenciesACGME tool box:    Record review       Chart stim. Recall    Checklist           Gl...
Assessment Challenges when Charting the 7 C’s•   Socially negotiated educational competencies•   Accreditation agencies ma...
Potential solutions?• Linking abstract competencies to  observable behaviors    •   Key competencies/Enabling competencies...
Word of caution“…a tendency to describe generalcompetencies in exhaustive detail, leadingto bulky, fragmented documents th...
Building your own assessment toolbox• Define the purpose of your assessment• Define the population you are assessing• From...
Exercise #2• Create your own toolbox using  worksheet #2• Discuss your toolbox with a colleague• Group discussion   •   Ch...
Microsoft clip-art, 2012                           44
In Summary• Define what you wish to assess by  designing a blueprint• Decide how you will assess• Understand the challenge...
Thank You!       Questions?Please give me feedback at    ctouchie@mcc.ca                             46
Useful references1. Epstein RM and Hundert EM. Defining and assessing   professional competence. JAMA 2002;287:226-2352. C...
Upcoming SlideShare
Loading in...5
×

Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s

915

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
915
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
57
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s"

  1. 1. Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s Claire Touchie, MD, FRCPC Associate Professor, University of Ottawa Chief Medical Education Advisor Medical Council of CanadaProperty of the Medical Council of Canada
  2. 2. Conflict of InterestNone 2
  3. 3. ObjectivesBy the end of the workshop, participants will be able to:• Define blueprinting in the context of specific assessment needs• Design an appropriate blueprint for the purpose of a specific assessment• Describe available tools for student performance assessments• Construct an assessment tool box to achieve assessment goals 3
  4. 4. Agenda• Assessment and blueprinting• Exercise #1: Designing a blueprint• Group Discussion• Leg stretch• Assessment tool boxes• Exercise #2: Assembling a tool box• Group Discussion• Wrap-up and Evaluations 4
  5. 5. Microsoft clip-art, 2012 5
  6. 6. Professional Competence Developmental Impermanent Context dependent 6
  7. 7. Dimensions of Professional Competence Cognitive Habits Technical Of Mind Professional Competence Affective Integrative Moral Contextual Relationship Epstein and Hundert, JAMA 2002; 287:226-235 7
  8. 8. The CanMEDS Seven C’s Royal College of Physicians and Surgeons of Canada 8
  9. 9. Charting the Seven C’s Continuum Curriculum Comprehensive Seven Clinical C’s Competency- Based Collaborative Case-based Epstein and Hundert, JAMA 2002; 287:226-235 9
  10. 10. Purposes of Assessment• Certify the competence of future practitioners • Minimal level of competence• Discriminate among candidates for advanced training • Rank competitively• Provide motivation and direction for learning • Provide formative feedback• Judge the adequacy of a training program • Evaluate teacher or courses Epstein and Hundert, JAMA 2002; Crossley et al, Medical Education, 2002 10
  11. 11. Purposes of Assessment Learning • To discover the worth • To improve the quality Feedback Assessment 11
  12. 12. Assessment Does Professional Behavior Authenticity Shows How Knows How Cognition Knows Modification of Miller from Crossley et al. Medical Education, 2002 12
  13. 13. “If you don’t know where you’regoing, you probably ain’t going to get there” Microsoft clip-art, 2012 13
  14. 14. BlueprintingDefining the “what” should be representedon your assessment• For licensure or certification • Practice analysis• For a course or rotation • Objectives 14
  15. 15. Blueprint“Specifies all the elements of performance relevant tothe assessment so that appropriate samples of activityand corresponding methods can be selected according totheir relative importance to the overall assessmentprocess.” Newble & Dawson, 1994“When the items of a test are judged to adequatelyrepresent well-defined domains ofcontent…generalizable samples…” Cronbach, 1971 15
  16. 16. Domains to be tested Items tested 16
  17. 17. BlueprintValid assessment: interpretation of theresults reflect what was meant to be tested• Appropriate blueprint • Defines the content of the assessment • Provides validity evidence 17
  18. 18. How do I blueprint for my assessment?1. Define the purpose of your assessment • Summative or Formative? • To decide on competency or to rank? • To provide feedback to the student or to the teacher?2. Tabulate curricular content • Topics vs clinical presentation vs objectives 18
  19. 19. Example of curricular content• Community acquired pneumonia• Hospital acquired pneumonia• Otitis media• Urinary tract infections• Meningitis• Endocarditis• Osteomyelitis/septic arthritis• Septicemia• Cellulitis/skin infections• HIV 19
  20. 20. How do I blueprint for my assessment?3. Provide relative weighting of the content Importance X frequency • Importance/impact 1. Non-urgent, little prevention potential 2. Serious, but not immediately life threatening 3. Life threatening emergency and/or high potential for prevention impact • Frequency 1. Rarely seen 2. Relatively common 3. Very common 20
  21. 21. How do I blueprint for my assessment?4. Sample opinion on weighting • Involve course chairs, teachers, evaluation coordinators, previous learners • Consensus method • Readjust weighting if necessary 21
  22. 22. Example of weighting Content presentation Importance Frequency IXF Weight CAP 3 3 9 0.22 HAP 3 1 3 0.073 Otitis media 1 3 3 0.073 UTI 2 3 6 0.15 Meningitis 3 1 3 0.073 Endocartitis 3 1 3 0.073 Osteo/Septic arthritis 2 1 2 0.049 Septicemia 3 2 6 0.15 Cellulitis/Skin infn 2 2 4 0.098 HIV 2 1 2 0.049 Total 41 22
  23. 23. How do I blueprint for my assessment?5. Decide total number of items on your assessment • Based on your weighting, you can decide how many items per content area 23
  24. 24. How do I blueprint for my assessment?6. Decide how you would like to break it down?• Tasks • Diagnosis/Investigation/Treatment/Counseling• Context of care • ED/Inpatient/Outpatient• Competencies 24
  25. 25. How do I blueprint for my assessment?Content No. of items Diagnosis Investigation TreatmentCAP 11 4 4 3HAP 3.65 1-2 0 1-2Otitis media 3.65 1-2 0 1-2UTI 7.5 2 3 2Meningitis 3.65 1 1 1-2Endocarditis 3.65 1 1-2 1Osteo/Septic arthritis 2.45 1 1 0Septicemia 7.5 2-3 1-2 3Cellulitis/Skin infection 4.9 2-3 0 2HIV 2.45 1 1-2 0 25
  26. 26. How do I blueprint for my assessment?Final blueprint often looks like a matrix• Patient group X Task• Clinical presentation X Context of Care• Clinical Activities X Competencies 26
  27. 27. Exercise #1• Using worksheet #1 start working on a blueprint for your purpose• Get together with colleagues and choose one blueprint to work on• Large group discussion 27
  28. 28. Microsoft clip-art, 2012 28
  29. 29. Assessment Tool BoxesWhat will I use to assess my blueprint? 29
  30. 30. How do I assess what I want to?Blueprint• Establishes what you wish to testAssessment tool box• Helps to determine how you will test 30
  31. 31. Assessing the 7 C’s 31
  32. 32. Assessing the CompetenciesProfessional Competence• More than a demonstration of isolated competence Eraut, 1994• “…when we see the whole, we see its parts differently than when we see them in isolation” Polanyi, 1969 32
  33. 33. Assessing the Competencies• Context specific• What may be considered “competent” in one context may not be in a different context. 33
  34. 34. Assessment tools for CompetenciesRCPSC suggest:• In-training evaluations (ITERs)• MCQs• SAQs• Essays• Oral examinations• OSCEs• Simulations• Log books Microsoft clip-art, 2012 34
  35. 35. CanMEDS Evaluations and PGEMed expert • ITER>Oral>MCQ>SAQCommunicator • ITER>Oral>OSCE>simOther roles • ITER>Oral>OSCE Chou et al. Medical Education, 2008 35
  36. 36. CanMEDS Evaluation and PGEPG directors levels of satisfaction with CanMEDsevaluations• Medical Expert: • satisfied to very satisfied• Communicator, collaborator, scholar, and professional • Neutral to satisfied• Manager and health advocate • Dissatisfied to neutral Chou et al. Medical Education, 2008 36
  37. 37. ACGME Competencies• Patient care• Medical knowledge• Practice-based learning and improvement• Inter-personal and communication skills• Professionalism• Systems-based practice 37
  38. 38. Assessment tools for ACGME CompetenciesACGME tool box: Record review Chart stim. Recall Checklist Global rating SP OSCE Simulations 3600 global rating Portfolios MCQs Oral exam Log books Patient survey 38
  39. 39. Assessment Challenges when Charting the 7 C’s• Socially negotiated educational competencies• Accreditation agencies mandate that competencies be assessed• Tools don’t match single competencies independently except for medical knowledge/expert• Other competencies/roles • Reflect personal attributes?• What is desirable VS what is observable? Lurie, Medical Education, 2012 39
  40. 40. Potential solutions?• Linking abstract competencies to observable behaviors • Key competencies/Enabling competencies• Mapping competencies to observable behaviors • Milestones, Entrustable Professional Activities (EPAs) 40
  41. 41. Word of caution“…a tendency to describe generalcompetencies in exhaustive detail, leadingto bulky, fragmented documents that losepractical value for education as theybecome less and less connected with thereal world.” ten Cate et al. Medical Teacher 2010 41
  42. 42. Building your own assessment toolbox• Define the purpose of your assessment• Define the population you are assessing• From your blueprint, what is best assessed with what tool• Is it going to be reliable? Feasible? Acceptable?• Will the result interpretation be valid? 42
  43. 43. Exercise #2• Create your own toolbox using worksheet #2• Discuss your toolbox with a colleague• Group discussion • Challenges? 43
  44. 44. Microsoft clip-art, 2012 44
  45. 45. In Summary• Define what you wish to assess by designing a blueprint• Decide how you will assess• Understand the challenges and limitations to assessing the 7 C’s• Stay tuned to further developments 45
  46. 46. Thank You! Questions?Please give me feedback at ctouchie@mcc.ca 46
  47. 47. Useful references1. Epstein RM and Hundert EM. Defining and assessing professional competence. JAMA 2002;287:226-2352. Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical Education 2002;36:800-8043. Lurie SJ. History and practice of competency-based assessment. Medical Education 2012;46:49-574. Coderre S, Woloschuk W, McLaughlin K. Twelve tips to blueprinting. Medical Teacher 2009;31:322-324 47
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×