This document discusses blueprinting and choosing appropriate assessment tools to evaluate student performance. It defines blueprinting as specifying the elements of performance to be represented on an assessment based on their importance. The presentation reviews designing a blueprint by defining the assessment purpose, tabulating curricular content, weighting domains, and deciding the number of assessment items. It also discusses assessing competencies like the CanMEDS 7 roles and ACGME competencies using tools such as in-training evaluations, MCQs, OSCEs, and simulations. Challenges in assessing competencies and potential solutions like linking competencies to observable behaviors are reviewed. The importance of defining assessment purpose and matching it with reliable and valid tools from a created toolbox is emphasized.

1. Blueprinting and Choosing
Appropriate Tools for Assessment
of Student Performance:
A Start to the Charting of the C’s
Claire Touchie, MD, FRCPC
Associate Professor, University of Ottawa
Chief Medical Education Advisor
Medical Council of Canada
Property of the Medical Council of Canada

2. Conflict of Interest
None
2

3. Objectives
By the end of the workshop, participants will be able to:
• Define blueprinting in the context of specific
assessment needs
• Design an appropriate blueprint for the purpose of a
specific assessment
• Describe available tools for student performance
assessments
• Construct an assessment tool box to achieve
assessment goals
3

4. Agenda
• Assessment and blueprinting
• Exercise #1: Designing a blueprint
• Group Discussion
• Leg stretch
• Assessment tool boxes
• Exercise #2: Assembling a tool box
• Group Discussion
• Wrap-up and Evaluations
4

5. Microsoft clip-art, 2012
5

6. Professional Competence
Developmental
Impermanent
Context dependent
6

7. Dimensions of Professional Competence
Cognitive
Habits
Technical
Of Mind
Professional
Competence
Affective
Integrative
Moral
Contextual
Relationship
Epstein and Hundert, JAMA 2002; 287:226-235 7

8. The CanMEDS Seven C’s
Royal College of Physicians and Surgeons of Canada
8

9. Charting the Seven C’s
Continuum
Curriculum Comprehensive
Seven
Clinical
C’s Competency-
Based
Collaborative Case-based
Epstein and Hundert, JAMA 2002; 287:226-235 9

10. Purposes of Assessment
• Certify the competence of future practitioners
• Minimal level of competence
• Discriminate among candidates for advanced
training
• Rank competitively
• Provide motivation and direction for learning
• Provide formative feedback
• Judge the adequacy of a training program
• Evaluate teacher or courses
Epstein and Hundert, JAMA 2002; Crossley et al, Medical Education, 2002 10

11. Purposes of Assessment
Learning
• To discover the worth
• To improve the quality
Feedback Assessment
11

12. Assessment
Does
Professional Behavior
Authenticity
Shows How
Knows How Cognition
Knows
Modification of Miller from Crossley et al. Medical Education, 2002 12

13. “If you don’t know where you’re
going, you probably ain’t going to get
there”
Microsoft clip-art, 2012
13

14. Blueprinting
Defining the “what” should be represented
on your assessment
• For licensure or certification
• Practice analysis
• For a course or rotation
• Objectives
14

15. Blueprint
“Specifies all the elements of performance relevant to
the assessment so that appropriate samples of activity
and corresponding methods can be selected according to
their relative importance to the overall assessment
process.” Newble & Dawson, 1994
“When the items of a test are judged to adequately
represent well-defined domains of
content…generalizable samples…” Cronbach, 1971
15

16. Domains to be tested
Items tested
16

17. Blueprint
Valid assessment: interpretation of the
results reflect what was meant to be tested
• Appropriate blueprint
• Defines the content of the assessment
• Provides validity evidence
17

18. How do I blueprint for my assessment?
1. Define the purpose of your assessment
• Summative or Formative?
• To decide on competency or to rank?
• To provide feedback to the student or to the
teacher?
2. Tabulate curricular content
• Topics vs clinical presentation vs objectives
18

19. Example of curricular content
• Community acquired pneumonia
• Hospital acquired pneumonia
• Otitis media
• Urinary tract infections
• Meningitis
• Endocarditis
• Osteomyelitis/septic arthritis
• Septicemia
• Cellulitis/skin infections
• HIV
19

20. How do I blueprint for my assessment?
3. Provide relative weighting of the content
Importance X frequency
• Importance/impact
1. Non-urgent, little prevention potential
2. Serious, but not immediately life threatening
3. Life threatening emergency and/or high potential for
prevention impact
• Frequency
1. Rarely seen
2. Relatively common
3. Very common
20

21. How do I blueprint for my assessment?
4. Sample opinion on weighting
• Involve course chairs, teachers, evaluation
coordinators, previous learners
• Consensus method
• Readjust weighting if necessary
21

22. Example of weighting
Content presentation Importance Frequency IXF Weight
CAP 3 3 9 0.22
HAP 3 1 3 0.073
Otitis media 1 3 3 0.073
UTI 2 3 6 0.15
Meningitis 3 1 3 0.073
Endocartitis 3 1 3 0.073
Osteo/Septic arthritis 2 1 2 0.049
Septicemia 3 2 6 0.15
Cellulitis/Skin infn 2 2 4 0.098
HIV 2 1 2 0.049
Total 41
22

23. How do I blueprint for my assessment?
5. Decide total number of items on your
assessment
• Based on your weighting, you can decide
how many items per content area
23

24. How do I blueprint for my assessment?
6. Decide how you would like to break it
down?
• Tasks
• Diagnosis/Investigation/Treatment/Counseling
• Context of care
• ED/Inpatient/Outpatient
• Competencies
24

25. How do I blueprint for my assessment?
Content No. of items Diagnosis Investigation Treatment
CAP 11 4 4 3
HAP 3.65 1-2 0 1-2
Otitis media 3.65 1-2 0 1-2
UTI 7.5 2 3 2
Meningitis 3.65 1 1 1-2
Endocarditis 3.65 1 1-2 1
Osteo/Septic arthritis 2.45 1 1 0
Septicemia 7.5 2-3 1-2 3
Cellulitis/Skin infection 4.9 2-3 0 2
HIV 2.45 1 1-2 0
25

26. How do I blueprint for my assessment?
Final blueprint often looks like a matrix
• Patient group X Task
• Clinical presentation X Context of Care
• Clinical Activities X Competencies
26

27. Exercise #1
• Using worksheet #1 start working on a
blueprint for your purpose
• Get together with colleagues and choose
one blueprint to work on
• Large group discussion
27

28. Microsoft clip-art, 2012
28

29. Assessment Tool Boxes
What will I use to assess my blueprint?
29

30. How do I assess what I want to?
Blueprint
• Establishes what you wish to test
Assessment tool box
• Helps to determine how you will test
30

31. Assessing the 7 C’s
31

32. Assessing the Competencies
Professional Competence
• More than a demonstration of isolated
competence Eraut, 1994
• “…when we see the whole, we see its
parts differently than when we see them
in isolation” Polanyi, 1969
32

33. Assessing the Competencies
• Context specific
• What may be considered “competent” in
one context may not be in a different
context.
33

34. Assessment tools for Competencies
RCPSC suggest:
• In-training evaluations (ITERs)
• MCQs
• SAQs
• Essays
• Oral examinations
• OSCEs
• Simulations
• Log books
Microsoft clip-art, 2012
34

35. CanMEDS Evaluations and PGE
Med expert
• ITER>Oral>MCQ>SAQ
Communicator
• ITER>Oral>OSCE>sim
Other roles
• ITER>Oral>OSCE
Chou et al. Medical Education, 2008 35

36. CanMEDS Evaluation and PGE
PG directors levels of satisfaction with CanMEDs
evaluations
• Medical Expert:
• satisfied to very satisfied
• Communicator, collaborator, scholar, and
professional
• Neutral to satisfied
• Manager and health advocate
• Dissatisfied to neutral
Chou et al. Medical Education, 2008 36

37. ACGME Competencies
• Patient care
• Medical knowledge
• Practice-based learning and improvement
• Inter-personal and communication skills
• Professionalism
• Systems-based practice
37

38. Assessment tools for ACGME Competencies
ACGME tool box:
Record review Chart stim. Recall
Checklist Global rating
SP OSCE
Simulations 3600 global rating
Portfolios MCQs
Oral exam Log books
Patient survey
38

39. Assessment Challenges when Charting the 7 C’s
• Socially negotiated educational competencies
• Accreditation agencies mandate that competencies
be assessed
• Tools don’t match single competencies
independently except for medical knowledge/expert
• Other competencies/roles
• Reflect personal attributes?
• What is desirable VS what is observable?
Lurie, Medical Education, 2012 39

40. Potential solutions?
• Linking abstract competencies to
observable behaviors
• Key competencies/Enabling competencies
• Mapping competencies to observable
behaviors
• Milestones, Entrustable Professional Activities
(EPAs)
40

41. Word of caution
“…a tendency to describe general
competencies in exhaustive detail, leading
to bulky, fragmented documents that lose
practical value for education as they
become less and less connected with the
real world.” ten Cate et al. Medical Teacher 2010
41

42. Building your own assessment toolbox
• Define the purpose of your assessment
• Define the population you are assessing
• From your blueprint, what is best
assessed with what tool
• Is it going to be reliable? Feasible?
Acceptable?
• Will the result interpretation be valid?
42

43. Exercise #2
• Create your own toolbox using
worksheet #2
• Discuss your toolbox with a colleague
• Group discussion
• Challenges?
43

44. Microsoft clip-art, 2012
44

45. In Summary
• Define what you wish to assess by
designing a blueprint
• Decide how you will assess
• Understand the challenges and
limitations to assessing the 7 C’s
• Stay tuned to further developments
45

46. Thank You!
Questions?
Please give me feedback at
ctouchie@mcc.ca
46

47. Useful references
1. Epstein RM and Hundert EM. Defining and assessing
professional competence. JAMA 2002;287:226-235
2. Crossley J, Humphris G, Jolly B. Assessing health
professionals. Medical Education 2002;36:800-804
3. Lurie SJ. History and practice of competency-based
assessment. Medical Education 2012;46:49-57
4. Coderre S, Woloschuk W, McLaughlin K. Twelve tips to
blueprinting. Medical Teacher 2009;31:322-324
47