Competency Workshop by SMA

1. Competency Workshop
Medbiquitous
May 2011
Susan Albright

2. Workshop Objectives
By the end of this workshop, participants will
be able to:
 Describe a process for developing
competency frameworks
 Identify challenges and potential barriers
to developing and implementing
competency frameworks
 Identify opportunities available with the
implementation of competency
frameworks in an electronic system

3. Agenda
 Introductions
 Part 1 – Background on the Tufts
Experience
 Activity
 Part 2 Assessment
 Discussion
 Break
 Part 3 – Developing the system
 Discussion
 Summary

4. Why Competencies?
 “Competency-based education is
characterized by the development of clearly
defined and explicitly published
competencies, mapping the curriculum to
achieve competencies, and assessment
process matched to competencies.”
(Ron Harden, Medical Teacher 2002 #24 Vol 2

5. What is Competency Based
Education
 CBE is an institutional process that moves education from
focusing on what academics believe graduates need to know
(teacher-focused) to what students need to know and be able
to do
 CBE is focused on outcomes (competencies) that are linked to
workforce needs as defined by the profession.
 CBE necessitates complex assessment through portfolios or
experiential learning assessement in field experience
 Large skill sets are broken down into competencies which
sequestial levels of mastery

6. CBE
 Competencies may reinforce one
another from basic to advanced as
learning progresses
 Continual refinement of competencies is
essential as this a process not a product
 Consistency in expectations across the
curriculum
 Consistency in grading
http://www.ceph.org/pdf/Competencies_TA.pdf
Council on Education for Public Health

7. Competency Frameworks
 ACGME
 Scottish Doctors
 Tomorrow’s Doctors
 The Tuning Project
 CanMeds
 MSOP AAMC

8. ACGME Competency Domains
 Patient care
 Medical knowledge
 Practice-based learning and
improvement
 Interpersonal and communication skills
 Professionalism
 Systems-based practice

9. AAMC Learning objectives
 The first of the four attributes is that physicians must be altruistic.
There are seven learning objectives, including the objective that before
graduation, the student can demonstrate compassionate treatment of
patients and respect for their privacy and dignity.
 The second attribute is that physicians must be knowledgeable; one of
the six learning objectives is that the student can demonstrate
knowledge of the normal structure and function of the body and of
each of its major organ systems.
 The third attribute is that physicians must be skillful; one of the eleven
learning objectives is that the student have knowledge about relieving
pain and ameliorating the suffering of patients.
 The last attribute is that physicians must be dutiful; one of the six
learning objectives is that the student have knowledge of the
epidemiology of common maladies within a defined population, and the
systematic approaches useful in reducing the incidence and prevalence
of those maladies.

10. Planning Process at the Tufts
Cummings Veterinary School
One year + planning process
involving:
•Educational Dean
•Chair of Rotation Directors
•All Rotation Directors
•Support Workshops provided by
the Tufts Center for Teaching and
Learning

11. Part 1
The Tufts Cummings Veterinary
School Experience
 Start with Competency Statements of
accrediting body – AVMA
 Localize to School Wide graduating
competencies for clinical rotations
 Workshops on CBE and competency
writing
 Build rotation/clerkship Competencies
 Produce Grading Rubrics

12. Vet School Experience
 Match each rotation back to the school wide
competencies
 All competencies come back to curriculum
committee for approval
 Informatics had not been identified by any
rotation – curriculum committee added these
competencies
 Curriculum committee added the preclinical
competencies

13. Helpful Tips
 Provide a good model
 Provide support – workshops
 Do not insist on uniformity
 Make sure grading rubrics match back
to competencies
 Identify it as a work in progress
 Hard part was getting the attention of
the rotation directors

14. Helpful Tips
 One person brings it all together
 Work with external bodies – curriculum
committee to provide overview

15. How to write Competencies
 Understand the level (school-wide,
course, content)
 Action Words
 Verbs followed by nouns
 Measureable and/or observable
 Performance based

16. How to write….
 Consider relationships up and down the
framework
 Do not use evaluative words in the
competency (this will come later)
 Who owns the competency? Who can
create? Who can modify?
 Feel free to use terms like ‘outcomes’,
learning objectives

17. Part 2
 Assessment and grading rubrics

18. How rubrics help:
 For Courses:
 Rubrics enable faculty to clearly communicate expectations for student performance to
students. They support more consistent and objective assessment of student work.
They also facilitate detailed feedback to students. When areas of strength and
weakness in performance are noted, students more easily comprehend the reasoning
behind their professor’s assessment of their work.
 A rubric is a scoring guide that seeks to evaluate a student's performance based on
the sum of a full range of criteria rather than a single numerical score.
 For Student Outcomes Assessment:
 Rubrics facilitate quality conversations among faculty about student learning, shorten
the time it takes to do SOA, and contribute to the validity and reliability of the
assessment process.
- Stevens and Levi, Ch. 2
www.uni.edu/chfasoa/IntroductiontoRubrics.ppt

19. Developing a Rubric cont.
 Scale
How well or poorly has the student done on the
task?
 Sophisticated, competent, partly competent, not yet competent.
 Exemplary, proficient, marginal, unacceptable.
 Advanced, Intermediate, Novice.
 Distinguished, proficient, intermediate, novice.
 Accomplished, developing, beginning. (Huba and Freed, 2000)
Note: Consider whether a scale needs more than three levels. Some
research indicates that information about student learning obtained
from a three-level scale is comparable to that obtained from a five-
level scale. More levels typically means more time spent on
assessment.

20. Developing a Rubric cont.
 Dimensions
 Dimensions break down a task into components
and identify the importance of these
components.
 Dimensions are descriptive, not evaluative (e.g.,
“organization” not “good organization”).
 Dimensions help students see that the work
they are doing is multidimensional and draws on
multiple abilities.

21. Developing a Rubric cont.
 Description of the Dimensions
 Rubrics should contain at least a description of the
highest level of performance.
 With experience and in response to the complexity of the
performance, descriptions of all levels can be written.
 Students need not fit cleanly into a single category. On
oral presentation skills, a student might speak in a clear
voice but lack eye contact.
 Descriptions for each level help students see that the
work they are doing does have varying levels of
achievement. There is a difference between poor and
excellent work.

22. Examples of Grading Rubrics
 handout

23. Competency Assessment-
TUSK mock-ups

24. Planning Process
After the competencies and
grading rubrics TUSK had a series
of meetings with :
•Educational Dean
•Chair of Rotation Directors
•All Rotation Directors
•Iterative process of
moving current process to
Online version integrated
with TUSK
• Create Powerpoint ”story
board” which developer
used as functional spec

25. Process to get to competency
assessment in TUSK
 TUSK meetings with vet school leaders
 Understand business processes
 Translate these to the online world
 Develop powerpoint story board to achieve
agreement of functional spec
 Iterate over 7 months
 Check in with all rotation directors
 Test and train
 Pilot test in two clerkships
 Release to all clerkships

26. View the Process through Every
Lens
 Course Administrators
 Create form
 Link to competencies
 Link Faculty to students
 Faculty Assessors
 Able to select/deselect students
 See picture of student
 Easy access to form
 Course Director
 See each assessor’s view as well as summary
 Able to add comments
 Able to override scores
 Student
 View the results
 Compilation of all clerkship results
 Registrar
 Access final grade (allow resubmit)

27. Competency Assessment
Features
 Prepare TUSK to accept competencies
 Link to School-wide/course competencies
 Configurable form for each course
 Multiple assessors
 Access to Registrar
 Show scores or only feedback
 Multiple rubrics per form
 Override score/grade
 Student retains access to scores over time

28. Overview
I. Admin View
a) Populate Competency Data
b) Populate Competency
Assessment Form & Data
c) Other Assessment Form Data
II. Faculty View
a) Access to the Assessment
b) Selecting Additional Students
c) Assessment Form
III. Director View
a) Assessment Results for All
Students
b) Completion Report
III. Director View (con’t)
c) Individual Student Results
d) Completed Faculty/Staff
Form
e) Summary Form
IV. Student View
a) Accessing Assessment
Results
V. Registrar View
IV. View final grades

29. I. Administrator View
Creating and Populating
Competency
Assessment Data

30. POPULATE COMPETENCY
DATA
Done at the School Administrator level

31. Step 1: Create Competency Category

32. Modify Competency Category

33. Add Sub-Competency Levels (if applicable)

34. Sort Competencies

35. Sort Child Competencies

36. POPULATE COMPETENCY
ASSESSMENT FORM DATA
Done at the Course Admin level

37. Step 2: Create Assessment Form(s)
Note: The “View Reports” button is only visible to the course director
“Balance Weights” still needs to be further discussed/clarified

38. Populate Assessment Form Metadata
Grading is numeric. There
will be an option in the
grade book to give an
equivalent letter or pass/fail
grade
Scoring Range defaults to
‘No’
Score Display:
• Score & Feedback
• Feedback Only
Scoring Value:
• Max value = 25
Frequency:
• Max No. = 10

39. Step 3: Populate Fields and Data
 Question type:
 Scaling
 Scaling w/Sub-question
 Single Select
 Single Select w/Sub-question
 Multi-level questions (options with sub-
questions)
 Link competencies to questions
 Free-text feedback option

40. Add Field: Scaling
Make text boxes bigger

41. Linking Competencies

42. Scaling w/ Sub-Questions

43. Add Field: Single Select

44. Single Select w/ Sub-level Questions

45. OTHER COMPETENCY
ASSESSMENT FORM DATA
Done at the Course Admin level

46. Considerations
 Display student images?
 Display elective information?
 Display summary comments
 Multiple assessors?
 Who selects assessors?

47. Considerations?
 Use Numeric scores?
 Display scores to students or just
comments?
 Display scores to assessors?
 360 degress review?
 What else?

48. Link Faculty/Student

49. II. Faculty View
Getting to the form

50. Accessing the Assessment(s)

51. Accessing the Form
Evaluator: J. Berg

52. Assessment Form – Scaling

53. Assessment Form – Single Select

54. III. Director View
Viewing assessment
results

55. Assessment Results – All Students

56. Assessment Results – Individual Student
Note: To view a form completed by a
faculty/staff click their name
Note: When ready, click here to fill out a final
assessment form for the student

57. Completed Form by Faculty/Staff

58. Summary Form
Hover over score to
see actual individual
scores
This is at the bottom of the form
Click to view student’s patent logs
in a pop-up window

59. Student’s Patient Log Summary
This opens up as a pop-up window

60. Completion Report

61. Send Email Reminders

62. IV. Student View
Access via the Grade
Book

63. Accessing Assessment Form
Note: Link to view final assessment form
completed by the course director

64. Accessing Assessment Form
J. Berg happens to also be the course director

65. V. Registrar View
View Final Grades

66. Grades by Course
Step 1: Select the course

67. Grades by Course
Step 2: Select the time period

68. Part 3
 Building the Framework

69. First steps
 A set of clear definitions
 Use cases
 A survey of Canadian, US and UK health
professional education organizations
(24 respondents)
 See definitions_and_usecases.pdf for
details (medbiquitous.org)

70. Comparison of published
frameworks
 Accreditation Council on Graduate Medical Education (ACGME)
Competencies8
 Acute Care Nurse Practitioner Competencies
 American Association of Critical-Care Nurses (AACN) General Patient
Care Competencies
 American Society of Health-System Pharmacists
 CanMEDS 20059
 CanMEDS Specialty Competencies
 Good Medical Practice (UK)
 Good Medical Practice (USA)
 The Scottish Doctor7
 Tomorrow's Doctors
 Tuning (MEDINE)
 Women’s Healthcare Competencies

71. DEVELOPMENT OF A
CONCEPTUAL MODEL

72. Outcomes for
Clinical Skills
A competency object . . .
Relates to other
competency objects . . .
Which can relate to
external resources . . . Learning
Object
Assess-
ment
Performance
data

76. Outcomes for
Clinical Skills
A competency object . . .
Relates to other
competency objects . . .
Which can relate to
external resources . . . Learning
Object
Assess-
ment
Performance
data
Current Specifications

77. Specs and Schemas Available
At:
 http://www.medbiq.org/working_group
s/competencies/index.html
 Open license, membership not required

78. Competencies –
individual
competency
definition
A standard format for a
single competency
definition, including an
identifier, categories
and references.
Competency Object Working draft
Being
implemented by
Tufts
May be
implemented by
VA
Essential for tying
curricula to competencies
and enabling competency
based education.
Competencies –
an interrelated
set of
competencies
A framework for
establishing the
relationships among
competencies in a
single framework
Competency
Framework
Working draft
Being
implemented by
Tufts
May be
implemented by
VA
Essential for integrating
existing competency sets
into curriculum
management systems.
Competencies –
evidence of
achievement
A format for
documenting
achievement of a
milestone or
competency.
Educational
Achievement
Work to begin in
July 2010
This will be important for
schools tracking
competence in a portfolio
and for residency
programs tracking
competence.
Competencies –
cross-mapping
A format for mapping
competencies from one
framework to
competencies from
another framework.
Informal proposal
from competency
working group
(planned work)
Should we arrive at the
point of having a national
competency framework,
this would be important
for schools that want to
see how local
competencies fit into the
national picture.

79. Spec needs Analysis of the Problem
from all perspectives
 Student View
 How am I doing?
 What do I have to do?
 How can I show my learning
 Faculty view
 How is student doing
 Grade/Rate/Comment on student
 View proof of learning
 See full range of expected learning outcomes
 Registrar
 Track grades
 Admin View
 What are the competencies
 Where are competencies taught - content/courses/themes/rotations/clinics
 Where are the gaps
Need tools to Enter, Store, view, Grade

80. Medbiquitous Competency Object
Standard
Copyright Medbiquitous consortium 2010 All rights reserved May 20 2010 p.11

81. What Info Needs to be
brought Together
 School wide competencies
 Course level competencies/learning
objectives
 Content addressing competencies and
learning objectives
 Element of time – when and how much
 How are competencies assessed

82. Where does TUSK get the
Data
 Schedule
 Content linked to schedule (upon
upload)
 Curriculum hours pulled from schedule
 Keywords linked to content

83. School Wide
Competencies:
Competency
Framework
Course
competencies
Course
competencies
Content
repository
w/
metadata
Content
competencie
s
Competency
Assessment
skills knowledge attitudes
Where are
they taught ?
How are they taught?
Where are they
assessed?
External
Competency
Framework
lecture
s
PBL
rotation
s
When?
(Links to
schedule)
Competencies

84. Schedule – Curriculum View

85. First year courses

86. Course Summary view

90. Methods to Display Complex Data
complex data displays

91. References
 Medical Teacher: Outcomes-Based Education Themed Issue Vol 29
Number 7 September 2007
 Lessons Learned from the Competency-Based Curriculum Initiative at
Baylor College of Dentistry. McCann, Ann L.; Babler, William J.; Cohen,
Peter A., Journal of Dental Education, v62 n2 p197-207 Feb 1998
 The new formal competency based curriculum at Indiana University
School of Medicine, Litzleman and Cottingham, Academic Medicine Vol
82, No. 4 April 2007
 An educational blueprint for the Brown Medical School, Smith MD
 Learning objectives for medical student education--guidelines for
medical schools: report I of the Medical School Objectives Project.
AAMC Acad Med. 1999 Jan;74(1):13-8.