This document provides an overview of a webinar presentation on universal assessment challenges in healthcare related education. It discusses establishing context through both external drivers like accreditation standards, and internal drivers from various stakeholder perspectives. It then covers specific challenges around capturing assessment data through coding test items and curriculum mapping, as well as analyzing the data through learning analytics to inform curriculum and instruction improvements.

1. 1866.429.8889 | 1.954.429.8889 learn.examsoft.com
Thank you for joining.
The webinar will begin shortly.
Webinar Name
Presenter Name
Date
Universal Assessment
Challenges in Health Care
Related Education
An ExamSoft Client Webinar

2. Universal Assessment Challenges in
Healthcare Related Educa7on
JANE M. SOUZA, PH.D.
ASST. DEAN OF ASSESSMENT, WEGMANS SCHOOL OF PHARMACY
CHAIR, ASSESSMENT LEADERSHIP TEAM, ST. JOHN FISHER COLLEGE
CHAIR, AMERICAN ASSOCIATION OF COLLEGES OF PHARMACY, ASSESSMENT SIG

3. Agenda
Establishing
Context:
External
and
Internal
Drivers
Capturing
Data:
Coding
Exam
Blueprin:ng:
Plan
for
Curriculum
Coverage
Learning
Analy:cs:
Effec:ve
use
of
the
data

4. CONTEXT:
External Drivers
Accredita:on
Standards
increasingly
calling
for
Ø documented
assessment
data
Ø greater
accountability
for
assessment
prac4ces
Basic
World
Federa:on
Medical
Educa:on
Standards:
Ø Strengthened
requirements
for
alignment
between
teaching
and
learning
Ø Forma:ve
assessments
with
provisions
for
feedback
Ø Assurance
that
graduate
outcomes
are
assessed1
1Hayes,
R.
2014.
The
poten4al
impact
of
the
revision
of
Basic
World
Federa4on
medical
Educa4on
Standards.
Medical
Teacher
36
(6):
459-­‐462.

5. CONTEXT:
External Drivers
Standards
for
Accredita:on
of
Medical
Educa:on
Programs
Leading
to
the
MD
Degree
Ø 8.2…uses
formally
adopted
medical
educa4on
program
objec:ves
Ø 8.4
…uses
a
variety
of
outcome
data
Ø 9.4…employs
a
variety
of
measures
for
the
assessment
of
student
achievement,
including
students’
acquisi4on
of
knowledge
Ø 9.7…ensures
that
each
medical
student
is
assessed
and
provided
with
formal
forma:ve
feedback…to
allow
sufficient
4me
for
remedia4on2
2Liaison
CommiYee
on
Medical
Educa4on.
2014.
Standards
for
Accredita4on
of
Medical
Educa4on
Programs
Leading
to
the
MD
Degree

6. CONTEXT:
Internal Drivers
• identify
student
weakness
to
remediate
early
• Collect
formative
data
for
continuous
improvement
of
teaching
strategies
and
curriculum
• document
curriculum
coverage
Student
Perspec:ve
Faculty
Perspec:ve
Admin
Perspec:ve

7. Old (Adequate) Curriculum Map

8. Curriculum Map Mee7ng New Expecta7ons

9. How do we disaggregate the test data?
Tag
or
“code”
each
test
item
–
or
dimension
on
a
performance
rubric
Ø Accredita4on
Standard
Ø Course
Learning
Outcome
Ø Level
of
Bloom’s
Taxonomy,
Miller’s
Pyramid
of
Assessment
Ø What
else
would
you
code?

10. NATIONAL BOARD OF MEDICAL EXAMINERS
USMLE Step 1
Behavioral
Sciences
Biochemistry
Biosta:s:cs
&
Epidemiology
Cardiovascular
System
Gastrointes:nal
System
General
Principles
of
Health
&
Disease
Gene:cs
Gross
Anatomy
&
Embryology
Hematopoie:c
&
Lymphore:cular
Systems
Histology
&
Cell
Biology
Immune
System
Microbiology
&
Immunology
S1.01
S1.02
S1.03
S1.04
S1.05
S1.06
S1.07
S1.08
S1.09
S1.10
S1.11
S1.12

11. Example
Ques:on
–
Step
One
Exam
3
A
32-­‐year-­‐old
woman
with
type
1
diabetes
mellitus
has
had
progressive
renal
failure
over
the
past
2
years.
She
has
not
yet
started
dialysis.
Examina4on
shows
no
abnormali4es.
Her
hemoglobin
concentra4on
is
9
g/dL,
hematocrit
is
28%,
and
mean
corpuscular
volume
is
94
m3.
A
blood
smear
shows
normochromic,
normocy4c
cells.
Which
of
the
following
is
the
most
likely
cause?
• Acute
blood
loss
• Chronic
lymphocy4c
leukemia
• Erythrocyte
enzyme
deficiency
• Erythropoie4n
deficiency
• Immunohemolysis
• Microangiopathic
hemolysis
• Polycythemia
vera
• Sickle
cell
disease
• Sideroblas4c
anemia
• b-­‐Thalassemia
trait
(Answer:
D)
3
Retrieved
from
hYp://www.usmle.org/step-­‐1/#ques4on-­‐formats
How
would
you
code
this
ques4on
to
document
coverage
of
USMLE
topics
and
assess
student
understanding
of
topics?

14. Accredita:on
Standards
Bloom’s
Taxonomy
School of Pharmacy Example
Faculty-­‐requested
category

16. Coding at the
course level

17. Miller’s Pyramid
Of
Assessment
MP
4
MP
3
MP
2
MP
1

18. Bloom’s
Taxonomy
BT
1
BT
2
BT
3

19. Sounds like a lot of work…..Why bother?

20. Data
from
faculty
created
test
items

21. *
The
Na4onal
Academies
Press
at
hYp://www.nap.edu/catalog.php?record_id=18738
*

23. OUTCOME
CATEGORIES
WEIGHT

24. Pre-release
Send instructions
Student Submission

25. Student Submission Onscreen Grading
grade

26. EngageStudents
Learning Analytics

27. Faculty use of the data
Course
Outcome
Assessment
1
Exams
#1
and
#2
2
Exam
#3,
Paper
#1
3
Exam
#2
4
Exam
#4,
Presenta:on
#1
5
Exams
#2,
#3,
and
#4
Course
Outcome
Assessment
Student
Average
#
Items
per
Outcome
What
ac:ons
will
be
taken
1
Exam
1,
2
85.6%
45
2
Exam
3,
Paper
1
79.5%
20
3
Exam
2
71.1%
56
4
Exam
4,
Pres.
1
90.2%
60
5
Exams
2,
3,
4
87.6%
19

28. Topic/Learning
Outcome
Instructor
Number
of
lecture
hours
Number
of
test
items
%/#
Knowledge
%/#
Applica:on
%/#
Synthesis
Immune
System
Dr.
H.
6
24
20%
(5)
30%
(7)
50%
(12)
Microbiology
&
Immunology
Dr.
A.
4
16
30%
(5)
30%
(5)
40%
(6)
EXAM
TOTAL
10
40
25%
(10)
30%
(12)
45%
(18)
Tool
#1:
Exam
Blueprint4
4
JMSouza,
JLMathews,
St.
John
Fisher
College

30. Ques4on
Performance

31. Student
use
of
the
data

32. Sample
Student
Exam
Report

35. Sample
Student
Longitudinal
Report
STUDENT
PEER
GROUP
OUTCOME

36. Advisor
–
Students
Report
Student
/
Advisee
1
Student/
Advisee
2
Student/
Advisee
3

37.
#
Exams/
Projects
#
Items/
Assignments
Group
Average
BT1
Knowledge
146
2951
84.73
%
BT2
Applica:on
136
998
84.20
%
BT3
Synthesis
47
83
81.88
%
S01
Cardiovascular
35
293
82.28
%
S01
Respiratory
23
71
85.17
%
S03
Gastrointes:nal
18
100
86.14
%
S04
Renal
23
110
77.58
%
S05
Neurologic
43
294
85.40
%
Tracking Longitudinal Data

38. Using analy7cs effec7vely
ü Create
exam
blueprints:
plan
coverage
of
outcomes
ü Review
exam-­‐level
results:
reflect
on
teaching
/
learning
ü Save
exam
summaries:
create
an
archive
ü Share
summaries
with
students:
engage
students
in
assessment
ü Generate
longitudinal
reports:
track
progress
over
4me
ü Ask
students
to
reflect
on
personal
longitudinal
reports:
plan
for
future
study
ü Use
the
data
as
evidence
of
teaching
effec4veness
ü Use
the
data
for
scholarship
of
teaching
and
learning
ü Use
the
data
to
support
program
effec4veness.

39. ü Data
provided
for
students
• Informs
study
• Guides
remedia4on
ü Data
provided
for
faculty
• Documents
coverage
of
course
outcomes
• Iden4fies
area
of
student
and
cohort
weaknesses
• Provides
opportuni4es
for
Scholarship
of
Teaching
&
Learning
(SToL)
• Provides
evidence
of
teaching
effec4veness
for
dossiers
ü Data
provided
for
administrators
and
accreditors
• Maps
curriculum
coverage
• Documents
student
performance
on
curricular
goals
• Documents
evidence-­‐based
changes

40. CONTEXT:
External Drivers
Accredita:on
Standards
increasingly
calling
for
Ø documented
assessment
data
Ø greater
accountability
for
assessment
prac4ces
Basic
World
Federa:on
Medical
Educa:on
Standards:
Ø Strengthened
requirements
for
alignment
between
teaching
and
learning
Ø Forma:ve
assessments
with
provisions
for
feedback
Ø Assurance
that
graduate
outcomes
are
assessed1
1Hayes,
R.
2014.
The
poten4al
impact
of
the
revision
of
Basic
World
Federa4on
medical
Educa4on
Standards.
Medical
Teacher
36
(6):
459-­‐462.

41. CONTEXT:
External Drivers
Standards
for
Accredita:on
of
Medical
Educa:on
Programs
Leading
to
the
MD
Degree
Ø 8.2…uses
formally
adopted
medical
educa4on
program
objec:ves
Ø 8.4
…uses
a
variety
of
outcome
data
Ø 9.4…employs
a
variety
of
measures
for
the
assessment
of
student
achievement,
including
students’
acquisi4on
of
knowledge
Ø 9.7…ensures
that
each
medical
student
is
assessed
and
provided
with
formal
forma:ve
feedback…to
allow
sufficient
4me
for
remedia4on2
2Liaison
CommiYee
on
Medical
Educa4on.
2014.
Standards
for
Accredita4on
of
Medical
Educa4on
Programs
Leading
to
the
MD
Degree

42. Thank you
Ques4ons?
Sugges4ons?
Jane
M.
Souza,
Ph.D.
JaneMarieSouza@gmail.com

43. 43
Click to edit Master title style
Click to edit Master subtitle style
For More Information:
Call: 1.866.429.8889
Email: info@examsoft.com
Visit: learn.examsoft.com