Incorporation of sound curriculum evaluation measures and related analysis can provide evidence to support changes within the curriculum that close content gaps, as well as support for individual interventions for academically at-risk students as early as possible in the course sequencing, to avoid the prospect of a “too little, too late” response to learning deficits. To promote the process of continual program evaluation and quality improvement, faculty are better defining the data they analyze to drive fine-tuning of curricula, with the ultimate goal of achieving all desired curriculum outcomes. However, many programs lack the assignment of these analysis tasks within their curriculum committee framework, and as a result, changes to testing policy may be implemented without much evidence-based reason, and may be carried out in a way that is irrespective of other curriculum revisions. In best cases, this lack of consistency with the analysis as well as the lack of attention to curriculum impact once these testing policies are implemented results in the lack of any observable increase in desired outcomes like improved pass rates. At worst, this situation results in the “wheels spinning” scenario, where faculty serving on the curriculum committee appear to make random but unrelated policy changes throughout the academic year, with no real clarity about what outcomes they are expecting from these interventions, and no way of accruing data after the fact that can be analyzed for any evidence of improvement. This webinar will address a common trend that is increasingly being adopted by faculty to avoid this type of scenario: the formulation of a testing committee. The discussion will encompass methods used to evaluate both total program outcome achievement and individual student performance, using methods for both internal and external curriculum evaluation, and will identify how faculty can incorporate consequences associated with students’ scores and other evaluation data within their testing policies that have been shown in research studies to improve outcomes. Another key role of the committee is the design and implementation of all testing-related policies within the curriculum, generally with approval of the overall curriculum committee, but also with input from the student affairs committee, as these testing policies relate to admission, progression, and graduation policies that are generally within the oversight of the student affairs committee. Finally, the testing committee will be described as the regulator of the school’s testing style manual with respect to item creation, editing, and removal of test items from the item bank used for teacher-made exams, based on a systematic review of item analysis data in concert with sound item writing skills designed to produce test items at the application-and-above level within the cognitive taxonomy.

1. 1
An ExamSoft Client Webinar
Building Testing
Committees that have
Authority to Create
Effective Change

2. Building
Tes,ng Commi0ees
that have authority to
create effec,ve change
AINSLIE T. NIBERT, PHD, RN, FAAN
JANUARY 20, 2015
2

3. Why form a Tes,ng Commi0ee?
Higher
Educa-on
Trends
•
Increased
demand
for
measurable
achievement
of
program
outcomes
• Surveillance
for
gaps
in
curriculum
becoming
an
ongoing
need
• Avoidance
of
inadequate/late
response
to
academically
at-­‐risk
students
• Consequences
affec?ng
their
admission,
progression,
gradua?on
• Increasing
faculty
re?rements
–
inexperienced
faculty
stretched
thin
• Curriculum
evalua?on
is
a
low
priority
un?l
re-­‐accredita?on
looms
• Less
experienced
faculty
lack
skills
in
test
item
wri?ng
and
exam
analysis
• Administra?on:
tes?ng
policies
lack
consistencies
with
their
adop?on,
maintenance,
and
revisions;
policy-­‐making
should
be
supported
by
evidence
3

4. Tes,ng Policies & Procedures:
Evidence-­‐based & Consistent
Well-­‐wri6en
test
policies
and
procedures,
carried
out
consistently,
ensure:
• Defensibility
of
Tests
used
in
each
course
across
the
curriculum
• Test
Blueprint
aligns
with
Course
Objec?ves
• Standard
scoring
procedures
are
used
for
each
test
administra?on
• Policies
are
constructed
using
APA’s
Code
&
are
applied
consistently
• Sta?s?cal
item
analysis
is
systema?c
and
applied
rou?nely
to
each
test
• Procedures
for
item
nullifica?on,
rescoring,
second
grader
review
are
set
up
• Mechanisms
for
documen?ng
test
outcomes
and
analysis
are
established
• Consistent
requirements
established
across
all
courses
• Computerized
reports
used
whenever
possible
to
streamline
effort
for
the
faculty
• Code
of
Fair
Tes?ng
Prac?ces
in
Educa?on
by
the
American
Psychological
Associa?on
applied
• See
hSp://www.apa.org/science/programs/tes?ng/fair-­‐code.aspx
4

5. A need for synergy: Tes,ng Policies rela,ng to
Admission, Progression, and Gradua,on
Tes-ng
Commi6ee
and
Student
Affairs
Commi6ee
–
policy
alignment
• The
tes?ng
commiSee’s
recommenda?ons
about
tes?ng
policy
• Can
overlap
with
policies
adopted
by
the
Student
Affairs
CommiSee
• Grading
policies
• Student
grievance
policies
• Admission
requirements
• Condi?ons
students
must
meet
to
be
considered
candidates
for
re-­‐admission
to
the
program
• Following
academic
failure
of
a
course
or
courses
• Following
withdrawal
from
a
course,
or
from
the
program,
for
any
reason
• Progression
requirements
• Gradua?on
requirements
The
tes?ng
commiSee
does
not
formulate
policy
independently,
but
reports
recommenda?ons
to
the
curriculum
commiSee
for
full
faculty
approval.
5

6. Curriculum evalua,on: Use aggregate
student test data to find/close the gaps
• Total
program
evalua?on
and
curriculum
revisions
should
be
accomplished
using
an
evidence-­‐based
approach
that
is
systema?c.
• Use
aggregate
student
response
data
to
establish
benchmarks;
track
each
cohort’s
performance
against
benchmarks
and
compare
to
previous
cohorts
• Use
aggregate
student
response
data
gathered
longitudinally
from
mul?ple
cohorts
to
iden?fy
weaknesses
consistently
seen
by
course
or
by
concept
across
mul?ple
courses;
seek
sources/causes
of
program
weaknesses
• When
curriculum
evalua?ons
are
substan?ated
by
both
faculty
expert
opinion
(subjec?ve
data)
and
aggregate
student
test
results
(scoring
data,
item
analysis
sta?s?cs
that
substan?ate
test
reliability
and
validity
–
objec?ve
data,)
the
resul?ng
revisions
are
specifically
targeted
at
areas
of
program
weaknesses
• Saves
faculty
?me
by
pinpoin?ng
areas
of
concern
&
streamlining
revisions
• Completes
the
“evalua?on
loop”
-­‐-­‐-­‐
just
collec?ng
the
data
is
insufficient
6

7. Avoid the “too li0le, too late”
response for the at-­‐risk student
Tes-ng
commi6ees
can
make
recommenda-ons
about
individual
student
interven-ons,
but
should
not
usurp
the
role
of
Student
Affairs.
• Recommend
updates
to
scoring
benchmarks
following
data
analysis
• Create
or
adopt
tools
for
longitudinal
student
data
collected
in
each
course
• Require
course
coordinators
to
submit
the
datasets
each
semester
• Establish
remedia?on
plans
–
create
student
learning
contract
templates
• Formulate
tools
that
summarize/document
results
following
implementa?on
of
individual
student
ac?on
plans
to
improve
performance
• Coordinate
with
Student
Affairs
and
Curriculum
(Faculty)
CommiSee
• Recommend
changes
to
Admission,
Progression,
and
Gradua?on
Policies
7

8. Commitment to adherence to
Best Prac,ces in Tes,ng
Ensure
that
tes?ng
commiSee
has
proper
representa?on
and
authority
◦ Representa?on
across
all
levels
of
the
program
◦ Sufficient
resources
to
stay
abreast
of
best
prac?ces
and
elevate
faculty
skills
Make
the
commiSee
a
sub-­‐commiSee
of
the
curriculum
commiSee
or
other
faculty
commiSee
that
represents
the
en?re
faculty
◦ Policy
and
procedures
recommended
by
the
tes?ng
commiSee
◦ But
are
approved
by
the
overall
faculty
commiSee
Enforcement
–
item
bank,
exam
bank,
style
guide,
etc
◦ CommiSee
must
have
an
avenue
through
administra?on
hierarchy
to
enforce
policies
established
and
carry
out
consequences
when
policies
are
ignored
or
thwarted
8

9. Best Prac,ce #1: Establish Tes,ng Commi0ee
Design
and
revise
the
program’s
tes?ng
policies
◦
Publishes
Guidelines
for
Exam
Development
◦ Wri?ng
style
protocol
–
apply
to
all
tests
◦ Cri?cal
thinking
items
–
increase
Applica?on
level
Exam
Administra?on
Policies
◦ Number
of
items/exam;
length
of
?me
alloSed/exam
◦ Proctoring
Guidelines
◦ Dissemina?on
of
Grades
(When?
How?)
◦ Test
Review
–
to
do
or
not
to
do;
how
to
do
it

10. …more Tes,ng Commi0ee Ac,vi,es
Systema?c
Item
Analysis
(see
also
#3
below)
◦
Use
sta?s?cal
parameters
for
analyzing
overall
test
and
individual
test
items
◦ Adhere
to
published
psychometric
standards
◦ Summarize
analysis
and
include
ac?on
plan
◦ Review
test
blueprint
as
needed
before
next
test
◦ Review
individual
items
earmarked
for
aSen?on
Compile/analyze
standardized
test
results
◦ Make
recommenda?ons
to
curriculum
commiSee
for
changes
based
on
findings
10

11. Best Prac,ce #2: Wri,ng Style Protocol
Establishing
clear
guidelines
for
faculty
leaves
liSle
room
for
ambiguity
and
helps
insure
uniformity
in
the
presenta?on
of
exams
throughout
the
curriculum:
Will
present
or
past
tense
be
used
for
test
items?
Will
op?ons
end
in
periods,
whether
or
not
there
are
complete
sentences?
Will
all
op?ons
begin
with
a
capital
leSer?
When
stressing
a
word
in
the
stem,
will
it
be
highlighted,
boldfaced,
italicized,
underlined?
Will
the
term
pa?ent
or
client
be
used?

12. Best Prac,ce #3: Increase propor,on of Applica,on-­‐
and-­‐above Items
Which
interven?on
is
most
important?
Which
interven?on,
plan,
assessment
data
is/are
most
cri?cal
to
developing
a
plan
of
care?
Which
interven?on
should
be
done
first?
What
ac?on
should
the
nurse
take
first?
Which
interven?on,
plan,
nursing
ac?on
has
the
highest
priority?
What
response
is
best?
12

13. Best Prac,ce #4: Use Uniform Sta,s,cal Parameters
for Item Analysis
What
is
the
minimum
acceptable
difficulty
level
for
a
test
item?
What
is
an
acceptable
discrimina?on
level
(PBCC)
for
a
test
item?
What
is
the
acceptable
number
of
mastery
items
to
include?
What
is
an
acceptable
reliability
coefficient
for
the
exam?
13

14. Systema,c Item Analysis used for all tests:
Use a 3-­‐Step Method
1.
Review
Difficulty
Level
2.
Review
Discrimina?on
Data
Item
Discrimina?on
Ra?o
(IDR)
Point
Biserial
Correla?on
Coefficient
(PBCC)
3.
Review
Effec?veness
of
Alterna?ves
Response
Frequencies
Non-­‐distracters
14
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd
ed.).Houston, TX: Health Education Systems, Inc.

15. 15
Parameter
Level
of
Acceptance
Item
Difficulty
30%-­‐90%
Item
Discrimina?on
Ra?o
25
%
and
above
PBCC
(point
biserial
correla?on
coefficient)
0.20
and
above
KR20
(Kuder-­‐Richardson
20)
0.70
and
above
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd
ed.).Houston, TX: Health Education Systems, Inc.
Use these Recommended Item Analysis Standards

16. PBCC
&
KR-­‐20:
Standards
of
Acceptance
for
Nursing
versus
General
Educa?on
exams
16
PBCC:
0.15
and
above
KR20:
0.60
-­‐
0.65
and
above
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd
ed.).Houston, TX: Health Education Systems, Inc.

17. Analyze
Student
Response
Frequencies
Target
revisions
of
op?ons
with
‘0’
responses
These
op?ons
are
not
plausible;
even
poor
performers
know
enough
to
avoid
these
choices
on
the
exam,
which
increases
prospect
for
success
by
guessing
When
an
op?on
is
chosen
by
fewer
than
30
test
takers,
this
generally
indicates
poor
item
discrimina?on
and
should
be
edited.
If
fewer
than
30
test
takers
took
the
test,
frequencies
are
not
reliable;
cumula?ve
data
should
be
used.
17

18. Make Curricular Evalua,on consistent by
adop,ng a Master Test Blueprint Format
18

19. Generate an electronic blueprint (using the
master format) for each exam in each course
19

20. Test Review Summary
20

21.
Summary: Tes,ng ‘Best Prac,ces’
Improve
Test
Blueprints
Adopt
a
uniform
item
style/format;
publish
the
guidelines
Invert
the
propor?on
of
Applica?on-­‐and-­‐above
test
items
as
compared
to
Knowledge/Comprehension
items
Use
a
Systema?c
Item
Analysis
methodology
Include
10%
alterna?ve
item
formats
–
across
all
tests
Remove
items
that
no
longer
align
with
na?onal
standards
or
current
clinical
prac?ce,
which
should
be
reflected
in
your
program
outcomes
◦ In
Nursing:
Remove
obsolete
nursing
diagnoses
◦ In
Nursing:
Eliminate
Trade
names
for
most
medica?ons
(use
generic
only)
For
competencies
that
must
be
tested
in
each
course,
be
consistent:
◦ In
Nursing:
10%
of
all
test
items
are
medica?on
calcula?ons
–
across
all
tests
in
all
courses
Revise
the
test
length/?me
allotments
–
be
consistent
across
all
tests
21

22. ….more Tes,ng ‘Best Prac,ces’
Establish
scoring
benchmarks
that
are
evidence-­‐based/outcome-­‐based
Determine
consistent
weight
of
standardized
exam
in
overall
grade
Iden?fy
consequences
for
failure
to
achieve
benchmarks
◦ Offer
addi?onal
(parallel)
versions
of
the
exam
–
mix
these
up!
◦ Allow
enough
?me
for
remedia?on
to
occur
between
aSempts.
◦ Assign
specific
remedia?on
ac?vi?es
(mul?ple
strategies)
◦ Date/?me
for
submission
of
these
should
be
specified
in
the
learning
contract.
Monitor
remedia?on
usage
for
student
adherence
to
contract.
◦ Selec?ons
should
be
based
on
weak
areas
iden?fied
on
scoring
report.
◦ Focus
on
building
confidence
–
this
is
not
a
puni?ve
ac?vity.
Model
effec?ve
study
habits
and
test
taking
strategies.
Use
proctoring
guidelines;
take
test
security
measures
◦ Inves?gate
suspected
breaches
and
enforce
published
policies.
22

23. A few words about vigilance with Test Security
1. Encourage
moral
behavior
(Academic
honesty
program
at
your
school
with
clear
language
placed
in
handbooks)
2. Discourage
chea?ng
a. Before
tes?ng
1. Minimize
access
to
exams
and
viewing
of
exam
content
2. Use
highest
levels
of
security
available
in
LMS
for
teacher-­‐made
tests
and
all
security
features
available
in
a
standardized
tes?ng
plaqorm
-­‐
protect
logins
&
access
codes;
ac?ve
dashboarding
3. Train
proctors
for
live
proctoring
ac?vi?es;
re-­‐train/re-­‐cer?fy
regularly
b. During
tes?ng
1. Establish
secure
environment
2. No
devia?ons
to
test
procedures
or
breakdown
of
environmental
security
allowed.
Ex:
leaving
room
equates
to
the
test
being
over
for
that
student
regardless
of
reason
3. Vigilant
proctoring
(proctor
physically
walks
around
the
room
or
rou?nely
accesses
ac?ve
dashboards
for
remote
tes?ng
3. Detect
chea?ng
with
Data
Forensics
and
take
ac?ons
as
needed
23

24. References
American
Psychological
Associa?on.
(2004)
Code
of
Fair
Tes?ng
Prac?ces
in
Educa?on.
Washington,
DC:
Joint
CommiSee
on
Tes?ng
Prac?ces.
hSp://www.apa.org/science/programs/tes?ng/fair-­‐code.aspx
Morrison,
S.,
Nibert,
A.,
&
Flick,
J.
(2006).
Cri$cal
thinking
and
test
item
wri$ng
(2nd
ed.).
Houston,
TX:
Health
Educa?on
Systems,
Inc.
Na?onal
Council
of
State
Boards
of
Nursing.
(2013)
2013
NCLEX-­‐RN
test
plan.
Chicago,
IL:
Na?onal
Council
of
State
Boards
of
Nursing.
hSps://www.ncsbn.org/3795.htm
Nibert,
A.
(2010)
Benchmarking
for
student
progression
throughout
a
nursing
program:
Implica$ons
for
students,
faculty,
and
administrators.
In
Capu?,
L.
(Ed.),
Teaching
nursing:
The
art
and
science,
2nd
ed.
(Vol.
3).
(pp.45-­‐64).
Chicago:
College
of
DuPage
Press.
Schroeder,
J.
(2013).
Improving
NCLEX-­‐RN
pass
rates
by
implemen?ng
a
tes?ng
policy.
Journal
of
Professional
Nursing,
29(2),
S43-­‐S47.
Sewell,
J.,
Culpa-­‐Bondal,
F.,
&
Colvin,
M.
(2008).
Nursing
program
assessment
and
evalua?on:
Evidence
based
decision
making
improves
outcomes.
Nurse
Educator,
33(3),
109-­‐112.
24

25. Have Ques,ons? Need More Info?
Thanks
for
your
?me
&
aSen?on
today!
25
866-429-8889