Presented by Melinda E. Lull, Ph.D., Associate Professor, Pharmaceutical Sciences, and Jennifer L. Mathews, Associate Professor, Pharmaceutical Sciences, both from Wegmans School of Pharmacy, St. John Fisher College For many courses, exams are a significant portion of the overall grade. These high-stakes exams can be the deciding factor in a student passing or failing a class. The consequences for failing a course can mean delayed graduation and lost employment dollars. In addition, exam data is often used to determine if curricular revisions are needed programmatically and work should be done in advance to ensure that the exam data is reflective of the quality of the course. To prepare faculty for writing high quality exams we conducted a series of workshops on learning outcomes, exam question construction, exam blueprints, coding to Bloom’s Taxonomy and the peer-review process. Each of these workshops helped to form the foundation of constructing exams which truly reflected student learning. In addition, we have implemented guidelines to create a consistent look to all exams. For example the stem of every question includes the point value, type of question, and if partial credit will be granted. We also have guidelines to assist in settings for exam postings and releasing results. Significant efforts have been made to ensure that students are presented with high-quality, consistent exams and that the grade they achieve is based only on their comprehension of the material.

1. Assessment Consistency:
Is Everyone on the Same Page?
Department of Pharmaceutical Sciences
Wegmans School of Pharmacy
St. John Fisher College, Rochester, NY
Mindy Lull, Ph.D.
Associate Professor
Jennifer Mathews, Ph.D.
Associate Professor

2. Participants will learn:
1. How to use faculty development workshops to
provide the foundation of writing quality
exams.
2. How to effectively use the peer-review process.
3. How to create consensus and guidelines for
creating and launching exams.

3. Why is consistency important?
• Consistency contributes to exam quality
• Exams should reflect content learning
• Reduces student confusion or frustration
• Exams are high-stakes
• Data are used for many purposes

4. Faculty Development for Consistency
• To prepare faculty for writing high quality exams,
we conducted a series of workshops:
– learning outcomes
– exam blueprints
– coding to Bloom’s Taxonomy
– peer-review process
• Each of these workshops helped to form the
foundation of constructing exams which truly
reflected student learning.

5. Develop clearly written
statements of key learning
outcomes.
Use results of
assessments to
improve teaching and
learning.
Design experiences
that provide
opportunities to
achieve outcomes.
Assess student
achievement of
outcomes
Cycle as described by Middles States
Commission on Higher Education
Cycle of Assessment

6. Clearly written outcomes and exam
maps
• We made the decision to use ExamSoft® as our
tool for course assessments.
• Before any exams were created using the
technology we had several faculty development
workshops.
• Development sessions led by NABP
– Revision of course learning outcomes by faculty in the
course
– Techniques for writing good exam questions
– Use of exam Maps
– Proper coding to Blooms Taxonomy
• All faculty are writing and coding their own exam
questions.

7. Exam map: is my exam reflective of
what I taught?
Course learning outcome: Based on
the site and mechanism of action,
predict the therapeutic and side
effects associated with
pharmacological agents used to
treat various disease states. Course Activity Lecture LO Knowledge Application Synthesis
Lecture 2 (9.11.12 – 9.13.12) APPE
lectures
Lecture, TP,
patient
scenario
EPO REMS
· pg. ; Bank question 1420 (MC)
Anemia
Outcome 4
1
Iron + Vitamin C
· pg. 9; Bank question 1525 (FITB)
Anemia
Outcome 4
1
Anemia from RBC destruction
• pg. ; Bank question 1527 (SA)
Anemia
Outcome 1,
2, 3
1

8. Continued updates and quick tips
• Use time during faculty retreats and meetings to
practice coding sample questions, to address
coding inconsistencies, holes in our curricular
coverage, questions/concerns
• Email reminders with “how to”
– Ex. How to write a good FITB question
• Binders for all faculty with sample materials
– Appendix B topics, WSOP program outcomes,
Bloom’s Taxonomy verbs

9. Resources: Sample “How to”

10. The peer review process
• “Peer-reviewed articles
provide a trusted form of
scientific
communication.”
• “Even if you are
unfamiliar with the topic
or the scientists who
authored a particular
study, you can trust peer-
reviewed work to meet
certain standardsof
scientific quality.”
Diagram and quotes from: http://undsci.berkeley.edu/article/howscienceworks_16

11. Peer review
• As academics we place value in the peer
review process.
• We should also place value in writing high
quality, peer-reviewed assessments.

12. Our Process
§ This is a voluntary process for faculty who want
to participate
§ 2-3 reviewers/exam
§ Faculty are invested in the process
§ Reviewers DO NOT make changes to exams, and
item writers can choose not to make suggested
changes
§ Student data is much more reflective of what
was learned versus could my test item be
deciphered

13. Quality of questions: Peer
reviewed test item
YES NO Comments/
Suggestions
1 Is the question clear to the reader?
2. Are there typographical errors?
3. Is the question coded to the
appropriate level of Bloom’s
Taxonomy?
4. Is the question coded appropriately
to the learning outcome(s)?
5. If the question has a history, did it
perform well in the past?
6. Is the content level appropriate for
the course?
7. Other considerations: “positive”
language, jargon free, limited
abbreviations, double-jeopardy, exam
time

14. PRTI: Peer Reviewed Test Items

15. How to learn from others
• Problem areas:
1. FITB questions
2. Short answer/essay (character limits)
3. Eliminating K-type
4. Select all that apply

16. We now have a validated test bank
• We now have a validated test bank.
• Over 3000 questions have been peer-
reviewed!

17. Creating Formatting Guidelines for
Consistency
• Ensures a consistent look to all exams and exam
reports
• Consider having a “Quick Tips” document that
outlines these for all faculty
• Guidelines
– Question format
– Exam creation and postings
– Exam administration
– Exam reports

18. Consistency in Question Format
• Can the student get a quick idea of the question?

19. Consistency in Question Format
• Are commonly used elements formatted the
same way?

20. Consistency in Question Format
• Are commonly used elements formatted the
same way?

21. Consistency in Question Format
• Are commonly used elements formatted the
same way?

22. Consistency in Exam Creation
• Prevent logistical errors in exam with a checklist:
– Correct # of points
– Check question groupings
– Override font and font size
– Check settings on partial credit, select-all-that-apply
and k-type question
– Export copy of exam for last-minute spelling errors
– Preview exams before posting

23. Consistency in
Exam Posting

24. Consistency in Exam Posting

25. Consistency in Exam Posting

26. Consistency in Exam
Administration
• Exam details
• Assessment details
• Policies
• Contact information

27. Consistency in Exam Reports

28. Conclusions
• Good exam consistency involves question
writing, exam creation, exam posting, exam
administration, and exam reports.
• Training is required to get all faculty on the same
page, and to emphasize the importance of
consistency.
• A “Quick-Tips” guide is useful for logistical
recommendations for consistency.

29. Thank you!
Contact us:
Melinda Lull: mlull@sjfc.edu
Jennifer Mathews: jmathews@sjfc.edu