Presented by Melinda Lull, Jennifer Matthews, and Jane Souza Vertical- Medical/ Nursing In a graduate level course, peer instructors were trained to create online quizzes for use as optional formative assessments by students. These quizzes were created using ExamSoft, and designed to reflect the format of summative in-class exams. Quizzes were made available to students, and performance was recorded. At the conclusion of each semester, quiz and exam psychometrics were analyzed and compared to determine the effect of quizzes on exam performance in the courses. Comparisons revealed improvements in scores between quizzes and exams and a benefit to students utilizing the resources, making them a useful instrument in the self-directed learning toolbox. Self-testing, a type of formative assessment, provides students with a study tool that can help identify areas of weakness. Despite their usefulness, this type of additional learning resource can often be time consuming for faculty members, limiting its successful use across multiple courses. The use of peer tutors has been shown to provide advantages to faculty members as well as students utilizing tutoring services and the tutors themselves. This session will instruct participants on the implementation and analysis of self-assessments using ExamSoft. At the end of the session, participants will be able to: 1) Demonstrate the process of enrolling peer instructors as exam item writers. 2) Demonstrate the use of the internal comments box as a means of providing feedback to peer instructors. 3) Describe the process for analyzing ExamSoft data in the comparison of quiz and exam performance.

1. 1ExamSoft Assessment Conference | #ASSESS2015
ExamSoft Assessment Conference
#ASSESS2015

2. 2ExamSoft Assessment Conference | #ASSESS2015
Assessing the Impact of Peer
Instructor Generated Formative
Self-Assessments
Presenters:
Melinda Lull Ph.D., Jennifer Mathews Ph.D.
Wegmans School of Pharmacy
St. John Fisher College, Rochester, NY
Jane Marie Souza, Ph.D.
University of Rochester, Rochester, NY

3. 3ExamSoft Assessment Conference | #ASSESS2015
Learning Objectives for the Session:
• Demonstrate the process of enrolling peer
instructors as exam item writers.
• Demonstrate the use of the internal
comments box as a means of providing
feedback to peer instructors.
• Describe the process for analyzing ExamSoft
data in the comparison of quiz and exam
performance.

4. 4ExamSoft Assessment Conference | #ASSESS2015
Why use self-assessments?
• Self-assessments increase course
performance. (Stewart et.al., 2014; West and Sadoski, 2011)
• Self-assessments increase self-directed
learning. (Nicol et.al., 2006)
• Students achieve better long-term retention
of material. (Nevid and Mahon, 2009)
• Students get more practice with the format of
in-class exams.

5. 5ExamSoft Assessment Conference | #ASSESS2015
Why use peer instructors?
• Peer instructors increase student
performance. (Santee and Garavalia, 2006)
• Students may feel more comfortable with
peers than faculty. (Carrol, 1996)
• Peer instructors themselves benefit from
reviewing material. (Haist et.al., 1997; Haist et.al., 1998)

6. 6ExamSoft Assessment Conference | #ASSESS2015
Training
Faculty
ExamSoft Infrastructure
Enrolling Peer Instructors
Students
Question Writing
ExamSoft
Implementation
Question Writing
Question Editing
Feedback
Launching Quizzes
Assessment
Utility
Impact
Students
Peer Instructors

7. 7ExamSoft Assessment Conference | #ASSESS2015
• ExamSoft Infrastructure
– Peer instruction requires separate folders
• Questions
• Assessments
• Course
Training Faculty
ExamSoft Infrastructure
Enrolling Peer Instructors

8. 8ExamSoft Assessment Conference | #ASSESS2015
• Enrolling Peer Instructors
– Peer Instructor “Faculty” Account
• E.g., tutors@sjfc.edu
– With limited permissions
• Create exam items only
• Only access tutoring courses question folder
• Can only save, not approve, questions
Training Faculty
ExamSoft Infrastructure
Enrolling Peer Instructors

9. 9ExamSoft Assessment Conference | #ASSESS2015
• Provide peer instructors with:
– Detailed ExamSoft instructions
– Resources on how to write good questions
with examples
• Considerations:
– Have them provide rationales for all questions
– Limit question types (e.g., essays)
Training Students
Question Writing
ExamSoft

10. 10ExamSoft Assessment Conference | #ASSESS2015
• This is a learning process for peer instructors
too!
Implementation
Question
Writing
Question Editing
Feedback

11. 11ExamSoft Assessment Conference | #ASSESS2015
• Faculty will ultimately be in control of:
– Reviewing all questions
– Assembling quizzes
– Launching quizzes
• Aim for consistency
– Across quizzes, courses, and with in-class exams
• Considerations:
– Posting multiple versions or allow for multiple
downloads
– Enable secure quiz and review
– Make passwords easily available (e.g., post on course
management site)
– Ensure adequate time to access
Implementation
Launching
Quizzes

12. 12ExamSoft Assessment Conference | #ASSESS2015
Implementation
Launching
Quizzes

13. 13ExamSoft Assessment Conference | #ASSESS2015
• Data Available from ExamSoft
– Number of exam takers
– Aggregate performance on quizzes and
individual questions
– Individual student performance on individual
questions
– Category performance*
Assessment
Utility
Impact

14. 14ExamSoft Assessment Conference | #ASSESS2015
Assessment Utility
Number per semester 9-12
# quizzes per exam 2-3
Average # of students taking each
assessment (% of class)
1st attempt: 53.9 (67.1%)
2nd attempt: 24.2 (30.2%)
% taking at least one quiz 95%
Average # of quizzes taken per student 8.5
Average of 6 semesters worth of data in 5 different courses

15. 15ExamSoft Assessment Conference | #ASSESS2015
• Comparisons using data mined from
ExamSoft
– Exam performance vs. quiz performance
– Exam vs. quiz performance broken down by
course learning objective
– Exam performance of quiz takers vs. non-quiz
takers
Assessment Impact
Students
Peer Instructors

16. 16ExamSoft Assessment Conference | #ASSESS2015
Assessment Impact
Students
Peer Instructors
* p<0.05
** p< 0.001
Example: Quiz vs. Exam Performance by Exam
By Learning OutcomeBy Exam

17. 17ExamSoft Assessment Conference | #ASSESS2015
Assessment Impact
Students
Peer Instructors
Student Perceptions of Tutoring Quizzes (n=37)
Question
Average
Score*
%
SA/A
I feel that the tutoring quizzes are a valuable resource. 4.65 95%
I feel that taking the tutoring quizzes increased my
confidence going into the exam.
4.43 92%
I feel that taking the tutoring quizzes increased my
performance on the exam.
4.27 81%
I feel that the tutoring quizzes accurately reflected the
material that was on the exams.
4.24 92%
I recommend continuing to offer tutoring quizzes in the
future.
4.76 98%
Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly
Agree); SA= Strongly Agree; A= Agree

18. 18ExamSoft Assessment Conference | #ASSESS2015
Assessment Impact
Students
Peer Instructors
Peer Tutor Perceptions of Tutoring Quizzes (n=8)
Question Average Score*
%
SA/A
Online tutoring quizzes were a valuable resource for
the students I was instructing.
4.63 100%
Writing online tutoring quiz questions increased my
knowledge of the course topics covered.
4.38 88%
Learning to write quiz questions was a valuable
learning experience.
4.63 100%
Based on a Likert Scale of 1-5 (1= Strongly Disagree; 5= Strongly Agree); SA= Strongly Agree; A= Agree

19. 19ExamSoft Assessment Conference | #ASSESS2015
Use of Self-Assessments in Pharmaceutical
Science Courses
• Started with a pilot in 1 pharmacology course
with 2 peer instructors
• Now implemented in 7 courses throughout
the pharmaceutical sciences curriculum
– 16 peer instructors involved
• All guided by same guidelines and using
same resources and instructions

20. 20ExamSoft Assessment Conference | #ASSESS2015
Conclusions
• Creating the infrastructure for self-assessments
in ExamSoft is similar to that of in-class
assessments, and only requires a few minor
modifications.
• Feedback demonstrates that students perceive a
benefit from self-assessments.
• Quiz and exam performance indicate that
students benefitted from self-assessments.
• Peer instructors perceived a benefit from
creating self-assessments.
• Faculty perceive a benefit and more courses are
utilizing self-assessments and peer instructors.

21. 21ExamSoft Assessment Conference | #ASSESS2015
References
• Carroll M. Peer tutoring: can medical students teach biochemistry. Biochem
Educ. 1996; 24:13-15
• Haist SA, Wilson JF, Fosson SE, Brigham NL. Are fourth-year medical students
effective teachers of the physical examination to first-year medical students. J
Gen Intern Med. 1997; 12:177-181
• Haist SA, Wilson JF, Brigham NL, Fosson SE, Blue AV. Comparing fourth-year
medical students with faculty in the teaching of physical examination skills to
first year students. Acad Med 1998; 3: 190-200
• Nevid JS, Mahon K. Mastery quizzing as a signaling device to cue attention to
lecture material. Teach Psychol. 2009; 36(1): 29-32
• Nicol DJ, Facfarlane-Dick D. Formative assessment and self-regulated learning:
a model and seven principles of good feedback practice. Stud High Educ. 2006;
31(2): 199-218
• Santee J and Garavalia L. Peer tutoring programs in health professions
schools. AJPE 2006; 70: Article 70
• Stewart D, Panus P, Hagemeier N, Thigpen J, Brooks L. Pharmacy student self-
testing as a predictor of examination performance. AJPE 2014; 78(2): Article 32
• West C and Sadoski M. Do study strategies predict academic performance in
medical school? Med Educ. 2011; 45(7): 696-703

22. 22ExamSoft Assessment Conference | #ASSESS2015
THANK YOU!
CONTACT US:
Mindy Lull: mlull@sjfc.edu
Jennifer Mathews: jmathews@sjfc.edu
Jane Marie Souza:
JaneMarie.Souza@rochester.edu