Recently, Dr. Misty Stutz and Dr. Kimberly Daugherty, from Sullivan College of Pharmacy, show how strong assessments begin with solid exam items. However, many faculty members struggle with writing questions, or even knowing if their questions are indeed valid. At Sullivan University College of Pharmacy, didactic examinations are created and delivered by utilizing an embedded-assessment software.

1. 1
Using ExamSoft Data for
Faculty Development
Misty M Stutz, PharmD and Kimberly K Daugherty, PharmD BCPS
Sullivan University College of Pharmacy
Thank you for joining.
The webinar will begin shortly.

2. 2
Please pose questions to
the presenter through the
“Questions” field of the
Go To Webinar tool on
the right side of your
screen.
All questions will be
addressed at the
conclusion of the
presentation.

3. Using ExamSoft Data for
Faculty Development
Misty M Stutz, PharmD
Chair, Clinical and Administrative Sciences
Kimberly K Daugherty, PharmD BCPS
Assistant Dean, Academic Affairs and Assessment

4. Objectives
 Describe how to use ExamSoft reports to pull data
needed for a faculty teaching scorecard
 Explain how faculty scorecards can be used to help
faculty identify how their item writing compares to
their peers
 Discuss how the report cards can be interpreted and
how they can be used
 Describe how the report cards are organized and
presented to the faculty member

5. College Info
 Three year accelerated program
 Four quarter curriculum
 Academic Year July –June
 Graduated 1st class 2011
 Private Institution
 Career-based university
 Teaching focus
 Two departments
 30 Clinical and
Administrative Sciences
faculty
 11 Pharmaceutical Sciences
faculty

6. FACULTY EVALUATION

7. Background
 CAS Department: 45% of promotion criteria is
excellence in teaching
 Need for faculty feedback and development
 New college of pharmacy
 ”Young faculty”
 Prior to 2014, annual evaluation based on
 (1) self-evaluation and
 (2) student survey data

8. Assessment of teaching excellence
 Data from students and alumni (Subjective)
 Student Surveys
 Focus Groups
 Data from peers (Subjective/Objective)
 Classroom Observation
 Materials review
 Assessment review
 Self-assessment and reflection (Subjective/Objective)
 Reflections
 Student Performance
*P.Piascik et al./Currents in Pharmacy Teaching and Learning 3 (2011) 238-248

9. Faculty Teaching
Scorecard

10. Strongly
Agree
Agree Disagree
Strongly
Disagree
Unable to
Comment
18. The performance
feedback I receive is
effective. (“effective”
changed to
“constructive” 2016)
2013 21.7% (5) 60.9% (14) 17.4% (4) 0.0% (0) 0.0% (0)
2014 20.7% (6) 69.0% (20) 6.9% (2) 3.4% (1) 0.0% (0)
2015 39.3% (11) 57.1% (16) 0.0% (0) 0.0% (0) 3.6% (1)
2016 37.8% (14) 56.8 (21) 5.4% (2) 0.0% (0) 0.0% (0)
National 25.8% (789) 50.0% (1527) 13.7% (419) 4.3% (131) 6.2% (190)
AACP Faculty Survey

11. Using ExamSoft to create objective data
 All quizzes and examinations in Examsoft are tagged by:
 SUCOP outcome
 Curriculum topic
 Faculty Name
 Blooms taxonomy
 Item analysis is conducted after each examination to determine
if questions should be rescored:
 Exam Average
 Low and high exam scores
 Point biscerals
 Difficulty index
 Item analysis data can also be used to help develop faculty
exam writing skills.
http://university.examsoft.com/h/i/218895432-evaluating-the-statistics/233580

12. Item analysis
 Difficulty Index (0.00-1.00): The difficulty index
measures the proportion of Exam Takers who
answered an item correctly. A higher value indicates
a greater proportion of Exam Takers responded to an
item correctly. A lower value indicates that fewer
exam takers got the question correct.
 SUCOP rules for question review: questions are
reviewed for potential rescoring if they have a
difficulty level of 0.60

13. Item analysis
 Point Bi-Serial (-1.00-1.00): The point bi-serial
measures the correlation between an Exam Taker's
response on a given item and how the Exam Taker
performed on the overall exam.
 SUCOP rules for question rescoring:
 0.30 and up is considered a very good item and is not
rescored unless the difficulty index is below about 0.30
 0.20-0.29 is considered a reasonably good question that
may only need minor adjustment and is reviewed for
potential rescoring
 <0.20 is considered marginal to poor and is usually rescored
or full credit given

15. 39
19
13
10
6
15
0
5
10
15
20
25
30
35
40
45
> 90% 80-90% 70-80% 60-70% 50-60% <50%
Percentageofcorrectanswers
Difficulty level of question
Percentage of total questions that exam takers got correct
CAS Faculty Average
How our faculty write questions

16. 39
76
79
81
74
39
57
0
10
20
30
40
50
60
70
80
90
>90% 80-90% 70-80% 60-70% 50-60% <50% Total
Percentageofpb>0.2
Difficulty level of question
Percentage of questions with point bi-serial of > 0.2
CAS Faculty Average
How our faculty write questions

17. 26
18
13
18
11
19
39
19
13
10
6
14
0
5
10
15
20
25
30
35
40
45
> 90% 80-90% 70-80% 60-70% 50-60% <50%
Percentageofcorrectanswers
Difficulty level of question
Percentage of total questions that exam takers got correct
"Hard" Item Writer Average
“Hard” item writer

18. 40
31
13
10
2
7
39
19
13
10
6
14
0
5
10
15
20
25
30
35
40
45
> 90% 80-90% 70-80% 60-70% 50-60% <50%
Percentageofcorrectanswers
Difficulty level of question
Percentage of total questions that exam takers got correct
"Easy" Item Writer Average
“Easy” item writer

19. 40
84
88 90
100
33
64
39
76
79 81
74
39
56
0
20
40
60
80
100
120
>90% 80-90% 70-80% 60-70% 50-60% <50% Total
Percentageofpb>0.2
Difficulty level of question
Percentage of questions with point bi-serial of > 0.2
"Good" Item Writer CAS Faculty Average
Example of good item writer
across all difficulty levels

20. 27
58
73
48
52
38
46
39
76
79
81
74
39
56
0
10
20
30
40
50
60
70
80
90
>90% 80-90% 70-80% 60-70% 50-60% <50% Total
Percentageofpb>0.2
Difficulty level of question
Percentage of questions with point bi-serial of > 0.2
"Poor" Item Writer CAS Faculty Average
Example of poor item writer
across all difficulty levels

21. 22
66
50
75
71
60
68
39
76
79
81
74
39
56
0
10
20
30
40
50
60
70
80
90
>90% 80-90% 70-80% 60-70% 50-60% <50% Total
Percentageofpb>0.2
Difficulty level of question
Percentage of questions with point bi-serial of > 0.2
"Mixed" Item Writer CAS Faculty Average
Example of item writer with bad “easy”
questions and good “hard” questions

22. Example of feedback
Analysis of Data
Your total # of questions: 348
It appears you are still writing more difficult
items than your peers and they are not
performing as well. Item writing should be on
your faculty development plan for this year.

23. HOW TO RETRIEVE DATA IN
EXAMSOFT

26. Total number of items written

27. Determine range of difficulty

28. Determine point bi-serial

29. For this faculty
• 20% of total items range in 80-90% difficulty
• 93% of those questions are >0.2 point bi-
serial

30. LIMITATONS

31. Limitations
 Tag questions appropriately
 Entering questions for another writer
 Limited number of questions for some faculty
 Items that are not analyzed. (Fill in blank,
essay, etc.)
 Mix of summative and formative assessments

32. Future Scorecards
 Tie questions to Blooms Taxonomy
 Faculty self-assessment to create faculty
development plan

33. Questions