This is my research project I completed for my grad school capstone

1. Implementing Online Quizzes
into a Math Course for
Elementary Education Majors
By: Alex Keefover

2. Introduction
• This semester I am working as a technology mentor with the mathematics
education professor at Fairmont State University as part of the requirements to
earn my graduate degree.
• This course teaches them the axioms of the real numbers, among other things,
but students have had trouble memorizing and applying these axioms in the past.
• During fall 2017, the class average for the first nine online quizzes was 76%. The
class average on the tenth quiz, which was comprehensive, was 45%.
• The professor wanted these numbers higher.

3. Reason for the Study and Guiding Question
• If elementary education teacher candidates do not learn these axioms, they run the
risk of not truly comprehending the math they are teaching.
• Online quizzes were implemented in the past to help students with learning the
axioms.
• However, these online quizzes could have been more effective by using different
features that were not being used such as question banks, various types of questions,
multiple attempts, and time limits.
• Thus, this action research will examine the following question: How do different
design features of online quizzes influence undergraduate elementary education
majors' learning of the basic axioms of the real numbers?

4. Literature Review Overview
• Research shows that online quizzes have helped students’ learning because they
give students incentive to study (Johnson & Kiviniemi, 2009; Wickline & Spektor,
2011), reduce anxiety (Brown & Tallon, 2015), and raise overall grades (Martins,
2016).
• Admittedly, there is not a lot of research on online practice quizzes in a
mathematics class. However, there have been quite a few studies on online
practice quizzes in social science courses.
• No research was found that concluded that online practice quizzes are ineffective.

5. The Literature On Best Online Quiz Design Methods
• All the research found online practice quizzes effective but differed on
implementation method.
• All this research offered some guidelines on how online quizzes should be
implemented and designed.
• They all agree on that the quizzes should be designed as formative assessments
meaning that unlimited attempts should be given and immediate feedback should be
given (Johnson & Kiviniemi, 2009; Grimstad & Grabe, 2004; Wickline & Spektor, 2011;
Lowe, 2015).
• Two of the studies suggested that the quizzes should be counted for a grade
(Johnson & Kiviniemi, 2009; Lowe, 2015), two suggested that they should not be
counted for a grade (Grimstad & Grabe, 2004; Wickline & Spektor, 2011), and one
suggested that they should be counted for bonus points (Martins, 2016).

6. Method Introduction
• This research was conducted at Fairmont State University with elementary
education teacher candidates in the Structure of the Real Numbers course.
• Specifically, the research was conducted in the Spring Semester of 2018 with a
class of 18 females and 1 male.
• All the students have passed or tested out of college algebra. Further, all
students had passed the Praxis Core exam, and 15 had already been accepted
into the Ed Program.
• On the Quizzes page of the course’s Blackboard site, students are shown quizzes
they have already taken and quizzes that are currently assigned.
• Only one quiz was assigned at a time.

7. Research Design
• The course gave 4 tests and a final exam. The instructor and I assigned 6 online
quizzes, not including the pre-test and post-test.
• All the online quizzes contained questions like those on the exams.
• The pre- and post-test were the same.
• Throughout the class, online quizzes were given before each test as formative
assessments to help students prepare for the tests.
• As the course progressed, the instructor and I altered the settings on the quizzes to
see how to make them most effective.
• These settings included number of attempts, time limit, number of questions in the
question banks, and number of questions on the quizzes.

8. Designing the Question Pools
• The question pools were created based on the operation (addition or
multiplication) and set (integers, whole numbers, rational numbers, real numbers,
and irrational numbers).
• Inside each pool, five question types were used: fill in multiple blanks, jumbled
sentence, multiple choice, either/or, and multiple answer (multiple choice where
students had to select all the correct answers).
• The questions in each pool were tagged with what level of difficulty they are
(easy, medium, or hard), what property it addresses, and the level of Bloom’s
Revised Taxonomy it addresses (Anderson & Krathwohl, 2001).

9. Sample
Questions
A sample of a jumbled sentence question and a
fill in multiple blanks question.

10. Creating the Question Pools and Quizzes
• For each pool, between 20 and 40 questions were created depending on the properties
addressed.
• Two multiple choice questions were created for each property tagged as easy for the level of
difficulty and remember under Bloom’s Revised Taxonomy (Anderson & Krathwohl, 2001).
• Two fill in multiple blanks questions or two jumbled sentence questions for each property were
created and these were labeled as hard and understand for level of difficulty and Bloom’s
Revised Taxonomy respectively.
• Also, an either/or question that asked the student if a property holds under the set for
subtraction or division was labeled as hard and analyze.
• Finally, five multiple answer questions were created with five possible answers that gave a
statement and asked students to choose all the properties that were exemplified in the
statement. These were labeled as medium and understand.

11. Creating the Question Pools and Quizzes
• The properties that were included in the pool were all and only the properties that satisfied the
set under the operation. The possible properties for each pool were closure, commutative,
associative, identity, inverse, zero (multiplication only), and distributive property of multiplication
over addition.
• Creating the pools with the labels like this was very time consuming but made creating the
quizzes simple.
• The quizzes were made by telling the learning management system from which set of questions
to pick a certain number of random questions.
• For example, the quiz on the integer properties of addition was set up to include one question
that was easy and addresses the associative property, one question that was easy and addresses
the closure property, and the same for the commutative, identity and inverse question. This
went on and on until the quiz contained at least one question for each property and level of
difficulty.

12. Quiz Settings
• The final quiz was created using the same method but used all the pools to choose
the questions from.
• The first quiz allowed two attempts, the last quiz allowed five attempts, and all the
other quizzes allowed three attempts.
• Each quiz had the following settings: the highest score on all attempts was the one
recorded, each attempt had to be completed in one sitting, they were timed, the
entire quiz was presented on one screen, and there was no specific order to the
questions.
• The students were given a week to complete the first and second quiz, five days to
complete the third, fourth, and fifth quiz, and two weeks and two days to complete
the last quiz.

13. Changes to Time Limit on Quizzes 1-4
• As we went through this study, some changes needed to be made.
• For most of the early quizzes, the students took less than half the time limit given
for most of the quizzes, so the time limit was reduced by 5-7 minutes for later
quizzes.
• However, the students tended to express anxiety about the time limit so it was
decided to give them a little more time that was needed to reduce anxiety.

14. A New Question Type
• Quiz five is the quiz where an addition was made.
• While reflecting on the quiz regarding the tests, it was recognized that there was a
question type on the tests that was not being addressed on the quizzes.
• On the tests, there was a problem where an algebraic problem was worked out and
the students had to say which property allowed each step to work.
• There was no quiz question like this thus far.
• So, one question like this was added to quiz five as a jumbled sentence question.
• A picture of this question is on the next slide.

15. A New Question Type
• The question that was added to quiz
5.

16. Changes to Time Limit on Quiz 5
• The original time limit set for quiz five was 15 minutes. However, after taking one
attempt, many students told the instructor that they did not have enough time to
complete the quiz due to the new question. So, five minutes was added to the
time limit.
• The interesting thing is the students only took, on average, 11 minutes to take the
quiz and only six out of 48 attempts took 15 minutes or more. Two possible
explanations of this are that the students just needed to orient to the new
question type then they were fine with it or just knowing there was a time limit
caused anxiety.

17. Original Incorrect Feedback
• The other change that was made was on the feedback that was given when students
answered a question incorrectly.
• On quizzes one through five, this feedback gave page numbers in the book that the
properties were defined on. We also gave the students the possible choices (on every
question except for fill in multiple blank questions) and which questions the students
answered incorrectly.
• We did not give the students what answer they chose. We felt that students should be
paying attention closely enough to remember the answer they chose.
• We also did not give students the correct answer because we wanted them to look it
up.

18. Updated Incorrect Feedback
• When the students were surveyed after quiz five, 63.2% of students either disagreed or strongly
disagreed that the feedback on the questions was helpful.
• So, to combat this, an online glossary was created that defined all the properties, the feedback
given when students answered incorrectly was changed, and the students were given what
answer they chose as feedback on the final quiz.
• It was still not given to the students the correct answer as the instructor and I still wanted the
students to correct their mistakes themselves.
• The incorrect feedback was changed to “Please go to https://studymate.com/?id=3ZgtUMjss
and look at the glossary. Check that: 1. You spelled the property and operation correctly. 2. You
gave the correct operation. 3. You gave the correct property. If you are still having trouble,
please message the instructor before your next attempt.”

19. Pre-Test V.S. Post-Test
• Gains were seen that suggest that
students have learned from the
quizzes. It was not expected that
grades on the post-test would be near
perfect since it was given before the
students completed all the quizzes
and exams. So, seeing this progress
when we did is promising to suggest
the quizzes helped the students learn
the properties.
0
1
2
3
4
5
Pre-Test [Total Pts: 5] Post-Test [Total Pts: 5]
Average Score on Pre-Test and Post-Test

20. Correlation Coefficients of Scores and Averages
• The correlation coefficient between the quiz scores and test scores is 0.36.
• Thus, there is a weak positive correlation between the quiz scores and test scores,
however, the quiz scores are not a strong indicator of test scores.
• The same method was used to find a correlation between quiz averages and test
averages where the correlation coefficient is 0.73, which suggests that the
students’ overall averages on the quizzes for the course may be helpful to
indicate what their overall averages may be on the tests for the course.

21. Quiz Averages V.S. Test Averages
0
20
40
60
80
100
120
20 21 22 23 24 25 26 27 28 29
TestAverages
Quiz Averages
Quiz Averages V.S. Test Averages

22. Correlation Coefficients of Test Average and Part of
Quiz Average
• Next, we recognized that every topic addressed on the tests was not addressed on the
quizzes.
• So, it was important to see if there was a correlation between the quiz scores and the
score only on the part of the test that was addressed by the quizzes.
• Since quizzes one and two prepared students for test one, those scores were averaged
together to use. The same method was used with quizzes three and four and test two.
For test three, quiz five was the only one given after test two to help students prepare
so it did not need averaged with another quiz.
• The three correlation coefficients, in order, are 0.79, 0.55, and 0.13.

23. Last Semester V.S. This Semester
• The last semester class average on the quizzes was 76%. With the data thus far,
this semester’s percentage is up to 92%. This could be explained by other
reasons (different students with different sets of knowledge coming in, et cetera),
but it does suggest that students performed better when the new quiz design was
implemented.

24. Survey Results
Statement Strongly Agree Agree Neither Agree nor
Disagree
Disagree Strongly Disagree Unanswered
In general, the quiz
questions were
easy to understand.
15.789% 68.421% 15.789% 0% 0% 0%
In general, I was
given too little time
to take the quizzes.
15.789% 15.789% 21.053% 31.579% 15.798% 0%
In general, I was
given too much
time to take the
quizzes.
0% 5.263% 21.053% 47.368% 26.316% 0%
In general, the
quizzes were a
good use of my
time to prepare me
for the tests.
31.579% 52.632% 0% 15.789% 0% 0%
The quizzes were
too hard.
0% 10.526% 10.526% 63.158% 15.789% 0%
The number of
attempts given on
the quizzes were
adequate to help
me learn all of the
properties.
52.632% 42.105% 5.263% 0% 0% 0%

25. Survey Results
Statement Strongly Agree Agree Neither Agree nor
Disagree
Disagree Strongly Disagree Unanswered
There were enough
questions that each
time I took a quiz, I
received different
questions.
5.263% 68.421% 5.263% 5.263% 10.526% 5.263%
The quizzes were
spread out enough
over the semester that
I was never
overwhelmed by
learning the
properties.
52.632% 47.368% 0% 0% 0% 0%
The quizzes gave me
an incentive to study
5.263% 63.158% 15.789% 15.789% 0% 0%
Completing the
quizzes reduced my
anxiety when
answering exam
questions on the
properties.
15.789% 36.842% 42.105% 5.263% 0% 0%
Having the quiz count
for a grade was
stressful. I would have
rather had them as
non-graded practice
quizzes.
5.263 10.526% 21.053% 47.368% 15.789% 0%

26. Written Survey Results
• On the survey where these questions were asked, the students were also asked
what they liked and what they did not like about the quizzes.
• Most of what was liked was how much practice it gave them on studying the
properties and the variety of question types.
• Most of what was disliked was the feedback given after an attempt and the time
limit.

27. Original Feedback V.S. Updated Feedback
• The students were surveyed again after the
new feedback with the glossary was
implemented.
• It is interesting to note that one student said
they strongly disagreed with all the
statements in this survey (including that they
had plenty of time to take the latest quiz),
but then in the written response said, “I
really liked the time limit on this quiz, I didn't
feel rushed at all.” So, I wonder if she did
not answer the questions as she intended. 0
10
20
30
40
50
60
70
80
90
100
% of students who found the original
feedback helpful
% of students who found the new feedback
helpful
Satifaction of the Feedback on the Quizzes

28. The Professor’s Thoughts on the Online Quizzes
• "My students this semester seem to be more committed to learning the
properties of real numbers than students in previous semesters. I attribute much
of this to the new quiz format, which ensures that students get some different
questions on each of 3-5 attempts on the quiz. The extra practice has helped
them pay more attention to the details of the properties (including their spelling),
which is evidenced on their in-class tests. I also notice that when I ask them what
property is being represented during class, they seem to respond more quickly
and more correctly than students I've had previously."

29. My Thoughts
• Looking at the pre- and post-test results, I would use online quizzes in my teaching practice to
help students learn.
• The students made significant progress in learning the properties of the real numbers.
• Also, online quizzes appeared to be an appropriate instructional strategy because it was easy to
create questions that could be graded automatically by a learning management system.
• The quiz scores did not prove to be a strong correlation to the test scores but the quiz averages
did prove to be a strong indicator of test averages. So, the professor may want to look at the
quiz averages to see if the students are ready for the tests rather than looking at the quiz scores.

30. Discussion of Correlation Coefficients
• When we look at the trend of the correlation coefficients we notice a downward trend. This may be
because each test was cumulative in regard to the properties but all the quizzes focused on a
particular set of numbers. This would suggest that, in the future, the quizzes should contain questions
from previous quizzes to match the material being added with each test.
• The last correlation coefficient from quiz five to the corresponding section on test three was the lowest
at 0.13. A possible reason that the last coefficient is so low is quiz five was where that new question
type was introduced without preparing the students for it first. In the future, it would be better to
orient the students to how that question works before having them complete the quiz, either by
showing them how to do it in class or having them complete a practice problem like it before the quiz.
This helps me with one of my professional learning goals which is to know how to prevent some issues
students may run into. Another possible reason was only one quiz was given before that test while
two were given before the other test. Thus, more than one quiz may be needed to predict how the
students will do on a test.

31. What I Think Made The New Online Quizzes
Effective
• Last semester’s quizzes did not utilize question pools or multiple attempts. The
implementation of these two features was the major changes done to the quizzes.
I see this as what made the quizzes effective as formative assessments. Thus, to
answer my research question, question pools and multiple attempts are design
features of online quizzes that have a positive effect on students’ performance.

32. Design Features That Had a Negative Impact
• The design features that had a negative impact on the students’ attitude of online
quizzes was the time limit and the original feedback given.
• I now know that when students take online assessments, they want to be able to
correct their mistakes using the online resources and not have to find where they can
correct their mistakes. Therefore, I know I need to keep the feedback as self-
contained as possible. This research found that a glossary is a good way to do this
where students still correct their own mistakes but it is easy for them to access.
• Also, I now know that time limits stress students on online quizzes so I need to make
it clear that when I use a time limit that the quizzes are low-stakes and the students
can message me if they have trouble completing the quiz in the time limit.

33. Future Research
• For the future, I would like to see this study done in multiple courses to have a bigger set of data
points. This research is limited due to it having so few data points.
• Also, I would make some changes to the design of the research.
• First, I would grade the quizzes and tests on the same scale so it would be easier to perform my
statistical analysis.
• Second, I would give an orientation of how to take the quizzes when the classes start to rule out
technical problems as a reason scores were low.
• Third, I would focus more on what feedback is effective and what feedback is not effective.
• Finally, I would use the same number and type of questions on each quiz to make them more uniform.
• These changes would narrow down the variables so the research can focus on the design of online
quizzes.

34. Works Cited
• Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., . . . Wittrock, M. C. (2000). A Taxonomy for Learning,
Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Pearson.
• Brown, M. J., & Tallon, J. (2015). The Effects of Pre-Lecture Quizzes on Test Anxiety and Performance in a Statistics Course. Education, 135(3), pp. 346-
350.
• Grimstad, K., & Grabe, M. (2004, April 1). Are Online Study Questions Beneficial? Teaching of Psychology, 31(2), pp. 143-146.
• Johnson, B. C., & Kiviniemi, M. T. (2009, January 2). The Effect of Online Chapter Quizzes on Exam Performance in an Undergraduate Social
Psychology Course. Teaching of Psychology, 36(1), pp. 33-37.
• Lowe, T. W. (2015, June 26). Online Quizzes for Distance Learning of Mathematics. Teaching Mathematics and Its Applications, pp. 138-148. Retrieved
March 17, 2018
• Martins, S. G. (2016, May 19). Weekly Online Quizzes to a Mathematics Course for Engineering Students. Teaching Mathematics and Its Applications,
pp. 56-63.
• Wickline, V. B., & Spektor, V. G. (2011, March 23). Practice (Rather Than Graded) Quizzes, With Answers, May Increase Introductory Psychology Exam
Performance. Teaching of Psychology, 38(2), pp. 98-101.