Evaluation Methodology

1. Sergey Sosnovsky, Saarland University / DFKI
Evaluation
Methodology
Batumi, 05/06/2015

2. Evaluation Goal
❖ Implement large scale evaluation trials of the reformed courses
to demonstrate that the reformed courses bring educational
value
❖ Improved grades (higher average grade, lower percentage of students
failing the course,…)
❖ Higher knowledge gain (on average, for underachieving students, for
female students, for CS students…)
❖ Lower drop-out rates (how to measure?)
❖ Increased motivation, improved attitude toward learning mathematics
❖ Positive evaluation by students (some students… of some features)
Which of these parameters
should we collect? All of them

3. Overall experiment design
❖ Longitudinal study
❖ Two groups: Group1 (controlled) and Group2
(experimental)
❖ Each group is observed for one semester
❖ Group1 - in 2015, Group2 - in 2016
❖ Group1 is taught with the old version of the course
❖ Group2 is taught with the new version of the course
❖ Main challenge: inter-group differences as uncontrolled
factors that would influence the results

4. End of the
project
Large-scale Evaluation Plan
now
Fall, 2015 Fall, 2016
teach the selected
course in the
traditional way
teach modified
versions of the
selected course
implement updated
courses within
Math-Bridge
compare student
performance, analyse the
results and verify the
impact of the modified
courses

5. Evaluation Plan (2015)
Demographic
+ Motivation
Questionnaire
Pre-test
Motivation,
& Affect Post-
Questionnaire
Post-test
Exam
Assign
Login
Fall, 2015

6. Evaluation Plan (2016)
Demographic
+ Motivation
Questionnaire
Pre-test
Math-Bridge
Introduction
Motivation,
Affect &
Math-Bridge
Post-
Questionnaire
Post-test
Exam
Assign
Login
Fall, 2016

7. Pre /post-tests
❖ SEFI competencies should be used as a framework for
developing tests questions.
❖ Isomorphism of questions can be achieved by using different
numerical values (avoid complex calculations that would
increase the complexity without influencing difficulty)
❖ Test must assess both the target competencies and prerequisite
competencies
Issues:
- Math-Bridge or paper? (depends on a course)
- how long? (20-40 min.)
- what is the nature of questions? (SEFI)

8. Assessment methodology

9. Pre-questionnaire
❖ Demographics
❖ Gender
❖ Major/speciality
❖ Background
❖ experience with e-Learning
❖ lifetime Math experience
❖ Motivation
❖ Interest in math
❖ Fear of failure
❖ Extrinsic value
❖ Expectations from the course (amount of learning, need for future studies/
career personal grade prediction)

10. Post-questionnaire
❖ Motivation
❖ Interest in math
❖ Fear of failure
❖ Extrinsic value
❖ How expectations from the course have been met
❖ Which aspect of the course they like/ did not like
❖ …standard course evaluation forms
❖ (during the 2nd semester): Math-Bridge-related
questionnaire