MetaMath: Evaluation Methodology

1. Sergey Sosnovsky, Saarland University / DFKI
MetaMath: Evaluation
Methodology
Tampere, 19-21/01/2015

2. Evaluation Goal
❖ Implement large scale evaluation trials of the reformed courses
to demonstrate that the reformed courses bring educational
value
❖ Improved grades (higher average grade, lower percentage of students
failing the course,…)
❖ Higher knowledge gain (on average, for underachieving students, for
female students, for CS students…)
❖ Lower drop-out rates (how to measure?)
❖ Increased motivation, improved attitude toward learning mathematics
❖ Positive evaluation by students (some students… of some features)
Which of these parameters
should we collect? All of them

3. Overall experiment design
❖ Longitudinal study
❖ Two groups: Group1 (controlled) and
Group2(experimental)
❖ Each group is observed for one semester
❖ Group1 - in 2015, Group2 - in 2016
❖ Group1 is taught with the old version of the course
❖ Group2 is taught with the new version of the course
❖ Main challenge: inter-group differences as uncontrolled
factors that would influence the results

4. Assessment methodology

5. Pre /post-tests
❖ SEFI competencies should be framework for developing tests
questions.
❖ Isomorphism of questions can be achieved by using different
numerical values (avoid complex calculations that would
increase the complexity without influencing difficulty)
❖ Test must assess both the target competencies and prerequisite
competencies
Issues:
- electronic or paper?
- when to organise?
- how long?
- what is the nature of questions?
- … then they will be graded

6. Pre-questionnaire
❖ Demographics
❖ Gender
❖ Major/speciality
❖ Background
❖ experience with e-Learning
❖ lifetime Math experience
❖ Motivation
❖ Interest in math
❖ Fear of failure
❖ Extrinsic valut
❖ Expectations from the course (amount of learning, need for future studies/
career personal grade prediction)
Challenge:
- cannot be anonymises
Issues:
- electronic or paper?
- when to organise?

7. Post-questionnaire
❖ Motivation
❖ Interest in math
❖ Fear of failure
❖ Extrinsic valut
❖ How expectations from the course have been met
❖ Which aspect of the course they like/ did not like
❖ …standard course evaluation forms
❖ (during the 2nd semester): Math-Bridge-related
questionnaire
Challenge:
- ensure that students fill
it in
Issues:
- electronic or paper?
- when to organise?