Toby Bailey's slides from the LTKB workshop 2011 - exploring use of MapleTA for assessment in mathematics

1. TA in Edinburgh
Toby Bailey
June 23, 2011

2. Maple TA
Maple TA is a system for e-assessment particularly aimed at
Mathematics.
It allows questions with numbers or formulas as answers,
not just multiple choice.
It allows algorithmically generated questions, so every
student gets a different example with the same underlying
pattern.
It uses Maple (a computer algebra system) as an underlying
engine for both the algorithmic generation, generation of
feedback and marking.

3. Some questions

4. Some questions

5. Some questions

6. Some questions

7. Some questions

8. Some questions

9. Algorithmically generated questions
Each student getting their own version of a question has
advantages:
Collaboration has to be more active. (Need to understand well
enough to change the numbers.)
It makes it more reasonable to allow multiple attempts at a
test.
One question provides lots of versions for practice.
One can (and we do) write feedback in the system using the
actual parameters of the students questions.
Reasonable to allow students to print out a copy of their
test and come back later to answer it.
But not all things you want to test (particularly outside
mathematics) are amenable.

10. The Maple engine
For maths purposes the ability to do advanced computer algebra is
very important.
Grading the answer to “Evaluate 3(x − 1)2 dx”
(x − 1)3 + C and x 3 − 3x 2 + 3x + C both correct.
Let S equal simplify( d/dx(students answer) - 3(x − 1)2 )
If this is non-zero then 0 marks
Else, if students answer contains precisely 2 indeterminates
then full marks
Else, if students answer contains one indeterminate then full
marks −1 (forgot constant of integration).
Also enables one to be much smarter about setting questions.

11. How have we used it?
Around half of the “weekly hand-ins” for Y1 maths for
sci/eng for the last 3 years.
10-point Y1 Calculus assessed entirely by 4 TA tests.
Each test can be taken multiple times before a final “due
date”.
A very different assessment dynamic from the “on-off exam”.
In Studio tutorials.
Group needs to agree an answer before entering it. “Buzz”
from instant response.

12. Course results
Average of best 5 of 6 TA assessments. Average of best 3 of
4 written assessments.
Exam mean ≈ 60. Both forms of CA mean ≈ 70.
TA correlates better with exam (0.7 versus 0.6).
Fewer students have very high TA scores but poor exam.

13. Reflections
It’s tempting to think of e-assessment as just a labour-saving
partial alternative to conventional CA but:
A big weakness of the “usual” cycle of lecture - problems -
hand-in - mark - return is that by the time the feedback
arrives the course has typically moved on.
Research suggests that attempt followed by immediate,
specific feedback and a new similar problem should be more
effective.
E-asessment provides a mechanism for implimenting what should
be a rather better learning model.

14. Negatives
Students may perceive e-assessment as a money-saving
exercise.
Have to sell it to students as something in addition to the
finite human resources that we have available.
Students find it unforgiving: “I just left out a few brackets
and it gave me zero marks.”
Question authoring is very time-consuming.

15. The Future
E-assessment is here to stay, but I am a long way from
understanding the “dynamics” of e-assessment.
How does repeated attmpts at (algorithmically generated)
e-assessment differ from the “usual exam system”?
Should we have infinitely repeatable practice tests and one-off
assessed tests?
Can we make doing the same question 10 times with different
numbers appealing?
Can we achieve novel things with e-assessment - e.g. “Just In
Time Teaching”
We give credit for what ought to be formative assessment.
The “chocolate cake and fruit salad problem”.
And in future, how do we balance TA with use of the systems that
come with textbooks?