Computer aided assessment (CAA) uses computer algebra systems to automatically mark mathematical work, allowing for immediate feedback. It can check student answers algebraically for equivalence rather than just matching answers. This addresses issues with multiple choice questions. Well-designed CAA questions can test for conceptual understanding and properties of functions. The system provides data on student misconceptions to inform feedback. Authoring questions requires balancing expressive power and ease of creation.

1. Computer Aided Assessment
(CAA) for mathematics
Chris Sangwin & Simon Hammond
Copyright c
Last Revision Date: June 1, 2009

2. 2
Introduction.
... NOT multiple choice questions ...
• Computer aided assessment (CAA)
• CAA with computer algebra
• Practical issues
Implementations
• Pedagogical issues
• Future directions

3. 3
JEM - Joining Educational Mathematics
eContentPlus Thematic Network
http://jem-thematic.net/
Founder members (15):
Universitat Politecnica de Catalunya, Helsingin Yliopisto, Tech-
nical University, Jacobs University, Universiteit van Amster-
dam, University of Birmingham, FernUniversitt Hagen, Maths
for More, NAG Ltd, Liguori Editore, ISN Oldenburg GmbH,
RWTH Aachen University, Univ. Nacional de Educacin a Dis-
tancia, Universitat Oberta de Catalunya, Universidade de Lis-
boa.

4. 4
Use of objective tests
Consider the following question:
Example question 1
Determine the following integral:
cos(x) sin(2x)dx.
As a multiple choice question:
◦ (2/3) cos3 (x) + C
◦ −(2/3) cos(x) + (2/3) sin3 (x) + C
◦ −(2/3) cos(x) + (1/3) sin(x) sin(2x) + C
◦ Don’t know.
How do we know the students don’t differentiate the
candidate solutions to check?

5. 5
Computer algebra marking
Computer algebra systems can be used to mark work.
This checks for algebraic equivalence.
(x + 1)2 ≡ x2 + 2x + 1
Useful for marking many routine problems.

6. 6
Fundamental idea
if simplify(sa-ta) = 0 then
mark := 1 else mark := 0

7. 7
STACK
System overview
The STACK system:
• internet based CAA system,
• uses very simple
Maxima (computer algebra), and
L TEX (type setting)
A
• All components open source (e.g. GPL).

8. 8
Demonstrating the STACK system
http://www.stack.bham.ac.uk/

9. 9
In learning and teaching
We are assessing a student provided answer.
This is an objective test.
This is
• not Multiple Choice Question;
• not string/regex match.
Other tests for the form of an answer.

10. 10
Input of mathematics
This is a fundamental but unsolved problem.
There are a number of options
1. Strict CAS syntax. eg. 2*(x-1)*(x+1)
2. “informal” linear text syntax. eg. 2(x-1)(x+1)
x(t-1) ?
3. Graphical input tool. eg. equation editor.
4. (Pen-based input ?)
5. (Geometry applet ?)
Not all groups of students are equal.

11. 11
Syntax innovations
Difficult to achieve!
Babbage 1830’s
“a profusion of notations [...] which threaten, if not duly cor-
rected, to multiply our difficulties instead of promoting our progress”
Babbage, C. (1827)
sin2 (x) sin−1 (x)
sin sin x = sin2 x
(composition)

12. 12
Structure in random problem sets
In practice, the numbers often do not matter.
Tuckey, C. O., Examples in Algebra, Bell & Sons, London, (1904)
Too much randomization destroys structure.
An underlying question space.

13. 13
Workshop task
Option A:
Context: end of first calculus course. (Age 18)
Write 6 questions which test whether a student can differentiate
elementary functions.
E.g. Differentiate cos(3x) with respect to x.
Option B:
Context: age 11.
Write 6 questions which test whether a student can add frac-
tions.

14. 14
Randomization
1. What could you randomize?
2. What would you randomize?
3. What are some likely incorrect answers?
4. What feedback would you like to provide?
... with a view to implementing these questions live.

15. 15
Issues
• Well-posed questions.
• Fair questions.
• Structure in question sets.
Schemes of work, vs isolated questions.
• Algebraic form of answers as a goal.

16. 16
Feedback
One third of feedback interventions decreased performance.
Kluger, A. N. and DeNisi, A., Psychological Bulletin (1996).
The nature of feedback determines its effectiveness.

17. 17
Processing answers
Test for algebraic equivalence if simplify(sa-ta) = 0 then
mark := 1 else mark := 0
Using mainstream CAS
• Get a lot very quickly,
Great for calculus and beyond.
• Elementary algebra can be a problem.
Maxima seems to be more suitable than most.

18. 18
Every CAS is different!
Input Maple Maxima Axiom
(numbers)
0.5-1/2 0.0
√ 0.0 0.0
4^(1/2) √4 2 2
1 1 1
4^(-1/2) 4
√
4 2 √2
-4^(1/2) −4 2i 2√−1
sqrt(-4) 2i 2i 2 −1
(indices)
a^n*b*a^m an bam an+m b bam an
(a^(1/2))^2 √a a √a
(a^2)^(1/2) a2 |a| a2
(collecting terms)
1+x^2-2*x x2 − 2x + 1 x2 − 2x + 1 x2 − 2x + 1
x/3+1.5*x+1/3 1.833x + .333 · · · 1.833x + 1
3
1.833x + 0.333 · · ·
5 5x 5
3*x/4+x/12 6
x 6 6
x
5 1 5 5
3/(4*x)+1/(12*x) 6 x 6x 6x

19. 19
Input Maple Maxima Axiom
(brackets)
-1*(x+3) −x − 3 −x − 3 −x − 3
2*(x+3) 2x + 6 2(x + 3) 2x + 6
2x−1
(2*x-1)/5+(x+3)/2 9
10
x + 13
10 5
+ x+3
2
9
10
13
x + 10
(x-1)^3/(x-1) (x − 1)2 (x − 1)2 x2 − 2x + 1
x2 −2x+1 x2 −2x+1
(x^2-2*x+1)/(x-1) x−1 x−1
x−1
1 9x2 +3x 9x2 +3x
(9*x^2+3*x)/(3*x) 3 x 3x
3x + 1
(other)
log(x^2) ln x2 2 log(x) log x2
log(x^y) ln (xy ) y log(x) log (xy )
log(exp(x)) ln (ex ) x x
exp(log(x)) x x x
cos(-x) cos(x) cos(x) cos(x)

20. 20
Issue: technical problems
• Mixed data types in polynomials
x/3 + 0.5?
• Unary minus (no simplification).
1 −1 1
− , , or .
1−x 1−x x−1
• Display,
1. Implicit multiplication, (xy, x · y, x × y)
2. i vs j,
√ 1
3. x vs x 2 .

21. 21
Language
Do we have a way to talk about these fine details?
Unhelpful phrases:
• simplify,
1 1000
e.g.22 = 4 or 22 = · · ·?
• “move over”

22. 22
Checking for properties
To mark
Example question 2
Give an odd function.
1. calculate f (x) + f (−x),
2. simplify,
3. check equality to zero.

23. 23
Creating examples/instances
Some questions ask for examples of objects.
They require higher level thinking.
Such questions are rare. (11.5 questions from 486 ≈ 2.4%)
Pointon and Sangwin, 2003
Perhaps because they are time consuming to mark.
STACK may mark some questions of this style.
Exemplar questions

24. 24
Students’ answers
Students show great variety in their answer, and method.
For example, 190 students were asked for two functions that
satisfy f (1) = 0.
Their answers were marked automatically.

25. 25
The students (N = 190) gave 93 ‘different’ answers.
Frequency
Frequency
1st Answer 2nd Answer
x2 − 2x 45 x3 − 3x 29
x2
2 −x 31 x2 − 2x 10
x3 x3
3 −x 11 3 −x 9
x2 − 2x + 1 7 (x − 1)2 8
x4
x2 − 2x + 3 7 4 −x 8
(x − 1)2 5 x4 − 4x 5
2x2 − 4x 5 ex−1 − x 1
x3 x2
3 − 2 5 ex−1 + e−x+1 1
0 4 ln(x) − x 1

26. 26
Two strategies emerged:
JL: Ok, just take the parabola and shift it one.
···
B: I said, x − 1 = 0, then integrated it.

27. 27
These problems can be used to generate (short) discussions.
• sorting the data,
• methods used,
• ‘exotic’ examples.
−1
f1 (x) = 0, f2 (x) = |x|(x − 2), f3 (x) = e (x−1)2 .

28. 28
Automatic feedback
Sophisticated automatic feedback may be provided by
computer algebra systems.
This
• is immediate,
• is based on properties of students’ answers,
• could be positive and encouraging,
• may be based on common mistakes,
• may be based on common misconceptions.

29. 29
Common misconceptions
Computer algebra can also test for a type of incorrect answer.
Misconceptions may be identified by
• educational research,
• previous teaching experience,
• examining answers from previous students

30. 30
Odd functions
On examining the odd functions given by students,
the majority of coefficients (= 1) are odd,
eg
3x5 , 5x7 , 7x5 − 3x.
Students’ concept image of an odd function requires odd coeffi-
cients.
Furthermore, f (x) = 0 is odd, but was absent.

31. 31
Functions that are odd and even.
When asked for a function that was both odd and even
35% gave the correct answer (eventually),
35% failed to answer the question.
Incorrect answers revealed that 24% of the students added an
odd and even function.
Examples include
x + x2 , x2 + x3 , x5 − x6 .
The computer algebra system can test for these misconceptions.

32. 32
Student feedback
What do you like about the system? Did you have any difficul-
ties? If so please describe them.
Feedback & partial credit
i like the way that you are given credit if your an-
swer is partially correct and also given guidance on
achieving the full mark for that question.
I like the fact that feedback is immediate, but I do
not like the fact that if I get an answer wrong I do
not know where in my working I have made the error

33. 33
Random questions
The questions are of the same style and want the
same things but they are subtly different which means
you can talk to a friend about a certain question but
they cannot do it for you. You have to work it all
out for yourself which is good.
Syntax problems
I feel the aim system is reasonably fair, however i
have lost a lot of marks in quiz 3 for simple syntax
errors

34. 34
Give me an example...
Recognising the turning points of the functions pro-
duced in question 2 was impressive, as there are a
lot of functions with stationary points at x=1 and
it would be difficult to simply input all possibilities
to be recognised as answers.

35. 35
Authoring questions
In authoring, there is tension:
1. Ability to use all features of CAS.
2. Ease of writing questions.
Not making question authors into programmers.

36. 36
Conclusion
Some important questions
• For what purposes is this tool useful?
• What properties do we want?
– Not “looks correct”.
– Not “select the correct answer”.
• What feedback should we give?