Try once, refine once (a pattern for formative e-assessment) Aliy Fowler University of Kent 28-04-09

Pattern summary A two-step approach to assessment/question answering/problem solving Encourages students to carefully consider initial answers Then, having received feedback, to give equal consideration to refining/correcting

A two-step approach to assessment/question answering/problem solving

Encourages students to carefully consider initial answers

Then, having received feedback, to give equal consideration to refining/correcting

The initial problem (source case) A University ab-initio Spanish module Large student numbers Skills-based course Provision of sufficient formative assessment meant unmanageable marking loads Impossible to provide immediate feedback leading to fossilisation of errors

A University ab-initio Spanish module

Large student numbers

Skills-based course

Provision of sufficient formative assessment meant unmanageable marking loads

Impossible to provide immediate feedback

leading to fossilisation of errors

The solution (source case) A CALL system designed to enable students to: Independently practise sentence translation Receive immediate (and robust) feedback on all errors Attend immediately to the feedback (before fossilisation can occur)

A CALL system designed to enable students to:

Independently practise sentence translation

Receive immediate (and robust) feedback on all errors

Attend immediately to the feedback (before fossilisation can occur)

The Pattern

Wider applicability Any skills-based learning situation where: Multiple errors/misconceptions are possible Feedback can be given which identifies the location/type of errors without revealing the correct answer(s) Feedback generation (and grading?) can be automated otherwise the resubmission element will contribute to staff overloading it needs to be E-assessable!

Any skills-based learning situation where:

Multiple errors/misconceptions are possible

Feedback can be given which identifies the location/type of errors

without revealing the correct answer(s)

Feedback generation (and grading?) can be automated

otherwise the resubmission element will contribute to staff overloading

it needs to be E-assessable!

Wider applicability Possible subject areas Mathematics Logic Computer Science: algorithms/programming databases mark-up languages CSS MFL

Possible subject areas

Mathematics

Logic

Computer Science:

algorithms/programming

databases

mark-up languages

CSS

MFL

Some theoretical justification Ferreira & Atkinson (2009) divide feedback strategies for language learning into: GAS (Given-Answer Strategies) target forms corresponding to students’ errors are given PAS (Prompting-Answer Strategies) students pushed to notice errors in their responses and repair the errors for themselves Found that in a tutorial context PAS seemed to promote more constructive student learning

Ferreira & Atkinson (2009) divide feedback strategies for language learning into:

GAS (Given-Answer Strategies)

target forms corresponding to students’ errors are given

PAS (Prompting-Answer Strategies)

students pushed to notice errors in their responses and repair the errors for themselves

Found that in a tutorial context PAS seemed to promote more constructive student learning

Some theoretical justification Nicol & Macfarlane (2006) maintain that good feedback practice includes: Activities which encourage reflection on both the processes and products of learning Providing opportunities to close the gap between current and desired performance Providing opportunities to repeat the same ‘task-performance-feedback’ cycle for example by allowing resubmission

Nicol & Macfarlane (2006) maintain that good feedback practice includes:

Activities which encourage reflection on both the processes and products of learning

Providing opportunities to close the gap between current and desired performance

Providing opportunities to repeat the same ‘task-performance-feedback’ cycle

for example by allowing resubmission

Some theoretical justification Sadler (1989) Can only tell whether learning results from feedback if students have the opportunity to act on the feedback Boud (2000) Unless students can use feed- back to produce improved work, neither they nor their teachers can gauge its efficacy

Sadler (1989)

Can only tell whether learning results from feedback if students have the opportunity to act on the feedback

Boud (2000)

Unless students can use feed- back to produce improved work, neither they nor their teachers can gauge its efficacy

How is the final mark arrived at? The two submissions are un equally weighted Best to give more weight to the first attempt since this ensures that students give careful consideration to the construction of their first answer but can improve their mark by refining the answer The marks ratio can vary (depending on assessment/feedback type) the more information given in the feedback, the lower the weight the second mark should carry

The two submissions are un equally weighted

Best to give more weight to the first attempt

since this ensures that students give careful consideration to the construction of their first answer

but can improve their mark by refining the answer

The marks ratio can vary (depending on assessment/feedback type)

the more information given in the feedback, the lower the weight the second mark should carry

How is the final mark arrived at? If the ratio is skewed too far in favour of the first attempt… students are less inclined to try hard to correct non-perfect answers If the ratio is skewed too far in favour of the second attempt… students exhibit less care over the construction of their initial answer

If the ratio is skewed too far in favour of the first attempt…

students are less inclined to try hard to correct non-perfect answers

If the ratio is skewed too far in favour of the second attempt…

students exhibit less care over the construction of their initial answer

Why “try once , refine once ”? The resubmission limit is important Prevents a “mindless” iterative approach to solving the problem In which students begin with a “stab-in-the-dark” Then proceed by allowing the system/tutor to guide them step-by-step to the correct answer often via numerous minimally altered attempts with little critical engagement

The resubmission limit is important

Prevents a “mindless” iterative approach to solving the problem

In which students begin with a “stab-in-the-dark”

Then proceed by allowing the system/tutor to guide them step-by-step to the correct answer

often via numerous minimally altered attempts

with little critical engagement

More theoretical justification Looking at the teaching of programming: Turkle & Papert (1990) Used the term bricolage to refer to the “try it and see” approach Deemed it a valid alternative to the “planning” approach However... Ben-Ari (2001) Bricolage is not “an effective epistemology for dealing with the massive amount of detailed knowledge must be constructed and organized in levels of abstraction” [sic]

Looking at the teaching of programming:

Turkle & Papert (1990)

Used the term bricolage to refer to the “try it and see” approach

Deemed it a valid alternative to the “planning” approach

However... Ben-Ari (2001)

Bricolage is not “an effective epistemology for dealing with the massive amount of detailed knowledge must be constructed and organized in levels of abstraction” [sic]

More theoretical justification Researching automatic feedback and resubmissions in Computer Science Malmi & Korhonen (2004) Found results indicating that allowing high or unlimited numbers of resubmissions discouraged “active pondering”: learners do not concentrate on finding the errors in their programs on their own. they use the automatic assessment system as a kind of debugger: “Try something and look at if it works”

Researching automatic feedback and resubmissions in Computer Science

Malmi & Korhonen (2004)

Found results indicating that allowing high or unlimited numbers of resubmissions discouraged “active pondering”:

learners do not concentrate on finding the errors in their programs on their own.

they use the automatic assessment system as a kind of debugger: “Try something and look at if it works”

More theoretical justification And in a follow-up paper Malmi & Korhonen (2005) noted: When multiple submissions were permitted, about 10% of students spent an unreasonable amount of time on exercises when measured against their success in the examination

And in a follow-up paper Malmi & Korhonen (2005) noted:

When multiple submissions were permitted, about 10% of students spent an unreasonable amount of time on exercises

when measured against their success in the examination

More theoretical justification Hattie & Timperley (2007) Receptivity to feedback can be affected by the degree of confidence students have in the correctness of their responses Kulhavy & Stock (1989) Feedback has its greatest effect when a learner expects his/her response to be correct and it turns out to be wrong since the learner will study the item more intently in order to correct the misconception

Hattie & Timperley (2007)

Receptivity to feedback can be affected by the degree of confidence students have in the correctness of their responses

Kulhavy & Stock (1989)

Feedback has its greatest effect when a learner expects his/her response to be correct and it turns out to be wrong

since the learner will study the item more intently in order to correct the misconception

More theoretical justification With the “Try once, refine once” pattern a higher proportion of the marks are given for the first attempt So students are likely to give initial answers in which they have a considerable degree of confidence Thus if an answer is found to be incorrect, it is then that feedback will be most effective

With the “Try once, refine once” pattern a higher proportion of the marks are given for the first attempt

So students are likely to give initial answers in which they have a considerable degree of confidence

Thus if an answer is found to be incorrect, it is then that feedback will be most effective

References Ben-Ari, M. (2001). Constructivism in computer science education. Journal of Computers in Mathematics and Science Teaching, 20 (1), 45–73. Boud, D. (2000). Sustainable assessment: rethinking assessment for the learning society. Studies in Continuing Education, 22 (2), 151-167. Ferreira, A. & Atkinson, J. (2009). Designing a feedback component of an intelligent tutoring system for foreign language. Knowledge Based Systems , doi:10.1016/j.knosys.2008.10.012 Hattie, J. & Timperley, H. (2007). The power of feedback. Review of Educational Research , 77 , 81-112 Kulhavy, R.W. & Stock, W.A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review , 1(4), 279–308.

Ben-Ari, M. (2001). Constructivism in computer science education. Journal of Computers in Mathematics and Science Teaching, 20 (1), 45–73.

Boud, D. (2000). Sustainable assessment: rethinking assessment for the learning society. Studies in Continuing Education, 22 (2), 151-167.

Ferreira, A. & Atkinson, J. (2009). Designing a feedback component of an intelligent tutoring system for foreign language. Knowledge Based Systems , doi:10.1016/j.knosys.2008.10.012

Hattie, J. & Timperley, H. (2007). The power of feedback. Review of Educational Research , 77 , 81-112

Kulhavy, R.W. & Stock, W.A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review , 1(4), 279–308.

References Malmi, L. & Korhonen, A. (2004). Automatic Feedback and Resubmissions as Learning Aid, Proceedings of the IEEE International Conference on Advanced Learning Technologies , ICALT’04 , 186-190 Malmi, L., Karavirta, V., Korhonen, A. & Nikander, J. (2005). Experiences on automatically assessed algorithm simulation exercises with different resubmission policies, ACM Journal of Educational Resources in Computing, 5 (3), http://doi.acm.org/10.1145/1163405.1163412 Nicol, D.J. & Macfarlane-Dick, D. (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31 (2), 199-218. Sadler, D.R. (1989). Formative assessment and the design of instructional systems. Instructional Science . 18 (2), 119-144. Turkle, S. & Papert, S. (1990). Epistemological pluralism: Styles and cultures within the computer culture. Signs: Journal of Women in Culture and Society , 16 (1), 128-148.

Malmi, L. & Korhonen, A. (2004). Automatic Feedback and Resubmissions as Learning Aid, Proceedings of the IEEE International Conference on Advanced Learning Technologies , ICALT’04 , 186-190

Malmi, L., Karavirta, V., Korhonen, A. & Nikander, J. (2005). Experiences on automatically assessed algorithm simulation exercises with different resubmission policies, ACM Journal of Educational Resources in Computing, 5 (3), http://doi.acm.org/10.1145/1163405.1163412

Nicol, D.J. & Macfarlane-Dick, D. (2006), Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31 (2), 199-218.

Sadler, D.R. (1989). Formative assessment and the design of instructional systems. Instructional Science . 18 (2), 119-144.

Turkle, S. & Papert, S. (1990). Epistemological pluralism: Styles and cultures within the computer culture. Signs: Journal of Women in Culture and Society , 16 (1), 128-148.