formative e-assessment: case stories, design patterns, and future scenarios Caroline Daly, Harvey Mellar, Yishay Mor, Norbert Pachler, Institute of Education, University of London

Overview Scoping study commissioned by JISC Short term, small budget, intended to inform future funding frameworks Established a commited user group of higher-education teachers & researchers Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform

Scoping study commissioned by JISC

Short term, small budget, intended to inform future funding frameworks

Established a commited user group of higher-education teachers & researchers

Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform

Methodology Desk research Literature review Comparing frameworks 5 Practical Enquiry Days Combination of collaborative reflection, report back from team, and guest plenaries Launch day, 3 Planet workshops, developers' day

Desk research

Literature review

Comparing frameworks

5 Practical Enquiry Days

Combination of collaborative reflection, report back from team, and guest plenaries

Launch day, 3 Planet workshops, developers' day

What is formative e-assessment? No consistent view in the literature From “anything test before the final” to “synonymous with learning” The use of digital means to support formative assessment Formative features of assessment, which are afforded by specific features of digital media

No consistent view in the literature

From “anything test before the final” to “synonymous with learning”

The use of digital means to support formative assessment

Formative features of assessment, which are afforded by specific features of digital media

OK, so what is Formative Assessment? “ An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence” (Dylan Wiliam)‏

“ An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence”

(Dylan Wiliam)‏

Formative = feedback + moments of contingency "... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another." (Leahy, Lyon, Thompson, and Wiliam 2005)‏

"... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another."

(Leahy, Lyon, Thompson, and Wiliam 2005)‏

Teacher Learner Peer Instruction Tasks Actions Tasks Actions

Wiliam's 5 stratagies

Conversational Framework (Laurillard)‏

Evidence Centred Design (Mislevy)‏ Highly developed Pattern based Oriented to large scale, automated systems Measurement centric Light on theory Less suitable for open activity designs

Highly developed

Pattern based

Oriented to large scale, automated systems

Measurement centric

Light on theory

Less suitable for open activity designs

A few cases Creature of the week CoMo Post 16 String Comparison Open Mentor ...

Creature of the week

CoMo

Post 16 String Comparison

Open Mentor

...

Creature of the week (Judy Robertson)‏ Situation large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life. Task Engage and motivate the students show examples of good work which others could learn from show students their work is valued. build a sense of community. http://purl.org/planet/Cases/creatureoftheweek

Situation

large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life.

Task

Engage and motivate the students

show examples of good work which others could learn from

show students their work is valued.

build a sense of community.

CoMo (Niall Winters, Yishay Mor)‏ Situation Royal Vet College. Hospital rotations as part of their training. Task Allow students to capture critical incidents in text and image. Support sharing of clinical experiences and co-reflection. http://purl.org/planet/Cases/CoMo

Situation

Royal Vet College.

Hospital rotations as part of their training.

Task

Allow students to capture critical incidents in text and image.

Support sharing of clinical experiences and co-reflection.

Post 16 string comparison (Aliy Fowler)‏ Situation Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students. Sixth Form level, grammatical consolidation and whole-sentence translation. Task Allow students to practise written language independently and receive feedback on errors in order to improve their language skills. http://purl.org/planet/Cases/Post16stringcomparison

Situation

Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students.

Sixth Form level, grammatical consolidation and whole-sentence translation.

Task

Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.

Solution A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.

A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.

Open mentor (Denise Whitelock)‏ http://purl.org/planet/Cases/OpenMentor

A few patterns.. Try Once, Refine Once Feedback on Feedback Classroom display

Try Once, Refine Once

Feedback on Feedback

Classroom display

Try Once, Refine Once (Aliy Fowler)‏ http://pul.org/planet/Patterns/TryOnceRefineOnce

Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.

Context Class size Large (30-300)‏ Content Skills facts Mode of instruction Blended / on-line. Computer tested.

Class size

Large (30-300)‏

Content

Skills facts

Mode of instruction

Blended / on-line. Computer tested.

Solution

Feedback on Feedback (Linda McGuigan)‏ http://purl.org/planet/Patterns/FeedbackonFeedback

Problem Good feedback should - Alert learners to their weaknesses. Diagnose the causes and dynamics of these. Include operational suggestions to improve the learning experience. Address socio-emotive factors. Tutors know this, but are pressed for time. Or not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching

Good feedback should -

Alert learners to their weaknesses.

Diagnose the causes and dynamics of these.

Include operational suggestions to improve the learning experience.

Address socio-emotive factors.

Context Large scale, technology supported, graded courses many tutors instructing many students. Feedback is mediated by technology that allows it to be captured and processed in real time Topic of study is subject to both grading and formative feedback.

Large scale, technology supported, graded courses

many tutors instructing many students.

Feedback is mediated by technology that allows it to be captured and processed in real time

Topic of study is subject to both grading and formative feedback.

Solution Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.

Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms.

In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.

Classroom Display http://purl.org/planet/Patterns/Classroomdisplay

Problem Rewards participation. Relates to learner's personal experiences. Window on student conceptions . Using learner generated content.. Needs to collate works in a single easy to access location. Learners uncomfortable about presenting their work in public Legal or other restrictions on sharing work.

Rewards participation.

Relates to learner's personal experiences.

Window on student conceptions .

Needs to collate works in a single easy to access location.

Learners uncomfortable about presenting their work in public

Legal or other restrictions on sharing work.

Context Class size: Small / medium (6-60)‏ Mode of instruction: Blended (preferable)‏ Time frame Continuous, over a period Pedagogy Involves construction / media production

Class size:

Small / medium (6-60)‏

Mode of instruction:

Blended (preferable)‏

Time frame

Continuous, over a period

Pedagogy

Involves construction / media production

Solution

Augmented domain map

Example scenario When using Try Once Refine Once , there is a risk that high-achievers do not receive feedback. So - Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors. Use Feedback on Feedback to alert tutors to the problem.

When using Try Once Refine Once , there is a risk that high-achievers do not receive feedback.

So -

Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors.

Use Feedback on Feedback to alert tutors to the problem.

Conclusions Tip of the iceberg Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios Collaborative elicitation of patterns from cases could be a potent form of professional development.

Tip of the iceberg

Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios

Collaborative elicitation of patterns from cases could be a potent form of professional development.

Thank you The Formative e-Assessment project: http://projects.lkl.ac.uk/feasst Final report http://telearn.noe-kaleidoscope.org/open-archive/browse?resource=1875 Yishay Mor http://people.lkl.ac.uk/yishay yishaym@gmail.com This presentation http://www.slideshare.net/yish/formative-eassessment-cases-patterns-and-scenarios