Formative e-assessment: case stories, design patterns, and future scenarios Norbert Pachler, Caroline Daly, Harvey Mellar, Yishay Mor Institute of Education, University of London

Overview Scoping study commissioned by JISC Short term, small budget, intended to inform future funding frameworks Established a committed user group of higher-education teachers & researchers Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform

Scoping study commissioned by JISC

Short term, small budget, intended to inform future funding frameworks

Established a committed user group of higher-education teachers & researchers

Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform

Methodology Desk research Literature review Frameworks in which to situate formative e-assessment 5 Practical Enquiry Days Combination of collaborative reflection, report back from team, and guest plenaries Launch day, 3 Planet workshops, developers' day

Desk research

Literature review

Frameworks in which to situate formative e-assessment

5 Practical Enquiry Days

Combination of collaborative reflection, report back from team, and guest plenaries

Launch day, 3 Planet workshops, developers' day

t o develop a domain map for formative e-assessment to c arry out a review of relevant literature t o delineate a set of key processes involved in effective formative e-assessment practice through case studies and patterns t o scope a vision for formative e-assessment t o make recommendations for practice and policy making Project aims

t o develop a domain map for formative e-assessment

to c arry out a review of relevant literature

t o delineate a set of key processes involved in effective formative e-assessment practice through case studies and patterns

t o scope a vision for formative e-assessment

t o make recommendations for practice and policy making

Issues from the literature – consensus? Formative (e)-assessment is concerned with learners making progress towards measurable attributes/knowledge/skills/understanding It is about working with mechanisms and practices which allow the gap to close between what they are currently able to achieve and what they might be able to achieve Increasing learners’ active responsibility for their part in the learning process is a main feature across a range of contexts and technologies … but not much further consensus…differences tend to focus around whether ‘assessment’ is treated as an ‘event’ or a ‘process’.

Formative (e)-assessment is concerned with learners making progress towards measurable attributes/knowledge/skills/understanding

It is about working with mechanisms and practices which allow the gap to close between what they are currently able to achieve and what they might be able to achieve

Increasing learners’ active responsibility for their part in the learning process is a main feature across a range of contexts and technologies

… but not much further consensus…differences tend to focus around whether ‘assessment’ is treated as an ‘event’ or a ‘process’.

Teacher and learner roles Cox et al 2008 (practice-based element of dentistry): 'a feedback process that provides information that can be used to fine-tune or modify what has already been done ‘….but by whom? What if… the teacher/tutor is ‘monitoring’ rather than ‘changing’? assessment processes are purely for self-assessment? the responses are automated?

Cox et al 2008 (practice-based element of dentistry):

'a feedback process that provides information that can be used to fine-tune or modify what has already been done ‘….but by whom?

What if…

the teacher/tutor is ‘monitoring’ rather than ‘changing’?

assessment processes are purely for self-assessment?

the responses are automated?

Feedback Bull & McKenna 2004 (computer-assisted assessment CAA)‏ ‘… assessments which assist learning by giving feedback which indicates how the student is progressing in terms of knowledge, skills and understanding of a subject. In CAA this often takes the form of objective questions with feedback given to the student either during or immediately after the assessment. Formative assessment may be monitored by the tutor, used purely for self-assessment, or used to contribute marks to a module grade’. (p. xiv) What if… feedback is complex (Shute 2008), or a two-way or multi-way process?

Bull & McKenna 2004 (computer-assisted assessment CAA)‏

‘… assessments which assist learning by giving feedback which indicates how the student is progressing in terms of knowledge, skills and understanding of a subject. In CAA this often takes the form of objective questions with feedback given to the student either during or immediately after the assessment. Formative assessment may be monitored by the tutor, used purely for self-assessment, or used to contribute marks to a module grade’. (p. xiv)

What if…

feedback is complex (Shute 2008), or a two-way or multi-way process?

- we contrast Bull with Nicol & Macfarlane-Dick’s 2006 criticisms of transmission-focused feedback: ‘ feedback messages are invariably complex and difficult to decipher’ in transmission contexts - students find it hard to take appropriate actions feedback tends to emphasis only cognition, not motivation and beliefs which are vital to internalization of knowledge and understanding transmission-based feedback is used in highly inventive ways, e.g. to direct learners within a carefully structured test environment towards appropriate pathways which enhance motivation and self-regulation (Winkley)‏ What types of knowledge, skills & understanding are to be included? How do feedback patterns fit into broader conceptual frameworks for learning and teaching?

- we contrast Bull with Nicol & Macfarlane-Dick’s 2006 criticisms of transmission-focused feedback:

‘ feedback messages are invariably complex and difficult to decipher’ in transmission contexts - students find it hard to take appropriate actions

feedback tends to emphasis only cognition, not motivation and beliefs which are vital to internalization of knowledge and understanding

transmission-based feedback is used in highly inventive ways, e.g. to direct learners within a carefully structured test environment towards appropriate pathways which enhance motivation and self-regulation (Winkley)‏

What types of knowledge, skills & understanding are to be included?

How do feedback patterns fit into broader conceptual frameworks for learning and teaching?

What is formative e-assessment? The contribution of more/faster/more frequent/ automated feedback to formative assessment ( Conole & Warburton, 2005 )?? The use of digital means to support formative assessment?? Formative features of assessment, which are afforded by specific features of digital media??

The contribution of more/faster/more frequent/ automated feedback to formative assessment ( Conole & Warburton, 2005 )??

The use of digital means to support formative assessment??

Formative features of assessment, which are afforded by specific features of digital media??

“ An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence” (Dylan Wiliam)‏

Formative = feedback + moments of contingency "... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another." (Leahy, Lyon, Thompson, and Wiliam 2005)‏

Wiliam's 5 strategies

Conversational Framework (Laurillard)‏

Our Methodology Focus on practitioner participation 5 practical enquiry days + on-line async. pre & post work Case stories -> design patterns -> scenarios

Focus on practitioner participation

5 practical enquiry days

+ on-line async. pre & post work

Case stories -> design patterns -> scenarios

Problem: Bad Design

the limit on growth is not the capacity to produce, but the knowledge to do it right. Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*)‏

Solution...

Design patterns Formative e-Assessment PED 5, Dec 2008 [describe] a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice (Alexander et al., 1977) ‏ C o n t e x t Problem Solution

Formative e-Assessment PED 5, Dec 2008 Problem Keep the rain out Context Cold, wet, poor. Method of solution Thatched roof Related Timber frame, Slanted roof, Chimney

example: activity nodes Formative e-Assessment PED 5, Dec 2008 Design problem Community facilities scattered individually through the city do nothing for the life of the city. Design solution Create nodes of activity throughout the community, spread about 300 yards apart. http://www.uni-weimar.de/architektur/InfAR/lehre/Entwurf/Patterns/030/ca_030.html

Problem: acceleration The world is changing. Fast. Faster. Teachers are learners. Students are researchers. We are all designers of our own and our peer's learning experiences. Son, this was my dad's mobile. I want you to have it.

The world is changing. Fast. Faster.

Teachers are learners.

Students are researchers.

We are all designers of our own and our peer's learning experiences.

Participatory Methodology for Practical Design Patterns Problem Acceleration -> need for effective protocols for sharing of design knowledge Context interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice. blended setting : co-located meetings + on-line collaborative authoring system.

Problem

Acceleration -> need for effective protocols for sharing of design knowledge

Context

interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice.

blended setting : co-located meetings + on-line collaborative authoring system.

Solution: a series of three* collaborative reflection workshops Case Stories Workshop Engender collaborative reflection among practitioners by a structured process of sharing stories. Pattern Mining Workshop Eliciting patterns by reflecting on and comparing case stories. Future Scenarios Workshop Validating and enhancing patterns by applying them to novel problems.

Case Stories Workshop

Engender collaborative reflection among practitioners by a structured process of sharing stories.

Pattern Mining Workshop

Eliciting patterns by reflecting on and comparing case stories.

Future Scenarios Workshop

Validating and enhancing patterns by applying them to novel problems.

Workshop I: Sharing case stories

Problem: telling a good story is not so easy Inexperienced story-tellers might - Take the context for granted Preach, apologise, market, or generalise Avoid inconvenient details Interactive feedback should help, but peers might - Be reluctant to criticize Attribute misunderstanding to their own faults Loose attention

Inexperienced story-tellers might -

Take the context for granted

Preach, apologise, market, or generalise

Avoid inconvenient details

Interactive feedback should help, but peers might -

Be reluctant to criticize

Attribute misunderstanding to their own faults

Loose attention

Three hats Solution: Three Hats

Some pointers for the case studies ‘ E-assessment… there is still much work to be done. One of the most important messages…is simply the variety and scope for imagining new forms of assessment – and of using technology to support it in imaginative ways ’ (Whitelock and Watt 2008, p. 153)‏ ‘ Modernising assessment’ involves blurring the boundaries between formative and summative processes (Elliott 2008) Rather than thinking in terms of ‘formative assessment’, it might be more appropriate to think in terms of how assessment can be used ‘formatively’ (Wiliam July 08)‏

‘ E-assessment… there is still much work to be done. One of the most important messages…is simply the variety and scope for imagining new forms of assessment – and of using technology to support it in imaginative ways ’ (Whitelock and Watt 2008, p. 153)‏

‘ Modernising assessment’ involves blurring the boundaries between formative and summative processes (Elliott 2008)

Rather than thinking in terms of ‘formative assessment’, it might be more appropriate to think in terms of how assessment can be used ‘formatively’ (Wiliam July 08)‏

Cases Assessment focus Technology used Technology role Socio-ped setting Institutional setting HE under-graduate + post-16 self instant feedback to individual st bespoke string comparator accuracy of language items String comparison HE distance tutor self-assess-ment graphical feedback to tutors web-based tool tutor socio-emotive feedback Open mentor HE vet training WBL self peer-peer t-st, t-group reflection & multi-player feedback social networking/ mob device recording/ reflecting clinical exp. Como: mobiles + flikr HE under-graduate t-st deliver tutor feedback Audacity & dicta-phones concepts in sociology Audiofiles M-level ITE peer-peer t-st t-group represent & share thinking wiki academic writing in teacher ed Academic writing

A few cases Creature of the week CoMo Post 16 String Comparison Open Mentor ...

Creature of the week

CoMo

Post 16 String Comparison

Open Mentor

...

Creature of the week (Judy Robertson)‏ Situation large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life. Task Engage and motivate the students show examples of good work which others could learn from show students their work is valued. build a sense of community. http://purl.org/planet/Cases/creatureoftheweek

Situation

large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life.

Task

Engage and motivate the students

show examples of good work which others could learn from

show students their work is valued.

build a sense of community.

CoMo (Niall Winters, Yishay Mor)‏ Situation Royal Vet College. Hospital rotations as part of their training. Task Allow students to capture critical incidents in text and image. Support sharing of clinical experiences and co-reflection. http://purl.org/planet/Cases/CoMo

Situation

Royal Vet College.

Hospital rotations as part of their training.

Task

Allow students to capture critical incidents in text and image.

Support sharing of clinical experiences and co-reflection.

Post 16 string comparison (Aliy Fowler)‏ Situation Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students. Sixth Form level, grammatical consolidation and whole-sentence translation. Task Allow students to practise written language independently and receive feedback on errors in order to improve their language skills. http://purl.org/planet/Cases/Post16stringcomparison

Situation

Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students.

Sixth Form level, grammatical consolidation and whole-sentence translation.

Task

Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.

Solution A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.

Open mentor (Denise Whitelock)‏ http://purl.org/planet/Cases/OpenMentor

A few patterns.. Try Once, Refine Once Feedback on Feedback Classroom display

Try Once, Refine Once

Feedback on Feedback

Classroom display

Try Once, Refine Once (Aliy Fowler)‏ http://pul.org/planet/Patterns/TryOnceRefineOnce

Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.

Context Class size Large (30-300)‏ Content Skills facts Mode of instruction Blended / on-line. Computer tested.

Class size

Large (30-300)‏

Content

Skills facts

Mode of instruction

Blended / on-line. Computer tested.

Solution

Feedback on Feedback (Linda McGuigan)‏ http://purl.org/planet/Patterns/FeedbackonFeedback

Good feedback should - Alert learners to their weaknesses. Diagnose the causes and dynamics of these. Include operational suggestions to improve the learning experience. Address socio-emotive factors. Tutors know this, but are pressed for time. Or not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching Problem

Good feedback should -

Alert learners to their weaknesses.

Diagnose the causes and dynamics of these.

Include operational suggestions to improve the learning experience.

Address socio-emotive factors.

Context Large scale, technology supported, graded courses many tutors instructing many students. Feedback is mediated by technology that allows it to be captured and processed in real time Topic of study is subject to both grading and formative feedback.

Large scale, technology supported, graded courses

many tutors instructing many students.

Feedback is mediated by technology that allows it to be captured and processed in real time

Topic of study is subject to both grading and formative feedback.

Solution Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.

Classroom Display http://purl.org/planet/Patterns/Classroomdisplay

Problem Rewards participation. Relates to learner's personal experiences. Window on student conceptions . Using learner generated content.. Needs to collate works in a single easy to access location. Learners uncomfortable about presenting their work in public Legal or other restrictions on sharing work.

Rewards participation.

Relates to learner's personal experiences.

Window on student conceptions .

Needs to collate works in a single easy to access location.

Learners uncomfortable about presenting their work in public

Legal or other restrictions on sharing work.

Context Class size: Small / medium (6-60)‏ Mode of instruction: Blended (preferable)‏ Time frame Continuous, over a period Pedagogy Involves construction / media production

Class size:

Small / medium (6-60)‏

Mode of instruction:

Blended (preferable)‏

Time frame

Continuous, over a period

Pedagogy

Involves construction / media production

Solution

Augmented domain map

Example scenario When using Try Once Refine Once , there is a risk that high-achievers do not receive feedback. So - Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors. Use Feedback on Feedback to alert tutors to the problem.

When using Try Once Refine Once , there is a risk that high-achievers do not receive feedback.

So -

Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors.

Use Feedback on Feedback to alert tutors to the problem.

What does ‘e’ add to formative assessment? I. Speed Speed of response is often important in enabling feedback to have an effect Supports rapid iteration – in many cases the ability to give feedback quickly means that the student’s next problem solving iteration can begin more quickly. II. Storage capacity Ability to access very large amounts of data (appropriate feedback/additional work/illustrations can be identified). III. Processing A utomation – in some situations the e-assessment system can analyse responses automatically and provide appropriate feedback. S calability – can often be the result of some level of automation. A daptivity – systems can adapt to students.

I. Speed

Speed of response is often important in enabling feedback to have an effect

Supports rapid iteration – in many cases the ability to give feedback quickly means that the student’s next problem solving iteration can begin more quickly.

II. Storage capacity

Ability to access very large amounts of data (appropriate feedback/additional work/illustrations can be identified).

III. Processing

A utomation – in some situations the e-assessment system can analyse responses automatically and provide appropriate feedback.

S calability – can often be the result of some level of automation.

A daptivity – systems can adapt to students.

IV. Communication Often the advantage of the ‘e’ is that it enables rapid communication of ideas across a range of audiences, and the technology allows this range to be controlled it can be just one person, a group, a class or more This communication aspect means that aspects of communication can be captured and given a degree of semi-permanence Semi- permanence supports the sharing of intellectual objects. V. Construction and representation Representation – the ability to represent ideas in a variety of ways and to move and translate between these representations Technology can support learners in the construction of representations of their own ideas. VI. Mutability Shared objects are not fixed. They can be changed easily and quickly.

IV. Communication

Often the advantage of the ‘e’ is that it enables rapid communication of ideas across a range of audiences, and the technology allows this range to be controlled it can be just one person, a group, a class or more

This communication aspect means that aspects of communication can be captured and given a degree of semi-permanence

Semi- permanence supports the sharing of intellectual objects.

V. Construction and representation

Representation – the ability to represent ideas in a variety of ways and to move and translate between these representations

Technology can support learners in the construction of representations of their own ideas.

VI. Mutability

Shared objects are not fixed. They can be changed easily and quickly.

Conclusions Tip of the iceberg Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios Collaborative elicitation of patterns from cases could be a potent form of professional development.

Tip of the iceberg

Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios

Collaborative elicitation of patterns from cases could be a potent form of professional development.

Thank you The Formative e-Assessment project: http://projects.lkl.ac.uk/feasst Final report http://telearn.noe-kaleidoscope.org/open-archive/browse?resource=1875 This event http://projects.lkl.ac.uk/feasst/april-28th/ This presentation http://www.slideshare.net/yish/formative-eassessment-a-scoping-study