Building a Bridge Between Theory and
Practice in Formative e-Assessment
e-Assessment in
Practice,
12 November,
Cranfield

Scoping a Vision for
Formative e-Assessment
• Commissioned by JISC
• June 2008 to January 2009
• Norbert Pachler, Harvey
Mellar, Caroline Daly, Yishay
Mor, Dylan Wiliam,
Emmanouil Agianniotakis
• http://feasst.wlecentre.ac.uk/
2
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Methodology
Desk research
Literature review
Comparing frameworks
Five Practical Enquiry Days
Combination of collaborative reflection, report back
from team, and guest plenaries
Launch day, three Pattern workshops, developers' day
3
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

What is formative e-assessment?
Formative?
‘practice’ assessment /
serial summative assessment synonymous with learning
The e?
The use of digital
means to support
Alternative view:
Formative features of assessment,
which are afforded by specific
features of digital media 4
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

OK, so what do we mean by
Formative Assessment?
“An assessment functions formatively when
evidence about student achievement elicited by
the assessment is interpreted and used to make
decisions about the next steps in instruction that
are likely to be better, or better founded, than the
decisions that would have been made in the
absence of that evidence”
(Dylan Wiliam)
5
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Formative = feedback + moments of
contingency
"... These create "moments of contingency," in
which the direction of the instruction will
depend on student responses. Teachers
provide feedback that engages students, make
time in class for students to work on
improvement, and activate students as
instructional resources for one another."
(Leahy, Lyon, Thompson, and Wiliam 2005)
6
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Participatory Methodology for
Practical Design Patterns
• Problem
– Acceleration → need for effective protocols for sharing
of design knowledge
• Context
– interdisciplinary communities of practitioners
engaged in collaborative reflection on a common
theme of their practice.
Son, this was my
dad's mobile.
I want you to 7
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
have it.

Problem: The Design
Divide
the gap between those who
have the expertise to develop
high-quality tools and
resources and those who
don’t (Mor & Winters, 2008*)
the critical resource is not the capacity to
produce, but the knowledge to do it right.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 8

Solution...
(in architecture)
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 9

What is a pattern?
• At is simplest, it is a
– Generalised solution to a problem
– Follows a specific structure
Cookbook: ingredients,
procedure, expected
What are we trying to
outcomes
achieve / solve?
Problem Solution
Context
When, Where,
Who
10
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

A burda pattern..
Season: Fall
For: Women
Garment Type: Dress
Style: Classic, Evening Wear, Romantic
Material: Taffeta
“if I copy a dress, I can only create
the same dress. If I have a pattern, I
can create many dresses” (Yim Ping
LENDEN)
11
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

As for
software
12
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

In learning / e-learning..
13
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

The Participatory Pattern
Workshops Methodology
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 14

Collaborative reflection workshop
Facilitate on-going design-
level conversation between
designers and practitioners
involved in diverse aspects
of the problem domain.
Open, trusting and convivial.
And at the same time
Critical, focused and output-directed.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 15

Collaborative reflection workshop
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 16

Case Stories Workshop
Engender collaborative
reflection among
practitioners by a
structured process of
sharing stories of
successful practice.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 17

Pattern Mining Workshop
Shift from anecdotes to
transferable design
knowledge by identifying
commonalities across
case stories, and
capturing them in a
semi-structured form.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 18

Future Scenarios Workshop
Validate design patterns
by applying them to
novel real problems in
real contexts.
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 19

A few cases
• Creature of the week
• CoMo
• Post 16 String Comparison
• Open Mentor
• ...
20
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Creature of the week (Judy
Robertson)
Situation
large class (138), first and second year computer science students.
assignment: create a virtual pet in Second Life.
Task
− Engage and motivate the students
− show examples of good work which others could
learn from
− show students their work is valued.
− build a sense of community.
21
http://purl.org/planet/Cases/creatureoftheweek
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

22
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

CoMo (Niall Winters, Yishay Mor)
Situation
− Royal Vet College.
− Hospital rotations as part of their training.
Task
− Allow students to capture critical incidents in text
and image.
− Support sharing of clinical experiences and co-
reflection.
23
http://purl.org/planet/Cases/CoMo
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

24
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Post 16 string comparison (Aliy
Fowler)
Situation
− Grammar school been piloting the ‘string comparison’ approach to
language teaching at post-16 for AS and A2 level students.
− Sixth Form level, grammatical consolidation and whole-sentence
translation.
Task
− Allow students to practise written language
independently and receive feedback on errors in
order to improve their language skills.
http://purl.org/planet/Cases/Post16stringcomparison
25
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Solution
A bespoke string (sequence) comparator was
designed; uses fine-granularity sequence
comparison to compare correct language
strings to a user’s answer. Students answer
questions and the comparator marks up errors
in their input using colour coding (and font
style) to highlight the different types of error. If
an answer contains errors the student is given
a second attempt in which to correct the
submission based on the feedback received.
26
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Open mentor (Denise Whitelock)
http://purl.org/planet/Cases/OpenMentor
27
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

A few patterns..
• Try Once, Refine Once
• Feedback on Feedback
• Classroom display
28
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Try Once,
Refine Once
(Aliy Fowler)
http://pul.org/planet/Patterns/TryOnceRefineOnce
29
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Problem
Lack of immediate feedback for students
leads to fossilisation of errors and
misconceptions
providing immediate feedback in an
iterative fashion can also hinder
effective learning since students are
able to "grope their way" step-by-step
to a correct solution without
necessarily having to think about each
answer as a whole.
30
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Context
Class size
− Large (30-300)
Content
− Skills facts
Mode of instruction
− Blended / on-line. Computer tested.
31
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Solution
32

Feedback
on
Feedback
(Linda
McGuigan)
33
http://purl.org/planet/Patterns/FeedbackonFeedback

Good feedback should - Problem
Alert learners to their weaknesses.
Diagnose the causes and dynamics of these.
Include operational suggestions to improve the learning
experience.
Address socio-emotive factors.
Tutors know this, but are pressed for
time. Or not aware of their feedback
strategies
Large teaching organisations are not equipped to
provide tutors with personal feedback on their
teaching 34

Context
Large scale, technology supported, graded
courses
− many tutors instructing many students.
Feedback is mediated by technology that allows it
to be captured and processed in real time
Topic of study is subject to both grading and
formative feedback.
35
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Solution
Embed a mechanism in the learning and teaching system
that regularly captures tutor feedback, analyses it, and
presents them with graphical representation of the types of
feedback they have given. Ideally, this should also include
constructive advice as to how to shift from less to more
effective forms.
In computer supported environments (e.g. VLEs), this
mechanism could be integrated into the system, providing
tutors with immediate analysis of their feedback, as well as
long-term aggregates.
36
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Classroom Display
http://purl.org/planet/Patterns/Classroomdisplay 37
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Problem
Using learner
generated content..
Rewards participation.
Relates to learner's personal
experiences.
Window on student conceptions.
Need to collate works in a single easy to access
location.
Learners uncomfortable about presenting their work in
public
Legal or other restrictions on sharing work.
38

Context
Class size:
− Small / medium (6-60)
Mode of instruction:
− Blended (preferable)
Time frame
− Continuous, over a period
Pedagogy
− Involves construction / media production
39
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Solution
40
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Augmented domain map
41

Example scenario
When using Try Once Refine Once there is a risk
Once,
that high-achievers do not receive feedback.
So -
Use Showcase Learning to celebrate students’
work and provoke feedback from peers and
tutors.
Use Feedback on Feedback to alert tutors to the
problem.
42
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Conclusions
Tip of the iceberg
Practitioners (educational / software)
acknowledge the value of patterns, when served
with side dishes of cases + scenarios
Collaborative elicitation of patterns from cases
could be a potent form of professional
development.
43
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

Thank you
The pattern language network project:
http://patternlanguagenetwork.org
The learning patterns project:
http://lp.noe-kaleidoscope.org/
The formative e-assessment project:
http://feasst.wlecentre.ac.uk/
Yishay Mor
This presentation: people.lkl.ac.uk/yishay
http://talks.yishaymor.org/e-assessmentinpractice2009 yishaym@gmail.com
http://www.slideshare.net/yish/eain @yishaym
44
Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009