Formative e-assessment: case
stories, design patterns, and
future scenarios
Harvey Mellar
London Knowledge Lab
Institute o...
Overview
Short term, scoping study commissioned by JISC, and supported by the Centre for
Excellence in Work-based Learning...
WHAT IS FORMATIVE
ASSESSMENT?
A definition
“An assessment functions formatively when
evidence about student achievement elicited by
the assessment is in...
Five strategies
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment.
Educational Assessment, Eva...
Moments of contingency
Teachers design their instruction to yield evidence about student
achievement, by carefully craftin...
PARTICIPATORY METHODOLOGY
FOR PRACTICAL DESIGN
PATTERNS
8
Case Stories Workshop
Engender
collaborative reflection
among practitioners by
a structured process of
sharing stories o...
9
Pattern Mining Workshop
Shift from anecdotes
to transferable design
knowledge by
identifying
commonalities across
case s...
The core template
• Context
– Where, when, who (all the things you can’t change)
• Problem (pick one!)
– We want to do A u...
11
Future Scenarios Workshop
Validate design
patterns by
applying them to
novel real
problems in real
contexts
CASES
Creature of the week
(Judy Robertson)
Situation
Large class (138), first and second year computer
science students. Assign...
CoMo (Niall Winters, Yishay Mor)
Situation
 Royal Veterinary College
 Hospital rotations as part of the training
Task
 ...
Open mentor (Denise Whitelock)
http://purl.org/planet/Cases/OpenMentor
PATTERNS
Try Once,
Refine Once
(Aliy Fowler)
http://purl.org/planet/Patterns/TryOnceRefineOnce
Problem
Lack of immediate feedback for students
leads to fossilisation of errors and
misconceptions
providing immediate fe...
Context
Class size
 Large (30-300)
Content
 Skillsfacts
Mode of instruction
 Blendedon-line. Computer tested
Solution
• Students are posed questions of a type which elicit answers that
can contain multiple errors
• If a student's a...
Feedback
on
Feedback
(Linda
McGuigan)
http://purl.org/planet/Patterns/FeedbackonFeedback
Good feedback should
 Alert learners to their weaknesses.
 Diagnose the causes and dynamics of these.
 Include operatio...
Context
Large scale, technology supported, graded
courses
 many tutors instructing many students
Feedback is mediated by ...
Solution
Embed a mechanism in the learning and teaching system
that regularly captures tutor feedback, analyses it, and
pr...
SCENARIO
High achievers
When using Try Once Refine Once, there is a risk
that high-achievers do not receive feedback
So
• Use Showc...
AUGMENTED DOMAIN MAP
Reminder of the five strategies
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment.
Educational...
REPORTS
http://www.jisc.ac.uk/media/documents/projects/scopingfinalreport.pdf
Practical design patterns for
teaching and learning with
technology
A book for Sense Publisher's 'Technology
Enhanced Lear...
RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios
RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios
RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios
RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios
Upcoming SlideShare
Loading in …5
×

RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios

884
-1

Published on

Research in Distance Education: impact on practice conference, 27 October 2010. Presentation in Assessment Strand by Dr Harvey Mellar, Institute of Education.
More details at www.cde.london.ac.uk.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
884
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • We define formative e-assessment as the use of ICT to support the iterative process of gathering and analysing information about student learning by teachers as well as learners and of evaluating it in relation to prior achievement and attainment of intended, as well as unintended learning outcomes, in a way that allows the teacher or student to adjust the learning trajectory
  • harvey
    Level 2

    http://patternlanguagenetwork.myxwiki.org/xwiki/bin/view/Groups.FormativeEAssessment/
  • Key findings from the literature
    The domain is complex and contentious:
    there is a wide heterogeneity in the literature, and frequent slippage between terms such as ‘assessment’ and ‘learning’, ‘formative’ and ‘summative’ and there are widely differing theoretical emphases
    a wide variety of perspectives and practices exist which prioritise different educational goals; components have been identified to reflect a variety of actors, learning intentions, roles and activities, and the mechanisms involved in enabling progression of learning towards measurable attributes
    From: ‘practice’ assessment, or serial (or repeated) summative assessment
    To: synonymous with learning
  • What does ‘e’ add?

    Speed
    Storage capacity
    Processing
    Communication
    Construction and representation
    Mutability

    Adaptivity’ is a core component of e-assessment processes indicating the flexible responsiveness on the part of learners and teachers which may or may not itself involve the use of technology.

    I. Speed
    Speed of response is often important in enabling feedback to have an effect
    Supports rapid iteration – in many cases the ability to give feedback quickly means that the student’s next problem solving iteration can begin more quickly.
    II. Storage capacity
    Ability to access very large amounts of data (so appropriate feedback/additional work/illustrations can be identified).
    III. Processing
    Automation – in some situations the e-assessment system can analyse responses automatically and provide appropriate feedback.
    Scalability – can often be the result of some level of automation.
    Adaptivity – systems can adapt to students.
    V. Communication
    Often the advantage of the ‘e’ is that it enables rapid communication of ideas across a range of audiences, and the technology allows this range to be controlled it can be just one person, a group, a class or more
    This communication aspect means that aspects of communication can be captured and given a degree of semi-permanence
    This semi permanence supports the sharing of intellectual objects.
    V. Construction and representation
    Representation – the ability to represent ideas in a variety of ways and to move and translate between these representations
    Technology can support learners in the construction of representations of their own ideas.
    By representation technology enables concepts to be ‘shaped’ and therefore affects their meaning, i.e. representation makes use of symbols which help meanings develop
    In representing their ideas in digital artefacts (creating these intellectual objects) learners open up a window on their thinking.
    VI. Mutability
    Shared objects are not fixed, they can change/be changed easily and quickly.


  • RIDE 2010 presentation - Formative e-assessment: case stories, design patterns, and future scenarios

    1. 1. Formative e-assessment: case stories, design patterns, and future scenarios Harvey Mellar London Knowledge Lab Institute of Education, University of London http://feasst.wlecentre.ac.uk/ http://www.jisc.ac.uk/whatwedo/projects/feasst.aspx
    2. 2. Overview Short term, scoping study commissioned by JISC, and supported by the Centre for Excellence in Work-based Learning for Education Professionals • Methodology • Desk research • Literature review • Comparing frameworks • Five Practical Enquiry Days • Combination of collaborative reflection, report back from teams, and guest plenaries • Launch day, three Planet workshops, developers' day (Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns - http://patternlanguagenetwork.wordpress.com) • Wiki for collaborative authoring of patterns http://purl.org/planet/Groups.FormativeEAssessment/
    3. 3. WHAT IS FORMATIVE ASSESSMENT?
    4. 4. A definition “An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence” (Dylan Wiliam)
    5. 5. Five strategies Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31.
    6. 6. Moments of contingency Teachers design their instruction to yield evidence about student achievement, by carefully crafting hinge-point questions, for example. These create ‘moments of contingency’, in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another. (Leahy, Lyon, Thompson and Wiliam, 2005)
    7. 7. PARTICIPATORY METHODOLOGY FOR PRACTICAL DESIGN PATTERNS
    8. 8. 8 Case Stories Workshop Engender collaborative reflection among practitioners by a structured process of sharing stories of successful practice
    9. 9. 9 Pattern Mining Workshop Shift from anecdotes to transferable design knowledge by identifying commonalities across case stories, and capturing them in a semi-structured form
    10. 10. The core template • Context – Where, when, who (all the things you can’t change) • Problem (pick one!) – We want to do A under condition B but are constrained by C • Solution (in any order that works for you) C o n t e x t Problem Solution When, Where, Who What are we trying to achieve / solve? Cookbook: ingredients, procedure, expected outcomes
    11. 11. 11 Future Scenarios Workshop Validate design patterns by applying them to novel real problems in real contexts
    12. 12. CASES
    13. 13. Creature of the week (Judy Robertson) Situation Large class (138), first and second year computer science students. Assignment: create a virtual pet in Second Life Task  Engage and motivate the students  show examples of good work which others could learn from  show students their work is valued.  build a sense of community.http://purl.org/planet/Cases/creatureoftheweek
    14. 14. CoMo (Niall Winters, Yishay Mor) Situation  Royal Veterinary College  Hospital rotations as part of the training Task  Allow students to capture critical incidents in text and image  Support sharing of clinical experiences and co- reflection http://purl.org/planet/Cases/CoMo
    15. 15. Open mentor (Denise Whitelock) http://purl.org/planet/Cases/OpenMentor
    16. 16. PATTERNS
    17. 17. Try Once, Refine Once (Aliy Fowler) http://purl.org/planet/Patterns/TryOnceRefineOnce
    18. 18. Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.
    19. 19. Context Class size  Large (30-300) Content  Skillsfacts Mode of instruction  Blendedon-line. Computer tested
    20. 20. Solution • Students are posed questions of a type which elicit answers that can contain multiple errors • If a student's answer is entirely correct a mark of 100% is awarded • If their answer contains errors, a mark is given which contributes to a percentage of the total mark for the question, along with detailed - yet generic - feedback on the location and type of the errors • Students are then permitted a second attempt in which to refine their answer • The mark for the 2nd attempt contributes to remaining percentage of the total mark for the question • Feedback on any remaining errors is also given, along with the correct answer(s) • No further attempts are permitted
    21. 21. Feedback on Feedback (Linda McGuigan) http://purl.org/planet/Patterns/FeedbackonFeedback
    22. 22. Good feedback should  Alert learners to their weaknesses.  Diagnose the causes and dynamics of these.  Include operational suggestions to improve the learning experience.  Address socio-emotive factors. Tutors know this, but are pressed for time, or are not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching Problem
    23. 23. Context Large scale, technology supported, graded courses  many tutors instructing many students Feedback is mediated by technology that allows it to be captured and processed in real time Topic of study is subject to both grading and formative feedback
    24. 24. Solution Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.
    25. 25. SCENARIO
    26. 26. High achievers When using Try Once Refine Once, there is a risk that high-achievers do not receive feedback So • Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors • Use Feedback on Feedback to alert tutors to the problem
    27. 27. AUGMENTED DOMAIN MAP
    28. 28. Reminder of the five strategies Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5-31.
    29. 29. REPORTS
    30. 30. http://www.jisc.ac.uk/media/documents/projects/scopingfinalreport.pdf
    31. 31. Practical design patterns for teaching and learning with technology A book for Sense Publisher's 'Technology Enhanced Learning' series Editors: Yishay Mor (London Knowledge Lab), Steven Warburton (King's College London) and Niall Winters (London Knowledge Lab) http://www.practicalpatternsbook.org/Home

    ×