Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Bridge Between Theory and Practice in Formative e-Assessment

2,002 views

Published on

invited talk at
e-Assessment in Practice, November 12, Cranfiled

http://talks.yishaymor.org/e-assessmentinpractice2009

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Building a Bridge Between Theory and Practice in Formative e-Assessment

  1. 1. Building a Bridge Between Theory and Practice in Formative e-Assessment e-Assessment in Practice, 12 November, Cranfield
  2. 2. Scoping a Vision for Formative e-Assessment • Commissioned by JISC • June 2008 to January 2009 • Norbert Pachler, Harvey Mellar, Caroline Daly, Yishay Mor, Dylan Wiliam, Emmanouil Agianniotakis • http://feasst.wlecentre.ac.uk/ 2 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  3. 3. Methodology Desk research Literature review Comparing frameworks Five Practical Enquiry Days Combination of collaborative reflection, report back from team, and guest plenaries Launch day, three Pattern workshops, developers' day 3 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  4. 4. What is formative e-assessment? Formative? ‘practice’ assessment / serial summative assessment synonymous with learning The e? The use of digital means to support Alternative view: Formative features of assessment, which are afforded by specific features of digital media 4 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  5. 5. OK, so what do we mean by Formative Assessment? “An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence” (Dylan Wiliam) 5 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  6. 6. Formative = feedback + moments of contingency "... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another." (Leahy, Lyon, Thompson, and Wiliam 2005) 6 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  7. 7. Participatory Methodology for Practical Design Patterns • Problem – Acceleration → need for effective protocols for sharing of design knowledge • Context – interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice. Son, this was my dad's mobile. I want you to 7 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 have it.
  8. 8. Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*) the critical resource is not the capacity to produce, but the knowledge to do it right. Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 8
  9. 9. Solution... (in architecture) Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 9
  10. 10. What is a pattern? • At is simplest, it is a – Generalised solution to a problem – Follows a specific structure Cookbook: ingredients, procedure, expected What are we trying to outcomes achieve / solve? Problem Solution Context When, Where, Who 10 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  11. 11. A burda pattern.. Season: Fall For: Women Garment Type: Dress Style: Classic, Evening Wear, Romantic Material: Taffeta “if I copy a dress, I can only create the same dress. If I have a pattern, I can create many dresses” (Yim Ping LENDEN) 11 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  12. 12. As for software 12 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  13. 13. In learning / e-learning.. 13 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  14. 14. The Participatory Pattern Workshops Methodology Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 14
  15. 15. Collaborative reflection workshop Facilitate on-going design- level conversation between designers and practitioners involved in diverse aspects of the problem domain. Open, trusting and convivial. And at the same time Critical, focused and output-directed. Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 15
  16. 16. Collaborative reflection workshop Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 16
  17. 17. Case Stories Workshop Engender collaborative reflection among practitioners by a structured process of sharing stories of successful practice. Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 17
  18. 18. Pattern Mining Workshop Shift from anecdotes to transferable design knowledge by identifying commonalities across case stories, and capturing them in a semi-structured form. Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 18
  19. 19. Future Scenarios Workshop Validate design patterns by applying them to novel real problems in real contexts. Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009 19
  20. 20. A few cases • Creature of the week • CoMo • Post 16 String Comparison • Open Mentor • ... 20 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  21. 21. Creature of the week (Judy Robertson) Situation large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life. Task − Engage and motivate the students − show examples of good work which others could learn from − show students their work is valued. − build a sense of community. 21 http://purl.org/planet/Cases/creatureoftheweek Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  22. 22. 22 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  23. 23. CoMo (Niall Winters, Yishay Mor) Situation − Royal Vet College. − Hospital rotations as part of their training. Task − Allow students to capture critical incidents in text and image. − Support sharing of clinical experiences and co- reflection. 23 http://purl.org/planet/Cases/CoMo Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  24. 24. 24 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  25. 25. Post 16 string comparison (Aliy Fowler) Situation − Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students. − Sixth Form level, grammatical consolidation and whole-sentence translation. Task − Allow students to practise written language independently and receive feedback on errors in order to improve their language skills. http://purl.org/planet/Cases/Post16stringcomparison 25 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  26. 26. Solution A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received. 26 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  27. 27. Open mentor (Denise Whitelock) http://purl.org/planet/Cases/OpenMentor 27 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  28. 28. A few patterns.. • Try Once, Refine Once • Feedback on Feedback • Classroom display 28 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  29. 29. Try Once, Refine Once (Aliy Fowler) http://pul.org/planet/Patterns/TryOnceRefineOnce 29 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  30. 30. Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole. 30 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  31. 31. Context Class size − Large (30-300) Content − Skills facts Mode of instruction − Blended / on-line. Computer tested. 31 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  32. 32. Solution 32
  33. 33. Feedback on Feedback (Linda McGuigan) 33 http://purl.org/planet/Patterns/FeedbackonFeedback
  34. 34. Good feedback should - Problem Alert learners to their weaknesses. Diagnose the causes and dynamics of these. Include operational suggestions to improve the learning experience. Address socio-emotive factors. Tutors know this, but are pressed for time. Or not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching 34
  35. 35. Context Large scale, technology supported, graded courses − many tutors instructing many students. Feedback is mediated by technology that allows it to be captured and processed in real time Topic of study is subject to both grading and formative feedback. 35 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  36. 36. Solution Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates. 36 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  37. 37. Classroom Display http://purl.org/planet/Patterns/Classroomdisplay 37 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  38. 38. Problem Using learner generated content.. Rewards participation. Relates to learner's personal experiences. Window on student conceptions. Need to collate works in a single easy to access location. Learners uncomfortable about presenting their work in public Legal or other restrictions on sharing work. 38
  39. 39. Context Class size: − Small / medium (6-60) Mode of instruction: − Blended (preferable) Time frame − Continuous, over a period Pedagogy − Involves construction / media production 39 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  40. 40. Solution 40 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  41. 41. Augmented domain map 41
  42. 42. Example scenario When using Try Once Refine Once there is a risk Once, that high-achievers do not receive feedback. So - Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors. Use Feedback on Feedback to alert tutors to the problem. 42 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  43. 43. Conclusions Tip of the iceberg Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios Collaborative elicitation of patterns from cases could be a potent form of professional development. 43 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009
  44. 44. Thank you The pattern language network project: http://patternlanguagenetwork.org The learning patterns project: http://lp.noe-kaleidoscope.org/ The formative e-assessment project: http://feasst.wlecentre.ac.uk/ Yishay Mor This presentation: people.lkl.ac.uk/yishay http://talks.yishaymor.org/e-assessmentinpractice2009 yishaym@gmail.com http://www.slideshare.net/yish/eain @yishaym 44 Yishay Mor / formative e-assessment / http://talks.yishaymor.org/e-assessmentinpractice2009

×