Published on

Participatory Pattern Workshops:
Application to formative e-assessment

An LKL seminar at Southampton University, Oct. 2009

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Southampton, Oct. 2009 Participatory Pattern Workshops: Application to formative e-assessment
  2. 2. the critical resource is not the capacity to produce, but the knowledge to do it right. Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*)‏
  3. 3. Solution... (in architecture)
  4. 4. What is a pattern? <ul><li>At is simplest, it is a </li></ul><ul><ul><li>Generalised solution to a problem </li></ul></ul><ul><ul><li>Follows a specific structure </li></ul></ul>When, Where, Who What are we trying to achieve / solve? Cookbook: ingredients, procedure, expected outcomes C o n t e x t Problem Solution
  5. 5. A burda pattern.. Season: Fall For: Women Garment Type: Dress Style: Classic, Evening Wear, Romantic Material: Taffeta “ if I copy a dress, I can only create the same dress. If I have a pattern, I can create many dresses” (Yim Ping LENDEN)
  6. 6. As for software
  7. 7. Participatory Methodology for Practical Design Patterns <ul><li>Problem </li></ul><ul><ul><li>Acceleration -> need for effective protocols for sharing of design knowledge </li></ul></ul><ul><li>Context </li></ul><ul><ul><li>interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice. </li></ul></ul><ul><ul><li>blended setting : co-located meetings + on-line collaborative authoring system. </li></ul></ul>Son, this was my dad's mobile. I want you to have it.
  8. 8. The Participatory Pattern Workshops Methodology
  9. 9. Collaborative reflection workshop Facilitate on-going design-level conversation between designers and practitioners involved in diverse aspects of the problem domain. Open, trusting and convivial. And at the same time Critical, focused and output-directed.
  10. 10. Collaborative reflection workshop
  11. 11. Case Stories Workshop Engender collaborative reflection among practitioners by a structured process of sharing stories of successful practice.
  12. 12. Tell us about... <ul><li>A specific incident </li></ul><ul><li>That happened to you </li></ul><ul><li>Where you confronted a challenge / problem </li></ul><ul><li>And resolved it successfully </li></ul>
  13. 13. Be a STARR <ul><li>Situation </li></ul><ul><ul><li>Describe the context in detail. </li></ul></ul><ul><li>Task </li></ul><ul><ul><li>What was the problem you were trying to solve? </li></ul></ul><ul><li>Action </li></ul><ul><ul><li>What did you do to solve it? </li></ul></ul><ul><li>Results </li></ul><ul><ul><li>What happened? Did you succeed? Did you adjust? </li></ul></ul><ul><li>& Reflections </li></ul><ul><ul><li>What did you learn? </li></ul></ul>http://www.slideshare.net/yish/star-case-study-template http://patternlanguagenetwork.myxwiki.org/xwiki/bin/view/Cases/
  14. 14. Tell it like it was <ul><li>You don’t know </li></ul><ul><li>Would have happened.. </li></ul><ul><li>Could have happened.. </li></ul><ul><li>Should have happened.. </li></ul><ul><li>Will Happen… </li></ul>You DO know, and only YOU know What happened
  15. 15. Three hats
  16. 16. Pattern Mining Workshop Shift from anecdotes to transferable design knowledge by identifying commonalities across case stories, and capturing them in a semi-structured form.
  17. 17. The core template <ul><li>Context </li></ul><ul><ul><li>Where, when, who (all the things you can’t change) </li></ul></ul><ul><li>Problem (pick one!) </li></ul><ul><ul><li>We want to do A under condition B but are constrained by C </li></ul></ul><ul><li>Solution </li></ul><ul><li>(in any order that </li></ul><ul><li>works for you) </li></ul>When, Where, Who What are we trying to achieve / solve? Cookbook: ingredients, procedure, expected outcomes C o n t e x t Problem Solution
  18. 18. Force Mapping <ul><li>Name the forces </li></ul><ul><li>Give them icons </li></ul><ul><li>Plot the links and mark + / - </li></ul>Forces: constraints or factors that influence the problem. The difficulty of solving the problem arises from tensions between competing forces.
  19. 19. Forces <ul><li>A ctors </li></ul><ul><li>B eliefs </li></ul><ul><li>C onditions </li></ul><ul><li>D esires </li></ul>
  20. 20. Future Scenarios Workshop Validate design patterns by applying them to novel real problems in real contexts.
  21. 21. the cycle of design research
  22. 22. the cycle of design research
  23. 23. formative e-assessment: case stories, design patterns, and future scenarios Caroline Daly, Harvey Mellar, Yishay Mor, Norbert Pachler, Institute of Education, University of London http://feasst.wlecentre.ac.uk/
  24. 24. Overview <ul><li>Scoping study commissioned by JISC </li></ul><ul><li>Short term, small budget, intended to inform future funding frameworks </li></ul><ul><li>Established a commited user group of higher-education teachers & researchers </li></ul><ul><li>Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform </li></ul>
  25. 25. Methodology <ul><li>Desk research </li></ul><ul><ul><li>Literature review </li></ul></ul><ul><ul><li>Comparing frameworks </li></ul></ul><ul><li>Five Practical Enquiry Days </li></ul><ul><ul><li>Combination of collaborative reflection, report back from team, and guest plenaries </li></ul></ul><ul><ul><li>Launch day, three Planet workshops, developers' day </li></ul></ul>
  26. 26. What is formative e-assessment? <ul><li>No consistent view in the literature </li></ul><ul><ul><li>From: ‘practice’ assessment, or serial (or repeated) summative assessment </li></ul></ul><ul><ul><li>To: synonymous with learning </li></ul></ul><ul><li>The use of digital means to support formative assessment </li></ul><ul><li>Formative features of assessment, which are afforded by specific features of digital media </li></ul>
  27. 27. OK, so what do we mean by Formative Assessment? <ul><li>“ An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence” </li></ul><ul><li>(Dylan Wiliam) ‏ </li></ul>
  28. 28. Formative = feedback + moments of contingency <ul><li>&quot;... These create &quot;moments of contingency,&quot; in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another.&quot; </li></ul><ul><li>(Leahy, Lyon, Thompson, and Wiliam 2005) ‏ </li></ul>
  29. 29. Wiliam's five strategies
  30. 30. Conversational Framework (Laurillard) ‏
  31. 31. Learner’s conception as practice = 20% Learner’s conception = 10% teacher’s Teacher’s conception Other learner’(s) conceptions Teacher-designed task practice environment Other learner’(s) conceptions as practice Offers answers, ideas Questions, offers ideas Feedback on action Action to achieve goal Shares practice attempt Adapts approach to task to current conception Adapt a Task practice environment for learners’ needs Reflects on feedback in relation to task and action Reflects on alternative practice Reflects on learners’ practice Hints, comments Present concepts Task goal Revises action This animation represents the full use of the Conversational Framework through a combination of Teaching Methods, such as lecture/book/web resource + tutorial/discussion environment + fieldwork/lab/simulation + collaboration environment Presents conception as product Questions Learner’s conception = 20% teacher’s Learner’s conception = 40% teacher’s Learner’s conception as practice = 40% Learner’s conception = 50% teacher’s Learner’s conception as practice = 50% Adapts approach to task to current conception Shares practice attempt Learner’s conception as practice = 60% Learner’s conception = 60% teacher’s It illustrates how the iteration between theory and practice, between teacher and learner, and other learners, is expected to contribute to enable the learner to approach an understanding equivalent to the teacher’s. Discursive/theoretical level Practice/practical level
  32. 32. Evidence Centred Assessment Design (Mislevy) ‏ <ul><li>PRO: </li></ul><ul><li>high level of abstraction, so room for implementation flexibility </li></ul><ul><li>delivery process was implementation independent </li></ul><ul><li>use of design patterns </li></ul><ul><li>ANTI </li></ul><ul><li>full architecture was beyond our needs </li></ul><ul><li>somewhat removed from the more open ended forms of work that we were seeing in practice </li></ul><ul><li>too close to implementation detail for where we were at </li></ul><ul><li>did not provide a theory of learning, or of formative assessment </li></ul>
  33. 33. A few cases <ul><li>Creature of the week </li></ul><ul><li>CoMo </li></ul><ul><li>Post 16 String Comparison </li></ul><ul><li>Open Mentor </li></ul><ul><li>... </li></ul>
  34. 34. Creature of the week (Judy Robertson)‏ <ul><li>Situation </li></ul><ul><ul><li>large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life. </li></ul></ul><ul><li>Task </li></ul><ul><ul><li>Engage and motivate the students </li></ul></ul><ul><ul><li>show examples of good work which others could learn from </li></ul></ul><ul><ul><li>show students their work is valued. </li></ul></ul><ul><ul><li>build a sense of community. </li></ul></ul>http://purl.org/planet/Cases/creatureoftheweek
  35. 36. CoMo (Niall Winters, Yishay Mor)‏ <ul><li>Situation </li></ul><ul><ul><li>Royal Vet College. </li></ul></ul><ul><ul><li>Hospital rotations as part of their training. </li></ul></ul><ul><li>Task </li></ul><ul><ul><li>Allow students to capture critical incidents in text and image. </li></ul></ul><ul><ul><li>Support sharing of clinical experiences and co-reflection. </li></ul></ul>http://purl.org/planet/Cases/CoMo
  36. 38. Post 16 string comparison (Aliy Fowler)‏ <ul><li>Situation </li></ul><ul><ul><li>Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students. </li></ul></ul><ul><ul><li>Sixth Form level, grammatical consolidation and whole-sentence translation. </li></ul></ul><ul><li>Task </li></ul><ul><ul><li>Allow students to practise written language independently and receive feedback on errors in order to improve their language skills. </li></ul></ul>http://purl.org/planet/Cases/Post16stringcomparison
  37. 39. Solution <ul><li>A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received. </li></ul>
  38. 40. Open mentor (Denise Whitelock)‏ http://purl.org/planet/Cases/OpenMentor
  39. 41. A few patterns.. <ul><li>Try Once, Refine Once </li></ul><ul><li>Feedback on Feedback </li></ul><ul><li>Classroom display </li></ul>
  40. 42. Try Once, Refine Once (Aliy Fowler)‏ http://pul.org/planet/Patterns/TryOnceRefineOnce
  41. 43. Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to &quot;grope their way&quot; step-by-step to a correct solution without necessarily having to think about each answer as a whole.
  42. 44. Context <ul><li>Class size </li></ul><ul><ul><li>Large (30-300)‏ </li></ul></ul><ul><li>Content </li></ul><ul><ul><li>Skills facts </li></ul></ul><ul><li>Mode of instruction </li></ul><ul><ul><li>Blended / on-line. Computer tested. </li></ul></ul>
  43. 45. Solution
  44. 46. Feedback on Feedback (Linda McGuigan)‏ http://purl.org/planet/Patterns/FeedbackonFeedback
  45. 47. Problem <ul><li>Good feedback should - </li></ul><ul><li>Alert learners to their weaknesses. </li></ul><ul><li>Diagnose the causes and dynamics of these. </li></ul><ul><li>Include operational suggestions to improve the learning experience. </li></ul><ul><li>Address socio-emotive factors. </li></ul>Tutors know this, but are pressed for time. Or not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching
  46. 48. Context <ul><li>Large scale, technology supported, graded courses </li></ul><ul><ul><li>many tutors instructing many students. </li></ul></ul><ul><li>Feedback is mediated by technology that allows it to be captured and processed in real time </li></ul><ul><li>Topic of study is subject to both grading and formative feedback. </li></ul>
  47. 49. Solution <ul><li>Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. </li></ul><ul><li>In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates. </li></ul>
  48. 50. Classroom Display http://purl.org/planet/Patterns/Classroomdisplay
  49. 51. Problem <ul><li>Rewards participation. </li></ul><ul><li>Relates to learner's personal experiences. </li></ul><ul><li>Window on student conceptions . </li></ul>Using learner generated content.. <ul><li>Needs to collate works in a single easy to access location. </li></ul><ul><li>Learners uncomfortable about presenting their work in public </li></ul><ul><li>Legal or other restrictions on sharing work. </li></ul>
  50. 52. Context <ul><li>Class size: </li></ul><ul><ul><li>Small / medium (6-60)‏ </li></ul></ul><ul><li>Mode of instruction: </li></ul><ul><ul><li>Blended (preferable)‏ </li></ul></ul><ul><li>Time frame </li></ul><ul><ul><li>Continuous, over a period </li></ul></ul><ul><li>Pedagogy </li></ul><ul><ul><li>Involves construction / media production </li></ul></ul>
  51. 53. Solution
  52. 54. Augmented domain map
  53. 55. Example scenario <ul><li>When using Try Once Refine Once , there is a risk that high-achievers do not receive feedback. </li></ul><ul><li>So - </li></ul><ul><li>Use Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors. </li></ul><ul><li>Use Feedback on Feedback to alert tutors to the problem. </li></ul>
  54. 56. Conclusions <ul><li>Tip of the iceberg </li></ul><ul><li>Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios </li></ul><ul><li>Collaborative elicitation of patterns from cases could be a potent form of professional development. </li></ul>
  55. 57. Thank you The pattern language network project: http://patternlanguagenetwork.org The learning patterns project: http://lp.noe-kaleidoscope.org/ The formative e-assessment project: http://feasst.wlecentre.ac.uk/ This presentation: http://www.slideshare.net/yish/ppw_feasst_sh Yishay Mor people.lkl.ac.uk/yishay yishaym@gmail.com @yishaym Niall Winters www.lkl.ac.uk/naill n.winters@ioe.ac.uk @nwin Harvey Mellar lkl.ac.uk/people/mellar.html [email_address]