Scoping a vision for formative e-assessment: Cases and design patterns

Problem I: common language Learners? Design Knowledge in TEL Developers Policy makers Teachers Researchers

Problem II: The void The Prophets will tell you what should be done The Explorers will tell you what they did Current discussion of learning and technology alternates between the abstract theoretical and the anecdotal. In between there is a shortage of design-level discourse. ?

Wanted: a design science of learning A science of design has - A value dimension A functional axis of decomposition Attention to representation (Mor & Winters, 2007)‏ Herbert Simon (1969): we need a scientific study of the man-made. At its core, the science of design. “ everyone designs who devises courses of action aimed at changing existing situations into desired ones ”

A science of design has -

A value dimension

A functional axis of decomposition

Attention to representation

(Mor & Winters, 2007)‏

The Design Knowledge Problem Expert := one who solves problems in a particular domain Expert := has domain design knowledge Experts do, Consultants talk Experts talk in jargon But..

Solution: sharing stories Stories (narratives) are a fundamental form of generating / sharing knowledge. (Bruner)‏ Thick descriptions of problems & solutions. Everyone likes a good story. http:// www.slideshare.net/yish/case -study-how-to-presentation

Stories (narratives) are a fundamental form of generating / sharing knowledge. (Bruner)‏

Thick descriptions of problems & solutions.

Everyone likes a good story.

But.. Narratives are not enough: The Aha! Factor How do we identify the key design element in a story? The fantasy factor How do we know its true? The (cognitive) load factor The world is changing too fast for us to take in all the good stories.

Narratives are not enough:

The Aha! Factor

How do we identify the key design element in a story?

The fantasy factor

How do we know its true?

The (cognitive) load factor

The world is changing too fast for us to take in all the good stories.

Design patterns [describe] a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice (Alexander et al., 1977) ‏ C o n t e x t Problem Solution

Problem Keep the rain out Context Cold, wet, poor. Method of solution Thatched roof Related Timber frame, Slanted roof, Chimney

example: activity nodes Design problem Community facilities scattered individually through the city do nothing for the life of the city. Design solution Create nodes of activity throughout the community, spread about 300 yards apart. http://www.uni-weimar.de/architektur/InfAR/lehre/Entwurf/Patterns/030/ca_030.html

Scenarios The ultimate proof of a pattern language is in its effectiveness as a tool for design. Ask participants to tell as “fantasy story”: a current design challenge as an “I wish” case story. Apply patterns to derive solution.

The ultimate proof of a pattern language is in its effectiveness as a tool for design.

Ask participants to tell as “fantasy story”: a current design challenge as an “I wish” case story.

Apply patterns to derive solution.

A few cases Creature of the week CoMo Post 16 String Comparison Open Mentor Academic writing Audio files CLICK ...

Creature of the week

CoMo

Post 16 String Comparison

Open Mentor

Academic writing

Audio files

CLICK

...

Creature of the week Situation large class (138) first and second year computer science students. assignment: create a virtual pet in Second Life. Task Engage and motivate the students show examples of good work which others could learn from show students their work is valued. build a sense of community.

Situation

large class (138)

first and second year computer science students.

assignment: create a virtual pet in Second Life.

Task

Engage and motivate the students

show examples of good work which others could learn from

show students their work is valued.

build a sense of community.

CoMo Situation Royal Vet College. Hospital rotations as part of their training. Task Allow students to capture critical incidents in text and image. Support sharing of clinical experiences and co-reflection.

Situation

Royal Vet College.

Hospital rotations as part of their training.

Task

Allow students to capture critical incidents in text and image.

Support sharing of clinical experiences and co-reflection.

Post 16 string comparison Situation Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students. Sixth Form level, grammatical consolidation and whole-sentence translation. Task Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.

Situation

Grammar school been piloting the ‘string comparison’ approach to language teaching at post-16 for AS and A2 level students.

Sixth Form level, grammatical consolidation and whole-sentence translation.

Task

Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.

Solution A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.

A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.

Open mentor

A few patterns.. Try Once, Refine Once Feedback on Feedback Classroom display Use my Stuff Round and Deep Showcase Learning Three Hats

Try Once, Refine Once

Feedback on Feedback

Classroom display

Use my Stuff

Round and Deep

Showcase Learning

Three Hats

Try Once, Refine Once http://patternlanguagenetwork.myxwiki.org/xwiki/bin/view/Patterns/TryOnceRefineOnce

Problem Lack of immediate feedback for students leads to fossilisation of errors and misconceptions providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.

Context Class size Large (30-300)‏ Content Skills facts Mode of instruction Blended / on-line. Computer tested.

Class size

Large (30-300)‏

Content

Skills facts

Mode of instruction

Blended / on-line. Computer tested.

Solution

Feedback on Feedback http://patternlanguagenetwork.myxwiki.org/xwiki/bin/view/Patterns/FeedbackonFeedback

Problem Good feedback should - Alert learners to their weaknesses. Diagnose the causes and dynamics of these. Include operational suggestions to improve the learning experience. Address socio-emotive factors. Tutors know this, but are pressed for time. Or not aware of their feedback strategies Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching

Good feedback should -

Alert learners to their weaknesses.

Diagnose the causes and dynamics of these.

Include operational suggestions to improve the learning experience.

Address socio-emotive factors.

Context Large scale, technology supported, graded courses many tutors instructing many students. Feedback is mediated by technology that allows it to be captured and processed in real time Topic of study is subject to both grading and formative feedback.

Large scale, technology supported, graded courses

many tutors instructing many students.

Feedback is mediated by technology that allows it to be captured and processed in real time

Topic of study is subject to both grading and formative feedback.

Solution Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms. In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.

Embed a mechanism in the learning and teaching system that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms.

In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.

Classroom Display http://patternlanguagenetwork.myxwiki.org/xwiki/bin/view/Patterns/Classroomdisplay

Problem Rewards participation. Relates to learner's personal experiences. Window on student conceptions . Using learner generated content.. Needs to collate works in a single easy to access location. Learners uncomfortable about presenting their work in public Legal or other restrictions on sharing work.

Rewards participation.

Relates to learner's personal experiences.

Window on student conceptions .

Needs to collate works in a single easy to access location.

Learners uncomfortable about presenting their work in public

Legal or other restrictions on sharing work.

Context Class size: Small / medium (6-60)‏ Mode of instruction: Blended (preferable)‏ Time frame Continuous, over a period Pedagogy Involves construction / media production

Class size:

Small / medium (6-60)‏

Mode of instruction:

Blended (preferable)‏

Time frame

Continuous, over a period

Pedagogy

Involves construction / media production

Solution

Task 1: highlight concepts Hazard: needs definition Asset: Key common concept

Task 2: design game – add formative assessment to e-Learning Agree on a scenario (who, what, where)‏ Each team member takes on a role (teacher, student, mgmt, admin)‏ State desires Find appropriate patterns Propose solution

Agree on a scenario (who, what, where)‏

Each team member takes on a role (teacher, student, mgmt, admin)‏

State desires

Find appropriate patterns

Propose solution

A scenario is.. Preferably, a real problem you are familiar with. e.g. extending a system you have in development / in use Define the context of use Environmental constraints Define the primary roles and workflows What is the problem to be solved?

Preferably, a real problem you are familiar with.

e.g. extending a system you have in development / in use

Define the context of use

Environmental constraints

Define the primary roles and workflows

What is the problem to be solved?

Coming up..

Deadline: December 23rd 2008 [email_address] http://www.iwm-kmrc.de/workshops/e-learning-patterns/

Review on formative e-Assessment Paper on the CoMo case study Symposium on pattern methodologies http://www.cal-conference.elsevier.com/

Review on formative e-Assessment

Paper on the CoMo case study

Symposium on pattern methodologies

http://www.cal-conference.elsevier.com/

July 8-12, 2009, Irsee Monastery, Bavaria Deadline : February 14, 2009 http://www.hillside.net/europlop/ http://flickr.com/photos/ademaraguiar/