The Conversational Framework: An approach to evaluating e-assessment patterns in terms of learning theory Diana Laurillard London Knowledge Lab Institute of Education Scoping a vision of formative e-assessment

A common framework for representing learning theory What does it take to learn (formal learning)? What are the requirements for formative assessment? Evaluating patterns of formative e-assessment

A common framework of representation ‘ Instructivist’ - Teacher-focused Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Concepts Actions Adapt actions Adapt Task practice environment Reflect Task goal Answers

A common framework of representation ‘ Constructionist’ - Practice-focused Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Concepts Actions Adapt actions Adapt Task practice environment Reflect Task goal Reflect Revisions Feedback Practice environment Answers

A common framework of representation ‘ Constructionist’ - Practice-focused Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Practice environment

A common framework of representation ‘ Social constructivist’ - Learner-focused Teacher’s ideas Learner’s ideas Learner’s practice Reflect Other learner(s) Ideas Ideas Practice environment Reflect Others’ practice

A common framework of representation ‘ Collaborative learning’ = ‘Social constructionism’? Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Practice environment Other learner(s) Ideas Ideas Reflect Others’ practice Adapt actions Draft outputs Draft outputs

What does it take to learn? - A Conversational Framework Instructivism + Constructionism + Social constructivism + Collaboration Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Practice environment Other learner(s) Ideas Ideas Reflect Others’ practice Adapt actions Draft outputs Draft outputs Concepts Answers Reflect Questions Outputs

The Conversational Framework An attempt to draw on the learning theories developed over the last century, and encapsulate them in a form that enables educators to test conventional and digital methods against them ( Dewey, Piaget, Vygostsky, Bruner, Papert, Pask, Anderson, Marton, Biggs, Brown, Lave, Wenger, Engestrom...)

Testing the value of digital learning technologies against the Conversational Framework

Lecture, Presentation, Book, Educational television, Audio… Powerpoint, Digital video, Animation, Podcast… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Reflect Questions Outputs Learning through acquisition …

Libraries, Catalogues, Journals, Resource banks… Online resource, Digital library, Website, Search engine… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Reflect Questions Learning through inquiry …

Tutorial, Seminar, Class discussion, Small group discussion… Online conferencing, Forum, Chat room, Discussion board… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Answers Reflect Questions Outputs Learning through discussion …

Problem sheet, practice exercises, project work… Interactive simulation, Spreadsheet, Data analysis tool, Game… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Answers Reflect Questions Outputs Learning through practice … Answers Outputs Feedback Answers Outputs Reflect

Laboratory, Small group work, Fieldwork, Workshop… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Answers Reflect Questions Outputs Learning through collaboration … CSCL environments, Wikis, Second Life, Multiplayer games… Reflect

Essay, program, solution, design, product, performance… Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Other learner(s) Ideas Ideas Reflect Other learner(s) Adapt actions Draft outputs Draft outputs Concepts Reflect Outputs Learning through production … Ppt, Program, Model, Website, Blog, Design, Digital video…

An integrated framework of our collective theories of learning provides a powerful challenge to digital technologies, showing that: To support the learning process fully we need to integrate all the technological capabilities available

The Conversational Framework – Challenging a learning design 1. access explanations and presentations of the theory, ideas or concepts? 2. ask questions about their understanding of the theory, etc, by providing the opportunity for answers from (i) the teacher, or (ii) their peers? 3. offer their own ideas and conceptual understanding, by providing comment on them from (i) the teacher, or (ii) their peers? 4. use their theoretical understanding to achieve a clear task goal by adapting their actions in the light of their understanding, or in response to comments or feedback? 5. repeat practice, by providing feedback on actions that enables them to improve performance? 6. repeat practice, by enabling them to share their trial actions with peers, for comparison and comment? 7. reflect on the experience of the goal-action-feedback cycle, by offering repeated practice at achieving the task goal? 8. discuss and debate their ideas with other learners? 9. reflect on their experience, by having to articulate or produce their ideas, reports, designs, performances, etc. for presentation to their peers? 10. reflect on their experience, by having to articulate or produce their ideas, reports, designs, performances, etc. for presentation to their teachers? How does your pattern of learning activities motivate students to:

Learner’s conception as practice = 20% Learner’s conception = 10% teacher’s Teacher’s conception Other learner’(s) conceptions The theory of the Conversational Framework - the learner’s developing experience Teacher-designed task practice environment Other learner’(s) conceptions as practice Learner’s conception = 20% teacher’s Learner’s conception = 40% teacher’s Learner’s conception as practice = 40% Learner’s conception = 50% teacher’s Learner’s conception as practice = 50% Learner’s conception as practice = 60% Learner’s conception = 60% teacher’s Discursive/theoretical level Practice/practical level “ the best predictor of quality of output is the number of iterations” Theory level Practice level Shares practice attempt Offers answers, ideas Questions, offers ideas Feedback on action Action to achieve goal Shares practice attempt Adapts approach to task to current conception Adapt a Task practice environment for learners’ needs Reflects on feedback Reflects on alternative practice Reflects on learners’ practice Hints, comments Present concepts Task goal Revises action Presents conception as product Questions Adapts approach to task to current conception

Feasst: Formative assessment strategies Strategy: clarifying and sharing learning intentions and criteria for success Strategy: engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding Strategy: providing feedback (extrinsic/intrinsic) that moves learners forward Strategy: activating students as instructional resources for one another, sharing outputs and ideas with peers Strategy: activating students as the owners of their own learning, learner self-regulation;

Strategy: clarifying and sharing learning intentions and criteria for success

Strategy: engineering effective classroom discussions and other learning tasks that elicit evidence of student understanding

Strategy: providing feedback (extrinsic/intrinsic) that moves learners forward

Strategy: activating students as instructional resources for one another, sharing outputs and ideas with peers

Strategy: activating students as the owners of their own learning, learner self-regulation;

Wiliam: Formative assessment concepts Teacher’s ideas Learner’s ideas Practice environment Learner’s practice Actions Adapt actions Adapt Task practice environment Task goal Reflect Revisions Feedback Practice environment Other learner(s) Ideas Ideas Reflect Others’ practice Adapt actions Draft outputs Draft outputs Hints/comments Reflect Questions Outputs Moments of contingency Forms of feedback to the learner Concepts

Task goal Learner’s conception as practice Teacher’s conception Other learner’(s) conceptions The Conversational Framework - the learners’ experience of formative assessment – collaborative version Teacher-designed task Other learner’(s) conceptions as practice Feedback on action Action to achieve goal Shares practice attempt Adapts approach to task Adapt a Task practice environment for learners’ needs Reflects on feedback Reflects on alternative practice Reflects on learners’ practice Revises action Adapts approach to task Shares practice attempt Offers answers, ideas Questions, offers ideas Theory level Practice level Learner’s conception Hints, comments Clarify learning intention Engineering effective discussions, tasks and activities that elicit evidence of learning Providing (intrinsic) feedback that moves learners forward Activating students as learning resources for one another Activating students as owners of their own learning Activating students as owners of their own learning Aspects of formative assessment – Dylan Wiliam Providing (extrinsic) feedback that moves learners forward

Task goal Learner’s conception as practice Teacher’s conception Other learner’(s) conceptions The Conversational Framework - the learners’ experience of formative assessment – individual version Teacher-designed task Other learner’(s) conceptions as practice Feedback on action Action to achieve goal Shares practice attempt Adapts approach to task Adapt a Task practice environment for learners’ needs Reflects on feedback Reflects on alternative practice Reflects on learners’ practice Revises action Adapts approach to task Shares practice attempt Offers answers, ideas Questions, offers ideas Theory level Practice level Learner’s conception Hints, comments Aspects of formative assessment – Dylan Wiliam Clarify learning intention Providing (intrinsic) feedback that moves learners forward Activating students as owners of their own learning Providing (extrinsic) feedback that moves learners forward

The Conversational Framework – Challenging a learning design dummy 1. access explanations and presentations of the theory, ideas or concepts? 2. ask questions about theory, etc, by providing the opportunity for answers from the teacher, or their peers? 3. offer their own ideas and understanding, by providing comment on them from the teacher or their peers? 4. use their understanding to achieve a goal by adapting their actions in the light of understanding, or feedback? 5. repeat practice, by providing intrinsic feedback on actions that enables them to improve performance? 6. repeat practice, by enabling them to share their trial actions with peers, for comparison and comment? 7. reflect on the experience of the goal-action-feedback cycle, by offering repeated practice at achieving the goal? 8. discuss and debate their ideas with other learners? 9. reflect on their experience, by having to articulate or produce their ideas, for presentation to their peers? 10. reflect on their experience, by having to articulate or produce their ideas for presentation to their teachers? How does your pattern of learning activities motivate students to: TryOnceRefineOnce pattern Lack of immediate feedback leads to fossilisation of errors; immediate feedback can hinder effective learning “ Feedback on any remaining errors is given, along with correct answer(s)” “ Students are posed questions to elicit answers that can contain multiple errors. A mark is given which contributes to a percentage of the total mark for the question, along with detailed - yet generic- feedback on the location and type of the errors.)” “ Students are permitted a second attempt in which to refine their answer. No further attempts are permitted.”

Learner’s conception as practice = 20% Learner’s conception = 10% teacher’s Teacher’s conception Applying the Conversational Framework to the TryOnceRefineOnce pattern Teacher-designed task practice environment Learner’s conception = 20% teacher’s Learner’s conception = 40% teacher’s Learner’s conception as practice = 40% “ the best predictor of quality of output is the number of iterations” Theory level Practice level Gaps can be filled by using technology to: Structure peer feedback to increase cycles of iterations Provide computer-based feedback to increase practice iterations Automate hints and comments in response to student actions Technology (‘e’) adds speed of response, iteration, intrinsic feedback (processing), re-use, and orchestration of peer collaboration Reflects on feedback Action to achieve goal Adapt a Task practice environment for learners’ needs Feedback Task goal Revises action Adapts approach to task to current conception

Gaps can be filled by using technology to:

Structure peer feedback to increase cycles of iterations

Provide computer-based feedback to increase practice iterations

Automate hints and comments in response to student actions

Summary The Conversational Framework can be used to evaluate and improve patterns of formative assessment through use of technology against established learning theory and the formative assessment criteria established in the Feasst project.