The document discusses technology supported assessment and feedback. It outlines the REAP project which took a principled approach to redesigning courses using technology. The project involved 6000 students across 19 pilots at 3 universities. The pilots showed benefits like reduced workload and improved learning outcomes. Formative assessment and feedback are important for student learning but current practices are unsatisfactory. The REAP project aimed to address this by embedding seven principles of good feedback practice into technology-supported assessment activities.

1. Martin Hawkseye-Learning Advisor (Higher Education)Technology supported assessmentIncreasing learner success with technology supported assessmentApril 28, 2011| slide 1

2. PlanDefining assessment and feedbackOutline the REAP Project and its philosophy towards a ‘principled’ approach to course redesign Some examples of REAP pilotsQuestions and discussionTechnology supported assessmentApril 28, 2011| slide 2

3. Defining assessment and feedbackAssessment – the narrow meaning is exam for external accreditationFeedback – the narrow sense is what the tutor writes about/on a finished piece of student workREAP was also interested in:Self-assessment and reflection - a form of internally generated feedbackPeer dialogue as feedback and peer assessment Technology supported assessmentApril 28, 2011| slide 3

4. DefinitionsWho is involved in formative assessment and feedbackTutorPeersExternal (e.g. placement supervisor)Computer generatedSelfTechnology supported assessmentApril 28, 2011| slide 4

5. Why take formative assessment and feedback seriously?Assessment is a key driver of student learningAssessment is a major cost in HE (staff time)Widely reported that students don’t read the feedbackDropout/retention – linked to academic experienceFirst year experience – students need regular and structured feedback opportunities.National student survey (NSS) – students are dissatisfied with feedback.QAA reports – main area of criticism in EnglandTechnology supported assessmentApril 28, 2011| slide 5

6. NSS 2007 - Assessment and Feedback ResultsTechnology supported assessmentNationally only 55% of students think feedback is prompt and had helped to clarify things they did not understand [Scotland: 48%]

7. Nationally only 63% of students agree that have received detailed comments on their work [Scotland: 49%]April 28, 2011| slide 6

8. Key messagesFormative assessment and feedback by others can only have an impact on learning when it influences a student’s own self-regulatory processes (adapted from Boud, 1995)Students are already self-assessing and generating feedback so we should build on this capacity (Nicol and Macfarlane-Dick, 2004)Technology supported assessmentApril 28, 2011| slide 7

9. REAP3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University)Targeting large 1st year classesMulti-disciplinary as well as faculty-wide (19 pilots, ~6000 students)Range of technologies: online tests, simulations, discussion boards, e-voting, e-portfolios, peer/feedback software, admin systems, VLEs, offline-onlineA ‘principled’ approach to designing and embedding assessment practicesTechnology supported assessmentApril 28, 2011| slide 8

10. Scaffolding self regulation: 7 principles of good feedback (assessment design)Clarify what good performance is (goals, criteria, standards).Facilitate reflection and self-assessment in learning Deliver high quality feedback to students: feedback that enables students to monitor and self-correctEncourage peer and tutor dialogue around learningEncourage positive motivational beliefs & self esteem through assessmentProvide opportunities to close the feedback loopUse feedback information to shape teachingSource: Nicol and Macfarlane-Dick (2006)Technology supported assessmentApril 28, 2011| slide 9

11. Two super principlesSuper-principle 1: developing learner self-regulation (empowerment) i.e steers to encourage ownership of learning – the seven principles discussed above.Super-principle 2: time on task and effort (engagement) i.e. steers on how much work to do and when – Gibbs and Simpson 4 conditionsCase examples from REAP – applying these principles/conditionsTechnology supported assessmentApril 28, 2011| slide 10

12. REAP Pilots (1)Department of Mechanical EngineeringStudents: 250Technology: Commercial online homework packages, electronic voting system (EVS)Assessment Activities: Weekly tests provide on-demand feedback on student performance for both students and tutors. Just-in-time teaching supported by interactive classes using EVS. Efficiencies: 60% reduction in assessment workload saving 102 staff hours. License cost of commercial packages £12.95 and £7 per student per annum. Learning Gains: Results from 2006/7 diet indicate strong class attainment maintained (90% pass rate, 65% average) "I think it’s managed to save a lot of time for ourselves and the tutors and given them more time to develop what they are going to talk about and give more time for them to speak to people individually if they need it." Student comment Technology supported assessmentApril 28, 2011| slide 11

13. REAP Pilots (2)Department of PsychologyStudents: 560Technology: Online collaborative group tasks supported by VLE message-boardAssessment Activities: Regular collaborative tasks support peer feedback processes and student engagement. Efficiencies: 50% of lectures replaced with online tasks. Staff time re-directed to support online tasks. Learning Gains: Significant overall improvement in average exam pass mark (51.1% in 2005/06 diet rising to 57.4% in 2006/07). Exam failure rate reduced from 13% to 5%. Course failure rate reduced from 12.1% to 2.8%. Technology supported assessmentApril 28, 2011| slide 12

14. September 2007JISC CETIS Assessment SIGREAP Pilots (3)Department of Hospitality & Tourism ManagementStudents: 200Technology: Podcasts, electronic voting system Assessment Activities: Regular podcast releases support changes to lectures to include interactive discussion using EVS.Efficiencies: 50% reduction in lectures (however, lectures delivered twice to smaller groups).Licensing and development costs associated with podcasts (c. £1k per 30 minutes podcast)Learning Gains: Significant gain in overall exam mark in 2006/07 diet (+12.2%) compared with 2005/06. Significant reduction (-25%) in students receiving non-qualification.

15. September 2007JISC CETIS Assessment SIGREAP Pilots (4)Department of Modern LanguagesStudents: 200Technology: Online homework and tests supported by VLE, electronic voting system (EVS)Assessment Activities: Diagnostic tests of student knowledge at start of year to inform teaching.Formative feedback from regular online testing and EVS classroom use. Efficiencies: Effective delivery of course made possible despite significant cuts in funding and staffing. Tutorials reduced by 50% and replaced with online tasks. Listening classes reduced from 360 to 160 hours. Saving of 200 staff hours. Learning Gains: Failure rate in final exam reduced from 24% to 4.6% compared with 2005/06 diet. "Having almost immediate feedback on marks was very useful as I was aware at every point as to how well I was coping" Student comment

16. September 2007JISC CETIS Assessment SIGREAP Pilots (5)School of Pharmacy - Pharmacy Practice 3Students: 240Technology: Online simulation tutorial Assessment Activities: Regular online tasks to improve student engagement, multiple opportunities for self-testing and regular feedback. Efficiencies: Savings in staff time due to reduced need for remedial revision work with individual students. Learning Gains: Significant gain in overall exam mark in 2006/07 (+16%) compared to 2005/06 diet. “The tutorial was an excellent resource and learning tool to supplement our class” Simulated exactly what it would be like to carry out a check on a prescription, allowing us to experience the difficulties involved and discover where we needed improvement” Student comments

17. REAP Pilot (6)School of Pharmacy - Foundation PharmacyStudents: 120Technology: e-portfolio, electronic feedback formAssessment Activities: Improved tutor feedback to students supported by feedback form, enhanced opportunities for reflection activities.Efficiencies: Staff workload increased during test implementation but reductions anticipated in future years.Learning Gains: No significant improvements in academic performance reported. 89% of students receiving feedback via the feedback form agreed the feedback was ‘useful’ or ‘very useful’. “The feedback form was especially useful, I found it easier to work from as it was segmented into the different aspects of the report I had written and had comments on both the strong and weak elements of my report” Student commentTechnology supported assessmentApril 28, 2011| slide 16

18. Slight detour A more detailed exampleAudio/Video FeedbackTechnology supported assessmentApril 28, 2011| slide 17

19. Question for youQuestion: What are the opportunities for change at UHI?Technology supported assessmentApril 28, 2011| slide 18

20. ChangeChallenge to ChangeJISC Typology of technology use in assessment and feedback doc http://jiscdesignstudio.pbworks.com/f/typology_tech_assess.docTechnology supported assessmentApril 28, 2011| slide 19

21. Do you have the policies in place to back change?Strathclyde Policies http://www.strath.ac.uk/staff/policies/academic/ and Feedback is a Dialogue site http://www.strath.ac.uk/learnteach/teaching/staff/assessfeedback/University of Dundee Online Assessment Policy and Procedureshttp://www.dundee.ac.uk/academic/learning/archive/200910/mtg2/papero.rtfTechnology supported assessmentApril 28, 2011| slide 20