More Related Content
Similar to Testa@Greeniwch - institutional approach to improving feedback and assessment practices using tESTA
Testa@Greeniwch - institutional approach to improving feedback and assessment practices using tESTA
- 1. TESTA@Greenwich Institutional approach toimproving feedback and assessment practices using TESTA at the University of Greenwich Monika Pazio Duncan McKenna
- 2. • Nationally andinternationally adopted • Mixed methods approach • Programme level assessment Original TESTA
- 3. Scale University of Greenwich% Satisfied Sector/Nationa l % Satisfied 2013 2014 2013 2014 Teaching 86 85 86 87 Assessment and Feedback 73 74 72 72 Academic Support 80 82 80 81 Organisation and Management 78 79 78 78 Learning Resources 85 87 84 85 Personal Development 84 84 82 82 Overall Satisfaction 85 86 85 86 University of Greenwich
- 4. • Aim –assessing suitability and effectiveness of TESTA for the UoG context • March – July 2014 • Target – programmes approaching validation • 8 programmes participated, 3 fully completed • Senior management involvement in nomination • Cross university SCA selection, scholarship (5 x £1000) • Followed the original TESTA process Pilot
- 5. Completion rates Pilot Research Component A BC D E F G H AEQ 80/110 students (71%) 36/112 students (32%) 26/70 Students (37%) 40/312 students (13%) 39/245 students (16%) 50/ 157 students (32%) 20/ 92 students (22%) 55/ 148 students (37%) Focus Groups 2 (3 and 4 students) 2 (6 and 8 students) 1 (2 students) 1 1 interview (1 x 3 1 interview 1x1)
- 6. The process Programmes appointedby the DLTs Initial meeting with the team questionnaires Focus groups audit Report writing Report presentation
- 7. • The process •Participant experience • Project team experience • Broader assessment profile • Effectiveness Evaluation
- 8. Evaluation – Theprocess • Timing • Difficulty is accessing staff and students • Difficulties with focus groups • Lack of engagement with programme teams –’why me?’ attitude • Low SCA engagement • Patchy workload
- 9. Evaluation – Theprocess Methodology • The questionnaire (the scales, the questions and ‘neutral’) • The focus on quantity of feedback without acknowledging the importance of quality - The nature of volume of oral feedback
- 10. Evaluation – Participantexperience “it’s very easy, I was worried when I agreed to it thinking what else do they want me to do but it didn’t require a lot of work from me” “Also thank you to you and Monica for making the evaluation process as smooth as it was.” “The findings provided a useful mapping of assessment practices that confirmed what we already suspected in a quantified manner. As such, it helped to inform the programme level assessment design.” “Overall we have reduced the number of assessments as well as reducing the reliance on purely written forms.”
- 11. • Low Engagement •Money not a long term motivator • Lack of knowledge of the nature of the programme BUT • Online presence • Resource development • Help with data analysis Evaluation – Project Team Student Change Agents
- 12. Evaluation – A&Fprofile • 33% students directly exposed (382 participants) • Over 1300 blog views • Over 260 video views • Summer of Innovation entry – Feed 360 (298 votes from 33 institutions)
- 13. • difficult tosay but…. • it makes PL reconsider assessment • triggers discussion about assessment design • provides data to feed into validation documentation • provides evidence for the PL to implement changes Evaluation - effectiveness
- 14. • Programmes contacted18 months prior to validation • AEQ - Change to scales and questions, distributed by the staff or SCA • Audit – Keele approach to audit, additional quality of feedback analysis, verbal vs written feedback ratio • SCA – recruited from the programme, non-monetary rewards, training programme and resources, personalised approach • No reports – detailed presentations TESTA@Greenwich changes to the process
- 15. • TESTA acceptedas a recommended/ mandatory tool for review • Recommendations inform changes • Springboard for further projects – TEL, students as producers Institutional impact
- 16. Institutional implementation Sustainability Centrally managed • Projectofficer • Responsibility for data collection on the team • Report • Team providing a service Centrally supported • Better cooperation between the team and the PL • PL participates in the data collection • Team helping • A presentation
- 17.
- 18. Thank you Contact details: MonikaPazio m.pazio@gre.ac.uk
Editor's Notes
- #3 A word of introduction to the main project: Designed by Winchetster team – Jessop and Gibbs Looks at programme level assessment as opposed to course levels Mixed methods approach – 3 components – audit, student focus groups and questionnaires Aims to contrast staff and students vie, identify pressure points and give recommendation to programme assessment design Trial with 4 universities Proved to be successful, adpated nationally and internationally – taking over the world
- #4 Some background to our context: Problem with feedback and assessment Statistics reflect the national trend, there is a need for improvement cross institutionally Some programmes do better than others, need for improvement, showing what works and what doesn’t Decisoin to pilot TESTA – project officer hired for that purpose
- #5 Aim – we did that in terms of the suitability of the tools The pilot started in march, when the project officer was hired, the cooperation with the programme teams began at the end of March, and for some didn’t start till May We targeted [rogrammes apporaching validation and asked the DLTs to nominate one or two from each faculty Overall we had 8 programmes, 3 completed
- #6 The table presents completion rates according to the programmes The highest one we had was 70% due to a very good cooperation with the programme team Programme G – 1 focus group 1 interview DEFHG process was not completed, for D and E we provided interim reports
- #8 Effectiveness was a tricky one but I’ll talk about this later
- #9 The major issue was the timing. We started late in the term which meant we needed to move quickly. Lack of cooperation with some programme teams resulted in process on those porgrammes starting very late and limited access to students. The timing also affected organisation of focus groups. While we could send online invitation to AEQs once the students left the campus there was no way of involving them into FG. We tried online, bribery with vouchers but it still had very little success. What postponed the process was reluctance to get involved from some porgramme teams. This is related to how the selection was organised. We asked DLTs to nominate a programme, because of that there was this feeling of ‘oh no I’m in trouble’ and TESTA was treated as a tool to punish and uncover everything that is wrong with the programme and report back to the management. This result in resistance. Due to timing SCA enagement was also relatively low. While they did participate the 3 hours per week they were paid for never happened. This was due to the fact that again the timing of the project collided with their assessment period. The majority of work had to be done when they were busy, when they had free time, there was no data collection/analysis tasks for them to perform. This meant more workload for the two of us.
- #10 The questionniare – we found that the questions wer repetitive and there was a number of questions that really refereed to one aspect of assessment and there was lack of questions that would quantitatively deal with for example variety of assessment, loading and bunching – issues important for Greenwich the audit focusd on the quantity of feedback, we found that it’s not so much the quantity that is important but the quality, that aspect was missing Oral feedback – again very problematic, relying on an estimate that was usually very high and difficult to check, also what to do with whole class feedback?
- #11 Easy – no additional workload Good source of information for validation process Plans for/ interest in making changes
- #12 The major issue was the timing. We started late in the term which meant we needed to move quickly. Lack of cooperation with some programme teams resulted in process on those porgrammes starting very late and limited access to students. The timing also affected organisation of cous groups. While we could send online invitation to AEQs once the students left the campus there was no way of involving them into FG. We tried online, bribery with vouchers but it still had very little success. What postponed the process was reluctance to get involved from some porgramme teams. This is related to how the selection was organised. We asked DLTs to nominate a programme, because of that there was this feeling of ‘oh no I’m in trouble’ and TESTA was treated as a tool to punish and uncover everything that is wrong with the programme and report back to the management. This result in resistance. Due to timing SCA enagement was also relatively low. While they did participate the 3 hours per week they were paid for never happened. This was due to the fact that again the timing of the project collided with their assessment period. The majority of work had to be done when they were busy, when they had free time, there was no data collection/analysis tasks for them to perform. This meant more workload for the two of us.
- #13 We did pretty good Despite the very bad timing we managed to get 33% of students on the programmes exposed the blog had 1300 views which meant more and more people were interested in TESTA, we also had questionnaire responses from porgrammes outside of our chosen 8
- #15 We’ve come to the end of roun1 of the game and taking ino account the issues we encoutered we introduce the following changes which essentially make TESTA@Greenwich:
- #16 Good feedback from the participants and the management mean that testa has been growing, it is now mandatory for 2 faculties to use in preparation for review Recommendations produced in the report were used to make changes, for examples some programmes reduced the no of assessment, some introduced greater variety, some amended their peer feedback process It also serves as a sprngboad for further projects to implement recommendations for example audio feedback etc; ideally we would want to involve students in curriculum co-design which is currently happening in one of the programmes where students are asked to design their ideal assessment diet, we want to introduce projects with student involvement across the university and testa gives a great opportunity for that
- #17 Institutional implementation means greater workload to a very small team of 2. Hence when thinking about sustainability and institutional implementation the model of current testa process had to be changed. We examined how other universities approached testa and we found that the models tend to be either centrally supported by educational development units, which means like in our case that there is a project officer that does the job for the programme leader, the PL involvement I minimal, it’s like a service that is provided, on the oter side of the spectrum were universities which offered no support, i.e. the resonsibility for testa, in whatecer way it was delieverd (in some cases only audit) was the PL’s responsibility. We decided to position ourselves in the middle of those two approaches, and support the process centrally. This means that we support the PL rather than do the job for them, this means greater engagement of the programme team in the process, and the final result is only produced in the form of a presentation to be shared with others