An evidenced-informed approach to     enhancing programme-wide            assessment             TESTA to FASTECHDr Tansy ...
Pre-Conference ActivitiesPre-reading:1) Gibbs & Simpson (2004) Conditions under whichassessment supports student learning....
Pre-conference questions1) What conditions do you see as most   important in student learning (Paper 1)?2) What is your re...
TESTA ‘Cathedrals Group’ Universities
Why TESTA has been compelling1) The research methodology2) It is conceptually grounded in assessment and   feedback litera...
Presentation Overview1) The Research Methodology (Tansy)2) Case study as a compelling narrative (Tansy)3) Trends in assess...
Two Paradigms          Transmission              Social constructivist model• Expert to novice                  • Particip...
1) Research Methodology•   triangulates data from three sources•   presented in a case study•   complex, ambiguous, textur...
Programme Audit•   How much summative assessment•   How much formative (reqd, formal, feedback)•   How many varieties of a...
Assessment Experience Questionnaire            version 3.3•   28 questions•   5 point Likert scale where 5 = strongly agre...
Focus groups• What kinds of assessment• How assessment influences your study  behaviour• Whether you know what quality wor...
Research Methodology                         ASSESSMENTPROGRAMME AUDIT          EXPERIENCE    (n=22)             QUESTIONN...
2) The cases are surprising, complex, puzzling    Here is one case from the TESTA data……
Case Study 1• Lots of coursework (47 tasks)• Very varied forms (15 types of assessment)• Very few exams (1 in every 10)• M...
But students:• Don’t put in a lot of effort and distribute their  effort across few topics• Don’t think there is a lot of ...
Your best guessesA.   Variety of assessment confuses studentsB.   Assessment in ‘bunched’ at certain timesC.   The feedbac...
What is going on?•   Teachers work hard, students less so.•   Feedback is too late to be useful•   Teachers have varied st...
3) Trends in assessment and feedback•   High summative assessment, low formative•   High variety (average 11; range 7-17)•...
Q&A
4) The effort narrative. TESTA data shows that:•   average of 12 summative per year•   24 teaching weeks, one every two we...
The more you write the better you become at it…and if we’ve only written 40 pieces over three yearsthat’s not a lot.So you...
TESTA plus HEPI quizWhich one is false?A) 1 in 3 UK students study for 20 hours or less a weekB) Students on only 1 out of...
Chat boxWhat ideas might encourage students to put in  effort regularly on degree programmes?• Type your responses in the ...
Strategies to encourage student effortChoose your top strategy to encourage effort:A) Raise expectations in first yearB) R...
Technologies that may help… What technologies might work to  spur on regular and distributed              effort?Type your...
5) The baffled student narrativeo The language of written criteria is difficult  to understando feedback does not always r...
I’m not a marker so I can’t really think like them... I don’thave any idea of why it got that mark.They have different cri...
Chat Box  What strategies might help students to    internalise goals and standards?• Type your responses in the text chat
Strategies to help students know what                ‘good’ isWhich strategy do you think helps most?A) Showing students m...
6) System-wide features make it difficult for students touse feedback and act on ito feedback often arrives after a module...
It’s rare that you’ll get it in time to help you on that same module.t’s rare that you’ll get it in time to help you on th...
Changes through TESTA        Structural        Thematic        Pedagogic         Module
Types of changes1. Reduced summative2. Increased formative assessment3. Streamlined variety4. Raised expectations of stude...
www.testa.ac.uk
Q&A
FASTECHFeedback and Assessment for Students with TechnologyWhat is FASTECH?•   R&D Project (3 yrs): ‘R’ primarily with TES...
FASTECH: a Pedagogical Goal                     … ability to manage own learning …                     In each assessment ...
Finally,                             for an excellent overview                         of technologies and pedagogiesJISC,...
DISCUSSIONto be continued in the conference discussion forumHow do you think using technology in A&F will        improve s...
ReferencesBlack, P. & D. William (1998) ‘Assessment and ClassroomLearning’, Assessment in Education: Principles, Policy an...
Jessop, T., El Hakim, Y. & Gibbs, G. (2011) TESTA: Research inspiringchange, Educational Developments 12 (4). In press.Jes...
Upcoming SlideShare
Loading in …5
×

TESTA to FASTECH Presentation

622 views

Published on

JISC Annual Online Expert Session, 22 November, 2011.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
622
On SlideShare
0
From Embeds
0
Number of Embeds
49
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 22 x 18 = 400 ish modules etc
  • TESTA to FASTECH Presentation

    1. 1. An evidenced-informed approach to enhancing programme-wide assessment TESTA to FASTECHDr Tansy Jessop & Yaz El Hakim, University of Winchester Professor Paul Hyland, Bath Spa University JISC Online Annual: 22 November 2011
    2. 2. Pre-Conference ActivitiesPre-reading:1) Gibbs & Simpson (2004) Conditions under whichassessment supports student learning.http://www2.glos.ac.uk/offload/tli/lets/lathe/issue1/articles/simpson.pdf2) Gibbs, G. & Dunbar-Goddet, H. (2007) The effects ofprogramme assessment environments on student learning.http://www.heacademy.ac.uk/assets/documents/teachingandresearch/gibbs_0506.pdf3) Jessop, T., Smith, C. & El Hakim, Y. (2011) Programme-wide assessment: doing ‘more with less’ from the TESTANTFS project. HEA Assessment & Feedback Briefing Paper.http://www.heacademy.ac.uk/assets/documents/assessment/2011_Winchester_SS_Briefing_Report.pdf
    3. 3. Pre-conference questions1) What conditions do you see as most important in student learning (Paper 1)?2) What is your response to the idea of institutional and programme ‘assessment environments’ which influence assessment and feedback patterns? (Paper 2)3) What are the main challenges and benefits of addressing assessment patterns on a whole programme? (Paper 3)
    4. 4. TESTA ‘Cathedrals Group’ Universities
    5. 5. Why TESTA has been compelling1) The research methodology2) It is conceptually grounded in assessment and feedback literature3) It’s about improving student learning4) It is programmatic in focus5) The change process is dialogic & developmental
    6. 6. Presentation Overview1) The Research Methodology (Tansy)2) Case study as a compelling narrative (Tansy)3) Trends in assessment & feedback (Tansy) Q&A4) The student effort narrative (Yaz)5) The bewildered student narrative (Yaz)6) Systems-failure on feedback narrative (Yaz) Q&A7) A way forward: FASTECH (Paul)
    7. 7. Two Paradigms Transmission Social constructivist model• Expert to novice • Participatory, democratic• Planned, packaged & ‘delivered’ • Messy and process-oriented• Feedback given by experts • Peer review• Feedback received by novices • Self-evaluation• One way traffic • Social process• Very little dialogue • Dialogue• Emphasis on measurement • Emphasis on learning• Competition • CollaborationMetaphor = mechanical system Metaphor = the journey
    8. 8. 1) Research Methodology• triangulates data from three sources• presented in a case study• complex, ambiguous, textured• open to discussion - not the ‘final word’• ‘before’ and ‘after’ data
    9. 9. Programme Audit• How much summative assessment• How much formative (reqd, formal, feedback)• How many varieties of assessment• Proportion exams to coursework• Word count of written feedback• How much ‘formal’ oral feedback• Criteria, learning outcomes, course docs
    10. 10. Assessment Experience Questionnaire version 3.3• 28 questions• 5 point Likert scale where 5 = strongly agree• 9 scales and one overall satisfaction question• Scales link to conditions of learning• Examples: – quantity and distribution of effort; – use of feedback; – quantity and quality of feedback; – clear goals and standards
    11. 11. Focus groups• What kinds of assessment• How assessment influences your study behaviour• Whether you know what quality work looks like• What feedback is like and how you use it
    12. 12. Research Methodology ASSESSMENTPROGRAMME AUDIT EXPERIENCE (n=22) QUESTIONNAIRE (AEQ n= 1200+) FOCUS GROUPS (n=50 with Programme 301 students) Team Meeting
    13. 13. 2) The cases are surprising, complex, puzzling Here is one case from the TESTA data……
    14. 14. Case Study 1• Lots of coursework (47 tasks)• Very varied forms (15 types of assessment)• Very few exams (1 in every 10)• Masses of written feedback on assignments (15,412 words)• Learning outcomes and criteria clearly specified….looks like a ‘model’ assessment environment
    15. 15. But students:• Don’t put in a lot of effort and distribute their effort across few topics• Don’t think there is a lot of feedback or that it very useful, and don’t make use of it• Don’t think it is at all clear what the goals and standards are……what is going on?
    16. 16. Your best guessesA. Variety of assessment confuses studentsB. Assessment in ‘bunched’ at certain timesC. The feedback is too late to be of any useD. Teachers don’t share a common standardE. Other• Select your response from the buttons (A B C D E) at the bottom-right of the list of participants• Type any additional comments into the text-chat
    17. 17. What is going on?• Teachers work hard, students less so.• Feedback is too late to be useful• Teachers have varied standards• Students see feedback as ‘modular’• Variety confuses students• Formative tasks are assigned low priority• Summative assessment drives effort
    18. 18. 3) Trends in assessment and feedback• High summative assessment, low formative• High variety (average 11; range 7-17)• Written feedback (ave7,153; r = 2,869-15,412 )• Low oral feedback (average 6 hours)• Watertight documents, tacit standards• Huge institutional and programme variations:o formative: summative ratios (134:1 cf 1:10)o oral feedback (37 minutes to 30 hours)
    19. 19. Q&A
    20. 20. 4) The effort narrative. TESTA data shows that:• average of 12 summative per year• 24 teaching weeks, one every two weeks• summative tasks end-loaded & bunched• leading to patchy effort• and surface learning• with an average three formative tasks a year….
    21. 21. The more you write the better you become at it…and if we’ve only written 40 pieces over three yearsthat’s not a lot.So you could have a great time doing nothing untillike a month before Christmas and you’d suddenlypanic. I prefer steady deadlines, there’s a gradualmove forward, rather than bam!In the second year, I kept getting such good marks Ithought “If I’m getting this much without putting inmuch effort that means I could do so much better ifI actually did do the hours” but it just goes up anddown really.
    22. 22. TESTA plus HEPI quizWhich one is false?A) 1 in 3 UK students study for 20 hours or less a weekB) Students on only 1 out of 7 TESTA programmes agreed that they were working hardC) Students work hardest when there is a high volume of formative assessment and oral feedbackD) Students work hardest when there is a high volume of summative assessment and written feedbackE) 1 in 3 UK students undertake > 6 hours of paid work a weekSelect your response from the buttons (A B C D E) at thebottom-right of the list of participants
    23. 23. Chat boxWhat ideas might encourage students to put in effort regularly on degree programmes?• Type your responses in the text chat
    24. 24. Strategies to encourage student effortChoose your top strategy to encourage effort:A) Raise expectations in first yearB) Require more formative assessmentC) Link formative and summative tasksD) Use more peer and self assessmentE) Design small, frequent assessed tasksSelect your response from the buttons (A B C D E) at thebottom-right of the list of participants
    25. 25. Technologies that may help… What technologies might work to spur on regular and distributed effort?Type your responses in the textchat
    26. 26. 5) The baffled student narrativeo The language of written criteria is difficult to understando feedback does not always refer to criteriao students feel that marking standards vary and are subjective and arbitraryo students sometimes use criteria instrumentally
    27. 27. I’m not a marker so I can’t really think like them... I don’thave any idea of why it got that mark.They have different criteria, build up their own criteria.Some of them will mark more interested in how you wordthings.You know who are going to give crap marks and who aregoing to give decent marks.
    28. 28. Chat Box What strategies might help students to internalise goals and standards?• Type your responses in the text chat
    29. 29. Strategies to help students know what ‘good’ isWhich strategy do you think helps most?A) Showing students models of good workB) Peer marking workshopsC) Lots of formative tasks with feedbackD) Plenty of interactive dialogue about standardsE) Self assessment activitiesSelect your response from the buttons (A B C D E) at thebottom-right of the list of participants
    30. 30. 6) System-wide features make it difficult for students touse feedback and act on ito feedback often arrives after a module, or after submission of the next tasko tasks are not sequenced or connected across modules, leading to lack of feed forwardo students sometimes receive grades electronically before their feedback becomes available on parchment in a dusty officeo technology has led to some depersonalised cut and pasting
    31. 31. It’s rare that you’ll get it in time to help you on that same module.t’s rare that you’ll get it in time to help you on that same module.You know that twenty other people have got the same sort ofcomment.I look on the Internet and say ‘Right, that’s my mark. I don’t needto know too much about why I got it’.I only apply feedback to that module because I have this fearthat if I transfer it to other modules it’s not going to transfersmoothly.You can’t carry forward most of the comments because youmight have an essay first and your next assignment might be aposter.
    32. 32. Changes through TESTA Structural Thematic Pedagogic Module
    33. 33. Types of changes1. Reduced summative2. Increased formative assessment3. Streamlined variety4. Raised expectations of student workload5. Sequenced and linked tasks across modules6. Practice based changes
    34. 34. www.testa.ac.uk
    35. 35. Q&A
    36. 36. FASTECHFeedback and Assessment for Students with TechnologyWhat is FASTECH?• R&D Project (3 yrs): ‘R’ primarily with TESTA tools; ‘D’ in disciplines and universities.• approach: teaching teams with students interpret ‘R’ data to determine goals of ‘D’.• activities: to address QA and QE issues, optimize sector engagement (fastech.ac.uk)• outputs: R&D findings, experiences & guides by teachers, students, others…Pragmatic Principles?• Fast: using readily-available technologies; quick to learn, easy to use …• Efficient: after start-up period; saves time & effort ( paper), productivity …• Effective: brings significant learning benefit to students, pedagogic impact …
    37. 37. FASTECH: a Pedagogical Goal … ability to manage own learning … In each assessment culture, this entails using technologies that help promoteStudent Teacherbaggage … baggage … transparency & S participation in all processes from• all can be design and management to feedback and revision strategic! (validity, reliability & fairness are not enough) and blocks: • ideas about role a reshaping of teacher & student responsibilities of assessmentand blocks: • unsure about• ideas about processes that enhance and create new: peer- value of feedback roles of S & T learning activities & collaborations (in/out of class); • assessment &• … self & peer assessment; recording, sharing & review marking conflated of students’ progress and achievements … • criteria & standards teacher revision of pedagogies, based upon records • … of student progress & achievement in learning attuning of assessment to address individual & distinctive needs & aspirations …..
    38. 38. Finally, for an excellent overview of technologies and pedagogiesJISC, Effective Assessment in a Digital Age. Bristol: HEFCE, 2010.Available at: www.jisc.ac.uk/digiassess (esp., pp. 14-15, 54-55)For resources associated with this publication:www.jisc.ac.uk/assessresourcePlease contact us for more info about TESTA and FASTECH:Tansy.Jessop@winchester.ac.ukYassein.El-Hakim@winchester.ac.ukp.hyland@bathspa.ac.ukWebsites: www.testa.ac.uk & www.fastech.ac.uk (from January 2012)Thank You
    39. 39. DISCUSSIONto be continued in the conference discussion forumHow do you think using technology in A&F will improve students’ learning?
    40. 40. ReferencesBlack, P. & D. William (1998) ‘Assessment and ClassroomLearning’, Assessment in Education: Principles, Policy and Practice. 5(1): 7-74.Bloxham, S. & P. Boyd (2007) Planning a programme assessment strategy.Chapter 11 (157-175) in Developing Effective Assessment in Higher Education.Berkshire. Open University Press.Boud, D. (2000) Sustainable Assessment: Rethinking assessment for thelearning society, Studies in Continuing Education, 22: 2, 151 — 167.Gibbs, G. & Simpson, C. (2004) Conditions under which assessment supportsstudents learning. Learning and Teaching in Higher Education. 1(1): 3-31.Gibbs, G., & Dunbar-Goddet, H. (2007) The effects of programme assessmentenvironments on student learning. Higher Education Academy.http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/gibbs_0506.pdfGibbs, G. & Dunbar-Goddet, H. (2009). Characterising programme-levelassessment environments that support learning. Assessment & Evaluation inHigher Education. 34,4: 481-489.
    41. 41. Jessop, T., El Hakim, Y. & Gibbs, G. (2011) TESTA: Research inspiringchange, Educational Developments 12 (4). In press.Jessop, T., McNab, N., and Gubby, L. (2012 forthcoming) Mind the gap: Ananalysis of how quality assurance procedures influence programmeassessment patterns. Active Learning in Higher Education. 13(3).Knight, P.T. and Yorke, M. (2003) Assessment, Learning and Employability.Maidenhead. Open University Press.Nicol, D. J. and McFarlane-Dick, D. (2006) Formative Assessment and Self-Regulated Learning: A Model and Seven Principles of Good Feedback Practice.Studies in Higher Education. 31(2): 199-218.Nicol, D. (2010) From monologue to dialogue: improving written feedbackprocesses in mass higher education, Assessment & Evaluation in HigherEducation, 35: 5, 501 – 517Sambell, K (2011) Rethinking Feedback in Higher Education. Higher EducationAcademy Escalate Subject Centre Publication.

    ×