ISSOTL12 Hamilton, CanadaDoing institutional change ‘the fourth way’:a whole institutional change programme for assessment...
The University of the Arts London’s overall curriculum was loosely defined in a number ofsignificant fields, and assessmen...
AnalysisThe literature on assessment in art and design emphasises connoisseurship, and tacitlyestablished shared standards...
P48, Fig 3.1 What to retain and what to abandon                                 Retain                                   A...
LinksThe Marking matrix, information about its development, and formal evaluation:http://www.arts.ac.uk/assessment/marking...
Upcoming SlideShare
Loading in …5
×

Issotl12 handout sw

503 views
434 views

Published on

ISSOTL12

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
503
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Issotl12 handout sw

  1. 1. ISSOTL12 Hamilton, CanadaDoing institutional change ‘the fourth way’:a whole institutional change programme for assessment and feedbackProfessor Shân WareingDean of Learning and Teaching DevelopmentUniversity of the Arts LondonE: s.wareing@arts.ac.ukFrom 5th November 2012Pro Vice Chancellor Learning and TeachingBuckinghamshire New UniversityE: shan.wareing@bucks.ac.ukFollow me on twitter @shanwareing Trigger: If participation prevails – if what matters most is left unreified – then there may not be enough material to anchor the specificities of coordination and to uncover diverging assumptions. Wenger 1998 p65SynopsisThis paper explores the model of quality assurance and professional developmentarticulated in Hargreaves and Shirley’s (2009) The Fourth Way through its application to acase study of institutional changes to assessment, using the model as an explanation of theinitial hostility of staff to the changes and of their ultimate success.Case StudyThe case study relates to an intervention to improve assessment practices at a large visualand creative arts university in London, where clarity around assessment standards andexpectations, consistency of grading, usefulness of feedback, and turn-around time, were ofconcern. 1
  2. 2. The University of the Arts London’s overall curriculum was loosely defined in a number ofsignificant fields, and assessment in particular was based on an assumption of apparentlytacitly shared standards. A national Quality Review in 2007 questioned the adequacy ofassessment policy and processes. The National Student Survey results showed students didnot perceive assessment to fair nor the criteria on which they were assessed to be clearbeforehand. With up to 50% of teaching undertaken by hourly paid staff, and a lack ofsocial spaces for staff, it seemed unlikely that the presumed shared standards based oninteraction amongst staff could be established satisfactorily. There was also researchsuggesting that students from under-represented backgrounds found it harder to deducetacit standards, with adverse impact on their achievement. UAL had a gap between theachievement of white students and Black and Minority Ethnic students which is 10% higherthan the national average gap. Anecdotally there were stories of very minimal andunhelpful feedback being given to students.The initiative consists of eight explicit standard criteria, a standard feedback form which hasbeen developed into an online form integrated with the student record system, and a fixedmatrix indicating key identifiers of achievement at different grades. There areundergraduate and taught postgraduate versions of each. Courses can adapt the feedbackform by indicating any criteria that are not applicable, and by customising the explanatorytext appearing below each criterion to make it more relevant to their students.Achievement is indicated using radio buttons against each of the criteria, and feedback canbe provided against each criterion either using standard text from the matrix or writingunique text. All markers are required to write free text ‘feed forwards’ to students. Thegrade is not the computational outcome of the achievement against each criterion. Thematrix in indicative only, but cannot be adapted: it has to appear in its standard form in allcourse handbooks.The purpose of the initiative was to introduce explicit reified standards into assessment, asthe basis for marker moderation discussions and for explaining assessment to students; toprompt more effective feedback on student work; to speed up the process of providing suchfeedback for markers; and to underpin course design. Feedback from staff suggests that thedevelopment has at least partly met all these goals.The new criteria and feedback forms were introduced first in September 2009, and for finalyear undergraduate students and students on taught postgraduate courses in September2010. There was an average increase of 7% in UAL’s NSS scores for assessment andfeedback in 2011, and of 7%-10% for the assessment and feedback questions in thePostgraduate Taught Experience Survey. A further increase of 2% occurred in the 2012 NSS.Additionally, there was an increase in 2010/11 in the proportion of 1st/2:1 degreesawarded, up c. 2% to 66% on the trend of proceeding years, when it has been in a range62%-64%. Student feedback obtained through focus groups was positive from the outsetabout the criteria.A qualitative evaluation of the initiative found that staff with a more pedagogicallydeveloped understanding of assessment found the new criteria and forms at least as goodor better than their previous practices, but staff with a more superficial understanding ofassessment found the criteria and forms constraining. 2
  3. 3. AnalysisThe literature on assessment in art and design emphasises connoisseurship, and tacitlyestablished shared standards, built up through community interactions (e.g. Orr 2006).However, this approach does not account for power imbalances between staff and students,nor the possibility of prejudice and bias in marking (Sabri 2011) nor for the logisticaldifficulties of developing shared standards amongst hourly paid staff in the absence ofcommunity spaces such as staff rooms.An ideal intervention to improve assessment might have involved an extensive professionaldevelopment programme, supporting all staff in acquiring a personal and scholarlyunderstanding of assessment, to underpin personal and locally crafted assessment andmarking practices. However, barriers to this approach included lack of resource to provideextensive professional development, including paying hourly paid staff to attend, logisticaland motivational difficulties to ensuring sufficient participation by staff, and the delay inrealising the benefits of the change programme to students.It was therefore decided to impose a universal assessment system, designed to addresssome of the known shortfalls in the existing approach. The new system was pedagogicallyrobust and underpinned by the scholarship of learning and teaching (particularly drawing onBloom 1956, Kolb 1984, Perry 1970 and Biggs 2003, and taking account of the disciplinaryneeds of the visual arts).Many academic staff initially saw the imposition of the assessment changes ascompromising their autonomy, a managerial challenge to their professional judgement, anda blow to the educational value of their teaching via an inflexible, inappropriate set ofstandards. However, student feedback was positive, and scores for students’ perception ofassessment and feedback in national surveys rose. Over two years of implementation, stafffeedback has improved dramatically: “The assessment tool ... speeded up the process ofassessment and allowed ... staff to give better informed, critical and useful feedback tostudents.”Hargreaves and Shirley’s analysis of education over 60 years charts a shift fromunquestioned professional autonomy, through a period of where transparency andstudents’ rights resulted in imposed and deprofessionalising standards. The most effectiveleadership and management processes balance support and accountability, freedom andconsistency, the empowerment of staff and the empowerment of students. Reactions tothe assessment change programme, and its ultimate success, can be explained through thelens of this model. 3
  4. 4. P48, Fig 3.1 What to retain and what to abandon Retain AbandonFirst Way Inspiration, innovation and autonomy Inconsistency and professional licenseSecond Way Urgency, consistency and all-inclusive Cut-throat competition and excessive equity standardisationThird Way Balance and inclusiveness, public Persistent autocracy, imposed targets, involvement, financial reinvestment, obsession with data, effervescent better evidence, and professional interactions networksFourth Way “A democratic and professional path to improvement that builds from the bottom, steers from the top, and provides support and pressure from the sides” p107P51 ‘you can’t just adopt the end product of something that too others years to develop...no two places are alike...nevertheless they can be cross-pollinated with other successfulreform initiatives and movements’Hargreaves, A. and Shirley, D. (2009) The Fourth Way: The inspiring future for Educational Change, London: Sage.Conclusion1) Its clearer in hindsight than it was at the time2) Assessment problems were exacerbated by the disciplinary culture, by headline debates about good assessment in HE, & by logistical problems such as c. 50%-75% of the teaching undertaken by fractional and hourly paid staff, no staff rooms, reduced budgets for staff development, and no time for lunch hours or coffee breaks3) Half the effort was on the solution, the other half was building the capacity to create and implement the solution. Thats why good practice isnt directly transferable4) Change management in mass higher education must enable professional autonomy within clear structures and systems, introduced collegiately and supportively.5) Leadership has a moral imperative, which is the source of its momentum 4
  5. 5. LinksThe Marking matrix, information about its development, and formal evaluation:http://www.arts.ac.uk/assessment/markingcriteria/index.htmlA formal evaluation of the marking criteria introduction:http://www.arts.ac.uk/assessment/assets/doc/Report-Evaluation_of_UAL_Marking_Criteria.pdfScreengrabs of the online assessment tool, developed from staff feedback after the markingcriteria and standard feedback sheet were introduced:https://picasaweb.google.com/114823604275354290558/OATScreengrabs181012?authuser=0&authkey=Gv1sRgCIf74Y2vw4-LywE&feat=directlinkEvidence of improvement in students’ perception of assessment:http://www.arts.ac.uk/assessment/assets/doc/UAL-NSS-assessment-results-2008-12.pdfReferencesArcher, L. (2007) ‘Widening Participation, Social Class and Ethnicity: Issues and Considerations’, Conference Presentation at University of the Arts London 2 nd Annual Learning and Teaching Conference 21st-22nd May 2007.Blair, B. (2006). "At the end of a huge crit in the summer, it was "crap" - I had worked really hard but all she said was "fine" and I was gutted." Art, Design and Communication in Higher Education 5(2): 83-95.Cowdroy, R. and Williams, A. (2006) ‘Assessing creativity in the creative arts’ Art, Design & Communication in Higher Education 5 (2) pp97-117Elton, L. (2006) ‘Assessing creativity in an unhelpful climate’ Art, Design & Communication in Higher Education 5 (2) pp119-130Gawande, A. (2010) The Checklist Manifesto. London: Profile BooksHargreaves, A. and Shirley, D. (2009) The Fourth Way: The inspiring future for Educational Change, London: Sage.Orr, S. (2006) ‘Assessment Practices in Art and Design’ Art, Design & Communication in Higher Education 5 (2) pp79-81Orr, S. (2010). We kind of try to merge our own experience with the objectivity of the criteria: the role of connoisseurship and tacit practice in undergraduate fine art assessment. Art, Design and Communication in Higher Education 9(1): 5-19Sabri, D. (2011) An Evaluation of marking criteria at the University of the Arts, London, London: UALSadler, D. Royce (1983) Evaluation and the Improvement of Academic Learning, Journal of Higher Education, 54: 60-79Sadler, D.Royce. (1989) Formative assessment and the design of instructional systems, Instructional Science 18: 119-144Sadler, R. D. (2009). Indeterminacy in the use of pre-set criteria for assessment and grading. Assessment and Evaluation in Higher Education 34(2): 159-179.Sadler, R. D. (2010) Transforming holistic assessment and grading into a vehicle for complex learning. In Assessment, Learning and Judgement in Higher Education. ed G. Joughin. London: SpringerWenger, E. (1998) Communities of Practice. Cambridge: Cambridge University Press 5

×