Electronic Management of Assessment – Critical Success Factors in Institutional Change


Published on

EMA critical success factors from HeLF surveys. EdMedia presentation 2014

Published in: Education, Business, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Electronic Management of Assessment – Critical Success Factors in Institutional Change

  1. 1. Electronic Management of Assessment – Critical Success Factors in Institutional Change Dr Barbara Newland, University of Brighton Dr Neil Ringan, Manchester Metropolitan University Lindsey Martin, Edgehill University
  2. 2. Electronic Management of Assessment (EMA) includes a wide range of activities so the following definitions are being used: Definitions eSubmission electronic submission of an assignment eMarking electronic marking (including offline marking eg in Word) eFeedback electronic feedback (ie text, audio, video but not hard copy) eReturn electronic return of marks
  3. 3.  A network of senior staff in institutions engaged in promoting, supporting and developing technology enhanced learning  Over 130 nominated Heads from UK Higher Education institutions  A regular programme of well attended events  Represents the interests of its members to various national bodies and agencies including the Higher Education Academy and JISC www.helf.ac.uk Heads of eLearning Forum (HeLF)
  4. 4. Aim  To identify critical success factors relating to institutional change with EMA from longitudinal research over the past three years which has:  Identified current practice with regard to EMA in UK HE  Provided a snapshot of the strategic overview identifying key issues relating to strategic change, policies and practices
  5. 5.  HeLF members were surveyed each March for 3 years from 2011 and asked to respond with regard to their knowledge of their own institution  The questions were a mixture of closed multiple-choice/multiple- answer and multiple selection as well as open response type  Participants were assured that all data collected in the survey would be held anonymously and securely  No personal data was asked for or retained unless the participant indicated a willingness to be contacted in the future  The results were analysed using quantitative and qualitative methods Methodology
  6. 6. Results  The results emphasised the importance of assessment  It is mission critical as it determines whether students achieve, progress and gain awards (and at which level)  Consequently, all stakeholders in the assessment process are wary of making errors and need to be convinced of the merits in changing long-established practices  The change in practice supported by technology is only worthwhile if it effectively enhances stakeholders’ experiences and/or improves the overall process workflow
  7. 7. Results  The following comments illustrate the scale of the change being undertaken:  ‘e-Submission is quite straightforward. However, we could have managed the process of e-marking and e- feedback better. Academic staff need a lot of time to come round to the idea if they are changing years of established practice.’  ‘Never underestimate the effort involved with winning hearts and minds of colleagues.’ (HeLF, 2014)
  8. 8. Critical Success Factors Institutional culture Regulations Policy Guidelines Support Implementation Plan Communication Responsibilities Monitoring Risks Technology Health and safety Resourcing – support staff Attitudes of academics, administrators, students Quality assurance
  9. 9. Institutional culture  An institutional culture encompasses all the critical success factors. The change process needs to align with this culture. In some universities there has have been mandatory policies and in others a more flexible approach or a mixture of both:  ‘It depends very much on the culture of your organisation. Here a partnership model between central services, and School / faculty admin and academics tends to work. Top down imposition of systems is less successful. In other institutional cultures this may be different.’  ‘Policy led the mandatory eSubmission. eFeedback, while voluntary, is being adopted by staff as it is available in the same interface.’  ‘One of the emerging ideas is to see submission, marking and feedback as separate entities, i.e. enabling students to submit and receive feedback online but staff to mark by whatever method is comfortable. Also not to insist on a particular feedback tool as long as the virtual point students pick up the feedback is consistent.’ (HeLF, 2014)
  10. 10. Institutional culture  The EMA process involves everyone in a university so identifying a leader of change can bring all the people together.  ‘Having a central 'head of elearning' role to bring everything together across the institution: to scour the Regulations and relevant policies to produce a set of key questions for consideration by each of the key stakeholder groups.’ (HeLF, 2014)
  11. 11. Institutional culture  Consultation of all stakeholders and support from senior management as well as the efforts of departmental champions and students is key:  ‘Consultation was important to choose a system that would fulfil the criteria of all stakeholders. The stakeholders themselves shaped that system. The drive from students was critical in making sure people engaged with this’.  ‘Senior Management support. Departmental champions. Peer and student pressure on academic staff concerning provision of eFeedback and eMarking.’ (HeLF, 2014)
  12. 12. Regulations  Assessment regulations will need to include EMA processes.  Often this requires just a small amendment to existing regulations. These will need to go through the committee process, which may take some time.  It may be helpful to consider how the current policy handles situations in which a student is unable to hand in a hard copy.  For example, the outcome for a student who hands in work late because his own computer fails could be the same as if the reason late submission had been missing a bus  Similarly, the outcome for a student who cannot hand in work as the university is shut due to snow could be the same if the student cannot hand in work because a university system has failed.
  13. 13. Policy and guidelines  Policies  clear, up-to-date  have to be agreed and approved by committees. This may take some time  Guidelines:  need to be made easily available to students, academics, administrators and departments  give clear instructions on effective ways of using the systems and explain the responsibilities in alignment with the regulations and policies  describe contingency plans for particular scenarios occurring.
  14. 14. Technology  Both the end user hardware and software and the central systems need to be robust, reliable and secure  User software should be user-friendly and have the features and functionality to improve the assessment process for students, academics and administrators  There are benefits if the software integrates with other central systems such as the student record system and the Virtual Learning Environment.  ‘Reliability / availability of both the VLE and Turnitin essential.’  ‘Achieving a smooth system that supports the process well (technological perspective’ (HeLF, 2014)  Some departments provide second monitors and/or laptops to academics for eMarking and eFeedback.
  15. 15. Support  Support and training both face-to-face and online is required for academic and administrative staff as well as Helpdesk support.  Learning technologists have a key role and the move to eSubmission has generally created more work for them (Newland, 2011)  Students may require support in accessing and understanding the method of eFeedback  Support will be required both in advance and then to handle urgent queries from students and staff as the deadline approaches and stress levels increase with the urgency of submitting on time and overcoming any problems in this process  Pedagogical support to encourage effective feedback practices can enhance student learning. The quality of eFeedback is dependent on the quality of the feedback
  16. 16. Implementation Plan  Identifies what, when, how and who is responsible  It needs to include the detail of the EMA processes e.g.  How to handle extensions, second marking, blind marking, re-sits.  Is it e-only Submission or e and paper?  If eFeedback is optional then who prints and who pays for it.
  17. 17. Communication  Always important during a change process  Not only about the process of the change but also highlighting the benefits of engaging in the change.  Needs to be two way so that the implementation can be modified as the result of feedback and people participating in the change process are aware of the impact of their feedback  This helps to develop a sense of ownership/partnership with the change rather than it being perceived as something enforced.
  18. 18. Responsibilities  EMA will change the roles of academics, administrators and learning technologists  This change may vary within courses and departments within a university  The consideration of responsibilities needs to be within a university’s EMA guidelines.
  19. 19. Monitoring and risks  Monitoring  The implementation of EMA needs to be monitored and exceptions, for example, to e-only Submission to be identified and reported  The monitoring can also highlight effective practice.  Risks  Assessment is mission critical and high risk in every aspect.  Appropriate planning for, and mitigation against, potential identified risks occurring should be embedded within EMA policies.
  20. 20. Health and safety  Concerns expressed by certain staff regarding the time spent looking at a screen. Usually, these concerns are already addressed through a university’s health and safety procedures  Many academics undertake eFeedback at home on their own computers, and appropriate consideration should be given to this aspect.
  21. 21. Resourcing  Implementing change requires resourcing not only in hardware and software but also for support and engagement in reviewing the whole EMA process.
  22. 22. Attitudes  The move to eSubmission is generally supported by students, academics and administrators. However, many academics are initially reluctant to engage with eMarking and eFeedback. Positive experiences from students and staff encourage the change.  ‘Positive student feedback through module evaluations and staff, student consultative committees. Exams & Assessments dept are positive about new processes.’ (HeLF, 2014)  Academic staff attitudes to eMarking are steadily changing as there is some recognition of the perceived value, benefits and efficiency and acknowledgment that it will happen in future.
  23. 23. Quality Assurance  eMarking and eFeedback can provide a more consistent student experience. For example, by using a course or departmental rubric  However, greater standardisation may deter innovation in assessment
  24. 24. Conclusion  The critical success factors in the use of EMA as it has moved from individual early adopters to more formalised institution–wide usage have been identified through a series of annual HeLF surveys  Consideration of these factors within an institutional context can enable the effective implementation of EMA  These factors involve everyone in a university and need to be considered simultaneously for effective implementation of, and change to, EMA
  25. 25. References  Gibbs, G., Simpson, C., & Macdonald, R. (2003). Improving student learning through changing assessment https://www.open.ac.uk/fast/pdfs/Gibbs%20and%20Simpson%202003.pdf  HeLF, (2014), www.helf.ac.uk  Newland, B., Martin, L., Ramsden, A. ( 2011). eSubmission – UK Policies, Practice and Support. Proceedings of the European Conference on e- Learning: 74-82.  Newland, B., Martin, L., Bird, A., & Masika, R. (2013). HeLF Electronic Management of Assessment Report, www.helf.ac.uk  Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self- regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31: 2, 199 — 218.  Sadler, D.R. (1989). Formative assessment and the design of instructional systems, Instructional Science, 18, 119-144.
  26. 26.  Dr Barbara Newland b.a.newland@brighton.ac.uk  Dr Neil Ringan n.ringan@mmu.ac.uk  Lindsey Martin Lindsey.Martin@edgehill.ac.uk Contact details