Presentation exploring the relationship between policy and practice in the development of e-assessment in higher education and the importance of establishing a policy framework - developed in collaboration with all key stakeholders - to support wider uptake among academic staff.

1. e-Assessment Policy and Practice –
Making the Connection
David Walker
(@drdjwalker)
University of Dundee

2. Overview
•
•
•
•
•
•
•
Locate e-assessment in context
Conditions for embedding e-assessment
Role of e-assessment in wider Technology
Enhanced Learning (TEL) strategy
Channels/stakeholders informing strategy/policy
Embedding inclusion in policy and practice
Institutional efforts and annual cycle of
e-assessment
Consider factors that will shape policy in the future

3. Not about bureaucracy…
You can view the full collection on Flickr:
http://www.flickr.com/photos/8181983@N05/ or by visiting www.vimrod.com

4. It’s about addressing…
“Do we have a policy on this?”
Colleague, University of Galway (2010)

5. What is e-Assessment?
You can view the full collection on Flickr:
http://www.flickr.com/photos/8181983@N05/ or by visiting www.vimrod.com

6. e-Assessment in Context
• Rapid developments in field of e-learning in last 10 years
• Expansion of higher education system (mass higher education) –
social, philosophical and practical implications
• e-assessment becoming particularly prevalent across the sector majority of activity within science/engineering-based disciplines
• Growing body of research underpinning practice
• Emphasis on alignment of learning and assessment methods with
learning outcomes (curriculum alignment)

7. Credibility Crisis?
e-assessment “…must satisfy critics of its
pedagogical fitness for purpose”.
McKenna, C. & Bull, J. (2000). Quality assurance of computer-assisted assessment:
practical and strategic issues. Quality Assurance in Education. 8(1), 24-31.
• Can e-assessment be used to assess appropriate levels of
learning and skills?
• Are appropriate quality assurance procedures in place to
ensure the validity/reliability/security of e-assessments?

8. Danger of Disjunction
“While assessment is one of the last things that university
teachers consider when planning and developing courses, it is
often the first thing that most students think about.”
Open University (Undated)

9. Embedding e-Assessment
•
Realising promise of e-assessment requires strategic
approach
•
Need to address concerns about security and reliability to
enhance credibility
•
Ensure policy/strategy is practical and sustainable and
aligned to institutional strategic aims
•
Foster sense of ownership of policy across institution

10. e-Assessment Policy
• Use of e- assessment systems and emerging
tools requires clear guidance
• Dundee first UK institution to develop a
Senate approved policy in this area
• Policy and Procedures (2003) - broadly
aligned with BS7988:2002
• Policy specifies procedures to be followed
if e-assessment used in modules
• Regularly reviewed/updated

11. Informing Policy
e-Learning Forum
t
en
En
vir
on
Middle-Out
m
on
vir
En
m
en
t
Top-Down
Institutional Aims / Sectoral
Agenda / Legal Duty
Learning Lunches
Bottom-Up
Environment

12. e-Assessment Stakeholders
The ‘Hidden Team’
‘Front of House’
IT Database Administrator
Examinations Office
Lecturer
Distance Student
Is the database correctly
configured?
Is it receiving and retrieving
correct data?
Is it properly secured behind a
firewall?
Are these results 100% reliable and
secure?
Do our procedures represent the best
possible practice?
Are procedures in place to address
exam irregularities?
Is my technology compatible?
Is my connection reliable?
Is this a replacement for F2F?
IT Desktop Administrator
eLearning/Assessment Manager
Does the desktop software
operate correctly?
Are the “exam accounts” secure
and available?
Are the PCs well maintained and
reliable?
Are the staffing levels correct?
Is our software choice appropriate?
Are the server specifications correct
to cope with the expected load?
Are staff education procedures in
place?
Which software should I use?
Who can electronically author my
questions?
Can we use textbook questions?
Is this assessment driving learning?
Are my questions too simple?
Is this technology reliable?
Do the students find it tedious?
Where can we obtain challenging
questions?
Are these assessments aligned with
our learning outcomes?
IT Server Administrator
Timetabling Officer
Is the server fully patched,
reliable and secure?
Is the software functioning
correctly and reliably
Do we have monitoring and is it
working?
Are the logging and backup
procedures working correctly?
Can the proposed exams be fitted
into the best rooms?
Are assessments receiving
appropriate prioritisation?
Disability Services Manager
Do procedures address needs of
students requiring reasonable
adjustments?
Is appropriate assistive technology
available
School Secretary
Can we book an IT suite for 240
students on Wednesday the 25th?
Two students have additional
requirements.
Learning Technologist
Are staff adequately trained and
aware of how the systems operate?
Are assessments correctly
scheduled?
Will the exam run smoothly?
Are contingency measures in place?
Student
This is nice and private
I am getting real feedback here
Do I know the answers?
Can I learn from this?
I am at home in this
online environment

13. Evolution of Policy
• Original policy redeveloped and re-approved in 2007 and expanded to
include guidance on use of emerging assessment tools e.g. ePortfolios,
electronic submission etc
• Continued development in assessment and Web 2.0 technologies and
approaches - prompted extensive revision to policy in 2009
• Revision process involved:
– Cross-team ‘framework’ meeting to identify key issues
– Input from specialists on matters ranging from technical to procedural.
Cross-team approach to policy development – input from members of the
‘hidden team’ important
– Review of best practice in field including policies produced by fellow UK
HE Institutions

14. Policy Issues
• Greater emphasis on roles and responsibilities and clearer
stipulation on time scales for registration and preparation
of electronic examinations
• Extended guidance on accessibility and disability support
• As use of e-assessment increases technical issues more
frequent – greater clarity required on how technical issues
will be managed
• Need for clarity on invigilation (aligned with University
regulations on conduct of examinations)

15. Inclusive Practice
•
Inclusion = supporting disabled staff/students and
staff/students who are situationally disabled
•
e-Assessment (TEL) has huge potential to enhance inclusion
– or introduce new barriers
•
Dundee has an active inclusion support network
•
Web Accessibility Service co-ordinator involved across the
range of TEL strategy activities – important contributor to
policy

16. Shared Resource

17. Policy or Practice?

18. Institutional Efforts
How can an institution help to realise the promise of
e-assessment?
• Enlist small groups across schools to pilot new initiatives over a
fixed period
• Ensure appropriate channels exist to disseminate good practice
e.g. Teaching Awards/Learning Lunches
• Offer appropriate academic professional development
opportunities – include both internal and external contributors
• Support a culture of professional development and a willingness
engage with new ideas/approaches

19. Annual Cycle of e-Assessment
Sept.
Online resit
exams
Testing &
tuning
Modules
begin
Annual
framework
meeting
Summer
upgrade
window
Online exams
June
December
Mid
semester
window
Online
exams
• Growth may occur each year
eAssessment
education
module
• Changes may be made to the
systems
• A framework allows the
process to be sustained

20. Summative e-Assessment
“Students can, with difficulty, escape from the effects of
poor teaching, they cannot… escape the effects of poor
assessment.”
(Boud, 1995)
• High-stakes: concerns about validity, reliability, credibility,
fidelity, fairness & transparency
• High salience
higher levels of test anxiety
• Clear policy and procedures crucial

21. Formative e-Assessment
“The clear advantage of online testing as part of formative
assessment is that it is in a private space and, in most cases, it can
be accessed at a time when the student feels appropriate.”
(Challis, 2005)
•
e-assessment widely used for formative assessment
•
e-Testing: automatic feedback to learners - allow students to
practice skills
•
•
Reflective journal/scientific log – comments left by academics
Most prominent role for e-assessment?

22. Student Views
"We need formative assessments! Online formatives should be
available after every block.”
“So it’s a bit more relaxed sitting on a computer because you can, you
can sit back and just click through whereas you’ve got all this going on
in your mind but...you don’t have the added thing of writing down.”
“I think that on the whole this type of assessment it can work, and it
can consolidate learning, and it can enable you to apply knowledge and
understanding to situations and problem solving but I think on the
other hand it can be a means to an end..."

23. Future of e-Assessment
• Secure, stable, performing systems - many institutions
moving to hosted solutions
• Innovation within and beyond existing (dominant)
assessment management systems
• Policy issues around use of mobile technologies
• Risk to large-scale testing posed by closure of IT suites
• Stronger links between educational developers/learning
technologists/academics – more effective assessments
• Increased sharing/sales of item banks
• Clearer divide between (compliance) ‘testing in’ corporate
sector and ‘assessment’ in education sector

24. Closing Remarks
• Successful e-assessment/technology enhanced learning
dependent on numerous components – strategic, technical and
procedural
• Range of stakeholders – academic staff, students, e-learning staff,
IT staff, administrators, senior management
• Challenge is to accommodate growth, consider risk and accept
rapid change
• Issues of sustainability – IT suite availability, staffing, licensing etc
• Policy/strategy should be informed through formal and informal
routes – need to achieve shared ownership

25. Closing Messages
“e-Assessment has come a long way and is here
to stay.”
Source: http://www.da.mod.uk/academy-events/e-assessment-in-practice

26. Contact Information
Dr David Walker
Library & Learning Centre
University of Dundee
Email: d.j.walker@dundee.ac.uk
Twitter: @drdjwalker