In Semester 1 of this academic year (2013-14) Newcastle University successfully delivered 71 exams, with more than 9000 student sittings across a multitude of venues across campus. The only additional software cost has been a site-wide Respondus Licence used to convert existing formatted questions and produce backup paper exam copies. By building on Blackboard's existing functionality, we have provided a streamlined process for building and running secure, locked down online exams with additional reporting and feedback functionality. This presentation outlines how the OLAF (Online Assessment and Feedback) project has evolved since 2007 to its present state today. The stakeholders involved in a delivering a complex cross-institutional service and the tools we have developed to assist us to keep track of multiple exam delivery across multiple PC venues. Our technical support team have developed a free in-house solution for providing a locked down browser, using Firefox portable and a compatible extension. During the session we will explore the pros and cons of Blackboard as an assessment system, outlining what can be achieved and where we are hitting limitations and whether we can do anything about these limitations. The existing assessment tools in Blackboard can meet the majority of needs for online examinations at present. We can use exported data from Blackboard to produce additional reports and student feedback. Yet there are limitations to the testing that can be conducted through Blackboard that will stop some users from fulfilling their requirements. There are still features that other software offers that staff would like to use, such as breaking tests into sections, more advanced text marking and so on. Should institutions be aiming for more online assessments with more sophisticated features, or should we stick with what the majority need to use? During the session we will encourage debate about short term/long term needs and strategies

1. Successfully Delivering
Large Scale Online
Summative Exams
Ashley Wright
Development Officer
Quality in Learning & Teaching
Newcastle University

2. OVERVIEW
• Background
• Processes
• Technical aspects
• Progress to date
• Benefits
• Issues
• Future developments
• Q and A

3. BACKGROUND
UTLC funded project in 2007 – Online
Assessment and Feedback (OLAF)
Aims:
• Improvement in student and staff
experiences in the areas of assessment and
feedback highlighted by the NSS
• Develop a product/service which would
deliver high stakes summative assessment
in a secure online environment

4. PROCESSES
• Swim lane diagram….highlights the nature
of this service
4 parties involved:
• Customer/staff
• QuILT (Quality in Learning & Teaching)
• Exams Office
• ISS (Information Systems & Services)

5. PROCESSES
Creating online exams -
https://blogs.ncl.ac.uk/olafaq/
• Directly in Blackboard/re-used with
modifications (easiest option)
• Convert from Word using Respondus – site
wide licence. Upload/download directly to
Bb via Respondus

6. TECHNICAL ASPECTS
• Around 1400 PCs on campus located in clusters
(Herschel largest with ~ 150) - http://m.ncl.ac.uk/iss/#clusfree
• Reboot cluster and login with generic user account
• WE mode replaced with FireFox Portable in Kiosk
mode, disabling ALT keys
• Page generated looking for today start date and
OLAF in the title

7. TECHNICAL ASPECTS
• Next step is login screen – altered using extension
in FireFox
• Open test in a new window so there is no left menu
• The important parts which change for each
assessment are the
course_assessment_id and the
course_id, these are the pk1 values in the
database for the relevant exam.

8. TECHNICAL ASPECTS
• Invigilators in charge during the exam sitting
• Next step is for students to enter the exam
password
• Password reset after 30 mins by Exams Office to
one contained on Yubikey

9. TECHNICAL ASPECTS
• Questions presented one at a time, timers not set,
force completion not set, unlimited multiple attempts
• Encouraged to view Question Completion Status

10. TECHNICAL ASPECTS
Submission Page altered – cannot view results
This
To that

11. TECHNICAL ASPECTS
• KEY - Checker page created by ISS to manage
exam volume
• Direct links into the relevant test.
• Red colour highlights a potential issue
• Info is gathered by a PHP web server connecting
directly to the Blackboard database
• Could be developed into a building block?

12. PROGRESS TO DATE

13. BENEFITS
• Staff time saved in terms of marking alternative
assessments (10 mins per sitting ~90000 mins in
Jan 2014)
• Reuse existing assessments, with modifications
• Paper!
• Staff and student familiarity
• Robust and confident in the performance of Bb

14. ISSUES
• Should it be titled OLA?
• Limitations on q types/-ve marking, layout, random
blocks
• Randomisation – id of Qs – specific sections
• Cluster spaces limited and technical glitches
• Save and Submit on final Q
• Forgotten login details by students!

15. FUTURE DEVELOPMENTS
• Incorporate text entry
• Use Tablets (exams of <= 1 hour)
• Multimedia (view/listen and answer
questions based upon this)
• Non Newcastle campus delivery, e.g
Singapore, Malaysia…a n other location
• What are Bb planning next?

16. ?

17. Q & A
Thanks for listening!
ashley.wright@ncl.ac.uk
@ashleyhwright