What was the impact on student perception measures when technology problems occur during an e-exam event?

1. Mathew Hillier
Naveen Kumar
Nirmani Wijenayake
University of New South Wales
Sydney, Australia
e-exams in a university bio-chemistry
laboratory: technology problems and
its impact on student experience

2. Focus of this paper – key research question
What was the impact on student perception
measures when technology problems occur
during an exam event?
Prior work has not addressed this specific
question!

3. Study context: UNSW Australia
On-campus
invigilated spaces
Lab computer
Secure browser
Classroom laptop
Secure browser
Exam hall laptop
Secure browser
Home BYO laptop
Browser + Remote
LMS (Moodle):
provides exam content
Off-campus remote
Safe Exam Browser: secures a computer for an exam and allows
whitelisted apps and resources to enable authentic assessment tasks.
Invigilation
service
The study was carried out within the broad context of the development of a university-wide digital assessment
strategy at UNSW.
This study focused on the ‘computer lab’ context – desktop computers supplied by the university.
Students used Safe Exam Browser + Moodle Quiz + Lab Archives (lab e-note book).
This paper reports on one pilot (Note: two other pilots followed later in the year).

4. Method
Case study: BioChem Lab exams, July 2019
Participants: 215 second year undergraduate students across 4 class groups.
Equipment: Windows lab PCs with Safe exam browser + Moodle Quiz + LabArchives (e-notes).
Data collection: Survey using Likert item responses.
Exam room procedure:
1. Students enter the room and were seated at a suitable bench.
2. Students can read a printed instruction sheet with steps to get started.
3. Once logged into the desktop computer, students then logged into the LMS using Chrome browser and then clicked on a link
leading to a SEB setting file. This then launched Safe Exam Browser. Students were required to again login to the LMS and
were then taken to the Moodle quiz start page.
4. When all students were ready to begin, the invigilator announced the start of the exam by providing the Moodle quiz
password. This enabled all students to start at the same time. Quiz auto-save occurred each 30 seconds.
5. Exam end: students used the Moodle quiz submission button and then exited SEB. The quiz was set to automatically submit
if the quiz timer expired.
6. Students were requested to log-off from the computer.
7. Students completed the post-exam survey before leaving the room.

5. Bio-Chem Pilot: SEB + Moodle + LabArchives
Digital open
book with
controlled
e-notes
Moodle quiz in
lock down
browser
(SEB)
This pilot: BIOC 11 July 2019
220 students across 4 labs
Future SEB rollout
Central labs 1400 seats (Feb 2020)
Other labs + 500 seats (later)
Total 1900 seats.
Safe Exam Browser (SEB) was used to secure the computer

6. Results – impact of technology problems on
student perceptions
The ratings given by students at the practice and two exam events.
The final exam (Post 2) experienced technical problems. This is reflected in the lower ratings.
Ease of use, reliability and security were significantly impacted (Sign Test: Post 1 v Post 2)
Chart shows the trend in ratings between events.
1
2
3
4
5
Pre Post 1 Post 2
I felt this particular exam suited the use of computers.
I felt the exam software was easy to use.
I felt the exam software was reliable against technical failures.
I felt the exam software was secure against cheating.
I would recommend this approach to doing exams to others.
Strongly
disagree
Strongly
agree
Z Sig.
-1.812 0.07
-3.816 0.00
-3.35 0.00
-1.925 0.05
-0.297 0.77
Sign Test
Significant differences as per Sign test results are shown
(see paper Table 2 and 3 for details).

7. Results – impact of technology problems on
student perceptions
Significant differences as per Mann-Whitney test results are shown in yellow (see paper Table 4).
-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4
[pre] Overall, I feel the e-Exam System is easy to use
[pre] I feel the e-Exam System is reliable against technical failures
[pre] I feel the e-Exam System is secure against cheating
[pre] I would recommend the e-Exam System to others
[Post] I felt this particular exam suited the use of computers
[Post] I felt the e-exam system was easy to use
[Post] I felt the e-exam system was reliable against technical failures
[Post] I felt the e-exam system was secure against cheating
[Post] I would recommend the e-exam system to others
[Post2] I felt this particular exam suited the use of computers
[Post2] I felt the e-exam system was easy to use
[Post2] I felt the e-exam system was reliable against technical…
[Post2] I felt the e-exam system was secure against cheating
[Post2] I would recommend the e-exam system to others
Post 2 event AM v PM sessions: difference ratings
The PM session of the Post 2 exam event experienced greater technical disruption. This is
reflected in the lower ratings given by the PM participants.
The chart shows the DIFFERENCE between the ratings given by AM and PM participants.

8. Other UNSW Lab Pilots in 2019
Strongly
agree
Post-exam student feedback (Mean)
BYOD USB Campus lab Campus lab Campus lab
SEB Moodle SEB Moodle SEB Moodle SEB Moodle
N: 107 26 216 17
3.8
4.5
4.4
4.2
3.5
4.0
3.6
3.5
1 2 3 4 5
I type faster then I handwrite
The written instructions were easy to follow
It was easy to start the exam
Overall I feel the software used for this exam was easy to use
I felt the eexam system was reliable against technical failures
I felt the eexam system was secure against cheating
I would recommend this approach to doing exams to others
I would like to use a computer for exams in the future
2018 Moansh SOLAR3010 BIOC2181 CHEM1011
Strongly
disagree
This paper!

9. Get in touch!
Dr Mathew Hillier
Academic Lead: Digital Assessment, Office of PVCE, UNSW
m.hillier@unsw.edu.au
http://mathewhillier.com
UNSW Digital Assessment tool kit (with cases etc by DA CoP group)
https://teaching.unsw.edu.au/digital-assessment-toolkit
Other Info:
DET national e-Exam project http://TransformingExams.com