High stakes assessments are a physical, logistical, and technical challenge, but the payoffs for a successful implementation are huge. AWS gives us the ability to scale from ad-hoc assessment to large scale end of semester exams - this provides the flexibility to deliver reliable, authenticate assessment at very large scale.

1. P U B L I C S E C T O R
S U M M I T
C anberra, AC T

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.P U B L I C S E C T O R
S U M M I T
Scalable Authentic Assessment for
the Information Age
Cliff Ashford
Delivery Lead, Research and Education Services
Monash University

3. ,

4. Any idea what he’s doing?

6. 6

7. What?
Creation
• Ways to test
knowledge
• Validating method
• Reuse/rework
• Security
Staging
• Quality assurance
• Scheduling
• Risk planning
• Security
• Measurement
Sitting
• Invigilation
• Monitoring
• Medical
• Cheating
• Security
• Measurement
Marking
• Management
• Bias
• Adjustment
• Security

8. Mountains
of paper
Whiteboard
administration
Error
prone
Student
experience
Limited
authenticity
Manual
analytics
Why?

9. Technology
Assessment
Services
Pedagogy
Monash
Pedagogy
Authentic assessment, sound
pedagogical design, faculty and
student engagement, education
innovation
Technology
Platform stability and
robustness, security, wi-fi,
scalability, technical
development, user experience,
laptops, BYOD, venue fit-out
Assessment
Services
Operational exam delivery,
event management, venue set
up, invigilation, exam
creation, BYOD, contingency
arrangements, delivering at
scale
Cliff Ashford, Delivery
Lead, eSolutions
Kris Ryan,
Pro Vice-Chancellor
(Education)
Vern Garth,
Assoc. Director,
Exam Services, SEBS
Who?

10. Who?

11. 2015 2016 2017 2018 2019 2020
Small Pilot
• Three units
• Faculty of law, business, economics, and IT
• Vendor managed software and hardware
• 100 students per sitting
• Multiple choice and short answer
• Wi-Fi network
“We need to do
something in this
area…”
Timeline…

12. 2015 2016 2017 2018 2019 2020
Market Scan
• Five vendors
• Seven local and international
universities
Re-examine Reasoning
• Core function of an institution
• Masters of own destiny
• Focus on student experience
• Understand all aspects of high risk activity
• Move beyond the concept of ‘exams’
Timeline…

13. 2015 2016 2017 2018 2019 2020
Let’s do it ourselves…
• Express a vision
• Identify platform
• Choose a partner
• Convince the university
• Engage with students
• … and later staff
Plan for the Future
Build for the Now
• Get everyone we can off paper
• Initiate BYOD policy, but provide laptops now
• Ensure absolute reliability
• Focus first on largest impact: Students
• Engage key innovators in the future state
Timeline…

14. Scale
8%
10,000
seats
Sem 2
2018
600 seats
Sem 1
2018
½%
30%
37,000
seats
Sem 1 2019
34%
42,000 seats
Sem 2 2019
80%
100,000 seats
Sem 2 2020

15. Scalable InterdisciplinaryEquitable
Independently
assessed
Evidence
based
Outcome
focussed
Non-negotiables

16. 80% ONLINE
Assuming 50% decrease in
marking time for long-form
(essay) questions only
$7,000,000*
Savings per annum
* Based on Monash findings combined
with literature review
The money slide

17. Off-the-shelf vs on
eAssessment: Buy vs Build (2018/19)
Buy Build

18. A lean process by
reducing manual
handling and possibilities
for error
Scalable efficiencies
Embracing automation
and the flexibility of cloud
services
Faster ROI by getting
something working
then iterate
enhancements
Leverage modern services
with analytics, security, and
privacy embedded from the
beginning
Sustainable – Learn
the problem space
and never repeat
mistakes
1 3
2 4
5
Industry best practice

19. Vital partners

20. Scalability
O Week Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Break Week 8 Week 9 Week 10 Week 11 Week 12 Swot Exam 1 Exam 2 Exam 3 Break
eAssessment: Typical Semester Load
Academic Student

21. The system

22. The Physical
Implementation

23. Caulfield
racecourse
Monash University
Triple redundant
network connection
Big, bold, and secure

24. 2 x 10Gb
dark fibre
40 trollies
1,500 laptops
Infrastructure
60 video
cameras
$80,000
patch leads
850m
cable ties
56 km of
network cable

25. The Response

26. Post exam surveys

27. • Overall, less time marking
• 100% legible responses
• Mark as soon as exam finishes
• Mark exams anywhere
• Chief examiners could track
progress of batches
• Access to analytics
• Exam duration same as paper
• True blind marking
• Same as in-semester experience
• Future possibilities
• Mock exams prepared students
What they liked so far… What they want next…
• Exam marking usability
• Question databank
• Extend question functionality
• Analytics locked until marking complete
• Clearer processes within faculty
• Gradebook and callista (SMS) integration
• Freehand support
Focus groups

28. “My goodness, this is absolutely wonderful. Where has this system
been all my academic life..?
“No exam bags, ease of use of the system, sharing among markers,
READING the student responses, no handwriting of marks, blinded
marking...bloody brilliant.”
Associate Dean (Education)
Faculty of Pharmacy and Pharmaceutical Sciences
Academic staff feedback

29. “Where goats eat the internet”

30. Headline outcomes
• 77% of students would recommend
eAssessment to other students
• 70% of students would prefer taking
an eAssessment to a paper exam
“I really loved it, I wish my other
exams this semester are also done
through e-assessment. It’s so much
easier to edit and clarify my thoughts
and I think it helped improve the
quality of my answers”
“Overall, being able to type the
exam significantly aided my
performance as I barely
handwrite nowadays.”
Invigilated and take-home mock
exams
Drop-in sessions
Co-designed interface
Library support
In-class support materials
Student feedback

31. Business and pedagogy
Scale
Authenticity
Paper
VR/AR Laptop/quiz
BYOD/quiz
BYOD/apps
Optimal solution
Laptop/appsWIL/OSCE

32. 2015 2016 2017 2018 2019 2020
Where next?
3D oesophagus model, courtesy of Virtro Entertainment Inc. Vancouver, BC

33. 2015 2016 2017 2018 2019 2020
Where next?

34. Project scale
• Core team 20
• Execution 250 - 300 Monash staff
• Managed Service for AWS hosting
Primary focus areas
• Student experience
• Resilience
• Performance
Core principles
• More flexible assessment enables better learning
opportunities
• Any aspect of the project may be vetoed by any
of the three sponsorship areas
Monash scale
• Students: 75,000
• ‘High-stakes’ sittings: 360,000
• On-campus per annum: 250,000
Development timeline
• Feb 2018: Development start
• May 2018: 600 sittings
• Nov 2018: 10,000 sittings
• Jun 2019: 37,000 sittings
• Nov 2019: 42,000 sittings
Operational scale
• Venue capacity 4,500 seats
• Supplied 1,500 laptops
Key takeaways