1. The Future of
Assessment
Challenges and
Opportunities
Dr Graeme Salter

2. Quick Quiz

3. Interested in working together?
Looking for the next challenge
Get in touch if we might be able to
work together …

4. Too late
Director
Entrepreneurship,
International and
Engagement

5. The Future of
Assessment
Let’s start with a story
In 1895,
H.G. Wells published
…

6. Time Travel Story

7. Hectic pace
Smartphones
ATMs
Aargh!!!

8. Retreated into an
older style building

9. something strangely familiar …
He had stumbled into a school

10. Not just schools

11. Testing Technology
Professor Steven Schwartz
Chair of the Australian Curriculum, Assessment
and Reporting Authority
“The testing technology used in
our grandparents’ day
(handwritten examinations)
is simply not suitable today”

12. Aimed to provide a wirelessly enabled
specialist educational laptop to every
NSW public school student in Years 9 to
12 by 2012.
Offer not taken up by the School my
children attended.
Principal didn’t want reliance on
technology when the HSC was still
paper-based.

13. Assessment Drives Learning
“Students can escape bad teaching,
Phil Race (2006) -
It is widely accepted now that
assessment is the major driver for
student learning
but they can’t escape bad
assessment”
David Boud

14. Escaping bad teaching
An example -
Professor Thomas
Reeves
University of Georgia

15. Good Assessment
There are some great assessment innovators
Examples include -
• No Tech
• Low Tech
• High Tech

16. No Tech
University of Cambridge's School of Clinical Medicine –
Trainee doctors are presented with different situations,
in which the patients are played by actors, such as
having to break bad news to family members and
explaining the next stage of treatment to a patient.

17. Technology isn’t always the solution
• Logitech presenter with timer
• Set the length of the presentation and
device vibrates with 5, 2 and 1 minute to go
• I wanted a 10 minute
warning
• Spent an hour looking
for how to adjust the
time to 10 minutes
• Any ideas about what
the non-technical
solution was?

18. Low Tech - Clickers
What do you do if the policy
is no mobile devices?

19. Even Lower Tech - Plickers
Paper clickers
Does anyone know what a ‘plicker’ is?

20. High Tech
3D Collaboration Studio – USC

21. High Tech
3D Collaboration Studio – USC

22. Challenges and
Opportunities

23. Challenge 1
Scalability - most academic staff are not
innovators or early adopters

24. The easy road
We often end up measuring the things that
are easy to measure (e.g. information
recall) (Race, 2006)

25. (Text-based) Online Quizzes

26. Administrative efficiency?
Gareth Mills, FutureLab –
“What has happened is the automation of
many of the easy-to-automate processes of
traditional assessment.
This includes the marking of multiple-choice
questions and the crunching and analysis of
big data.
The application of technology has tended to
serve the needs of administrative efficiency
rather than trigger genuine
transformation”

27. Shift the Curve

28. (Short) Exemplars and PD

29. Identifying Good Assessment

30. Publicly Acknowledged
Congratulated by Dean to all of School
Looking at using them as mentors in the …

31. Peer Assisted Teaching Scheme
Google ‘Peer Assisted Teaching Scheme’ for more info

32. Need to help colleagues
Race (2006) –
“We need to help colleagues –
• reduce the burden of assessment
(for students as well as for us)
• while at the same time improving its
quality”

33. Marking Tools
Turnitin
Educanon
Teammates
and many more …

34. Teammates (Peer Review)

35. Teammates (Peer Review)

36. Teammates (Peer Review)

37. Teammates (Peer Review)

38. Teammates (Peer Review)

39. Teammates (Peer Review)

40. Responding to Challenge
Techniques include -
• Address the ‘Why Change’ issue
Highlight and demonstrate benefits
• Professional development opportunities
• Exemplars
• Identifying good practice
• Encouraging excellence (and effort)
• Assistance schemes (eg. PATs)
• Tools that reduce the burden

41. Challenge 2
Technology can encourage poor, as well
as good, assessment practices

42. Quick Quiz
Are multiple choice exams an accurate
measure of one’s knowledge?
A. Yes
B. A and C
C. A and B
D. All of the above

43. The problem with MCQs
Most commonly used to promote memorisation
and factual recall (surface learning)
I forgot to make a back-up copy of my brain,
so everything I learned last semester was lost

44. MCQ Construction
1. Real-world Scenarios
Before: What is the first concern of an emergency worker?
After: You arrive at the scene of an accident where people
are panicked and yelling. Three people appear to be
injured. What action will you take first?
2. Analysis of Visuals
Before: Select the best definition of active listening.
After: Which video best demonstrates active listening
during a call with an irate customer?
3. The Answer Plus The Reason Why
Before: What are three signs of edema?
After: A patient entered the hospital with edema of both
lower extremities. What action should the nurse take and
why?

45. Integrated with other activities
‘Fundamentals of Human Physiology’
• previously used only reports as assessment
• changed one assessment item to regular
quizzes
• provided formative feedback to students
• provided feedback to lecturer which was
used to shape future seminars

46. Graded Alternatives
NB. This is a SURFACE example!!
What is the capital of Australia?
A Melbourne
B Sydney
C Glasgow
D Canberra
E Auckland
• The more complex the question, the more important
partial marks become.
0.5
0.5
-1
1
0

47. Graded Alternatives
Blah, blah… What is the approximate ε450nm for the
coloured compound formed in the assay?
A. 0.199 mM-1 cm-1
B. 250 mM-1 cm-1
C. 4 mM-1 cm-1
D. 2 mM-1 cm-1
E. 19.6 mM-1 cm-1
Each option reveals a
particular mistake
Especially good with a
multi-step calculation
Can reward process

48. Dealing with misconceptions
One study found that learners performed
better when misconceptions were taken
into account
but even better when the teacher was well
aware of the misconceptions

49. Adaptive Mechanics Network
Misconceptions specifically addressed

50. Confidence Based Marking

51. MCQs and Context
Authentic, real-world contexts critically
important (not only for MCQs)

52. learn.alleninteractions.com/7-steps-improved-elearning-challenge

56. Virtual Health Assessment

57. Simulation using PowerPoint

58. Simulation using PowerPoint

59. Simulation using PowerPoint

60. Simulation using PowerPoint

61. Simulation using PowerPoint
To create an image hotspot cover it with a
transparent shape and hyperlink from the
shape

62. Simulation using PowerPoint
BONUS – Can save as an interactive PDF

63. Why use PowerPoint
when other tools allow richer interactions?
Looking at scalability by empowering
(sometimes reluctant) academic staff

64. Gamification & Effort
Gordon Commission Report –
• Assessment results are more likely to be
meaningful if students give maximum effort.
• Electronic game designers seem to have
found ways to get students to give that
effort

65. The Crimescene Game

70. Why aren’t we doing more of this?
The technology is easy
Storyboarding and content is the time consuming part

71. Add Video
Use of video for patient interviews
Students get to compare with expert (doctor) interview

72. Add virtual worlds

74. Challenge 3
Where do we get the resources?

75. Don’t waste opportunities
HD video camera
always with
you!

76. Construction Management

77. Construction Management

78. Capture it!
Alumni and industry talks
great, but usually to a
small audience
• Captured video could be
used in many ways
• eg. welcome videos to
demonstrate relevance (more
likely to listen than grey-
haired Directors of
Engagement)

79. Cognitive Surplus
Have you hear of cognitive surplus?
Learning how to use free time
more constructively for creative
acts rather than consumptive
ones, particularly with the advent
of online tools that allow new
forms of collaboration

80. PhD Research
Tap into cognitive surplus by using student
assessment tasks to develop Open
Educational Resources (OER)
Note - our research found that most digital
natives were content consumers not
content creators
(apart from uploading pictures to Instagram
and Facebook)

81. Some are creators

82. Student Generated Content

83. Student Generated
Assessment Content
Fellenz (2004) developed what he called the
‘multiple-choice item development
assignment’ (MCIDA).
Students were briefed on MCQ construction and in
tutorials they had opportunities to discuss,
question and critique MCQs and to learn how to
classify them in relation to Bloom’s (1956)
taxonomy of educational objectives

84. Student Generated Feedback
MICT unit - students making little change
based on feedback until …
student work put up on screen in front of all
students and critiqued by all
may be confronting for some students (can use
techniques such as sandwich feedback)

85. Challenge 4
Assessing what hasn’t been taught
(eg. presentation skills in computer
projects)
Assuming students learn by osmosis

86. Spot the mistakes
https://www.youtube.com/watch?v=ATfY8dvbuFg

87. Sandwich Feedback
Any ideas what I mean by sandwich
feedeback?
How I learned to do better presentations

88. Challenge 5An illustration
from a horse whisperer…
Well, you’ve been a
pretty good horse.
Hard working, not the
fastest, but …
No, stupid, I
said I wanted a
feedbag NOT
feedback

89. Not everyone wants feedback
Some students focus on marks

90. “I spend all this time
providing feedback
and they don’t even
use it”

91. Multi-part assessments
Rather than an end of semester
assignment, could have a multi-part
assignment where feedback feeds into
the next part
Eg. presentation skills practice sessions
(I would rather see learning and
improvement than assessment used
simply for sorting our grades)

92. Challenge 6
Mass Education System

93. Can lead to …
but I hate
markingI love
teaching

94. and avoiding hard to assess skills
We sometimes avoid assessing skills that
are hard to assess (at least in bulk)
eg. let’s not set a presentation for our
class of 1,000 students
Some of these skills are what employers
are asking for

95. Quick Quiz

96. Quick Quiz
Engineering graduates are the richest of
their prosperous peers, with an average
wealth of $US25.8 billion
However, the report suggests that multi-
millionaires in the making might be better
off forgoing university altogether, as almost
a third of the wealthiest people in the
world do not have degrees.

97. Carnegie Mellon Principles
We assess what we value – not what is
easy

98. Challenge 6
One size fits all

100. Carnegie Mellon Principles
One size does not fit all
Individual programs and courses use
different approaches that are appropriate
in different contexts for different
objectives and goals

101. Negotiated Assessments
Learning contract negotiated
Project must meet listed specifications (eg.
include programming or multimedia)
Project can be for
• you to learn a new skill
• for a potential business
• for an existing business
• for another unit
Golf store example

102. Learner Profiles
Team roles for group projects based on -
• Personality surveys
• Self-reported

103. Adaptive Systems

104. Challenge 7
Are we there yet?
Sometimes we might just not be ready
(technologically or logistically)

105. December 2012
NAPLAN test to go online by 2016

106. April 2014

107. August 2014
many schools will not be ready …in less than two years

108. Challenge 8
That’s not the way we do it!
This is also our greatest opportunity –
pedagogical re-engineering

109. Use of Tools in Assessment
We use tools, not to cheat, but as a
way to increase our capacity for
critical and creative thought.
Gareth Mills, Futurelab

110. iPad Initiative
As part of the University's Blended
Learning Strategy, the use of technology
such as iPads, laptops and smartphones
in tutorial rooms and lecture theatres is a
common sight.
Breaking ground in this area, the School of
Business unit titled Managing Service
and Experience successfully trialled the
use of iPads during its Autumn
session final exam.

112. The iPad Exam

113. Understanding
“I think it should be a test of
understanding rather than a test
of remembering”
Dr Colin Sheringham

114. ‘When Ready’ Assessment
No one feels cheated because their
driving test is not on the same day!

115. ‘When Ready’ Assessment
We might have more well-balanced, when-
ready assessments rather than the set
piece, once-a-year, no re-sits culture that
drives assessment at the moment. If we
can get technology assessment to scale
in an important arena like driving, why
not in others?
Gareth Mills, FutureLabs

116. Multi-disciplinary Opportunities
Low-cost 3D-printable Prosthetic Foot
Swinburne University
High-end prosthetic foot can retail for as
much as $5000 (USD)
3D printed -$15 wholesale

117. Multi-disciplinary Opportunities

118. Multi-disciplinary Opportunities

119. Responding to the Challenges
• Assessment tasks that encourage meaningful
learning
• Varied assessment tasks
• Meaningful contexts
• Rich, detailed feedback
• Adaptive feedback
• Personalistion
• Using technology for pedagogical as well as
administrative purposes
• Using analytics for pedagogical as well as
administrative purposes

120. Discussion
What trends do you see happening in the
future of assessment?

121. Questions?
The Future of
Assessment
Challenges and
Opportunities
Dr Graeme Salter