Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Future of
Assessment
Challenges and
Opportunities
Dr Graeme Salter
Quick Quiz
Interested in working together?
Looking for the next challenge
Get in touch if we might be able to
work together …
Too late
Director
Entrepreneurship,
International and
Engagement
The Future of
Assessment
Let’s start with a story
In 1895,
H.G. Wells published
…
Time Travel Story
Hectic pace
Smartphones
ATMs
Aargh!!!
Retreated into an
older style building
something strangely familiar …
He had stumbled into a school
Not just schools
Testing Technology
Professor Steven Schwartz
Chair of the Australian Curriculum, Assessment
and Reporting Authority
“The t...
Aimed to provide a wirelessly enabled
specialist educational laptop to every
NSW public school student in Years 9 to
12 by...
Assessment Drives Learning
“Students can escape bad teaching,
Phil Race (2006) -
It is widely accepted now that
assessment...
Escaping bad teaching
An example -
Professor Thomas
Reeves
University of Georgia
Good Assessment
There are some great assessment innovators
Examples include -
• No Tech
• Low Tech
• High Tech
No Tech
University of Cambridge's School of Clinical Medicine –
Trainee doctors are presented with different situations,
i...
Technology isn’t always the solution
• Logitech presenter with timer
• Set the length of the presentation and
device vibra...
Low Tech - Clickers
What do you do if the policy
is no mobile devices?
Even Lower Tech - Plickers
Paper clickers
Does anyone know what a ‘plicker’ is?
High Tech
3D Collaboration Studio – USC
High Tech
3D Collaboration Studio – USC
Challenges and
Opportunities
Challenge 1
Scalability - most academic staff are not
innovators or early adopters
The easy road
We often end up measuring the things that
are easy to measure (e.g. information
recall) (Race, 2006)
(Text-based) Online Quizzes
Administrative efficiency?
Gareth Mills, FutureLab –
“What has happened is the automation of
many of the easy-to-automate ...
Shift the Curve
(Short) Exemplars and PD
Identifying Good Assessment
Publicly Acknowledged
Congratulated by Dean to all of School
Looking at using them as mentors in the …
Peer Assisted Teaching Scheme
Google ‘Peer Assisted Teaching Scheme’ for more info
Need to help colleagues
Race (2006) –
“We need to help colleagues –
• reduce the burden of assessment
(for students as wel...
Marking Tools
Turnitin
Educanon
Teammates
and many more …
Teammates (Peer Review)
Teammates (Peer Review)
Teammates (Peer Review)
Teammates (Peer Review)
Teammates (Peer Review)
Teammates (Peer Review)
Responding to Challenge
Techniques include -
• Address the ‘Why Change’ issue
Highlight and demonstrate benefits
• Profess...
Challenge 2
Technology can encourage poor, as well
as good, assessment practices
Quick Quiz
Are multiple choice exams an accurate
measure of one’s knowledge?
A. Yes
B. A and C
C. A and B
D. All of the ab...
The problem with MCQs
Most commonly used to promote memorisation
and factual recall (surface learning)
I forgot to make a ...
MCQ Construction
1. Real-world Scenarios
Before: What is the first concern of an emergency worker?
After: You arrive at th...
Integrated with other activities
‘Fundamentals of Human Physiology’
• previously used only reports as assessment
• changed...
Graded Alternatives
NB. This is a SURFACE example!!
What is the capital of Australia?
A Melbourne
B Sydney
C Glasgow
D Can...
Graded Alternatives
Blah, blah… What is the approximate ε450nm for the
coloured compound formed in the assay?
A. 0.199 mM-...
Dealing with misconceptions
One study found that learners performed
better when misconceptions were taken
into account
but...
Adaptive Mechanics Network
Misconceptions specifically addressed
Confidence Based Marking
MCQs and Context
Authentic, real-world contexts critically
important (not only for MCQs)
learn.alleninteractions.com/7-steps-improved-elearning-challenge
Virtual Health Assessment
Simulation using PowerPoint
Simulation using PowerPoint
Simulation using PowerPoint
Simulation using PowerPoint
Simulation using PowerPoint
To create an image hotspot cover it with a
transparent shape and hyperlink from the
shape
Simulation using PowerPoint
BONUS – Can save as an interactive PDF
Why use PowerPoint
when other tools allow richer interactions?
Looking at scalability by empowering
(sometimes reluctant) ...
Gamification & Effort
Gordon Commission Report –
• Assessment results are more likely to be
meaningful if students give ma...
The Crimescene Game
Why aren’t we doing more of this?
The technology is easy
Storyboarding and content is the time consuming part
Add Video
Use of video for patient interviews
Students get to compare with expert (doctor) interview
Add virtual worlds
Challenge 3
Where do we get the resources?
Don’t waste opportunities
HD video camera
always with
you!
Construction Management
Construction Management
Capture it!
Alumni and industry talks
great, but usually to a
small audience
• Captured video could be
used in many ways
•...
Cognitive Surplus
Have you hear of cognitive surplus?
Learning how to use free time
more constructively for creative
acts ...
PhD Research
Tap into cognitive surplus by using student
assessment tasks to develop Open
Educational Resources (OER)
Note...
Some are creators
Student Generated Content
Student Generated
Assessment Content
Fellenz (2004) developed what he called the
‘multiple-choice item development
assignm...
Student Generated Feedback
MICT unit - students making little change
based on feedback until …
student work put up on scre...
Challenge 4
Assessing what hasn’t been taught
(eg. presentation skills in computer
projects)
Assuming students learn by os...
Spot the mistakes
https://www.youtube.com/watch?v=ATfY8dvbuFg
Sandwich Feedback
Any ideas what I mean by sandwich
feedeback?
How I learned to do better presentations
Challenge 5An illustration
from a horse whisperer…
Well, you’ve been a
pretty good horse.
Hard working, not the
fastest, b...
Not everyone wants feedback
Some students focus on marks
“I spend all this time
providing feedback
and they don’t even
use it”
Multi-part assessments
Rather than an end of semester
assignment, could have a multi-part
assignment where feedback feeds ...
Challenge 6
Mass Education System
Can lead to …
but I hate
markingI love
teaching
and avoiding hard to assess skills
We sometimes avoid assessing skills that
are hard to assess (at least in bulk)
eg. let’...
Quick Quiz
Quick Quiz
Engineering graduates are the richest of
their prosperous peers, with an average
wealth of $US25.8 billion
Howe...
Carnegie Mellon Principles
We assess what we value – not what is
easy
Challenge 6
One size fits all
Carnegie Mellon Principles
One size does not fit all
Individual programs and courses use
different approaches that are app...
Negotiated Assessments
Learning contract negotiated
Project must meet listed specifications (eg.
include programming or mu...
Learner Profiles
Team roles for group projects based on -
• Personality surveys
• Self-reported
Adaptive Systems
Challenge 7
Are we there yet?
Sometimes we might just not be ready
(technologically or logistically)
December 2012
NAPLAN test to go online by 2016
April 2014
August 2014
many schools will not be ready …in less than two years
Challenge 8
That’s not the way we do it!
This is also our greatest opportunity –
pedagogical re-engineering
Use of Tools in Assessment
We use tools, not to cheat, but as a
way to increase our capacity for
critical and creative tho...
iPad Initiative
As part of the University's Blended
Learning Strategy, the use of technology
such as iPads, laptops and sm...
The iPad Exam
Understanding
“I think it should be a test of
understanding rather than a test
of remembering”
Dr Colin Sheringham
‘When Ready’ Assessment
No one feels cheated because their
driving test is not on the same day!
‘When Ready’ Assessment
We might have more well-balanced, when-
ready assessments rather than the set
piece, once-a-year, ...
Multi-disciplinary Opportunities
Low-cost 3D-printable Prosthetic Foot
Swinburne University
High-end prosthetic foot can r...
Multi-disciplinary Opportunities
Multi-disciplinary Opportunities
Responding to the Challenges
• Assessment tasks that encourage meaningful
learning
• Varied assessment tasks
• Meaningful ...
Discussion
What trends do you see happening in the
future of assessment?
Questions?
The Future of
Assessment
Challenges and
Opportunities
Dr Graeme Salter
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
The Future of Assessment - Moodleposium 2015
Upcoming SlideShare
Loading in …5
×

The Future of Assessment - Moodleposium 2015

523 views

Published on

Technology is changing the way we deliver education. It is also changing our methods for assessment - in both good and bad ways!

Published in: Education
  • Be the first to comment

  • Be the first to like this

The Future of Assessment - Moodleposium 2015

  1. 1. The Future of Assessment Challenges and Opportunities Dr Graeme Salter
  2. 2. Quick Quiz
  3. 3. Interested in working together? Looking for the next challenge Get in touch if we might be able to work together …
  4. 4. Too late Director Entrepreneurship, International and Engagement
  5. 5. The Future of Assessment Let’s start with a story In 1895, H.G. Wells published …
  6. 6. Time Travel Story
  7. 7. Hectic pace Smartphones ATMs Aargh!!!
  8. 8. Retreated into an older style building
  9. 9. something strangely familiar … He had stumbled into a school
  10. 10. Not just schools
  11. 11. Testing Technology Professor Steven Schwartz Chair of the Australian Curriculum, Assessment and Reporting Authority “The testing technology used in our grandparents’ day (handwritten examinations) is simply not suitable today”
  12. 12. Aimed to provide a wirelessly enabled specialist educational laptop to every NSW public school student in Years 9 to 12 by 2012. Offer not taken up by the School my children attended. Principal didn’t want reliance on technology when the HSC was still paper-based.
  13. 13. Assessment Drives Learning “Students can escape bad teaching, Phil Race (2006) - It is widely accepted now that assessment is the major driver for student learning but they can’t escape bad assessment” David Boud
  14. 14. Escaping bad teaching An example - Professor Thomas Reeves University of Georgia
  15. 15. Good Assessment There are some great assessment innovators Examples include - • No Tech • Low Tech • High Tech
  16. 16. No Tech University of Cambridge's School of Clinical Medicine – Trainee doctors are presented with different situations, in which the patients are played by actors, such as having to break bad news to family members and explaining the next stage of treatment to a patient.
  17. 17. Technology isn’t always the solution • Logitech presenter with timer • Set the length of the presentation and device vibrates with 5, 2 and 1 minute to go • I wanted a 10 minute warning • Spent an hour looking for how to adjust the time to 10 minutes • Any ideas about what the non-technical solution was?
  18. 18. Low Tech - Clickers What do you do if the policy is no mobile devices?
  19. 19. Even Lower Tech - Plickers Paper clickers Does anyone know what a ‘plicker’ is?
  20. 20. High Tech 3D Collaboration Studio – USC
  21. 21. High Tech 3D Collaboration Studio – USC
  22. 22. Challenges and Opportunities
  23. 23. Challenge 1 Scalability - most academic staff are not innovators or early adopters
  24. 24. The easy road We often end up measuring the things that are easy to measure (e.g. information recall) (Race, 2006)
  25. 25. (Text-based) Online Quizzes
  26. 26. Administrative efficiency? Gareth Mills, FutureLab – “What has happened is the automation of many of the easy-to-automate processes of traditional assessment. This includes the marking of multiple-choice questions and the crunching and analysis of big data. The application of technology has tended to serve the needs of administrative efficiency rather than trigger genuine transformation”
  27. 27. Shift the Curve
  28. 28. (Short) Exemplars and PD
  29. 29. Identifying Good Assessment
  30. 30. Publicly Acknowledged Congratulated by Dean to all of School Looking at using them as mentors in the …
  31. 31. Peer Assisted Teaching Scheme Google ‘Peer Assisted Teaching Scheme’ for more info
  32. 32. Need to help colleagues Race (2006) – “We need to help colleagues – • reduce the burden of assessment (for students as well as for us) • while at the same time improving its quality”
  33. 33. Marking Tools Turnitin Educanon Teammates and many more …
  34. 34. Teammates (Peer Review)
  35. 35. Teammates (Peer Review)
  36. 36. Teammates (Peer Review)
  37. 37. Teammates (Peer Review)
  38. 38. Teammates (Peer Review)
  39. 39. Teammates (Peer Review)
  40. 40. Responding to Challenge Techniques include - • Address the ‘Why Change’ issue Highlight and demonstrate benefits • Professional development opportunities • Exemplars • Identifying good practice • Encouraging excellence (and effort) • Assistance schemes (eg. PATs) • Tools that reduce the burden
  41. 41. Challenge 2 Technology can encourage poor, as well as good, assessment practices
  42. 42. Quick Quiz Are multiple choice exams an accurate measure of one’s knowledge? A. Yes B. A and C C. A and B D. All of the above
  43. 43. The problem with MCQs Most commonly used to promote memorisation and factual recall (surface learning) I forgot to make a back-up copy of my brain, so everything I learned last semester was lost
  44. 44. MCQ Construction 1. Real-world Scenarios Before: What is the first concern of an emergency worker? After: You arrive at the scene of an accident where people are panicked and yelling. Three people appear to be injured. What action will you take first? 2. Analysis of Visuals Before: Select the best definition of active listening. After: Which video best demonstrates active listening during a call with an irate customer? 3. The Answer Plus The Reason Why Before: What are three signs of edema? After: A patient entered the hospital with edema of both lower extremities. What action should the nurse take and why?
  45. 45. Integrated with other activities ‘Fundamentals of Human Physiology’ • previously used only reports as assessment • changed one assessment item to regular quizzes • provided formative feedback to students • provided feedback to lecturer which was used to shape future seminars
  46. 46. Graded Alternatives NB. This is a SURFACE example!! What is the capital of Australia? A Melbourne B Sydney C Glasgow D Canberra E Auckland • The more complex the question, the more important partial marks become. 0.5 0.5 -1 1 0
  47. 47. Graded Alternatives Blah, blah… What is the approximate ε450nm for the coloured compound formed in the assay? A. 0.199 mM-1 cm-1 B. 250 mM-1 cm-1 C. 4 mM-1 cm-1 D. 2 mM-1 cm-1 E. 19.6 mM-1 cm-1 Each option reveals a particular mistake Especially good with a multi-step calculation Can reward process
  48. 48. Dealing with misconceptions One study found that learners performed better when misconceptions were taken into account but even better when the teacher was well aware of the misconceptions
  49. 49. Adaptive Mechanics Network Misconceptions specifically addressed
  50. 50. Confidence Based Marking
  51. 51. MCQs and Context Authentic, real-world contexts critically important (not only for MCQs)
  52. 52. learn.alleninteractions.com/7-steps-improved-elearning-challenge
  53. 53. Virtual Health Assessment
  54. 54. Simulation using PowerPoint
  55. 55. Simulation using PowerPoint
  56. 56. Simulation using PowerPoint
  57. 57. Simulation using PowerPoint
  58. 58. Simulation using PowerPoint To create an image hotspot cover it with a transparent shape and hyperlink from the shape
  59. 59. Simulation using PowerPoint BONUS – Can save as an interactive PDF
  60. 60. Why use PowerPoint when other tools allow richer interactions? Looking at scalability by empowering (sometimes reluctant) academic staff
  61. 61. Gamification & Effort Gordon Commission Report – • Assessment results are more likely to be meaningful if students give maximum effort. • Electronic game designers seem to have found ways to get students to give that effort
  62. 62. The Crimescene Game
  63. 63. Why aren’t we doing more of this? The technology is easy Storyboarding and content is the time consuming part
  64. 64. Add Video Use of video for patient interviews Students get to compare with expert (doctor) interview
  65. 65. Add virtual worlds
  66. 66. Challenge 3 Where do we get the resources?
  67. 67. Don’t waste opportunities HD video camera always with you!
  68. 68. Construction Management
  69. 69. Construction Management
  70. 70. Capture it! Alumni and industry talks great, but usually to a small audience • Captured video could be used in many ways • eg. welcome videos to demonstrate relevance (more likely to listen than grey- haired Directors of Engagement)
  71. 71. Cognitive Surplus Have you hear of cognitive surplus? Learning how to use free time more constructively for creative acts rather than consumptive ones, particularly with the advent of online tools that allow new forms of collaboration
  72. 72. PhD Research Tap into cognitive surplus by using student assessment tasks to develop Open Educational Resources (OER) Note - our research found that most digital natives were content consumers not content creators (apart from uploading pictures to Instagram and Facebook)
  73. 73. Some are creators
  74. 74. Student Generated Content
  75. 75. Student Generated Assessment Content Fellenz (2004) developed what he called the ‘multiple-choice item development assignment’ (MCIDA). Students were briefed on MCQ construction and in tutorials they had opportunities to discuss, question and critique MCQs and to learn how to classify them in relation to Bloom’s (1956) taxonomy of educational objectives
  76. 76. Student Generated Feedback MICT unit - students making little change based on feedback until … student work put up on screen in front of all students and critiqued by all may be confronting for some students (can use techniques such as sandwich feedback)
  77. 77. Challenge 4 Assessing what hasn’t been taught (eg. presentation skills in computer projects) Assuming students learn by osmosis
  78. 78. Spot the mistakes https://www.youtube.com/watch?v=ATfY8dvbuFg
  79. 79. Sandwich Feedback Any ideas what I mean by sandwich feedeback? How I learned to do better presentations
  80. 80. Challenge 5An illustration from a horse whisperer… Well, you’ve been a pretty good horse. Hard working, not the fastest, but … No, stupid, I said I wanted a feedbag NOT feedback
  81. 81. Not everyone wants feedback Some students focus on marks
  82. 82. “I spend all this time providing feedback and they don’t even use it”
  83. 83. Multi-part assessments Rather than an end of semester assignment, could have a multi-part assignment where feedback feeds into the next part Eg. presentation skills practice sessions (I would rather see learning and improvement than assessment used simply for sorting our grades)
  84. 84. Challenge 6 Mass Education System
  85. 85. Can lead to … but I hate markingI love teaching
  86. 86. and avoiding hard to assess skills We sometimes avoid assessing skills that are hard to assess (at least in bulk) eg. let’s not set a presentation for our class of 1,000 students Some of these skills are what employers are asking for
  87. 87. Quick Quiz
  88. 88. Quick Quiz Engineering graduates are the richest of their prosperous peers, with an average wealth of $US25.8 billion However, the report suggests that multi- millionaires in the making might be better off forgoing university altogether, as almost a third of the wealthiest people in the world do not have degrees.
  89. 89. Carnegie Mellon Principles We assess what we value – not what is easy
  90. 90. Challenge 6 One size fits all
  91. 91. Carnegie Mellon Principles One size does not fit all Individual programs and courses use different approaches that are appropriate in different contexts for different objectives and goals
  92. 92. Negotiated Assessments Learning contract negotiated Project must meet listed specifications (eg. include programming or multimedia) Project can be for • you to learn a new skill • for a potential business • for an existing business • for another unit Golf store example
  93. 93. Learner Profiles Team roles for group projects based on - • Personality surveys • Self-reported
  94. 94. Adaptive Systems
  95. 95. Challenge 7 Are we there yet? Sometimes we might just not be ready (technologically or logistically)
  96. 96. December 2012 NAPLAN test to go online by 2016
  97. 97. April 2014
  98. 98. August 2014 many schools will not be ready …in less than two years
  99. 99. Challenge 8 That’s not the way we do it! This is also our greatest opportunity – pedagogical re-engineering
  100. 100. Use of Tools in Assessment We use tools, not to cheat, but as a way to increase our capacity for critical and creative thought. Gareth Mills, Futurelab
  101. 101. iPad Initiative As part of the University's Blended Learning Strategy, the use of technology such as iPads, laptops and smartphones in tutorial rooms and lecture theatres is a common sight. Breaking ground in this area, the School of Business unit titled Managing Service and Experience successfully trialled the use of iPads during its Autumn session final exam.
  102. 102. The iPad Exam
  103. 103. Understanding “I think it should be a test of understanding rather than a test of remembering” Dr Colin Sheringham
  104. 104. ‘When Ready’ Assessment No one feels cheated because their driving test is not on the same day!
  105. 105. ‘When Ready’ Assessment We might have more well-balanced, when- ready assessments rather than the set piece, once-a-year, no re-sits culture that drives assessment at the moment. If we can get technology assessment to scale in an important arena like driving, why not in others? Gareth Mills, FutureLabs
  106. 106. Multi-disciplinary Opportunities Low-cost 3D-printable Prosthetic Foot Swinburne University High-end prosthetic foot can retail for as much as $5000 (USD) 3D printed -$15 wholesale
  107. 107. Multi-disciplinary Opportunities
  108. 108. Multi-disciplinary Opportunities
  109. 109. Responding to the Challenges • Assessment tasks that encourage meaningful learning • Varied assessment tasks • Meaningful contexts • Rich, detailed feedback • Adaptive feedback • Personalistion • Using technology for pedagogical as well as administrative purposes • Using analytics for pedagogical as well as administrative purposes
  110. 110. Discussion What trends do you see happening in the future of assessment?
  111. 111. Questions? The Future of Assessment Challenges and Opportunities Dr Graeme Salter

×