EdTech in Assessment: Sinner or
Innovations and Trends in
Wednesday 29th November
Slides available on Slideshare at:
• Senior Lecturer/Flexible Learning Advisor, AUT
• FLLinNZ 2004 - Ako Academy 2007
• Lecturer/eLearning Advisor, Curtin Uni 2011-2012
• Learning Design Coordinator, UniMelb 2012-2017
• Director, Learning Design and Systems, Learning
and Teaching DirectorateEastern Health 2017 --->
Effective assessment has always been a challenge in
education, but appears to be even more so with the
rapid advances being made with new technologies.
However, technology is just one disruptor as we start to
question and reflect on the nature of assessment itself.
This session with explore some of the challenges we
face in education around assessment and technology,
which might be our saviour as it has been our bane.
Perhaps it is time for a paradigm shift in assessment to
move into the future with confidence?
Ascilite blog post – 31st July 2017
AUT is now working on
“implementing mātauranga Māori
into our curriculum.” Robert Hogg,
mātauranga Māori literally means
1. Start with the basics, 21st century
pedagogy, LOs, reliability, validity and
2. Tried and true challenges, F2F, online
3. Some ideas for solutions to these (and
other) challenges – practice, tech
4. On the horizon?
5. Te Wero – What are you going to do?
Assessment in a 21st century pedagogy
• Timely and meaningful
• Relevant tasks
• Self and peer assessment
• Clear, transparent goals and
TeachThought - A Diagram Of 21st Century Pedagogy -
“Good assessment can be a learning experience”
NZ Herald – 15th November 2017
Learning Objectives – “the sinister 16”
verbs that are are passive, internal and/or otherwise unobservable
• Have a knowledge of
• Be aware of
• Be conscious of
• Be familiar with
Potter, M.K., and Kustra, E. (2012). Course design for Constructive Alignment. – A primer
on learning outcomes and the SOLO Taxonomy. Centre for Teaching and Learning,
University of Windsor.
Reliability and Validity
Reliability – will the instrument produce consistent
results between different assessments?
Validity – are we measuring what we say we are
measuring? Is the assessment valid?
“Professor Millet’s study looked at data from more
than 50,000 course sections at a North American
university over several years. He found that
variability did not occur only within particular
departments, but also among individual faculty
Millet, I. (2016). The relationship between grading
leniency and grading reliability
Traditional written papers (2-8 pages)
Audio Recordings (student created)
Open discussion (discussion board)
Paired discussion (2-5 students)
Response to video
Twitter summaries (distillation)
Screencasts (student created)
Bailey, S., Hendricks, S., Applewhite, S. (2015). Student
Perspectives of Assessment Strategies in Online Courses, Journal
of Interactive Online Learning, v.13, n. 3, 112-25.
Assessment is like a box of
Chocolates – Chapter 1 by
Lisa Maxfield, California
State University, Long
Beach, In “Coming to
Terms with Student
Administrators” edited by
Peggy Maki (2010).
Adds a layer of
safety (physical and
Formative assessment is
more than a or a
• An opportunity to Fail
• Fun? Gamification?
eTools for assessment – many already in LMS (to some degree)
Deploy assessments (peer
AES – automated essay scoring
Blockchain - authentication /
badging / certification
AI - Artificial intelligence
analyticsCheating on assessments
F2F: the old ways are still being used!
“…writing cheat notes on their bodies and calculators, using
information stored on smartphones, and copying the work of those
sitting next to them.”
Dough, P. (2016). Universities bust more than 80 exam cheats. NZ Herald
“Illicit notes and copying others remain common methods adopted
by cheaters, alongside more modern trends such as using
“Universities found students hiding notes in the toilets and taking a
toilet break to check them, sneaking notes into the exam hall, or
whispering to others during the exam. Incidents of plagiarism and
passing notes were also uncovered.”
F2F: However, there are ‘new’ technologies
Thai university students caught using spy cameras, linked to smartwatches to cheat on medicine
Arthit Ourairat, the rector of Rangsit University, posted pictures of the hi-tech cheating equipment on
his Facebook page, announcing that the entrance exam in question had been cancelled after the plot
Smartwatches that allow pupils to 'cheat' in exams for sale on Amazon. Deputy head says such devices
are making exams a 'nightmare to administer’
Good luck bringing your watch into your exam - Massey University has banned watches from
examination rooms. http://netguide.co.nz/story/good-luck-bringing-your-watch-your-exam/
Apple Watch banned from HSC exams and schools -
Australia New ZealandStudents at Sydney University use impersonators to sit their
exams - August 11th 2015
Sydney University medical students invented patients for
assignments - June 6th 2015
Macquarie University revokes degrees for students caught buying
essays in MyMaster cheating racket - May 28th 2015
Sydney University, University of NSW and UTS crack down on
cheating students - April 25th 2016
Deakin University students kicked out for 'contract cheating’ - May
18th 2016 http://www.theage.com.au/victoria/deakin-university-students-
University of Sydney's medical school in second cheating
controversy May 26th 2016 -
Student comes up with 'ingenious' way to cheat in university exams
– May 5th, 2017
Uni cheats: hundreds punished – April 19, 2014
Unis frown on note-sharing website as bright students cash in –
August 12, 2014
Police change exams after training detectives share 'cheat sheets‘ –
July 31, 2017
“It’s conceivable that
someone could pay an
extra $1,000 a class—
to simply hire someone
to earn their degree.”
Newton, D. (2015). Cheating in Online Classes is
Now Big Business. The Atlantic.
Essay mills“The results were “alarming”, with the quality of purchased essays being “higher than expected”, says Dr Lines, who warned that the use
of essay mills might therefore be “much, much higher” than previously thought.
Posing as a student, she had purchased 13 undergraduate history essays, required to be 2,000 words long and on the subject of the end of
the Second World War, at a cost of between A$166 and A$529 (£95 and £304).
Of these, only two received failing grades of 49 per cent or lower. Seven received a pass (between 50 and 64 per cent), three got a credit
(65 to 74 per cent), and one was judged worthy of a high distinction (above 85 per cent).
She also purchased 13 master’s essays, with the same word limit and a similar title but requiring more sources to be cited, at a cost of
between A$181 and A$713.
Of these, six received fails, often because they appeared to misunderstand the question. But three got passes, two a credit, one a
distinction (75 to 84 per cent) and one a high distinction.
Significantly, Turnitin plagiarism detection software did not perceive anything wrong in any of the undergraduate essays, and
identified the copying of material in only three of the postgraduate papers.” SafeAssign?
Havergal, C. (2016). Essay mills turning out high-quality essays undetected. Times Higher
• Fabris, C. (2015). Another Use for
Yik Yak on Campus? Cheating on
Exams. Chronicle of Higher
• Other Social
How far do we go?
• Waddell, K. (2016). Iraq Shut Down
Its Internet to Prevent Sixth-
graders From Cheating. The Atlantic
(should/could) we do?
• A policy of student education around
assessment /academic integrity and quality
• Transparency not just in assessment and then
process, but the expectations of the students
role in the process
• The Academic integrity of the institution
needs to be reinforced throughout the
Learners need to know
Why they are being assessed
How they are going to be assessed
What the ‘rules’ are (both sides, feedback)
The value of the assessment (where it leads)
Creating a Climate and Culture of Cheating
•“A 2010 survey of Yale undergraduates by The Yale
Daily News showed that most had never read the
school’s policy on academic honesty, and most
were unsure of the rules on sharing or recycling
Perez-Pena, R. (2012). Studies Find More Students Cheating, With High Achievers No Exception. New York Times
•Harvard students take pledge not to cheat
Cougjlan, S. (2015). BBC News - http://www.bbc.com/news/business-34769435
Online Honour Code - “Online faculty should give consideration to
incorporating an online honour code in their syllabus if their
institution has not already adopted an honour code.”
Update Course Guides - ”… recommend that additional information be
included in the syllabus to clarify faculty expectations of academic
behaviour. As students indicated that the use of textbooks, notes, or
other sources is not seen as cheating behaviour, a statement
clarifying those perceptions as being correct or incorrect is needed
in the course syllabus.”
Jones, I. S., Blankenship, D., & Hollier, G. (2013). Am I Cheating? An Analysis of Online Students' Perceptions
of Their Behaviors and Attitudes. Psychology Research, 3(5), 261.
Face to Face vs Online Cheating
Spaulding(2009) - No significant differences between students perception of their own
integrity in the face to face delivery as opposed to the online delivery of subjects
• There is a difference in perception that the online delivery system provides more
opportunity to cheat
Jones et al (2013) - Students self-reported that they did use notes or a book in an online
exam but did not agree that it was really cheating. They had varying definitions for what was
determined as cheating, some of the answers were classified as
• Substitution of materials - 22.4%
• Collusion - 9.4%
• Copying - 11%
• Other definitions that were not say to categorise
Spaulding, M. (2009). Perceptions of academic honesty in online vs. face-to-face classrooms. Journal of interactive online
learning, 8(3), 183-198.
Jones, I. S., Blankenship, D., & Hollier, G. (2013). Am I Cheating? An Analysis of Online Students' Perceptions of Their Behaviors
and Attitudes. Psychology Research, 3(5), 261.
Cooperation and Collaboration Vs. Cheating?
Boom in websites selling lecture notes to
Alastair Weng shares the secrets to his
academic success for less than the price of a
The University of Melbourne science student is
among thousands of young Australians who are
converting their lecture and textbook notes into
CAPS continual assessment
of practical skills
• Use devices the
students have to collect
evidence of their
• Metadata provides
some level of
“The proliferation of learning management systems
suggests that no one system is sufficiently feature rich,
or adequately flexible and extensible enough to meet
everyone’s needs or even most institutions’
Ira Fuchs, vice president for research in information technology,
Mellon Foundation. (2004, July) Learning Management Systems: Are
we there yet?
The full article is here, it’s a but dated but Ira has always
been a visionary.
A few simple things we can do to
‘confound’ low level cheating
• If using the MCQs in Blackboard
remember that you can allow the
random ordering of answer, but only if
it is turned on in the Test Question
• In creating questions you can then
specify that the answer be shown in a
A few simple things we can do to
‘confound’ low level cheating
• Then, when you reply the
test/quiz you can also set the
option to randomise the
questions in their
presentation to the students
• Change the tone of the conversation
• A similarity checker rather than a plagiarism detector
• Use it for education in the first year, moving to more punitive
in advancing years of study, when students should know better
• Ferguson, R. (2016). Online program Turnitin causing anxiety
among students: report. The Australian - Higher Education.
Can Turnitin really help
students to improve
• Keyboard recognition
• Web Camera observations
• Identification of student
• Still not there…yet?
• Schaffhauser, D. (2016). How Students Try To
Bamboozle Online Proctors. Campus
What’s the answer?
• “Over-reliance on technological
methods in the fight against
student cheating risks a
technological arms race.”
• “The only reason I imagine
students stop cheating is
because they’re being trusted.”
• Barthel, M. (2016). How to Stop Cheating in College. The Atlantic.
Discussions around the nature of
• Should we look at assessment from
• Case studies, authentic assessment,
challenge/problem based learning
• Students cheat for good grades. Why not make the classroom about
learning and not testing? http://qz.com/409195/students-cheat-for-
• 10 million + Video lectures watched
• 2 million + Assessments submitted
• 1.2 million + learners enrolled in five years
• Over 100,000 members in the course’s Facebook groups
• Learners from over 190+ countries
• Average course rating of 4.63 out of 5
• 36% of learners are from emerging economies
- By the numbers
• In-video quizzes, great formative
• Interactive video - https://corp.hapyak.com/
What’s on the
The UniMelb Academic
Integrity website is being
redeveloped and offers an
Academic Integrity Module
(AIM) in the LMS -
Students are given access to selected
article/s before the ‘lecture’ and can
edit/comment beside the relevant text.
Perusall prepares a report for lecturers
before the class highlighting the ‘hot topics’
that students have identified. Lecturers can
view the students
questions/comments/discussion and then
use them as prompts in the class.
A better/new way to teach/access?
Some key benefits
• better dialogue and communication that can overcome distance and
• immediate and learner-led assessment through interactive online
tests and tools in the hand (such as voting devices and internet
connected mobile phones)
• authenticity through online simulations and video technologies and
risk-free rehearsal of real-world skills in professional and vocational
• fast and easy processing and transferring of data
• improved thinking and ownership through peer assessment,
collection of evidence and reflection on achievements in e-portfolios
• making visible skills and learning processes that were previously
difficult to measure
• a personal quality to feedback, even in large-group contexts
• JISC’s guide (2010) and podcast https://www.jisc.ac.uk/podcasts/podcastpress-
• Identify some low hanging fruit in
automated assessment that you can
quickly implement to give students
timely and meaningful feedback
• Reflect on the nature of assessment
in your context; is there anything you
can modify to improve the
assessment process for your
students, you and your institution
• Slide 1 – Male student sitting at desk from www.Pixabay.com and are released under Creative Commons CC0, which makes them safe to use without asking
for permission or giving credit to the artist - even for commercial purposes.
• Slide 3 – Exam at the University of Melbourne from UniMelb’s Learning Environments Photo collection. http://imagelibrary.le.unimelb.edu.au/digital/index.jsp
• Slide 4 - HOTEL pedagogy map - Creative Commons - http://hotel-project.eu/content/learning-theories-map-richard-millwood
• Slide 5 - Ascilite blog post - http://blog.ascilite.org/considering-indigenous-pedagogy/ - Teaching adults posters, Ako Aotearoa -
• Slide 6 - Yunkaporta, Tyson (2009) Aboriginal pedagogies at the cultural interface. Professional Doctorate (Research) thesis, James Cook University. - Ascilite
blog post - http://blog.ascilite.org/considering-indigenous-pedagogy/
• Slide 7 – Assessment Curation site - https://www.scoop.it/t/rubrics-assessment-and-eproctoring-in-higher-education
• Slide 8 – Te Wero warrior from Pixabay – (see slide 1 for creative commons info).
• Slide 9 – Students studying screenshot from http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11943854
• Slide 12 – Student studying screenshot from https://www.facultyfocus.com/articles/online-education/assessment-strategies-students-
• Slide 13 – Forrest Gump movie poster from Paramount Pictures – Book cover from Amazon.com
• Slide 14 – Chocolate box menu by https://www.trenancechocolate.co.uk/
• Slide 15 - Motion Background from Pixabay.com (see slide 1 for creative commons info).
• Slide 17 – See slide 3.
• Slide 18 – Watch images from article URLs beside each image.
• Slide 19 - Police hat - http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=11895973
• Slide 20 – Screenshot from https://www.noneedtostudy.com/myclass/
• Slide 21 – Screenshot from URL on page.
• Slide 23 - Screenshot from URL on page.
• Slide 24 – Contract Cheating document from www.teqsa.gov.au
• Slide 25 - Screenshot from URL on page.
• Slide 26 - Screenshot from URL on page.
• Slide 27 – Screenshot of NZQA twitter feed and Ako poster from Ako Aotearoa website.
• Slide 31 - Screenshot from URL on page.
• Slide 32 – Both images from Pixabay.com (see slide 1 for creative commons info).
• Slide 33 – LMS vendor logos from their respective websites.
• Slide 34 and 35 – BB logo and screenshots from quiz module in BB LMS.
• Slide 36 - Screenshot from URL on page.
• Slide 37 – Turnitin logo from company website, and photo from UniMelb’s Learning Environments Photo collection.
• Slide 38 and 39 - Turnitin logo from company website, and Screenshot from URL on page.
• Slide 40 - Screenshot from URL on page.
• Slide 41 and 42 – Personal photos taken with permission from MSU Springfield.
• Slide 43 - Image from Pixabay.com (see slide 1 for creative commons info).
• Slide 44 - Screenshot from URL on page.
• Slide 45 - Photo from UniMelb’s Learning Environments Photo collection. http://imagelibrary.le.unimelb.edu.au/digital/index.jsp
• Slide 46 – All images from Pixabay.com (see slide 1 for creative commons info).
• Slide 47 and 48 – Screenshots from UniMelb MOOC lectures and video lecture quizs.
• Slide 49 – Article from The Australian online – April 16, 2016.
• Slide 50 – Screenshots from the UniMelb Academic Integrity website.
• Slide 51 – Cadmus: Not on my campus - https://umsu.unimelb.edu.au/cadmus-what-is-it/ - Is Cadmus watching you -
• Slide 52 - Photo from UniMelb’s Learning Environments Photo collection. http://imagelibrary.le.unimelb.edu.au/digital/index.jsp - Screenshot of
Paul Denny from his UniMelb Website.
• Slide 53, 54 and 55 – Screenshot from URL on page.
• Slide 56 – PeerMark screenshot from Turnitin in the UniMelb LMS.
• Slide 57 - Perusall screenshot for website - http://perusall.com/ First slide of Raoul’s Thresa’s presentation from http://melbourne-
• Slide 58 - Screenshot from URL on page.
• Slide 59 – Te Wero Warrior Image from Pixabay.com (see slide 1 for creative commons info).
• Slide 60 - Photo from UniMelb’s Learning Environments Photo collection. http://imagelibrary.le.unimelb.edu.au/digital/index.jsp
EdTech in Assessment: Sinner or
Innovations and Trends in Assessment
Wednesday 29th November 1.55pm-2.40pm
Slides available on Slideshare at: