Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Assessment Conference Dunedin 2017 -


Published on

Slide deck for my plenary session at the Assessing Learning conference, Wednesday 29th November, 2017 at Otago Polytechnic.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Assessment Conference Dunedin 2017 -

  1. 1. EdTech in Assessment: Sinner or saviour? Innovations and Trends in Assessment stream Peter Mellow Wednesday 29th November 1.55pm-2.40pm Slides available on Slideshare at:
  2. 2. • Senior Lecturer/Flexible Learning Advisor, AUT 1987-2011 • FLLinNZ 2004 - Ako Academy 2007 • Lecturer/eLearning Advisor, Curtin Uni 2011-2012 • Learning Design Coordinator, UniMelb 2012-2017 • Director, Learning Design and Systems, Learning and Teaching DirectorateEastern Health 2017 --->
  3. 3. Abstract Effective assessment has always been a challenge in education, but appears to be even more so with the rapid advances being made with new technologies. However, technology is just one disruptor as we start to question and reflect on the nature of assessment itself. This session with explore some of the challenges we face in education around assessment and technology, which might be our saviour as it has been our bane. Perhaps it is time for a paradigm shift in assessment to move into the future with confidence?
  4. 4. theories-map-richard-millwood Learning Theory Map, available as a PDF download or an interactive web page
  5. 5. Indigenous pedagogy Ascilite blog post – 31st July 2017 AUT is now working on “implementing mātauranga Māori into our curriculum.” Robert Hogg, AUT lecturer mātauranga Māori literally means Māori knowledge
  6. 6. Tyson Yunkaporta’s PhD identified eight key strands of indigenous pedagogy 1. Story sharing 2. Non-linear 3. Community links 4. Deconstruct – reconstruct 5. Symbols and images 6. Non-verbal 7. Learning map 8. Land links Indigenous pedagogy This map attempts to represent the language, social or nation groups of Aboriginal Australia. It shows only the general locations of larger groupings of people which may include clans, dialects or individual languages in a group. It used published resources from 1988-1994 and is not intended to be exact, nor the boundaries fixed. It is not suitable for native title or other land claims. David R Horton (creator), © AIATSIS, 1996. No reproduction without permission. To purchase a print version visit: Used with permission.
  7. 7. Curation • /t/rubrics- assessment-and- eproctoring-in- higher-education
  8. 8. Todays Menu 1. Start with the basics, 21st century pedagogy, LOs, reliability, validity and student voice 2. Tried and true challenges, F2F, online 3. Some ideas for solutions to these (and other) challenges – practice, tech 4. On the horizon? 5. Te Wero – What are you going to do?
  9. 9. Assessment in a 21st century pedagogy • Timely and meaningful feedback • Relevant tasks • Self and peer assessment • Clear, transparent goals and objectives TeachThought - A Diagram Of 21st Century Pedagogy - learning/a-diagram-of-21st-century-pedagogy/ “Good assessment can be a learning experience” NZ Herald – 15th November 2017
  10. 10. Learning Objectives – “the sinister 16” verbs that are are passive, internal and/or otherwise unobservable • Understand • Appreciate • Comprehend • Grasp • Know • See • Accept • Have a knowledge of • Be aware of • Be conscious of • Learn • Perceive • Value • Get • Apprehend • Be familiar with Potter, M.K., and Kustra, E. (2012). Course design for Constructive Alignment. – A primer on learning outcomes and the SOLO Taxonomy. Centre for Teaching and Learning, University of Windsor.
  11. 11. Reliability and Validity Reliability – will the instrument produce consistent results between different assessments? Validity – are we measuring what we say we are measuring? Is the assessment valid? “Professor Millet’s study looked at data from more than 50,000 course sections at a North American university over several years. He found that variability did not occur only within particular departments, but also among individual faculty members.” Millet, I. (2016). The relationship between grading leniency and grading reliability
  12. 12. Lowest: Quizzes Traditional written papers (2-8 pages) Group projects Middle: Audio Recordings (student created) Open discussion (discussion board) Paired discussion (2-5 students) Highest: Response to video Twitter summaries (distillation) Screencasts (student created) Field experiences Interviews Work samples Student Voice Bailey, S., Hendricks, S., Applewhite, S. (2015). Student Perspectives of Assessment Strategies in Online Courses, Journal of Interactive Online Learning, v.13, n. 3, 112-25.
  13. 13. ASSESSMENT Assessment is like a box of Chocolates – Chapter 1 by Lisa Maxfield, California State University, Long Beach, In “Coming to Terms with Student Outcomes Assessment: Faculty and Administrators” edited by Peggy Maki (2010).
  14. 14. Adds transparency Avoids surprises Adds a layer of safety (physical and emotional)
  15. 15. Formative assessment is more than a  or a  It provides: • Feedback • An opportunity to Fail • Fun? Gamification?
  16. 16. eTools for assessment – many already in LMS (to some degree) Organising assessments Grade centre Deploy assessments (peer matching) MCQ Automate feedback Grade/Analyse/QA/Authenticate assessments AES – automated essay scoring Blockchain - authentication / badging / certification AI - Artificial intelligence Badges/gamification Learning Analytics/Assessment analyticsCheating on assessments
  17. 17. F2F: the old ways are still being used! “…writing cheat notes on their bodies and calculators, using information stored on smartphones, and copying the work of those sitting next to them.” Dough, P. (2016). Universities bust more than 80 exam cheats. NZ Herald “Illicit notes and copying others remain common methods adopted by cheaters, alongside more modern trends such as using smartphones.” “Universities found students hiding notes in the toilets and taking a toilet break to check them, sneaking notes into the exam hall, or whispering to others during the exam. Incidents of plagiarism and passing notes were also uncovered.”
  18. 18. F2F: However, there are ‘new’ technologies Thai university students caught using spy cameras, linked to smartwatches to cheat on medicine exam. Arthit Ourairat, the rector of Rangsit University, posted pictures of the hi-tech cheating equipment on his Facebook page, announcing that the entrance exam in question had been cancelled after the plot was discovered. on-exam/7399728 Smartwatches that allow pupils to 'cheat' in exams for sale on Amazon. Deputy head says such devices are making exams a 'nightmare to administer’ cheat-in-exams-for-sale-on-amazon-a6910666.html Good luck bringing your watch into your exam - Massey University has banned watches from examination rooms. Apple Watch banned from HSC exams and schools - 20150820-gj3px2.html
  19. 19. Australia New ZealandStudents at Sydney University use impersonators to sit their exams - August 11th 2015 impersonators-to-sit-their-exams-20150810-givhs0.html Sydney University medical students invented patients for assignments - June 6th 2015 patients-for-assignments-20150604-ghgsy2 Macquarie University revokes degrees for students caught buying essays in MyMaster cheating racket - May 28th 2015 students-caught-buying-essays-in-mymaster-cheating-racket-20150527-ghba3z Sydney University, University of NSW and UTS crack down on cheating students - April 25th 2016 20160421-gobq1m.html Deakin University students kicked out for 'contract cheating’ - May 18th 2016 kicked-out-for-contract-cheating-20160517-goxm1y.html University of Sydney's medical school in second cheating controversy May 26th 2016 - school-in-second-cheating-controversy-20160525-gp3g3h.html Student comes up with 'ingenious' way to cheat in university exams – May 5th, 2017 Uni cheats: hundreds punished – April 19, 2014 Unis frown on note-sharing website as bright students cash in – August 12, 2014 Police change exams after training detectives share 'cheat sheets‘ – July 31, 2017
  20. 20.
  21. 21. Online “It’s conceivable that someone could pay an extra $1,000 a class— to simply hire someone to earn their degree.” Newton, D. (2015). Cheating in Online Classes is Now Big Business. The Atlantic. 11/cheating-through-online-courses/413770/
  22. 22. Essay mills“The results were “alarming”, with the quality of purchased essays being “higher than expected”, says Dr Lines, who warned that the use of essay mills might therefore be “much, much higher” than previously thought. Posing as a student, she had purchased 13 undergraduate history essays, required to be 2,000 words long and on the subject of the end of the Second World War, at a cost of between A$166 and A$529 (£95 and £304). Of these, only two received failing grades of 49 per cent or lower. Seven received a pass (between 50 and 64 per cent), three got a credit (65 to 74 per cent), and one was judged worthy of a high distinction (above 85 per cent). She also purchased 13 master’s essays, with the same word limit and a similar title but requiring more sources to be cited, at a cost of between A$181 and A$713. Of these, six received fails, often because they appeared to misunderstand the question. But three got passes, two a credit, one a distinction (75 to 84 per cent) and one a high distinction. Significantly, Turnitin plagiarism detection software did not perceive anything wrong in any of the undergraduate essays, and identified the copying of material in only three of the postgraduate papers.” SafeAssign? Havergal, C. (2016). Essay mills turning out high-quality essays undetected. Times Higher Education. essays-undetected-study
  23. 23. Contract cheating – Tracey Bretag Workshop 1.55pm-3pm Thursday Plenary 3.30pm Thursday
  24. 24. Contract Cheating publications/good-practice-note- addressing-contract-cheating- safeguard-academic-integrity
  25. 25. Social Media • Fabris, C. (2015). Another Use for Yik Yak on Campus? Cheating on Exams. Chronicle of Higher Education - us/another-use-for-yik-yak-on- campus-cheating-on-exams/56543 • Other Social Media channels?
  26. 26. How far do we go? • Waddell, K. (2016). Iraq Shut Down Its Internet to Prevent Sixth- graders From Cheating. The Atlantic - /archive/2016/05/iraq-shut-down-its- internet-to-prevent-sixth-graders- from-cheating/482946/
  27. 27. What can (should/could) we do? • A policy of student education around assessment /academic integrity and quality control vigilance • Transparency not just in assessment and then process, but the expectations of the students role in the process • The Academic integrity of the institution needs to be reinforced throughout the educational journey Learners need to know Why they are being assessed How they are going to be assessed What the ‘rules’ are (both sides, feedback) The value of the assessment (where it leads)
  28. 28. Creating a Climate and Culture of Cheating •“A 2010 survey of Yale undergraduates by The Yale Daily News showed that most had never read the school’s policy on academic honesty, and most were unsure of the rules on sharing or recycling their work.” Perez-Pena, R. (2012). Studies Find More Students Cheating, With High Achievers No Exception. New York Times •Harvard students take pledge not to cheat Cougjlan, S. (2015). BBC News -
  29. 29. Additional Solutions Online Honour Code - “Online faculty should give consideration to incorporating an online honour code in their syllabus if their institution has not already adopted an honour code.” Update Course Guides - ”… recommend that additional information be included in the syllabus to clarify faculty expectations of academic behaviour. As students indicated that the use of textbooks, notes, or other sources is not seen as cheating behaviour, a statement clarifying those perceptions as being correct or incorrect is needed in the course syllabus.” Jones, I. S., Blankenship, D., & Hollier, G. (2013). Am I Cheating? An Analysis of Online Students' Perceptions of Their Behaviors and Attitudes. Psychology Research, 3(5), 261.
  30. 30. Face to Face vs Online Cheating Spaulding(2009) - No significant differences between students perception of their own integrity in the face to face delivery as opposed to the online delivery of subjects • There is a difference in perception that the online delivery system provides more opportunity to cheat Jones et al (2013) - Students self-reported that they did use notes or a book in an online exam but did not agree that it was really cheating. They had varying definitions for what was determined as cheating, some of the answers were classified as • Substitution of materials - 22.4% • Collusion - 9.4% • Copying - 11% • Other definitions that were not say to categorise Spaulding, M. (2009). Perceptions of academic honesty in online vs. face-to-face classrooms. Journal of interactive online learning, 8(3), 183-198. Jones, I. S., Blankenship, D., & Hollier, G. (2013). Am I Cheating? An Analysis of Online Students' Perceptions of Their Behaviors and Attitudes. Psychology Research, 3(5), 261.
  31. 31. Cooperation and Collaboration Vs. Cheating? Boom in websites selling lecture notes to university students Alastair Weng shares the secrets to his academic success for less than the price of a coffee. The University of Melbourne science student is among thousands of young Australians who are converting their lecture and textbook notes into cash. buying-university-notes-brings-in-big-bucks- 20160701-gpwgfm.html
  32. 32. CAPS continual assessment of practical skills • Use devices the students have to collect evidence of their achievements • Metadata provides some level of authentication Camera GPS Compass Barometer 3-axis gyro Accelerometer Proximity sensor Ambient light sensor
  33. 33. The LMS “The proliferation of learning management systems suggests that no one system is sufficiently feature rich, or adequately flexible and extensible enough to meet everyone’s needs or even most institutions’ requirements." Ira Fuchs, vice president for research in information technology, Mellon Foundation. (2004, July) Learning Management Systems: Are we there yet? The full article is here, it’s a but dated but Ira has always been a visionary. ng-management-systems-are-we-there-yet.aspx
  34. 34. A few simple things we can do to ‘confound’ low level cheating • If using the MCQs in Blackboard remember that you can allow the random ordering of answer, but only if it is turned on in the Test Question Settings • In creating questions you can then specify that the answer be shown in a random order
  35. 35. A few simple things we can do to ‘confound’ low level cheating • Then, when you reply the test/quiz you can also set the option to randomise the questions in their presentation to the students
  36. 36. /news/2017/11/14/professors- have-mixed-reactions- blackboard-plan-offer-tool- grading-online
  37. 37. Turnitin • Change the tone of the conversation • A similarity checker rather than a plagiarism detector • Use it for education in the first year, moving to more punitive in advancing years of study, when students should know better • Ferguson, R. (2016). Online program Turnitin causing anxiety among students: report. The Australian - Higher Education. turnitin-causing-anxiety-among-students-report/news- story/215211e1d2f54f1e1e1d481a43f68a63
  38. 38. •
  39. 39. Can Turnitin really help students to improve their writing? • ImproveWriting?Product=Turnitin&Notification_Languag e=English&Lead_Origin=Social%20-%20Media%20-%20- %20-%20Blog%20-%20-%20- %20Newsletter&source=Webcast%20-%20-%20- %20White%20-%20Paper
  40. 40. Revision Assistant • 7/07/06/plagiarism-powerhouse-turnitin- revises-the-writing-process.aspx
  41. 41. Proctoring The Proctoring Center @ MSU Springfield, MO 2017
  42. 42. Proctoring
  43. 43. Online Proctoring • Keyboard recognition • Web Camera observations • Identification of student • Still not there…yet? • Schaffhauser, D. (2016). How Students Try To Bamboozle Online Proctors. Campus Technology. 06/how-students-try-to-bamboozle-online- proctors.aspx
  44. 44. What’s the answer? • “Over-reliance on technological methods in the fight against student cheating risks a technological arms race.” • “The only reason I imagine students stop cheating is because they’re being trusted.” • Barthel, M. (2016). How to Stop Cheating in College. The Atlantic. cheating-in-college/479037/
  45. 45. Discussions around the nature of assessment • Should we look at assessment from another perspective? • Case studies, authentic assessment, challenge/problem based learning • Students cheat for good grades. Why not make the classroom about learning and not testing? good-grades-why-not-make-the-classroom-about-learning-and-not- testing/
  46. 46. Certificate/Diploma Degree Masters PhD Micro-credentials ‘NanoDegree’ Specialisation
  47. 47. • 10 million + Video lectures watched • 2 million + Assessments submitted • 1.2 million + learners enrolled in five years • Over 100,000 members in the course’s Facebook groups • Learners from over 190+ countries • Average course rating of 4.63 out of 5 • 36% of learners are from emerging economies Melbourne MOOCs - By the numbers
  48. 48. • In-video quizzes, great formative assessment • Interactive video - Melbourne MOOCs
  49. 49. What’s on the Horizon? The UniMelb Academic Integrity website is being redeveloped and offers an Academic Integrity Module (AIM) in the LMS - http://academicintegrity.unimel Cadmus
  50. 50. What the University says:
  51. 51. What the Students say:
  52. 52. The power of Peers
  53. 53.
  54. 54.
  55. 55.
  56. 56. Students are given access to selected article/s before the ‘lecture’ and can edit/comment beside the relevant text. Perusall prepares a report for lecturers before the class highlighting the ‘hot topics’ that students have identified. Lecturers can view the students questions/comments/discussion and then use them as prompts in the class. http://melbourne- 468/Do-old-habits-die-hard.pdf A better/new way to teach/access?
  57. 57. Some key benefits • better dialogue and communication that can overcome distance and time constraints • immediate and learner-led assessment through interactive online tests and tools in the hand (such as voting devices and internet connected mobile phones) • authenticity through online simulations and video technologies and risk-free rehearsal of real-world skills in professional and vocational education • fast and easy processing and transferring of data • improved thinking and ownership through peer assessment, collection of evidence and reflection on achievements in e-portfolios • making visible skills and learning processes that were previously difficult to measure • a personal quality to feedback, even in large-group contexts • JISC’s guide (2010) and podcast release-effective-assessment-in-a-digital-age-06-sep-2010
  58. 58. Te Wero • Identify some low hanging fruit in automated assessment that you can quickly implement to give students timely and meaningful feedback • Reflect on the nature of assessment in your context; is there anything you can modify to improve the assessment process for your students, you and your institution
  59. 59. Resources and references • UNSW (video)- assessment • Cornell’s CTE - ideas/assessing-student-learning/peer-assessment.html • Reading - self-assessment/peer-assessment/eia-peer- assessment.aspx • UTex - learning/feedback/peer-assessment • Academic honesty and plagiarism - UniMelb:
  60. 60. Images : • Slide 1 – Male student sitting at desk from and are released under Creative Commons CC0, which makes them safe to use without asking for permission or giving credit to the artist - even for commercial purposes. • Slide 3 – Exam at the University of Melbourne from UniMelb’s Learning Environments Photo collection. • Slide 4 - HOTEL pedagogy map - Creative Commons - • Slide 5 - Ascilite blog post - - Teaching adults posters, Ako Aotearoa - • Slide 6 - Yunkaporta, Tyson (2009) Aboriginal pedagogies at the cultural interface. Professional Doctorate (Research) thesis, James Cook University. - Ascilite blog post - • Slide 7 – Assessment Curation site - • Slide 8 – Te Wero warrior from Pixabay – (see slide 1 for creative commons info). • Slide 9 – Students studying screenshot from • Slide 12 – Student studying screenshot from prefer/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BrXoveSypRC2Mbhe3MIcogg%3D%3D • Slide 13 – Forrest Gump movie poster from Paramount Pictures – Book cover from • Slide 14 – Chocolate box menu by • Slide 15 - Motion Background from (see slide 1 for creative commons info). • Slide 17 – See slide 3. • Slide 18 – Watch images from article URLs beside each image. • Slide 19 - Police hat - • Slide 20 – Screenshot from • Slide 21 – Screenshot from URL on page. • Slide 23 - Screenshot from URL on page. • Slide 24 – Contract Cheating document from • Slide 25 - Screenshot from URL on page. • Slide 26 - Screenshot from URL on page. • Slide 27 – Screenshot of NZQA twitter feed and Ako poster from Ako Aotearoa website. • Slide 31 - Screenshot from URL on page.
  61. 61. Images : • Slide 32 – Both images from (see slide 1 for creative commons info). • Slide 33 – LMS vendor logos from their respective websites. • Slide 34 and 35 – BB logo and screenshots from quiz module in BB LMS. • Slide 36 - Screenshot from URL on page. • Slide 37 – Turnitin logo from company website, and photo from UniMelb’s Learning Environments Photo collection. • Slide 38 and 39 - Turnitin logo from company website, and Screenshot from URL on page. • Slide 40 - Screenshot from URL on page. • Slide 41 and 42 – Personal photos taken with permission from MSU Springfield. • Slide 43 - Image from (see slide 1 for creative commons info). • Slide 44 - Screenshot from URL on page. • Slide 45 - Photo from UniMelb’s Learning Environments Photo collection. • Slide 46 – All images from (see slide 1 for creative commons info). • Slide 47 and 48 – Screenshots from UniMelb MOOC lectures and video lecture quizs. • Slide 49 – Article from The Australian online – April 16, 2016. • Slide 50 – Screenshots from the UniMelb Academic Integrity website. • Slide 51 – Cadmus: Not on my campus - - Is Cadmus watching you - • Slide 52 - Photo from UniMelb’s Learning Environments Photo collection. - Screenshot of Paul Denny from his UniMelb Website. • Slide 53, 54 and 55 – Screenshot from URL on page. • Slide 56 – PeerMark screenshot from Turnitin in the UniMelb LMS. • Slide 57 - Perusall screenshot for website - First slide of Raoul’s Thresa’s presentation from http://melbourne- • Slide 58 - Screenshot from URL on page. • Slide 59 – Te Wero Warrior Image from (see slide 1 for creative commons info). • Slide 60 - Photo from UniMelb’s Learning Environments Photo collection.
  62. 62. EdTech in Assessment: Sinner or saviour? Innovations and Trends in Assessment Peter Mellow Wednesday 29th November 1.55pm-2.40pm Slides available on Slideshare at: