Using technology to provide feedback on learning strategies

3,143 views

Published on

Feedback is problematic, and in higher education the situation is degrading quickly due to pressures to cater to massive cohorts while maintaining effectiveness. Technology can help achieve scalable feedback, and we show a few examples illustrating the process.

Published in: Education

Using technology to provide feedback on learning strategies

  1. 1. EksoBionicsflickr.com Using technology to provide feedback on learning strategies Abelardo Pardo (@abelardopardo) Faculty of Engineering and IT slideshare.net/abelardo_pardo Digital Learning Week
 University of South Australia
 16 March 2016
  2. 2. Abelardo Pardo Using technology to provide feedback on learning strategies 2 MichaelCoghlanflickr.com About
 Feedback Feedback
 Higher Ed Tech to scale
 feedback Examples
  3. 3. Abelardo Pardo Using technology to provide feedback on learning strategies 3 MichaelCoghlanflickr.com About
 Feedback
  4. 4. Abelardo Pardo Using technology to provide feedback on learning strategies "information provided by an agent (teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding” 4 Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi: 10.3102/003465430298487 FarukAteşflickr.com
  5. 5. Abelardo Pardo Using technology to provide feedback on learning strategies Innovations designed to strengthen the frequent feedback that students receive about their learning yield substantial learning gains 5 Black, P., & Wiliam, D. (1998). Assessment and Classroom Learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. doi:10.1080/0969595980050102 GarethChristopherflickr.com
  6. 6. Abelardo Pardo Using technology to provide feedback on learning strategies If you could choose one… • More than 500 meta-analyses of student achievements • 100 factors with potential influence • Feedback in top five • (74 meta-analyses) Most effective form: video, audio, computer-assisted instructional feedback, and/or related goals 6 Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand
  7. 7. Abelardo Pardo Using technology to provide feedback on learning strategies 7 Feedback is inherent in and a prime determiner of processes that constitute self regulated learning Galflickr.com Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3), 245-281
  8. 8. Abelardo Pardo Using technology to provide feedback on learning strategies Feedback models 1. Task Level (understanding, performance) 2. Process Level (what to do to understand, perform) 3. Self-regulation level (detecting and directing effort) 4. Self level (personal evaluation and affect) 8 Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi: 10.3102/003465430298487 Orkomedixflickr.com
  9. 9. Abelardo Pardo Using technology to provide feedback on learning strategies What is the recipe? 1. encourage contact between student and faculty 2. develops reciprocity and cooperation among students 3. uses active learning techniques 4. gives prompt feedback 5. emphasizes time on task 6. communicates high expectations 7. respects diverse talents and ways of learning 9 Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. American Association for Higher Education Bulletin, 39, 3-7. 
 Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as lever. American Association for Higher Education Bulletin, 49, 3-6.
  10. 10. Abelardo Pardo Using technology to provide feedback on learning strategies 10 Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge. Perceived as an administrative chore 
 instead of a pedagogical necessity MarcinWicharyflickr.com
  11. 11. Abelardo Pardo Using technology to provide feedback on learning strategies 11 Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2). Feedback has a Cinderella status Bitslammerflickr.com
  12. 12. Abelardo Pardo Using technology to provide feedback on learning strategies NicolasNovaflickr.com Feedback is inherently problematic 12 Higgins, R., Hartley, P., & Skelton, A. (2010). Getting the Message Across: The problem of communicating assessment feedback. Teaching in Higher Education, 6(2), 269-274. doi:10.1080/13562510120045230 Feedback as communication
  13. 13. Abelardo Pardo Using technology to provide feedback on learning strategies 1) Construct an understanding 2) Use it to regulate 13 Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199-218. doi:10.1080/03075070600572090 JamesButlerflickr.com
  14. 14. Abelardo Pardo Using technology to provide feedback on learning strategies 14 MichaelCoghlanflickr.com About
 Feedback Feedback
 Higher Ed
  15. 15. Abelardo Pardo Using technology to provide feedback on learning strategies 15 JoelSageflickr.com “sustainability of feedback is under threat… pervasive student concerns about the provision of feedback in an era of mass education" Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.
  16. 16. Abelardo Pardo Using technology to provide feedback on learning strategies 16 AllenLaiflickr.com Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge. Downward spiral: Student disenchantment mounts when feedback is uninformative and 
 as a consequence, less feedback is provided Solution: provide high-value feedback, rethink the role of student, enhance relation between guidance and feedback
  17. 17. Abelardo Pardo Using technology to provide feedback on learning strategies Feedback is delicate. It must be positive. High fragility of confidence felt by many students 17 Ferguson, P. (2011). Student perceptions of quality feedback in teacher education. Assessment & Evaluation in Higher Education, 36(1), 51-62. doi:10.1080/02602930903197883 Doranflickr.com
  18. 18. Abelardo Pardo Using technology to provide feedback on learning strategies The feedback question gets systematically lower values in student surveys 18 Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a decade of National Studies. University of Melbourne: Centre for the Study of Higher Education. Eleafflickr.com
  19. 19. Abelardo Pardo Using technology to provide feedback on learning strategies • Ambiguity or opacity (What do you mean?) • Negativity 19 Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education, 32(6), 601-611. doi:10.1080/02602930601116946 • Lateness • Uncertainty about criteria and contexts Loonyhikerflickr.com
  20. 20. Abelardo Pardo Using technology to provide feedback on learning strategies 20 PostmodernSleazeflickr.com Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: the challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698-712. doi:10.1080/02602938.2012.691462 Conventional feedback: Mark 1 Impractical due to resources Reserved for important concepts only Could be extended by technology if
 careful selection is applied Mark 2 From mechanistic to responsive Curriculum-wide view Foster self-regulation, feedback
 seeking skills, calibrated 
 judgement, etc.
  21. 21. Abelardo Pardo Using technology to provide feedback on learning strategies 21 Active Learning Works
  22. 22. Abelardo Pardo Using technology to provide feedback on learning strategies 22 2 Hours 2 Hours 3 Hours Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema, Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
  23. 23. Abelardo Pardo Using technology to provide feedback on learning strategies 23 Students are less likely to engage in pre-class activities if they are not interactive, do not provide formative feedback, and not coherently linked with the face-to-face activities O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher Education, 25, 85-95. doi:10.1016/j.iheduc.2015.02.002 DanKlimkeflickr.com
  24. 24. Abelardo Pardo Using technology to provide feedback on learning strategies 24 MichaelCoghlanflickr.com About
 Feedback Feedback
 Higher Ed Tech to scale
 feedback
  25. 25. Abelardo Pardo Using technology to provide feedback on learning strategies Instructive or process feedback is not common in learning analytics 25 Gasevic, D., Dawson, S., & Siemens, G. (2015). Let's not forget: Learning analytics are about learning. TechTrends, 59(1), 64-75. ChefCookeflickr.com
  26. 26. Abelardo Pardo Using technology to provide feedback on learning strategies 26 HungarianSnowflickr.com Intelligence Amplification Hierarchies of regulators to achieve amplification We measure intelligence as the capacity to make the appropriate selection Ashby, W. R. (1957). An Introduction to Cybernetics. London: Chapman & Hall Ltd.
  27. 27. Abelardo Pardo Using technology to provide feedback on learning strategies 27 Baker, R. S. (2016). Stupid Tutoring Systems, Intelligent Humans. International Journal of Artificial Intelligence in Education. doi: 10.1007/s40593-016-0105-0 Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UK. Journal of Interactive Media in Education, 2016(1). doi:10.5334/jime.394 Change in focus, from artificial intelligence to
 intelligence amplification Tutors are informed about students struggling but it is up to them to decide if/when to act
  28. 28. Abelardo Pardo Using technology to provide feedback on learning strategies 28 BrandonMartin-Andersonflickr.com • Can data guide the provision of effective feedback • Can this provision be done at scale?
  29. 29. Abelardo Pardo Using technology to provide feedback on learning strategies 29 JoshRepennekerFlickr.com Can technology and data be used to change the economic model of feedback, achieve scale and maintain effectiveness?
  30. 30. Abelardo Pardo Using technology to provide feedback on learning strategies Simplistic View Emerging “We are really shifting the role of an educator to someone who is more of a data- enabled detective” Max Ventilla 
 AltSchool founder
 (emphasis added) 30 Mead, Rebecca (2016). Learn Different. Silicon Valley disrupts education. The New Yorker, Annals of Technology, March 7 Issue
  31. 31. Abelardo Pardo Using technology to provide feedback on learning strategies 31 MichaelCoghlanflickr.com About
 Feedback Feedback
 Higher Ed Tech to scale
 feedback Examples
  32. 32. Abelardo Pardo Using technology to provide feedback on learning strategies 32 • First year engineering course • Computer Systems • Flipped Blended Learning • ~300 students • Preparation activities with formative feedback and
 summative assessment • 42% satisfied with feedback (meh)
  33. 33. Abelardo Pardo Using technology to provide feedback on learning strategies 33 2 Hours 2 Hours 3 Hours Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema, Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer. 2014 Edition
  34. 34. Abelardo Pardo Using technology to provide feedback on learning strategies 34
  35. 35. Abelardo Pardo Using technology to provide feedback on learning strategies 35 1. realtime feedback 2. convey engagement 3. compare with rest of cohort 4. reset weekly 5. one click away from notes Approach 1 (2014) SeanDreilingerflickr.com
  36. 36. Abelardo Pardo Using technology to provide feedback on learning strategies 36 Tracking
  37. 37. Abelardo Pardo Using technology to provide feedback on learning strategies 37 Technology Technology Week N
  38. 38. Abelardo Pardo Using technology to provide feedback on learning strategies 38 Weekly real time feedback
  39. 39. Abelardo Pardo Using technology to provide feedback on learning strategies 39 No statistically significant difference in the rating of feedback (2013 edition, M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17
  40. 40. Abelardo Pardo Using technology to provide feedback on learning strategies 40 1. encourage contact between student and faculty (forum) 2. uses active learning techniques 3. gives prompt feedback 4. emphasizes time on task 5. kind, specific, helpful Approach 2 (2015)Ben Manson flickr.com BenMansonflickr.com
  41. 41. Abelardo Pardo Using technology to provide feedback on learning strategies 41 Week N Technology Human
  42. 42. Abelardo Pardo Using technology to provide feedback on learning strategies 42 Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge, Edinburgh. 1. Group students 2. Each group has an estimated midterm score 3. Groups are determined by conditions in engagement measures 4. Can I act on this information? Student Clustering based on Engagement
  43. 43. Abelardo Pardo Using technology to provide feedback on learning strategies 43 You should take a more careful look at how symbols are encoded in the video. Would you be able to encode/ decode UAL symbols without looking at the video? Good initial work. However, did you understand the trick to handle encoding with a variable number of bits? Would you be able to provide an example? Good work. Would you be able to come up with your own machine language and your encoding scheme? Remember that it has to be unambiguous. Thorough work with the task about machine language encoding. Give it a quick review before the midterm. Q1 Q2 Q3 Q4 Instructor
  44. 44. Abelardo Pardo Using technology to provide feedback on learning strategies 44 Algorithm
  45. 45. Abelardo Pardo Using technology to provide feedback on learning strategies 45 Automatic Email
  46. 46. Abelardo Pardo Using technology to provide feedback on learning strategies 46 1. Data collected weekly 2. Email sent at end of week 3. 4 weeks before the midterm nateOneflickr.com
  47. 47. Abelardo Pardo Using technology to provide feedback on learning strategies 47 Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03; 2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6
 Effect size (Cohen’s d) = 0.49. Medium positive effect
  48. 48. Abelardo Pardo Using technology to provide feedback on learning strategies 48 Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79; 2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002
 Effect size (Cohen’s d) = 0.21. Small positive effect
  49. 49. Abelardo Pardo Using technology to provide feedback on learning strategies 49 MatthewandHeaderflickr.com Feedback: high impact/risk Higher Ed is in dire need for feedback revamp Hybrid methods: human intelligence amplification Data supported approaches allow effective scaling
  50. 50. EksoBionicsflickr.com Using technology to provide feedback on learning strategies Abelardo Pardo (@abelardopardo) Faculty of Engineering and IT slideshare.net/abelardo_pardo Digital Learning Week
 University of South Australia
 16 March 2016

×