Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Feedback at scale with a little help of my algorithms

887 views

Published on

Talk exploring how to use data to provide scalable feedback in learning experiences. The solutions explored propose the use of algorithms to enhance how humans instructors provide feedback to students more effectively

Published in: Education
  • Be the first to comment

  • Be the first to like this

Feedback at scale with a little help of my algorithms

  1. 1. Abelardo Pardo (@abelardopardo)
 Faculty of Engineering and IT slideshare.net/abelardo_pardo Queen’sUniversityflickr.com Feedback at scale with a little help from my algorithms SusanaFernandezflickr.com Research Seminar Universitat Pompeu Fabra Barcelona Spain 5 May 2016
  2. 2. Feedback at scale with a little help from my algorithmsAbelardo Pardo 2 RishiSflickr.com About me Teaching First year engineering course Computer systems 350 students Plenary 2 hour lecture + 
 2 hour tutorial + 
 3 hour laboratory session Active learning Heavy use of technology Research Educational technology Learning analytics Data-guided feedback Naturalistic experiments
  3. 3. Feedback at scale with a little help from my algorithmsAbelardo Pardo 3 gingiberflickr.com Feedback and Data Decision Trees Data driven rubric Design and 
 production
  4. 4. Feedback at scale with a little help from my algorithmsAbelardo Pardo 4 gingiberflickr.com Feedback and Data
  5. 5. Feedback at scale with a little help from my algorithmsAbelardo Pardo 5 Ericflickr.com Systematically low ratings of the feedback provided to the students
  6. 6. Feedback at scale with a little help from my algorithmsAbelardo Pardo 6 Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a decade of National Studies. University of Melbourne: Centre for the Study of Higher Education. Eleafflickr.com The feedback question gets systematically lower values in student surveys
  7. 7. Feedback at scale with a little help from my algorithmsAbelardo Pardo 7 Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education, 32(6), 601-611. doi:10.1080/02602930601116946 • Lateness • Uncertainty about criteria and contexts Loonyhikerflickr.com • Ambiguity or opacity (What do you mean?) • Negativity Loonyhikerflickr.com
  8. 8. Feedback at scale with a little help from my algorithmsAbelardo Pardo 8 Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge. Perceived as an administrative chore 
 instead of a pedagogical necessity MarcinWicharyflickr.com
  9. 9. Feedback at scale with a little help from my algorithmsAbelardo Pardo 9 JoelSageflickr.com “sustainability of feedback is under threat… pervasive student concerns about the provision of feedback in an era of mass education” Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.
  10. 10. Feedback at scale with a little help from my algorithmsAbelardo Pardo 10 Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2). Feedback has a Cinderella status Bitslammerflickr.com
  11. 11. Feedback at scale with a little help from my algorithmsAbelardo Pardo 11 Active Learning Works
  12. 12. Feedback at scale with a little help from my algorithmsAbelardo Pardo 12 AllenLaiflickr.com Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge. Downward spiral: Student disenchantment mounts when feedback is uninformative and still,
 less feedback is provided Solution: provide high-value feedback, rethink the role of student, enhance relation between guidance and feedback
  13. 13. Feedback at scale with a little help from my algorithmsAbelardo Pardo 13 BrandonMartin-Andersonflickr.com • Can data guide the provision of effective feedback • Can this provision be done at scale?
  14. 14. Feedback at scale with a little help from my algorithmsAbelardo Pardo 14 DmitryGrigorievflickr.com Learning Analytics: measure, collect, analyse data about learners to understand and improve their learning and the environment in which it occurs
  15. 15. Feedback at scale with a little help from my algorithmsAbelardo Pardo 15 EustaquioSantimanoflickr.com Collect Report Analyze Act Refine
  16. 16. Feedback at scale with a little help from my algorithmsAbelardo Pardo 16
  17. 17. Feedback at scale with a little help from my algorithmsAbelardo Pardo 17 EustaquioSantimanoflickr.com Collect Report Analyze Act Refine
  18. 18. Feedback at scale with a little help from my algorithmsAbelardo Pardo 18 gingiberflickr.com Feedback and Data Decision Trees
  19. 19. Feedback at scale with a little help from my algorithmsAbelardo Pardo 19 WilliamMurphyflickr.com • 13 Week first year Engineering • Weekly activities (formative/summative) • Videos, MCQ, Exercises, dashboard • n = 272, Weeks 2-5 and 7-13 Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge, Edinburgh. doi:10.1145/2883851.2883870
  20. 20. Feedback at scale with a little help from my algorithmsAbelardo Pardo 20 OliverBraubachflickr.com Objective 1. Data indicators close to learning design 2. Predictive model 3. Bridge between model and application 4. Straightforward delivery method
  21. 21. Feedback at scale with a little help from my algorithmsAbelardo Pardo 21 LouishPixelflickr.com • Event counts from interactive course material • Midterm/final exam scores • Recursive partitioning • Divide cohort into performance categories
  22. 22. Feedback at scale with a little help from my algorithmsAbelardo Pardo 22 Data collected • Indicators are directly connected with learning design • Data structure shaped by the schedule (weeks) • Data available in a per-week basis • What is the expected midterm/final score in week n?
  23. 23. Feedback at scale with a little help from my algorithmsAbelardo Pardo 23 Result Example • Week 10 • Predicted score at leaves (out of 40) • Conditions at nodes • If (EXC.in >=22) and (VID.PL < 8.5) then score = 6
  24. 24. Feedback at scale with a little help from my algorithmsAbelardo Pardo 24 • Each leaf node represents a group of students with their estimated score. • Example: 6, 8.3, 8.4, 9.4, 9.9, 10, 15 (out of 40) • Intervention: suggested work before exam Result interpretation
  25. 25. Feedback at scale with a little help from my algorithmsAbelardo Pardo 25 shabnammayetFlickr.com RMSE: Root mean square error, MAE: Mean absolute error Performance
  26. 26. Feedback at scale with a little help from my algorithmsAbelardo Pardo 26 gingiberflickr.com Feedback and Data Decision Trees Data driven rubric
  27. 27. Feedback at scale with a little help from my algorithmsAbelardo Pardo 27 2 Hours 2 Hours 3 Hours Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema, L. Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer. 2014 Edition
  28. 28. Feedback at scale with a little help from my algorithmsAbelardo Pardo 28
  29. 29. Feedback at scale with a little help from my algorithmsAbelardo Pardo 29 1. realtime feedback 2. convey engagement 3. compare with rest of cohort 4. reset weekly 5. one click away from notes Approach 1 (2014) SeanDreilingerflickr.com
  30. 30. Feedback at scale with a little help from my algorithmsAbelardo Pardo 30 Tracking
  31. 31. Feedback at scale with a little help from my algorithmsAbelardo Pardo 31 Technology Technology Week N
  32. 32. Feedback at scale with a little help from my algorithmsAbelardo Pardo 32 Weekly real time feedback
  33. 33. Feedback at scale with a little help from my algorithmsAbelardo Pardo 33 No statistically significant difference in the rating of feedback (2013 edition, M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17
  34. 34. Feedback at scale with a little help from my algorithmsAbelardo Pardo 34 1. encourage contact between student and faculty (forum) 2. uses active learning techniques 3. gives prompt feedback 4. emphasizes time on task 5. kind, specific, helpful Approach 2 (2015)Ben Manson flickr.com BenMansonflickr.com
  35. 35. Feedback at scale with a little help from my algorithmsAbelardo Pardo 35 Week N Technology Human
  36. 36. Feedback at scale with a little help from my algorithmsAbelardo Pardo 36 You should take a more careful look at how symbols are encoded in the video. Would you be able to encode/ decode UAL symbols without looking at the video? Good initial work. However, did you understand the trick to handle encoding with a variable number of bits? Would you be able to provide an example? Good work. Would you be able to come up with your own machine language and your encoding scheme? Remember that it has to be unambiguous. Thorough work with the task about machine language encoding. Give it a quick review before the midterm. Q1 Q2 Q3 Q4 Instructor
  37. 37. Feedback at scale with a little help from my algorithmsAbelardo Pardo 37 Algorithm
  38. 38. Feedback at scale with a little help from my algorithmsAbelardo Pardo 38 Automatic Email
  39. 39. Feedback at scale with a little help from my algorithmsAbelardo Pardo 39 1. Data collected weekly 2. Email sent at end of week 3. 4 weeks before the midterm nateOneflickr.com
  40. 40. Feedback at scale with a little help from my algorithmsAbelardo Pardo 40 Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03; 2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6
 Effect size (Cohen’s d) = 0.49. Medium positive effect
  41. 41. Feedback at scale with a little help from my algorithmsAbelardo Pardo 41 Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79; 2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002
 Effect size (Cohen’s d) = 0.21. Small positive effect
  42. 42. Feedback at scale with a little help from my algorithmsAbelardo Pardo 42 gingiberflickr.com Feedback and Data Decision Trees Data driven rubric Design and 
 production
  43. 43. Feedback at scale with a little help from my algorithmsAbelardo Pardo 43 DepartmentforBusinessInnovation&Skillflickr.com • Content creation • Interactive • Multi-view (tutor, bilingual) • Focus on content (not style) • Platform agnostic • Design-embedded analytics
  44. 44. Feedback at scale with a little help from my algorithmsAbelardo Pardo 44 DavidMichalczukflickr.com • Markup language • Batch processing • Sphinx-doc • Webdav gateway for publishing • Self-contained web site • Extensible language: macros
 for design elements
  45. 45. Feedback at scale with a little help from my algorithmsAbelardo Pardo 45
  46. 46. Feedback at scale with a little help from my algorithmsAbelardo Pardo 46
  47. 47. Feedback at scale with a little help from my algorithmsAbelardo Pardo 47 Single source approach •Plain text •Version control •Distributed production •Suitable for ed designers
  48. 48. Feedback at scale with a little help from my algorithmsAbelardo Pardo 48 Back annotation
  49. 49. Feedback at scale with a little help from my algorithmsAbelardo Pardo 49 “The interactive quizzes during class are excellent” 
 “I love the quizzes” ‘Learning was supported by useful electronic learning resources’ Agreement increased from 78% to 98% Professional Practice of Radiography (PG). A/Prof. Mark McEntee
  50. 50. Feedback at scale with a little help from my algorithmsAbelardo Pardo 50 https://bitbucket.org/abelardopardo/reauthoring •Activity duration •MCQ •Video embedding •Difficulty/Usefulness 2D grid •Tutor view •Tutor feedback •Coupled with tracking • Automatic back-annotation
  51. 51. Feedback at scale with a little help from my algorithmsAbelardo Pardo 51 FabienCAMBIflickr.com • Feedback is delicate and messy • Data can help scaling • Good focal point for analytics • Provide intuitive tools to 
 instructors • Authoring is a challenge
  52. 52. Abelardo Pardo (@abelardopardo)
 Faculty of Engineering and IT slideshare.net/abelardo_pardo Queen’sUniversityflickr.com Feedback at scale with a little help from my algorithms SusanaFernandezflickr.com Research Seminar Universitat Pompeu Fabra Barcelona Spain 5 May 2016

×