Recommended
More Related Content
What's hot
What's hot (20)
Similar to Personalized Remedial Recommendations for SQL Programming Practice System (Jordan Barria-Pineda in APCSE20 at UMAP20)
Similar to Personalized Remedial Recommendations for SQL Programming Practice System (Jordan Barria-Pineda in APCSE20 at UMAP20) (20)
Recently uploaded
Recently uploaded (20)
Personalized Remedial Recommendations for SQL Programming Practice System (Jordan Barria-Pineda in APCSE20 at UMAP20)
- 1. Personalized Remedial Recommendations for SQL Programming Practice System Jordan Barría-Pineda ( @j_barria), Kamil Akhuseyinoglu ( @cskamil1), Kerttu Pollari-Malmi, Teemu Sirkiä, Lauri Malmi, Peter Brusilovsky Adaptive and PersonalizedComputer Science Education (APCSE) workshop @ UMAP’20 1
- 2. 1 2 3 4 Motivation Related Work Proposed Approach Results5 Study APCSE Workshop 6 Discussion / Conclusion Outline 2 Limitations and Future Work 6 7
- 5. 1 Motivation Research Gap Explainable Recommender Systems Benefits Domains Online shopping Movies Music Technology- Enhanced Learning Persuasi- veness Trust Efficiency Level of knowledge Learning goals APCSE Workshop 5 Recommenda- tions’ effects
- 6. Exploring the Need for Explainable Artificial Intelligence (XAI) in Intelligent Tutoring Systems (ITS) (2019)Vanessa Putnam and Cristina Conati at IUI’ 19 Complementing educational recommender systems with open learner models (2020) Solmaz Abdi et. al. at LAK’ 20 2 Related work APCSE Workshop Improving Student-System InteractionThrough Data- driven Explanations of Hierarchical Reinforcement Learning Induced Pedagogical Policies. (2020) Guojing Zhou et. al. at UMAP’20 6 Overall positive students’ attitude towards explanations (why?, how?) for hints in ITS (specially when disagreeing with the system) Explaining the tutor’s decisions to students through data- driven explanations could improve the student-system interaction in terms of students’ engagement and autonomy. Some students stated feeling of unfairness when given recommendations through OLM “explanations”
- 7. 3 Proposed approach Mastery Grids for SQL Programming APCSE Workshop 7
- 8. Jordan Barria-Pineda and Peter Brusilovsky. 2019. Making Educational RecommendationsTransparent through a Fine-GrainedOpen Learner Model. In IUI’19Workshops. APCSE Workshop 8 Mastery Grids with transparent recommenda- tions 3 Proposed approach
- 9. 3 Proposed approach Learning Content Recommendations APCSE Workshop 9 Roya Hosseini, I-Han Hsiao, Julio Guerra, and Peter Brusilovsky. 2015. What Should I Do Next? Adaptive Sequencing in the Context of Open Social Student Modeling. In Design for Teaching and Learning in a Networked World, Gráinne Conole, Tomaž Klobučar, Christoph Rensing, Johannes Konert, and Elise Lavoué (Eds.). Springer International Publishing, Cham, 155–168.
- 10. 3 Proposed approach Remedial Learning Content Recommendations APCSE Workshop 10
- 11. 3 Proposed approach Remedial Learning Content Recommendations APCSE Workshop Most recent success rate Estimated level of knowledge 11 Learning activity i Problematic concepts j Difficulty score
- 12. 3 Proposed approach Remedial Learning Content Recommendations APCSE Workshop Recommended learning activities + difficult - difficult 12 Difficulty scoring Available learning activities Only learning activities with misconceptions
- 13. 3 Proposed approach – Explanatory interfaces + 1 2 3 4 Explanations 1 2 3 4 APCSE Workshop 13
- 14. 3 Proposed approach – Explanatory interfaces APCSE Workshop
- 15. 4 Study APCSE Workshop 18 25 20 20DB management course February May Spring 2019 term 15
- 16. 4 Study APCSE Workshop 18 25 20 20DB management course Main activity in the system February May Spring 2019 term 16
- 17. 5 Results APCSE Workshop Data Analysis 1) Recommendations’ persuasiveness • Access rate • Conversion rate • Persistence rate 2) Effects of recommendations in learning 3) Subjective feedback - - - Morepersuasiveness 17
- 18. 5 Results - Persuasiveness APCSE Workshop Access rate 18 +
- 19. 5 Results - Persuasiveness APCSE Workshop Marginal interaction effects Conversion rate 19
- 20. 5 Results - Persuasiveness APCSE Workshop Marginal interaction effect Persistence rate 20 +
- 21. 5 Results – Effects on Learning APCSE Workshop Main predictor: Success rate NO EFFECT of explanatory treatments 21 Multiple regression
- 22. 5 Results – Subjective feedback APCSE Workshop 22 +
- 23. 6 Discussion / Conclusions APCSE Workshop Remedial recommendations affect learning content selection behavior of students Explanations can help students to add their own judgment on the top of recommendation to make more balanced decisions about working or not on an open recommended problem (rather than rushing to anything that is recommended) 23 With full explanations (i.e., more information about the generation), the recommendations’ quality was more likely to be critically judged + This help could be useful in online learning contexts where either student modeling or recommendation approaches are not highly accurate.
- 24. 7 Limitations / Future Work Students accessed practice content at the end of their course, a chance for recommendations to be less than perfect was relatively high. Marginally significant effects 24 Replicating the study in a setup where students would use the system throughout the whole term Exploring explanations to explain other information (e.g. input data) and recommendations focused on other learning goals (e.g. reinforcing existing knowledge) APCSE Workshop
- 25. Thanks for your attention! If you have any question please ask me or email me to jab464@pitt.edu In case you are teaching Java, Python or SQL classes we are open for collaborations! @j_barria
Editor's Notes
- Educational Recommender Systems (EdRecSys) are different in nature from conventional Recommender Systems (RecSys) -mostly related to e-commerce- as the main goal of EdRecSys is supporting students learning’ instead of maximizing users’ satisfaction from consuming the recommended items. Thus, research on transparency for traditional RecSys is hard to transfer from e-commerce contexts to educational scenarios, as the level of knowledge of the end-user (i.e. the student) is crucial for generating and evaluating the impact of the recommendations on students’ learning.
- A classroom experiment was run where we implemented a Mastery Grids version in an introductory Java course with a rule-based recommendation engine designed for supporting students’ knowledge expansion [1] (see Fig. 1). The group that received explanations exhibited more persuasive recommendations in terms of adopting the suggested material as part of their learning process.
- College students 2X2 design
- These results, combined, make us hypothesize that the inclusion of individual explanatory elements can lead students to think a little bit more about the appropriateness of the recommendations, especially when they have the meta-cognitive ability to do this (more likely to happen at the end of the course where students have a higher degree of self-awareness about their own knowledge).
- This shows that students overall persistence on problems, once they start working on them, is lower when they only had access to partial explana- tions (opposite phenomenon discovered for access rate).
- No significant differences in system’s satisfaction.
- Rec activities highlighted with stars were on average more attractive As the data show, the learners exhibited lower conversion rates for recommended problems than for non-recommended ones when any type of explanations (i.e., visual or textual) were provided. This result suggests that both explanation types enabled students to better add their own judgment on the top of recommendation and make a more balanced decision about working or not on an open recommended problem rather than rushing to anything that is recommended.
- when the students accessed practice content at the end of their course, a chance for recommendations to be less than perfect was relatively high since the system was not able to track knowledge gained by the students through most of their work in class