This presentation offers data-driven reflections and lessons learned from the development and deployment of a learnersourcing adaptive educational system called RiPPLE, which to date, has been used in more than 50-course offerings with over 12,000 students. Our reflections are categorised into examples and best practices on (1) assessing the quality of students’ contributions using accurate, explainable and fair approaches to data analysis, (2) incentivising students to develop high-quality contributions and (3) empowering instructors with actionable and explainable insights to guide student learning. The paper associated with the presentation is available at https://dl.acm.org/doi/abs/10.1145/3448139.3448143

1. Charting the Design and Analytics Agenda of
Learnersourcing Systems
Dr Hassan Khosravi
The University of Queensland
Brisbane, QLD, Australia
h.khosravi@uq.edu.au
Professor Shazia Sadiq
The University of Queensland
Brisbane, QLD, Australia
shazia@itee.uq.edu.au
Professor Dragan Gasevic
Monash University
Clayton, Vic, Australia
Dragan.Gasevic@monash.edu
A/Prof Gianluca Demartini
The University of Queensland
Brisbane, QLD, Australia
g.demartini@uq.edu.au

2. Charting the Design and Analytics Agenda of Learnersourcing Systems 2
Challenges in Teaching Large Classes
Learnersourcing is emerging as a viable and pedagogically justified approach of addressing these
challenges.
Help students develop creativity,
critical thinking, communication and
collaboration skills
Provide timely and authentic feedback
Provide learning content and
instructions that cater to the diverse
academic ability of the learners
https://httpessaysahmyookhs.wordpress.com/21st-century-skills/

3. Charting the Design and Analytics Agenda of Learnersourcing Systems 3
Overview
Data Driven reflections and lessons learned from design,
development and deployment of RiPPLE
Suggestions for developing learnersourcing system
The RiPPLE system
Learnersourcing

4. Charting the Design and Analytics Agenda of Learnersourcing Systems 4
Learnersourcing: a pedagogically
supported form of crowdsourcing that
mobilises the learner community as
experts-in-training to contribute to
teaching and learning activities while
being engaged in meaningful learning
experiences themselves.
Learnersourcing
Kim, J, 2015. Learnersourcing: improving learning with collective learner
activity. Ph.D. Dissertation. Massachusetts Institute of Technology.
Khosravi, H. and Demartini, G, Sadiq, S., & Gasevic D. (2021). Charting the
Design and Analytics Agenda of Learnersourcing Systems. Lak’11 (in
press)

5. Charting the Design and Analytics Agenda of Learnersourcing Systems 5
Learnersourcing in Action

6. Charting the Design and Analytics Agenda of Learnersourcing Systems 6
Benefits
• Promotes critical learning, creativity, collaboration
and communication.
• Recognises them as partners in learning.
• Helps them develop evaluative judgement.
• Leads to the development of large repositories
study material.
Challenges
• Assessing quality of contributions
• Incentivising students to contribute
• Enabling Instructors to provide oversight with
minimal additional work.
Learnersourcing Benefits and Challenges

7. Charting the Design and Analytics Agenda of Learnersourcing Systems 7
Problem, Aim and Contribution
Problem
best practices for design and development of
learnersourcing systems are unknown
Aim
Contribute to development of best practices
for implementation of learnersourcing
systems
https://blog.marketo.com/2014/02/how-to-set-goals-you-can-meet.html
https://www.searchenginejournal.com/question-improve-your-problem-solving/327995/#close
Contribution
Share data-driven reflections and lessons
learned from design, implementation and
deployment of a learnersourcing system

8. Overview
Data Driven reflections and lessons learned from design,
development and deployment of RiPPLE
Learnersourcing The RiPPLE system
Suggestions for developing learnersourcing system

9. Charting the Design and Analytics Agenda of Learnersourcing Systems 9
The RiPPLE System
9
Key Features
Partnering with students to
develop a repository of
high-quality study material.
Learnersourcing Adaptive learning
Recommending learning resources
to each student based on
their learning needs.
Engaging students
with interactive in-class or self-
paced learning activities
Peer learning
Helping students find
effective study partners
that suit their availability
and learning needs.
Additional features
Interactive learning activities
RiPPLE is an open access educational platform that aims to provide a personalised, active and social learning
experience.
Khosravi et al. in the Journal of Learning Analytics (2019)
For more info visit http://ripplelearning.org/

10. Charting the Design and Analytics Agenda of Learnersourcing Systems 10
Adoption and Recognition
RiPPLE is increasingly being recognised as an exemplar tool for using AI in education.
15 Peer-reviewed articles
2 best paper nomination
Featured as an exemplar
educational technology in
EDUCAUSE Horizon 2019 Report
Academics and researchers
from over 50 universities have
accessed RiPPLE
After six semesters, RiPPLE has been adopted by courses in Medicine, Pharmacy,
Psychology, Education, Business, Computer Science & Biosciences.
60
Course offerings
14000+
Users
1,000,000+
Resources studied
30,000+
Study resources created

11. Charting the Design and Analytics Agenda of Learnersourcing Systems 11
Learnersourcing in RiPPLE
(1) Students author learning resources. (2) Resources are moderated by peers and tutors.
(3) Based on the moderations a decision is made. (4) RIPPLE flags questionable resources for spot checking.

12. Overview
Suggestions for developing learnersourcing system
Learnersourcing The RiPPLE system
Data driven reflections and lessons learned from design,
development and deployment of RiPPLE

13. Charting the Design and Analytics Agenda of Learnersourcing Systems 13
2. Incentivising high quality contributions
1. Quality control of student generated content
3. Empowering instructors with explainable and actionable insights
Data Driven Reflections and Lessons Learned

14. Charting the Design and Analytics Agenda of Learnersourcing Systems 14
Can Students Accurately Evaluate the Quality of Learning Resources?
• Data presence 64k moderations from 2.4k
learners and 28 instructors on 15K resources.
• Overall, ratings provided by students strongly
correlate with ratings from experts
Abdi et al. in IEEE Transactions on Educational Technologies (2021)
• When instructors reject a resource, the chance of
students also rejecting is 16%.
• Identifying low-quality resources based on peer
review is challenging

15. Charting the Design and Analytics Agenda of Learnersourcing Systems 15
1.1 Developing Accurate and Explainable Consensus Approaches
Arrieta, A et al. (2020). Explainable Artificial Intelligence (XAI):
Concepts, taxonomies, opportunities and challenges toward
responsible AI. Information Fusion, 58, 82-115.
How can we accurately, transparently and fairly evaluate student-generated content?
TPR TNR AUC TPR TNR AUC
Majority Vote 0.96 0.20 0.58 0.99 0.10 0.54
Mean 0.96 0.20 0.58 1.00 0.14 0.57
Median 1.00 0.20 0.60 1.00 0.14 0.57
Expectation Maximisation 0.94 0.4 0.67 1.00 0.33 0.67 High
Trust Propagation 0.81 0.62 0.71 0.88 0.52 0.70 Medium
Sentiment-Alignment 0.94 0.53 0.74 1.00 0.29 0.64
Relatedness - BERT 0.94 0.60 0.77 0.98 0.24 0.61
Length×Relatedness 0.92 0.73 0.83 0.91 0.76 0.84
Bagged Decision Tree 0.89 0.53 0.71 0.91 0.81 0.86
Random Forest 0.91 0.6 0.75 0.91 0.67 0.79
Boosting 0.87 0.6 0.73 0.91 0.67 0.79
RPART 0.79 0.87 0.83 0.90 0.86 0.88
Comment-based Models
Type
Baseline Models
Ensemble Models
Explainable ?
Very High
Very Low
Low
Method
INFS NEUR
Probabilistc models
Darvishi et al., to appear in the proceedings of
the Learning at Scale conference (2020)

16. Charting the Design and Analytics Agenda of Learnersourcing Systems 16
1.2 Employing Human-in-the-loop and Responsible AI
Based on: https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems
Full Automation: AI as a Big Red Button Designing with a Human in the Loop
Benefits
ü Significant gains in transparency
ü Effective incorporation of human judgment
ü Less pressure on building “perfect”
algorithms.
Challenges
Data and algorithmic bias
Lack of transparency and trust in decision
making

17. Charting the Design and Analytics Agenda of Learnersourcing Systems 17
1.2 Employing Human-in-the-loop in RiPPLE
Challenging the outcome of a moderation

18. Charting the Design and Analytics Agenda of Learnersourcing Systems 18
1.3 Employing Appropriate Criteria for Evaluating Resources
Criteria: Alignment, Correctness, Clarity Alignment, Correctness + Clarity, Difficulty, Critical thinking
Gyamfi et al. in Assessment and Higher Education (2021)

19. Charting the Design and Analytics Agenda of Learnersourcing Systems 19
Quality control of student generated content
Empowering instructors with explainable and actionable insights
Data Driven Reflections and Lessons Learned
Incentivising high quality contributions

20. Charting the Design and Analytics Agenda of Learnersourcing Systems 20
Participation inequality. In most online
communities, 90% of users are lurkers who
never contribute, 9% of users contribute a
little, and 1% of users account for almost all
the action.
Incentivising high quality contributions
Many streaming, gaming, and social media
platforms are designed with the prime
intention of increasing engagement without
considering the quality of the engagement.
So how can learnersourcing systems be designed to encourage a large portion of the student population
to contribute high-quality learnersourcing activities?
Photo by camilo jimenez on Unsplash
Source: https://www.nngroup.com/articles/participation-inequality/

21. Charting the Design and Analytics Agenda of Learnersourcing Systems 21
2.1 Modelling students in Learnersourcing Systems
Abdi et al. in proceedings of AI in Education (2020 and 2021)
1. Discouraging learnersourcing contributions.
2. Missing the opportunity to leverage learnersourcing data in modelling students
3. Advancing the belief that learnersourcing does not contribute to learning.

22. Charting the Design and Analytics Agenda of Learnersourcing Systems 22
2.1 Open Learner Models for Learnersourcing Systems
Abdi et al. in proceedings of AI in Education (2020 and 2021)

23. Charting the Design and Analytics Agenda of Learnersourcing Systems 23
• Using RiPPLE Formatively
• Engagement following 90-9-1 rule
• Low amount of high-quality engagement
2.2 Tying Learnersourcing to Assessment
• Using RiPPLE Summatively
• High amount of low quality engagement.
• Requires supporting assessment logistics

24. Charting the Design and Analytics Agenda of Learnersourcing Systems 24
2.3 Employing Gamification Mechanisms
Profile page Leaderboard Google Analytics

25. Charting the Design and Analytics Agenda of Learnersourcing Systems 25
Quality control of student generated content
Incentivising high quality contributions
Data Driven Reflections and Lessons Learned
Empowering instructors with explainable and actionable insights

26. Charting the Design and Analytics Agenda of Learnersourcing Systems 26
3.1 The Analytics Toolbox

27. Charting the Design and Analytics Agenda of Learnersourcing Systems 27
3.2 Spot checking in RiPPLE
Spot checking in RiPPLE
• Identifying resources that have passed moderation but are likely to be incorrect or ineffective.
• Uses human-driven metrics (e.g., high-disagreement in moderation evaluations) and data-
driven metrics (e.g., low discrimination) to recommend resources for inspection.

28. Charting the Design and Analytics Agenda of Learnersourcing Systems 28
3.2 Explainable Spot Checking in RiPPLE

29. Overview
Learnersourcing The RiPPLE system
Data driven reflections and lessons learned from design,
development and deployment of RiPPLE
Suggestions for developing learnersourcing system

30. Charting the Design and Analytics Agenda of Learnersourcing Systems 30
Suggestions for Developing Learnersourcing Systems
Develop accurate and
explainable consensus
approaches
Enable users to raise
concerns against
automated decisions
Reflect on the criteria
used in the evaluation
student generated content
Use open learner models
to present student
contributions
Support logistics to tie
leanersourcing to
assessment
Employ various
gamification
mechanisms
Empower instructors with
actionable and explainable
insights
conduct rigorous
empirical studies to
evaluate the platform.

31. Charting the Design and Analytics Agenda of Learnersourcing Systems 31
1. Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021). Evaluating the Quality of Learning Resources: A
Learnersourcing Approach. IEEE Transactions on Learning Technologies, 14(1), 81-92.
2. Abdi, S., Khosravi, H., & Sadiq, S. (2020, July). Modelling Learners in Crowdsourcing Educational Systems.
In International Conference on Artificial Intelligence in Education (pp. 3-9). Springer, Cham.
3. Arrieta, A et al. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges
toward responsible AI. Information Fusion, 58, 82-115.
4. Darvishi, A., Khosravi, H., & Sadiq, S. (2020, September). Utilising Learnersourcing to Inform Design Loop Adaptivity.
In European Conference on Technology Enhanced Learning (pp. 332-346). Springer, Cham.
5. Gyamfi, G., Hanna, B. E., & Khosravi, H. (2021). The effects of rubrics on evaluative judgement: a randomised
controlled experiment. Assessment & Evaluation in Higher Education, 1-18.
6. Jiang, Y., Schlagwein, D., & Benatallah, B. (2018, June). A Review on Crowdsourcing for Education: State of the Art
of Literature and Practice. In PACIS (p. 180).
7. Kim, J. (2015). Learnersourcing: improving learning with collective learner activity (Doctoral dissertation,
Massachusetts Institute of Technology).
8. Khosravi, H., Kitto, K., & Williams, J. J. (2019). RiPPLE: A Crowdsourced Adaptive Platform for Recommendation of
Learning Activities. Journal of Learning Analytics, 6(3), 91-105.
References