The value of students developing the capacity to make accurate judgements about the quality of their work and that of others has been widely recognised in higher education literature. However, despite this recognition, little attention has been paid to the development of tools and strategies with the potential both to foster evaluative judgement and to support empirical research into its growth. This paper provides a demonstration of how educational technologies may be used to fill this gap. In particular, we introduce the adaptive learning system RiPPLE and describe how it aims to (1) develop evaluative judgement in large-class settings through suggested strategies from the literature such as the use of rubrics, exemplars and peer review and (2) enable large empirical studies at low cost to determine the effect-size of such strategies. A case study demonstrating how RiPPLE has been used to achieve these goals in a specific context is presented.

1. Fostering and Supporting Empirical Research on Evaluative Judgement
Fostering and Supporting Empirical Research on Evaluative
Judgement via a Crowdsourced Adaptive Learning System
Dr Hassan Khosravi
The University of Queensland
Brisbane, QLD, Australia
h.khosravi@uq.edu.au
George Gyamfi
The University of Queensland
Brisbane, QLD, Australia
g.gyamfi@uq.net.au
Dr. Barbara Hanna
The University of Queensland
Brisbane, QLD, Australia
b.hanna@uq.edu.au
Dr. Jason Lodge
The University of Queensland
Brisbane, QLD, Australia
jason.lodge@uq.edu.au

2. Fostering and Supporting Empirical Research on Evaluative Judgement 2
Overview
What is Evaluative Judgement (EJ)? Related work and challenges
Fostering and supporting empirical research on
evaluative judgement via a system called RiPPLE
A case study demonstrating how RiPPLE can facilitate
empirical research on EJ is presented

3. Fostering and Supporting Empirical Research on Evaluative JudgementDevelopment and Adoption of an Adaptive Learning System
Case Study
The RiPPLE Platform
Evaluative Judgement and Related Work

4. Fostering and Supporting Empirical Research on Evaluative Judgement 4
Evaluative judgement (EJ) is the capability to make
decisions about the quality of work of oneself and
others.
Hailed as the skill that will allow students to:
1. Use feedback effectively,
2. Develop expertise in their field and
3. Extend their understanding beyond current work
to future endeavours, including lifelong learning.
What is Evaluative Judgement?

5. Fostering and Supporting Empirical Research on Evaluative Judgement 5
Strategies to Foster Evaluative Judgement
Rubrics Self-assessment Feedback ReflectionExemplars Peer-assessment
q Current research on EJ’s potential has been largely theoretical.
q Little to no empirical work exist to verify the effect size of the proposed strategies.

6. Fostering and Supporting Empirical Research on Evaluative Judgement 6
Facilitating Evaluative Judgement via Educational Technologies
Peer grading and evaluation system
q Most educational technologies are built without the aim of supporting research.
q They do not enable data harvesting or conducting controlled experiments.

7. Fostering and Supporting Empirical Research on Evaluative Judgement 7
Two Success Stories
§ Data from PeerWise has
been used by 80
publications.
§ Research supported:
impact of gamification and
the ability of students to
develop high-quality
learning resources.
Associate Professor
Paul Denny
§ Data from ASSISTments has
enabled 27 publications.
§ Research supported:
adaptive learning and the
personalisation of feedback.
Professor Neil Heffernan

8. Fostering and Supporting Empirical Research on Evaluative Judgement 8
Aim
Introduce system that fosters and supports empirical
research on evaluative judgement.

9. Fostering and Supporting Empirical Research on Evaluative JudgementDevelopment and Adoption of an Adaptive Learning System
Case Study
Evaluative Judgement and Related Work
The RiPPLE Platform

10. Fostering and Supporting Empirical Research on Evaluative Judgement 10
The RiPPLE Platform
Content creation Content moderation Adaptive practice
Peer study recommendations Clicker-based in-class activity See ripplelearning.org

11. Fostering and Supporting Empirical Research on Evaluative Judgement 11
Content Creation
MCQs Worked Examples Notes

12. Fostering and Supporting Empirical Research on Evaluative Judgement 12
Self and Peer Assessment

13. Fostering and Supporting Empirical Research on Evaluative Judgement 13
Assessment Feedback

14. Fostering and Supporting Empirical Research on Evaluative Judgement 14
Delivering an Adaptive Learning Experience
1. The Learner model visualises the current
knowledge stage of a student
2. The recommender system suggests effective
resources on topics that the student is developing at
an appropriate level of difficulty

15. Fostering and Supporting Empirical Research on Evaluative Judgement 15
RiPPLE Demo

16. Fostering and Supporting Empirical Research on Evaluative Judgement 16
Support Ethical Low Cost Empirical Educational research
Develop guidelines
referring to consent,
transparency and
benevolence
Develop mechanisms
for instructors to run
controlled
experiments.
Provide access to rich
analytics to support
observational studies.
Develop mechanisms for
instructors to run quasi-
experiments.

17. Fostering and Supporting Empirical Research on Evaluative JudgementDevelopment and Adoption of an Adaptive Learning System
The RiPPLE Platform
Evaluative Judgement and Related Work
Case Study

18. Fostering and Supporting Empirical Research on Evaluative Judgement 18
• Aim: provides an example of how RiPPLE may be used to conduct empirical
research on evaluative judgement.
• Research questions
• RQ1. How do students’ subjective evaluations of learning resources compare with those
of domain experts?
• RQ2. What is the impact of practice over time on students’ ability to judge the quality of
learning resources?
Case Study
Location: The University of Queensland (UQ)
Course: Introduction to Information System
Number of students: 512
Number of experts: 6
Number of items: 2,355
Number of ratings: 31,143
Duration: 13 Weeks

19. Fostering and Supporting Empirical Research on Evaluative Judgement 19
Experimental Setting
q Exclude students who answered less than
25 questions (319 remaining).
q Group students into high performing (top
27%), low performing (bottom 27%) and
medium performing (rest)
Students
q Exclude items that had less than 10 ratings
from each group of students (1632 remaining)
q Group items into high, average and low
quality.
q Randomly select 14 items from each group
Items
q 6 domain experts were recruited
q Each expert independently reviewed the
selected 42 items.
Experts

20. Fostering and Supporting Empirical Research on Evaluative Judgement 20
• RQ1. How do students’ subjective evaluations of learning resources compare with those of
domain experts?
Results – RQ1
• This analysis reveals a strong and
positive correlation (r=0.832, p<0.05)
between ratings from the two groups.

21. Fostering and Supporting Empirical Research on Evaluative Judgement 21
RQ2. What is the impact of practice over time on students’ ability to judge the quality of
learning resources?
Results – RQ2
Domain experts’ ratings as the gold standard
to compute Root Mean Squared
Error(RMSE)
A statistically significant association
between weekly RMSE value at p=0.0226
and a mild inverse correlation (reduction in
student error in quality ratings) between
week r=-0.602.

22. Fostering and Supporting Empirical Research on Evaluative Judgement 22
• On the effects of rubrics on evaluative judgement: A Randomised Controlled
Experiment – measuring time on task, confidence and outcome
Examples of RCT experiments
Controlled Group Experiment Group

23. Fostering and Supporting Empirical Research on Evaluative Judgement 23
• On the effects of self-assessment on evaluative judgement: A Randomised
Controlled Experiment - measuring time on task, confidence and outcome
Examples of RCT experiments
Controlled Group Experiment Group

24. Fostering and Supporting Empirical Research on Evaluative Judgement 24
• On the effects of feedback on evaluative judgement: A Randomised Controlled
Experiment - measuring time on task, confidence and outcome
Examples of RCT experiments
Controlled Group Experiment Group

25. Fostering and Supporting Empirical Research on Evaluative Judgement 25
Summary
What is Evaluative Judgement (EJ)? Related work and challenges
Fostering and supporting empirical research on
evaluative judgement via a system called RiPPLE
A case study demonstrating how RiPPLE can facilitate
empirical research on EJ is presented

26. Fostering and Supporting Empirical Research on Evaluative Judgement
Fostering and Supporting Empirical Research on Evaluative
Judgement via a Crowdsourced Adaptive Learning System
Dr Hassan Khosravi
h.khosravi@uq.edu.au
George Gyamfi
g.gyamfi@uq.net.au
Dr. Barbara Hanna
b.hanna@uq.edu.au
Dr. Jason Lodge
jason.lodge@uq.edu.au

27. Fostering and Supporting Empirical Research on Evaluative Judgement 27