The value of students developing the capacity to make accurate judgements about the quality of their work and that of others has been widely studied and recognised in higher education literature. To date, much of the research and commentary on evaluative judgement has been theoretical in nature, focusing on perceived benefits and proposing strategies seen to hold the potential to foster evaluative judgement. Their efficacy remains largely untested. The rise of educational tools and technologies which generate data on learning activities at an unprecedented scale, alongside insights from the learning analytics and educational data mining communities, provide new opportunities for fostering and supporting empirical research on evaluative judgement. Accordingly, this paper offers a conceptual framework and an instantiation of the framework in the form of an educational tool called RiPPLE for data-driven approaches to investigate the enhancement of evaluative judgement. Two case studies, demonstrating how RiPPLE can foster and support empirical research on evaluative judgement are presented.

1. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement
Development of educational tools that enable large-scale ethical
empirical research on evaluative judgement
Dr Hassan Khosravi
The University of Queensland
Brisbane, QLD, Australia
h.khosravi@uq.edu.au
George Gyamfi
The University of Queensland
Brisbane, QLD, Australia
g.gyamfi@uq.net.au
Dr Barbara Hanna
The University of Queensland
Brisbane, QLD, Australia
b.hanna@uq.edu.au
Dr Jason Lodge
The University of Queensland
Brisbane, QLD, Australia
jason.lodge@uq.edu.au

2. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 2
Overview
Evaluative Judgement Educational Tools and Educational Research
Conceptual Model & a Referenced Implementation Case Studies and Data-Driven Reflections

3. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 3
Overview
Best short paper nomination
Invited submission to the Journal of Learning Analytics
(under review)

4. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 4
Evaluative judgement (EJ) is the capability to make
decisions about the quality of work of oneself and
others.
• Recognised as the skill that empowers students to:
1. Use feedback effectively
2. Develop expertise in their field
3. Develop the autonomy to think critically
4. Become reflexive and lifelong learners with
knowledge of their evaluative potential.
What is Evaluative Judgement?

5. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 5
Strategies to Foster Evaluative Judgement
Rubrics Self-assessment Feedback ReflectionExemplars Peer-assessment
q Current research on EJ’s potential has been largely theoretical.
q Empirical work to verify the effect size of the proposed strategies is emerging.

6. Development of educational tools that enable large-scale ethical empirical research on evaluative judgementDevelopment of educational tools that enable large-scale ethical empirical research on evaluative judgement
Evaluative Judgement Educational Tools and Educational Research
Conceptual Model & a Referenced Implementation Case Studies and Data-Driven Reflections

7. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 7
Facilitating Evaluative Judgement via Educational Technologies
Peer grading and evaluation system
q Most educational technologies are built without the aim of supporting research.
q They do not enable data harvesting or conducting controlled experiments.

8. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 8
Two Success Stories
§ Data from PeerWise has
been used by 80
publications.
§ Research supported:
impact of gamification and
the ability of students to
develop high-quality
learning resources.
Associate Professor
Paul Denny
§ Data from ASSISTments has
enabled 27 publications.
§ Research supported:
adaptive learning and the
personalisation of feedback.
Professor Neil Heffernan

9. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 9
Aim
Present a conceptual framework and an instantiation of the
framework in the form of an educational tool called RiPPLE for
data-driven approaches to investigate the enhancement of
evaluative judgement.

10. Development of educational tools that enable large-scale ethical empirical research on evaluative judgementDevelopment of educational tools that enable large-scale ethical empirical research on evaluative judgement
Evaluative Judgement Educational Tools and Educational Research
Conceptual Model & a Referenced Implementation Case Studies and Data-Driven Reflections

11. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 11
The Conceptual Model
• The proposed framework consists of seven main considerations for the
implementation of educational tools that promote both the development of
evaluative judgement and research into it.

12. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 12
• An overview of the referenced implementation for a tool called RiPPLE based
on the proposed conceptual model.
A Referenced Implementation

13. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 13
The RiPPLE Platform
Content creation Content Evaluation Adaptive practice
Peer study recommendations Clicker-based in-class activity See ripplelearning.org
50
Course offerings
Over 12,000
Users
Over 15,000
Resources Created
Over 1,000,000
Resources viewed

14. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 14
Content Creation
Multi-Choice and Multi-Answer Worked Examples Notes

15. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 15
Can Students Create High-Quality Resources?
“60% of all explanations classified as being of high or outstanding
quality. Overall, 75% of questions met combined quality criteria”
“People with greater expertise tend to make assumptions about student
learning that turn out to be in conflict with students’ actual performance
and developmental propensities.”
“Crowdsourcing can efficiently yield high-quality assessment
items that meet rigorous judgmental and statistical criteria. ”
Overall, 86% of the examples classified in this multiple cohort investigation were
found to be a ‘High quality question’ by being coherent, correct, requiring more
than just simple factual recall, possessing a valid solution and reasonable
distractors.

16. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 16
Self and Peer Assessment

17. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 17
Assessment Feedback

18. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 18
Delivering an Adaptive Learning Experience

19. Development of educational tools that enable large-scale ethical empirical research on evaluative judgementDevelopment of educational tools that enable large-scale ethical empirical research on evaluative judgement
Evaluative Judgement Educational Tools and Educational Research
Conceptual Model & a Referenced Implementation Case Studies and Data-Driven Reflections

20. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 20
• Aim: provides an example of how RiPPLE may be used to conduct empirical
research on evaluative judgement.
• Research questions
• RQ1. How do students’ subjective evaluations of learning resources compare with those
of domain experts?
• RQ2. What is the impact of practice over time on students’ ability to judge the quality of
learning resources?
Case Study – Observational study
Location: The University of Queensland (UQ)
Course: Introduction to Information System
Number of students: 512
Number of experts: 6
Number of Resources: 2,355
Number of ratings: 31,143
Duration: 13 Weeks

21. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 21
• RQ1. How do students’ subjective evaluations of learning resources compare with those of
domain experts?
Results – RQ1
• This analysis reveals a strong and
positive correlation (r=0.832, p<0.05)
between ratings from the two groups.

22. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 22
RQ2. What is the impact of practice over time on students’ ability to judge the quality of
learning resources?
Results – RQ2
Domain experts’ ratings as the gold standard
to compute Root Mean Squared
Error(RMSE)
A statistically significant association
between weekly RMSE value at p=0.0226
and a mild inverse correlation (reduction in
student error in quality ratings) between
week r=-0.602.

23. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 23
Data Driven Reflections - Instructors vs Students
Data
• #Courses= 6
• #Expert Spot Checks = 1,011
• # Learner Ratings on the spot-checks= 3,464
Results
• When instructors provide a rating of 3, 4 or 5 there is
92% agreement and 8% disagreement
• When instructors provide a rating of 1 or 2 there is
19% agreement and 81% disagreement
• The task of identifying low-quality resources based on
solely students judgements maybe challenging.

24. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 24
• On the effects of rubrics on evaluative judgement: A Randomised Controlled
Experiment – measuring time on task, confidence and outcome
Case Study – Controlled Experiment
Control Group Experiment Group
Presented in the following talk

25. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 25
• On the effects of self-assessment on evaluative judgement: A Randomised
Controlled Experiment - measuring time on task, confidence and outcome
Experiments Under Investigation
Control Group Experiment Group

26. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 26
• On the effects of feedback on evaluative judgement: A Randomised Controlled
Experiment - measuring time on task, confidence and outcome
Experiments Under Investigation
Control Group Experiment Group

27. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 27
Data-Driven Reflections – Rubric Criteria
Criteria: Alignment, Correctness, Clarity Criteria: Alignment, Correctness + Clarity, Difficulty, Critical thinking

28. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 28
Data-Driven Reflections – Rating vs Confidence
Data
• #Courses= 19
• #Learners= 2,410
• #Experts= 28
• #Resources= 14,559
• #Learner Moderations= 64,044
Results
• students are most likely to provide a high rating with a
high level of confidence
• The Dunning-Kruger effect, in which low-performing
students tend to overestimate their abilities may be
one of the factors that contribute to the large number
of students who self-assess their confidence as high

29. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement 29
Summary
Evaluative Judgement Educational Tools and Educational Research
Conceptual Model & a Referenced Implementation Case Studies and Data-Driven Reflections

30. Development of educational tools that enable large-scale ethical empirical research on evaluative judgement
Development of educational tools that enable large-scale ethical empirical research
on evaluative judgement
Dr Hassan Khosravi
h.khosravi@uq.edu.au
George Gyamfi
g.gyamfi@uq.net.au
Dr. Barbara Hanna
b.hanna@uq.edu.au
Dr. Jason Lodge
jason.lodge@uq.edu.au