IS Peer Grading a
Valid Assessment for
HENG “PATRICK” LUO & ANTHONY C. ROBINSON
JOHN A. DUTTON E-EDUCATION INSTITUTE, PENN STATE
The Problem of Assessment
Why is assessment needed in MOOCs?
◦ Differs a MOOC from an educational website
◦ Closes the loop
Why is assessment difficult in MOOCs?
◦ Massive - workload
◦ Open – heterogeneous student population
◦ Online – technology affordances
◦ Course – reliability and validity emphasized
Why peer grading might be the
◦ Pros: immediate, easy-to-setup
◦ Cons: no feedback, can’t assess complex
Teaching Assistant Grading
◦ Pros: accurate, meaningful feedback (maybe)
can assess complex assignments
◦ Cons: expensive, takes time, “unsustainable”
Local Learning Center (blended)
◦ Pros: accurate, meaningful feedback
◦ Cons: difficult to build collaboration, only
accessible to certain students.
Findings from Literature Review
Peer Grading Reliability
◦ Measured by the agreement among peer graders
◦ Generally positive findings regarding reliability
◦ Calculated by Pearson r, Proportion of Variance, or Intraclass Correlation
◦ Sometimes mistaken for validity, much smaller body of research
Peer Grading Validity
◦ Measured by the similarity between peer grading and instructor grading
◦ Positive findings reported in various contexts
Peer Grading in MOOCs
◦ Few empirical studies
◦ Mixed experiences reported from both instructors and students
Coursera’s Peer Grading System
Each submitted assignment is randomly
graded by 3-5 peers.
Instructor provides the grading rubric
and builds it into the Peer Grading
Evaluation can be quantitative,
qualitative, or both.
Students can grade their own
Median Score is used as the final peer
RQ1: Can Coursera’s peer review system provide a reliable and valid assessment of student
assignments in a MOOC?
RQ2: Does the use of median score provide a more valid assessment than the use of mean score
when calculating the final peer grading scores?
RQ3: What are the perceived effects of peer grading on students’ MOOC learning experience?
The structure of peer grading assignment data in Coursera’s database
Attributes of a submitted peer grading assignment
Finding - Reliability
95% Confidence Interval F Test with True Value 0
Lower bound Upper bound Value df1 df2 Sig
Single Measures .262 .240 .284 2.774 1824 7300 .000
Average Measures .640 .613 .665 2.774 1824 7300 .000
Intraclass Correlation Coefficient (Case 1) for Peer Grading Scores (N=1825)
Clarity of the
Quality of the
Aesthetics of the
Single Measures .216 .215 .176 .210
Average Measures .579 .578 .516 .571
Intraclass Correlation Coefficients (Case 1) for the Four Criterion Scores (N=1825)
instructor_grading peer_grading_median peer_grading_mean self_grading
instructor_grading 1 .619** .662** .341**
peer_grading_median 1 .952** .279**
peer_grading_mean . 1 .464**
Pearson’s Correlation Coefficient between Instructor, Peer, and Self Grading Scores (N=93)
**. Correlation is significant at the 0.01 level (2-tailed)
Question Questions (1-5 from strongly disagree to strongly agree) N Positive
The peer review process helped me to grow and improve as a spatial thinker. 2121 63% 3.67
I received fair grades on my Lesson 5 Mapping Assignment from my peers. 1694 62% 3.75
The feedback my peers gave me on my Lesson 5 Mapping Assignment was useful. 1719 61% 3.69
The peer grading activity in this course made me feel more engaged in the course. 2027 63% 3.70
The peer grading activity in this course made me feel more connected with other students. 2039 57% 3.57
The peer grading activity in this course provided me with opportunities to review and/or
reflect on the course content.
2044 72% 3.88
I would recommend keeping the peer grading assignment in future offerings of this course. 2185 70% 3.90
*. Agreed or strongly agreed responses from the students are considered as positive responses
Students’ Ratings of the Seven Statements Regarding the Peer Grading Activity in MGR
The inter-rater reliability of peer grading is low.
The use of multiple student graders can significantly increase the peer grading reliability.
Peer grading seems to be a valid assessment of student assignment (both median-based or
Self-grading is less valid and tends to produce inflated scores.
General perception towards the peer grading activity in MOOC is positive. Students think it is
fair, useful, and beneficial to their MOOC learning experience.