This document discusses the use of peer review and rubrics in the Canvas learning management system at Harvard Kennedy School and Harvard School of Engineering and Applied Sciences. It provides examples of how policy analysis and engineering courses used peer review in Canvas to provide students feedback on assignments. Both faculty and students found the peer review process in Canvas to be useful, though some challenges were noted, such as limitations in Canvas functionality and difficulties navigating peer feedback. Lessons learned focused on strategies to improve the peer review experience for students and faculty.

1. Rubrics and Peer-Review in Canvas: Examples from
FAS and HKS
Daniel Jamous, Senior Instructional Technologist, Academic
Technology Services, Harvard University Information Technology
Alice Heath, Master in Public Policy Candidate, Harvard Kennedy
School
Kelly Miller, Lecturer on Applied Physics, Harvard John A. Paulson
School of Engineering and Applied Sciences
Salimah Samji, Manager of Building State Capability Program, Center
for International Development, Harvard Kennedy School
Allison Pingree, Director of Professional Pedagogy, Strengthening
Learning and Teaching Excellence (SLATE), Harvard Kennedy School
IT Summit
June 2 2016

2. Using Canvas for Peer Review at HKS:
Policy Analysis & Design (API-505)
Allison Pingree & Alice Heath
Harvard IT Summit
Thursday, June 2, 2016 - 2:20– 3:15 pm

3. Course Context: API-505, Policy Analysis & Design
• Required for all 1st-year Masters of Public Policy Students
• 4 sections, each taught by a different faculty
• Integrates other core courses, leading to deliverables that reflect the
practice of real-world public policy: memos, oral briefings, slide decks
• One section (course head, Prof. John Donahue, chair of MPP Program)
made most extensive use of peer review and rubrics in Canvas
• Two other sections used peer review in Canvas with simplified rubric; 4th
used peer review via e-mail

4. Why Use Peer Review with a Rubric?
Why in Canvas?
• We knew from Spring Exercise [experiential capstone] that the draft-feedback-
rewrite cycle is a great tool for learning.
• We also knew from prior versions of this course that the faculty simply couldn’t
give detailed feedback on every draft, especially not in time for it to be useful.
• We’d learned from observing the Negotiation course that the MPPs—especially
in cohorts—were skilled at dealing with each other candidly but thoughtfully.
• Course focus itself is the evaluation of options along various criteria - rubric
allowed us to be more transparent about quality, and practice what we preach.
• We needed a way to manage the logistical complexity of pairing up students
and tracking their exchanges.

8. Overall, How Did it Go?
In-class poll of students:
• very strong support for the usefulness of the peer reviews (~40% extremely useful, 30%
definitely useful, 20% somewhat useful, and 10% or less not useful)
End-of-course evaluations: mentioned 5x as one of “2-3 most valuable aspects” of course:
• “Requiring us to provide peer feedback on the first memo assignment was very helpful--
forced us to do something that we would not have done otherwise, but was really
valuable.”
• “The peer feedback mechanism was excellent. Our group found it so valuable that we
organized peer feedback for the last assignment even though it wasn't required.”
Faculty:
• “I think it was one of the best features of the course this year. . . . Canvas, while not
without its glitches, is a huge leap forward.”

9. But Not Without Its Challenges . . .
1) Canvas does not allow you to set a deadline within peer review, or to grade
the quality of the peer review.
2) There are several different ways to give feedback on Canvas peer review,
some of which miss the main point of doing peer review.
• Line edits
• Attach files
• Use a rubric
• Write a general comment
3) From a student’s perspective, finding their peers’ review comments is not
intuitive -- there are several routes to see different types of feedback.

10. But Not Without Its Challenges . . .
1) Canvas does not allow you to set a deadline within peer review, or to grade
the quality of the peer review.
2) There are several different ways to give feedback on Canvas peer review,
some of which miss the main point of doing peer review.
• Line edits
• Attach files
• Use a rubric
• Write a general comment
3) From a student’s perspective, finding their peers’ review comments is not
intuitive -- there are several routes to see different types of feedback.

13. But Not Without Its Challenges . . .
1) Canvas does not allow you to set a deadline within peer review, or to grade
the quality of the peer review.
2) There are several different ways to give feedback on Canvas peer review,
some of which miss the main point of doing peer review
• Line edits
• Attach files
• Use a rubric
• Write a general comment
3) From a student’s perspective, finding their peers’ review comments is not
intuitive -- there are several routes to see different types of feedback.

18. But Not Without Its Challenges . . .
1) Canvas does not allow you to set a deadline within peer review, or to grade
the quality of the peer review.
2) There are several different ways to give feedback on Canvas peer review,
some of which miss the main point of doing peer review.
• Line edits
• Attach files
• Use a rubric
• Write a general comment
3) From a student’s perspective, finding their peers’ review comments is not
intuitive -- there are several routes to see different types of feedback.

20. Lessons Learned
• Strategies
• If you want to set a deadline or grade the peer review, create a new assignment.
• Create a document with instructions on how to give and view peer feedback.
• Use screenshots.
• Test the instructions from a student’s perspective.
• Benefits
• To students:
• Peers offered valuable insights
• Giving feedback to others focused them on assessment criteria, which
clarified how to improve their own work
• To Faculty:
• Logistics far more streamlined and simplified (vs. manual/email)
• Easier to check completion
• Required fewer documents – rubric integrated
• Ability to use anonymous feedback

21. AAP 50 and Canvas

22. AP50
lectures group work exams projects
• Physics as a Foundation for Science &
Engineering
• new class in SEAS Sept. 2012
• 30 students first year, now 75 students
• undergraduate engineering students
• project and team based
• “Flipped classroom”
Applied Physics 50

24. Team & Project Based Learning

25. AP50
lectures group work exams projects
• 3 projects per semester
• 3 groupings per semester
• final report for each project
• teaching staff provides feedback on
reports via rubrics uploaded on Canvas
• teams resubmit reports (via Canvas) for
a final grade
Canvas for Groups and Reports

31. Our$challenge:$How$to$diffuse$our$ideas$and$
create$a$community$of$practice?
• Huge%demand%for%PDIA,%short%supply%of%faculty
• Change%agents%in%developing%countries%cannot%
afford%to%attend%an%executive%education%course
– Sustainable%change%requires%teaching%teams%who%
work%together
– There%is%a%spacing%effect%of%learning%in%one’s%own%
environment%
• Why%not%experiment%with%a%MOOC?

32. Our$answer:$Canvas$Catalog
• We%already%had:
– Blog:%International%following%(155%countries)
– 34%high%quality%3P5%min%videos%%(why?)
• Canvas%Catalog
– No%Harvard%ID%required%to%sign%up
– Easy%to%use
– Do%It%Yourself%(DIY)

33. What$did$we$do?
• Launched%our%first%MOOC%on%Nov.%8,%2015
– 6%weeks
– Weekly%assignments
• Written%answers%(no%quiz)
• 4%conceptual%and%2%openPended
– Weekly%reflection%exercises
– Average%of%3P5%hours%of%work
– Peer%review%and%rubric%tool%
• to%grade%and%to%learn%from%each%other

34. Example$of$assignment$rubric

35. 241$people$from$53$countries$completed$Part$I
Top$10$
Countries
# of
Students
United%States 34
Nigeria 31
Peru 21
Great%Britain 15
India 8
Kenya 7
Tanzania 7
Pakistan 7
Nepal 6
TimorPLeste 6

36. What$did$we$learn?

37. 79%$rated$the$Peer$reviews
good$or$higher

38. “The$peer$reviews$were$truly$amazing."A"huge"part"of"it"was"helpful"and"
usually"highlighted"areas"I"may"not"have"considered.”
“Very"useful"and"have"never$had$it$so$good$in$all$the$online$programs$i have"
undertaken"in"the"past”
“I"think"the"peer"review"was"a"very"good"opportunity"to"engage"with"like=
minded"persons"that"provided$competent$reflection$and$criticism.”
“Generally"good."In"fact"I"think"the"act"of"peer"reviewing"is"more"valuable"than"
the"act"of"being"peer"reviewed."Doing"the"reviewing"forces"me"to"think"
seriously"about"another"person's"take"on"the"question.”
“It"is$informative$to"read"others"reflections,"and"adds$to$the$community$feel$of"
the"course.”
Student$Feedback$on$Peer$Reviews

39. “A"little"disappointing."A"couple"offered"interesting"comments"that"were"worth"
thinking"about"but"others$were$delayed$or$not$completed.”
“Peer"review"assignments"were"very"useful"except"that"it"was"very"difficult"to"
find"course"colleagues"to"comment"on"work"consistently."It"looks"like"people"
were$very$busy$and$could$not$find$much$time$to$peer9review$work.”
“Sometimes,"a"reviewer"can"score"a"student"low,"because"of"his/her"limited$
understanding$of$the$concepts.”
“Some"reviewers"do"not"simply"understand"the"concept"or"the"problem,"hence"
they"provide"minimal"contribution."
Student$Feedback$(cont’d)

40. Valuable$feedback$and$peer$interaction

41. Generic$no$value$add

42. Limited$understanding$of$the$concepts

43. Limited$understanding$of$concept/language

44. Where$to$next?
• More%specific%questions/prompts%for%reviewer
– What%did%you%like?
– What%would%you%suggest%for%improvement?
– How%does%this%differ%from%what%you%learnt?
• Assignments
– Hired%graders%
• Flipped%classroom%Fall%2016
– Opportunity%to%test%peer%review%in%residential%
course%(calibrated%reviews)