Given at Waseda University Center for Higher Education Studies, July 31, 2019. Discusses the emerging annotation platforms, Perusall and Hypothesis, to promote deeper analysis of assigned reading

1. Increasing student engagement with
assigned reading: annotation,
crowdsourcing, and redaction
Gavin Porter PhD
Lecturer, Program in Immunology
Harvard Medical School
Boston, Massachusetts

2. Topics over the years….
• Ken Bain’s “What the
best teachers do” book
club
• Re-considering the
Lecture as a teaching
format
• Grading issues
• Carl Wieman Teaching
Practices Inventory
• Best/worst ways to start
the semester
• Best ways to finish the
semester (or class
period)

3. How my teaching led to today’s café topic
• Course 1: Research Methods in Immunology
• Course 2: Pedagogy course (“How to be a better
teacher”) for Harvard biomedical PhD students and
post-docs
• Course 3: Deconstructing Immunology presentations
– Linked to weekly seminar speaker
– Try to understand the invited speaker’s research papers
– Thus, very much a “paper-based” course – we need to
read and understand primary research articles……how to
best help students to do that???

4. Outline – Increasing engagement with reading
• Student reading habits
– What does the research say?
• Turning reading from a solitary to
a collective activity
– What is Perusall?
• What can it do/How do I get started?
• Experience sharing from my
Immunology MMSc paper analysis-
based course
• How does the annotation assessment
fit with good pedagogy practices?
• Other powers of the “crowd” and
redaction

5. Student reading habits – what does
the research say?

6. What % of students will claim to have done
required reading prior to a class?

7. The amount students claim to read, is
below what they actually read……
The Physics Teacher, 44(6), pp.338-342. 2006
• In a study of multiple
Physics courses
– 37% of students
regularly read the
textbook
– less than 13% read often
and before lecture
Journal of Instructional Psychology, 31(3), pp. 227-232. 2004
• In a study of multiple
Psychology courses
– Average of only 28% of
students did the
assigned reading before
it was covered in class

8. Can a student do well in your course
without doing the required reading?
• What will teachers say?
• What will students say?

9. • Only 4% of professors think that a student could
score an A or B without doing the assigned reading
for a course, while 34% of the students thought they
could do so…….
The importance that professors attach to reading
is much higher than the importance attached by
students……
Journal of College Teaching & Learning, 3(3), pp.51-56.

10. How does doing (or not doing…) the reading
beforehand affect tutorials/discussion sections?

11. How does doing (or not doing…) the reading
beforehand affect tutorials/discussion sections?
The Journal of Higher Education, 69(4), pp.384-405.
• Only 20% of students identified “not having done the
reading” as a reason to not participate in discussions
– Students can key off of each other’s discussions without
necessarily being well-informed via reading
– Teachers would rather have a less-than-fully-informed
discussion in tutorials compared to silence…….
– Tutorial-based discussion is not as strong of a motivator as
we’d like it to be……explore other strategies…….where to
turn???

12. Outline – Increasing engagement with reading
• Student reading habits
– What does the research say?
• Turning reading from a solitary to
a collective activity
– What is Perusall?
• What can it do/How do I get started?
• Experience sharing from my
Immunology MMSc paper analysis-
based course
• How does the annotation assessment
fit with good pedagogy practices?
• Other powers of the “crowd” and
redaction

13. What is Perusall?
• It’s an annotation platform for reading
• Turns reading from a solitary to a
collective activity
– With Perusall, students and teachers can:
• Pose and answer questions about assigned
reading
• Add links to other papers to help with student
understanding, and nurture further interest
• Post multimedia to help with explanations
• “Upvote” quality student annotations and
questions
• Get notified when somebody answers their
question
Notice the word play…..

16. Works with PDFs (free) or textbooks from the following publishers…….

17. What’s the comparator to the
Annotation platform?
• For my class it was a template that students
would need to complete and fill-in prior to the
class which discusses the paper
The ”before” condition

19. Trying to encourage
multiple readings by
the students
Prompting a succinct
summary
It’s not easy to
understand some of
the papers……
This is a high level
skill….goal for the
future
Need to
read >1
paper

20. Prompt for figure
analysis….

21. Student “X” templated assignment

22. Student “X”

23. Template-based approach
• Students are following the same recipe for
analysis each week
• I can make some comments and hand back –
this is mostly on an individual student basis
• Got a bit tedious by the end of the semester,
particularly for multi-panel figures……
– Was looking for a way to liven things up……

24. This paper from sem 2
had 20 sub-panels
A through T – quite
tedious to apply the template
approach to all 20 chunks…….
and this is just one figure out of
10

25. The basics……getting started with Perusall in 5
easy steps
• Interface is extremely intuitive – no problems for me at
all…..support documentation is great
1. Go to www.perusall.com click on Get started now
– You can sign in with Google, Twitter, or with your email

26. Perusall how to……5 easy steps
2. Create a new course

27. Perusall how to……5 easy steps
3. Add students and populate course with the
readings you want – you can upload your own
pdfs
My prompts to the students
in the syllabus….
The PDFs that I uploaded….

28. Perusall how to……5 easy steps
4. Look for annotations made by students by
clicking the conversations tab; you can “seed” some
conversation by making your own annotations

29. Perusall how to……5 easy steps
5. Add thought-provoking follow-up
annotations, encourage students, upvote
responses, answer questions (as you see fit……)
Blown up on
next slide….

30. Automatically gives a blank comment/question box below any student annotation
…..similar to social media
• Can add links and multimedia
• Links to relevant explanatory papers were
deemed really helpful by students
• Students also added papers that helped me as a
teacher!!!
• Can also add #hashtags
• Students are alerted via email when someone replies
to their post or answers their question
• Even has emojis…….

31. Student 1
Student 2
myself
You have the context for the annotation,
as you can see the highlighted part of
the paper (in purple) that the
annotations are referring to……….

32. Student 1
Student 2
myself
• Students can pose questions
• Other students can answer them
• Students can also post links to relevant
studies
• The linked paper was very helpful as
a follow-up
• Instructor can weigh in with more
nuance……

33. Can post diagrams if that is
helpful….not limited to text
only conversations…..

34. Questions are flagged with an orange
color……

35. Continued from previous thread…..
Instructor can see which threads
are getting the most attention

36. We had a “best hashtag” contest each
week……fun for students……

37. Grounding usage of the annotation platform
with good pedagogical practices…..
• Common Educational Technology refrain…….“it’s not just
the technology, it’s how it’s being used”
– 6 relevant pedagogy points derived via working with the
annotation platform
1. We know interleaving material, and retrieval practice is good for
retention……..the teacher can prompt this with annotations
2. Teacher cannot always predict what students will be confused
about, good to elicit this prior to the class session
3. If students answer each other’s questions, it can decrease teacher
burden (…..and empower students!!!)
4. “Mine” the collective class knowledge – students may think of
things in ways you didn’t and find resources that you didn’t…….
5. Students begin to appreciate something from a didactic lecture
more when they see its application in the primary literature
6. Students are proposing new experiments to each other……this is
exactly the kind of skill they need in the future --
#authenticassessment
• Can show the experiences in my class that highlight the
above points…….

38. 1. You can prompt students to recall
previous course content……..
And flesh out the details if necessary……

39. In this case, all the students were
confused about the use of a particular
word in the paper. I probably wouldn’t
have anticipated a problem
with this piece of vocabulary without
reading this annotation thread…..
Instructor can correct the misconception
2. Teachers cannot always predict beforehand what
students will be confused about…..

40. Related issue -- some student annotations prompted me to dig
into the biology in ways I wouldn’t have done otherwise…..
Correct the
student’s
misconcept
ion
Dig into the
biology

41. You can point students to further beneficial reading……
• I would never have made this type of reading recommendation
without seeing that particular student annotation!!!

42. 3. Students-helping-students can partially decrease
teacher burden and empower students
• Student explained a key reagent and provided link to a follow-up paper
• Other students (and the instructor….) can upvote this help
– (“socialized” function…..everyone enjoys getting ”likes” on their social media
posts)
Student posted a helpful explanation
Post got ”upvoted”
Helped a classmate

43. Students indicate
confusion points
and in some cases
resolve the
confusion before
the instructor
does……
Indicating
confusion.
Posing a question.
Another
student further
refines the
question…
Resolution
+ link to
helpful
diagram

44. 4. “Mine” the collective class
knowledge – students may think
of things in ways you didn’t and
find resources that you didn’t…….

45. Instances where the collective class
knowledge really helped!!!
• Students can weigh in publicly with
their experience
• They feel valued for sharing in a forum
that includes the whole class, makes
the data in the paper more “real”, as it
is linked to a clinical case that your
classmate already experienced and can
provide more information on!!!!
• Personal experience with cases
mentioned in the papers…..
• Antibiotic prescription practices…..
• Vaccine routes of administration
discussion…..
• All of this was a great supplement to
the instructor’s knowledge!!!

46. 5. Students begins to appreciate something from a didactic
lecture after reading about it in the primary literature…….
• Argues in favor of using primary sources in teaching…..

47. 6. Students are proposing new
experiments to each other……
This is exactly where we want to go in regards to
skills for future independent researchers!!!!!
#authenticassessment

48. Template vs. Annotation approaches
Templated
( more “traditional”) assessment
Annotation-based assessment
(Perusall)
Student output Uniform – tedious to read
“Chunked” – comes in at deadline
Unique -- exciting to read, draws from each
student’s distinctive background
“Flows” – real-time steady stream of output
Harnessing collective
class knowledge
Didn’t really afford this opportunity
– solitary turn-ins
Promoted this opportunity – reading the
article collectively; comments are shaped by
the student’s unique background
Common student
questions
Need to be flagged by the teacher
and remembered between reading
lengthy assignments
Can be flagged (upvoted) by students
themselves
Students-helping-
students
Students don’t see each other’s
output
Possible and common – the student
questions are there for everyone to see…..
Number of “turns of
consideration”
Probably only 1 – student brings up
a point, gets addressed in the
handed-in assignment…..whether
the student follows-up (reading,
diagram, etc.) is not known
More than 1. Longer threads, can see
questions answered and new questions
being posed……more amenable to a
“dialogic” style of feedback. Between the
teacher-student and the students
themselves
Instructor comments
on student text
The individual student benefits from
comments on their individual
assignment
Students can benefit from comments on
threads that are not their own, and can see
corrections of other student’s
misconceptions

49. The body of annotations made by students can
also serve as a basis for research
• MOOCs,LMSs – data explosion – any mouse
click by students is a potential data point

50. • Provide a basic scoring rubric that might
be helpful to instructors
The body of annotations made by students can
also serve as a basis for research

53. Perusall is not the only annotation platform, the research is
not only limited to data mining/quantitative approaches
Could also pique the interest of those
in media studies and the humanities……

54. A few open questions for annotation
platforms.….
• Does annotation content and overall usage vary
between two different annotation platforms
(Perusall and Hypothes.is) with the same group of
students and similar content?
• Is there a relationship between annotation timing
(with respect to deadline) and annotation quality?
• Student voice in annotation usage/best practices
for students and teachers?

56. Outline – Increasing engagement with reading
• Student reading habits
– What does the research say?
• Turning reading from a solitary to
a collective activity
– What is Perusall?
• What can it do/How do I get started?
• Experience sharing from my
Immunology MMSc paper analysis-
based course
• How does the annotation assessment
fit with good pedagogy practices?
• Other powers of the “crowd” and
redaction

58. Advantage of circulating throughout a lecture hall -- I
noticed that multiple students were doing this.........
Learning science (or business, or engineering, or
law…….anything!!!!) involves learning a lot of new
vocabulary

60. Power of student crowdsourcing – evolving “corpus” of science
vocabulary

61. Redaction
• I have used this many times in abstract writing
assignments
• Give the paper (redacted of any identifying
information, redacted of the abstract)
• Get the students to write the abstract based
on the rest of the paper
For example……

62. Original Redacted – abstract and any identifying info

63. Redaction could be applied to many parts of the
paper, not just the abstract……
• Scientific research articles follow the IMRD structure
– Introduction
– Methods
– Results
– Discussion
• Can redact the discussion section, get the students to
generate the discussion section through through their own
deliberation
• Then unblind the discussion section and compare the
student products versus the author’s product
– Students want to know how their analysis compares with the
authors of the paper; this is the direction they want to go in
(novice to expert…..) in the future
– Essence of good feedback:
• Students need a target
• Need to know how different they are from the target

64. • 3 conditions for effective feedback, learners
need to:
– Possess a concept of the standard being aimed for
– Be able to compare their current level of
performance with the standard
– Engage in appropriate action which leads to some
closure of the gap between the two
Considered a seminal paper in assessment

65. “Take-home” message
• Reading compliance might be less than you
think among your students, please consider
ways to increase reading engagement in your
future courses
• Annotation platforms
– Turn the solitary into the collective……
• Other crowdsourced activities…..
– Can also help with language learning for your students
– Also turns the solitary into the collective……
• Redaction of key information – can the students
reproduce what the experts have written?

66. For further consideration…….
• Link to PDFs of all the papers mentioned
• Perusall
–www.perusall.com
–tps://gking.harvard.edu/files/gking/files/p
h.pdf
• Hypothes.is for Educators
–https://web.hypothes.is/education/

67. Network
• Harvard
– Curriculum Fellows
• Emory
– CFDE
– TATTO
• PhD teacher training
• HKU
– CETL
– TeLi
I’m happy to make connections back to any of these for Waseda University’s benefit…….

68. Thank you for your attention
• Happy to have some discussion…….