Slides from Bioscience Pedagogic Research Journal Club meeting at the University of Leicester, UK. The meeting discussed "Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom" a study by Louis Deslauriers and colleagues at Harvard University.

1. PedR Journal Club (November 2019)
Discussion of:
Measuring actual learning versus feeling of learning in
response to being actively engaged in the classroom
Deslauriers et al. (2019) PNAS 116:19251-12957
Led by Dr Chris Willmott
Molecular & Cell Biology Dept
University of Leicester

2. Initial 5W1H analysis
Where?
Who?
What (overview)?
When?
Why?
How (detail)?

3. Initial 5W1H analysis
Where?
- Single-centre study @ Harvard University
- Participants on Physics courses

4. Initial 5W1H analysis
Who?
- Lead author = Louis Deslauriers
Director, Science Teaching & Learning
Faculty of Arts and Sciences
(previously Uni of British Columbia)
- Kelly Miller = frequent co-author with
Eric Mazur (esp Peer Instruction)

5. Initial 5W1H analysis
What (overview)?
“We compared students’ self-reported perception of
learning with their actual learning under controlled
conditions in large enrollment introductory college
physics courses” (from Abstract, p19251)
“Identical class content and handouts” delivered via:
(1) Active instruction (best practice) or
(2) Passive instruction (by experienced staff)

6. Initial 5W1H analysis
When?
- Published September 2019
- Submitted December 2018
- Study repeated in Spring and Fall semesters,
but no year identified

7. Initial 5W1H analysis
Why?
“Despite active learning being recognized as a
superior method of instruction in the classroom, a
major recent survey* found that most college STEM
instructors still choose traditional teaching methods.”
(Significance, p19251)
* Stains et al. [n=31] (2018)
Anatomy of STEM teaching in North American universities
Science 359:1468-1470

8. Initial 5W1H analysis
How (detail)?
- Cross-over study
- Active v Passive teaching of same content
- Run twice (total 149 participants)
- Follow-up 1-2-1 interviews (n = 17)

9. “Calculus” module
15 weeks, 2 x 90 min sessions per week
Weeks 1-11
“Interactive lecture”
FCI scores
CLASS scores
Force Concept Inventory (FCI)
Basic knowledge about mechanics
Colorado Learning Attitudes about
Science Survey (CLASS)
Measures alignment to expert
thinking about physics

10. “Calculus” module
15 weeks, 2 x 90 min sessions per week
ACTIVE
Weeks 1-11
“Interactive lecture”
PASSIVE
Topic 1
STATICS
FCI scores
CLASS scores
PASSIVE
Strictly didactic, fluent delivery
prioritises, no group activities
Students fill gaps in handouts
as instructor explains
ACTIVE
Students work together to
attempt problems prior to
solution being revealed

11. “Calculus” module
15 weeks, 2 x 90 min sessions per week
ACTIVE
Weeks 1-11
“Interactive lecture”
PASSIVE
Topic 1
STATICS
FCI scores
CLASS scores
PASSIVE
Strictly didactic, fluent delivery
prioritises, no group activities
Students fill gaps in handouts
as instructor explains
ACTIVE
Students work together to
attempt problems prior to
solution being revealed
“The crucial difference
between the 2 groups was
whether students were told
directly how to solve each
problem or were asked to
try to solve the problems
themselves in small groups
before being given the
solution.” (p19252)

12. “Calculus” module
15 weeks, 2 x 90 min sessions per week
ACTIVE
Weeks 1-11
“Interactive lecture”
PASSIVE
Topic 1
STATICS
FCI scores
CLASS scores
FOL
then TOL
Feeling of Learning (FOL)
Self-reflection on learning
5-point Likert scale
Test of Learning (TOL)
12 Multiple Choice Questions

13. “Calculus” module
15 weeks, 2 x 90 min sessions per week
ACTIVE
Weeks 1-11
“Interactive lecture”
Weeks 13-15
“Interactive lecture”
ACTIVEPASSIVE
Topic 1
STATICS
Topic 2
FLUIDS
PASSIVE
FCI scores
CLASS scores
FOL
then TOL
FOL
then TOL

14. Controls (1)
1) Both instructors had training in active learning - identical,
informed by best practices
2) Both instructors had comparable experience in delivering
fluent, traditional lectures
3) Lecture slides, handouts, and written feedback provided
during each class identical for both groups
4) Students randomly assigned to two groups; groups
indistinguishable by metrics used
5) Cross-over study: each student experienced both types of
instruction

15. Controls (2)
6) Students had no exposure to either of new instructors
before intervention
7) Protocol repeated in two different cohorts, with same
results (n=149, total)
8) Instructors did not see the TOLs, prepared by different
author
9) Author of TOLs did not have access to course materials or
lecture slides; wrote tests based only on detailed learning
objectives

16. Key findings
TOL TOLFeeling of Learning Feeling of Learning

17. Follow-up interviews
- n=17, drawn from both Spring & Fall cohorts
- “Representative” sample re FCI, CLASS, grades
- Structured, one-to-one
- Instruction in active classroom disjointed (n=15)
- Frequent interruptions (n=14)
- Concerns errors would not be corrected (n=10)
- General feeling of frustration and confusion (n=14)
[no discussion of interview methodology, coding, etc]

18. Follow-up interviews
- Correlation “the instructor was effective at teaching”
and “I feel like I learned a great deal from this
lecture”, more than half an SD greater than those
who felt instructor not effective
- Interviews probed interpretation of the teaching
efficacy question:
1. Clarity of explanation
2. Organisation of presentation
3. Smooth flow of instruction

19. Key findings
- Cognitive fluency of good lecturers can mislead
students into believing they have learned more than
they actually have
- Novices in a subject are ill-equipped to judge how
much they have learned (Dunning-Kruger effect)
- Students unfamiliar with active learning methods
may misconstrue cognitive struggle as a bad thing,
unaware of its merits for aiding learning

20. And now for the odd bit…
- Not odd in the research, odd in the reporting
- “We carried out a semester-long intervention to see
if these attitudes could be changed…” (p 19255)
- Crucial part of the story, but minimal description in
formal paper (some in Supplementary Information)
re intervention or evaluation method (“a survey”)
- Some percentages offered, but no description of
research instrument

21. Changing attitudes
- “As the success of active learning crucially depends
on student motivation and engagement, it is of
paramount importance that students appreciate,
early in the semester, the benefits of struggling
with the material during active learning.” (p19255)
- “We recommend that instructors intervene early on
by explicitly presenting the value of increased
cognitive efforts associated with active learning.
Instructors should also give an examination (or
other assessment) as early as possible so students
can gauge their actual learning.” (p19256)

22. Changing attitudes
- “ The success of active learning will be greatly
enhanced if students accept that it leads to deeper
learning - and acknowledge that it may sometimes
feel like exactly the opposite is true.” (p19256)

23. Implications (for us)?
- Limitations of FOL as measure of academic worth?
- Modes of teaching that are better for learning may
involve more effort on part of students and
therefore receive lower satisfaction scores
Caution re Module review surveys, NSS, etc
“…student evaluations of teaching should be used
with caution as they rely on students’ perceptions
of learning and could inadvertently favor inferior
passive teaching methods over research-based
active pedagogical approaches” (p19256)

24. Implications (for us)?
- Limitations of FOL as measure of academic worth?
- Modes of teaching that are better for learning may
involve more effort on part of students and
therefore receive lower satisfaction scores
Caution re Module review surveys, NSS, etc
- Priming of student expectations may be crucial to
successfully introducing active learning
- You CAN get PedR published in major science
journals (PNAS, Science)