A 2+ hour in-depth workshop about Just-in-Time Teaching (with some scholarly-teaching ideas peppered in).

1. Name
School
Department
JUST IN TIME TEACHING:
AN IN-DEPTH WORKSHOP
@ TLTS PRE-CONFERENCE WORKSHOP
DR. JEFF LOATS
DEPARTMENT OF PHYSICS
MSU DENVER

2. WARM-UP: TEACHING HERITAGE
Thinking about the college instructors you've had
experiences with (including yourself), where do you
think their methods and attitudes come from? Why
do you think they teach the way that they do?
~70% → “We Teach The Way We Were Taught”
~30% → Teacher personality/comfort/style
~20% → Training
~20% → Experiences while teaching
~10% → Research or scholarship
(rough estimates compiled over time)

3. WARM-UP: TEACHING HERITAGE
Snippets:
“Their methods and attitudes come from the way
they were taught.”
“the classrooms are conducive to lecture”
“the students are expecting a regular style course
and are trained in such.”

4. WARM-UP: TEACHING HERITAGE
“I think it certainly has to do with how they were
taught. Or once they think they have a good
strategy they stick with it forever without taking
in to consideration how the students learn and
change with time. Or worse yet they are just not
open to continually improving teaching.”
“Unless you are invested in the Scholarship of
Teaching and Learning, you have nothing else to
go on other than the experiences you had in the
classes you took as a graduate student. ”

5. ASIDE: LEARNING STYLES
“I think that many teachers teach in a way that
makes sense to them, according to their learning
style […]”
Best current evidence: Learning styles don’t exist
References:
• “The Myth of Learning Styles”
by Cedar Riener and Daniel Willingham
• YouTube: Learning Styles Don’t Exist
• Scholarly review: “Learning styles: Concepts
and evidence”, Pashler et al, 2008

6. WARM-UP: TEACHING HERITAGE
Thinking about the college instructors you've had
experiences with (including yourself), where do you
think their methods and attitudes come from? Why
do you think they teach the way that they do?
~70% → “We Teach The Way We Were Taught”
~30% → Teacher personality/comfort/style
~20% → Training
~20% → Experiences while teaching
~10% → Research/evidence
(rough estimates compiled over time)

7. THE EVIDENCE STANDARD
Teachers can feel bombarded…
I strive to be a scholarly teacher …
• Apply the rigor we bring to our academic
disciplines to the discipline of teaching.
• Choose teaching methods that are strongly
informed by the best empirical evidence
available.
Contrast teaching your subject with treating a
medical condition…

8. In your teaching do you have a method for holding
students accountable for preparing for class?
Previous anonymous poll results (compiled):
A) I don’t, but I ask/threaten really well
B) I use a paper method (quiz, journal…)
C) I use a digital method (clickers, etc.)
D) I use Just-in-Time Teaching
E) I have some other method
17%
51%
11%
5%
17%
(𝑁~230)

9. OVERVIEW
1. Motivation for change
2. Basics of Just-in-Time Teaching
3. Mock example
4. Choose our own adventure
5. Summaries

10. PHYSICS EDUCATION REVOLUTION
Eric Mazur, Physicist at Harvard:

11. “ALL SIMILARLY (IN)EFFECTIVE…”

12. University of
Washington
CU Boulder
University of Illinois
at Urbana-Champaign

13. TECHNIQUE & TECHNOLOGY
Technique:
Just-in-Time Teaching
Technology:
Online question & response tools
Learne
r
Teacher

14. JUST-IN-TIME TEACHING
Online pre-class assignments
called WarmUps
First half - Students
• Conceptual questions, answered in sentences
• Graded on thoughtful effort
Second half - Instructor
• Responses are read “just in time”
• Instructor modifies that day’s plan accordingly.
• Aggregate and individual (anonymous) responses
are displayed in class.
Learne
r
Teacher

15. Students have developed a robot dog
and a robot cat, both of which can
run at 8 mph and walk at 4 mph.
A the end of the term, there is a race!
The robot cat must run for half of its
racing time, then walk.
The robot dog must run for half the
race distance, then walk.
A) The cat wins B) The dog wins C) They tie
15

16. WARM-UP: ROBODOG VS.
ROBOCAT
Predict which one will win the race, and explain
why you think so.
Previous:
~33% → Robocat!
~58% → Robodog!
~0% → They tie!
~8% → Can’t tell!
Alternate view:
~8% → Good math
~8% → Bad math
~33% → Good reasoning
~33% → Bad reasoning
~8% → Invalid arguments
~8% → No reasoning 

17. WARM-UP: ROBODOG VS.
ROBOCAT
Invalid arguments:
“I'm going to say the cat simply because cats are
better than dogs.”
“[…] And............because everyone knows cats
can outrun dogs. :)”

18. WARM-UP: ROBODOG VS.
ROBOCAT
“The cat will win because running for half the
time can cover more than half the race distance
the dog will run.”
“The cat will win. It covers more distance
running than walking while the dog covers
exactly half the distance running.”

19. Consider a typical day in your class. What fraction
of students did their preparatory work before
coming to class?
Previous anonymous poll results (compiled):
A) 0% - 20%
B) 20% - 40%
C) 40% - 60%
D) 60% - 80%
E) 80% - 100%
19
29%
32%
20%
14%
5%
(𝑁~238)

20. STUDENT PREPARATION
SCHOLARSHIP
Quotes from Sappington, Kinsey, & Munsayac (2002)
“72% of Connor-Greene’s (2000) sample
reported that they rarely or never read their
assignments by the due date.”
“Burchfield and Sappington (2000):
On any given day, less than a third of
students in this population had
adequately prepared for class.”
USPIRG survey: 70% of students
admit that they sometimes don’t
even obtain required textbooks.

21. JITT STRUCTURE & RESPONSE
RATES
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
%Responsed
Class #
Response Rate by Day
College Physics I, N = 78
Worth 10% of final grade
Due 10 PM the night before class
Assignments available for prior 2-3 days
College Physics I

22. WARMUP QUESTIONS
• Every-day language
• Occasional simple comprehension question
• Mostly “higher level” questions
• Any question is better than none (don’t be precious)
Connections to evidence:
–Pre-class work reduces working memory load
during class.
–Multimodal practice (not learning styles):
JiTT brings reading, writing and discussion as
modes of practice.

23. METACOGNITION
Two questions in every WarmUp:
First:“What aspect of the material did you find
the most difficult or interesting.”
Last: “How much time did you spend on the pre-
class work for tomorrow?”
Connections to evidence:
–Practicing metacognition is beneficial:
Students regularly evaluate their own
interaction with the material.

24. JUST-IN-TIME TEACHING
A different student role:
• Actively prepare for class
(not just reading/watching)
• Actively engage in class
• Compare your progress & plan accordingly
A different instructor role:
• Actively prepare for class with you
(not just going over last year’s notes )
• Modify class accordingly
• Create interactive engagement opportunities
Learne
r
Teacher

25. MAZUR AFTER 1 YEAR

26. ELSEWHERE?

27. Writing Good Questions

28. FEATURES OF A GOOD QUESTION
28
What would a “good” response look like?
– A paragraph? (too long)
– One word? (too short)
Make sure the reading is needed to respond (but a
sentence straight out of the book shouldn’t work).
Make sure a beginner can take a crack at the question
Be concrete:
– “Explain in 2-3 sentences.”
– “Give two brief examples.”
– “Explain how you got your estimate.”
“Game out” their responses a bit.

29. WRITE A QUESTION AND SHARE...
29
Consider an intro. course in your discipline.
Consider a topic you discuss early in that course.
Write one question… shoot for “higher level.”
Good keywords: apply, analyze, evaluate, sketch,
use, compare, estimate, etc.
Take 3 minutes, write it at the top of a sheet…
Trade papers with a neighbor and answer their
question (as best you can) on the back of their
page.

30. Break!

31. BREAK TIME!
31
Chew on these…
Are you best lecturer in the world on the
topics you teach?
In the 21st-century, how should students
spend their 15 hours per credit with
you?

32. WARM-UP: BIGGEST “TAKE AWAY”
Deslauriers, et al., 2011: “Improved Learning in a
Large-Enrollment Physics Class” (3 pages)
Controlled comparison of learning achieved
using two instructional approaches.
3 hours of traditional lecture given by an
experienced highly rated instructor.
VS.
3 hours of instruction given by a trained but
inexperienced instructor using research-based
best-practices.

33. WARM-UP: BIGGEST “TAKE AWAY”
What was the biggest “take away” idea that you
got from the article?
~50% → Active engagement shows impressive
learning outcomes compared to
lecture.
~30% → Changes were well-received by
students
~10% → Wondered why we still use poor
methods?

34. WARM-UP: BIGGEST “TAKE AWAY”
“There is a need to avoid lecture-based
approaches to instruction. In order to engage
students, strategies we use should be based on
current research.”
“ ‘A typical study in the domain of physics
demonstrates how student learning is improved
from one year to the next when an instructor
changes his or her approach,...’
That speaks loudly to me.”

35. WARM-UP: BIGGEST “TAKE AWAY”
“I don't need to be afraid of trying new teaching
strategies based on research in the Scholarship of
Teaching and Learning. I might assume that since I
don't already have a lot of experience with these
techniques, they are going to bomb in class due to
my inexperience and thus I'll be afraid to even try.
This article points out the "proven" strategies beat
out "experience in teaching". I can have confidence
in trying new things by relying on the Scholarship of
Teaching & Learning research even if I don't already
have a lot of experience. ”

36. COMBINED IMPACT
36
Deslauriers, et al. (2011):
“We found increased student attendance, higher
engagement, and more than twice the learning in the
section taught using research-based instruction.”
With standard deviations around 13%, this meant an
effect size of 2.5!
“[…] other science and engineering classroom
studies report effect sizes less than 1.0. An effect
size of 2, obtained with trained personal tutors, is
claimed to be the largest observed for any
educational intervention.”

37. Consider:
“If I stopped using any “innovative” teaching
techniques, and just tried to do the best I could
with straight-up lecture, there would be no
concrete external consequences for me.”
A) Strongly disagree
B) Disagree
C) Neither agree nor disagree
D) Agree
E) Strongly agree
37

38. ACTIVE-LEARNING METASTUDY
In 2014, Freeman, et al. published a huge meta-
analysis in Proceedings of the National Academy
of Sciences. Narrowing over 600 studies down to
225 that met quality/methodology criteria.
Key findings were both large and consistent:
• Students in traditional-lecture courses were 50%
more likely to fail.
• Students in active-learning courses gained about
half a standard deviation on exams (~6%)

39. IS THIS A MORAL ISSUE?
From the paper: “If the experiments analyzed here had
been conducted as randomized controlled trials of
medical interventions, they may have been stopped for
benefit—meaning that enrolling patients in the control
condition might be discontinued because the treatment
being tested was clearly more beneficial.”
From Freeman: “The impact of these data should be
like the Surgeon General’s report on “Smoking and
Health” in 1964–they should put to rest any debate
about whether active learning is more effective than
lecturing.”

40. Pause for questions…

41. SHARE AGAIN
41
Trade papers again with a different person and
answer their question on the front of their page
(as best you can). Don’t peek at your peer’s
response, please.
Take a few minutes to consider the two responses
you got.
Talk with your neighbor about what you see in
the responses...

42. Which topic would you like to spend our
remaining time on?
A) Evidence for effectiveness
B) Best tools for JiTT
C) Getting student “buy-in”
42

43. Evidence For Effectiveness

44. STUDIED EFFECTIVENESS
Used at hundreds of institutions
Dozens of studies/articles, in many disciplines:
Bio, Art Hist., Econ., Math, Psych., Chem., etc.
–Increase in content knowledge
–Improved student preparation for class
–Improved use of out-of-class time
–Increased attendance & engagement in class
–Improvement in affective measures

45. JITT VS. FINAL GRADE
CORRELATIONS
College Physics I, Fall 2013
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90 100
CumulativeScore(withoutwarm-ups)
WarmUp Score
WarmUps vs. Cumulative Score
Correlation r = 0.71

46. PROGRESSIVE EXAMS
CORRELATIONS
College Physics I:
Important disclosure: This was not a hypothesis we were
testing, it appeared as we analyzed the data. Could be
0.18
0.33
0.43
0.54
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Mini Exam
(week 4)
Exam 1
(week 7)
Exam 2
(week 11)
Final Exam
(week 16)
NoneWeakStrongModerate
Correlations between Total WarmUp Score
and Sequence of Exams

47. Mean on 1-5 scale
Preparation for class 4.06
Engagement during
class 3.93
STUDENT SURVEY RESULTS
9% 10%
81%
10%
18%
73%
10%
22%
68%
0%
20%
40%
60%
80%
100%
Harmful Neutral Helpful
How did WarmUps affect your...
Preparation Engagement Learning
N = 781

48. STUDENT SURVEY QUOTES
Physics:
“Initially, it was hard for me to get used to the
warm-ups. It seemed like along with the
homework assignments there was a lot of things
to do. Eventually I got used to it and ultimately
the warmups really helped me to learn the
material and stay caught up with the class.”
“If it weren't for warm ups, the amount of time I
spent reading the book would have dropped by
75%”

49. Best Tools for JiTT

50. WHAT TOOLS TO USE?
The crucial part:
Daily reading, grading & using responses
• Automatic full credit for any response
• View all responses to a question together
• Grade responses on the same page with
minimal clicks
Wishlist:
Easy (quick!) individual feedback

51. SMALL ASIDE: TEXT EXPANDER
51
Every professor should have this!
You define a snippet like “ttyl” which instantly
gets replaced by “Talk to you later!”
Windows:
– Texter, PhraseExpres
(FREE, some advanced features, some flaws)
– Breevey ($40, worth it if you hit problems)
– AutoHotKey (free advanced automation tool)
Mac:
– TypeIt4Me, TextExpander, Typinator
(All cost $20-$30. Generally worth it!)

52. WHAT TOOLS TO USE?
• CMS/LMS (Blackboard, D2L, Moodle, etc.)
Ready to use, tools… imperfect  awful
• Free service from JiTTDL.org.
Designed just for JiTT. Additional website, not
very “shiny” by 2015 standards.
• Students email responses
Easy… also overwhelming and awful
• Blogging tools (WordPress)?
• New tools (TopHat? Learning Catalytics?)

53. Getting Student Buy-In
(a.k.a. “the sales pitch”)

54. OVERARCHING MESSAGE
Communicating with your students (humans)
• Message (explicit statements)
• Attitude (subtext, body language, etc.)
Consistent subtext:
"I am here to help you learn, and I have thought
about your learning trajectory carefully."
Consistent attitude:
I am comfortable and relaxed about my part of
this partnership.

55. THE SALES PITCH
13,000 hours in invisible contract indoctrination
Mindfulness in what we say and what we do.
Day 1 – Keep justifications short. Emphasize
purpose over mechanics.
Day 2 – Discuss their first experience & response
rates. Remind them about structure & purpose,
but mostly show them.
Day 3 – Return to “different roles” for both.
Demonstrate value, be consistent

56. STUDENTS: BUSY-WORK
DETECTORS
K-12 represents more than 13,000 hours of class
Students are experts at detecting what really
matters to an instructor:
• What does the instructor do with class time?
• What does the instructor talk about?
• Does the instructor push against the usual
“invisible contract” of the classroom?

57. DEMONSTRATE VALUE & BE
CONSISTENT
Demonstrating that you value JiTT
• Thank them for giving you insight
• Bring at least one “difficult/interesting” item from
WarmUp to class each day.
• Give non-verbal cues that you value discussing
WarmUps as much as other course components.
Be consistent with:
• Assignment releases and due dates/times
• Follow-up in class
• Summative assessments (e.g., exam questions) that
build on WarmUp questions.

58. Summary Slides

59. WHAT MIGHT STOP YOU?
In terms of the technique:
Time, coverage, not doing your part, pushback…
In terms of the technology:
Learning curve, tech. failures, perfectionism…
In any reform of your teaching:
Reinventing, no support, too much at once…

60. A POSSIBLE PLAN
Choose one course you will teach next term.
A. Write two questions for each class meeting:
1. One lower-level (maybe multi-choice?).
One higher-level (sentences).
2. Give yourself 10 minutes to write each one
B. Write a standard (1st) metacognitive question
C. Discuss one question at the top of class, and
one in the middle. Use the metacognitive
responses as break points or highlights.

61. MY SUMMARY
JiTT is an easy, research-based technique that
you can consistently integrate into your teaching.
From an evidence-based perspective, JiTT
addresses often-neglected areas.

62. YOUR SUMMARY
If you want to implement JiTT, what is your next
concrete action?
MSU Denver: Join the JiTT FLC next term
Tomorrow at 9 AM, panel presentation on JiTT
Email: jeff.loats@gmail.com
Twitter: @JeffLoats
Slides: www.slideshare.net/JeffLoats

63. On-Demand Slides

64. JITT REFERENCES & RESOURCES
Simkins, Scott and Maier, Mark (Eds.) (2010) Just in Time Teaching: Across the Disciplines, Across the Academy,
Stylus Publishing.
Gregor M. Novak, Andrew Gavrin, Wolfgang Christian, Evelyn Patterson (1999) Just-in-Time Teaching: Blending
Active Learning with Web Technology. Prentice Hall. Upper Saddle River NJ.
K. A. Marrs, and G. Novak. (2004). Just-in-Time Teaching in Biology: Creating an Active Learner Classroom Using
the Internet. Cell Biology Education, v. 3, p. 49-61.
Jay R. Howard (2004). Just-in-Time Teaching in Sociology or How I Convinced My Students to Actually Read the
Assignment. Teaching Sociology, Vol. 32 (No. 4 ). pp. 385-390. Published by: American Sociological Association
Stable URL: http://www.jstor.org/stable/3649666
S. Linneman, T. Plake (2006). Searching for the Difference: A Controlled Test of Just-in-Time Teaching for Large-
Enrollment Introductory Geology Courses. Journal of Geoscience Education, Vol. 54 (No. 1)
Stable URL:http://www.nagt.org/nagt/jge/abstracts/jan06.html#v54p18
Sappington J, Kinsey K and Munsayac K (2002) Two studies of reading compliance among college students.
Teaching of Psychology 29(4): 272–274.
Louis Deslauriers, Ellen Schelew and Carl Wieman (2011). Improved Learning in a Large-Enrollment Physics Class.
Science, Vol. 332 no. 6031 pp. 862-864 DOI: 10.1126/science.1201783