Slides from Jeff Loats' plenary talk on Just in Time Teaching at the Lilly West 2012 conference.

1. Just in Time Teaching
A 21st Century Brain-Based Technique
Jeff Loats
Metropolitan State College of Denver Department of Physics
Lilly West 2012
“Learning technologies should be designed to
increase, and not to reduce, the amount of personal
contact between students and faculty on intellectual
issues.”
Study Group on the Conditions of Excellence
in American Higher Education, 1984

2. Big Ideas in Brain-Based Learning
2
Many myths out there… what actually works?
From Linda Nilson, we have four categories where we
have useful, evidence-based information:
• Focus and attention
• Using emotions appropriately
• Repetition and practice
• Feedback
As in all things, evidence rules.

3. Big Ideas in Brain-Based Learning
3
• Focus and attention
–No such thing as multitasking, etc.
• Using emotions appropriately
–A little anxiety is good, a bit more is bad, etc.
• Repetition and practice
• Feedback
Just in Time Teaching plays a role here

4. Feedback That Works
4
“Improvement of performance is actually a function of
two perceptual processes. The individual’s perception
of the standards of performance, and her/his
perception of his/her own performance.”
The Feedback Fallacy – Steve Falkenberg
(via Linda Nilson)

5. Workshop Overview 5
1. The motivation for change
2. Just in Time Teaching and Brain-Based Learning
3. Try it as students and then as instructors
4. Evidence for effectiveness, feedback from students
5. Roadblocks and bottlenecks
6. My summary

6. 6
The Physics Education Revolution
• About ~20 years ago, physics teachers began
treating education as a research topic!
• Their findings were grim:
From Mazur, New Faculty
Workshop presentation, 2004
• Eric Mazur
(at Harvard!)
found this:
"But the students do
fine on my exams!"

7. How Do Others Do?
7
Conclusion: Traditional physics lectures are all similarly
(in)effective for improving conceptual understanding.

8. 8
Enter Physics Education Research (PER):
An effort to find empirically
tested ways to improve the situation.
Many top university physics departments now
have a PER research group.
University of Washington University of Colorado

9. 9
Technique & Technology
Technique:
Just in Time Teaching (JiTT)
Technology:
Web based question & response tools

10. Just in Time Teaching
10
“Just-in-Time Teaching (JiTT for short) is a teaching
and learning strategy based on the interaction between
web-based study assignments and an active learner
classroom.” ~ Gregor Novak, Co-Author of the JiTT book
• Online assignments (“WarmUps”), completed
before class to promote preparation and thought.
• Responses are read “just in time”.
• Instructor modifies that day’s plan accordingly.
• Aggregate and individual (anonymous) responses
are displayed in class.

11. What JiTT is Not
11
JiTT techniques rely heavily on web-based tools.
JiTT is not about
… online courses or distance learning.
… computer-graded homework.
… delivering content via the web.
The goals of JiTT:
• Student preparation.
• Obvious communication loop.
• Improve student ownership and buy-in of class.
• Establish a community effort towards learning.

12. Questions – Practice & Repetition 12
• WarmUp questions about (new) material:
– Every-day language.
– Some simple comprehension questions .
– Mostly higher level questions (a la Bloom).
– Perhaps any question is better than none.
• Brain-based connections:
– Pre-class work reduces the working memory load
during class, a factor experts easily neglect.
– Multimodal practice (not learning styles):
Reading, writing and discussion are modes of
practice that JiTT brings to the classroom daily.

13. Questions – Deliberate Practice 13
• “What aspect of the material did you find the most
difficult or interesting. Give this a bit of thought and
be specific in your answer.”
• “How much time did you spend on the pre-class
work for tomorrow?”
[Results: A pretty steady average of ~40 minutes
across many courses/levels/cohorts.]
• Brain-based connection:
– Forced practice at metacognition:
Students regularly evaluate their own interaction
with the material.

14. Closing The Loop 14
• Student responses are:
– Graded on thoughtful effort that demonstrates
they did the reading.
– Sampled and categorized by the instructor in
some way to create a class response profile.
– Quoted anonymously in class to demonstrate a
variety of aspects: Misconceptions, good
efforts, superior explanations, metacognition, etc.
• Closing the loop:
– Instructor responds to some students digitally.
– Class time emphasizes interactive engagement.

15. Effective Feedback 15
Brain-based connection (from Falkenberg):
– Feedback doesn’t work if students don’t correctly
perceive the performance standards.
– Feedback doesn’t work if students cannot
correctly evaluate their own performance.
• JiTT offers chances to clarify standards in low-
stakes situations. Allows us to show model
responses that we did not generate.
• Closing the loop allows students to judge whether
they have correctly evaluated their own
performance.

16. JiTT Web Tools
16
• JiTT can be done using many different online tools:
– Course management systems (Blackboard).
– Free service from the JiTT Digital Library.
– Email (hard to manage).
– Blogging tools (WordPress).
Best features to have:
• All student responses on one webpage
• Auto-grading: 2/2 for anything by default.
• Click to email students from the response page.
• “Frequently sent responses” somewhat automated.

17. Mock WarmUp Question
17
Students have developed a robot dog
and a robot cat, both of which can
run at 8 mph and walk at 4 mph.
A the end of the term, there is a race!
The robot cat is programmed to run
for exactly half of its racing time.
The robot dog is programmed to run for
exactly half the racing distance.
Which one wins the race? Explain your reasoning.

18. Some Brainstorming
18
• Imagine an introductory course in your discipline.
• Imagine a topic you discuss early in that course.
• On the white card, write down one question, of either
of the two following types:
– A “low level” question (remember, understand):
Terms: “Define, repeat” or “describe, explain”
– A “higher level” question (apply, analyze, evaluate)
Terms: “Sketch, use” or “compare, estimate”
• Take 4-5 minutes, then I will ask you to trade your
question with a neighbor. Answer each other’s
questions as if you were a student who didn’t read.

19. Generating Questions
19
• Higher-level questions (3rd or 4th on Bloom’s
Taxonomy) seem like better JiTT questions.
• But class discussion is stimulated by both types.

20. Does It Work?
20
Mazur: After 1 year of using active engagement
methods:
• This is a big
jump in
conceptual
understanding.
• Is this just about
new energy being
put into an
old class?
(Always a tough confounding factor in assessing
new teaching techniques.)

21. Does It Work Outside Harvard?
21
• This method was
then used in classes
all over the country
with impressive
results all around.
• A conceptually
focused class with
responsibility placed
on students creates improved conceptual gains.
• The evidence indicates that problem solving skills
are improved when time is devoted to concepts.

22. Effectiveness of JiTT
22
• Back in 2004 JiTT was used by hundreds of
faculty, in more than 25 disciplines at over 100
institutions.
Yeah, sure… but does it work?
• Dozens of publications in different fields indicate
concrete differences in JiTT driven classes:
– Improved student preparation for class
– Improved use of out-of-class time and…
– Increased attendance & engagement in class
– Improvement in affective measures

23. Does It Work In Biology?
23
• Marrs, K., 2005,
Assessment of JiTT on Student Learning
In several biology courses the author documents:
Improved study habits & preparation before class
From “References for Just in Time Teaching”:
http://serc.carleton.edu/sp/pkal/justintime/references.html#pedagogy

24. Does It Work In Biology?
24
• Marrs, K., 2005,
Assessment of JiTT on Student Learning
‘Crammed’ in ‘Crammed’ in
Biology N100 other courses
A students 16% 44%
B students 34% 63%
C students 41% 65%
D students 64% 71%
F students 68% 69%
From “References for Just in Time Teaching”:
http://serc.carleton.edu/sp/pkal/justintime/references.html#pedagogy

25. Does It Work In Biology?
25
• Marrs, K., 2005,
Assessment of JiTT on Student Learning
In several biology courses the author documents:
Improved study habits & preparation before class
Increased retention rates (DFW from 29% to 21%)
Increased cognitive gains on conceptual knowledge
… AND increased content knowledge!
From “References for Just in Time Teaching”:
http://serc.carleton.edu/sp/pkal/justintime/references.html#pedagogy

26. Does It Work In Art History?
26
• Cookman, 2009
Using JiTT to Foster Active Learning
in a Humanities Course
This chapter (available online) describes Thinking
About Reading questions (TARs). His methods,
student reactions and lessons learned are all
included.
The course is History of 20th Century Photography.
Part of “Just in Time Teaching Across the Disciplines and Across the Academy” Book
Google this title.

27. Does It Work In Art History?
27
• Cookman, 2009
Using JiTT to Foster Active Learning
in a Humanities Course
Sample:
“Summarize in your own words Knightley’s
argument that Capa staged this photograph.
Summarize in your own words Whelan’s argument
that the photograph is authentic. Whose argument do
you find more convincing? Why?”
Part of “Just in Time Teaching Across the Disciplines and Across the Academy” Book
Google this title.

28. Does It Work In Sociology?
30
• Howard, J. R., 2004
Just-in-Time Teaching in Sociology or How I
Convinced My Students to Actually Read the
Assignment
Example question used to force synthesis (2 of 2):
What evidence suggests that perhaps we don't value our
children as much as we say we do?
“One quarter of the homeless population is children.
One third of college freshman [sic] are enrolled in
remedial classes. Seven million children are
assaulted each year. What evidence suggests that
perhaps we DO value children?”

29. Student Feedback
31
• From College Physics II, Fall 2010 (N = 75):
Agreed or
The WarmUps have… Strongly Agreed
…helped me to be more prepared for
class than I would otherwise be. 80%
…helped me to be more engaged in
class than I would otherwise be. 75%
…helped me to learn the material better
than I otherwise would 64%
…been worth the time they required to
complete 64%

30. Student Feedback
32
• From College Physics II, Fall 2010 (N = 75):
“These warmups helped me stay caught up with the
material because sometimes in other classes, since we
are not required to read, I end up cramming the
material at the last minute. So these helped to read a
little bit everyday.”
“I appreciate that the questions require thought. It is
beneficial to read the material and really have to use
the ideas, yet not be afraid to make mistakes or
incorrect assumptions”

31. Student Feedback
33
• From College Physics II, Fall 2010 (N = 75):
“Though I sometimes grumble and groan about
completing them, they have helped me stay on top of
the coursework. Additionally, my wrong answers have
been just as instructive in my learning as the right
ones. Introducing a problem, requiring my
thought, before coming to lecture, then covering the
answer, has given me a greater sense of relevance of
the material in several instances.”

32. Integrated “Lecture” Cycle
5-10
• Discussion of WarmUp minutes
Overall results each
Student responses with discussion
Questions/extensions from anyone
• Traditional “lecture” on topic
(motivated by WarmUp)
• Learner-centered pedagogy
(I use Peer Instruction with clickers)

33. 35
What Might Stop You?
In terms of the technique?
In terms of the technology?

34. What Might Stop You: Techniques
36
• The time investment for instructor is large the first
time, falling to an extra ~30 minutes after that.
• As with nearly any learner-centered technique, the
amount of material “covered” will likely shrink.
– To “cover” more or have the students learn
more… not a questions I can answer for you.
• Your actions must reflect the value you place on
their efforts on the WarmUps. Do your part.
• JiTT (used well) places more responsibility for
learning on the students. Students may balk.

35. What Might Stop You: Technology
37
• All technology has both a learning curve and flaws.
– Practice with your technology beforehand.
– Give yourself “outs” so that if it goes badly it
doesn’t cause a stampede.
– Find a community of instructors to help you
through rough patches (Local? Online?).
• Visit a class that uses these techniques.
• Don’t reinvent the wheel!
• Don’t do it all at once. (“10% rule”, or 5 min a day)

36. My Summary
38
• In terms of content and changes to the
classroom, Just in Time Teaching may be among the
easiest research-based instructional strategies that
you can consistently integrate into your teaching.
• From a brain-based learning perspective, JiTT
addresses areas that are often neglected by
traditional techniques.
• Student report that WarmUps make them better
prepared for class and help them learn the material.
• As with all reforms, we should be prepared to find
that students know less than we might hope.

37. Works Cited
• Falkenberg, S. (1996). “The Fedback Fallacy”. Retrieved March, 2012
from http://people.eku.edu/falkenbergs/feedback.htm
• Nilson, L. (20011) ”The Mind Has a Mind of Its Own”, teleseminar
recorded by Emphasis on Excellence, Jun 2011
• Mazur, E. 2004 ”Introduction to Peer Instruction” talk presented at
New Physics & Astronomy Faculty Workshop, 2004, UMD.
• Hake, R.R. 1998a. “Interactive-engagement vs traditional methods: A
six thousand-student survey of mechanics test data for introductory
physics courses,” Am. J. Phys. 66(1): 64-74; online at
<http://www.physics.indiana.edu/~sdi/ajpv3i.pdf> (84 kB).
Jeff Loats
jloats1@mscd.edu
303-900-2175
(please take a card)