Carl Wieman Science Education Initiative 2009-10 End of Year Event Overview Author: Carl Wieman

1. 2009-10 End of Year Event
Talks
Overview of CWSEI progress (lots of data!)– Carl Wieman
Improving Student Study Habits: results of interventions
Sara Harris & Louis Deslauriers
Interactive Engagement: examples from UBC classes (video)
Sarah Gilbert & department members
Poster session 11‐1:30 room 101
Details on everything being done and learned
Workshop & Discussion
1:30 – 3:00pm, room 101 – How to Most Effectively Measure the
Learning that Matters (workshop led by Carl Wieman)
g ( p y )
3:15 – 4:30pm, room 101 – Incorporating Writing in the Science
Curriculum; what and how? (discussion)

2. CWSEI “Trinity” for each course
What should What are
students students
1st: Learning goals. (what learn? learning?
should students be able to do?)
should students be able to do?)
2nd: Good assessment Which instructional
approaches
h
(validated tests) improve
student
learning?
3rd: Improved teaching methods
: Improved teaching methods
(research based, improve learning)
Materials, assessment tools, homework, notes …
Materials assessment tools homework notes
saved, reused, improved.
Making teaching more effective, and more rewarding f
M ki t hi ff ti d di for
faculty and students

3. Carl Wieman Science Education Initiative
Started 3 years ago ⇒ widespread impro ement in science
ears idespread improvement
education.
Departments at various scales and levels of maturity
Large scale mature-- Earth and Ocean Sciences
Large scale younger-- Physics and Astronomy
Computer Science
Math
Smaller scale programs -- Chemistry Statistics Life
Chemistry, Statistics,
Sciences
$2 M gift from David Cheriton for math and comp sci
comp. sci.

4. Today--focus on data
1. How many courses/faculty transformed?
2.
2 How much better is the learning?
a. learning
b. engagement
c. innovative problem solving
3. But does it stay learned? (retention)
4. Reaching all students. Turning low performers into high
5. Blizzard of data on improvement from across the
departments (appetizer for posters)

5. 1. How widespread is the change-- EOS, most mature,
full 3 year effort
effort.
24 courses transformed.
18 with formal CWSEI support
pp
6 with strong informal and moral support
~ 26 faculty involved
typical new things
•clearly articulated learning goals for students and
faculty
•pre-reading assignments & quizzes
•clicker questions and peer discussion
•worksheets & in-class group activities
•group exams
•team projects
team
•pre-post testing to measure learning, ...
much more active learning and feedback,

6. EOSC 111: Laboratory ‐ Completed ‐ All hands on,
Exploration of Planet Earth ‐ First yr Lab ‐ lots of group work,
Sara Harris
S i ‐ approx 100 per semester ‐ i di id l d
00 individual and group
quizzes
EOSC 112: The Fluid Earth: ‐ been through about 2 ‐ clickers
Atmosphere and Ocean years of EOS‐SEI ‐ online quizzes
Sara Harris, Roger
, g ‐ service course for anyone ‐ article readings, quizzed,
y g ,q ,
Francois, William Hsieh at UBC with feedback (rubrics)
‐ about 350/year (split
between 2 sections)
EOSC 114: Natural ‐ Completed June’08 Clickers
Disasters st year exploratory
‐ 1 year exploratory on‐line assignments
on line assignments
R. Stull and many others course
‐ over 1000 stu. per year
EOSC 210: Earth Science ‐ Completed Clicker Qs, in each lecture.
for Engineers
for Engineers ‐ Lecture and lab
Lecture and lab Activities and discussions
Activities and discussions
Erik Eberhardt, Uli Mayer, ‐ 230 each September in most lectures. Labs with
Stuart Sutherland group work and hands on
activities
EOSC 211: Computer ‐ Second teaching term In‐class worksheets, pair‐
Methods in the Earth, Sept. 2010 programming, name‐
Ocean and Atmospheric ‐ 2nd year programming sticks, pair and small group
Sciences course lecture/lab discussions, class
Richard Pawlowicz, ‐ 55 students enrolled last discussions
Catherine Johnson
Catherine Johnson term

7. EOSC 212: Topics in Earth ‐ Completed June ‘09 ‐ team‐based quizzes and
and Planetary Sciences ‐ 2nd year “science inclass activities &
M. Bostock, M. Jellinek thinking” course discussions
‐ 20 to 40 students per
p ‐ article reading and
g
year question posing workshop
style classes
‐ peer assessed
presentations & posters
EOSC 220: Introduction to Complete
EOSC 220: Introduction to ‐ Complete 3x5 cards used to answer
3x5 cards used to answer
Mineralogy ‐ mandatory intro. lab questions in class, in‐class
Mary Lou Bevier course for EOS students activities, class discussion,
‐ 120 students enrolled labs have group work and
group quizzes
EOSC 221: Petrology ‐ Completed Wake up exercises
Maya Kopylova ‐ Lecture and lab (integrating activities into
‐ 100 each January each lecture), some 3x5
cards, labs with group
work and hands on, some
work and hands on some
"authentic activity" labs
EOSC 223: Field ‐ Minor support summer lectures have regular
techniques 2009 activities and 3x5 cards to
Mary Lou Bevier ‐ Lectures and Field get feedback,
component Field activities
EOSC 252: Physics of ‐ First teach term ‐ lab exercises
geologic materials completed ‐ in‐class demonstrations
F. Herrmann ‐ 2nd yr “physics” course with worksheets
‐ 20 – 30 students each
20 30 students each ‐ aiming for interactive
aiming for interactive
year lecturing next yr.
etc. for 3 more pages

8. 2. But do these changes improve student outcomes?
(learning, engagement, ...)
Hard to tell in most courses because no pre-transform
data.
data
Data from example courses where similar transformations,
and good pre t
d d transform and post t
f d t transform data.
f d t
Louis Deslauriers and Ellen Schelew (physics)--- cleanest
comparison study of teaching methods ever done.
Will be landmark in science education research
(as soon as they write it up for publication)

9. new-- Louis Deslauriers (PD) and Ellen Schelew (grad std)
Perfect comparison of teaching methods: identical sections (260 each),
intro phys. 153, same material & time.
___I___________ _____II_________
Experienced highly rated Experienced highly rated
instructor
instructor-- trad. lecture & ~2 cl.
2 instructor trad.
instructor--trad. lecture & ~2 cl.
2
questions questions
same preparation
same attendance
same attendance wk 1-11
wk 1-11
same engagement
same midterm 1 & 2 grades
Wk 12-- competition
elect-mag waves elect-mag
elect mag waves
Louis and Ellen (inexper.) regular instructor
research based teaching intently prepared lecture
common exam on EM waves

10. transformed section
•pre class reading assignments with quizzes
•pre-class
•in-class small group activities
•clicker questions with student-student discussion
•targeted instructor feedback guided by observations
of student thinking

11. Results II. Trad I. Transformed.
1. Attendance pre 58% 58% (wk 10 & 11)
during 58 % 81%
2. Engagement pre 50% 50% (wk 10 & 11)
(back ½ room) during 50 % 85%
3. Learning (test) 41(1)% 74(1) %
above guess (23%)
g ( ) 18% 51%
S. D. = 13%
trad trans
trad. ⇒0.58 x 0.5 = 29% engaged
ad ⇒0 58 0 5 9% e gaged 18 51
for above average instructor
trans. ⇒0.81 x 0.85 = 69% engaged
other things practiced: scientific discourse, critiquing scientific
arguments, sense-making, collaboration.

12. But how did students feel about it?
“Q1 I really enjoyed the interactive teaching technique during
Q1.
the three lectures on E&M waves (Ch32).”
70 63
57
ents
60
umber of stude
50
40
30
20 12
10
Nu
2 0
0
strongly
Strongly Agree Neutral Disagree Strongly
agree disagree
agree
Q2 I feel I would have learned more if the whole phys153
course would have been taught in this highly interactive style.
80
70 67
mber of students
60
50
40 36
30
21
20
Num
8
10 2
0
Strongly Agree Neutral Disagree Strongly
agree disagree

13. Q6 I found the pre-reading to be very helpful to my learning:
70 66
60
Numb of students
s
50
40 35
28
30
ber 20
10
3 1
0
gy
Strongly Agree
g Neutral Disagree
g Strongly
gy
agree disagree
Q5 What contributed most to my learning during these three
lecture on E&M waves:
60
51
Number of students
50
39
40
30
22
r
20 13
8
10
0
clicker in-class
Trying to figure Trying to work instruc. pre-
The instructor The pre-reading The pre-reading pre-read
out the answer out the answers explanation to quiz
quest.
to clicker to the in-class explan.
activity l
the clicker reading
d quiz
questions activities to c.q. or in-
questions or
class activities
activ.

14. Q8 In class, the group discussions with my neighbors were very
helpful to my learning:
80
70
70
60
Numb os students
s
50
40
33
ber
30
22
20
10 7
0
0
Strongly agree Agree Neutral Disagree Strongly
disagree

15. What does such a class l k l k
h d h l look like?
See upcoming video clips session.
p g p

16. Measuring student (dis)engagement. Erin Lane
Watch random sample group (10-15 students). Check
against li t of di
i t list f disengagement b h i
t behaviors each 2 min.
h i
time (minutes)

17. What about advanced upper division courses?
Physics 408-- d
Ph i 408 advanced optics
d ti
Taught by same instructor for several years--
continually working t i
ti ll ki to improve.
He radically transformed this year.
Ended up covering same material in less time.
Midterm exam grades:
Pre transformation (lecture)
P t f ti (l t ) 56 +/ 3 1%
+/-3.1%
Post transformation 77 %
(Exams different, but three experts did blind rating of the
different
exams. All concluded post transformation exam more difficult)

18. What about learning to think more innovatively?
g y
Learning to solve challenging novel problems
Jared Taylor and George Spiegelman
“Invention activities”-- practice coming up with
Invention activities
mechanisms to solve a complex novel problem.
Analogous to mechanism in cell.
2008-9-- randomly chosen groups of 30, 8 hours of
invention activities.
This year, run in lecture with 300 students. 8 times
per term. (video clip)

19. Plausible mechanisms for biological process student never
encountered before
6.0
Average Number
5.0
olutions
4.0
mber of So
3.0
Num
2.0
1.0
0.0
Control Structured Inventions (Outside Inventions (During
Problems (tutorial) of Lecture) Lecture)

20. Average Time to First Solution Thread
14.0
12.0
10.0
n)
Time (min
8.0
6.0
4.0
2.0
0.0
Control
C t l SPSA (O t id of
(Outside f IA (O t id of
(Outside f IA (D i L t )
(During Lecture)
Lecture) Lecture)

21. 3. So research based teaching achieves much better
learning & much greater engagement.
Does it stay learned?
y
(retention)

22. 3. Mastery of quantum mechanics concepts-short
& long term Deslauriers & Wieman to be published
score on quantum mech. concept survey
q p y
100
90
88
%)
85
ncept Surve Score (%
80 interactive engagement/practice
70
68
ey
superb traditional 65
60
lecturer
50
Con
40
30
0 5 10 15 20
Retention interval (Months)

23. Deslauriers, Lane,
4. Bringing up the bottom of the distribution Harris, Wieman
“What do I do with the weakest students? Are they just
hopeless, or is there anything I can do to make a difference?”
a. To get such big improvements in average, have to impact
entire distribution
b. Data on how to transform lowest performing students
into medium and high.
Intervened with bottom 25% of students after midterm 1.
•Phys250 ( gp y program, high selective and demanding),
y (engphys p g , g g),
bottom 25% averaged +20% improvement on midterm 2!
•EOS climate science course. Very broad range of students.
• Averaged +30% improvement!

24. nonintervention
intervention
100
100
90
80
70
score
student exam scores
60
50
40
M1
30
MIDTER
20
M 2
10
0 %0 0 20 40 60 80 100 120 140 160 180
student number
student number
midterm 1 score What magic does this?
,& X, midterm 2 score Listen to next talk.
•~All UBC science students can b successful
All i t d t be f l
•A little help on how to learn goes a long way

25. Large scale survey (~ 600) and interviews on factors that
UBC science students perceive as affecting academic
performance
Ashley Welsh
An early finding
•Students overwhelmingly recognize they do not know how to
study effectively Is seen as major barrier to success but find
effectively. success,
little help in learning how to study.

26. masses of other data
will overwhelm you with blizzard of info
Go to posters to get details and more results

27. Math 152 - Assessment of
Matlab “for” loop mastery
70% Correct
60% Incorrect
50% Blank
40%
30%
20%
10%
0%
2008 2009

28. Math 184--
intro calculus “workshops” part of course.
workshops course
Last year collected data on how they were functioning,
(observations, surveys, examine correlation of student
marks with numerous factors.)
This yea , made changes based o t e data
s year, ade c a ges on the data.

29. Math 184 Workshops –
Correlation between workshop
attendance and course grades
g
Pearson Correlation Coeffficient
0.60
0.50 relevant # is
0.40
(corr. coeff.)2
0.30
r
0.20
0.10
0.00
2008 2009

30. Math 184 Workshops
Student Survey
The workshop problems ….
"… provide useful practice for
p p "… are related to material covered
…
solving problems on tests" in class"
50%
2008 50%
2008
40% 2009 40% 2009
30% 30%
20% 20%
10% 10%
0% 0%
Strongly Agree Neutral Disagree Strongly Strongly Agree Neutral Disagree Strongly
Agree Disagree Agree Disagree

31. EOSC 211: Computer Methods in the
Earth Sciences
Introduced technique of “Pair-Programming”
from comp sci ed research:
Compared to previous year:
• Labs are completed about 15% faster
p
• Lab marks are about 10% higher
• Students are MUCH happier with the
transformed course
t f d
31

32. EOS Impact of TA training program
6. I consider myself to be an
effective teacher.
12
10
8
6 Pre
2008 4 Post
2
0
Agree Neutral Disagree
6. I consider myself to be an
y
effective teacher.
90
80
70
2009 60
50 Pre
40
Post
30
20
10
0
Agree Neutral Disagree

33. EOS 212-- using model based reasoning
Pre & post‐test scores pre
12
ts
er of student
10 post
8
6
Numbe 4
2
0
90‐100 80‐89 70‐79 60‐69 50‐59 <49
score (%) bins
( )
100%
90%
80%
70%
MUCH more time on
EOSC212
EOS collecting data
60%
50% a LITTLE more time on on time students
40% EOSC212
30%
Roughly equal time on
studying in courses.
20%
EOSC212.
10%
0% a LITTLE less time on
EOSC212. Relative amount of
MUCH less time on
EOSC212. time for different
courses across sci.i

34. Phys 109 & Sci 1 Intro physics lab
“invention” activity to develop scientific reasoning
invention
before activityy
after activity
1.0
ction of students
s
use histograms use standard
s
deviation
frac
0
student characterization of data

35. see posters to learn more about these and many more
Conclusions
1. It is possible to make widespread transformation
in UBC science teaching-- many courses, many faculty.
2. CWSEI transformations lead to
•much greater engagement,
•much greater learning,
learning
•happier students.
Looking forward to great progress in coming year

36. third year quantum mechanics course--
Common questions on QM spin
pre transform
pre-transform 2009 final exam 68%+/-3%
68%+/ 3%
2010 midterm 76%+/-2%
(spent half as much ti
( t h lf h time on t i )
topic)

37. physics lab diagnostic measurements
showing improvements, but more work needed

38. EOSC 211 Lab marks
39