Presentation given as part of the Liverpool John Moores University (LJMU) learning and teaching conference 2017

1. Interactive Engagement Strategies
For large and diverse classes
(a choose-your-adventure workshop)
14-15 JUNE 2017
LIVERPOOL JMU
TEACHING & LEARNING CONFERENCE
“VISIONS FOR LEARNING”

2. Choose your own adventure
Lectures as 2-way conversations
Interaction (in / out of class)
Assessments (student-generated)
Assessments (exams)

3. Lectures

4. Context - class time (specifically lectures)
CC BY-NC 2.0 https://flic.kr/p/f3ynHx!
Derek Bruff: Class time reconsidered!
http://prezi.com/donq036eunko/class-time-reconsidered/!
https://www.brookes.ac.uk/services/ocsld/resources/20reasons.html!

5. Source ar)cle : Poh, M.Z., Swenson, N.C., Picard, R.W., "A Wearable Sensor for Unobtrusive, Long-term Assessment of Electrodermal Ac)vity," IEEE Transac)ons on Biomedical Engineering,
vol.57, no.5, pp.1243-1252, May 2010. doi: 10.1109/TBME.2009.2038487
Ac)vity: Jared Stang, UBC Physics

6. Weekly rhythm for our 1A class!
Poh, M.Z., Swenson, N.C., Picard, R.W., "A Wearable Sensor for Unobtrusive, Long-term Assessment of Electrodermal Ac)vity," IEEE Transac)ons on Biomedical Engineering, vol.57, no.5, pp.1243-1252,
May 2010. doi: 10.1109/TBME.2009.2038487

7. Weekly rhythm for our 1A class!

8. Learning gains on PI !

9. 1. Pre class material!
© Jorge Royan / http://www.royan.com.ar / CC-BY-SA-3.0, via Wikimedia Commons!

10. 1. Pre class material - DIY media!

11. 1. Pre class material - DIY media!

12. Learning gains on PI !

13. Reproduced from Eric Mazur
(search “Confessions of a converted lecturer” on YouTube)

14. 1!
2!
3!
A ball initially at rest is!
thrown upwards, comes !
back down & is caught!
!
Which of the following is a!
plausible graph of the!
acceleration of the ball !
with time?!

15. Source activity: Simon Lancaster (UEA, UK) Ross Galloway (Edinburgh, UK)

16. A large truck collides head on with a
small compact car.
Which of the following statements is true?
1. The forces on the car is greater
2. The force on the truck is greater
3. The force on the car and truck are
equal
4. Can’t specify without knowing mass
and speed of vehicles

21. R R Hake !
American Journal of Physics: Volume 66, Issue 1, Pages 64-74!
http://dx.doi.org/10.1119/1.18809!

22. Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111
Wieman commentary PNAS !
http://www.pnas.org/cgi/doi/10.1073/pnas.1407304111
“This meta-analysis makes a powerful case that any college or university that is teaching
its STEM courses by traditional lectures is providing an inferior education to its students”

23. 1. Real (and perceived) workloads!
2. Dealing with resistance !
3. Suitability of teaching spaces!
4. About ‘covering content’!
5. Loss of (total) control !
!
… and the upsides too. !

24. Interaction /
communication

25. FOSTERING INTERACTION
a case study
Simon Bates
Senior Advisor, Teaching and Learning
Academic Director, CTLT
Professor of Teaching, Department of Physics and Astronomy

26. OVERVIEW
Landscape of LT tools and applications
My course context
Tools matter!
Implementation
Features

27. Content Authoring Content Delivery Publisher Simula4ons
Adobe Crea)ve Suite Livescribe Arts File Share Kaltura (*)
UBC Blogs / WordPress
(Content) (*)
Cengage (PIC) (*) Google Cardboard
Ar)culate Studio MediaSite (*) Cengage iLrn
LearnDash
(WordPress) (*)
Macmillan (PIC) (*) Layar
Audacity Microso] Excel
Connect (Content
Management) (*)
Library Online
Course Reserves
(LOCR) (*)
UBC iTunes (*) McGraw-Hill (PIC) (*) Mannequin Simula)ons
Bb Collaborate Voice
Authoring (*)
Microso] PowerPoint Document Cameras Lyryx UBC Wiki (*) Pearson (PIC) (*) Mo)on Control
Camtasia (*) One bucon Studio (*) DropBox
Modern Digital
Image Database
(MDID)
UBC YouTube (*) Sapling Learning (PIC) (*) NeoReality
Collabora)ve Learning
(*) Annota)on System
(CLAS)
Panopto Drupal (*) Moodle (*) Vimeo Wiley (PIC) (*) Phet Simula)ons
Explain Everything Dspace Omeka Vitalsource Praxis
Final Cut Pro Prezi edX (*) Owncloud WebAssign VCER
Entrada Perusall Wikipedia
Hot Potatoes
Respondus Studymate
(*)
Evernote Piazza (*) Wis)a
Jing Snagit
Exam Prep
Database (*)
Plone Workspace (*) PorLolios
Keynote Timeline JS Github Podcasts Zimbra Chalk and Wire
Lectora Google Docs Reflector Zoomify Connect (ePorLolio) (*)
Lightboard (*) VideoScribe (*) HTML 5 Flash Cards SourceTree Pep
iBooks SugarSync
Assessment Response System
Adap)ve Compara)ve
Judgement (ACJ) (*)
Mechanical TA Aplia
Auto Mul)ple Choice
(AMC)
Moodle (*) iClicker (*)
Calibrated Peer Review
(CPR) (*)
Open Badges UBC (*) Kahoot!
Cengage iLrn Pearson MyTest Learning Cataly)cs
Crowdmark PeerScholar Poll Everywhere
Entrada PeerWise (*) REEF polling (*)
Exam Prep DB (*) Remark Top Hat
ExamSo] Respondus Quiz (*)
Form Builder (*) Scantron
Grade Grinder Turni)n (*)
iPeer (*) WebAssign
LearnDash (WordPress)
(*) Webwork (*)
Analy4cs Course Evalua4on Course Admin Other
Arts Datamart
Blackboard Outcomes
Assessment (*)
Cengage iLrn 3D prin)ng
Blackboard Outcomes
Assessment (*)
CoursEval (*) Connect (Grade Center) (*) Google Earth
Connect (Performance Center)
(*)
SEoT Datamart (*) Doodle iStudiez Pro
Google Analy)cs TeachEval (*) Entrada SCORM (*)
IBM SPSS Google Calendar
Tin Can API (Experience
API)
JMP LearnDash (WordPress) (*)
Microso] Excel Moodle (*)
SEoT Datamart (*) One45
Stata Turni)n (*)
Tableau UBC Blogs / WordPress (*)
WebAssign
LEARNING TECHNOLOGY TOOLS - FUNCTIONAL MAP
CONTENT ASSESSMENT
COURSE MANAGEMENT & EVALUATIONINTERACTIONS
Discussion Survey Tool Social Media VC
Connect (Discussions) (*)
Connect (Enterprise Surveys)
(*)
Facebook Adobe Connect
Piazza (*) Connect (Surveys) (*) Figure 1 Bb Collaborate Web Conf. (*)
PulsePress (*) FluidSurveys (*) Google+ BlueJeans (*)
Slack Gravity Forms (WordPress) LinkedIn Google Hangouts
UBC Blogs / WordPress
(Discussion) (*)
LimeSurvey Twicer Lifesize
Qualtrics Skype
Survey Monkey VC (MedIT)
Vovici WebEx
Bold = Integrated Tool
(*) = Supported by LT Hub

28. CONTEXT
First year intro Physics courses
P101 / P117
Non-majors
Flipped class design
Heterogeneous cohort

29. TOOLS MATTER
Connect discussions vs Piazza

30. TOOLS MATTER
Connect discussions vs Piazza

32. IMPLEMENTATION
“No email” policy
Give reasons to engage
Support TA: to monitor and respond

33. A BUNCH OF NICE FEATURES…..

41. Assessment: student
Generated material

47. Selected results & analysis
Engagement - how do students use the system?
Benefits - what is the impact on learning?
Question quality - how good is what students produce?
Relevant publications:
Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002
(2014)
Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International
Journal of Science Education, 1-15 (2014).
Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) 10, 020105
Student-generated assessment - Education in Chemistry (2013) 13 1

48. Typical implementation
Minimum participation requirements for
each of two assessment exercises (PW1,
PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves
Winter Semester: 3 sections, ~800 students

49. Not so typical implementation
Writing original questions is a
demanding activity
Extensive scaffolding exercises
Revisited in subsequent tutorials

51. Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] 998 [1.6]
Answers 11393 [17.2] 11807 [18.7]
Comments 4901 [7.4] 5509 [8.7]
PW 1 PW 2

52. Engagement with PeerWise

54. Engagement with PeerWise

55. Engagement with PeerWise

56. Engagement with PeerWise
Generally, students did
• Participate beyond minimum requirements
• Engage in community learning, correcting errors
• Create problems, not exercises
• Provide positive feedback
Generally, students did not
• Contribute trivial or irrelevant questions
• Obviously plagiarize
• Participate much beyond assessment periods
• Leave it to the last minute (sort of….)

58. 58
Correlation with learning

59. 59
Correlation with learning

60. 60
Correlation with learning

61. 61

62. 62
Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare & assess various option possibilities; often conceptual
Synthesize
Ideas and topics from disparate course sections combined.
Significantly challenging problem.

63. 63
Textp>0.05, NS

64. 64
Question/Explanation Quality
Score Level Description
0 Missing No explanation provided or explanation incoherent/irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient explanation/justification/ Some
aspects may be unclear/incorrect/confused.
3 Good Clear and detailed exposition of correct method & answer.
4 Excellent
Thorough description of relevant physics and solution strategy.
Plausibility of all answers considered. Beyond normal expectation for
a correct solution

65. 65

66. 66
Results (UoE 2010-11)
2 successive years of the same course (N=150, 350)!
‘High quality’ questions: 78%, 79%!
Over 90% (most likely) correct, and 3/5 of those wrong were !
identified by students. !
69% (2010) and 55% (2011) rated 3 or 4 for explanations!
Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level. !

67. 67
Bottomley & Denny Biochem and Mol Biol Educ. 39(5)
352-361 (2011)
107 Year 2 biochem students
56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the
United States reported that 93% of the questions asked on
examinations in introductory biology courses were at the
lowest two levels of the revised Bloom’s taxonomy”
Comparison with literature

68. 68

69. 69

70. Assessment: exams

71. IAD Course Organisers Forum – Edinburgh, Oct 2015
Two-stage exams!
assessments of, for and as learning !
Simon.Bates@ubc.ca
@simonpbates
bit.ly/batestalks

72. Disclaimers!
• Not ours; not new
• Similar to elements of well-established pedagogy e.g. TBL
• But….. Significant due to ease, effectiveness and take-up

74. Overview
• Define two-stage exams (UBC style)
• Discuss advantages and disadvantages of two-stage exams
• Take your questions on logistics of administering them
• Describe some research done with two-stage exams at UBC

75. Two-stage Exams
The basics:
Summative assessments
During the exam (midterms / final / in-class test)
– Students complete then hand in individual exam
– Get into groups of 4 to work on a group exam (for about ½
the time of the individual portion)
What they look (and sound!) like
http://www.cwsei.ubc.ca/resources/SEI_video.html

76. Two-stage exams
• The Group Exam is identical to the individual exam
(+/- one or two “difficult” questions)
• Students work on the group exam collaboratively
Consensus
• The group only gets one exam sheet!

77. Two-stage exams at UBC
At UBC: now well over 100 courses
Physics, Chemistry, Biology, Math, Statistics, Earth, Ocean,
and Atmospheric Sciences, Computer Science, Forestry,
Pharmacy, Psychology, and Land and Food Systems
Many class formats / assessment types
900 student final exams (that was me J )
450 student 1st year lectures
20 person laboratories
20 student 4th year seminars
5 student graduate classes

78. Advantages
• Immediate feedback
• Models classroom behaviour
• Engaged students
• Teaching!

79. Advantages
Your list:
• Nearly instant feedback / reflection
• More collaboration
• Engagement
• Responsibility
• Different perspectives
• Happier students learn better
• Develops group work skills
• Lower achieving students get extra explanation
• Reduces anxiety

80. Advantages
• Higher achieving students benefit from explaining and
reinforcing knowledge
• ALL students participate!
• Quieter students get a chance to contribute even in large
classes
• Students like it and believe it helps their learning

81. Some Student Comments Phys 250
and EOSC 114
“Some problems, its a good way to find out
what you did wrong on the individual exam
almost immediately.”
“Great idea! The group exams give you a
chance to go over your answers to the exam
while you still care about the questions.”

82. Some Student Comments Phys 250
and EOSC 114
“Discussion over tricky questions facilitate
learning immediately and the answer/
concept is stuck in your brain FOREVER!”
“You actually learn what you got
wrong right away from a student
perspective”

83. Disadvantages
Your list
Compressing the grades
Group composition
Loud students
No agreement

84. Possible Disadvantages
• “Only certain subjects”
• Social loafing
• Dominant group members
• Sidetracked by process
• Assigning marks
• A & D
• Student inexperience with groups!
• Time! Shorter exams
• Getting “convinced” of wrong answers
• Incompatible with curving as standard

85. Some Student Comments Phys 250
and EOSC 114
“… That said, there was the issue of excessive
discussion in the group exam. That is, there were
several times where a part of a question was
contentious within our group and the ensuing
debate, frequently ended only by calling over
<instructor>, often took up so much time that doing
the last few questions was hurried and messy.”

86. Feedback from EOSC 114
“Group Exams are…”

87. Posi)ve ‘because…’ comments
Discussion 48
Learn why you were wrong 37
New perspectives 29
Better grades 21
Instant feedback 16
Review 10
Build confidence 8
Understand questions better 6
Learn techniques from others 6
Other 27
Total 208

88. Nega)ve ‘because…’ comments
Coming to consensus 21
Time consuming 13
Unbalanced knowledge in group 6
Convinced of wrong answer 3
Realize did poorly individually 3
Worth too much 2
Other 8
Total 56

89. How Groups Choose (EOSC 111)

92. Research evidence for effec)veness
Gilley, B.H. and Clarkston, B. (2014) Collaborative Testing: Evidence of Learning in a Controlled In-Class Study of Undergraduate Students
Journal of College Science Teaching Vol. 43 No. 3

96. Resources
• Journal articles:
– Gilley & Clarkston (2014) Journal of College Science
Teaching Vol. 43 No. 3
– Rieger & Heiner (2014) Journal of College Science
Teaching Vol. 43 No. 4
– Wieman, Rieger & Heiner (2014) Phys. Teach. Vol 52
• For more information and videos visit:
http://blogs.ubc.ca/eoassei/two-stage-exams/
• Misc articles:
http://www.macleans.ca/education/multiple-choice-multiple-students/
http://blogs.ubc.ca/catherinerawn/2014/07/22/two-stage-exam-introduction-and-resources/
https://teachingcommons.stanford.edu/teaching-talk/turn-exam-learning-experience-two-stage-exams
http://www.artsrn.ualberta.ca/mengel/huco5002014/?p=356

97. Resources

98. www.cwsei.ubc.ca!

99. http://blog.peerinstruction.net/ !

100. http://flippedlab.learning.ubc.ca/!

101. http://diy.open.ubc.ca/!