Keynote given at BCIT Faculty of Nursing Scholarship Day, May 2016

1. The
21st Century Educator
Simon Bates
simon.bates@ubc.ca
@simonpbates
bit.ly/batestalks
Students as partners in learning & teaching

2. The
21st Century Educator
Students as partners in learning & teaching
“Anatomy” of skills
and values
Example driver of
change: technology
Case study example
Deeper engagement
with assessment,
learning

3. CC BY-NC-SA 2.0 https://flic.kr/p/2ZdABF

4. Technology - scale and pace
Slide credit: Eric Grimson (MIT)

5. Technology - scale and pace
Slide credit: Eric Grimson (MIT)

7. Technology - reach and unbundling
https://www.youtube.com/watch?v=QWEq3xifCDw

8. Technology - disruptions
Graph extracted from http://vikparuchuri.com/blog/on-the-automated-scoring-of-essays/

9. Technology - implications
Changing the
of many aspects of life,
…and learning is included
what, where, when, how,
from whom and with whom

10. So what are the
we need to embrace, develop and refine?
skills, values and habits

12. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

13. Case study - students as partners
“How can I get my students to
engage more effectively with
formative assessment
opportunities in the course?”

14. a"web&based"MCQ"repository"created"by"students"

15. Ins$tu$ons((signing(up(per(year:(
!
2009:! ! !22(
2010:! ! !66(
2011:! ! !204(
2012:! ! !266(
2013!(Jan,Jun):! !214(
Growing(content(repository:(
!
Courses:! !2,500(
Logins/month:! !75,000(
Ques<ons:! !600,000(
Answers:! !12,000,000(
Answers
(20,000,000)
Questions
(1,000,000)

16. Student'ownership'over'
learning'resource'
Student'familiarity'with'
social'so7ware'
Leveraging'student'
energy'and'crea9vity'

18. Badges' Points' Leaderboards'

19. Selected results & analysis
Engagement - how do students use the system?
Benefits - what is the impact on learning?
Question quality - how good is what students produce?
Relevant publications:
Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4),
045002 (2014)
Student-Generated Content: Enhancing learning through sharing multiple-choice questions.
International Journal of Science Education, 1-15 (2014).
Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014)
10, 020105
Student-generated assessment - Education in Chemistry (2013) 13 1

20. Typical implementation
Minimum participation requirements for each
of two assessment exercises (PW1, PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves
Winter Semester: 3 sections, ~800 students

21. Not so typical implementation
TOCCLT'.'
-
QestIsct
How 1o...
q/t4osTer
Submit ond onswer questions
on topics in lhe torget region,
just obove lhe physics
you hove olreody mostered.
Dislroclers
) l*"*"ò
rNrrltng
Ih s region corrk:ins llte
plrysics knowledgo ond colrco¡ls
you connol leorn yel
becouse the louncalions
crrc nol in plocc
PHYSICS TOPICS IN YOUR TARG T REGION
buo'5ot't1 'Den¡¡lu
B¿:ovvì 1 J
F:rcq - YOUR CHOSEN TOPIC
à..
auC'q^ x@
bôo'osro
pn ro, d logroo,
o 9tS
COMMON MISCONCEPÎIONS AND ERRORS
(Sec hHÞ:/jÞhy¡.udofo¡.cdu/CJP/trêconcoÞllo¡t.pdt
tor o llil ol common mhconcepllons)
¿K N(,qht %cce Açxs no exi¡þ
, c.rrrçrJ On oblec'-t 5 õÞYJrnr cìü?À
I in rtuiò. ü
lrh'brot4ont eprç,e .I¡1enÀs crq
den-s,{^ c>Ç c.,þ¡ec , cìo+ .*¿n:r
oÇ Çtu,8,
fr- btrrqon! $crCe c¿cs Àoc¡:'cr
fr¿lt{e- r *hcrn ú?
Trr¿e. $ììfifflsz hrrr: ¿lÕoolr-g lnnS
(**= tcpo þ/rnl
{otr-* qrÇ bq.:. 0" Ooz
Physics knowledge
ond conccpluol underslonding
you lrove olrcody
conslrucleC ¡n your heod
¡,<'t.lerJ rt)^,¿z! Þ.rrp r- rr't¡ ,l?
Chcck lhol yovr on3Íêr ¡t rêo!ànoblê ðnd potrlble
'f<,,cc
Photo by Seth Casteel
http://www.littlefriendsphoto.com
Permission to use agreed
Writing original questions is a
demanding activity
Extensive scaffolding
exercises
Revisited in subsequent
tutorials

22. Copyright 2013 Graham Fowell / The Hitman, re-producedwith permission, Educa&on In Chemistry, Vol 50 No 1 (2013)

23. Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] 998 [1.6]
Answers 11393 [17.2] 11807 [18.7]
Comments 4901 [7.4] 5509 [8.7]
PW 1 PW 2

24. Engagement with PeerWise

26. Engagement with PeerWise

27. Engagement with PeerWise

28. Engagement with PeerWise
Generally, students did
• Participate beyond minimum requirements
• Engage in community learning, correcting errors
• Create problems, not exercises
• Provide positive feedback

29. Engagement with PeerWise
Generally, students did not
• Contribute trivial or irrelevant questions
• Obviously plagiarize
• Participate much beyond assessment periods
• Leave it to the last minute (sort of….)

30. Engagement with PeerWise

31. Correlation with learning

32. Correlation with learning
Quartiles
Q4 – top 25%
Q3 – upper middle
Q2 – lower middle
Q1 – bottom 25%
22 students did not
take the FCI

33. Correlation with learning

34. Correlation with learning
1st year Chemistry N=172
University of Edinburgh

35. Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare &assess various option possibilities; often conceptual
Synthesize
Ideas and topics from disparate course sections combined. Significantly
challenging problem.
Text

36. 1 2 3 4 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Bloom's Taxonomy: Question Quality
Textp>0.05, NS
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
PercentageofSubmittedQuestions
First semester N = 350
Second semester N = 252

37. Question/Explanation Quality
Description of explanation quality
Score Level Description
0 Missing
No explanation provided or explanation incoherent/
irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient explanation/
justification/ Some aspects may be unclear/incorrect/
confused.
3 Good Clear and detailed exposition of correct method & ans
4 Excellent
Thorough description of relevant physics and solution
strategy. Plausibility of all answers considered. Beyond
normal expectation for a correct solution

38. 0 1 2 3 4 0 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Explanation Quality

39. Results (UoE 2010-11)
2 successive years of the same course (N=150, 350)
‘High quality’ questions: 78%, 79%
Over 90% (most likely) correct, and 3/5 of those wrong were
identified by students.
69% (2010) and 55% (2011) rated 3 or 4 for explanations
Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.

40. Bottomley & Denny Biochem and Mol Biol Educ. 39(5)
352-361 (2011)
107 Year 2 biochem students
56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the
United States reported that 93% of the questions asked on
examinations in introductory biology courses were at the
lowest two levels of the revised Bloom’s taxonomy”
Comparison with literature

42. Resources
Student-facing system
http://peerwise.cs.auckland.ac.nz/
All the research studies referenced and scaffolding materials referred
to are accessible through the PeerWise community site http://
www.peerwise-community.org/