This document discusses ways to improve assessment in education by better measuring student activity and reasoning. It advocates using a blended approach with a spectrum of assessment tools rather than just traditional testing. New techniques discussed include "Hockey Lines" for debriefing and engagement, and "Turk Talk" which uses cues and free text input for hybrid virtual patients. The document also discusses tracking student activity using xAPI standards and doing analysis of student pathways through simulations to identify issues. It acknowledges there is a lot of work needed to generate new assessment questions and cases but sees potential in automated item generation and sharing items through platforms like OpenLabyrinth.

1. Education Research:
measuring activity
and reasoning
David Topps
Thessaloniki, Jun 2015

2. Notices & Disclaimers
• Declaration of Interest
– I advocate the use of OpenLabyrinth
– I am the OpenLabyrinth Dev Consortium lead
– Open source project – I (sadly) do not make money from this
• Copyright disclaimer
– Most images are Creative Commons
– Remainder used under Section 29 (Fair Dealing)
Copyright Act of Canada (2012)
– Do not assume you can reuse all these images

3. Measure it to improve it
• “If you can’t measure something, you can’t
understand it. If you can’t understand it, you
can’t control it. If you can’t control it, you
can’t improve it.” – James Harrington

4. We do a terrible job
Assessment is
our Achilles’ heel

5. Assessment - fail
• Assessment as fact recall
• Assessment as pass/fail

6. OSCEs
• Judging your pilot…

7. Easy to measure
• Drunks and lamp posts

8. Design with the end in mind
• Mug shot

9. Blended
• Spectrum of tools and approaches
• Not just fancy mannequins
• Practica Continua

10. Sidebar
• Two new techniques in learning design
– Hockey Lines
– Turk Talk

11. Keeping them engaged
• Hockey Lines
– This fits with debriefing, activity and
engagement but not with measurement –
include it?

12. Turk Talk
• Cueing of responses
• Free text input
• The Maryland Project

13. Turk Talk
• Hybrid Virtual Patient
• The Mechanical Turk

14. Turk Talk

15. Turk Talk
• Hybrid Virtual Patient

16. Measuring Performance
• Design with THE END in mind

17. Measuring Performance
• RCTs are the ultimate… but ultimate
means last, not best.

18. Measuring Performance
• Don’t sneer at observational studies
– 2 famous papers from John Ioannidis
• Why Most Published Research Findings Are False,
PLOS Medicine, 2005
• Any casualties in the clash of randomised and
observational evidence? – BMJ, 2001

19. Hi-fi Sim

20. SxT
• Not a mediocre car hire firm
• SCT (Script Concordance)
– Bimodal reference panels
• SJT (Situational Judgment)
– Robert Hale, 15y longitudinal data
• SDT (Signal Detection)
– Receiver Operator Curves

21. Tracking activity

22. Tracking activity
• ADL xAPI
– …a.k.a. TinCan
– Lightweight format (“Bob did this”)
– …to Learner Record Store

23. Hype cycle of BIG data

24. Humour me

25. Small data
• Interesting examples
– Virtual learners
– Maladaptive learner pathways
– Time pressures

26. Virtual Learners
• Faculty development
• Virtual patients 
virtual residents
• Situational Judgment
Testing
• Larry, the resident
from hell

27. Larry is awful
• He plays every trick in the book
– Cheats, lies, goes skiing
• Faculty had to rate Larry
– x/22 – how many failed Larry?
Failure to Fail

28. Visual Path Analysis
• Graph analytics
• Normal exploratory path

29. Visual Path Analysis
• Jeff Foxworthy approach
• Just Git ‘er done!

30. Visual Path Analysis
• Going in circles
• Stuck because
– Poor knowledge
– Poor case design

31. Visual Path Analysis
• Color-coded nodes
• Easy to see
– Better
– Worse

32. Visual Path Analysis
• Placement
– Location
– Location
– Location!
– Author has placed the
more desirable nodes
down the right side

33. Discrete items?
• Why ask discrete questions?
• We solve problems as series of decisions

34. Adaptive testing
• Conditional logic
– from branching cases
• Layered triplets
– vs isolated questions and stems

35. Time pressure
• Lots of studies on critical tasks
• What about routine chores?
– Rushing Roulette cases
– Series of lab tests
– In-Tray exercise
• Big difference
– Learners vs faculty
– Expecting merge over series

36. Internet of Things
• Move from HSVO (hifi, hi-cost)

37. Apple HealthKit
• Data security
• Open source
– Like WebKit
– On GitHub already

38. So who is gonna do all this?
…lots of work
to generate
all these questions

39. Seven basic plots

40. Automatic item generation
• Game changer
• xAPI pull, not just push
• qBanq
– Also pull, not push
– Nobody wants
to push!

41. Who wants to play?
• New multi-part funding
• xAPI item sharing
info@openlabyrinth.ca