David Topps gave a presentation at the WAVES Symposium in June 2016. He discussed scenario-based learning and how it can be used for more than just medical education. He also talked about measuring learning through activity metrics collected from various sources, and using cognitive computing to analyze large amounts of data from online programs. The presentation covered a wide range of topics in a non-linear way and emphasized learning from mistakes with safe environments and feedback.

1. Oh no! What did he do now?
David Topps
WAVES Symposium
June 2016

2. Notices & Disclaimers
• Declaration of Interest
– I advocate the use of OpenLabyrinth
– I am the OpenLabyrinth Dev Consortium lead
– Open source project – I (sadly) do not make money from this
• Copyright disclaimer
– Most images are Creative Commons
– Remainder used under Section 29 (Fair Dealing)
Copyright Act of Canada (2012)
– Do not assume you can reuse all these images

3. Warning: branching pathways!
• This is a non-linear presentation!
– Vagus = wanderer

4. Scenario Based Learning
• Not just for medical education
• Ruth Colvin Clark’s book
• General principles
– Process matters
– Using the right tool/modality
– Learning from your mistakes

5. “...the way that you do it”
“Focus not only on correct or incorrect
answer but the process used to derive
that answer”
Ruth Colvin Clark

6. Learning from your mistakes
– Countless YouTube videos belie that
– In medicine, we bury our mistakes!

7. Medical Error
• Institute of Medicine report
– “To Err is Human”
– Originally 1M deaths/year
– Downgraded to only(!) 100k/yr
• Why no furor from the public?
– “I’m glad my doctor’s not like that”
– We are still very trusted

8. Trust levels for professionals

9. Causes of Error
• Avoid blame
• Systems errors
• New IoM report blaming cognitive biases
– Improving diagnosis in medicine, Balogh et al 2015
– 25 years of literature to support this

10. 50/50 Study
• Geoff Norman and Sandra Monteiro
• Just released at CCME last month
• Brilliant methodology
• Tested ‘Society to Improve Diagnosis in Medicine’
• 100% more biases identified if wrong
• Kappa scores showed no agreement on biases
• Only bias was hindsight

11. Learning from your mistakes
• Need a safe environment to practise

12. Scenario Based Learning
• Using the right tool for the job
• Roger Kneebone’s work

13. How does Disney do it?

14. HSVO project
• Blended sims
– Laerdal SimMan 3G
– Virtual Human Dissection
– Virtual POV using a
camera array
– Glued together by
OpenLabyrinth
virtual patients

15. HSVO project
• Lessons learned…
• Activity metrics
• Lessons about activities
– Team learning
• Long quiet spells
• Flurries of activity
• All too tightly coupled

16. Riding a bike
• With John Seely Brown
• Handlebar exercise

17. You think my stuff is scattered?

18. 70/20/10 model

19. Do we learn from our mistakes?
• Only with measurement and feedback
• Buffalo Dash pilot
– Failure to Fail

20. Not everybody learns
• Big 5
– Not the same as the game hunter’s big five
– …although some of them are just lion about it!
– …and we don’t want to be Buffaloed
– (consider adding 8 buffalos sentence
• https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_
buffalo_buffalo_buffalo_Buffalo_buffalo )

21. Big 5

22. Not everybody learns
• Conscientiousness Index

23. Not everybody learns
• Conscientiousness Index
• -3SDs
– The secretaries already know!
– We have the data

24. Oh no! What did he do now?

25. Not just the right answer
• Not good enough for high school math test
• What about the sums?

26. Activity Metrics
• xAPI stuff
• Simple way of blending across modalities
• Lots of sources

27. Activity Metrics
• xAPI data from:
– OpenLabyrinth
– WordPress
– H5P widgets
– Arduino sensors
• Heart rate
• Galvanic skin
response

28. What’s the fuss about Big Data?
• Framework
–Volume, Variety, Velocity
–Debated… more V’s
• TED talk by Kenneth Cukier
–www.ted.com/talks/kenneth_cu
kier_big_data_is_better_data

29. Lots and lots of data
• BIG DATA principles
• Volume
• Velocity
• Variety
• At a personal level

30. Big Data’s 4th V
• Visualization
• Examples from
– GrassBlade
– OpenLabyrinth
– Watershed
– D3.js
– Example case with xAPI  EPAs  Competencies

31. Big Data & MOOCs
• Bigger data
– Not like LHC etc
• Karolinska’s work with OpenLabyrinth
– How many?

32. The Challenge of MOOCs
• Drop-off in numbers
• Meaningful assessment
– Various approaches
– No “badge”
• Nobody using Mozilla Badges

33. Not Enough Data!

34. Fool some of…
“You may fool all the people some of the time;
you can even fool some of the people all the
time; but you can’t fool all of the people all the
time.” [Abraham Lincoln]

35. How does BitCoin work?
• Principles of cryptocurrencies
• Guarantee identity of participants is real
– Without revealing who they are
• Two pieces of data connected
– With a guaranteed common identifier
– Linked by a block-chain
– Any encryption is crackable with enough effort
– Block chain makes it not worth the effort
• Much more than money
– Ethereum: https://www.ethereum.org/

36. Not Enough Data
• Collect so much data that you cannot fake it
• The effort of faking it is more than doing it

37. Not enough data
• Scores, MCQs etc
• (Clinical) reasoning pathways
– Broader uses of OpenLabyrinth
• Sensors
– Mug shots from web cams
– Keyboarding “fist”
– Office-level logistic data
• Participation data

38. Participation Scoring
• Traditional way of scoring distance learning
• What constitutes participation?
– Word counts
– Message posts
– Automatic text parsing
• Too basic for our needs
– Peer scoring
• Too prone to buddy effects

39. Scalable Human Interaction
• Not enough human/teacher interaction
• Turk Talk
– Scalable from groups of 8…
• Layered approach
• All text is recorded
• Can be analysed

40. Who wants to score that text?
• Bane of Teachers everywhere

41. Cognitive Computing to the Rescue
• Neural networks on steroids
• Much more than automated text processing
• Big data is cheap

42. Cognitive Computing Platforms
• IBM Watson
• Amazon
• Google
• Microsoft

43. Web 4.0
• Web 2.0 = Web Services
• Web 3.0 = Semantic Web
• Web 4.0 = Cognitive computing

44. SyntaxNet
• Google Research
• Parsey McParseFace
• http://googleresearch.blogspot.ca/2016/05/a
nnouncing-syntaxnet-worlds-most.html

45. Triplets
• Semantic indexing
– RDF triplets
• xAPI
– RDF triplets
• Triplets x triplets
– 3 x 3 = 9

46. Wrap Up
How to finish?

47. The Answer is 42
• Life, the Universe and Everything
• Douglas Adams
• 9 x 6 = ...
• 3 x 3 x 3 Hypercube in 4 dimensions
– 9 vertices
– 6 aspects
– Magic number is 42

48. The Singularity
• Chess
• Go
• Quibbling about the Turing Test
– https://en.wikipedia.org/wiki/Turing_test
• Exponential curves and false summits

49. What matters?
• “Not everything that can be counted counts,
and not everything that counts can be
counted” – Einstein (but not)
• Futility of RCTs in education research

50. The Drunk & the Lamp Post