UCL's Professor Patty Kostkova talk at Facultad de Informática on October 13th

1. 1
Digital Health: From Online
Evidence to Serious Games
“Stop the Bugs”
Patty Kostkova
UCL
London, UK

2. Healthcare at the end of 20th century: Internet
“The impact of the Internet has largely been
unforeseen, and it may have a revolutionary role in
retooling the trillion-dollar health care industry in the
United States” (June Forkner-Dunn 2003).
2

3. Healthcare in 21st century: mobile
 31% (up from 17% in 2010) of cell phone
users have used their phone to look up
health information
3

4. Information Overflow
 The information you have
is not the information you
want
 The information you
want is not the
information you need
 The information you
need is not known
 The information that’s
known can’t be found in
time
4

5. 5
Projects
hosted by
the NeLI
portal
 National electronic Library of Infection (NeLI, formerly NeLCD) – professionals oriented
 Bugs and Drugs on the Web (AR DL) – public oriented
 Training in Infection Manual (TII) – for trainees
 National Resource for Infection Control (NRIC) – for professionals
 eBug – EU DG SANCO project – education pack and games for children about AR
 SeaLife – semantic web browser for life sciences
 Medicines Support Unit for Optometrists – in collaboration with Department of Optometry, City Uni.
 WHO Lab Resources evaluation project
 IFH project – International Scientific Forum for Home Hygiene

6. 1. Quantitative evidence
NRIC monthly site usage 2005‐2011
40,000
35,000
30,000
25,000
20,000
15,000
10,000
5,000
0
Sep‐
05
Nov‐
05
Jan‐
06
Mar‐
06
May‐
06
Jul‐
06
Sep‐
06
Nov‐
06
Jan‐
07
Mar‐
07
May‐
07
Jul‐
07
Sep‐
07
Nov‐
07
Jan‐
08
Mar‐
08
May‐
08
Jul‐
08
Sep‐
08
Nov‐
08
Jan‐
09
Mar‐
09
May‐
09
Jul‐
09
Sep‐
09
Nov‐
09
Jan‐
10
Mar‐
10
May‐
10
Jul‐
10
Sep‐
10
Nov‐
10
Jan‐
11
Mar‐
11
May‐
11
Page Views Page Views on NRIC before and after ECCMID, 19th - 22nd April 2008
1000
900
800
700
600
500
400
300
200
100
0
Fri 18 Sat 19 Sun 20 Mon 21 Tue 22 Wed 23 Thu 24 Fri 25 Sat 26 Sun 27
Date
Page Views

7. Searching/browsing behaviour
7

8. 8

9. Does the accessibility of information
make the search for knowledge any
9
easier?

10. Google, msn searches that led to the AR site between 1 Sep & 31st Jan
38883 searches
10
 7161 included the word alcohol
 12949 antibiotic
 6515 chest
 3467 bacteria
 2085 virus
 1852 acne
 1149 flu
 …. And 250 included "cat"
• Some users have an addiction problem:
• Quit antibiotics
• Quit working
• Many users are concerned about animal
welfare:
• Antibiotics in disadvantaged animals
• information on cat colds
• Others have an international approach:
• swedish acne remedy
• mrsa chinese formula

11. Professional vs public Needs: Media
Influence or Outbreak Early Warning
3 Datasets:
 Professional Information Needs: The
NeLI/NRIC webserver logs
 Public Information Needs: The Google
Trends data
 Media coverage: Guardian Open Platform
API

14. Knowledge and attitude Change: Pre and Post Questionnaires: Bugs and
14
Drugs Site for Public
Pre and Post Use Questionnaires
– Used to evaluate differences in patient knowledge and attitudes
before and after using the site
– Users were free to browse the site between questionnaires
– First study took place in the Science Museum London as part of
‘live science’
– 227 visitors took part of which 177 completed both questionnaires
– Study repeated at Nottingham City Hospital Open Day and also
Oxford University Medical School

15. Summary of results - Science Museum
15

16. Knowledge & Attitude: Bugs and Drugs
Relationship between knowledge change and
5% users
decreased
knowledge score
and increased
attitude score
-1 -0.5 0 0.5 1
16
attitude change
1
0.8
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
Change in knowledge
Change in attitude
24% users
increased
knowledge score
and decreased
attitude score
7% users
decreased
knowledge score
and decreased
attitude score
11% users increased
knowledge score and
increased attitude
score

17. Community of Practice – Social Networks

18. FEM wiki – social networks and training
wiki resource for epidemiologists
18

19. FEM wiki Launch 2010 and beyond
300+ users registered at
ESCAIDE event.
Now: over 600 users
A red bar represents a single user’s period of activity.
19

20. SN vs active forum contributions

21. SN Roles – evolution in light of
actions in the CoP
 red - 1st replies
 blue -1st posts and 1st reply
 green nodes - 1st posters

22. Evolution of SN

23. Over 3 million
tweets were collected
during May 2009 to
March 2010
containing the word
flu by CeRC pilot
study

24. “I have swine flu”
12,954
“I have the flu”
12,651

25. Self Reporting Flu
3000
2500
2000
1500
1000
500
0
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
Total Tweets
Week
US
UK

26. Epidemic Intelligence:
Early Warning and Outbreak Prediction

27. 11th June 2009

29. Hour-by-Hour breakdown of the most
popular resources posted to Twitter

30. The Dashboard Platform

31. Medi+Board

32. Edugames4all
DG SANCO funded project to develop web games teaching pupils
about microbes, hand and respiratory hygiene, and appropriate use
of antibiotics

33. Project Background
 European wide, DG SANCO funded, antibiotic
and hygiene teaching resource for junior and
senior school children.
 17 countries are involved covering 62% of the
European population.

34. Learning Objectives…
 Microbes are all shapes
and sizes:
 Microbes exist in various locations:
(inside the body)
(in the kitchen)
(on the skin’s surface)

35. Teaching by playing:
Good and Bad Microbes
• Player transports lactobacillus to make yogurt
DEMO

36. Teaching by playing:
Food Hygiene
The food sorting game is used to show
hand, respiratory and respiratory hygiene.
DEMO

37. Antibiotics and Vaccine Use
• Player throws white blood cells
• Player delivers full course of antibiotics to
infection
• DEMO

38. Junior Game Evaluation:
Statistically significant knowledge gain
 Level 3, Q1: “We use good microbes to
make things like bread and yogurt” (p <
0.001, chi = 14.46)
 Level 1, Q1: “If you cannot see a microbe it
is not there” (p = 0.02, chi =5.60)
 Level 2, Q2: “Soap can be used to wash
away bad bugs” (p =0.02, chi=5.28)

39. Learning through Mechanics or
Text? - using these LO:
 Statistically significant
◦ LO1: “Soap can be used to wash away bad
bugs”
◦ LO3: “We use good microbes to make things
like bread and yogurt”
 Statistically not significant
◦ LO2: “Our bodies have nature defences that
protect us”.
◦ LO4: “All microbes and bad for us”.

40. Demographics
 60 children
 Age: 6-11 years old
 Recruited on a voluntary basis
 30 children in each group
 Group 1:
◦ Text taught LO1 and LO2
◦ Mechanics taught LO3 and LO4
 Group 2:
◦ Text taught LO3 and LO4
◦ Mechanics taught LO1 and LO2

41. Results
 The results do not show a statistical significant for
either method of teaching the LOs – delivering
each LO remain subjective to the topic
 “We use good microbes to make things like bread
and yogurt” was the only statistically significant
LO and the one where mechanics was more
successful than teaching through text. Also, it was
the most successful in the large evaluation using
combined text+mechanics approach - > this
mechanics works 
 Work in progress: larger studies, more LOs and
combination of methods

42. Seamless Evaluation Framework
LOs Scoring
Plot
Outline
Quest
Set
Conversa
tion
Nodes
Assets
Rules
Story
Rules
Game
Mechanics
Conversati
on Rules
Animation
s
+
+
+
+
Education
Layer
Mission
Layer
Quest
Layer
Conversation
Layer
+ Presentation
Layer

43. Study: Integrated assessment vs
external assessment
 49 kids age 6 – 13
 Two groups:
◦ Integrated assessment: How to be a Millionaire Show?
◦ Non integrated assessment: survey in the game

44. Results
Game perceived as
 “fun” (p=0.34, CI=95%)
 “having fun by playing the game” (p= 0.27,
CI=95%)
 “nice to play” (p=0.65, CI=95%)

45. Results continued
 63% with integrated evaluation preferred it
 20% reported that the integrated evaluation increased
their game experience
 30% said game more interesting, another 30% did not
affect them in any way, 15% stated the quiz did not
affect their game experience but prefer not to have it
integrated in the game
 5% do not want the quiz as part of the game
 60% did not notice that their knowledge was evaluated
through the integrated evaluation.

46. IDS Detective Game
 Interactive Digital Storytelling Game
 Target group: senior school children (12-16 years)
 Several Missions
◦ Bad Bacteria at BBQ – importance hand hygiene
◦ When Bugs go Wild – prudent antibiotics use
◦ Gambling Never Pays - prudent antibiotics use

47. Microvision…

48. Investigative dialogue: Jamie
Grimesworth (chef)

49. Collecting Evidence
DEMO PUZZLE

50. Seamless Evaluation Framework
LOs Scoring
Plot
Outline
Quest
Set
Conversa
tion
Nodes
Assets
Rules
Story
Rules
Game
Mechanics
Conversati
on Rules
Animation
s
+
+
+
+
Education
Layer
Mission
Layer
Quest
Layer
Conversation
Layer
+ Presentation
Layer

51. Seamless Evaluation Framework

52. Seamless Evaluation
“Debriefing phase”: Post test – towards the
end of the game – measure knowledge
change

53. Case Study

54. Seamless Evaluation – the Evaluation
Aims to assess:
 Player’s perception of the assessment (21 participants – that did
not leave the survey incomplete)
 Effectiveness of the game in delivering the LOs (145 participants
– participants who finish playing the game)
Participants:
 A mix of persons playing the game online and in a
controlled environment
 First the persons played the game and then they were
asked to fill a survey

55. SE Evaluation
Players perception of the integrated assessment:
Preferred assessment method: 94% preferred the evaluation
integrated in the game flow

56. Training mission: “Nathan – clean
up the locker room!”
 Based on “when bugs go wild”
mission
 Nathan, non-player character,
failed to clean the locker room
 Dirty socks in footballers’ cocker
room
 Nathan claims” these are clean but
smell off “deodorant”
 -> microbision shows presence of
microbes
 E-Copter lab tests shows these are
fungi
 Nathan admits the failure and
clears the room

57. Quantifying it: Usability Evaluation
 Evaluation in a “London School”
 15 kids participated
◦ 8 played with training mission
◦ 7 without training mission
◦ Choosing non-control evaluation by selecting only
those who declared that they have never played IDS
type of game would demonstrate improvement but
we wanted to know “how much” and is it “usable”
◦ Small numbers: practical issues in working at schools,
difficulties engaging enough target group children,
asking the right questions etc.

58. Usability Evaluation
 SUS (system Usability
Scale) adapted to games
 system -> game
 use -> play
Results: avg SUS score
Tutorial group = 61.25
(δ=8.95)
Non-Tutorial group = 60
(δ=19.94)
Not statistically significant
p=0.84) considering a
confidence interval of
95%.
# Question p
1 I think I would like to play this game frequently.
0.78
2 I found the game unnecessarily complex.
0.18
3 I thought the game was easy to play.
0.20
4 I think I would need the support of a teacher or other
expert to be able to play this game.
0.47
5 I found various functions in this game were well
integrated.
0.13
6 I thought there was too much inconsistency in this
game.
0.37
7 I would imagine that most people would learn to play
this game very quickly.
0.45
8 I found the game very awkward to use.
0.04
9 I felt very confident playing the game.
0.13
10 I needed a lot of help before I could get to play this
game.
0.16

59. SUS Results Comparison
 The only statistically significant difference is for question
eight – I found the game awkward to use (p=0.04) for a
confidence interval of 95%.

60. Summary
 Training mission needed to reduce drop out
rates in children unfamiliar with the IDS
games
 It worked: 1st evaluation confirmed
satisfaction, were not put off, ease of use
 2nd evaluation confirmed
◦ USABILITY with/without training similar (SUS
score)
◦ Kids who played with training found the game less
awkward to use
 NOT SURPRISING RESULTS – evidence we
achieved our goal 

61. Number of games users per country:
Junior Games Senior Game

62. Number of website visitors, page views and
resource downloads per month since January 2008

63. Media Coverage
 Invited Speaker – “Idea Champion” at BMJ Panel:
◦ "The idea most likely to make the biggest impact on healthcare by 2020" (the NHS Innovation
Expo 2011, Excel, London, March 2011)
◦ ECDC Invited Speaker European meeting on “Epidemic Intelligence” (Stockholm, October 2010)
 BMJ feature
◦ Can Twitter predict disease outbreaks?
◦ BMJ 2012; 344 doi: 10.1136/bmj.e2353 (Published 17 May 2012
 Media Coverage
◦ “Epidemic Inteligence”Scientific Film
◦ BMJ: Medical Innovation: Sabreena Malik
◦ Social networking sites can help prevent pandemics:
(AFP) (15th December 2010) and replicated across more than
30 online news outlets
63
“Patty Kostkova and her colleagues at City ehealth Research
Centre, City University, London, showed that the 2009 H1N1 flu
outbreak could have been identified on Twitter one week before
it emerged in official records from general practitioner reports
“E-Bug uses
interactive
games to teach
children about
infection control”

64. CeRC staff – THANKS!
CeRC
Gawesh Jawaheer
Gayo Diallo
Sue Wiseman
David Farrell
Steve D’Souza
Gemma Madle
Jane Mani-Saada
Anjana Roy
Julius Weinberg
Mike Catchpole
Faiza Hansraj
Nancy Lai
Sandy Beverage
John Lawrenson
Martin Szomszor
Lisa Lazareck
DasunWeerasinghe
Ros Nyugi
Christina Daly
Ed de Quincey
Helen Oliver
David Fowler
Simon Hammond
UCL: Stephan Garbin
Justin Moser, Wendy Pan
64

65. Contact
 Thank you to CeRC, ISI Foundation and
UCL
 p.kostkova@ucl.ac.uk
 Sites:
◦ www.neli.org.uk
◦ www.edugames4all.org
◦ www.nric.org.uk
◦ www.femwiki.com (hosted by ECDC)