1.
‘Alexa, are vaccines safe?’
Evaluating voice assistant responses to
consumer health questions
Emily Alagha, MSIS, AHIP - Clinical Information Specialist & Data
Management Librarian - Georgetown University Medical Center
Rachel Helbing, MLIS, MS, AHIP - Director of Library Services for
Health Sciences - University of Houston

2.
Objectives
1. Background
2. Purpose
3. Data collection
4. Results
5. Clinical implications

3.
Voice
Assistants

4.
Siri
Google Assistant
Alexa

5.
Online Misinformation

6.
Vaccination rates

7.
Outbreaks

8.
Online Misinformation
Vaccination rates
Outbreaks

9.
Quality & accuracy
Siri
Google Assistant
Alexa
vaccine safety & use

10.
Smart speaker brand owned
©2019MintelGroupLtd.AllRightsReserved.
ConfidentialtoMintel.
Read on mintel.com
Source:
Base:
Fieldwork:
554 internet users aged 18+ who own a smart speaker
March 2019 Lightspeed/Mintel
What brands of smart speakers do you own? Please select all
that apply.
72%
30%
14%
6%
5%
1%
1%
0% 25% 50% 75% 100%
Amazon
Google
Apple
JBL
Sonos
I don't know
Other, please specify
14

11.
Selecting Test Questions
The Centers for Disease
Control (CDC) immunization
FAQ page and the CDC infant
immunization FAQ page

12.
AnswerThePublic.com
search for ‘vaccines’

13.
Test Questions
54 total

14.
Data Collection
1. Set up iPads
2. Asked & recorded answers
3. Scored with rubric
4. Calculated results

15.
Siri
5.16 5.10
0.98
Google Assistant Alexa
Mean
Scores

16.
Voice Assistant Alexa Google Assistant Siri
VA provided same
response to both
researchers
62.96% 70.37% 68.52%
VA understood question
and provided answer
25% 94% 94%
Average spoken words
per answer
29 21 13
Most cited source Wikipedia.org CDC.gov CDC.gov

17.
Limitations
• Data collected in 2018; may not reflect recent algorithm
updates
• Scoring rubric has not been assessed for reliability and
may need future refinements
• May not reflect real use cases that incorporate tailored
search results
• Overall scores for all assistants may have been lower if
prioritizing audio only output

18.
Implications: Web Search

19.
Implications: Third Party Apps

20.
Implications: Third Party Apps

21.
Conclusions
• Siri and Google provided evidence-based links; little
audio output
• Alexa was worst at natural language comprehension;
high audio output
• Many information providers are entering voice
information provision via traditional websites and
third-party apps
• Information professionals should push for better
usability and increased transparency about health
information partnerships

22.
References
1. https://about.ads.microsoft.com/en-us/insights/2019-voice-report
2. https://pixabay.com/photos/voice-control-voice-commands-2598422/
3. Brand icons from https://www.flaticon.com
• https://www.flaticon.com/authors/simpleicon (Google)
• https://www.freepik.com (Apple & Amazon)
4. https://www.pewresearch.org/fact-tank/2013/02/01/majority-of-adults-look-online-for-health-
information/
5. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194811
6. https://www.ncbi.nlm.nih.gov/pubmed/26974260
7. https://www.nature.com/articles/d41586-018-07034-4
8. https://www.washingtonpost.com/national/health-science/percentage-of-young-us-children-
who-dont-receive-any-vaccines-has-quadrupled-since-2001/2018/10/11/4a9cca98-cd0d-11e8-
920f-dd52e1ae4570_story.html?utm_term=.f3d142106107

23.
References
9. https://www.cdc.gov/measles/cases-outbreaks.html
10. https://9to5google.com/2019/04/29/google-assistant-siri-market-share/
11. https://www.healthcareitnews.com/news/uchealth-launches-alexa-based-virtual-assistant-
voice-enabled-patient-engagement
12. https://www.theguardian.com/media/2019/aug/27/bbc-to-launch-alexa-rival-that-will-grasp-
regional-accents
13. https://www.theguardian.com/society/2019/jul/10/nhs-teams-up-with-amazon-to-bring-alexa-
to-patients
14. https://www.wired.com/story/amazon-alexa-search-for-the-one-perfect-answer/
15. https://earthtoplanet.com/the-future-of-medicine/