How well do Voice Assistants answer Consumer Health Questions?
1.
‘Alexa, are vaccines safe?’
Evaluating voice assistant responses to
consumer health questions
Emily Alagha, MSIS, AHIP - Clinical Information Specialist & Data
Management Librarian - Georgetown University Medical Center
Rachel Helbing, MLIS, MS, AHIP - Director of Library Services for
Health Sciences - University of Houston
14.
Data Collection
1. Set up iPads
2. Asked & recorded answers
3. Scored with rubric
4. Calculated results
15.
Siri
5.16 5.10
0.98
Google Assistant Alexa
Mean
Scores
16.
Voice Assistant Alexa Google Assistant Siri
VA provided same
response to both
researchers
62.96% 70.37% 68.52%
VA understood question
and provided answer
25% 94% 94%
Average spoken words
per answer
29 21 13
Most cited source Wikipedia.org CDC.gov CDC.gov
17.
Limitations
• Data collected in 2018; may not reflect recent algorithm
updates
• Scoring rubric has not been assessed for reliability and
may need future refinements
• May not reflect real use cases that incorporate tailored
search results
• Overall scores for all assistants may have been lower if
prioritizing audio only output
21.
Conclusions
• Siri and Google provided evidence-based links; little
audio output
• Alexa was worst at natural language comprehension;
high audio output
• Many information providers are entering voice
information provision via traditional websites and
third-party apps
• Information professionals should push for better
usability and increased transparency about health
information partnerships