This document describes a user study conducted to understand how medical experts search medical literature. 46 medical experts performed search tasks on 3 different systems using topics provided by a medical library. The systems varied from a basic Boolean system to one incorporating topic modeling. The topic modeling system was found to be the most difficult to use. Experts issued fewer queries for familiar topics and viewed fewer results. The study aims to help improve search tools for medical literature based on expert search behavior.

1. Search in Medical Text
Sarvnaz Karimi
National ICT Australia (NICTA)
The University of Melbourne
1 / 51

2. “What makes medical doctors use computers?”
2 / 51

3. Medicine and Computer Science
Data: Users:
biomedical literature biomedical researchers, medical
doctors/students, curators
clinical records hospital staff, medical doctors
medical social media drug companies, health authorities
3 / 51

4. Challenges for Computer Scientists
Data: Challenges:
biomedical literature creation of systematic reviews,
experts searching in the literature
clinical records search in medical records
medical social media discovery of drug side-effects
4 / 51

5. 1Systematic Reviews:
A Complex Search Episode for Evidence Based
Policy and Practice
5 / 51

6. A long term smoker with chronic obstructive air-
ways disease (COPD) who has recently quit
smoking has breathing difficulties. What are the
suitable non-drug therapies to improve the pa-
tient’s breathing?
(example by Prof. Paul Glasziou)
6 / 51

7. Is adjunctive vitamin A effective in children
diagnosed with non-measles pneumonia?
(Cochrane collaboration)
7 / 51

8. A clinician applying research to practice needs to
know:
What? interventions match the patient’s conditions
What? quality of evidence and applicability
What? duration, dosage, ...
8 / 51

9. Growth of medical scientific literature archive
(MEDLINE)
9 / 51

10. Evidence-Based Medicine (EBM)
Background Information/Expert Opinion
Randomized Controlled Trials (RCTs)
Critically Appraised Individual Articles
Critically Appraised Topics
Systematic
Reviews
Cohort Studies
Case−controlled Studies
Information
Filtered
Unfiltered
Information
QualityofEvidence
EBM applies the best available evidence to clinical decision-making.
10 / 51

11. A sample systematic review
Title: Vitamin A for non-measles pneumonia in
children
Main question: Is adjunctive vitamin A effective
in children diagnosed with non-measles pneumo-
nia?
Inclusion criteria: Only parallel-arm, randomized
controlled trials (RCTs) and quasi-RCTs, in which
children (younger than 15 years of age) with non-
measles pneumonia were treated with adjunctive
vitamin A, were included...
Methods: We searched The Cochrane Library,
Cochrane Central Register of Controlled Trials
(CENTRAL 2010, issue 3) which contains the
Acute Respiratory Infections Group’s Specialised
...
Main results: Six trials involving 1740 children
were included. There was no significant reduc-
tion in mortality...
11 / 51

12. Systematic reviewing process
develop criteria for including studies
Define a clear review question and
Systematic review
Presenting the results, interpreting
the findings, and drawing
conclusions
?
Search
Selecting studies and collecting data
undertaking meta−analysis
Analysing the data and
12 / 51

13. A sample MEDLINE query
1. exp vitamin A/
2. vitamin A.mp
3. retinol.mp
4. exp dietary supplements/
5. or/1-4
6. exp pneumonia/
7. pneumonia$.mp
8. exp pneumonia, bacterial/
9. exp pneumonia, lipid/
10. exp pneumonia, mycoplasma/
...
14. exp pneumonia, viral/
15. exp respiratory tract infections/
16. acute adj respiratory.mp
17. respiratory adj infection.mp
18. respiratory adj disease.mp
19. or/6-18
20. 5 and 19
13 / 51

14. A sample MEDLINE query
1. exp vitamin A/
2. vitamin A.mp
3. retinol.mp
4. exp dietary supplements/
5. or/1-4
6. exp pneumonia/
7. pneumonia$.mp
8. exp pneumonia, bacterial/
9. exp pneumonia, lipid/
10. exp pneumonia, mycoplasma/
...
14. exp pneumonia, viral/
15. exp respiratory tract infections/
16. acute adj respiratory.mp
17. respiratory adj infection.mp
18. respiratory adj disease.mp
19. or/6-18
20. 5 and 19
13 / 51

15. Scale of evidence inclusion
Documents
to be read in
full−text
To be actually
included in the
review
(500−2000)
Boolean Query Output
(4,000 −− 10,000)
Title & Abstract
(10−100)
14 / 51

16. Where can we help?
Our contributions on introducing ranked retrieval is published in:
* S. Karimi, S. Pohl, F. Scholer, L. Cavedon, J. Zobel, Boolean versus Ranked Querying for
Biomedical Systematic Reviews, BMC Medical Informatics and Decision Making, Vol 10,
Number 58, 2010
* D. Martinez, S. Karimi, L. Cavedon, T. Baldwin, Facilitating Biomedical Systematic Reviews
Using Ranked Text Retrieval and Classification, ADCS 2008, December 2008
15 / 51

17. To assist in query formulation for an initial search
strategy
Suggesting key-terms and synonyms e.g neoplasm for cancer
Bag-of-words to Boolean Suggesting structure to specified query
terms. Template queries already exist for limited inclusion criteria.
16 / 51

18. Consistency verification
Automatic verification against inclusion criteria
Automatic self-consistency verification: If a reviewer selects one
document, but later chooses to ignore a similar one, the system
should flag this possible inconsistency.
17 / 51

19. Dynamic relevance feedback
Document selection process is currently paper-based.
A dynamic relevance feedback approach that is active during the
document selection process could rank the remaining documents
based on estimated importance.
Dynamic relevance feedback might identify additional documents
that exist in the collection but were missed by the initial search
strategy.
18 / 51

20. Analysis and Meta-analysis
There are tools that assist analysing already extracted numerical data
from one or multiple studies, but the input to these tools should first be
extracted manually from text. Automatic information extraction can
save hours.
19 / 51

21. Review update
Updating the review with new evidence so that it remains relevant.
Treatment X works. Treatment Y is preferred over X.
Year 2005 Year 2010
20 / 51

22. Literature survey is hard!
21 / 51

23. 2User-Study:
Medical Expert’s Search Behavior
22 / 51

24. Subject: Library needs your help
Volunteers needed
Study : Improving Tools for Searching Medical Literature
(Alfred Health Ethics Committee approved)
Dear All, I am writing to you as a participant in a Library training class at the Ian Potter
Library in 2010... probably realise that systems for online searching are often complex
and not that easy to use...The Ian Potter Library is participating in a study together with
NICTA (University of Melbourne) and RMIT, looking at ways of improving search tools
for medical literature (see attached). We need volunteers..
Volunteers required - Improving tools for searching medical literature
This study aims to improve quality of search results in the biomedical domain. The
research team needs participants with (bio)medical background, especially medical
students/researchers, to carry out search tasks using search tools. The session will
take about 40 minutes. All participants receive movie vouchers. Alfred Hospital HREC
number 22/10. Further information...
23 / 51

25. Why user study?
How should a biomedical search engine look like?
What are the needs of specific users of biomedical search tools?
• Users’ behaviour, searching and querying style,...
Which one of our proposed systems is more effective?
24 / 51

26. Subjects
Experts: educational background in biomedical sciences and
related domains.
Non-experts: absolutely no education or working experience in
biomedical domains.
We recruited 46 experts of which 2 were assigned to a pilot study, and
6 did not finish the tasks, and also recruited 9 non-experts.
25 / 51

27. User study format
Subjects were asked to imagine that they should write a short report
about each given topic. Their goal was to carry out searches to find
useful articles that they would want to read in order to prepare their
report.
Each subject was asked to complete the following:
1. Opening questionnaire
2. Search phase, consisting of six tasks. For each task:
• Pre-task questionnaire to establish prior familiarity with topic
• Search for useful documents
• Post-task questionnaire about search experience
3. Closing questionnaire
26 / 51

28. Tasks assigned to the subjects
1 exercise therapy for cystic fibrosis
2 families and grief in the ICU
3 cognitive behaviour therapy for postnatal depression
4 vitamin D and dementia
5 ankle injuries and gait analysis
6 prevention of type 2 diabetes in developing countries
These topics were previously referred to health librarians
in Ian Potter library of Alfred Hospital in Melbourne, by
either students or staff.
27 / 51

29. Search systems and interfaces
System A: A Boolean retrieval system similar to PubMed. Results
were ordered by date. Very complicated multi-line Boolean
querying was supported.
System B: A combination of ranked and Boolean system. Both
ranked and Boolean querying were supported. If a query was
Boolean, the output was ranked based on the keywords.
System C: Topic modelling based system. The output of the
queries were topic modelled (LDA) and then ranked under each
topic.
28 / 51

30. Preferred system and difficulty of using the systems
Only a slight difference between A and B (not-ranked and
ranked), but C (topic-modelled) was significantly less liked.
Between A and B, the ranked results of system B were slightly
but significantly better liked.
System C (topic modelling) was rated hardest to work with.
29 / 51

31. Topic Familiarity and its effect on querying
Tasks Queries Ranked Boolean Complex Total query
entered queries queries Boolean terms
Not familiar 147 438 (3.0) 154 (1.0) 271 (1.8) 13 (0.1) 1840 (13.2)
Familiar 71 204 (3.0) 51 (0.7) 148 (2.1) 5 (0.1) 960 (15.9)
Very familiar 10 14 (1.4) 5 (0.5) 9 (0.9) 0 (0.0) 92 (9.2)
p-value 0.0172 0.0184 0.0334 0.6511 0.001
* The table shows the sum for each category, with the mean indicated in parentheses.
The number of queries entered varied with their level of topic familiarity.
More queries for topics that subjects were not familiar with.
The number of ranked or Boolean queries employed by searchers varies significantly with
the level of familiarity.
For very familiar topics, users employ fewer query terms.
30 / 51

32. Familiarity, visited result pages, and documents
selected as relevant
Tasks Result pages Items
viewed saved
Not familiar 147 494 (3.4) 999 (6.8)
Familiar 71 253 (3.6) 425 (6.0)
Very familiar 10 28 (2.8) 57 (5.7)
p-value 0.2801 0.0535
No significant relationship was found between prior familiarity and the number
of result pages viewed.
The number of items saved (relevant) did not vary significantly with topic
familiarity.
31 / 51

33. Familiarity based on the pre-task questionnaire and
Difficulty based on post-task questionnaire
Difficulty
Easy Medium Hard
Not familiar 78 44 25 147
Familiar 44 21 16 81
122 65 41
p-value=0.7678
No relation was confirmed between familiarity and perceived difficulty
of working with the systems, in other words being familiar did NOT
make the task easier or harder.
32 / 51

34. 3Drug Side-Effects:
What Do Patient Forums Reveal?
33 / 51

35. Drug side-effect
A drug side-effect is an effect (positive or negative)
that is secondary to the one intended.
Some side-effects are severe, such as organ failure,
high blood sugar, stroke, heart disease, neuropathy,
and some are mild, such as nausea, and dizziness.
Adverse side-effects that are unknown claim many
lives each year.
34 / 51

36. Side-effect discovery
or demand
+
Volunteers
Clinical trials
35 / 51

37. Post-marketing Surveillance
Clinical trials are expensive, sometimes out-dated, time
consuming, and often small-scale.
Professionals and drug users can report mostly severe
side-effects in official web-sites.
Patient social networks and forums – such as DailyStrength, and
AskPatient – collect feedback directly from drug consumers.
Data in such forums may be of questionable reliability, but it
provides indications of real side-effects, both mild and severe.
36 / 51

38. A new era in side-effect discovery
or demand
+Clinical trials
Volunteers
update
feedback
37 / 51

39. Trade-off in using data from social media
Advantages:
large amount of data
data generated by a large variety of people who share
information through personal blogs and public forums.
Disadvantages:
(Medical social data is difficult to access and process)
data is scattered over multiple sources.
availability of useful resources is limited (ownership).
data often contains noise (informal language, or mis-spelled
specialised terms) so traditional methods for pre-processing such
as POS tagging, chunking, and sentence segmentation may not
work well.
38 / 51

40. What you may see in a medical forum
User A Side effects from MedicineX therapy?
Post 1 . . . Since taking MedicineX for about 3 years, some time in the last
year or so I began to experience significant ringing in the ears. . . .
User B Re: Side effects from MedicineX therapy?
Post 2 I haven never heard about it. But I had nausea, vomiting and fever.
User C Re: Side effects from MedicineX therapy?
Post 3 It is not true at all. MedicineX is one of medicines which have least
side-effects. In fact, my heart related symptoms became better.
User D Re:Re: Side effects from MedicineX therapy?
Post 4 I didn’t have nausea or vomiting but had a skin rash for a few days.
User E Warning!!! BLOOD CLOTHS IN MY LUNG!!!
Post 5 After using MedicineX for 3.5 years, my doctor found a blood cloths
in my lung . . .
User A Thank you
Post 6 thx. My doctor told me my ear ringing was not MedicineX but . . .
39 / 51

41. Everybody’s different
All previous studies are focused on
extracting mentions of adverse effects and
mostly ignore the contributing factors that
are patient-dependent.
We are interested in extracting both
adverse and beneficial side-effects along
with background information on the
patients that could contribute to their
positive or negative experience.
This is particularly interesting because
clinical trials do not cover all possible patient
conditions.
40 / 51

42. Entities to be extracted
Entity Example
Disease “After 3 years of having Ativan keep the anxiety in check,
...”
Symptom “My heart was racing and ..”
Drug “I must be addicted to Xanax”
Duration “Began taking 5 mg daily(broke the 10mg pill in half) for
4 weeks”
Dosage “Began taking 5 mg daily ...”
Frequency “Began taking 5 mg daily ..”
Positive side-effect “I’m taking this for my back pain but it has been reducing
my stress as well.”
Negative side-effect “Sometimes causes drowsiness.”
Lack of negative side-effect “I feel dizzy and low but no vomiting.”
Lack of positive side-effect “I was feeling even more energetic initially but it doesnt
work like that any more”
Positive outcome “No apparent side effects thus far and results have been
very effective for the pain.”
Negative outcome “Problem is you build up a tolerance and eventually the
drug quits working as has been my case.”
Gender of patient “I was prescribed this for anxiety when my teenage
daughter was driving my wife and I into”
Age “I’m in my forties”
41 / 51

43. Relations to be extracted
Relation Description
Drug-Drug If a patient explicitly mentions that taking two named
drugs together had any effect or no effect, then the two
drugs are annotated by a positive, negative, or no ef-
fect relation.
MedicineA MedicineB... was fine till I started taking as well...
Dosage-Frequency The frequency in which a dosage is taken is annotated
by a for relation.
Dosage-Duration The prolong of intake for a specific dosage is annotated
with a for relation.
Drug-Dosage The dosage which a drug is taken is annotated with a
taken relation.
42 / 51

44. Data
We gathered data for ten different drugs from two different
forums: AskPatient2 and eHealth Forum3.
A total of 5,996 posts (40,871 sentences) was collected.
We only relied on free-text comments in each post.
The annotation is ongoing by two annotators.
2
http://www.askapatient.com/
3
http://ehealthforum.com/
43 / 51

45. A Survey:
What do people think about medicine and social
media?
44 / 51

46. Who participated?
# Participants Gender Age Range Education
Group A 83 61% M 2% under 21 57% G
39% F 83% 21-39 35% B
15% above 40 8% under
Group B 379 42% M 7% under 21 20% G
57% F 69% 21-39 7% B
24% above 40 73% under
All 462
Group A: survey posted on Facebook, e-health forum, and Yahoo health forum
Group B: Amazon Mechanical Turkers
B: bachelor degree, G: Graduate degree
M: Moderately, V: Very
45 / 51

47. How healthy our participants were? Do they trust their
doctors?
# Participants Healthy Trust Doctors
Group A 83 45% V 56% V
43% M 34% M
12% not 10% little/none
Group B 379 53% V 68% V
41% M 26% M
6% not 12% little/not
Group A: survey posted on Facebook, e-health forum, and Yahoo health forum
Group B: Amazon Mechanical Turkers
B: bachelor degree, G: Graduate degree
M: Moderately, V: Very
46 / 51

48. Do people use medical social networks, forums, blogs,
or medical information on Internet? Do people share
their experiences with drug side effects?
Generic Social Medical Social Int. search Trust Int. Share
Group A 83% M to E 24% yes 48% M to E 38% well 4% M to E
13% S 76% no 47% S 51% little 17% S
4% N 5% N 11% none 79% N
Group B 79% M to E 21% yes 56% M to E 50% well 31% M to E
14% S 79% no 39% S 38% little 30% S
7% N 5% N 12% none 54% N
Group A: survey posted on Facebook, e-health forum, and Yahoo health forum
Group B: Amazon Mechanical Turkers
N: never, S: sometimes, M: moderately often, E: extremely often
Not so healthy people share more than very healthy people (53% vs 38%).
47 / 51

49. What’s next
We propose finding patterns of side-effect reporting, both using
heuristics and automatically extracted rules. The outcome can be
used in enriching side-effect ontologies.
One of our contributions will be providing the research community
with a rich annotated collection that is large enough for
experimentation and diverse in the types of drugs and annotated
concepts.
The existing literature does not provide a comparison over
previous approaches, mainly due to lack of availability of a
standard and publicly accessible dataset. We intend to conduct a
comprehensive comparison of existing methods as well as our
own techniques.
48 / 51

50. Summary
There are many areas in medicine and health which can benefit from
more effective search in text:
Techniques used for extensive search in biomedical literature for
answering focused clinical questions (systematic reviewing) are
still way behind the state-of-the-art search technology.
Domain-experts search differently from laymen and biomedical
search engines should accommodate these differences.
Analysing medical social media is one method of capturing
previously undiscovered drug side-effects.
49 / 51

51. Expert Subjects (Opening questionnaire)
Category Number of Subjects
Gender female 27 (71%)
male 11 (29%)
Position allied health 12 (32%)
biomedical researcher 11 (29%)
medical student 9 (24%)
health librarian 3 (8%)
nurse 1 (3%)
Search tool used PubMed 33 (87%)
Google Scholar 31 (82%)
Ovid 22 (58%)
EBSCO 15 (40%)
Other 7 (18%)
Satisfaction very satisfied 3 (8%)
satisfied 30 (79%)
borderline 5 (13%)
unsatisfied 0 (0%)
50 / 51

52. Expert Subjects (Cont.)
Category Number of Subjects
Search tool usage daily 7 (18%)
weekly 14 (37%)
monthly 13 (34%)
rarely 4 (10%)
Database used Medline 30 (79%)
Journals@Ovid Full Text 17 (45%)
CINAHL 15 (35%)
Cochrane Systematic Reviews 12 (32%)
PsycINFO 11 (29%)
EMBASE 6 (16%)
AMED 5 (13%)
Other 12 (32%)
51 / 51