CoCites is a new search tool for scientific literature. The method uses citations to identify related articles. This presentation shows how the method works, how it was investigated, and how it can be used for reviews and meta-analyses

1. Using CoCites for searching scientific literature
systematic reviews and meta-analyses
A. Cecile J.W. Janssens, PhD
Research professor of epidemiology
Rollins School of Public Health, Emory University
@cecilejanssens

2. Disclosure
• I developed CoCites
• I hold a patent on the search method behind it

3. Finding literature on ‘new’ topics
• Which keywords should I use?
• Did my search retrieve all relevant articles on the topic?
• Is the retrieved article(s) a key article on the topic?
Researchers never sure about these answers
We face this problem with every article and every grant we write
Missing relevant articles suggests lack of knowledge and may impact peer review

5. Searching literature
Best-guess
keywords
Find some
relevant articles
Exhaustive
query
Find ‘all’
relevant articles
Related Articles in
PubMed, WOS, Scopus

7. Screening citations
• Seems intuitive, as researchers cite
articles related to their topic
• Studies show little value of searching
citations after keyword search
• Without exceptions, these looked at
– Direct citations, mostly backward
(references only)

8. Example
Imagine you are writing a review on exercise in women with
gestational diabetes, and your best guess search finds:
What other articles are published on exercise in women with are
pregnant and have diabetes?

9. 22 articles according to reference list
152 citations in Google Scholar,
65 in Web of Science, 51 in NLM,
22 in PubMed

10. Google Scholar / Google Erudito / Google Tutkija / Google Studie

11. CoCites: Retrieving articles that are cited together
Higher ranking
=
Co-cited more frequently
with search article
=
More similar to search article

13. In 2018: cited 22 times In 2020: cited 51 times

14. Title
1 Resistance exercise and glycemic control in women with gestational diabetes mellitus
2 Resistance exercise decreases the need for insulin in overweight women with gestational diabetes mellitus
3 Effect of treatment of gestational diabetes mellitus on pregnancy outcomes
4 A Multicenter, Randomized Trial of Treatment for Mild Gestational Diabetes.
5 A comparison of glyburide and insulin in women with gestational diabetes mellitus
6 Hyperglycemia and adverse pregnancy outcomes
7 Metformin versus insulin for the treatment of gestational diabetes
8 RCT of diet versus diet plus cardiovascular conditioning on glucose-levels in gestational diabetes
9 Effects of a partially home-based exercise program for women with gestational diabetes
10 IADP Study Groups Recommendations on the Diagnosis and Classification of Hyperglycemia in Pregnancy
11 Type 2 diabetes mellitus after gestational diabetes: a systematic review and meta-analysis
12 A walking intervention improves capillary glucose control in women with gestational diabetes mellitus
13 Guidelines of the ACOG for exercise during pregnancy and the postpartum period
14 Regular Exercise During Pregnancy to Prevent Gestational Diabetes A Randomized Controlled Trial
15 EXERCISE IN GESTATIONAL DIABETES - AN OPTIONAL THERAPEUTIC APPROACH
16 Summary and recommendations of the Fifth Int. Workshop-Conference on Gestational Diabetes Mellitus
17 Home-Based Exercise Training Improves Capillary Glucose Profile in Women with Gestational Diabetes
18 CRITERIA FOR SCREENING-TESTS FOR GESTATIONAL DIABETES
19 The effect of maternal exercise during pregnancy on abnormal fetal growth
20 Pre-gravid physical activity and reduced risk of glucose intolerance in pregnancy: the role of insulin sensitivity
21 Aerobic exercise and submaximal functional capacity in overweight pregnant women - A randomized trial
22 Gestational diabetes: The consequences of not treating
23 A lifestyle intervention of weight-gain restriction: diet and exercise in obese women with gestational diabetes
24 Exercise: The alternative therapeutic intervention for gestational diabetes
25 Gestational diabetes and the incidence of type 2 diabetes - A systematic review
Yellow: articles on interventions in gestational diabetes / insulin during pregnancy
In red: articles on exercise
Title
2 Resistance exercise decreases the need for insulin in overweight women with gestational diabetes mellitus.
14 Regular exercise during pregnancy to prevent gestational diabetes: a randomized controlled trial.
9 Effects of a partially home-based exercise program for women with gestational diabetes.
12 A walking intervention improves capillary glucose control in women with gestational diabetes mellitus
8 Randomized trial of diet versus diet plus cardiovascular conditioning on glucose levels in gestational diabetes.
Physical activity before and during pregnancy and risk of gestational diabetes mellitus: a meta-analysis.
Prevention of gestational diabetes: feasibility issues for an exercise intervention in obese pregnant women.
Exercise during pregnancy improves maternal glucose screen at 24-28 weeks: a randomised controlled trial.
No effect of the FitFor2 exercise programme on ... in pregnant women …at risk for gestational diabetes
15 Exercise in gestational diabetes. An optional therapeutic approach?
Hyperglycemia and adverse pregnancy outcomes.
Exercise during pregnancy and gestational diabetes-related adverse effects: a randomised controlled trial.
Simple lifestyle recommendations and the outcomes of gestational diabetes. …
Effect of exercise intensity and …in pregnant women at low and high risk for gestational diabetes.
17 Home-based exercise training improves capillary glucose profile in women with gestational diabetes.
Exercise training in pregnancy reduces offspring size without changes in maternal insulin sensitivity.
ACOG: exercise during pregnancy and the postpartum period.
23 A lifestyle intervention of weight-gain restriction: diet and exercise in obese women with gestational diabetes.
Home-Based Exercise Improves Fitness and Exercise Attitude and Intention in Women with GDM.
Resistance exercise training during pregnancy and newborn's birth size: a randomised controlled trial.
16 Summary and recommendations of the Fifth Int. Workshop-Conference on Gestational Diabetes Mellitus.
Effect of treatment of gestational diabetes mellitus on pregnancy outcomes.
The effect of an antenatal physical exercise programme on …in overweight and obese pregnant women.
13 Guidelines of the ACOG for exercise during pregnancy and the postpartum period.
The effects of mindfulness eating and yoga exercise on … of pregnant women with gestational diabetes.
Top 25 co-citations in 2018 Top 25 co-citations in 2020

15. From pilot study: studies that were ‘missed’
• Non-English publications:
Often not cited, sometimes because they are not retrievable through Medline/Embase (but through e.g.,
Linahl and Koreamed)
• Conference abstracts:
Have no citations, generally not cited (only when later published as full article), often not available in
Medline/Embase
• Dissertations, ‘grey’ literature:
Not retrievable through Medline/Embase
• Newer publications:
that are not yet cited, hence not co-cited
• Older publications:
that are no longer cited by newer studies

16. Finding recent and older studies

17. CoCites method: two steps
1 2
Janssens & Gwinn, BMC Med Res Methodol 2015

18. Search procedure

19. Does it work? How would you investigate?
Reproducing literature searches of 250 systematic reviews:
If we knew the 2 highest-cited articles, can we use these to find all
other included articles while screening fewer?

20. Performance metrics
Sensitivity (did we find all articles in the review?):
Number of retrieved studies
Number of studies included in published meta-analysis
Efficiency (did we screen fewer articles than the authors):
Number of screened articles
Number of screened articles in published meta-analysis

21. Did CoCites require screening fewer articles?
Published reviews: median 794 articles (IQR 273, 2132) per review
CoCites: median 873 articles (IQR 540, 1204) per review

22. Did CoCites retrieve all included articles?
Median 75% of articles (IQR 50, 90) retrieved per review

23. Even when method did ‘not’ work …
Title Frequency
Ultrasonographic characteristics as a criterion for repeat cytology in benign thyroid nodules 7
Revised American Thyroid Association Management Guidelines for Patients with Thyroid Nodules and Differentiated Thyroid Cancer 5
Contribution of scintigraphy and ultrasonography to the prediction of malignancy in thyroid nodules with indeterminate cytology 4
The Bethesda System for Reporting Thyroid Cytopathology 3
The thyroid nodule 3
Fine-Needle Biopsy Should Be Performed in Solid Hypoechoic Thyroid Nodules Greater Than One Centimeter 3
Preoperative Diagnosis of Benign Thyroid Nodules with Indeterminate Cytology 3
Natural history of benign solid and cystic thyroid nodules 3
Role of repeated fine-needle aspiration of thyroid nodules with benign cytologic features. 3
How to combine ultrasound and cytological information in decision making about thyroid nodules 3
Repeated fine-needle aspiration of the thyroid in benign nodular thyroid disease: Critical evaluation of long-term follow-up 3
Value of repeat ultrasound-guided fine-needle aspiration in nodules with benign cytological diagnosis 3
Usefulness of repeated fine-needle cytology in the follow-up of non-operated thyroid nodules 3
Risk of malignancy in nonpalpable thyroid nodules: Predictive value of ultrasound and color-Doppler features 2
Current approaches to primary therapy for papillary and follicular thyroid cancer 2
Increasing incidence of thyroid cancer in the United States, 1973-2002 2
SPECTRUM OF THYROID DISEASE IN A COMMUNITY - WHICKHAM SURVEY 2
Thyroid incidentalomas: Management approaches to nonpalpable nodules discovered incidentally on thyroid imaging 2
Repeat thyroid nodule fine-needle aspiration in patients with initial benign cytologic results 2
SIGNIFICANCE OF NONTOXIC THYROID NODULES 2
Value of re-aspirations in benign nodular thyroid disease 2
Initially non-diagnostic ultrasound-guided fine needle aspiration cytology of thyroid nodules: value and management 2
The Role of [F-18]-2-Fluoro-2-Deoxy-D-Glucose-Positron Emission Tomography in Thyroid Nodules With Indeterminate Fine-Needle Aspiration Biopsy
Systematic Review and Meta-Analysis of the Literature
2
How to Manage Thyroid Nodules With Two Consecutive Non-Diagnostic Results on Ultrasonography-Guided Fine-Needle Aspiration 2
Value of repeated fine needle aspirations of the thyroid: An analysis of over ten thousand FNAs 2
Yellow = search article; Blue is cited in review but not included in its analysis (just additional citation)

24. … it worked
Title Frequency
Ultrasonographic characteristics as a criterion for repeat cytology in benign thyroid nodules 7
Revised American Thyroid Association Management Guidelines for Patients with Thyroid Nodules and Differentiated Thyroid Cancer 5
Contribution of scintigraphy and ultrasonography to the prediction of malignancy in thyroid nodules with indeterminate cytology 4
The Bethesda System for Reporting Thyroid Cytopathology 3
The thyroid nodule 3
Fine-Needle Biopsy Should Be Performed in Solid Hypoechoic Thyroid Nodules Greater Than One Centimeter 3
Preoperative Diagnosis of Benign Thyroid Nodules with Indeterminate Cytology 3
Natural history of benign solid and cystic thyroid nodules 3
Role of repeated fine-needle aspiration of thyroid nodules with benign cytologic features. 3
How to combine ultrasound and cytological information in decision making about thyroid nodules 3
Repeated fine-needle aspiration of the thyroid in benign nodular thyroid disease: Critical evaluation of long-term follow-up 3
Value of repeat ultrasound-guided fine-needle aspiration in nodules with benign cytological diagnosis 3
Usefulness of repeated fine-needle cytology in the follow-up of non-operated thyroid nodules 3
Risk of malignancy in nonpalpable thyroid nodules: Predictive value of ultrasound and color-Doppler features 2
Current approaches to primary therapy for papillary and follicular thyroid cancer 2
Increasing incidence of thyroid cancer in the United States, 1973-2002 2
SPECTRUM OF THYROID DISEASE IN A COMMUNITY - WHICKHAM SURVEY 2
Thyroid incidentalomas: Management approaches to nonpalpable nodules discovered incidentally on thyroid imaging 2
Repeat thyroid nodule fine-needle aspiration in patients with initial benign cytologic results 2
SIGNIFICANCE OF NONTOXIC THYROID NODULES 2
Value of re-aspirations in benign nodular thyroid disease 2
Initially non-diagnostic ultrasound-guided fine needle aspiration cytology of thyroid nodules: value and management 2
The Role of [F-18]-2-Fluoro-2-Deoxy-D-Glucose-Positron Emission Tomography in Thyroid Nodules With Indeterminate Fine-Needle Aspiration Biopsy
Systematic Review and Meta-Analysis of the Literature
2
How to Manage Thyroid Nodules With Two Consecutive Non-Diagnostic Results on Ultrasonography-Guided Fine-Needle Aspiration 2
Value of repeated fine needle aspirations of the thyroid: An analysis of over ten thousand FNAs 2
Yellow = articles on same topic as search article, in red same technology
These articles were just not included in the meta-analysis, (for reasons we do not know)

25. Did CoCites retrieve all included articles?
Related to method:
• Number of citations for search articles
Related to our study design:
• Random selection of review,
minimally evaluated for quality
• Use of ‘other’ databases  articles
not in WOS/PubMed
– Percentage of articles in PubMed
• Heterogeneous topics (apples and
oranges)  search articles not on
same topic
– Similarity score: degree of co-citation
(citation overlap) between query articles
Median 75% of articles (IQR 50, 90) retrieved per review

26. Percentage of articles retrieved
Overall: median 75%

27. Percentage of articles retrieved
Overall: median 75%
Excluding ‘poor reviews’, CoCites retrieves 88% of articles that could have been
retrieved through PubMed

28. Other findings
• 88% retrieved articles represented 91% of the data
• Highest-cited articles, if not known in advance, are easily found by using CoCites for
infrequently-cited articles
Highest-cited  more likely frequently co-cited  rank high in results
• Strategy to reduce number of screened articles: filter on similarity score
– Preliminary findings: extremely promising

29. Results for alternative search articles

30. Across synonyms
Method is based on expert evaluations
Experts decide which articles to cite
(together)
Even if terminology is different

31. Notes for use (1)
• Searching highly-cited articles,
tends to return highly-cited articles
• Commentaries and reviews are
cited for diverse reasons  results
are diverse as well, with highly-
cited articles

32. Notes for use (2)
If you search multiple articles at the
same time:
1. Make sure articles are on same topic
Like: if you had to cite one, you could
pick any of them
2. Articles that are cited more often
than others ‘dominate’ the search
results ( limit number of citations
per article)
3. More citing articles is not evidently
better; too many makes it inefficient
( a few 100 seems enough)
 CoCites

33. Notes for use (3)
Citation search needs larger set of search
articles. Perform co-citation search first
• Start with 1 or 2 articles that meet your
search criteria, that are what you want
to find
• Perform co-citation search and screen
the results
• Add relevant articles to search set
• (Repeat co-citation search, if pertinent)
• Perform citation search on expanded set

34. Notes for use (4)
Still beta-version, more to come, e.g.:
• Sort/filter results
• Download to citation manager
Current solution:
• Export search results to excel
• Select and copy PMID
• Paste in PubMed
• Send results to citation manager

35. CoCites: one limitation and many strengths
• Missed studies: mostly abstracts, foreign articles and smaller studies
• Sensitive: retrieves all ‘major’ articles almost without exception
• Often more efficient than keyword-based searching, especially when no uniform
terminology; also: screening most relevant studies first
• Dependable: when search articles are cited frequently enough
• Simple: less dependent on expertise/experience of (re)searcher
• Intuitive and transparent: understandable what method is doing
• Reproducible: same search articles on same date = same results