This is a talk given at The 1st Workshop on the Impact of Recommender Systems with ACM RecSys 2019. Mendeley Suggest is a popular academic paper recommender, serv- ing over 1.5M researchers in 2018. We attempt to assess the extent Mendeley Suggest helps its users in their research in two areas: help- ing researchers keep up with the most prominent development in the field and help researchers find relevant literature. Our findings indicate that the recommender significantly increases the chance that a user finds important research and decreases the amount of time she needs to spend on searching. We observe that the effect is much greater than the number of accepted recommendations and propose that it is due to an increase in reading activity that Mendeley Suggest recommendations spur. Time-series analyses are presented to back up this hypothesis. Our results highlight the potential of academic paper recommenders in furthering science.

1. The Impact of
Recommenders on
Scientific Article Discovery:
September 2019
Minh Le, Deep Kayal, Andrew Douglas
The Case of Mendeley Suggest

2. Agenda
1. About Mendeley and Mendeley Suggest
2. How to measure the impact of a scientific article recommender?
3. Measuring Mendeley Suggest’s impact
4. Conclusions
2

3. How many of you…
• Use Mendeley last month?

4. How many of you…
• Use Mendeley Suggest last month?

5. Mendeley – Mendeley Suggest
5
Website Desktop Mobile

6. Goals of scientific article discovery
6
High coverage Timeliness
As a researcher, I want to capture the relevant literature with…
Efficiency

7. Measuring coverage and timeliness
7
Relevant
literature
Relevant
literature
100%
Researcher B
Researcher A

8. Measuring coverage and timeliness
0%
10%
20%
30%
40%
50%
60%
70%
80%
Jan-19 Feb-19 Mar-19
Researcher A Researcher B
11

9. Measuring recommender impact
0%
10%
20%
30%
40%
50%
60%
70%
80%
Jan-19 Feb-19 Mar-19
Researcher A Researcher B
12
Experiment
group
Control
group

10. Measuring recommender impact
0%
10%
20%
30%
40%
50%
60%
70%
80%
Jan-19 Feb-19 Mar-19
Researcher A Researcher B
13
Users
Non-users

11. Groups of users
• Users (Jan 2018 - Jul 2019):
− “Heavy”: top 5% users who click
on the most recommendations
− “Frequent”: top 25%
− “Infrequent”: click on at least one
• Non-users: did not open any rec.
in the period but stay active on
Mendeley
14

12. Goal sets
• Optimal set of articles
to read is not known
• Approximation 1: most-
cited articles within a
field
• Approximation 2: cited
article per person
15

13. Highly cited articles
• Most-cited articles in
each ASJC code
• ASJC = All Science
Journal Classification
Codes (~300 in total)
• Examples:
• Biochemistry: “Directed
Evolution of Protein
Catalysts”
• Pharmacology, Toxicology
and Pharmaceutics (all):
“An updated overview on the
development of new
photosensitizers for
anticancer photodynamic
therapy”
16

14. Highly cited articles
• Most-cited articles in
each ASJC code
• ASJC = All Science
Journal Classification
Codes (~300 in total)
• Examples:
• Biochemistry: “Directed
Evolution of Protein
Catalysts”
• Pharmacology, Toxicology
and Pharmaceutics (all):
“An updated overview on the
development of new
photosensitizers for
anticancer photodynamic
therapy”
17

15. Citable articles (personalized)
• Articles that a person
cited during Jan 2018 –
Jul 2019
• Research is
collaborative: my co-
authors might cite an
article that I didn’t read
18

16. Citable articles (personalized)
• Articles that a person
cited during Jan 2018 –
Jul 2019
• Research is
collaborative: my co-
authors might cite an
article that I didn’t read
19
0
-2 years -1 year

17. Citable articles (personalized)
• Articles that a person
cited during Jan 2018 –
Jul 2019
• Research is
collaborative: my co-
authors might cite an
article that I didn’t read
• Stages of research:
“discovery”,
“development”,
“finalization”
20
0
-2 years -1 year

18. Direct and indirect effect
21
Opening ~2.5
recommendations
per week

19. Direct and indirect effect
• Indirect effects of Mendeley Suggest might include:
• Upon reading relevant paper, a researcher might follow forward and backward
citations to gain a more exhaustive understanding of her field.
• A researcher might discover a new topic by serendipity
• We study S-frequent users in days they use Suggest and days they do not
use Suggest
• We measure:
• Additions to Mendeley library
• Annotations on papers
22

20. Paper additions
23

21. Annotations
24

22. “The “Mendeley suggest”
feature has been very
helpful in finding further
articles related to what I am
working on”
25

23. “I like it as a way to scan
literature quickly, it’s mostly
useful for finding papers.
Then later when I get to the
office I read it.”
26

24. Conclusions
• We found encouraging effects of Mendeley Suggest on users:
− Better scientific article discovery: higher coverage, timeliness, and efficiency
− Higher engagement with papers
• Limit:
− Observational study
• Future work:
− Follow up on the impact of Mendeley Suggest
− Better recommender systems
27

25. Thank you
Contact: Minh Le <m.le@elsevier.com>
We’re hiring!

26. Image credits
• Show of hands: http://www.themorrisagency.co.uk/
• Venn diagram: http://templatelab.com
• Clock: https://www.cleanpng.com
• Cogwheel and meter: https://www.vectorstock.com
29