Finding Insights in ALMS
for Research Evaluation

Richard Cave, @richardcave
I.T. Director, PLOS
Charleston Conference 201...
Article-Level Metrics at PLOS
• Article-Level Metrics (ALM) allow the
comprehensive analysis of the post-publication
activ...
ALMs provide the flexibility to look at a wide
range of numbers

3
Authors want to see the impact of their article

4
What about funders and institutions?
• Funders want to monitor the research that they
support
– Wellcome Trust spends appr...
Numbers are
GREAT

but what’s the
impact of the
research?!?

6
How easy it for institutions and funders to
track research impact?
•
•
•
•

It’s not easy!
Aggregate data for all articles...
ALM Reports
almreports.plos.org
Launched June 18
Allows researchers, institutions &
funders to:
• Create a report of the A...
But more important, provides a way to provide
context of ALMs with search
Search based on:
• Keyword
• Author name & count...
For researchers at your institution, can
show impact for all of their articles

10
For institutions and funders - Get specific!
(author_affiliate:"Stanford University") AND
(financial_disclosure:"National ...
Page views and citations over time
Standford University + National Cancer Institute
Total usage includes page views and do...
Page views and Mendeley bookmarks over
time
Standford University + National Cancer Institute
Total usage includes page vie...
Article usage and citations by subject
category
Standford University + National Cancer Institute
Rectangle size correlates...
Research collaborations
Standford University + National Cancer Institute

15
Wellcome Trust evaluation
Adam Dinsmore at Wellcome Trust:
• Identified and exported the entire corpus of PLOS articles
as...
For Publishers
• Improve Business Intelligence
– Optimize publicity efforts with more targeted campaigns (Call For
Papers,...
But beware of dragons
• Data issues = Garbage in, garbage out
• Author names
• Funder names
• “Welcome Trust” vs. “Wellcom...
ALM info
• General information on ALM
http://article-level-metrics.plos.org/
• ALM Reports
http://almreports.plos.org/
– F...
Upcoming SlideShare
Loading in …5
×

Finding Insights in Article-Level Metrics for Research Evaluation

5,860 views

Published on

The use of Article-Level Metrics (ALMs) as an indicator of an article’s quality and impact has dramatically increased in the last year. Publishers continue to add ALMs to research articles and new organizations have been created to aggregate ALMs across multiple fields including usage, citations, and social media. Using ALMs, researchers, librarians, funders, and the general public are able to gain insight into research articles that are the most widely read and used. PLOS launched ALM Reports (http://almreports.plos.org/) which allow users to view ALMs for any set of PLOS articles and visualize the data results. This allows users to quickly explore and compare ALMs for a large number of articles by searching for papers published by researchers at their institutions, for papers funded by specific funding agencies, or by searching on generic terms within an article. The application can be used to access up-to-date information on research papers, to view data on the downstream impact of the research, and to measure evidence of wider engagement with the research. These insights provide a powerful way to evaluate impact of research across many articles in a single view.

Published in: Technology, Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,860
On SlideShare
0
From Embeds
0
Number of Embeds
935
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Let’s assume that authors want to see the impact of their article
  • Now you can drill-down into who is funding research at your institution
  • Show Michael B. Eisen - http://almreports.plos.org/reports/metrics/10465
  • Now you can drill-down into who is funding research at your institution
  • Very basic: if people cannot trust the data, they will not use it.All the powerful benefits to ALMs – transforming research impact assessment, research discovery and navigation of literature, etc. None of this is possible if we do not have a robust system that ensures data integrity, one which is not just comprised of machine monitoring but human guardianship as well as community norms.
  • ALMs can also be used to show where articles are used. This graph geolocates authors of articles. But the same geolocation can be show for Tweets.
  • Finding Insights in Article-Level Metrics for Research Evaluation

    1. 1. Finding Insights in ALMS for Research Evaluation Richard Cave, @richardcave I.T. Director, PLOS Charleston Conference 2013
    2. 2. Article-Level Metrics at PLOS • Article-Level Metrics (ALM) allow the comprehensive analysis of the post-publication activity around a paper, employing usage stats, citations and altmetrics. http://article-level-metrics.plos.org/alm-info/ • Started in 2009 at PLOS • Lots of data, tracking over a dozen sources 2
    3. 3. ALMs provide the flexibility to look at a wide range of numbers 3
    4. 4. Authors want to see the impact of their article 4
    5. 5. What about funders and institutions? • Funders want to monitor the research that they support – Wellcome Trust spends approximately £650 million on research per year. What’s the impact? – Médecins Sans Frontières – how do they know the reach of their article? • Institutions want to see how their researchers are making a change – Stanford University – do librarians know what research was published by Stanford researchers? 5
    6. 6. Numbers are GREAT but what’s the impact of the research?!? 6
    7. 7. How easy it for institutions and funders to track research impact? • • • • It’s not easy! Aggregate data for all articles? Hard to do Follow tweets? Not likely News/blog stories? Hard to find correct sources 7
    8. 8. ALM Reports almreports.plos.org Launched June 18 Allows researchers, institutions & funders to: • Create a report of the ALMs for a set of PLOS articles • View a summary of the metrics along and data visualizations. 8
    9. 9. But more important, provides a way to provide context of ALMs with search Search based on: • Keyword • Author name & country • Institutional affiliation • Publication date • Subject areas • Funder 9
    10. 10. For researchers at your institution, can show impact for all of their articles 10
    11. 11. For institutions and funders - Get specific! (author_affiliate:"Stanford University") AND (financial_disclosure:"National Institutes of Health" ) (author_affiliate:"Stanford University") AND (financial_disclosure:"Wellcome Trust") (author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute") (author_affiliate:"Stanford University") AND (financial_disclosure:"National Cancer Institute" AND financial_disclosure:"Doris Duke Foundation") 11
    12. 12. Page views and citations over time Standford University + National Cancer Institute Total usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Scopus citations and bubble color with the PLOS journal. 12
    13. 13. Page views and Mendeley bookmarks over time Standford University + National Cancer Institute Total usage includes page views and downloads from PLOS and PMC. Bubble size correlates with Mendeley bookmarks and bubble color with the PLOS journal. 13
    14. 14. Article usage and citations by subject category Standford University + National Cancer Institute Rectangle size correlates with total views (HTML and PDF) from PLOS website. Color intensity correlates with number of Scopus citations.
    15. 15. Research collaborations Standford University + National Cancer Institute 15
    16. 16. Wellcome Trust evaluation Adam Dinsmore at Wellcome Trust: • Identified and exported the entire corpus of PLOS articles associated with the Wellcome • Analysed the ALMs associated with them and benchmarked against the sector average • Produced a consumer report Findings: • Wellcome Trust associated PLOS articles are viewed, saved and cited more often than the average for the biomedical research sector • They are discussed by the blogosphere more often than the sector average http://article-level-metrics.plos.org/files/2013/10/Dinsmore.pptx 16
    17. 17. For Publishers • Improve Business Intelligence – Optimize publicity efforts with more targeted campaigns (Call For Papers, conference presence, etc.) – Launch new collections of papers in areas of interest – Find new fields growing at the intersections between communities of interest • Deepen Editorial Capacities – Direct editorial resources to promote new research of interest in growing fields – Identify promotional opportunities for papers and authors (writing press releases, front matter content, editorial overviews, etc.) – Uncover potential new authors • Deliver a Richer Product – Maximize front matter visibility – Create more targeted editorial overviews – Enhance press reviews and promotional content for high-impact papers 17
    18. 18. But beware of dragons • Data issues = Garbage in, garbage out • Author names • Funder names • “Welcome Trust” vs. “Wellcome Trust” • Institutional names • A dozen ways to enter “University of California, Berkeley” • Will be helped in the future by: • ORCID • FundRef 18
    19. 19. ALM info • General information on ALM http://article-level-metrics.plos.org/ • ALM Reports http://almreports.plos.org/ – Future: ALM Reports will be open source but will have limitations (no search functionality) • ALM Application is Open Source (Ruby on Rails) https://github.com/articlemetrics/alm – Recently launched at Copernicus Publications – OJS has a beta trial 19

    ×