The document discusses the importance and methods of measuring scholarly and public impact through metrics, which can be categorized into traditional bibliometrics and emerging altmetrics. It highlights the various applications of these metrics in evaluating researchers for hiring, tenure, funding, and understanding broader scholarly outputs beyond just citations. The document emphasizes the need for a nuanced understanding of impact metrics, proposing that both bibliometrics and altmetrics serve complementary roles in assessing academic achievements.

1. Measuring Scholarly
and Public Impact:
Let’s Talk Metrics
RACHEL BORCHARDT MONDAY, MAY 16TH, 2017
BORCHARD@AMERICAN.EDU CONFERENCE FOR HIGH-IMPACT RESEARCH

2. Why metrics?
 Quantifiable measures of research impact
 Differentiate articles, journals, authors, institutions
 Imperfect calculations of a fuzzy concept

3. How are metrics used?
 Evaluate researchers:
 Hiring
 Tenure/promotion
 Merit awards
 Grant funding
 Benchmark programs and institutions
Image from AU Faculty Manual, http://www.american.edu/loader.cfm?csModule=security/getfile&pageid=4304081

4. What are metrics
 Traditional Bibliometrics
 Based on citation counts
 Emerging Altmetrics
 Based on online tools
Image from Plum Analytics

5. Categories of bibliometrics
 Article-level
 Journal-level
 Author-level
 Institutional-level

6. Traditional bibliometrics for journals
 Impact Factor and Citation Counts, created to measure
 Journals and journal articles
 Scholarly (journal) impact
 Initially created for librarians, then largely adopted by STEM
Image from Journal Citation Reports (library database)

7. Some Impact Factor limitations
 Retrospective measurement
 Narrow measurement
 Confined by what is indexed as a citation
 Can’t be compared between disciplines
 Other known issues: language, co-authorship, disproportionate article
citation impact
Image from Essential Science Indicators (library database)

8. One way to correct – disciplinary context
Image from Journal Citation Reports (library database)

9. Contextual limitations – “Political
Science” category
List from http://www.auspsa.org.au/publications

10. Other major bibliometrics
 Metrics from Web of Science: Eigenfactor, Immediacy Index, Cited Half-Life
 Metrics from Elsevier/Scopus: SJR, SNIP, IPP, CiteScore
 Metrics from Google Scholar: H5-index, H5-median
Image from http://www.journalmetrics.com

11. SJR comparison – “Political Science
and International Relations”

12. H5-index comparison – Political
Science
List from Google Scholar Metrics

13. Another alternative – citation distributions
http://www.nature.com/news/beat-it-impact-factor-publishing-elite-turns-against-controversial-metric-1.20224

14. Why altmetrics?
 Measures broader scholarly outputs
 Books, presentations, data sets, software code, and
more
 More measures of scholarship
 Views, downloads, shares, mentions in blogs, policy
papers, and more
 More immediate metrics
 Measures beyond scholarly impact
 Policy, public, and more
https://becker.wustl.edu/impact-assessment

15. What are altmetrics?
 Definition: The creation and study of new metrics based on the social web for
analyzing and informing scholarship.
Altmetrics.org
 My definition: The umbrella classification of non-citation based metrics.
http://altmetrics.org/manifesto ; http://twitter.com/jasonpriem

16. Categories of altmetrics
Article-level [individual work]
 Journal-level
 Author-level
 Institutional-level

17. Where do altmetrics come from?

18. How do we collect altmetrics?
 Directly from the individual tools
 From publishers (views, download data)
 From (some) library databases
 From scholarly networks
 Through aggregating tools
Slideshare views
PLOS article metrics Web of Science usage ResearchGate metrics Altmetric metrics

19. “Impact” and metrics
 Inherently fuzzy and subjective
 Attention
 Engagement
 Some correlate to bibliometrics; some don’t

20. Deciphering impact - ImpactStory

21. ImpactStory contextual elements

22. Deciphering impact from tools - Altmetric

23. Altmetric contextual elements
https://www.altmetric.com/details/585373

24. Impact correlations
https://arxiv.org/abs/1203.4745; http://onlinelibrary.wiley.com/doi/10.1002/asi.23309/full

25. Discipline-specific impact
https://www.researchtrends.com/issue-30-september-2012/the-use-of-big-datasets-in-bibliometric-research/

26. Limitations of impact: example
 https://www.altmetric.com/details/2313862/policy-documents
 https://www.healthypeople.gov/2020/topics-objectives/topic/Adolescent-
Health
 https://www.google.com/search?q=%22Trends+in+adolescent+and+youn
g+adult+health+in+the+United+States%22+site%3A.gov&oq=%22Trends
+in+adolescent+and+young+adult+health+in+the+United+States%22+si
te%3A.gov&aqs=chrome..69i57.3775j0j8&sourceid=chrome&ie=UTF-8

27. Current uses of altmetrics
 External
 Applying for funding, promotion/tenure
 Showcasing achievements
 Internal
 Better understand engagement with research
 Who, why, where

28. More on grant funders
https://www.nature.com/naturejobs/science/articles/10.1038/nj7471-397a

29. Grant funders: NSF
 Broader Impacts
 https://www.nsf.gov/od/oia/special/broaderimpacts/

30. Grant funders: Wellcome Trust
http://www.hefce.ac.uk/pubs/rereports/year/2015/metrictide/

31. Putting it all together - suggestions
 Develop personal impact goals
 Discipline-specific, research-specific, university-specific, etc.
 Demonstrate achievement of those goals
 Tell your story
 Bibliometrics, altmetrics, and qualitative measures
 Complementary roles
 Track early and often!

32. Getting help
 Rachel Borchardt
 borchard@american.edu
 202-885-3657
 (I work year-round!)
 Metrics guide, http://tinyurl.com/AUmetrics
 Borrow a copy of my book

33. Questions?