NISO Webinar: Beyond Publish or Perish: Alternative Metrics for Scholarship

http://www.niso.org/news/events/2012/nisowebinars/alternative_metrics/ NISO Webinar:Beyond Publish or Perish: Alternative Metrics for Scholarship November 14, 2012Speakers: Martin Fenner, Jason Priem, Aalam Wassef

Technical Lead ALM Project Martin Fennerhttp://www.flickr.com/photos/wallyg/156079118/

Journal-based metrics correlate poorlywith numbers for individual article 3

PLOS ONE is not a second-tier journalCombined Scopus citation counts for all PLOSBiology and PLOS Medicine articles published in2009, as well as top 200 PLOS ONE articles in 2009 Citation counts collected November 8, 2012 4

The megajournal has changed how we use thejournal as a filter 5

Citations are only a small fraction of how a paper is reused and discussed 100% 22.2% Article-Level Metrics from November 8, 2012 for 63,771 PLOS Papers 0.3% 6

The PLOS Article-Level Metrics projectstarted in 2009 and tracks usage, citationsand social web activity for all PLOS articlesUsage Citations Social WebPLOS Journals CrossRef Mendeley(HTML, PDF, XML) Scopus CiteULikePubMed Central Web of Science Facebook(abstract, fulltext, PubMed Central TwitterPDF, figures) Research Blogging PLOS Comments Wikipedia 7

Why use article-level metrics? 10

Should article-level metrics be used to evaluate the impact of research?• Can numbers reflect the impact of research, across disciplines and over time?• Do the currently available metrics really measure impact or something else?• Does the use of metrics for evaluation create undesired incentives?• How easily can metrics be changed by self-promotion and gaming? 11

What do we measure? 12

Gaming and Spam 13

Metrics as a discovery tool 14

Data-driven stories about the post-publication reception of a paper 15

http://blogs.bbsrc.ac.uk/index.php/2012/09/summertime-reprise/http://dx.doi.org/10.1371/journal.pcbi.1000204 16

Business intelligence for ajournal, university of funder 18

http://bit.ly/alm36 19

Tweets about Big Food CollectionData collected August 19, 2012. 21

Identify Open Access articles published by PLOS,funded by the European Commission SeventhFramework Programme (FP7)Match articles with grant numbersShow the impact of these publications PLOS Search API CORDIS 2562 Articles 1166 Articles 22

Author locations from the ENGAGE projectFirst authors (blue) and last authors (green).European Network for Genetic and Genomic Epidemiologyhttp://www.openaire.eu/en/component/openaire/project_info/default/530?grant=201413 23

Scopus 24

Mendeley 25

Thank you Wordle of altmetrics manifesto by AJ Cann http://www.flickr.com/photos/ajc1/6795008004/ 26

Altmetrics and Revolutions:New media, new metrics, and a new scholarly communications system. Nov 14, NISO

In the beginning: ...was the letter. • slow • costly • duplication • best solution given available technology

The First RevolutionIn 1665, Oldenburg published ThePhilisophical Transactions of theRoyal Society (now OA :).This applied the best availabletechnology (the printing press) tovastly improve dissemination.

The First Revolutionpromoted homogeneity ofoutputs.The standardized article was born ofthe need for industrial-scale replicationand interchangeability.

The Second Revolutionwill promote diversity ofoutputs.Since publication is nearly free, itbecomes trivial to capture the missingpieces of the scholarly record.

conversationstoriesanalysisdata

Looking beneath the foliage: Web promises new tools for conversation. reference managers blogs social bookmarking social networks

Examples: Mendeley2 million userlibraries200 milliondocumentsuploaded, 75Munique (MEDLINEhas 18 million...)

Examples: Twitter In one month, over 58k citations from Twitter to scholarly articles (citwaitions?) It is like having a jury preselect what will probably interest you…. Occasionally there will be something that people will link to, and it will change what I think, or what I’m doing, or what I’m interested in. -study participant

Examples: Twitter

Bring on themetrics.

A network of ideas: bibliometrics.In 1961, Garfield creates theScience Citation Index. • replaces expert judges with crowdsourced judgements • based on existing patterns of use: mining, not asking. • And thats awesome!

But only part of the picture1. Only one type of person: academics.2. Only one kind of resource: scholarly articles.3. Only one kind of use: using to support ascholarly article.

What about all the other uses? Reading, annotating, bookmarking, saving, discussing, sharing, etc?"...there are Heart of scholarly Spotting emergingundoubtedly highly communication research fronts willuseful journals that is "visits, personal require trackingare not cited contacts, and "formal andfrequently." letters." (Bernal, informal(Garfield, 1972) 1944) communication" (Kuhn, 1977)

What about all the other uses?We start to confuse“the kind of use we cantrack” with useand “citation impact”with impact.

Bibliometrics mined impacton the first scholarly Web. altmetrics minesimpact on the next one.

Why altmetrics?• We can more fairly evaluate researchers, publications, and institutions if we look at the whole impact picture.• We can assess impact faster• Etc, etc.• Much more important: quantifying impact means we can teach machines what’s important. This changes the game. And the game needs to be changed.

Journals need an upgrade.• First journals went hand in hand with the Scientific Revolution. Applied the most advanced technology available to the problem of spreading scholarship.• Today’s journals are still the best scholarly communication system possible using 17th-century technology.• They’ve got some problems: o Slow o Restrictive format: function follows form o Closed o Inconsistent quality control o Hard to innovate

We don’t use the Web.• Berners-Lee created the Web as a scholarly communication tool.• Today the Web has revolutionized everything but scholarly communication.• Online journals are essentially paper journals, delivered by faster horses. In the early days, CERN maintained a list of all the world’s Web servers. Haha, CERN ur so krazy.• But today, we can fit every single meaningful outlet for scholarship (25k journals + some conferences) on one list, too.

But what if we did?The Decoupled Journal article: a case study.

The second revolution has started.Once we have alt-citation data, it’s too useful toignore; alternative filters and even certification pathsbased on this data will open.As Peter Vinkler says, citation graph data is likeChekhov’s gun: once on stage, it has to be fired.

A wise man, that Chekov.

Questions?Jason Priem @jasonpriem, http://jasonpriem.org This is a living document; if a particular version is important to you, make sure you link to that revision.

http://www.niso.org	807
https://twitter.com	18
http://translate.googleusercontent.com	6
https://si0.twimg.com	1
http://www.egloos.com	1
http://inforan.egloos.com	1

NISO Webinar: Beyond Publish or Perish: Alternative Metrics for Scholarship

by National Information Standards Organization (NISO) on Nov 14, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 834

Statistics

NISO Webinar: Beyond Publish or Perish: Alternative Metrics for Scholarship Presentation Transcript