Downloaded 17 times
![Metric opacity: PageRank
The 2002 Google vs Scientology case [1]
ad-hoc adjustment of rankings to comply with DMCA
SEO: reverse-engineering PageRank
The Google dance [2]
short-term fluctuation in PageRank
SEO strategies to control it
[1] McCullagh (2002). Google Yanks Anti-Church Sites, Wired
http://www.wired.com/politics/law/news/2002/03/51233
[2] MetaMend. What is the Google Dance?
http://www.metamend.com/google-dance.html](https://image.slidesharecdn.com/coasp11-110922003027-phpapp02/75/Transparency-in-measures-of-scientific-impact-9-2048.jpg)
![Metric transparency: altmetrics
Alternative measures of the impact of scientific artifacts (papers, datasets,
slides) based on large-scale online data [3]
Provide transparent, verifiable impact indicators linked to data sources
Provide real-time, multidimensional, semantically rich impact indicators
Disclose areas of impact invisible to traditional bibliometrics and usage factors
Track references and reuse of scholarly contents inside and outside academia
[3] Priem, Taraborelli, Roth, Neylon (2010). The altmetrics manifesto,
http://altmetrics.org/manifesto](https://image.slidesharecdn.com/coasp11-110922003027-phpapp02/75/Transparency-in-measures-of-scientific-impact-13-2048.jpg)
![Metric transparency: altmetrics
How to validate altmetrics-based impact? [4]
Using altmetrics to redefine impact
extra-academic usage (industry, education)
reuse of open scholarly contents [5]
[4] Priem et al. (2011). Evaluating altmetrics, in preparation
[5] Neylon (2011) Re-use as Impact. ACM WebSci ’11 Altmetrics Workshop
http://altmetrics.org/workshop2011/neylon-v0/](https://image.slidesharecdn.com/coasp11-110922003027-phpapp02/75/Transparency-in-measures-of-scientific-impact-17-2048.jpg)
Uploaded byDario Taraborelli
Transparency in measures of scientific impact
This document discusses transparency issues with current metrics used to measure scientific impact, such as PageRank and Journal Impact Factor. It argues that these metrics lack transparency in their algorithms and are susceptible to gaming. As an alternative, it proposes altmetrics, which provide transparent, verifiable impact indicators linked to open data sources. Altmetrics track references and reuse of scholarly works both within and outside of academia. They aim to give a more comprehensive view of impact by measuring extra-academic usage and reuse of open scholarly content. The document calls for more transparent APIs and measures of reuse to better capture scientific impact.
More Related Content
Similar to Transparency in measures of scientific impact
Transparency in measures of scientific impact
- 1. Transparency in measures of scientific impact
- 2. What do impactmetrics measure?
- 3.
- 4.
- 5.
- 6.
- 7. Metric opacity Metric disconnected from what it measures Metric consumer can: • Make metric-based inferences • Compare item score over time • Rank other items by the same metrics
- 8. Metric opacity: PageRank Semi-publicalgorithm, arbitrary adjustments.
- 9. Metric opacity: PageRank The2002 Google vs Scientology case [1] ad-hoc adjustment of rankings to comply with DMCA SEO: reverse-engineering PageRank The Google dance [2] short-term fluctuation in PageRank SEO strategies to control it [1] McCullagh (2002). Google Yanks Anti-Church Sites, Wired http://www.wired.com/politics/law/news/2002/03/51233 [2] MetaMend. What is the Google Dance? http://www.metamend.com/google-dance.html
- 10. Metric opacity: JournalImpact Factor Public algorithm, arbitrary adjustments
- 11. Metric opacity: JournalImpact Factor Denominator variables (citable items) Self-citation (from author-level to journal-level self-citation) The JIF dance Editorial policies to maximize JIF publish more reviews negotiate citable items Controlling self-citation: the case of niche journals
- 12. Benefits of metricopacity Increase trust in the metric itself Provide easy to consume impact proxy “My website has a PageRank 7” “You increased your Klout score by 2 points” “I have 12 papers published in 3+ JIF journals” Better control of gaming Consequences of metric opacity Lack of verifiability Metric lock-in Metric abuse
- 13. Metric transparency: altmetrics Alternativemeasures of the impact of scientific artifacts (papers, datasets, slides) based on large-scale online data [3] Provide transparent, verifiable impact indicators linked to data sources Provide real-time, multidimensional, semantically rich impact indicators Disclose areas of impact invisible to traditional bibliometrics and usage factors Track references and reuse of scholarly contents inside and outside academia [3] Priem, Taraborelli, Roth, Neylon (2010). The altmetrics manifesto, http://altmetrics.org/manifesto
- 14. Metric transparency: altmetrics altmetricsservices readermeter.org citedin.org total-impact.org altmetric.com
- 15.
- 16.
- 17. Metric transparency: altmetrics How to validate altmetrics-based impact? [4] Using altmetrics to redefine impact extra-academic usage (industry, education) reuse of open scholarly contents [5] [4] Priem et al. (2011). Evaluating altmetrics, in preparation [5] Neylon (2011) Re-use as Impact. ACM WebSci ’11 Altmetrics Workshop http://altmetrics.org/workshop2011/neylon-v0/
- 18. Metric transparency: altmetrics DanielMietchen (2011) Reusing open-access materials. http://bit.ly/OA_reuse
- 19. Metric transparency: altmetrics DanielMietchen (2011) Reusing open-access materials. http://bit.ly/OA_reuse
- 20. Conclusions stop worrying about impact help us rethink impact create APIs to expose (and facilitate the reuse of) published contents usage data help us measure reuse transparently/reproducibly
- 21. Thanks Credits VTVM 12 voltsDC: CC-BY-NC image by 3D King Wildcat vs. Cheetah: CC-BY image by JD Hancock Dario Taraborelli. Transparency in measures of scientific impact "http://nitens.org/docs/slides/coasp11.pdf" "dtaraborelli@wikimedia.org " "@ReaderMeter"