What’s all the h about? A Summary of Performance Metrics for Academics and Journals

  • 318 views
Uploaded on

The attraction of metrics is irresistible. The emergence of the quantified self as an emergent phenomenon to improve performance, health, and daily functioning has dramatically reshaped cultural …

The attraction of metrics is irresistible. The emergence of the quantified self as an emergent phenomenon to improve performance, health, and daily functioning has dramatically reshaped cultural perceptions of sharing and the relative good. The science of publica- tions is not free from this movement with a proliferation of metrics associated with both individuals and their scientific products.
A dominant index, h, is central to the discussion associated with singular point estimate metrics versus the ineffable quality of the science we produce. The tension between quantity and quality or the modern redux of metrics versus quality is an excellent starting point in examination of the role metrics can play in improving or impeding scientific discovery. This dichotomy is of course fallacious but useful as means to test ideas associated with assigning merit to peer-reviewed publications. A brief overview of metrics is pro- vided in this talk including h with a strong emphasis on the theory of merit for the current dissemination pipeline in science.

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
318
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. context quantities merit elite metrics timeline h analysis significance contrasts options application
  • 2. data are the currency of science quantities
  • 3. +data are the currency of science quantities
  • 4. the greater good = feedback, filtering, change in behaviors.
  • 5. quality versus quantity
  • 6. volume of scientific literature immense
  • 7. constellation of ideasvolume of scientific literature immense
  • 8. merittime filter performance loss
  • 9. rank legacy+ sort quality - group impact forms
  • 10. unfortunately, much of the filtering not based on reading but numbers & tendencies merit
  • 11. criticalassumption
  • 12. 30 25 cites/yr/paper IFCites/yr/paper and IF 20 15 10 5 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 Effect size
  • 13. 0.6 0.5 0.4Effect size 0.3 0.2 0.1 0 reject accept Support hypotheses
  • 14. LOG citations per publication 0 1 2 35 4 6LOG total funding 7
  • 15. Why is this merit concept important?
  • 16. Box 1 Fig. a) Fig. b) 25 30 20 25 20 15Number of HC Number of HC 15 10 10 5 5 0 0 100 200 300 400 0 5000 10000 15000 20000 Number of Articles Number of Citations Fig. c) Fig. d) 70 40 60 50 30Number of HC Number of HC 40 20 30 20 10 10 0 0 elite 0.0 0.1 0.2 0.3 0.4 0.5 0 20 40 60 80 Proportion Citations to Most Cited Article Number of Journals
  • 17. Box 2 Fig. a) Fig. b) 0.8 80 ● 0.6 ● ● 60 ● ● Proportion CitationsNumber of HC ● ● ● ● ● ● ● 0.4 ● 40 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● 20 0.2 ●● ● ● ● ● ●● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ●● ● ● ● ●● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●●● ● ●●●● ●● ● ● ●●●● ● ● ●●● ●●●●●●● ● ● ●●●● ● ● ●●●● ● 0 0.0 ●●●● ● ● ● ●●●● ● ● ● ● ● 0 1 2 3 4 5 0.1 0.2 0.3 0.4 0.5 0.6 log(Articles in Nature or Science) Proportion Articles in Nature and Science Fig. c) Fig. d) ● ● ● ● ● ● ● ● ● ● 0.8 30 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 ● Proportion Citations ● ● ● ● ● ●Number of HC ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4 ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● 10 ●● ● ● ●●● ●● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● 0.2 ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 0.2 0.4 0.6 0.8 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Proportion Articles in Favorite Journal Proportion Articles in Favorite Journal
  • 18. Box 3 Fig. a) Fig. b) ● 0.8 120 ● ● ● 100 ● 0.6 ● ● ● ● 80 Proportion Citations ● ● ● ●Number of HC ● ● ● ● ● 0.4 ● ● ● ● ● 60 ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● 40 ● ● ● ●● ●● ● ● ●● ● ● ● ● 0.2 ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ● 20 ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ●● ● ●● ● ● ● ● ●●●● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ●● ●●● ● ● ● ●●● ●●●●● ●● ●● ●●● ● ●● ● ●●●●●● ● ● ●●● ●●●●●● ● ● ●● ● ● ●● ●●● ● ● ●●● ● 0 0.0 ● ● 0.0 0.2 0.4 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Proportion Single Authored Articles Proportion Single Authored Articles Fig. c) Fig. d) 1.0 30 ● ● ● ● ● ● ● 25 0.8 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20 ● ● ● ● ● ● ●● ● Proportion Citations ● ● 0.6 ● ● ●Number of HC ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● 15 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ●● 0.4 ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ● ●● ● ●● ●● 10 ● ● ● ● ● ●● ● ●● ● ●● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● 0.2 ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● 0 0.0 ● ● ●● ● ● 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Proportion First Authored Articles Proportion First Authored Articles
  • 19. the metric elite misses not only diversity of people but ideas
  • 20. i10-index 2011 q2-index hg-index m-index e-index r-index a-index w-index g-index 2006 JIF h-index 2005mcpp # citations 1902 # publications 1905 timeline
  • 21. Why isn’t impact factor or citations sufficient?
  • 22. h
  • 23. h relies on citations to papers not journals not skewed by singletons not influenced by large body of uncited papers minimizes politics of publication useful for similar stage comparisons applied to any group + - counts citations regardless of whydoes not account for variations in average numbers of pubs ignores number & position of authors limited by total number so juniors disadvantaged increasing h at high levels difficult so compression data looks backwards not forward analysis
  • 24. psychology contrasts
  • 25. economics
  • 26. finance
  • 27. marketing
  • 28. quality quantityh solution
  • 29. however at larger scales, N still important
  • 30. Matthew effectovervalued scientists do publish significantly more
  • 31. m options
  • 32. i10
  • 33. h derivations such as h-core or g index
  • 34. the solution is composite.
  • 35. still numbers, just a different box
  • 36. volume of scientific literature immense application
  • 37. curation connectionsvolume of scientific literature immense application
  • 38. collections departments journals individuals application
  • 39. curation journals/collections editors dissemination subject editors pyramid referees crowdsource readers reputation economy writers not based on citation capital
  • 40. curation by connection - big dataindividuals ideas data place utility not by merit or citations - by relationships
  • 41. sciencemerit ideasmetricsmetrics
  • 42. filters publicationshowever, mini-manuscripts, figshare, slideshare,pre-print servers, and data publications aretransforming the publication process in scienceand providing new opportunities for discovery.
  • 43. metrics that illuminate & provide insight will be criticalultimately, publications in all forms are extensions of learning
  • 44. citations to datasets/figures in talkCostas, R. and Borodons, M. 2007. The h-index: Advantages, limitations and its relation with other bibliometric indicators atthe micro level. - The Journal of Informetrics 1: 193-203.Harzing, A. W. and van der Wal, R. 2008. Comparing the Google Scholar h-index with the ISI Journal Impact Factor. -Resarch in International Management Products & Services for Academics Report.: 1-25.Lortie, C. J., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R. and Tregenza, T. 2007. Publication bias and merit in ecology. -Oikos 116: 1247-1253.Lortie, C. J., Aarssen, L. W., Parker, J. N. and Allesina, S. 2012. Good news for the people who love bad news: an analysis ofthe funding of the top 1% most highly cited ecologists. - Oikos 121: 1005-1008.Lortie, C. J., Aarssen, L. W., Budden, A. E. and Leimu, R. 2012. Do citations and impact factors relate to the real numbers inpublications? A case study of citation rates, impact, and effects sizes in ecology and evolutionary biology. - ScientometricsDOI: 10.1007/s11192-012-0822-6.Marnett, A. 2013. H-Index: What It Is and How to Find Yours. - Benchfly blog.Priem, J., Piwowar, H. and Hemminger, B. M. 2012. Altemtrics in the wild: using scoial media to explore scholarly impact. -arXiv 1203.4745v1.Wardle, D. A. 2010. Do ‘Faculty of 1000’ (F1000) ratings of ecological publications serve as reasonable predictors of theirfuture impact? . - Ideas in Ecology and Evolution 3: 11-15.