Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
context               quantities   merit       elite metrics                timeline      h        analysis significance   ...
data are the currency of science   quantities
+data are the currency of science   quantities
the greater good = feedback, filtering, change in behaviors.
quality versus quantity
volume of scientific literature immense
constellation of ideasvolume of scientific literature immense
merittime   filter   performance       loss
rank      legacy+   sort      quality      -    group   impact forms
unfortunately, much of the filtering not based   on reading but numbers & tendencies                                       ...
criticalassumption
30                        25                                  cites/yr/paper                                              ...
0.6              0.5              0.4Effect size              0.3              0.2              0.1               0       ...
LOG citations per publication                0                        1                                         2         ...
Why is this merit concept important?
Box 1                                                 Fig. a)                                                             ...
Box 2                                                 Fig. a)                                                             ...
Box 3                            Fig. a)                                                                                  ...
the metric elite misses not only diversity of people                     but ideas
i10-index        2011       q2-index       hg-index       m-index       e-index       r-index       a-index       w-index ...
Why isn’t impact factor or citations sufficient?
h
h relies on citations to papers not journals                 not skewed by singletons   not influenced by large body of unc...
psychology             contrasts
economics
finance
marketing
quality   quantityh solution
however at larger scales, N still important
Matthew effectovervalued scientists do publish significantly more
m    options
i10
h derivations such as h-core         or g index
the solution is composite.
still numbers, just a different box
volume of scientific literature immense   application
curation      connectionsvolume of scientific literature immense         application
collections                 departments  journals                  individuals              application
curation                journals/collections                               editors     dissemination                subjec...
curation by connection - big dataindividuals      ideas      data       place      utility      not by merit or citations ...
sciencemerit ideasmetricsmetrics
filters           publicationshowever, mini-manuscripts, figshare, slideshare,pre-print servers, and data publications aretr...
metrics that illuminate & provide insight will be criticalultimately, publications in all forms are extensions of learning
citations to datasets/figures in talkCostas, R. and Borodons, M. 2007. The h-index: Advantages, limitations and its relatio...
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
What’s all the h about? A Summary of Performance Metrics for Academics and Journals
Upcoming SlideShare
Loading in …5
×

What’s all the h about? A Summary of Performance Metrics for Academics and Journals

667 views

Published on

The attraction of metrics is irresistible. The emergence of the quantified self as an emergent phenomenon to improve performance, health, and daily functioning has dramatically reshaped cultural perceptions of sharing and the relative good. The science of publica- tions is not free from this movement with a proliferation of metrics associated with both individuals and their scientific products.
A dominant index, h, is central to the discussion associated with singular point estimate metrics versus the ineffable quality of the science we produce. The tension between quantity and quality or the modern redux of metrics versus quality is an excellent starting point in examination of the role metrics can play in improving or impeding scientific discovery. This dichotomy is of course fallacious but useful as means to test ideas associated with assigning merit to peer-reviewed publications. A brief overview of metrics is pro- vided in this talk including h with a strong emphasis on the theory of merit for the current dissemination pipeline in science.

Published in: Education
  • Be the first to comment

  • Be the first to like this

What’s all the h about? A Summary of Performance Metrics for Academics and Journals

  1. 1. context quantities merit elite metrics timeline h analysis significance contrasts options application
  2. 2. data are the currency of science quantities
  3. 3. +data are the currency of science quantities
  4. 4. the greater good = feedback, filtering, change in behaviors.
  5. 5. quality versus quantity
  6. 6. volume of scientific literature immense
  7. 7. constellation of ideasvolume of scientific literature immense
  8. 8. merittime filter performance loss
  9. 9. rank legacy+ sort quality - group impact forms
  10. 10. unfortunately, much of the filtering not based on reading but numbers & tendencies merit
  11. 11. criticalassumption
  12. 12. 30 25 cites/yr/paper IFCites/yr/paper and IF 20 15 10 5 0 -5 -4 -3 -2 -1 0 1 2 3 4 5 Effect size
  13. 13. 0.6 0.5 0.4Effect size 0.3 0.2 0.1 0 reject accept Support hypotheses
  14. 14. LOG citations per publication 0 1 2 35 4 6LOG total funding 7
  15. 15. Why is this merit concept important?
  16. 16. Box 1 Fig. a) Fig. b) 25 30 20 25 20 15Number of HC Number of HC 15 10 10 5 5 0 0 100 200 300 400 0 5000 10000 15000 20000 Number of Articles Number of Citations Fig. c) Fig. d) 70 40 60 50 30Number of HC Number of HC 40 20 30 20 10 10 0 0 elite 0.0 0.1 0.2 0.3 0.4 0.5 0 20 40 60 80 Proportion Citations to Most Cited Article Number of Journals
  17. 17. Box 2 Fig. a) Fig. b) 0.8 80 ● 0.6 ● ● 60 ● ● Proportion CitationsNumber of HC ● ● ● ● ● ● ● 0.4 ● 40 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● 20 0.2 ●● ● ● ● ● ●● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ●● ●● ● ● ● ●● ●● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ●● ●● ●●● ● ● ● ●● ● ● ● ●● ● ● ● ●●●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●●●● ● ●●●● ●● ● ● ●●●● ● ● ●●● ●●●●●●● ● ● ●●●● ● ● ●●●● ● 0 0.0 ●●●● ● ● ● ●●●● ● ● ● ● ● 0 1 2 3 4 5 0.1 0.2 0.3 0.4 0.5 0.6 log(Articles in Nature or Science) Proportion Articles in Nature and Science Fig. c) Fig. d) ● ● ● ● ● ● ● ● ● ● 0.8 30 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 ● Proportion Citations ● ● ● ● ● ●Number of HC ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.4 ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● 10 ●● ● ● ●●● ●● ●● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● 0.2 ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ● ●●● ● ● ● ● ●●● ● ● ● ●● ● ● ● ●●● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 0.2 0.4 0.6 0.8 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Proportion Articles in Favorite Journal Proportion Articles in Favorite Journal
  18. 18. Box 3 Fig. a) Fig. b) ● 0.8 120 ● ● ● 100 ● 0.6 ● ● ● ● 80 Proportion Citations ● ● ● ●Number of HC ● ● ● ● ● 0.4 ● ● ● ● ● 60 ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● 40 ● ● ● ●● ●● ● ● ●● ● ● ● ● 0.2 ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ● ● 20 ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ●● ●● ●● ● ●● ● ● ● ● ●●●● ●● ● ● ●● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ● ●●● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ●● ● ● ●● ● ●● ●●● ● ● ● ●●● ●●●●● ●● ●● ●●● ● ●● ● ●●●●●● ● ● ●●● ●●●●●● ● ● ●● ● ● ●● ●●● ● ● ●●● ● 0 0.0 ● ● 0.0 0.2 0.4 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Proportion Single Authored Articles Proportion Single Authored Articles Fig. c) Fig. d) 1.0 30 ● ● ● ● ● ● ● 25 0.8 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 20 ● ● ● ● ● ● ●● ● Proportion Citations ● ● 0.6 ● ● ●Number of HC ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● 15 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ●● ●● 0.4 ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ●●● ● ● ● ●● ● ●● ●● 10 ● ● ● ● ● ●● ● ●● ● ●● ●● ●● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ●● ● ●● ●●● ● ● ● ●● ●● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● 0.2 ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ● ● ● ● ● ●● ● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ●● ● ● ● 0 0.0 ● ● ●● ● ● 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 Proportion First Authored Articles Proportion First Authored Articles
  19. 19. the metric elite misses not only diversity of people but ideas
  20. 20. i10-index 2011 q2-index hg-index m-index e-index r-index a-index w-index g-index 2006 JIF h-index 2005mcpp # citations 1902 # publications 1905 timeline
  21. 21. Why isn’t impact factor or citations sufficient?
  22. 22. h
  23. 23. h relies on citations to papers not journals not skewed by singletons not influenced by large body of uncited papers minimizes politics of publication useful for similar stage comparisons applied to any group + - counts citations regardless of whydoes not account for variations in average numbers of pubs ignores number & position of authors limited by total number so juniors disadvantaged increasing h at high levels difficult so compression data looks backwards not forward analysis
  24. 24. psychology contrasts
  25. 25. economics
  26. 26. finance
  27. 27. marketing
  28. 28. quality quantityh solution
  29. 29. however at larger scales, N still important
  30. 30. Matthew effectovervalued scientists do publish significantly more
  31. 31. m options
  32. 32. i10
  33. 33. h derivations such as h-core or g index
  34. 34. the solution is composite.
  35. 35. still numbers, just a different box
  36. 36. volume of scientific literature immense application
  37. 37. curation connectionsvolume of scientific literature immense application
  38. 38. collections departments journals individuals application
  39. 39. curation journals/collections editors dissemination subject editors pyramid referees crowdsource readers reputation economy writers not based on citation capital
  40. 40. curation by connection - big dataindividuals ideas data place utility not by merit or citations - by relationships
  41. 41. sciencemerit ideasmetricsmetrics
  42. 42. filters publicationshowever, mini-manuscripts, figshare, slideshare,pre-print servers, and data publications aretransforming the publication process in scienceand providing new opportunities for discovery.
  43. 43. metrics that illuminate & provide insight will be criticalultimately, publications in all forms are extensions of learning
  44. 44. citations to datasets/figures in talkCostas, R. and Borodons, M. 2007. The h-index: Advantages, limitations and its relation with other bibliometric indicators atthe micro level. - The Journal of Informetrics 1: 193-203.Harzing, A. W. and van der Wal, R. 2008. Comparing the Google Scholar h-index with the ISI Journal Impact Factor. -Resarch in International Management Products & Services for Academics Report.: 1-25.Lortie, C. J., Aarssen, L. W., Budden, A. E., Koricheva, J., Leimu, R. and Tregenza, T. 2007. Publication bias and merit in ecology. -Oikos 116: 1247-1253.Lortie, C. J., Aarssen, L. W., Parker, J. N. and Allesina, S. 2012. Good news for the people who love bad news: an analysis ofthe funding of the top 1% most highly cited ecologists. - Oikos 121: 1005-1008.Lortie, C. J., Aarssen, L. W., Budden, A. E. and Leimu, R. 2012. Do citations and impact factors relate to the real numbers inpublications? A case study of citation rates, impact, and effects sizes in ecology and evolutionary biology. - ScientometricsDOI: 10.1007/s11192-012-0822-6.Marnett, A. 2013. H-Index: What It Is and How to Find Yours. - Benchfly blog.Priem, J., Piwowar, H. and Hemminger, B. M. 2012. Altemtrics in the wild: using scoial media to explore scholarly impact. -arXiv 1203.4745v1.Wardle, D. A. 2010. Do ‘Faculty of 1000’ (F1000) ratings of ecological publications serve as reasonable predictors of theirfuture impact? . - Ideas in Ecology and Evolution 3: 11-15.

×