Measuring is knowing - or is it?

633 views

Published on

Presentation at APE Berlin, 2010

1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
633
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Measuring is knowing - or is it?

    1. 1. Video of this talk: http://river-valley.tv/measuring-is-knowing-or-is-it/ Measuring is Knowing Or is it? Jan Velterop – APE – 20 January MMX
    2. 2. NSF Workshop onScholarly Evaluation Metrics Washington DC, December 2009
    3. 3. About Citations and Usage (More about Citiations than Usage)
    4. 4. Mixed feelings
    5. 5. Mixed feelings useless
    6. 6. Mixed feelings useless meaningless
    7. 7. Mixed feelings useless meaningless interesting
    8. 8. Mixed feelings useless meaningless interesting just play the game
    9. 9. Every ‘usage’ has its metric
    10. 10. Every metric has its ‘usage’
    11. 11. How science is linked
    12. 12. And metrics less so
    13. 13. Herbert van de Sompel Johan BollenMichael Kurtz Peter Binfield Jevin West Jorge Hirsch Jan Velterop Julia Lane Alexis-Michel Mugabushaka
    14. 14. growth of informal science communication re-use of data as impact metrics for data sets measuremetrics suitable to take decisions? local metrics vs global ones Eigenfactor vs Impact Factor web metrics poorly understood h-bar index refinement of h- focus on networks index multiple author, junior author, senior author article level metrics relative funding decisions increasing inputs not always leading to increased output how to quantify exposure in lay newsscience is non-linear media? attributed nano-publications
    15. 15. Current knowledge transfer
    16. 16. Current knowledge transfer Signal to noise ratio?
    17. 17. Current knowledge transfer – an analogy – needle transport
    18. 18. Current knowledge transfer – an analogy – needle transport Nano- publications are the “needles”
    19. 19. Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cellsinto virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorlyunderstood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show thatHeliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 hafter infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor ofapoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosisrepeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically inducedapoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNAinterference studies revealed that the gene is probably essential for virus pathology and replication. Ascoviruses [16747851] Ascoviruses [16747851] Induction [26277419] Induction of Pathology [26277419 Pathology 13817745] [13817745] Insect [839958] Cell of Insect [283053 839958] Cell [283053] Ascoviruses > induction of pathology > insect cells [16747851] > [26277419 13817745] > [283053 839958]
    20. 20. Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cellsinto virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorlyunderstood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show thatHeliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 hafter infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor ofapoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosisrepeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically inducedapoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNAinterference studies revealed that the gene is probably essential for virus pathology and replication. Ascoviruses [16747851] Ascoviruses [16747851] Induction [26277419] Induction of Pathology [26277419 Pathology 13817745] [13817745] Insect [839958] Cell of Insect [283053 839958] Cell [283053] Ascoviruses > induction of pathology > insect cells [16747851] > [26277419 13817745] > [283053 839958] Semantic codification of assertions
    21. 21. Author<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> Provenance<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description>
    22. 22. Author<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> Provenance<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> They are “nano-publications” because they can be cited
    23. 23. In essence the NSF workshop was all about Impact (Though “Impact” was not defined)
    24. 24. Let me help define “Impact”
    25. 25. But first…Why do scientists publish?
    26. 26. The record“keeping the minutes of science” The of S Rec c ord ienc e
    27. 27. The record“keeping the minutes of science” e of Scienc ord The Rec
    28. 28. Transfer ofinformation and knowledge
    29. 29. Credit in the ego-system; the acknowledge economy
    30. 30. Credit in the ego-system; the acknowledge economy Often seen as impact
    31. 31. “Scientific impact is the soundyou hear if you drop a featherdown the Grand Canyon andwait for the echo.”Paraphrasing Don Marquis, 1878-1937
    32. 32. “The notion of impact is incoherent, likelyto reward the sensationalist and second-rate ... and risks turning academics intodoor-to-door salesmen for vulgarisedversions of their increasingly market-oriented products.”Stefan Collini, professor of intellectual history and Englishliterature at the University of Cambridge
    33. 33. Impact Factorg-indexh-indexh-b-indexEigenfactor
    34. 34. Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor
    35. 35. Lazy Man’s Credit League Tables?Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor
    36. 36. Lazy Man’s Credit League Tables?Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor In Defiance of Science?
    37. 37. Defiance of Science?
    38. 38. Defiance of Science?Higher precision than needed for a space rendez-vous
    39. 39. Apart from journal level , also article level metrics:
    40. 40. Apart from journal level , also article level metrics: usage citations blog posts bookmarks comments/ratings trackbacks
    41. 41. The less I understand it, the morequality it is bound to have
    42. 42. The less I understand it, the morequality it is bound to haveHow about that as a metric?
    43. 43. Anyway, what does it mean, all these metrics and the resulting ranking of McJournals * and McArticles*?
    44. 44. Anyway, what does it mean, all these metrics and the resulting ranking of McJournals * and McArticles*? * obviously, in this context ”Mc” just means: falling within a narrow band of variability
    45. 45. not just self-referencing ivory tower influence?
    46. 46. not just self-referencing ivory tower influence?
    47. 47. Number of career decisions based purely on citation counts of one sort or another
    48. 48. Number of career decisions based purely on citation counts of one sort or another
    49. 49. Number of career decisions based purely on citation counts of one sort or another Just in case this graph makes it to the web – it’s totally fictional!
    50. 50. Number of career decisions based purely on citation counts of one sort or another Just in case this graph makes it to the web – it’s totally fictional! But believable, too!
    51. 51. Akin tophrenology?
    52. 52. Akin tophrenology?
    53. 53. EigenfactorAkin to down- loads g-indexphrenology? h-index h-b-index
    54. 54. EigenfactorAkin to down- loads g-index Impactphrenology? Factor h-index h-b-index
    55. 55. Societal Impact
    56. 56. 79-page document‘Citation’ mentionedexactly twice.‘Impact Factor’ notmentioned at all.‘Quality’ mentioned50 times.
    57. 57. The Scientific Egosystem is an Acknowledge Economy where ROI means Return on Influence
    58. 58. The Scientific Egosystem is an Acknowledge Economy where ROI means Return on Influence Shouldn’t that be Societal Influence?
    59. 59. “Not everything that can be counted counts; not everything that counts can be counted.” Albert Einstein .
    60. 60. Are the most cited articles also the ones worthciting most?
    61. 61. Are the most used articles also the ones worthusing most?
    62. 62. Are the most cited and used articles also theones worth most to current and future society?
    63. 63. the old story of goals and means
    64. 64. Julia Lane (NSF) One thing that bugs me is that people answer the question ‘what“ impact has your funding had’ with things like ‘I hired 3 postdocs and 2 support staff.’ ” Societal impact? How has your research – done with our money – made the world a better place?
    65. 65. Can we truly know?
    66. 66. Ivory towerFoundation
    67. 67. Thank you!velterop@conceptweballiance.org

    ×