• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Measuring is knowing -  or is it?
 

Measuring is knowing - or is it?

on

  • 611 views

Presentation at APE Berlin, 2010

Presentation at APE Berlin, 2010

Statistics

Views

Total Views
611
Views on SlideShare
611
Embed Views
0

Actions

Likes
1
Downloads
7
Comments
1

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Measuring is knowing -  or is it? Measuring is knowing - or is it? Presentation Transcript

  • Video of this talk: http://river-valley.tv/measuring-is-knowing-or-is-it/ Measuring is Knowing Or is it? Jan Velterop – APE – 20 January MMX
  • NSF Workshop onScholarly Evaluation Metrics Washington DC, December 2009
  • About Citations and Usage (More about Citiations than Usage)
  • Mixed feelings
  • Mixed feelings useless
  • Mixed feelings useless meaningless
  • Mixed feelings useless meaningless interesting
  • Mixed feelings useless meaningless interesting just play the game
  • Every ‘usage’ has its metric
  • Every metric has its ‘usage’
  • How science is linked
  • And metrics less so
  • Herbert van de Sompel Johan BollenMichael Kurtz Peter Binfield Jevin West Jorge Hirsch Jan Velterop Julia Lane Alexis-Michel Mugabushaka
  • growth of informal science communication re-use of data as impact metrics for data sets measuremetrics suitable to take decisions? local metrics vs global ones Eigenfactor vs Impact Factor web metrics poorly understood h-bar index refinement of h- focus on networks index multiple author, junior author, senior author article level metrics relative funding decisions increasing inputs not always leading to increased output how to quantify exposure in lay newsscience is non-linear media? attributed nano-publications
  • Current knowledge transfer
  • Current knowledge transfer Signal to noise ratio?
  • Current knowledge transfer – an analogy – needle transport
  • Current knowledge transfer – an analogy – needle transport Nano- publications are the “needles”
  • Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cellsinto virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorlyunderstood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show thatHeliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 hafter infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor ofapoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosisrepeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically inducedapoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNAinterference studies revealed that the gene is probably essential for virus pathology and replication. Ascoviruses [16747851] Ascoviruses [16747851] Induction [26277419] Induction of Pathology [26277419 Pathology 13817745] [13817745] Insect [839958] Cell of Insect [283053 839958] Cell [283053] Ascoviruses > induction of pathology > insect cells [16747851] > [26277419 13817745] > [283053 839958]
  • Ascoviruses (AVs) induce a unique pathology in their insect host cells causing cleavage of the cellsinto virion-containing vesicles. The mechanism by which AVs induce vesicle formation is poorlyunderstood. It is postulated that the virus initially induces apoptosis leading to cell fragmentation.The apoptotic bodies are however, rescued by the virus to form the vesicles. Here we show thatHeliothis virescens AV (HvAV-3e) is able to inhibit chemically induced apoptosis from around 16 hafter infection. Analysis of the genome of the virus indicated the presence of a putative inhibitor ofapoptosis (orf28) gene that encodes a protein with an imperfect baculovirus inhibitor of apoptosisrepeat (BIR) and a RING domain. Transiently expressed orf28 did not inhibit chemically inducedapoptosis suggesting that the protein may not serve as an inhibitor of apoptosis. Nevertheless, RNAinterference studies revealed that the gene is probably essential for virus pathology and replication. Ascoviruses [16747851] Ascoviruses [16747851] Induction [26277419] Induction of Pathology [26277419 Pathology 13817745] [13817745] Insect [839958] Cell of Insect [283053 839958] Cell [283053] Ascoviruses > induction of pathology > insect cells [16747851] > [26277419 13817745] > [283053 839958] Semantic codification of assertions
  • Author<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> Provenance<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description>
  • Author<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> Provenance<rdf:Description rdf:about=”http://www.nbic.nl/cwa/relation/#26277419#13817745"><cwa:typeRelation rdf:resource=” http://predicate.conceptwiki.org/index.php/#2121378”/><cwa:direction>1,2</cwa:direction><cwa:strength>1.0</cwa:strength><cwa:author rdf:resource=”http://people.conceptwiki.org/index.php/#85094810”/><cwa:provenance rdf:resource=”http://article.conceptwiki.org/index.php/#121646370”/><cwa:timestamp>1240641052059</cwa:timestamp><cwa:annotated_by rdf:resource=”http://people.conceptwiki.org/index.php/#43065817”/><cwa:annotation rdf resource=”http://www.virusdb.org/viruses/av/Heliothis_virescens_insect”></rdf:Description> They are “nano-publications” because they can be cited
  • In essence the NSF workshop was all about Impact (Though “Impact” was not defined)
  • Let me help define “Impact”
  • But first…Why do scientists publish?
  • The record“keeping the minutes of science” The of S Rec c ord ienc e
  • The record“keeping the minutes of science” e of Scienc ord The Rec
  • Transfer ofinformation and knowledge
  • Credit in the ego-system; the acknowledge economy
  • Credit in the ego-system; the acknowledge economy Often seen as impact
  • “Scientific impact is the soundyou hear if you drop a featherdown the Grand Canyon andwait for the echo.”Paraphrasing Don Marquis, 1878-1937
  • “The notion of impact is incoherent, likelyto reward the sensationalist and second-rate ... and risks turning academics intodoor-to-door salesmen for vulgarisedversions of their increasingly market-oriented products.”Stefan Collini, professor of intellectual history and Englishliterature at the University of Cambridge
  • Impact Factorg-indexh-indexh-b-indexEigenfactor
  • Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor
  • Lazy Man’s Credit League Tables?Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor
  • Lazy Man’s Credit League Tables?Journal Impact FactorIndividual g-indexIndividual h-indexTopic h-b-indexJournal Eigenfactor In Defiance of Science?
  • Defiance of Science?
  • Defiance of Science?Higher precision than needed for a space rendez-vous
  • Apart from journal level , also article level metrics:
  • Apart from journal level , also article level metrics: usage citations blog posts bookmarks comments/ratings trackbacks
  • The less I understand it, the morequality it is bound to have
  • The less I understand it, the morequality it is bound to haveHow about that as a metric?
  • Anyway, what does it mean, all these metrics and the resulting ranking of McJournals * and McArticles*?
  • Anyway, what does it mean, all these metrics and the resulting ranking of McJournals * and McArticles*? * obviously, in this context ”Mc” just means: falling within a narrow band of variability
  • not just self-referencing ivory tower influence?
  • not just self-referencing ivory tower influence?
  • Number of career decisions based purely on citation counts of one sort or another
  • Number of career decisions based purely on citation counts of one sort or another
  • Number of career decisions based purely on citation counts of one sort or another Just in case this graph makes it to the web – it’s totally fictional!
  • Number of career decisions based purely on citation counts of one sort or another Just in case this graph makes it to the web – it’s totally fictional! But believable, too!
  • Akin tophrenology?
  • Akin tophrenology?
  • EigenfactorAkin to down- loads g-indexphrenology? h-index h-b-index
  • EigenfactorAkin to down- loads g-index Impactphrenology? Factor h-index h-b-index
  • Societal Impact
  • 79-page document‘Citation’ mentionedexactly twice.‘Impact Factor’ notmentioned at all.‘Quality’ mentioned50 times.
  • The Scientific Egosystem is an Acknowledge Economy where ROI means Return on Influence
  • The Scientific Egosystem is an Acknowledge Economy where ROI means Return on Influence Shouldn’t that be Societal Influence?
  • “Not everything that can be counted counts; not everything that counts can be counted.” Albert Einstein .
  • Are the most cited articles also the ones worthciting most?
  • Are the most used articles also the ones worthusing most?
  • Are the most cited and used articles also theones worth most to current and future society?
  • the old story of goals and means
  • Julia Lane (NSF) One thing that bugs me is that people answer the question ‘what“ impact has your funding had’ with things like ‘I hired 3 postdocs and 2 support staff.’ ” Societal impact? How has your research – done with our money – made the world a better place?
  • Can we truly know?
  • Ivory towerFoundation
  • Thank you!velterop@conceptweballiance.org