No stories without evidence, no evidence without stories
•
3 likes•1,797 views
Talk given at Sydney University on 4 August 2015. Across many parts of our lives we are faced with the increasing availability of data to support decision making. With every element of the research process moving online, there are many new sources of data, as well as improved old sources of data, that can provide information on the performance, value and use of research and researchers. But there is a problem. The proliferation of proxy data, and their naive equation with such weakly defined concepts as “quality” and “excellence”, have lead to a reliance on rankings and quantitative measures as institutional targets. More than this the adoption of these instrumental targets has lead us away from a critical discussion of institutional values, indeed of what the institution is for. I will argue that it is only by moving away from such vague terms as “quality”, “excellence” and “impact” and focussing on institutional values and a well articulated mission that institutions of scholarship will continue to be relevant for the future. It is through interrogating the goals of the institution that the enormous potential resource of data on the research enterprise can be realised. Using the data effectively will allow us a window on how knowledge actually moves and is used. In combination with a clear sense of institutional goals this provides great opportunities for institutions to differentiate themselves from the pack.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to No stories without evidence, no evidence without stories
Similar to No stories without evidence, no evidence without stories (20)
More from Cameron Neylon
More from Cameron Neylon (20)
Recently uploaded
Recently uploaded (20)
No stories without evidence, no evidence without stories
- 1. No Evidence without Stories, No Stories Without Evidence Responsible use of metrics in research assessment 1 Cameron Neylon - 4 August 2015 - University of Sydney @cameronneylon - http://orcid.org/0000-0002-0068-716X
- 2. A QUESTION OF VALUES 2Nichole Burrows: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
- 8. More data is the answer! 8https://www.flickr.com/photos/blprnt/3261815454 CC BY
- 9. 9 DATA AND MEASUREMENT Scott Akerman: http://www.flickr.com/photos/sterlic/4299631538 CC BY
- 12. More data is the answer? 12https://www.flickr.com/photos/blprnt/3261815454 CC BY
- 14. Citations •Traditional and valued •Less consistent than most think – WoS, Scopus, CrossRef, (Europe)PMC •Track a very specific kind of usage •Only track a very specific kind of
- 15. Usage •Page views, downloads •Measure usage by a wider community •Measure greater range of use •Inconsistent between publishers •Rarely get demographics of viewers
- 16. Bookmarks •Academic and “public” facing sites – People collecting literature – Building a library •Measure greater range of use vs cites •Focus on use by researchers (Mendeley)
- 17. Conversations/Social Media •Twitter, Facebook, Blogs, News sites – Sharing literature within communities – Conversations about literature •Open to the public (but often actually driven by/involving researchers)
- 18. A story… 18
- 19. 19altmetric.com
- 20. 20
- 21. 21
- 22. 22
- 23. 0 20 40 60 80 0 5000 10000 15000 PLOS papers authored by University of Cape Town staff Usage data from the PLoS journal website (combined HTML views and PDF downloads). Circle size correlates with CrossRef citation counts. Red circles represent papers with "HIV" in title. Months Total Views Not an obvious outlier... 23
- 25. 25
- 27. ...who I can talk to. 27
- 28. 28
- 29. 29
- 30. 30See http://bit.ly/NJOyUn for the whole conversation
- 32. 32 DISSECTING IMPACT Steve Evans: https://www.flickr.com/photos/babasteve/3140688308 CC BY
- 33. Impact 33
- 42. 42 Research
- 47. 47
- 48. 48
- 49. 49
- 50. 50
- 51. 51
- 52. 52
- 53. 53
- 54. 54
- 55. 55
- 56. 56
- 62. 62
- 63. “Stories that persuade with data” Anita de Waard 63
- 64. “Stories that persuade with data” Anita de Waard 64 evidence
- 65. TELLING A GOOD STORY 65Magenta Rose: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
- 66. 66
- 69. 69
- 71. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Citations for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of CrossRef citations, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 71
- 72. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Mendeley for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Mendeley bookmarks, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views A Collaboratively-Derived Science-Policy Research Agenda 72
- 73. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Facebook for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Facebook activity, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 73
- 74. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Wikipedia for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 74
- 75. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Wikipedia for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views Afghanistan's Ethnic Groups Share a Y-Chromosomal Heritage Structured by Historical Events 75
- 77. 77Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY There are four pillars to our research vision: •an unstinting commitment to research excellence •a willingness to harness that excellence to address some of the vital national and global challenges of our time •a desire to engage with the communities for whom our research has real meaning and consequences and from whom we can learn – whether in the private or public sector, local or global •new initiatives to develop, nurture and support our researchers; that is, to develop not only brilliant minds, but also research leaders.
- 79. 79 Academic quality is highly context - specific, and it is sensible to think in terms of research qualities, rather than striving for a single definition or measure of quality. The central problem identified here is that academic quality is a complex notion that cannot easily be reduced to quantification – the use of proxy variables runs the risk of misrepresenting the qualities of research contributions and may lead to unintended consequences. As PLOS noted in its submission to this review, “it is unclear whether any unique quality of research influence or impact is sufficiently general to be measured”. Wilsdon et al. (2015) - The Metric Tide
- 81. RESPONSIBILITY USE OF METRICS 81Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY
- 82. RESPONSIBILITY USE OF METRICS 82Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY indicators
- 84. …into a shared mission 84Daniel Spiess: https://www.flickr.com/photos/deegephotos/5984387881 CC BY-SA
- 85. Get informed… 85
- 86. Sign DORA 86
- 87. …and mean it 87
- 88. Above all… 88
- 90. It’s just good scholarship… 90
- 91. 91