Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
No Evidence without Stories,
No Stories Without Evidence
Responsible use of metrics in research assessment
1
Cameron Neylo...
A QUESTION OF VALUES
2Nichole Burrows: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
3https://www.flickr.com/photos/schnappi/6116266037 CC BY
4
MissionMissionValues
Mission
5
AssessmentAssessment
Mission
6
AssessmentAssessment
7
http://www.flickr.com/photos/drachmann/327122302 CC-BY-SA
More data is the answer!
8https://www.flickr.com/photos/blprnt/3261815454 CC BY
9
DATA AND MEASUREMENT
Scott Akerman: http://www.flickr.com/photos/sterlic/4299631538 CC BY
10http://www.flickr.com/photos/verzo/8020565592/ CC BY
11http://www.flickr.com/photos/andrein/372192048/ CC BY-SA
More data is the answer?
12https://www.flickr.com/photos/blprnt/3261815454 CC BY
Types of Data
•Citations
•Usage
•Bookmarks
•Conversation
Citations
•Traditional and valued
•Less consistent than most think
– WoS, Scopus, CrossRef,
(Europe)PMC
•Track a very spec...
Usage
•Page views, downloads
•Measure usage by a wider
community
•Measure greater range of use
•Inconsistent between publi...
Bookmarks
•Academic and “public” facing sites
– People collecting literature
– Building a library
•Measure greater range o...
Conversations/Social Media
•Twitter, Facebook, Blogs, News sites
– Sharing literature within
communities
– Conversations a...
A story…
18
19altmetric.com
20
21
22
0 20 40 60 80
0
5000
10000
15000
PLOS papers authored by University of Cape Town staff
Usage data from the PLoS journal we...
...but interesting
interactions...
24
25
…driven by this
person...
26
...who I can talk to.
27
28
29
30See http://bit.ly/NJOyUn for the whole conversation
This. Is Impact.
31
32
DISSECTING IMPACT
Steve Evans: https://www.flickr.com/photos/babasteve/3140688308 CC BY
Impact
33
Impact
34
Research
Economic
Cultural
Education
Health
Environment
Impact
35
Research
Economic
Cultural
Education
Health
Environment
36
Research
Economic
Cultural
Education
Health
Environment
37
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs
38
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs
39
Research
Economic
Cultural
Education
Health
Environment
Research
Outputs
40
Research
Research
Outputs
41
Research
Research
Outputs
42
Research
43
Research
Citation
44
Research
Bookmark
Citation
45
Research
Bookmark
Citation
Previous
co-authorship
46
Research
Bookmark
Citation
Previous
co-authorship
Conference
co-attendance
Social media
conversation
47
48
49
50
51
52
53
54
55
56
57https://www.flickr.com/photos/see-through-the-eye-of-g/5392290809 CC BY
58https://www.flickr.com/photos/jenny-pics/3239638494 CC BY
59https://www.flickr.com/photos/garycycles2/2575676610 CC BY
60https://www.flickr.com/photos/53487196@N08/5439253136 CC BY
61https://www.flickr.com/photos/jenny-pics/3239638494 CC BY
62
“Stories that persuade with data”
Anita de Waard
63
“Stories that persuade with data”
Anita de Waard
64
evidence
TELLING A GOOD STORY
65Magenta Rose: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
66
67m01229: https://www.flickr.com/photos/39908901@N06/8538482488 CC BY
68Martin Howard: https://www.flickr.com/photos/martinhoward/2709364519 CC BY
69
70
http://ivory.idyll.org/blog/tag/skepticism.html
12 13 14 15 16
0
5000
10000
15000
Usage vs. Citations for Harvard Authors 01−04/12
220 PLOS articles from Harvard Universi...
12 13 14 15 16
0
5000
10000
15000
Usage vs. Mendeley for Harvard Authors 01−04/12
220 PLOS articles from Harvard Universit...
12 13 14 15 16
0
5000
10000
15000
Usage vs. Facebook for Harvard Authors 01−04/12
220 PLOS articles from Harvard Universit...
12 13 14 15 16
0
5000
10000
15000
Usage vs. Wikipedia for Harvard Authors 01−04/12
220 PLOS articles from Harvard Universi...
12 13 14 15 16
0
5000
10000
15000
Usage vs. Wikipedia for Harvard Authors 01−04/12
220 PLOS articles from Harvard Universi...
76Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY
77Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY
There are four pillars to our research vision:
•an...
“Qualities” not “Quality”
78
79
Academic quality is highly context - specific, and it is
sensible to think in terms of research qualities, rather
than ...
80
MissionMissionValues
RESPONSIBILITY USE OF METRICS
81Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY
RESPONSIBILITY USE OF METRICS
82Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY
indicators
Develop the stories…
83https://www.flickr.com/photos/jblmmwr/8027451065 CC BY
…into a shared mission
84Daniel Spiess: https://www.flickr.com/photos/deegephotos/5984387881 CC BY-SA
Get informed…
85
Sign DORA
86
…and mean it
87
Above all…
88
89Joe Flintham: https://www.flickr.com/photos/joeflintham/5631590491 CC BY-SA
It’s just good scholarship…
90
91
92https://www.flickr.com/photos/ell-r-brown/4655401891 CC BY
@cameronneylon
cn@cameronneylon.net
http://orcid.org/0000-0002-0068-716X
93
Upcoming SlideShare
Loading in …5
×

No stories without evidence, no evidence without stories

1,478 views

Published on

Talk given at Sydney University on 4 August 2015.

Across many parts of our lives we are faced with the increasing availability of data to support decision making. With every element of the research process moving online, there are many new sources of data, as well as improved old sources of data, that can provide information on the performance, value and use of research and researchers.

But there is a problem. The proliferation of proxy data, and their naive equation with such weakly defined concepts as “quality” and “excellence”, have lead to a reliance on rankings and quantitative measures as institutional targets. More than this the adoption of these instrumental targets has lead us away from a critical discussion of institutional values, indeed of what the institution is for.

I will argue that it is only by moving away from such vague terms as “quality”, “excellence” and “impact” and focussing on institutional values and a well articulated mission that institutions of scholarship will continue to be relevant for the future. It is through interrogating the goals of the institution that the enormous potential resource of data on the research enterprise can be realised. Using the data effectively will allow us a window on how knowledge actually moves and is used. In combination with a clear sense of institutional goals this provides great opportunities for institutions to differentiate themselves from the pack.

Published in: Science
  • Be the first to comment

No stories without evidence, no evidence without stories

  1. 1. No Evidence without Stories, No Stories Without Evidence Responsible use of metrics in research assessment 1 Cameron Neylon - 4 August 2015 - University of Sydney @cameronneylon - http://orcid.org/0000-0002-0068-716X
  2. 2. A QUESTION OF VALUES 2Nichole Burrows: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
  3. 3. 3https://www.flickr.com/photos/schnappi/6116266037 CC BY
  4. 4. 4 MissionMissionValues
  5. 5. Mission 5 AssessmentAssessment
  6. 6. Mission 6 AssessmentAssessment
  7. 7. 7 http://www.flickr.com/photos/drachmann/327122302 CC-BY-SA
  8. 8. More data is the answer! 8https://www.flickr.com/photos/blprnt/3261815454 CC BY
  9. 9. 9 DATA AND MEASUREMENT Scott Akerman: http://www.flickr.com/photos/sterlic/4299631538 CC BY
  10. 10. 10http://www.flickr.com/photos/verzo/8020565592/ CC BY
  11. 11. 11http://www.flickr.com/photos/andrein/372192048/ CC BY-SA
  12. 12. More data is the answer? 12https://www.flickr.com/photos/blprnt/3261815454 CC BY
  13. 13. Types of Data •Citations •Usage •Bookmarks •Conversation
  14. 14. Citations •Traditional and valued •Less consistent than most think – WoS, Scopus, CrossRef, (Europe)PMC •Track a very specific kind of usage •Only track a very specific kind of
  15. 15. Usage •Page views, downloads •Measure usage by a wider community •Measure greater range of use •Inconsistent between publishers •Rarely get demographics of viewers
  16. 16. Bookmarks •Academic and “public” facing sites – People collecting literature – Building a library •Measure greater range of use vs cites •Focus on use by researchers (Mendeley)
  17. 17. Conversations/Social Media •Twitter, Facebook, Blogs, News sites – Sharing literature within communities – Conversations about literature •Open to the public (but often actually driven by/involving researchers)
  18. 18. A story… 18
  19. 19. 19altmetric.com
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. 0 20 40 60 80 0 5000 10000 15000 PLOS papers authored by University of Cape Town staff Usage data from the PLoS journal website (combined HTML views and PDF downloads). Circle size correlates with CrossRef citation counts. Red circles represent papers with "HIV" in title. Months Total Views Not an obvious outlier... 23
  24. 24. ...but interesting interactions... 24
  25. 25. 25
  26. 26. …driven by this person... 26
  27. 27. ...who I can talk to. 27
  28. 28. 28
  29. 29. 29
  30. 30. 30See http://bit.ly/NJOyUn for the whole conversation
  31. 31. This. Is Impact. 31
  32. 32. 32 DISSECTING IMPACT Steve Evans: https://www.flickr.com/photos/babasteve/3140688308 CC BY
  33. 33. Impact 33
  34. 34. Impact 34 Research Economic Cultural Education Health Environment
  35. 35. Impact 35 Research Economic Cultural Education Health Environment
  36. 36. 36 Research Economic Cultural Education Health Environment
  37. 37. 37 Research Economic Cultural Education Health Environment Research Outputs
  38. 38. 38 Research Economic Cultural Education Health Environment Research Outputs
  39. 39. 39 Research Economic Cultural Education Health Environment Research Outputs
  40. 40. 40 Research Research Outputs
  41. 41. 41 Research Research Outputs
  42. 42. 42 Research
  43. 43. 43 Research Citation
  44. 44. 44 Research Bookmark Citation
  45. 45. 45 Research Bookmark Citation Previous co-authorship
  46. 46. 46 Research Bookmark Citation Previous co-authorship Conference co-attendance Social media conversation
  47. 47. 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. 51
  52. 52. 52
  53. 53. 53
  54. 54. 54
  55. 55. 55
  56. 56. 56
  57. 57. 57https://www.flickr.com/photos/see-through-the-eye-of-g/5392290809 CC BY
  58. 58. 58https://www.flickr.com/photos/jenny-pics/3239638494 CC BY
  59. 59. 59https://www.flickr.com/photos/garycycles2/2575676610 CC BY
  60. 60. 60https://www.flickr.com/photos/53487196@N08/5439253136 CC BY
  61. 61. 61https://www.flickr.com/photos/jenny-pics/3239638494 CC BY
  62. 62. 62
  63. 63. “Stories that persuade with data” Anita de Waard 63
  64. 64. “Stories that persuade with data” Anita de Waard 64 evidence
  65. 65. TELLING A GOOD STORY 65Magenta Rose: https://www.flickr.com/photos/130132803@N07/18778753910 CC BY
  66. 66. 66
  67. 67. 67m01229: https://www.flickr.com/photos/39908901@N06/8538482488 CC BY
  68. 68. 68Martin Howard: https://www.flickr.com/photos/martinhoward/2709364519 CC BY
  69. 69. 69
  70. 70. 70 http://ivory.idyll.org/blog/tag/skepticism.html
  71. 71. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Citations for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of CrossRef citations, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 71
  72. 72. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Mendeley for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Mendeley bookmarks, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views A Collaboratively-Derived Science-Policy Research Agenda 72
  73. 73. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Facebook for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Facebook activity, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 73
  74. 74. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Wikipedia for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views 74
  75. 75. 12 13 14 15 16 0 5000 10000 15000 Usage vs. Wikipedia for Harvard Authors 01−04/12 220 PLOS articles from Harvard University authors, published January to April 2012. Bubble size correlates with number of Wikipedia pages, and color with PLOS journal. Data collected April 11, 2013. Age in Months Total Views Afghanistan's Ethnic Groups Share a Y-Chromosomal Heritage Structured by Historical Events 75
  76. 76. 76Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY
  77. 77. 77Philip Choi: https://www.flickr.com/photos/superturtle/121564208 CC BY There are four pillars to our research vision: •an unstinting commitment to research excellence •a willingness to harness that excellence to address some of the vital national and global challenges of our time •a desire to engage with the communities for whom our research has real meaning and consequences and from whom we can learn – whether in the private or public sector, local or global •new initiatives to develop, nurture and support our researchers; that is, to develop not only brilliant minds, but also research leaders.
  78. 78. “Qualities” not “Quality” 78
  79. 79. 79 Academic quality is highly context - specific, and it is sensible to think in terms of research qualities, rather than striving for a single definition or measure of quality. The central problem identified here is that academic quality is a complex notion that cannot easily be reduced to quantification – the use of proxy variables runs the risk of misrepresenting the qualities of research contributions and may lead to unintended consequences. As PLOS noted in its submission to this review, “it is unclear whether any unique quality of research influence or impact is sufficiently general to be measured”. Wilsdon et al. (2015) - The Metric Tide
  80. 80. 80 MissionMissionValues
  81. 81. RESPONSIBILITY USE OF METRICS 81Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY
  82. 82. RESPONSIBILITY USE OF METRICS 82Joel Dinda: https://www.flickr.com/photos/mwlguide/5788522109CC BY indicators
  83. 83. Develop the stories… 83https://www.flickr.com/photos/jblmmwr/8027451065 CC BY
  84. 84. …into a shared mission 84Daniel Spiess: https://www.flickr.com/photos/deegephotos/5984387881 CC BY-SA
  85. 85. Get informed… 85
  86. 86. Sign DORA 86
  87. 87. …and mean it 87
  88. 88. Above all… 88
  89. 89. 89Joe Flintham: https://www.flickr.com/photos/joeflintham/5631590491 CC BY-SA
  90. 90. It’s just good scholarship… 90
  91. 91. 91
  92. 92. 92https://www.flickr.com/photos/ell-r-brown/4655401891 CC BY
  93. 93. @cameronneylon cn@cameronneylon.net http://orcid.org/0000-0002-0068-716X 93

×