Measuring Impact: Towards a data citation metric

576 views

Published on

How the ViBRANT and eMonocot projects are building tools, including a modified implementation of Bourne and Fink's 'Scholar Factor', the Biodiversity Data Journal, and Scratchpad's user metrics and statistics modules.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
576
On SlideShare
0
From Embeds
0
Number of Embeds
36
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • This work has been done under the ViBRANT and eMonocot projects. Much of this relates to Scratchpads which are in a simple terms an online content management system for biodiversity data (if you really want to know more you may want to leave and go next door as there is a summary of the project going on there).
  • These projects focus on data collection and curation, and making it available for reuse. Both projects use Scratchpads for a large part of this process.
  • In all large projects we like to know how well we are doing. But this is a pretty selfish use of statistics and metrics.
  • Communities are the core of these projects. We build tools for communities.
  • so we also need to consider how we build metrics tools for communities.
  • This might be allowing people to see how much they have contributed to a project, or conversely letting people using the project see who has contributed
  • We’d like a way to show people how the content they have generated is being used: who links to it, has it been shared socially, has it been traditionally cited, have people reused the data for novel purposes.
  • As part of ViBRANT there are a few citizen science projects so we also need to consider how we build metrics tools for communities
  • Collecting observation data via mobile phone
  • We’re working on a citizen science platform for Scratchpads
  • Metrics provide an added layer of incentive for citizen scientists – they could form the basis of a gamification reward system and allow people to quanify their involvement.
  • People who reuse their data. All of these different people need metrics to answer specific questions.
  • Perhaps the stereotypical metric – the h-index
  • We’ve found a way of creating a number out of one aspect of our output, and many people use it as proxy for the rest of what scientists do.
  • There has been a move towards alternative metrics, this example uses the altmetric
  • But there are still some problems with this method
  • Some parts of the metric, like Mendeley, can be considered to be fairly scholarly
  • Scratchpads metrics module
  • Scratchpads statistics module
  • It’s possible to publish data from a Scratchpad in the recently launched Biodiversity Data Journal, so an opportunity for some metrics activity.
  • Through CrossRef DOI
  • The Scratchpads statistics module provides information about the content on a given Scratchpad.
  • Filtering by user or taxonomic term means it’s easy to find out how much people have contributed, and how content is available for a given taxon.
  • If you’re looking for references or images you know how rich a resource this site might be.
  • Might give an indication of how active a community is
  • Do other people find the content useful?
  • This isn’t perfect, in fact it’s pretty experimental. So it will be on an opt-in basis.
  • Not every community wants the same set of criteria.
  • Traditional citations still play a part.
  • You also get credit for software you have written and datasets that you have created
  • Also get credit for grant and manuscript reviews. This is perhaps the trickiest part – but we are working with Pensoft to include this for reviews of papers in their journals.
  • Some of these things take more effort than others, so they are weighted.
  • So we have seen the theoretical model, now for the implementation we have created.
  • The user metrics module deals with the aggregation, weighting and display of metrics data.
  • A number of helper modules are responsible for getting the required data from external sources.
  • It’s easy to write a module that can supply data – maybe as little as 20 lines of code.
  • Modular design allow for a mix-and-match set of functionality, that also has some degree of future proofing. As a maybe slightly extreme example…
  • … we could add Facebook….
  • .. And remove legacy systems
  • One of the most basic web metrics available
  • We have done studies on links to Scratchpad content in articles on Google Scholar – and there has been a noticeable increase in recent years.
  • Also take some ideas from the altmetric community.
  • Measuring Impact: Towards a data citation metric

    1. 1. Measuring Impact Ed Baker & The Scratchpads Team
    2. 2. ViBRANT and eMonocot
    3. 3. ViBRANT and eMonocot • Projects that focus on data collection, curation and reuse
    4. 4. ViBRANT and eMonocot • Projects that focus on data collection, curation and reuse • Would like metrics to know how well we are doing
    5. 5. ViBRANT and eMonocot
    6. 6. ViBRANT and eMonocot Projects based around communities
    7. 7. ViBRANT and eMonocot Projects based around communities Provide tools for communities
    8. 8. ViBRANT and eMonocot Projects based around communities Provide tools for communities • Quantify user contribution to multi-user projects
    9. 9. ViBRANT and eMonocot Projects based around communities Provide tools for communities • Quantify user contribution to multi-user projects • Quantify how content generated by the community is used/reused
    10. 10. Citizen Science
    11. 11. Citizen Science • COMBER
    12. 12. Citizen Science • COMBER • anymals+plants
    13. 13. Citizen Science • COMBER • anymals+plants • Citizen Science profile
    14. 14. Citizen Science • COMBER • anymals+plants • Citizen Science profile Metrics allow for more competitive activities and for users to quantify their involvement
    15. 15. Who needs impact metrics?
    16. 16. Who needs impact metrics? • The authors of data
    17. 17. Who needs impact metrics? • The authors of data • People who employ data authors
    18. 18. Who needs impact metrics? • The authors of data • People who employ data authors • People who reuse the data
    19. 19. Who needs impact metrics? • The authors of data • People who employ data authors • People who reuse the data Am I useful?
    20. 20. Who needs impact metrics? • The authors of data Am I useful? • People who employ data authors • People who reuse the data Do we have our priorities right?
    21. 21. Who needs impact metrics? • The authors of data Am I useful? • People who employ data authors • People who reuse the data Do we have our priorities right?
    22. 22. Who needs impact metrics? • The authors of data Am I useful? • People who employ data authors • People who reuse the data Hidden gem? Do we have our priorities right?
    23. 23. Who needs impact metrics? • The authors of data Am I useful? • People who employ data authors • People who reuse the data Hidden gem? Avoid like the plague? Do we have our priorities right?
    24. 24. Our premise
    25. 25. We want people to share data…
    26. 26. …people want to please the “powers that be” …
    27. 27. … the powers that be like statistics.
    28. 28. So we made some
    29. 29. But the people weren’t happy.
    30. 30. But the people weren’t happy. Of all the things I do you only care about papers?
    31. 31. But the people weren’t happy. The only way you value my papers is by citation in other papers? Of all the things I do you only care about papers?
    32. 32. But the people weren’t happy. The only way you value my papers is by citation in other papers? Of all the things I do you only care about papers? The impact of my career is measured by one number?
    33. 33. So some people made their own
    34. 34. But there are still problems
    35. 35. But there are still problems This STILL only deals with my publications!
    36. 36. But there are still problems This STILL only deals with my publications! How do I get credit for data?
    37. 37. But there are still problems This STILL only deals with my publications! How do I get credit for data? I wrote some useful biodiversity software – what about that?
    38. 38. Are alt metrics useful?
    39. 39. Are alt metrics useful?
    40. 40. Are alt metrics useful? These are fairly scholarly
    41. 41. Are alt metrics useful? These measure broader public impact These are fairly scholarly
    42. 42. Are alt metrics useful? These measure broader public impact These are fairly scholarly • A good start
    43. 43. Are alt metrics useful? These measure broader public impact • A good start • Much broader than just scholarly citations These are fairly scholarly
    44. 44. Are alt metrics useful? These measure broader public impact • A good start • Much broader than just scholarly citations • Actually ignores traditional citations These are fairly scholarly
    45. 45. Are alt metrics useful? These measure broader public impact • A good start • Much broader than just scholarly citations • Actually ignores traditional citations These are fairly scholarly • Still only applied to papers
    46. 46. What are we doing?
    47. 47. What are we doing?
    48. 48. What are we doing?
    49. 49. What are we doing?
    50. 50. Biodiversity Data Journal Data papers (descriptions of data)
    51. 51. Biodiversity Data Journal Data papers (descriptions of data) • Allows data to become citable
    52. 52. Biodiversity Data Journal Data papers (descriptions of data) • Allows data to become citable • Allows data to participate in standard credit/metric systems
    53. 53. Biodiversity Data Journal Data papers (descriptions of data) • Allows data to become citable • Allows data to participate in standard credit/metric systems • … allows data to participate in alt metrics
    54. 54. Scratchpads Statistics Module What content exists on a Scratchpad site?
    55. 55. Scratchpads Statistics Module What content exists on a Scratchpad site? • Filter by user or taxonomic term
    56. 56. Scratchpads Statistics Module What content exists on a Scratchpad site? • Filter by user or taxonomic term • How much content is there?
    57. 57. Scratchpads Statistics Module What content exists on a Scratchpad site? • Filter by user or taxonomic term • How much content is there? • What kind of content is it?
    58. 58. Scratchpads Statistics Module What content exists on a Scratchpad site? • Filter by user or taxonomic term • How much content is there? • What kind of content is it? • How many registered users?
    59. 59. Scratchpads Statistics Module What content exists on a Scratchpad site? • Filter by user or taxonomic term • How much content is there? • What kind of content is it? • How many registered users? • How often is the content viewed?
    60. 60. Scratchpads Statistics Module Available now: http://antkey.org/stats [scratchpad.url]/stats
    61. 61. Scratchpads Metrics Module Puts the Scratchpad content in a broader context
    62. 62. Scratchpads Metrics Module Puts the Scratchpad content in a broader context • Users and content
    63. 63. Scratchpads Metrics Module Puts the Scratchpad content in a broader context • Users and content • Opt-in (not enabled by default)
    64. 64. Scratchpads Metrics Module Puts the Scratchpad content in a broader context • Users and content • Opt-in (not enabled by default) • Modular (pick and choose what to include)
    65. 65. Scratchpads Metrics Module Puts the Scratchpad content in a broader context • Users and content • Opt-in (not enabled by default) • Modular (pick and choose what to include) • Available by end of the year
    66. 66. Scratchpads User Metrics A partial implementation of the Scholar Factor proposed by Bourne and Fink
    67. 67. Scratchpads User Metrics A partial (modified) implementation of the Scholar Factor proposed by Bourne and Fink Citations
    68. 68. Scratchpads User Metrics A partial (modified) implementation of the Scholar Factor proposed by Bourne and Fink Citations Scholar Factor
    69. 69. Scratchpads User Metrics A partial (modified) implementation of the Scholar Factor proposed by Bourne and Fink Citations Software / Data Scholar Factor
    70. 70. Scratchpads User Metrics A partial (modified) implementation of the Scholar Factor proposed by Bourne and Fink Citations Software / Data Scholar Factor Grant / manuscript reviews
    71. 71. Scratchpads User Metrics A partial (modified) implementation of the Scholar Factor proposed by Bourne and Fink Citations Software / Data Scholar Factor Grant / manuscript reviews
    72. 72. Scratchpads User Metrics Scholar Factor
    73. 73. Scratchpads User Metrics Scratchpads User Metrics Module Scholar Factor
    74. 74. Scratchpads User Metrics Google Scholar (h-index) Scratchpads User Metrics Module Scholar Factor
    75. 75. Scratchpads User Metrics Google Scholar (h-index) Scratchpad Scratchpads User Metrics Module Scholar Factor
    76. 76. Scratchpads User Metrics Google Scholar (h-index) Scratchpad GitHub Scratchpads User Metrics Module Scholar Factor
    77. 77. Scratchpads User Metrics Google Scholar (h-index) Scratchpad GitHub Scratchpads User Metrics Module Scholar Factor ?
    78. 78. Scratchpads User Metrics Modular Google Scholar (h-index) Scratchpad GitHub Scratchpads User Metrics Module Scholar Factor ?
    79. 79. Scratchpads User Metrics Modular Google Scholar (h-index) Scratchpad GitHub Scratchpads User Metrics Module Scholar Factor ?
    80. 80. Scratchpads User Metrics Modular Google Scholar (h-index) Scratchpad GitHub Scratchpads User Metrics Module Scholar Factor ?
    81. 81. Same modular structure for content
    82. 82. Same modular structure for content • Number of web links to page
    83. 83. Same modular structure for content • Number of web links to page • Scholarly links (Google Scholar)
    84. 84. Same modular structure for content • Number of web links to page • Scholarly links (Google Scholar) • Social links
    85. 85. Summary We’re not there yet.
    86. 86. Summary We’re not there yet, but we have …
    87. 87. Summary We’re not there yet, but we have… • Modular framework to add services as they become available
    88. 88. Summary We’re not there yet, but we have… • Modular framework to add services as they become available • multi-dimensional – not just papers and citations
    89. 89. Summary We’re not there yet, but we have… • Modular framework to add services as they become available • multi-dimensional – not just papers and citations • Not a one size fits all approach
    90. 90. Summary We’re not there yet, but we have… • Modular framework to add services as they become available • multi-dimensional – not just papers and citations • Not a one size fits all approach

    ×