Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Value impact researchdataservices_esip_2017

139 views

Published on

Presentation to the Earth Science Information Partners (ESIP) in Bloomington Indiana 27 July 2017. Presentation covers value and economic impact studies by Charles Beagrie Ltd and our CESSDA SaW cost -benefit advocacy toolkit. A particular focus given to Earth Sciences.

Published in: Science
  • Be the first to comment

Value impact researchdataservices_esip_2017

  1. 1. Earth Science Information Partners (ESIP) Workshop July 2017 Neil Beagrie (Charles Beagrie Ltd) Measuring the Value and (Economic) Impact of Research Data Services Illustration by Jørgen Stamp digitalbevaring.dk CC BY 2.5 Denmark
  2. 2. About us • Charles Beagrie Ltd – independent consultancy in digital archive, library, science and research sectors. • Neil Beagrie consultant specialising in cost-benefit analysis, research data curation and digital preservation, e.g. KRDS projects. • Prof John Houghton economist specialising in applying economic methods to information technology policy, science and technology policy, and Open Access.
  3. 3. Presentation Overview • Previous work(note focussing on cost-benefit) • ESDS Impact Study • ADS/BADC/EBI Impact Studies • CESSDA Cost-Benefit Advocacy Toolkit • Key Challenges and Lessons Informal – happy to take questions as we go
  4. 4. Why measure value and (economic) impact? • Advocacy: proving (to non-researchers) that current services or future services are a worthwhile investment; • Complementary to other approaches such as data/article citation analysis; • Can show more of the impact on research, teaching & learning, etc. Note however: • It is challenging to do; • Methods well-established but impact data hard to obtain and to use.
  5. 5. Previous Work Big Science and Innovation Study for BIS July 2013 • Desk review of c. 100 studies internationally; • 3 studies highlighted to BIS as being particularly good examples of ‘good practice’ in the measurement of economic impacts: • Berkeley Lab 2010 • Human Genome Project 2011 • Economic and Social Data Service (ESDS) 2012
  6. 6. The ESDS Study
  7. 7. Best Practice from ESDS study • Applies range of methods; • Includes counter- factual; • Data collection: users & depositors, research & teaching; • Data weighting - survey value responses weighted to reflect the overall pattern of use from weblogs; • Case studies/ KRDS benefits illustrate benefits and impact pathways.
  8. 8. ESDS Study: Returns on Investment (ROI) ROI - service Benefit/cost ratio of net economic value to ESDS operational costs ROI - research data creation scenarios -Increase in returns from data re-use facilitated by ESDS (counterfactual) 5.4 to 1 Up to 10 to 1
  9. 9. ESDS Study: Researcher Efficiency Gains Impact of using ESDS data and services on research efficiency (after Beagrie et al 2012, p77, Figure 15) Economic and Social Research Council © 2012 CC-BY licensed
  10. 10. The Value and Impact Studies
  11. 11. Economic & Social Data Service (ESDS) Archaeology Data Service British Atmospheric Data Centre European Bioinformatics Institute Value + Impact Analyses of Data Services John Houghton + Neil Beagrie 4 joint studies to date. 5th in progress. Methods applied to:
  12. 12. Approaches and Methods • We combined quantitative and qualitative approaches. • All studies used: • Desk-based analysis of existing literature and reports; • Existing management and internal data collected by the data services; and • Original data collection in the form of online surveys, together with semi-structured interviews.
  13. 13. Indicative Scale (currently £1.00 = $1.31)
  14. 14. The Value and Impact of the British Atmospheric Data Centre
  15. 15. Key FindingsThe economic analysis shows that: • increase in research efficiency estimated as at least £10 million per annum. • Increase in ROI from the data re-use facilitated by BADC estimated as between £11 million and £34 million over thirty years (net present value) from one-year’s investment. The qualitative analysis in the user survey showed that: • 81% of the academic respondents reported that BADC was very or extremely important for their academic research, and • 53% of respondents reported that it would have a major or severe impact on their work if they could not access BADC data and services. In the depositor survey: • 52% suggested that being unable to deposit data with BADC would have a major or severe impact on them.
  16. 16. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. CESSDA SaW Toolkit
  17. 17. ESDS Study: Context • UK Data Archive one of the largest European social science data archives: perhaps the largest • UK Data Archive established for 40 years: has built up collections and users over time • Only economic impact study for any social science data archive to date • How to extend work for other social science archives?
  18. 18. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. CESSDA-SaW Task 4.6 Capturing and communicating the value and economic impact of social science data services. Develop a benefit/cost advocacy programme and supporting tools; assembling an evidence base to support the negotiation with ministries and funding organisations; support advocacy with other core stakeholders such as data creators and data users. Toolkit published April 2017.
  19. 19. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. CESSDA-SaW User Requirements Survey
  20. 20. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Survey Q2 THE DATA SERVICE'S CURRENT STAFFING IS APPROXIMATELY:
  21. 21. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Toolkit Components • Factsheets • ROI, Benefits, Costs • Worksheets • Benefits Summary for a Data Archive • Archive Development Canvas • Case studies • ADP, FSD, LiDA, UKDS • Selected External Tools • CCeX, KRDS, CDMA, ESDS Impact, ADP surveys , etc • User Guide
  22. 22. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. The Factsheets
  23. 23. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. ROI Factsheet (1)
  24. 24. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. ROI Factsheet (2)
  25. 25. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Counter-factuals – “Costs of Inaction” “Ideally, economic impact assessments should estimate the counterfactual – i.e. what would occur in the absence of the facility…However, counterfactuals are rarely addressed in the [c.100] studies reviewed due to lack of data. We found two exceptions that address this issue partially. One is the evaluation of the economic impacts of ESDS (2012) which partially explores the counterfactual through a users’ survey…Another exception is a review of economic impacts of large-scale science facilities in the UK (SQW, 2008) … however, this estimation is not done rigorously and relies mostly on the estimation of the local benefits.” Big Science and Innovation - Report to BIS - Technopolis 2013
  26. 26. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. ROI Factsheet (3)
  27. 27. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Benefits Factsheet (1)
  28. 28. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Benefits Factsheet (2)
  29. 29. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Benefits Factsheet (3)
  30. 30. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Costs Factsheet (1)
  31. 31. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Costs Factsheet (2)
  32. 32. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Bringing it all together: The Archive Development Canvas
  33. 33. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939.
  34. 34. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. The Archive Development Canvas (detailed)
  35. 35. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. The Archive Development Canvas (mapped)
  36. 36. Key Challenges and Lessons
  37. 37. EBI Report: User population • How ‘actual direct users’ were estimated from web stats: • Unique host log of 10.8 million (Annual figure for 2014) adjusted to reflect key considerations: • People behind unique IPs in web stats usually over-estimated by a factor of 7 (Formitchev, 2010) • EMBL-EBI accessed by organisational IPs (survey: 4185 responses from 3622 unique IPs): Factor in 1.16 per unique IP • Respondents accessed 9 services on average – 10.8 million figure captured across EMBL-EBI resources • Total user population estimate: 198,000 unique, actual direct users
  38. 38. Online Surveys (BADC) • Two independent online surveys were conducted by CB Ltd; • Survey questionnaires developed iteratively - small group of BADC customers helped test pilot versions; • Given the nature of subject and diverse customer base substantial effort was needed to design questionnaires; • Significant effort spent to reduce the likely burden on questionnaire recipients.
  39. 39. Online surveys (BADC) Significant time spent on: • wording of the invitations to participate • advance notice of the surveys via websites, news feeds, social media and email shots Five £50 Amazon vouchers offered in a draw for participants Surveys had high response rates and completion rates
  40. 40. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Toolkit challenges & lessons • diverse scale / type of organisations • complexity of topic / local resources
  41. 41. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Effort Grading Levels hours days months
  42. 42. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement number 674939. Design criteria • Easy for overloaded individuals/ smaller services • Short documents • Good Infographics • Synthesis • Making existing tools easier to use/tailored to (social science) data services
  43. 43. Conclusions • Value and (economic) impact analysis doable but challenging; • Difficulties and critical points for economic analysis include: • Limited knowledge of real user populations; • Survey sample and survey design; • Complexity of mature data services and their resources; • "Costs of inaction" data - counter-factuals for data services. • CESSDA SaW toolkit approach adaptable to other disciplines and can tailor for/utilise sector specific data and needs.
  44. 44. Questions?

×