MICS @ SwafS

1. Methods for measuring
citizen-science impact
online
SwafS citizen-science cluster meeting on impact
10 November 2021
Uta Wehn, IHE Delft
Luigi Ceccaroni, Earthwatch

2. 2
• MICS framework development for impact assessment
• Co-design of impact journeys in specific case-study sites
• General platform to measure impact
Content

3. Framework development
3
Literature
review
Initial
framework
Progress the state of the art
Increase accessibility
Set of qu
Targeted lit
review
Measuring Impact of Citizen Science
mics.tools
Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021)
Impact assessment of citizen science: state of the art
and guiding principles for a consolidated
approach. Sustainability Science
https://doi.org/10.1007/s11625-021-00959-2

4. Framework development
4
Initial
framework
Progress the state of the art
Increase accessibility and
adoption
Set of questions to
capture the value of
impact indicators
Targeted literature
review and analysis
mics.tools
platform
Additional
guidance
Measuring Impact of Citizen Science
mics.tools

5. Framework development
5
Initial
framework
Progress the state of the art
Increase accessibility and
adoption
Set of questions to
capture the value of
impact indicators
Targeted literature
review and analysis
[mics.tools]
MICS platform
Additional
guidance
Measuring Impact of Citizen Science
mics.tools
Impact assessment in
the MICS
case studies

6. 6
• Workshop with practitioners & researchers from NBSs held
at the 2019 River Restoration conference in Liverpool
• 11 Interviews with 10 CS project coordinators (WP2/WP3)
• Screened 503 projects, shortlisted 130 projects, in-depth
analysis of 24 reference projects for IA methodologies,
tools and indicators
• Review of 9 general IA frameworks, 77 CS-specific IA
frameworks & empirical insights -> CS literature review
collaboration with selected WeObserve Impact CoP
members, joint paper
Developing the MICS conceptual framework
Inductive
MICS
Conceptual
Framework
Deductive

7. Accommodate range of purposes of
citizen-science impact assessment
Non-linear impact journeys
rather than impact silos
IA data collection methods &
information sources
Relative vs absolute impact
2
1
3
4
Guiding principles for MICS conceptual framework
Comparison of IA results
across citizen-science projects
5 Cumulative enhancement of
the framework over time
6 Wehn, U., Gharesifard, M., Ceccaroni, L. et al. (2021) Impact assessment
of citizen science: state of the art and guiding principles for a consolidated
approach. Sustainability Science https://doi.org/10.1007/s11625-021-
00959-2

8. Implementing the MICS conceptual framework
8
Theory of change elements

9. 9
MICS Indicator
Characteristics Description
Name of Indicator Informative short name for the indicator
Description Succinct description of the indicator (what is it an indicator of)
Domain
Main domain to which the indicator contributes insights [society,
science, economy, environment, governance]
Qualitative/Quantitative
Type of data (quantitative or qualitative).
In some cases, the basic data will be qualitative (interview transcripts,
national reports or similar) which require coding / categorisation in
order to be useful for monitoring purposes.
Primary/Secondary data
Indicator based on:
 Primary data (that needs to be collected) or
 Secondary data (already existing)
 estimates
Source of data
 Primary data: specification of data collection method (e.g.
survey, interview, focus group discussion, observation, etc.)
 Secondary data: specification of existing database, document
(specific page numbers, exact tables etc.), and direct links to
the data source in question (if available)
 Estimates (summaries): expert judgement, e.g. project
coordinator
Time-series
For primary data:
 Time-series required (yes/no; if yes, how frequent)
For secondary data:
 Time-series available (yes/no) and (if applicable) actual years for
which data are available
Unit of measurement (observation)
Unit of measurement (observation), e.g. nominal, ordinal, interval
Unit of analysis
Unit of analysis (e.g. country, organisation, individuals (e.g. citizens),
publications etc.)
Analytical level (logic model)
Analytical level in the intervention logic model to which the indicator is
oriented (i.e. context, input, process, output, outcome, impact).
Links with indicators in other
domains
Domain(s)/indicator(s) to which the indicator (also) relates
Data collection
Details on how data should be collected for this indicator
 Method (survey / questionnaire, data retrieved from
databases, structured/semi-structured/explorative interviews,
focus groups, observation, desk research, document analysis,
ethnographic field studies, etc.)
 Questions asked
 Type & # of respondents / informants, incl. size of this
population
Indicator building
Explanation of how results are calculated from collected data.
For quantitative sub-indicators, specify the unit (e.g. #, %, etc.)
Availability of data
Extent of data availability for the indicator and sub-indicators (e.g. low,
moderate, high)
Implementing the MICS conceptual framework

10. 10
Economy - Indicators ID
Indicator Characteristics
Name of Indicator Company growth International trade & investment Innovation & research Competit
Primary/Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (surveys) and Secondary data Primary (s
Description <copy from D1.10>
Qual/Quantitative Quantitatve Quantitative & qualitative Quantitatve Quantitat
Source of data dedicated survey dedicated survey dedicated survey dedicated
Time-series Yes, survey should be repeated Yes, survey should be repeated Yes, survey should be repeated Yes, surve
Unit of measurement (observation) absolute values & nominal (see data collection) absolute value absolute value absolute v
Unit of analysis Organisation Organisation Organisation Organisat
Analytical level Outcome; Impact
Links with indicators in other domains n.a. n.a. n.a. n.a.
Data collection
Survey - questions:
How many jobs are currently directly related to [CO topic] and
enabling technologies?
What is the nature of these jobs? (junior, medior, senior
position(s)) [nominal]
How many of your products/services are relevant for the
provision of COs?
What was your organisation's annual turnover in [year]?
What is your organisation's market share in the business of
COs?
How many clients does your organsition have in the CO
business?
Survey - questions:
How many international clients does your organsition have in the
CO business?
What specific customer segments does your organisation serve
related to COs?
How much has your organisation invested in CO-related activities in
[year]?
Survey - questions:
How many IPRs related to COs and enabling technologies
(patents, trademarks, copyright, other know-how rights)
does your organisation hold?
How many CO-related research projects is your organisation
currently involved?
In total, what is your organisation's budget (income & own
investment) in these CO-related research projects?
Survey - q
What valu
does your
Which ma
How man
for CO-rel
How man
organisati
Indicator building
Items:
# subject-related jobs
Nature of jobs
# of CO related products/services
Turnover
Market share in the business of Cos
# of clients in CO business and enabling technologies
Indicators:
# of international clients CO business and enabling tech-nologies
Customer segments (sectors) related to CO
Amount of investment in CO-related activities
<explain relation of analysed results across items above>
Indicators:
IPR (patents, trademarks, copyright, other know-how rights)
# of CO-related research projects
Total budget of CO-related research projects
<explain relation of analysed results across items above>
Indicators
# of reven
# of mark
# of CO to
<explain r
Science &
Technology
Society Economy Environment Governance
MICS Indicator
Characteristics Description
Name of Indicator Informative short name for the indicator
Description Succinct description of the indicator (what is it an indicator of)
Domain
Main domain to which the indicator contributes insights [society,
science, economy, environment, governance]
Qualitative/Quantitative
Type of data (quantitative or qualitative).
In some cases, the basic data will be qualitative (interview transcripts,
national reports or similar) which require coding / categorisation in
order to be useful for monitoring purposes.
Primary/Secondary data
Indicator based on:
 Primary data (that needs to be collected) or
 Secondary data (already existing)
 estimates
Source of data
 Primary data: specification of data collection method (e.g.
survey, interview, focus group discussion, observation, etc.)
 Secondary data: specification of existing database, document
(specific page numbers, exact tables etc.), and direct links to
the data source in question (if available)
 Estimates (summaries): expert judgement, e.g. project
coordinator
Time-series
For primary data:
 Time-series required (yes/no; if yes, how frequent)
For secondary data:
 Time-series available (yes/no) and (if applicable) actual years for
which data are available
Unit of measurement (observation)
Unit of measurement (observation), e.g. nominal, ordinal, interval
Unit of analysis
Unit of analysis (e.g. country, organisation, individuals (e.g. citizens),
publications etc.)
Analytical level (logic model)
Analytical level in the intervention logic model to which the indicator is
oriented (i.e. context, input, process, output, outcome, impact).
Links with indicators in other
domains
Domain(s)/indicator(s) to which the indicator (also) relates
Data collection
Details on how data should be collected for this indicator
 Method (survey / questionnaire, data retrieved from
databases, structured/semi-structured/explorative interviews,
focus groups, observation, desk research, document analysis,
ethnographic field studies, etc.)
 Questions asked
 Type & # of respondents / informants, incl. size of this
population
Indicator building
Explanation of how results are calculated from collected data.
For quantitative sub-indicators, specify the unit (e.g. #, %, etc.)
Availability of data
Extent of data availability for the indicator and sub-indicators (e.g. low,
moderate, high)
Implementing the MICS conceptual framework

11. Guidance for participatory evaluation
in the case studies
11
Measuring Impact of Citizen Science
mics.tools
Co-creation of Impact Journeys
CS Impact Assessment Compendium per case study
‘One stop’ collection of Impact Assessment instructions & findings per
case study

12. Co-creating Impact Journeys
Case study – Outfall Safari, UK
12
… and Project managers
Citizen scientists…
Impact journey workshops (online)

13. 13
Measuring Impact of Citizen Science
mics.tools
Refining & validating the impact journeys
Co-creating Impact Journeys
Case study – Outfall Safari, UK

14. 14
Economy - IndicatorsID
IndicatorCharacteristics
Name of Indicator Company growth Internationaltrade & investment Innovation & research Competitiveness
Primary/Secondary data Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata Primary(surveys)andSecondarydata
Description <copyfromD1.10>
Qual/Quantitative Quantitatve Quantitative &qualitative Quantitatve Quantitative &qualitative
Source of data dedicatedsurvey dedicatedsurvey dedicatedsurvey dedicatedsurvey
Time-series Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated Yes,surveyshouldbe repeated
Unit of measurement (observation) absolute values &nominal(see data collection) absolute value absolute value absolute value
Unit of analysis Organisation Organisation Organisation Organisation
Analytical level Outcome; Impact
Links with indicators in other domains n.a. n.a. n.a. n.a.
Data collection
Survey- questions:
How manyjobs are currentlydirectlyrelatedto [CO topic] and
enablingtechnologies?
What is the nature of these jobs?(junior,medior,senior
position(s))[nominal]
How manyof yourproducts/services are relevant forthe
provisionof COs?
What was yourorganisation's annualturnoverin[year]?
What is yourorganisation's market share inthe business of
COs?
How manyclients does yourorgansitionhave inthe CO
business?
Survey- questions:
How manyinternationalclients does yourorgansitionhave inthe
CO business?
What specific customersegments does yourorganisationserve
relatedto COs?
How muchhas yourorganisationinvestedinCO-relatedactivities in
[year]?
Survey- questions:
How manyIPRs relatedto COs andenablingtechnologies
(patents,trademarks,copyright,otherknow-how rights)
does yourorganisationhold?
How manyCO-relatedresearchprojects is yourorganisation
currentlyinvolved?
Intotal,what is yourorganisation's budget (income &own
investment)inthese CO-relatedresearchprojects?
Survey- questions:
What value proposition(s)relatedto COs andenablingtechnologies
does yourorganisationhave?
Whichmarket segments does yourorganisationserve?
How manydifferent revenustreams does yourorgansiationhave
forCO-relatedvalue propositions?
How manypartners forCos andenablingtechnologies does your
organisationhave?
Indicator building
Items:
# subject-relatedjobs
Nature of jobs
# of CO relatedproducts/services
Turnover
Market share inthe business of Cos
# of clients inCO business andenablingtechnologies
<explainrelationof analysedresults across items above>
Indicators:
# of internationalclients CO business andenablingtech-nologies
Customersegments (sectors)relatedto CO
Amount of investment inCO-relatedactivities
<explainrelationof analysedresults across items above>
Indicators:
IPR (patents,trademarks,copyright,otherknow-how rights)
# of CO-relatedresearchprojects
Totalbudget of CO-relatedresearchprojects
<explainrelationof analysedresults across items above>
Indicators:
# of revenue streams
# of market segments served
# of CO topic-relatedpartners
<explainrelationof analysedresults across items above>
Availability of data low low low low
Statistical robustness No validationconducted
Feasibility / Replicability moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding moderatelyresource demanding
Comments/caveats
Bottom up (online workshops) Science-based (MICS conceptual framework)
Co-creating Impact Journeys
Case study – Outfall Safari, UK
From impact journeys to monitoring schemes

15. 15
Measuring of impacts
Indicators to measure
short- & long-term
impacts
Co-creation & validation
of impact journey
Co-creating Impact Journeys
Case study – Outfall Safari, UK
From impact journeys to monitoring schemes

16. mics.tools
16
Measuring Impact of Citizen Science
mics.tools

17. The MICS platform (still under development)
17
Measuring Impact of Citizen Science
mics.tools

18. The MICS platform (still under development)
18
Measuring Impact of Citizen Science
mics.tools

19. The MICS platform (still under development)
19
Measuring Impact of Citizen Science
mics.tools

20. The MICS platform (still under development)
20
Measuring Impact of Citizen Science
mics.tools
Citizen science
Agricultural and veterinary sciences
Art theory and criticism
Biological sciences
Chemical sciences
Earth sciences
Engineering
Environmental sciences
Information and computing sciences
Language, communication and culture
Law and legal studies
Mathematical sciences and statistics
Medical and health sciences
Philosophy and religious studies
Physical sciences
Psychology and cognitive sciences
Studies in human society
Technology

21. The MICS platform (still under development)
21
Measuring Impact of Citizen Science
mics.tools

22. The MICS platform (still under development)
22
Measuring Impact of Citizen Science
mics.tools
Environment
24

23. The MICS platform (still under development)
23
Measuring Impact of Citizen Science
mics.tools
What engagement approaches does the project use?
 I don’t know
Previous Next
 Gamification
 Real-time validation on
observation/data submission
 Notification of updates to contributed
observations
 Tips of the day
 Personalised recommendations
 Feedback on data quality
 Active and responsive social-media
presence ("responsive" in the sense
that questions from participants are
responded to)
 Storytelling or exchanging of
experiences
 Invitation to or suggestion of events
 Workshops
 Measurement campaigns
 Active and responsive pilot / case
study presence ("responsive" in the
sense that questions from participants
are responded to)
 A forum, or other platform on a
website
 Awareness raising (ad-hoc
demonstrations, meetings and
appearances from partners)
 Photo contests
 Social events not included in previous
options
 Creation of champions, ambassadors
or leaders
 Organic conversation

24. The MICS platform (still under development)
24
Measuring Impact of Citizen Science
mics.tools
Does the project have a formal engagement strategy?
 Yes
 No
 I don’t know
Want an example engagement strategy that you can use?
Previous Next

25. The MICS platform (still under development)
25
Measuring Impact of Citizen Science
mics.tools
Is the engagement strategy shared with participants?
 Yes
 No
 I don’t know
Previous Next

26. The MICS platform (still under development)
26
Measuring Impact of Citizen Science
mics.tools
Progress

27. Example 1 of additional guidance
27
Does the project have a code of ethics?
 Yes
 No
 I don’t know
Want a template code of ethics you can use?
Previous Next
Measuring Impact of Citizen Science
mics.tools

28. Example 2 of additional guidance
28
Do participants gain new knowledge from
taking part in the project?
 Yes, and it has been measured.
 Yes, but it hasn’t been measured.
 No
 I don’t know
Interested in learning how to measure this?
Previous Next
Measuring Impact of Citizen Science
mics.tools

29. The MICS platform (still under development)
29
Measuring Impact of Citizen Science
mics.tools

30. The MICS platform (still under development)
30
Measuring Impact of Citizen Science
mics.tools

31. The MICS platform (still under development)
31
Measuring Impact of Citizen Science
mics.tools

32. The benefits
32
Measuring Impact of Citizen Science
mics.tools
• A space to think about impact
• Resources and templates
• Guidance on impact-data collection
• An impact-oriented project description
• A quantified assessment
• Access to state-of-the-art AI algorithms

33. Website: mics.tools
Twitter: @MICSproject
Facebook: MICS Project
LinkedIn: MICS Project
Contacts
• Luigi Ceccaroni
Coordinator
lceccaroni@earthwatch.org.uk
• Uta Wehn
u.wehn@un-ihe.org
• Claire Williams
Project manager
cwilliams@earthwatch.org.uk
Measuring Impact of Citizen Science
mics.tools 33
This project has received funding from the European Union’s Horizon
2020 research and innovation programme under grant agreement
No 824711.