Can you crowd-source an evaluation report? Here's what happened when I tried http://stevepowell.blot.im/focus-docs-worked

1. Focus docs:
Crowdsourcing your evaluation
Steve Powell, UNEP brown-bag, Nairobi
March 2015
steve@pogol.net

2. What’s the problem?
• Greedy ToRs
• … mean evaluation is the act of balancing
narratives
• Evaluation matrix: way to go around the
houses
• Recommendations out of thin air?

3. Evaluation matrix
Team Web
survey
Document
analysis
Youth Youth in
villages
…
QA: relevance
QA1: stakeholder
relevance
QA1x: stakeholder
relevance of
component x
x x
…
QM: what shall we
do about ….
x x
…
x x

4. Around the houses

5. Focus Docs: the steps
• Preparation
• Clarify Privacy
• Face to face
• Inititialise questions
• Invite
• Moderate
• Summarise
• Write

6. Evalcrowd discussionThe evaluator
Results: strategy 1
Stakeholders
Results: strategy 2
Conclusions: strategy 1
Conclusions: strategy 2
“These are the mandatory questions for the report…”
post questions
give answers
ask evaluator for new evidence
add evidence
add comments &”likes”
“.. But you can add your own”
posts some questions,
moderates,
challenges,
adds new evidence,
synthesises into final report
Using Focus Docs to crowdsource your evaluation

7. Some other advantages of the
Focus Doc procedure
• Break down answers by type of respondent e.g. by gender
• discussion between all respondents almost simultaneously
• Respondents quote and/or link to additional evidence
• allows respondents to flag up where evidence seems to be
missing, i.e. where the evaluator can help
• frees up some evaluator time to carry out some more in-
depth investigations where really needed
• allows the evaluator to focus on issues which are critical
and/or controversial for contributors and/or minority
opinions
• makes it easy to bring in subject experts
• allows social network analysis of the responses

8. Disadvantages of Focus Docs?
• Low familiarity with web tools
Security/Privacy
• Potential data loss – need for backups, etc ….
• Fear of “on the record” contributions

9. Evaluator role (1): Focus on addressing hidden topics
• What themes are we not talking about?
• … would we rather not talk about?
• some themes, some groups?
• Do some groups get left out? Why?
• Rewards / sanctions for saying …
• (Focus Docs disadvantage = advantage!)

10. Evaluator role
2. The spirit and the letter
3. Irreplaceable!
4. Still face-to-face, but less
5. Address bias
6. Moderating the process and interaction
7. Ensuring participation & equal contribution