This presentation was held at the FoME Symposium, Oct. 1-2, 2015 at DW Akademie Bonn. FoME stands for "Forum Medien und Entwicklung". It's the German Forum "Media and Development", a network of institutions and individuals active in the field of media development cooperation. Find the Symposium’s documentation here: http://fome.info/2015

1. Zürcher FachhochschuleZürcher Fachhochschule 1
What to do without baselines?
Not often used methods in media evaluation
11th FoMe Symposium
Focus on Impact: Advanced Methods and Concepts in Media Development
Bonn, 1 October, 2015
Christoph Spurk
IAM Institute of Applied Media Studies

2. Zürcher Fachhochschule 2
What is Evaluation?
• Systematic assessment of policies, programs and
projects regarding
- Concept
- Implementation
- Results and effects (effectiveness)
• Objectives
- Formative Learning
- Summative Judgement

3. Zürcher Fachhochschule
Five Core Approaches in Evaluation*
• Needs Assessment (Relevance)
• Assessment of Program Theory or Design
• Process Evaluation
• Impact Evaluation (Results, Effects, Impact)
• Efficiency Assessment and Cost-effectiveness
*(Rossi et al. 2004, p54 ff.)
3

4. Zürcher Fachhochschule
Basic Requirements for Evaluation
• Clear Objectives of (Media) Intervention
– Based on Analysis of Context and comprehensive planning process
– Surprisingly often not done in current competitive environment
• Baseline study before Intervention
– Gathers data objectives of Intervention
– Provides insights (Needs!)
– Enables to measure later against this base in a before-after-design =
extent of change
• Results chains with indicators
4

5. Zürcher Fachhochschule
Basic Challenges in Evaluation
• Practice:
– Often no baseline data
– Even less data on «control groups» / «counterfactual»
– And still impact evaluation required!
• What to do and what not to do?
• Do we need «new» methods or rather not?
5

6. Zürcher Fachhochschule
What is often done – but should not
• Ask beneficiaries what the change is / was
– Most Significant Change Technique as Method
- «What was the most significant change through the radio
programme?»
- Strong influence from evaluator’s side – biased
- Social desirability in answers even more enhanced.
– Posing the most difficult evaluation questions directly
- «What do you think is the contribution of Radio XYZ to democratic
participation?»
- «How much has your farming income increased due to the radio
programme in the last hree years?»
• Unbalanced scales in answer options
- Strongly agree / agree / somehow agree / don’t agree 6

7. Zürcher Fachhochschule
What can be done? – Is there a way out?
1. Prepare a baseline study with control groups for next phase of
project
2. Use the «theory based» evaluation method as an alternative
design, but still second best.
3. Story checking as an additional data collection method
7

8. Zürcher Fachhochschule
I. «Theory based» evaluation
• Assessment of the «plausability of impacts»
• Requirements
– Knowledge from evidence-based research about what works
and what not
– Assessment whether program elements in intervention reflect
this knowledge or not.
– If program is in line with evidence, it has a greater plausibility
to achieve impacts.
8

9. Zürcher Fachhochschule
Example: Civil Peace Service (CPS)
• No baseline data at the beginning of project
• Evidence
• existed from previous research what civil society functions did work
in particular phases of conflict
• based on comparative research in various conflict settings
• Assessment:
• Most CPS program interventions were working on long-term civil
society functions (socialization, social cohesion) that were hardly
relevant in ongoing violent conflict, and
• not on short-term functions that were much in need (protection,
monitoring).
• Thus, the plausibility of impact was assessed as low.
9

10. Zürcher Fachhochschule
Potential for media programs - Example
• Farmer information project
– wants to know about the changes in farmer practice (organic farming!)
– No baseline data on farmer practice before project start
• Application of theory-based evaluation
– Evidence exists from previous research what type of information small-
scale farmers like and need for adoption:
- Technical plus market and economic information
- No top down single solutions but variety of options
– Potential to assess content of programmes regarding those requirements
– Results of content analysis would tell to what extent the information is
plausible to be implemented.
– Insights could provide the project with valuable insights for future
improvement
10

11. Zürcher Fachhochschule 11
II. Checking stories as exploration
• SjCOOP: Improving Science Journalism in Africa and the Arab
World
– Better quality in reporting
– Publishing science stories that achieve change (“impact stories”)
- enlightening public debate about science (health, agriculture)
- Contributing to evidence-based policy making or immediate change
• Have stories really achieved an “impact”?
– Selection of potential stories
– Strong verification process of claimed impact by evaluators
- Timeline
- Context/ History
- Media Echo
- Verification of reactions with sources and experts/stakeholders

12. Zürcher Fachhochschule
Impact : Science story level
12
Government
Follow-up
Story 2 (radio)
Follow-up
Story 3 (TV)
Follow-up
Story 1
Initial
science story
Media Echo
Public Agenda
Public reactions
Local administration

13. Zürcher Fachhochschule
CD4 Machines in Western Uganda
Problem:
• CD4 Machines in Fort Portal, did not work
• Patients had to walk 1,5 hour
Story:
• Hope Mafaranga publishes articles in New Vision 29/30th Nov.
2011
Media Echo:
• Radio stations and Ugandan News Agency pick story 29 Nov. to
2 December 2011.
Impact:
• Brand new CD4 Machines two weeks later

14. Zürcher Fachhochschule
Condoms in Northern Kenya
Problem:
• Acute shortage of free condoms in Isiolo region, Kenya
• Shortage leads to washing, re-using condoms
or using plastic bags.
Story by Violet Otindo:
- Television K24 airs14-minute special on 27th
February 2011
Media Echo:
-BBC Radio Swahili – 15th
March
-Citizen and Nation TV -22nd
March
-Further coverage in Kenya and in Danish and Norwegian newspapers
Impact:
- Health Minister admits shortage on 22nd March
- Emergency delivery of 100 000 condoms by Red Cross Kenya, 25th March
-Kenya government ordered larger supplies with UNFPA
- Parliament debated the issue on 29th March

15. Zürcher Fachhochschule
Checking stories yields more insights
• Typology of stories
– Putting an issue on public agenda
– Short-term impact: immediate change
– Long-term impact : changing policies
– Uptake of science
• Enabling and hindering factors
– Media Echo – sometimes, but not always
– NGO – media – civil society relationship: NGOs take issue up
– Context: Government responsiveness
– Framing of story: shaping a new frame
– Role models of African journalists – change agents
15

16. Zürcher Fachhochschule
Lessons
• Lessons to improve the impact of stories:
– Achieve stronger media echo
– Cooperation amongst media
– Target NGO – media relationships / partnerships
– Take government’s responsiveness into account
• Evidence
– What makes stories more plausible to have an impact
16