This document discusses methods for evaluating the impact of private sector support programs. It begins by outlining different approaches to impact evaluation, including testing causal effects and understanding contributing factors. The document then describes mixing quantitative and qualitative methods to evaluate both attribution of outcomes to interventions and contributions of interventions to outcomes. Specifically, it discusses using a cohort study, contribution scoring, and comparative case studies to evaluate the Dutch PRIME program. Results show some effects on business practices and outcomes like profits and exports. The document concludes by synthesizing the mixed methods approach into an evidence-based contribution narrative.

1. Mixing methods to assess the
impact of private sector support:
experiences from the Dutch
PRIME-programme
Giel Ton
Thursday 19th October
13.00 – 14.30
IDS Convening Space

2. Outline
• Evaluation of impacts:
o evaluation questions
o intervention logic
• Mixing methods:
o span of direct influence (attribution/contribution)
o scan validity threats to likely conclusions from core method
• Application in practice and results:
o Cohort design (yearly online survey)
o Contribution score (treatment intensity)
o Comparative case studies
• Overall synthesis: an evidence-based contribution narrative

3. PART ONE – EVALUATION OF
IMPACTS

4. RANDOMISTAS SYNERGY REALISTAS
Did the
intervention
cause the
change?
Testing and
modifying
theory
What combination
of factors, next to
the intervention,
caused the
change?
Generating and
modifying
theory
DESIGN PRINCIPLES
ANTICIPATE
GENERALISATION
Approaches to impact evaluation

5. What works?
Under what
conditions?
For whom?
What works, for
whom and under
what conditions?

6. RANDOMISTAS SYNERGY REALISTAS
DESIGN PRINCIPLES
Did the
intervention
cause the
change?
Testing and
modifying
theory
What
combination of
factors caused
the change?
Generating and
modifying
theory
ANTICIPATE
GENERALISATION
INTERVENTION
LOGIC
THEORY-BASED
EVALUATION
Approaches to impact evaluation

7. THE HOLY GRAIL:
THEORY-BASED EVALUATION

8. Define the intervention logic

9. PSD interventions
• CBI (RVO - Dutch Enterprise Agency) - support imports form
development countries to the European market through business
training, market information, international fairs, sector development
studies, etc.
• PUM Senior experts - support SMEs in business management,
investment planning and business links through experienced former
managers (peer-to-peer) through two week on-site visits.

10. Theory of change of CBI

11. PART TWO – MIXING METHODS

12. RANDOMISTAS SYNERGY REALISTAS
DESIGN PRINCIPLES
Regressional
analytic
methods
Configurational
comparative
methods
VALIDITY
THREATS
Attribution of
outcomes to
intervention
Contribution of
intervention to
outcomes
SPAN OF
DIRECT
INFLUENCE
COMBINE
CAUSAL LOGICS
Quantitative
research
Qualitative
research
Methods in impact evaluation

13. RANDOMISTAS SYNERGY REALISTAS
DESIGN PRINCIPLES
Regressional
analytic
methods
Configurational
comparative
methods
VALIDITY
THREATS
Attribution of
outcomes to
intervention
Contribution of
intervention to
outcomes
SPAN OF
DIRECT
INFLUENCE
COMBINE
CAUSAL LOGICS
Quantitative
research
Qualitative
research
Methods in impact evaluation

14. Policy makers’ interest
Policy interest
in effectiveness

15. Implementing agency’s influence
Effect size
Attribution
Contribution

16. Research method limitations
Net-effects
impact research
(attribution)

17. Research methods limitations
Net-effects impact
research
(attribution)
Monitoring
(contribution
approach)

18. Dutch government guidelines (DGIS 2017)

19. RANDOMISTAS SYNERGY REALISTAS
DESIGN PRINCIPLES
Regressional
analytic
methods
Configurational
comparative
methods
VALIDITY
THREATS
Attribution of
outcomes to
intervention
Contribution of
intervention to
outcomes
SPAN OF
DIRECT
INFLUENCE
COMBINE
CAUSAL LOGICS
Quantitative
research
Qualitative
research
Methods in impact evaluation

20. Four types of validity threats to conclusions
(Shadish, W. R., T. D. Cook and D. T. Campbell (2002). Experimental and Quasi-Experimental
Designs for Generalized Causal Inference, Houghton Mifflin Co. Boston, MA.)
 Statistical conclusion validity: how are inferences drawn from the
data-set?
● comply with proven methods and correct interpretation of
statistical analysis.
 Internal validity: how is causality attributed?
● What is the mechanism how interventions contribute to the
observed change.
 Construct validity: how are generalizations made from the
categories used in the evaluation to broader units of representation?
● Careful and precise definitions and concepts.
 External validity: to what extent are the findings generalizable?
● Define the ‘generalization domain’.

21. EXPLORING VALIDITY THREATS TO THE CORE METHOD
TYPE OF
VALIDITY
THREAT
MAIN THREAT
SUGGESTED ADDITIONAL
DESIGN ELEMENTS
RESULT/OBSERVATION
STATISTICAL
CONCLUSION
INTERNAL
CONSTRUCT
EXTERNAL

22. Core method: cohort study / Rolling baseline

23. TYPE OF
VALIDITY
THREAT
MAIN THREAT INCORPORATED ADDITIONAL DESIGN ELEMENTS
STATISTICAL
CONCLUSION
 Differences between
cohorts averages have to
controlled for but are not
known in advance and
not always observable.
 Econometric methods will check for selection
bias between the cohorts.
INTERNAL
 Correlation is not
causation
 The self-selection of
businesses into the
programme makes that
conclusions only apply to
firms similar to the
supported ones.
 A qualitative comparative case study approach
(N=6) to provide information on the processes
and factors that explain the effectiveness of the
support.
 The characteristics of the treated firms in these
case studies are compared with the averages
in the sector
CONSTRUCT
 The treatment is very
diverse
 A factor/cluster analysis to make a typology of
similar interventions (sector, region, type of
support provided)
EXTERNAL
 Country and sector
context matters (e.g.
economic crisis/growth
periods, political
situation, policy
developments)
 Explorative multivariate analysis of the final
data-set to identify heterogeneity of effects.
 Careful conclusions (restricted generalisation
domain of the causal inferences)

24. PART THREE – APPLICATION IN
PRACTICE AND RESULTS

25. Cohort study / Rolling baseline

26. Cohort
2013
Cohort
2014
Cohort
2015
Difference
2015-2013
Difference
2015-2014
Performance
 Sales, logarithms 12.62 12.46 12.35 -0.24** -0.08
 Employment, logarithms 3.44 3.37 3.38 -0.06 0.01
Type of firm
 Cooperative 0.01 0.01 0.02 0.01** 0.01***
 Individual 0.31 0.32 0.30 0.00 -0.02
 Joint Stock 0.09 0.11 0.06 -0.03* -0.05***
 Limited Liability 0.54 0.51 0.56 0.01 0.03
 Other 0.05 0.04 0.06 0.01 0.02
Region
 Africa 0.37 0.38 0.38 0.01 -0.01
 Asia 0.29 0.27 0.31 0.01 0.04
 Europe 0.18 0.17 0.14 -0.05** -0.04
 Latin America 0.16 0.18 0.17 0.03 0.01
Poverty status
 Least Developed 0.27 0.32 0.34 0.06** 0.01
 Lower Middle Income 0.47 0.42 0.43 -0.03 0.02
 Other Low Income 0.02 0.02 0.02 0.00 0.00
 Upper Middle Income 0.24 0.24 0.20 -0.03 -0.03
Sector
 Agriculture 0.17 0.17 0.17 0.00 0.00
 Manufacturing 0.50 0.46 0.46 -0.05* -0.01
 Service 0.32 0.37 0.38 0.05* -0.01
Other
 Female owner 0.09 0.20 0.03 -0.06*** -0.17***
 Received one mission 0.24 0.17 0.12 -0.12*** -0.05**
Inter-cohort comparison of characteristics
Slightly
smaller
firms
Similar
profile
More
intensive
support
Less
female-
headed
firms
Less in
Eastern
Europe

27. PRIME online survey (Qualtrics)

28. Effects on intermediate outcomes

29. Question 1 - level of practices:
Construct validity check
Objective business practices Subjective business practices
Keeping financial records via other Financial management
Verifying financial statements through
control outside the company or external
control by a certified auditor
Financial management
Written marketing plan Marketing techniques to increase sales of your product
or service
Website Marketing techniques to increase sales of your product
or service
Promotion material Marketing techniques to increase sales of your product
or service
All employees have written labor contracts Ways to retain, motivate and train employees
Systems and policies in place to monitor and
ensure worker safety.
Ways to retain, motivate and train employees
System to learn about the clients' opinions
on products and services.
Leading, planning and organizing the business
Documented quality assurance system of
the business processes.
Quality requirements of (inter)national buyers
System in place to monitor effects on the
environment.
Effects of the business on the environment

30. 0
.2.4.6.8
1
1 2 3 4
Pract.-Financial management
Fin. records via computer
0
.2.4.6.8
1
1 2 3 4
Pract.-Financial management
External audit
0
.2.4.6.8
1
1 2 3 4
Pract.-Marketing techniques
Marketing plan
0
.2.4.6.8
1
1 2 3 4
Pract.-Marketing techniques
Website
0
.2.4.6.8
1
1 2 3 4
Pract.-Marketing techniques
Promotion
0
.2.4.6.8
1
1 2 3 4
Pract.-Employees
Labor contract
0
.2.4.6.8
1
1 2 3 4
Pract.-Employees
Safety
0
.2.4.6.8
1
1 2 3 4
Pract.-leadership
Clients opinion
0
.2.4.6.8
1
1 2 3 4
Pract.-Quality req.
Quality assurance system
0
.2.4.6.8
1
1 2 3 4
Pract.-Environment
Monitor environment
Fitted line for subjective and ojective practice answers are in blue
Area of symbol proportional to number of observations
Construct validity test
Horizontal and vertical axis are self assessed practices and proxy indicators for business
practices levels respectively

31. Effects on intermediate outcomes

32. Question 2 & 3: Change & Influence
PRIME Contribution score
• An indicator for CBI contribution: How did your company’s knowledge change
over the past 12 months? + How did CBI influence the change?
How did your company’s
knowledge change over
the past 12 months?
Did CBI influence this
change?
CBI contribution
score (0-8)
CBI Contribution
index (0-100%)
Strong decrease No effect 0 0
Decrease No effect 0 0
No change No effect 0 0
Increase No effect 0 0
Strong increase No effect 0 0
Increase Very little 1 13
Strong increase Very little 2 25
Increase Some 3 38
Strong increase Some 4 50
Increase Quite a bit 5 63
Strong increase Quite a bit 6 75
Increase A lot 7 88
Strong increase A lot 8 100
No contribution
Full contribution
50 contribution

34. Slide: 34

35. (1) (2) (3) (5) (6)
VARIABLES Sales
Permanent
Employment
Non-Permanent
Employment Change in the profits Export
Lagged average
contribution score
-PUM/100 2.066 0.327 1.165 1.639** 0.817
(3.988) (0.509) (0.882) (0.779) (1.703)
Constant 5.730 0.470 -0.247 1.955 0.00481
(4.963) (0.600) (1.034) (1.309) (2.892)
Observations 38 149 106 128 43
R-squared 0.933 0.446 0.490 0.468 0.700
Controls Yes Yes Yes Yes Yes
Country FE Yes Yes Yes Yes Yes
Activity FE Yes Yes Yes Yes Yes
Regressions with Contribution Score as intensity
of treatment

36. Heterogeneity analysis for CBI - export promotion
Effect on contribution score
(Estimates for) the effect of firm size, region, income group, access to finance and
sector on CBI contribution score for firm practices
In practices
transfer CBI
is more
successful in
SSA then in
Asia and Latin
American
regions
Less in the
upper middle
countries
In agriculture
sector, the
success is
lower than in
services.
small
medium
large
Asia
EUR
L. America
MENA
SSA
LDC
low
lower middle
upper middle
easy
neutral
difficult
v. difficult
Agriculture
Consumer
Industry
Service
Size
Region
Income group
Access to Finance
Sector
-4 -2 0 2

37. Heterogeneity analysis for PUM - management counceling
Effect on contribution score
(Estimates for) the effect of firm size, region, income group, access to finance and
sector on PUM contribution score for firm practices PUM is more
successful in
in LDC and
Low income
countries,
In the
Services and
Trade sector;
in the
agriculture
the success is
lower.
and higher in
firms with
female
owners.
micro
small
medium
large
Asia
EUR
L. America
Africa
LDC
low
lower middle
upper middle
easy
neutral
difficult
v. difficult
Agriculture
Industry_Engineering
Service and Trade
Female owner
Size
Region
Income group
Access to Finance
Sector
Owner Gender
-3 -2 -1 0 1 2

38. PART FOUR – COMPARATIVE CASE
STUDIES

39. Comparative case studies
Two missions (2014/15 and 2016/17) of 10 days
each to interview SMEs, support organisations,
sector policy, etc.
• Bolivia - Tourism sector
• Peru - High value agriculture
• Uganda - Coffee
• Bangladesh - IT-sector
• Myanmar - textile
• Indonesia - Fishery & aquaculture

40. Additionality
Barriers and enablers
Implementation efficiency
Indirect effects
INSTITUTIONAL
ENVIRONMENT
(e.g. sector policies and
platforms)
SME CHARACTERISTICS
(e.g. size, gender, firm age)
MARKET AND COMPETITION
(e.g. interfirm cooperation,
sector features, market
outlook)
ADDITIONAL KNOWLEDGE
TO THE SME
(e.g. additional to other
information sources)
ADDITIONAL TO LOCAL
BUSINESS TRAINING
PROVIDERS
(e.g. willingness to pay)
COMPLEMENTARITY
TO OTHER SUPPORT
(e.g. other SME support
programmes, access to
finance)
BROKERING / RECRUITMENT
(e.g. social networks and
incentives for local
representatives)
SELECTION
(e.g. application procedure
and intake requirements)
FOLLOW-UP
(e.g. backstopping of
missions, complementary
actions)
SPILL-OVER
(e.g. employment of
suppliers, multiplier effects)
INNOVATION
(e.g. product development,
branding, cluster effects)
NEGATIVE EXTERNALITIES
(e.g. environmental,
cultural, political)
SUSTAINABLE ECONOMIC DEVELOPMENT
INCREASED
TURNOVER
IMPROVED PRACTICES IN SMALL AND MEDIUM ENTERPRISES
IMPROVED
EMPLOYMENT
INCREASED
PROFIT
INCREASED
SUSTAINABILITY
INCREASED
EXPORTS
INCREASED KNOWLEDGE
IN BUSINESS SUPPORT
ORGANISATIONS
INCREASED KNOWLEDGE IN
SMALL AND MEDIUM
ENTERPRISES
KNOWLEDGE TRANSFER AND COACHING
BUSINESS LINK
PROMOTION

41. Case study analysis plan
Table XX. Case study design and research questions
Research
question
Research objects Within-case analysis Cross-case analysis
What works? Support modalities Identify the different
support modalities used in
the sector
Compare similar support
modalities across cases
For whom? Beneficiaries Differentiate participation
and effects between larger
and smaller, and younger
and older firms
Differentiate participation
and effects between more
and less developed
countries
Under what
conditions?
Sector & country Firms operate under
similar economic and
political conditions
Cases cover various
sectors, with different
economic and political
conditions
Enablers and barriers Identify plausible enablers
and barriers of
effectiveness
Compare enablers and
barriers across cases
Policy recommendations:
How can CBI and PUM improve the effectiveness of their support?

42. PART FIVE – OVERALL SYNTHESIS

43. Overall synthesis = assembling the evidence
behind the contribution narrative

44. Overall synthesis = assembling the evidence
behind the contribution narrative
• Activities improve knowledge
& practices (online survey)
• The promoted ‘ good practices’
are considered to contribute to
improved performance of the
sector (case studies)
• Better SME performance is
good for sustainable economic
development (literature
review, secondary data)

45. Takeaways
• Try the three-step process for mixing methods
o refine evaluation questions based on an intervention logic
o anticipate validity threats to the expected type of conclusions
o maximise the scope for comparative research and benchmarking
• Be aware of the ‘ span of direct influence’ and the limits to attribution
• Ask evaluative self-assessment questions along observable indicators
• Collate evidence in a contribution narrative to report on development
impact (accountability)

46. Thanks!
www.primepartnership.nl
Former PRIME Research Coordinator
Wageningen Economic Research
Giel Ton
g.ton@ids.ac.uk
Institute of Development Studies
Centre for Development Impact