This document discusses evaluation of business start-up policies. [1] It outlines principles of evaluation including relevance, effectiveness, efficiency and sustainability. [2] It presents approaches to evaluation such as using a logical framework, indicators, and both qualitative and quantitative methods. [3] Finally, it emphasizes that evaluation should be better integrated into the entire policy process from design to implementation.

1. OECD-European Commission
Capacity-building seminar
‘Financing Business Start-up by Disadvantaged and
Under-Represented Groups’, Trento, 27-29 June 2012
Evaluation of Business Start-up Policies
Jonathan Potter
Senior Economist
OECD Centre for SMEs, Entrepreneurship & Local Development (CFE)
jonathan.potter@oecd.org
www.oecd.org/cfe

2. Presentation structure
1. Principles of evaluation
2. Evaluation approaches
– logical framework
– indicators
– qualitative and quantitative evaluation
3. Conclusions

3. 1.1 What is evaluation?
Papaconstantinou and Polt (1997):
“Evaluation refers to a process that seeks to
determine as systematically and objectively as
possible the relevance, efficiency and effectiveness
of an activity in terms of its objectives”
Purpose is to provide decision makers with the
best information to answer crucial questions . . .

4. 1.2 Evaluation questions
• Is the activity relevant?
• Is it achieving its objectives?
• Is it cost effective and proportional to what it
seeks to achieve?
• Should the activity be continued or terminated?
• If continued, how can it be improved?

5. 1.3 Assessment criteria
Criterion Explanation
Appropriateness (relevance) Extent to which the programme
objectives are relevant to the needs of
the economy, and address a market
failure or social inequity
Effectiveness Extent to which the activity is
achieving the programme objectives,
net of what would have happened in
the absence of the initiative
Efficiency Extent to which costs are minimised
in achieving the programme’s
objectives
Sustainability To what extent can benefits be
expected to last after the intervention
is completed
Process management Extent to which the programme is
well managed

6. 1.4 The counterfactual
Outputs, e.g. Assistance
number of option
businesses
started by Do nothing
the target option
group Additionality
Non-
additionality
Baseline Project
Time
What would happen without the programme
Our interest is in the difference between the assisted output and the
non-assisted output

7. 1.5 Evaluation in the policy cycle
Definition of
rationale for
policy
Appraisal of
Feedback policy
options
Evaluation of
Implementation
results
of programmes
(benefits –
and projects
costs)
Monitoring
of progress

8. 2.1 The logical framework – concept
• Analytical planning tool to clarify causal
relations in a policy intervention
• Operational chain of activities, immediate
results and subsequent impacts
• Can be used at programme level and at project
level
• Can be used for strategy design and for
monitoring and evaluation of projects

9. 2.2 Logic framework – microfinance and business support
project
Resources
Activities Outputs Outcomes Impacts
/inputs
E.g. public E.g. numbers E.g. number of E.g. reduction E.g. reduction
investment in of loans new firms in number of in
guarantees of provided; created by participants unemployment
non- hours of unemployed who are in the target
performing mentoring people; unemployed; group;
loans; fund provided; new increased increase in increased
administration contacts with skills; number of entrepreneur-
cost; public banks and improved starts; increase ship rate
cost of business business in survival of
business support proposals; new starts;
support agencies relationships increased
with banks lending to
target group

10. 2.3 EQUAL Business Creation Theme Logic Model

11. 2.4 Indicators
Type Examples
Inputs Funds allocated
Total budget
Funds allocated as % of budget
Activities Number of people contacted
Number of people having received services
Number of proposals for the creation of new businesses
Number of advisors
% of beneficiaries belonging to target groups
Outputs/outcomes Attitudes to entrepreneurship
Business start rate
Survival rate of enterprises
Reduction in unemployment rate
Other Satisfaction of beneficiaries with the programme
Satisfaction of managers with the programme

12. 2.5 Methods for evaluation
Qualitative approach
• Based on stakeholder and beneficiary interviews
• Deep questioning on processes and changes in
decisions and behaviour
• Strength is that it explores processes and engages
participants in policy learning
Quantitative approach
• Compares performance of assisted and matched
firms and individuals
• Accounts for selection bias
• Strength is that is gives robust impact estimates
12

13. 3. Conclusions
• Evaluation is critical to policy success but is not
yet properly integrated in policy process
• At this stage of the EU policy process the
emphasis is on ex ante evaluation, implying:
– development of logical frameworks
– Identification of indicators to measure, targets to
reach and creation of a monitoring programme
• Once activities have been implemented, will
need to consider the nature of evaluations to
commission; may need some data collection now