Presentation by Jonathan Potter, OECD LEED Senior Policy Analyst, and Stuart Thompson, OECD LEED Policy Analys, tat the seminar organised by the OECD LEED Trento Centre for the Officers of the Autonomous Province of Trento on 13 November 2015. https://www.trento.oecd.org

1. EVALUATION OF SME
AND
ENTREPRENEURSHIP
PROGRAMMES
Jonathan Potter and Stuart Thompson
Trento
13th November 2015

2. Structure of the presentation
1. Role and importance of evaluation
2. Approaches to evaluation – measures and
techniques
3. Evaluation examples
4. Planning and organising evaluation
5. Some conclusions

3. Part 1
Role and importance of evaluation

4. OECD Istanbul Ministerial Declaration
Recognised
“The need to develop a strong evaluation
culture in ministries and agencies
responsible for SME policies and
programmes”
“Evaluation provides a means of ensuring
that SME programmes remain cost-effective
and adapt to changing conditions in a
dynamic world”

5. What do we mean by evaluation?
Papaconstantinou and Polt (1997):
“Evaluation refers to a process that seeks to
determine as systematically and objectively as
possible the relevance, efficiency and effectiveness
of an activity in terms of its objectives”
Purpose is to provide decision makers with the
best information to answer crucial questions, e.g.
• Does my programme address the key issues?
• What is working and not working?
• What can I do better next time?

6. Why evaluate?
• Learning from experience
• To establish the impact of policies and
programmes against their objectives
• To make informed decisions about the allocation
of funds across a portfolio
• To show the taxpayer whether the programme is
a cost-effective use of public funds
• To stimulate informed debate
• To achieve continued improvements in the
design and administration of programmes

7. Some evaluation questions
• Is the activity still relevant?
• Is it achieving its objectives?
• Is it cost effective and proportional to what it
seeks to achieve?
• Should the activity be continued or terminated?
• If continued, how can it be improved?
• Can it be used in different contexts?
• What issues will occur when scaling up a pilot
project?

8. Part 2
Approaches to evaluation – measures
and techniques

9. Inputs
Resources required
for the programme
(finance, people etc)
Often defined in
advance, needs to
be sufficient
↓
Activities
Actual actions taken
(training courses,
advice services etc)
Determined from
technical
considerations and
experience
↓
Outputs
Immediate measure
of activities
(number of trainees,
people advised)
Determined from
previous experience
and evaluations
↓
Outcomes
Change in behaviour
of target (number of
start-ups etc)
Requires detailed
analysis and
research
↓
Impacts
Overall effect
(noting what would
have happened
anyway etc)
Often only
calculable well after
the programme is
complete
Measures at different levels

10. Measure Definition Typical Questions
Relevance The extent to which the activity is suited to
the priorities and policies of the target
group, recipient and government.
To what extent are the objectives of the programme
still valid?
Are the activities and outputs of the programme
consistent with the overall goal and the attainment of
its objectives?
Effectiveness The extent to which the intervention's
objectives were achieved, or are expected to
be achieved, taking into account their
relative importance.
To what extent were the objectives achieved / are
likely to be achieved?
What were the major factors influencing the
achievement or non-achievement of the objectives?
Efficiency The outputs in relation to the inputs. It
signifies that the intervention uses the least
costly resources possible in order to
achieve the desired results.
Were activities cost-efficient?
Were objectives achieved on time?
Was the programme or project implemented in the
most efficient way compared to alternatives?
Impact The positive and negative changes
produced by an intervention, directly or
indirectly, intended or unintended on the
local social, economic, environmental and
other development indicators.
What has happened as a result of the programme or
project?
What real difference has the activity made to the
beneficiaries?
How many people have been affected?
Sustainability Sustainability is concerned with measuring
whether the benefits of an activity are likely
to continue after funding has been
withdrawn. Projects need to be
environmentally as well as financially
sustainable.
To what extent did the benefits of a programme or
project continue after funding ceased?
What were the major factors which influenced the
achievement or non-achievement of sustainability of
the programme or project?
Typical evaluation measures

11. Monitoring
Step I Take up of program
Step II Recipients’ opinions
Step III Recipients’ views of the differences made by the
program
Evaluation
Step IV Comparison of the performance of ‘assisted’
with ‘typical’ ventures
Step V Comparison with ‘matched’ ventures
Step VI Taking account of selection bias
Approach - six steps

12. The ideal approach –
Randomised control group

13. Example - GATE
• Open to all, but targeted at the
unemployed
• Applicants randomly given assistance or
not
• No barriers to using other sources of
advice
• Result showed robust evidence of an
impact
Impaq International (2009), Growing America Through Entrepreneurship:
Final Evaluation of Project GATE

14. Regression model
N
N
N N
N
N N
N
N
N
N
N
N
N
N N
N
NN
N
B
B
B
B B
B
BB
B
B
B
B
B
B
B
B
B
B
Outputvariable(resultofavarietyof
factorsaffectingfirmcharacteristics)
Firm characteristics (control variables and
value of policy support)
Beneficiaries
Non-beneficiaries

15. Matching (propensity scoring)

16. Differences in differences

17. Regression discontinuity

18. Part 3
Evaluation examples

19. Example – Prince’s Trust Youth
Business Initiative
Greene, Francis J (2005), Evaluating Youth Entrepreneurship: the Case
of the Prince’s Trust
• The Prince’s Trust is one of the longest running
programmes to support youth entrepreneurship in the
United Kingdom. It uses a mix of soft (mentoring) and
hard support (loans and grants) and has maintained the
same approach over a long period.
• The programme’s specific aims are:
• Supporting individuals who would be unlikely to
start up an enterprise
• Deepening the human capital of participants
• Improving survival rates

20. Prince’s Trust - methodology
• Survey of recipients
• Survey of non-recipients
• Survey of non-survivors
• Matched survey with sample selection
• 3600 respondents
• Survival rates assessed over 3 years

21. Prince’s Trust - results
• Only mentoring appears to have a positive
effect
• This is at variance with simpler
evaluations which only looked at the
views of participants, who were very
positive
• No evidence that entrepreneurship
support assisted employability

22. Example – Start-up grants to
unemployed immigrants in Germany
• Germany has had long term policies of providing
support and “welfare bridges” to the unemployed i.e.
continued subsidies even when they are self-employed.
Exact levels of benefits have varied over time.
• Immigrants have different characteristics from the
general population and may respond to policies
differently.
• There have been a number of positive evaluations of the
scheme as a whole, but these do not mean that it works
in the same way for all groups of the unemployed, for
example immigrants.
Robinson, Eric (2011), Start-Up Grants to Unemployed Immigrants in
Germany: A Means to Reach Employment Parity?

23. Methodology
• Use of long term socio-economic
database, the German Socio Economic
Panel – including more than 140,000
individuals tracked between 1983 and
1993.
• Regression techniques, controlling for
sex, age, marital status, education, work
experience, sector, type of self-
employment

24. Results
• Unlike natives, foreigners are more likely
to lapse back into unemployment when
start-up benefits are high. This difference
is statistically significant.
• Consideration of skill levels suggests that
this is because high benefits give an
incentive for ill-prepared individuals to
start up business.

25. Part 4
Planning and managing evaluations

26. Evaluation and the policy cycle
Programming
What is the objective?
Identification
What are the barriers?
Formulation
How can we address it?
Resourcing
Can we mobilise resources?
Implementation
Is it on track?
Ex Post Evaluation
Did it produce the results?
Ex Ante
EvaluationOngoing/Interim
Evaluation
Meta evaluations
Benchmarking
Terminal
Evaluation

27. • A timetable for evaluations
• Responsibility for commissioning or
performing evaluations
• Quality assurance
• Mobilisation of resources
Evaluation Framework - management

28. • Identifying stakeholder needs
• Deciding what to evaluate
• Key Performance Indicators
• Timely and accurate information
• Qualitative information
• Information regarding assumptions
• Baselines
• Counterfactual
Evaluation framework - information

29. Getting the information?Cost
Complexity
Administrative
reports
Management
reports
Available
statistics
Adapted available
statistics
Interviews with
beneficiaries
Specialised
surveys

30. Dealing with the results
• Is there logic in the report?
• Is it peer-reviewed? Quality assured?
• Have things changed?
• What needs to be public?
• What is politically necessary?

31. Part 5
Conclusions

32. Conclusions
• It is important to evaluate consistently and robustly
across programmes
• Effective and efficient evaluation requires planning
from the beginning of implementation
• Evaluation often involves deciding on proportionality –
the amount of effort involved in gathering data against
the usefulness of the result
• Managing authorities need to develop capacity in
commissioning evaluations rather than in actually doing
them
• Full impact evaluations can be difficult

33. Questions