Presentation by Professor Stephen Roper to International Conference - Taiwan. Experimenting with industrial policy: The UK’s experience of industrial policy making using randomised control trials (RCTs)

1. Experimenting with industrial policy: The UK’s
experience of industrial policy making using
randomised control trials (RCTs)
Stephen Roper
Enterprise Research Centre and Warwick Business School,
Coventry, CV4 7AL, UK
Stephen.Roper@wbs.ac.uk

2. UK policy context
• 'Dear chief secretary, I'm afraid to tell you there's no money left’. Note left
for David Laws new Chief Secretary to the UK Treasury elected in 2010.
Started period of ‘austerity’ government
• In policy terms:
– Real terms cuts in public spending
– Removal of some organisations – notably Regional Development Agencies
– And, new emphasis on value for money in policy making
• This latter point has stimulated:
– Focus on ‘impact’ in public support for science and social science
– Creation of ‘What Works’ Centres….
– More interest in rigorous policy evaluation and RCTs
• (Also potentially important has been the increasing prominence of
behavioural economics and the role of the ‘Behavioural Insights team)

3. More broadly …
• ‘Experimental methods’ of policy evaluation are well-established in education and
social policy and development economics.
• Typically such evaluations involve individual human subjects facing some common
socio-economic problem, and the random allocation of subjects to a treatment and
control group. Differences in outcomes between the treatment and control groups are
then attributed to the effect of the policy intervention
• In terms of industrial policy, however, experimental policy evaluation approaches
remain marginal, with non-experimental, ex post policy evaluations remaining the
norm.
• Potter and Storey (2007), for example, provide an extensive review of OECD best
practice without any mention of either the application or potential for experimental
methods.
• Similarly, experimental approaches are largely ignored in UK government guidance (BIS,
2009) and evaluating innovation policy (Laredo, 1997)

4. Aims of paper
• Explore the rationale for RCTs in industrial policy design
and implementation
• Consider the recent UK history and the four RCTs
implemented to date
• Identify some of the implementation and practical
issues involved
• Consider questions of interpretation and the nature of
‘evidence’ for policy evaluation

5. On experiments …
• Experimental methods – based on randomised allocation – offer a way of
avoiding issues of selection rather than correcting ex post (with its attendant
difficulties)
• But experiments themselves are subject to series of potential implementation
issues which can reduce their ‘validity’ (or ability to provide true
representation of a treatment effect
• Threats to internal validity – i.e. treatment v control
– Small sample issues (Bruhn and McKenzie, 2009)
– Substitution bias (Heckman and Smith , 1995)
– Signalling bias (e.g. Meuleman and Maeseneire, 2012)
• Threats to external validity – i.e. in-scheme v not in-scheme
– Macro biases (information flows, social interaction, norm formation) (Garfinkel et
al., 1990)
– Participation bias (differing characteristics) (Burtless, 1995)
– Randomisation bias (Heckman and Smith, 1995).

6. … and the importance of replication
• Issues related to the validity (either external or internal) suggest the potential
value of triangulation or meta-analytical reviews across a number of RCTs
• In medicine the Cochrane Library have established routines for assessing the
quality of individual experiments and for synthesis
• In the context of medicine, however, interventions are primarily mechanical: a
treatment which works with one human being is very likely to work with another
• In industrial support, however, the situation is more complex given the
heterogeneity of firms, the social and interactive nature of business and the
importance (and diversity) of the contexts in which firms operate.
• Internal and external validity may therefore be more difficult to maintain in
industrial policy RCTs than in medicine.
• Arguably this means that standards of evidence should also be higher, requiring
consistent evidence from multiple RCTs from different contexts, before robust
conclusions should be drawn

7. Entrepreneurship and innovation
policy shifts in the UK since 2010
• Move towards ‘place-based’ policy in entrepreneurship policy;
innovation policy become more national. De-coupling?
• Perhaps key change has been closure of Regional Development
Agencies (which used to do both)
• Entrepreneurship policy now Local Economic Partnerships (39 in
England); Innovation policy national agency (InnovateUK or TSB)
• Also development of ‘Growth Hubs’ intended to provide/co-ordinate
local services for growth companies
• Remember too UK part of EU and so EU initiatives such as Horizon
2020 also potentially important

8. RCT1: Creative Credits -2010-12
• Creative Credits: a B2B voucher scheme
intended to establish innovation
partnerships between SMEs and creative
services suppliers, e.g. designers.
– Manchester City Region – Sept 2009 to
October 2010
– 150 Creative Credits worth £4,000 with
£1,000 firm contribution to stimulate
collaborative project
– 672 applications, 22 per cent funded,
rest to control
• Outcome measures (measured after 6 and 12
months):
– Short-term – project additionality –
more projects?
– Longer term
• output additionality (sales, innovation),
• behavioural additionality (innovation
intention),
• network additionality (partnering
intention)
Application
Randomised
Creative
Credit
150 companies
No Creative
Credit
422 companies
Follow-up
12 months
Follow-up
12 months

9. RCT1: Collecting evaluation data –
in practice
• Four surveys over two years so
attrition was a problem – used
some small financial incentives
• By survey 4. Treatment -78 per cent
response (n=117), control group
52.2 per cent (n= 157)
• Clearly bias between groups but
comparison of baseline
characteristics suggests NO
systematic bias within groups
• Longitudinal sample therefore
considered ‘representative’ and
preserved internal validity

10. RCT1: On additionality…
• Comparison of treatment and control groups suggest:
• Project additionality - firms receiving Creative Credit were 84 per cent
more likely to go ahead with their project – a strong positive effect
• Output additionality – (product/service innovation, process innovation,
sales growth) - strong positive effect after 6 months, no effect after 12
months – a transitory effect (and misleading Interim Report)
• Behavioural and network additionality – no significant effects after either
6 or 12 months – a negative result
• Results confirmed by 2-stage Heckman models

11. RCT2: Growth Vouchers
• Launched in January 2014 and ends in March 2015.
Provides subsidised strategic business advice to
around 20,000 small businesses through randomly
allocated vouchers. Budget £30m.
• Each voucher is worth up to £2,000 excluding VAT. It
can be used to cover half the cost of advice in one
of five areas:
– Raising finance and managing cash flow;
– Recruiting and developing staff;
– Improving leadership and management skills;
– Marketing, attracting and keeping customers; and
– Making the most of digital technology.
• Firms not getting a voucher will be given some
more general information on other sources of
potential support as well as getting access to the
website with private suppliers of advice.
• Growth Vouchers may also help to tell us which
types of advice are most valuable to firms.
• Plan is to follow up for 2-3 years after the end of
the period of support. Programme is targeted at
firms which do not usually use business advice.

12. RCT3: Growth Impact Pilot
• March 2014 March 2015 - Aim to
examine the impact of business
coaching on growth. Trial linked
to major government programme
called ‘Growth Accelerator’.
• SMEs provided with subsidised
leadership and management
training and some then given
access to additional business
coaching
• Coaching linked to access to
finance, business development or
innovation
Enrolment
Leadership and
management
training
Randomised
Business
coaching
C. 300 firms
No business
coaching
c. 300 firms
Follow-up
2 years
Follow-up
2 years

13. RCT4: University Growth
Vouchers
• Launching October 2014 – March 2015 with
target of 1250 SMEs.
• Aims to test whether firms who receive
subsidised leadership and management
education training (10 workshops,
networking events and mentoring) from six
UK Business Schools perform better than
those who have received no business advice.
• Growth Vouchers the scheme operates on a
co-funding basis with £2000 Growth
Voucher per business and the business
matching this funding. (Business schools are
receiving some additional funding for
participation).
• Analysis planned over two years. Careful
analysis of attrition to maintain internal
validity
Enrolment
Diagnostic
Workshop
Randomised
Receive Growth
Voucher
C. 500
No Growth
Voucher
C. 750
Follow-up
2 years
Follow-up
2 years

14. Assessing validity …
Internal validity External validity
RCT1: Creative Credits Proven robust Problematic – small
regional experiment
RCT2: Growth Vouchers Large numbers should be
okay but not proven yet
Larger numbers so should
be okay – not proven
RCT3: Growth Impact Pilot Okay. Careful design and
allocation
Small sample so may be
problematic as RCT1
RCT4: University Growth
Vouchers
Okay. Careful design and
allocation
Medium sample so may be
okay – not proven

15. Implications for the future
• By 2016 early results from all four RCTs. What will this mean?
• Actually hard to know!
• Key questions about whether can preserve internal validity of experiments
e.g. sample biases and attrition may be a problem in smaller trials (RCT3,
RCT4). External validity potential issue in all trials (again particularly
smaller ones)
• English saying: ‘One swallow doesn’t make a summer’. Similarly might also
argue that ‘One RCT doesn’t constitute robust evidence’. Do we need
replication as the Cochrane Review approach suggests?
• Another key issue is unwarranted generalisation. RCTs can provide good
evidence on very specific interventions and delivery mode and that is all.
Not generalizable to all business advice, all mentoring etc. Slight changes
may improve/upgrade effect. So need to be cautious in interpretation

16. Conclusions
• Creative Credits project (RCT1) did suggest feasibility of RCT approach to
industrial policy:
– Randomisation - > internal validity
– Longitudinal monitoring -> captured varying output effects
– Qualitative work -> generated causal insights
• UK rapidly developing some experience in this evaluation approach both
design and operational aspects (and further RCTs planned)
• Studies have yet to contribute to evidence base but potentially powerful
addition but ….
• Need to be somewhat cautious about generalising from single trials and
also generalising beyond the real ‘test’ provided by the trials.

17. Thanks
Contact:
Stephen.roper@wbs.ac.uk