TCI Cluster Evaluation Meeting Oslo

Cluster Evaluation
TCI Cluster Evaluation Working Group Meeting | Oslo, Norway
7-8 September 2017

AGENDA
Thursday
• Introductions and TCI evaluation WG ‘story so far’
• Welcome and introduction to clusters in Norway
• Theme One: Programme level evaluation
– Norway’s programme evaluation
– Reaction/experiences of others and questions
– Small group discussions around set of guiding questions
– Reporting back and next steps
Friday
• Re-cap and overnight reflections on day 1 discussions
• Theme Two: Capturing the human element of clusters
– Review of results and insights from survey pilots / alternative approaches
• Other issues / emerging opportunities in cluster evaluation

AGENDA
Thursday
• Welcome and introduction to clusters in Norway
• Introductions and TCI evaluation WG ‘story so far’
• Theme One: Programme level evaluation
– Norway’s programme evaluation
– Reaction/experiences of others and questions
Friday
• Re-cap and overnight reflections on day 1 discussions
• Theme Two: Capturing the human element of clusters
– Review of results and insights from survey pilots / alternative approaches
• Other issues / emerging opportunities in cluster evaluation

TCI CLUSTER EVALUATION WG
• A forum for learning collectively around these common,
complex & important evaluation challenges
• Sharing learning
• Identifying gaps
• Trying new approaches

2013 2014 2015 2016 2017
Workshop
Forres
TCI
Conference
Kolding
TCI
Conference
Monterrey
TCI
Conference
Eindhoven
Workshop
Belfast
Workshop
Rzeszow
Workshop
Barcelona
TCI
Conference
Daegu
Coming
Up in
2017
Oslo
Bogota
TCI CLUSTER EVALUATION WG

SOME KEY OUTPUTS
• Booklet on Designing Cluster Evaluation (2014)
• Cluster evaluation boardgame (2014)
• Perfect cluster evaluation framework (2016)
• Principles to guide cluster evaluation (2016)
• Set of firm-level survey questions aimed at capturing the human element
of clusters (2016)
• Poster & short paper presented at OECD Blue Sky Forum (2016)
• Collected together on website:
www.tci-network.org/evaluation

PROGRAMME LEVEL EVALUATION:
GUIDING DISCUSSION QUESTIONS
• What do programme level evaluations add to cluster initiative level
evaluations?
• What are they key aspects that should be included in such evaluations?
• What methods/data can illustrate the systemic effects and wider value of
clusters?
• Should/can we account for interactions between cluster programmes &
other policies?
• Where are opportunities to learn from each other? Does a ‘standardised’
approach make sense?

A ”generally accepted” effect logic for cluster
programmes, but still lots of questions
...contribute to increased interactive
learning and collaborative research
and innovation projects
...which contributes to increased
innovation, international attractive-
ness, productivity and growth
Activities to
strengthen or
upgrade a
cluster/
innovation
environment...
Input/Resources Activities Results/Outcomes Effects
3-10 years >10 years
Structural
capital
(tangibles)
Social capital
(intangibles)
Results/Outcomes
3-10 years

In particular, how can we better understand
the ”miracle” in the middle?

Need to evaluate impacts both on collaborative
dynamics and economic performance...but how?
Data/Indicators:
? Number/type/strength of engagement of actors
in cluster initiative
? Number/type/strength of
alliances/collaborations among cluster
participants
? Number/type/strength of
alliances/collaborations with related actors
outside the cluster
Methods of data collection/analysis:
- Surveys
- Interviews
- Social network analysis
Data/Indicators:
- Number/quality of publications and patents
(and other IP)
- Number of new products/processes/ services
- Number of new firms/firm growth
- Level of investments attracted (VC, FDI)
- Firm-level revenue/growth; export/growth;
employment/growth; and wages/growth
Methods of data collection:
- Surveys
- Interviews
- Business registers/national statistics
Impacts on Collaborative Dynamics (aka the human element)
(engagement, linkages/interaction, collaboration/collective action)
Impacts on Economic Performance
(intermediate outcomes and productivity)
Note: see Giuliani et al. (2014) for additional elaboration on the ”two-stage” evaluation process

• Internal and external linkages/ network
ties (structural)
– quantity of new linkages
– type/proximity of partner
– quality of linkage
• Engagement/trust/commitment
(relational)
– type of engagement
– level of (company) commitment/reciprocity
• Shared vision and identity (cognitive)
– common vision
– collective action
(inspired by both academic/theoretical frames
and existing monitoring/evaluation practices)
What are the dimensions of collaborative
dynamics we think are important to understand?
Collaborative Dynamics can be
characterized by...
• Increased interaction and knowledge sharing
between different types of actors
• Increased trust and deeper types of
collaboration (from information and
knowledge sharing to strategic collaboration)
• Participating actors’ perception of benefits
from pursuing joint activities (addressing
common goals)
• Participating actors’ commitment to collective
action (without guaranteed reciprocity)
• Participating actors’ perception and support
of a shared rationale or value proposition for
collective action
• Participating actors’ perception and support
of a shared identity
(see description of ”The Perfect Cluster”)
Proposed Dimensions and Indicators

A proposed way to monitor this
development (through a firm-level survey)
Based on theory and practice…a survey in four parts
A. Company information (2)
(could be provided by cluster manager)
B. Economic data (4-5) (could be sourced through business registers)
C. Perceived value of collaborative strength (5) (responses indicate companies’
perception of social capital/shared vision and value of collective action)
D. Collaborative dynamics (responses indicate the type of engagement and
dynamics within the cluster initiative, and companies’ perception of social
capital/shared vision and value of engaging in collaborative activities)
1. Number and type of new linkages (4)
2. Engagement in collaborative activities (6)
3. Perceived value of collaborative activities (6)
4. Brief example of how collaboration within the cluster has provided added value

Pilot experience
Pilots reported at Eindhoven
• Australia
– Issues with combining needs and
questions (govt, cluster mgrs, TCI survey)
– Resulting survey different
– Planned tests with 2 initiatives
• Plymouth, UK
– Tested with one initiative
– Revisions to questions required (wording
and fewer)
– Question on linkages (D1) problematic
– Additional tests with 2 initiatives
• Colombia
– Tested with 40 cluster initiatives
– No big modifications made, but critique
from firms on responding to question on
linakges (D1)
Additional pilots
• Basque country
• Catalonia
• Scotland

Presentation to TCI Evaluation Working
Group
Evaluation for Scottish Enterprise
Evaluation of Leadership role of
Scotland Food and Drink

Scotland Food and Drink
 Food and drink – the key sector for Scotland?
 Industry body for the sector
 Brings together business, government agencies,
research, trade bodies – public private partnership
 Membership body, but also government funding
 Leadership role for whole of sector

Evaluation Context
 Separated gov funding – services and leadership role
 Evaluation (interim) of leadership role funding
o Strategy/shared vision
o Setting agenda – common themes/challenges
o Building reputation and common identity
o Influencing government and policy
 NOT evaluation services (eg export support, meet the
buyer, market intelligence)
 Challenge to separate, but interesting to evaluate value
of strategic role

Evaluation
 Stakeholder interviews, focus groups, research and
reporting, company survey
 Co survey
o 54 responses (members 39% and non members 61%)
 Included some WG survey questions
o Perception and value of collaboration, and return on
investment
o C2, D2, D3, D4
o Already capturing co data (A&B), co size and turnover

Results - general
 Good understanding of SF&D Leadership role
o Promoting sector, building reputation
o Focal point for collaboration
o Establishing strategy and common identity
 In SF&D absence
o Collaboration more difficult
o Growth of sector/exports less
 Over two-thirds of companies reported that
SF&D’s industry leadership role has had a
positive impact on the food and drink industry

Results – survey questions
 Companies’
perceptions of
social
capital/shared
vision and
value of
collective
action (i.e.
perceived value
of collaborative
strength) – C2
 Generally +ve
 Members>
non-members
40
31
36
31
25
23
22
21
6
13
7
10
16
13
14
11
5
6
5
5
7
11
9
11
2
3
1
2
4
4
1
2
4
We are convinced that working with others provides
long-term benefits to our company to build the
reputation of the sector
long-term benefits to our company for consumer
and market intelligence
long-term benefits to our company for access to
new markets
long-term benefits to our company to influence
policy
long-term benefits to our company for skills and
workforce development
long-term benefits to our company for
product/technology development
In my experience, other members of the sector are
open and willing to exchange information (about
e.g. suppliers, clients) and experience/expertise in
order to tackle common issues
When my company has a challenge that cannot be
resolved in isolation, we usually first turn to
someone in the sector to help find a solution
Agree Somewhat
Agree
Neither/Nor Somewhat
Disagree
Disagree

 Type of
engagement and
level of
commitment of
companies to
collaborative
activities – D2
 Almost three-
quarters of
companies had
taken part in
some form of
collaborative
activity over the
last 18 months
(74%).
26
18
18
16
15
11
9
6
Developing knowledge/research
Supporting access to new domestic markets
Improving market intelligence and strategic
focus
Fostering innovation
Supporting internationalisation
Other
Attracting or developing talent
Attracting investment

 ROI of that
engagement–
D3
 Almost all of
the
collaborations
undertaken by
companies
(87%) were
valued as high
or moderate
10
9
8
11
6
5
7
4
9
4
4
1
6
2
2
2
2
2
2
2
1
1
1
1
1
Developing knowledge/research
Supporting access to new domestic markets
Improving market intelligence and strategic
focus
Fostering innovation
Supporting internationalisation
Other
Attracting or developing talent
Attracting investment
High Moderate Low Don't Know

Survey feedback
 Overall results suggests that companies value
collaboration, and perceive a high return on investment
from that collaboration.
 Helps identify level of collaborative activity and types of
relationship
 Shows the value and importance placed of collaboration
 Important evidence for SF&D as building strategy
 Survey – challenge with survey size
o D1 too much detail
o C3,4,5 – of importance but “squeezed out”
o C1 – seen as a given so dropped

What’s happened?
 SF&D built new strategy (in parallel to evaluation)
o Major industry engagement to inform
o Building on success story of growth
o More to do in innovation and exports
o Collaboration at the core
 Evaluation helped support further government funding
for Leadership role to support
 http://www.foodanddrink.scot/industry/strategy.aspx

Madeline Smith, SmithKelvin
madelinesmithkelvin@gmail.com
Evaluation undertaken by EKOS and SmithKelvin
for Scottish Enterprise and public sector partners

Reflections on Basque Country
Pilot Cluster Survey
TCI Cluster Evaluation Working Group Meeting | Oslo, Norway
7-8 September 2017

BACKGROUND
• SPRI conducts annual ex-ante and ex-post evaluation of the
action plans of supported cluster associations
– Assesses strategic plan, budget, staff, composition, participation and
relevance according to a set of criteria
• Objective of this new (in 2017) pilot is to better understand the
perceptions of cluster members about the impacts of cluster
activities on their competitiveness
– Survey to cluster members
– Designed and agreed with cluster associations, and incorporating
ideas/questions from TCI cluster evaluation working group
– Initial pilot with 4 cluster associations conducted in May 2017
– Being revised with plans to roll out to all associations in 2018

SURVEY STRUCTURE
• PART 1: Involvement of the firm in the activities of the cluster
– How many people participate in different types of activities
– Perception on degree of participation of firm as a whole
• PART 2: Specific areas of cluster cooperation
– Which areas are considered most important for the firm’s competitiveness?
– What type of role does the association currently play in each area?
– What type of role should the association play in each area?
• PART 3: Impacts of cluster cooperation (employment, exports, etc)
– Perceptions of impacts in last 5 years
– Expected impacts in next 5 years
• PART 4: Overall collaboration within the cluster
– Perceptions on common vision, openness to collaboration, etc.
• PART 5: Background information on the firm & respondent TCI SURVEY
PARTS A & B
TCI SURVEY
PART C
(Q1, 3, 4, 5)
TCI SURVEY PART D
(ELEMENTS Q2, 3)

PART 2 EXAMPLE: AREAS OF COOPERATION

INSIGHTS FROM PART 2
• Areas of cooperation given most importance:
– Technology watch and competitive intelligence
– Promoting networking and interaction among cluster members
• And least importance:
– Joint procurement and buying centres
– Developing the supply chain
• Associations are seen to take most leadership in:
– Promoting the visibility and knowledge of the cluster towards the outside
• And are seen not to intervene in:
– Joint procurement and buying centres
– Developing the supply chain
• Areas where associations should take greater leadership
– Development and implementation of standards, certifications and regulations

PART 4: OVERALL CLUSTER COLLABORATION
TCI SURVEY
PART C
(Q1, 3, 4, 5)

CAPTURING HUMAN ELEMENT:
GUIDING DISCUSSION QUESTIONS
• What specific revisions could be made to the survey?
• Are there specific questions that could be included in more general firm-
level surveys?
• How can other approaches complement surveys in capturing
collaborative dynamics?
• What should be the next steps to bring together the data/experiences that
are emerging?

OTHER ISSUES / EMERGING
OPPORTUNITIES
• Review of publications on cluster evaluation
• Evaluation session at the Bogota conference
• Cluster evaluation and smart specialization strategies
• Cluster evaluation ‘beyond GDP’
• Cluster evaluation and the agenda 2030 for sustainable development
• Collaboration with next round of European Cluster Observatory: smart
guide to cluster evaluation
• Volunteer for next meeting (Spring 2018?)

TCI Cluster Evaluation Meeting Oslo

More Related Content

What's hot

Similar to TCI Cluster Evaluation Meeting Oslo

More from TCI Network

Recently uploaded

TCI Cluster Evaluation Meeting Oslo