Application of various designs in evaluation of regional programs

Anna Borowczak, Ph.D. candidate
supervised by Prof. Pawel Churski
Regional Analysis Department, Institute of Socio-Economic Geography & Spatial Planning
Poznań, 25th of June 2015

1. Reasons justyfying the topic of disseration.
2. Scientific objective and research questions,
time-frame and territorial scope.
3. Choice of references.
4. Data collection and methods.
5. Structure of the content.
6. Terminology.
7. Theoretical background.
8. Model of evaluation for regional operational programs.
9. Empirical validation of evaluation model with three independent evaluation
designes.
10. Conclusions and discussion.
Part of the research presented in this dissertation was funded by the
National Centre of Science
in the framework of a research project no UMO-2011/01/N/HS5/00100
managed by the Author of this dissertation.
Agenda.

• Evaluation is an important tool for public administration, while it serves the
purpose of getting a market feedback, when public bodies need to assess their non-profit
investments;
• Evaluation researches play key role in regional development, because they
constitute substantial part of empirical research;
• Nevertheless, evaluation research practice is neither sufficiently supported
by theoretical work, nor it is commensurately reflected in literature on socio-
economic geography;
• This may be a major reason for poor methodological quality of evaluation research,
which is backed by the following observation: „evaluation researches are characterized by the lack of
consistent methodology rooted in literature, limited number of methods, lack of more complex cause and effect analysis,
reaching beyond planned indicators towards real changes evoked by the intervention, including changes of relations (...)
Evaluation designs are often simply compliant with guides issued by the European Commission, being mostly technical
and imitative [Olejniczak, 2009].
• But the other reason lies in intrinsic methodological challenges
of evaluation research, stemming from the characteristic co-operation between
public body and evaluator in the process of shaping the methodology of the each
research;
Justification of the topic.

This dissertation seeks to:
• Enhance methodological discussion in the field of evaluation of regional
development.
• Make an atempt to incorporate evaluation research into regional science.
• Help public administration to achieve best results with respect to scientific
knowledge.
• Help evaluation professionals to design better evaluation studies.
• Improve reliability of evaluation studies in regional development.
Gaps addressed in the dissertation.

Scientific objectives & research questions.
Establishing an evaluation model including the outcomes of empirical validation
carried out with three independent evaluation designs
Construction of evaluation
model for regional programs
Empirical verification
and validation
of evaluation model
with three independent
evaluation designs
Determining advantages
and limitations of each
evaluation design
Main objective
Theoretical and methodological
objective:
Cognitive objective:
Applicable dobjective:

Research questions:
1. What theoretical aspects of regional development should be considered when
drafting terms of reference for evaluation research?
2. What are information needs declared by public bodies with regards to
investments in regional development?
3. What factors determine the application of particular evaluation design in
evaluation of regional programs?
4. Which evaluation design offers relevant information as seen from both practical
and theoretical angles on regional development?
5. Do the outcomes of evaluation designs differ among each other? To what
degree are they similar/ contradicting?
6. Do the outcomes allow for reducing the level of insecurity while making
decisions on reshaping the next interventions in regional develoment?
Scientific objectives & research questions.

Subject of evaluation research:
embraces investments from two generations of programs:
Integrated Regional Operational Program (2004-2006)
Regional Operational Program (2007-2013)
Time scope:
years 2003 – 2010
enables an analysis of two crtical moments:
(2003) before any intervention started and
(2010) cummulation of investments from two programs
based on accessible data.
Territorial scope:
• Poznański subregion (61) – at NUTS 3 level:
• suburban region with 565 thous. inhabitants
• meso-scale, enables analysing both macro- (measured
GDP) and microeconomic interdependencies.
Time-frame and territorial scope.
Source: Author’s own elaboration.

Choice of references.
Evaluation
Category Author, edition
Definition, types
and functions
Alkin, M., 2004; Chelimsky, E., 1997; Campbell D.T., 1979; Dahler-Larsen P., 2005;
Drobniak A., 2009; Kierzkowski T. 2002; Korporowicz L., 1997; Mathison S., 2004;
Mizerek H., 2002; Olejniczak K., 2008 (a,b); Rossi P.H., Lipsey M.W., Freeman H.E., 2004;
Sanders J., 1994; Scriven M., 1982, 1992; Shadish,W.R., Jr., Cook T.C., Leviton L., 1991;
Stern E., 2004; Stufflebeam D.L., Madaus G.F., Kellaghan T., 2000; Vedung, V., 1997.
Methodology Anderson, A.A. (undated), Morton M.H., 2009; Barkley D.L., 2008; Baslé M., 2006;
Bradley J., 2006; Chen H. T., 1990, 2004; Churski P., Borowczak A., 2010; Connell, J.P,
Kubisch, A.C., 1998; Cook T.D., Campbell D.T., 1979; Donaldson S.I., 2007, Dutkowski M.,
2008; Górniak J., Worek B., Krupnik S., 2007; Guba E., Lincoln Y.S., 1989; Haber A., 2007;
Henry G.T, 1998; Jasiński M., Kowalski K., 2007; Komornicki T., Rosik P., Śleszyński P.,
Pomianowski W., 2010; Leeuw F. L., 1991, 2008; Malik K., 2009, 2011; Malik K., Heffner
K., 2010; McLaughlin J.A., Jordan G.B., 1999; Owen J.M., Rogers P.J., 1999; Patton M.Q,
2002; Pawson R., 2006; Pawson R., Tilley N., 1997, 2004; Potter J., 2009; Pylak K., 2009;
Roberts P., 2006; Trochim, W., 1989(a,b); Trzciński R., 2009; Weiss C.H, 1972; Wholey J.,
1983. W tym uzupełniająco: Antoszkiewicz J., 1990, Babbie, E., 2004, Bickman, L. 1987;
Chojnicki Z., 1999, 2008; Gruszczyński, L.A., 2003; Hammersley M., Atkinson P., 2000;
Heffner C.L., 2004; Kuhn T., 1968; Nowak S., 1965; Rogacki H., 2003; Such J., Szcześniak
M., 2006; Sztumski J., 1995; Tokarski T., 2011.

Regional
developmentEvaluation
Legal and
administrative
aspects of
regional policy
evaluation in
Europe &
Poland
Bachtler J., 2000, 2001; Bachtler J., Poverari L., Taylor S., 2000; Bachtler J, McMaster I.,
2008; Batterbury S., 2006; Bienias S.,2012; Bober J., 2007; Dutkowski M., 2008; Górniak
J., 2007; Haber A., 2007; Malik K., Bedrunka K., 2008; Mazur S., 2007; Mierzejewska L.,
2009; Olejniczak K., 2009; Żuber P., Bienias S., 2008
Theoretical
aspects of
regional
development
Barro R., Sala-i-Martin X., 1995; Budd L., Hirmis A.K., 2004; Camagni R., 2002;
Chądzyński i in., 2007; Chojnicki Z., 1999, 2008; Chojnicki Z., Czyż T., 1991; 2006; Churski
P., 2008, 2009; 2011; Czapiński J., Panek T. , 2009; Czyż T., 2008, 2012; Dijkstra L.,
Annoni P., Kozovska K., 2011; Domański B., 2004; Domański B., Gwosdz K., 2008;
Domański R., 2002, 2008; Giannias D., Liargovas P., Manolas G., 1996; Golejewska A.,
Gajda D., 2012; Gorzelak G., 2003, 2004, 2009; Gorzelak G., Jałowiecki B. 2001;
Gorzelak G., Smętkowski M., 2005; Grosse T.G., 2002; Huggins R., Izushi, H., 2008; Klasik
A., 2002; Kliber P., Malaga M., 2003; Kudłacz T., 1999; Markowski T., 2009; Molle W.,
2007; Nermend K., 2008; Orłowski W., 2003; Parysek J.J., 2008; Porter M., 1990; Potter
J., 2009; Sengupta J., 2011; Stackelberg K., Hahne U., 1998; Stimson R., Stough R.,
Roberts B., 2003; Storper M., 1997;

Regional
development
Contemporary
conditions for
regional
development in
Europe and
Poland
Ahner D., 2008; Bachtler J., Wren C., 2006; Bachtler J., Raines P., 2012, Bachtler J., Yuill
D., 2001; Barca F., 2009; Barca. F., McCann P., Rodriguez-Pose A., 2012; Begg I., 2001,
2008; Domański B., 2008, 2012; Faludi A., 2009; Fenge R., Meier V., 2008; Fesus G.,
Roller E., 2011; Gaffey V., 2010; Grosse T.G., 2008; Kukliński A., 2003; Markowski T.,
2009; Rodriguez-Pose A., Fratesi U., 2004; Ross-Larson B., 2008; Steiner M., 2013;
Strzelecki Z., 2011, 2013; Swianiewicz P., 2011; Szlachta J, 2011, 2012; Vanthillo T.,
Verhetsel A., 2012; Wierzbicki A., 2008; Zaucha J.,
Szlachta J., 2012.

Data collection & methods.
Published secondary data:
Eurostat, Polish Central Statistical Office, Polish Central Custums Office,
evaluation research databases by European Commission and Polish Ministry of
Development.
Unpublished secondary data:
• project database of Polish Ministry of Development, Regional Voivodship Office
and Regional Marshall Office (n2004-2006=148, n2007-2013=225);
• Terms of Reference regarding ex-post evaluation issued by the Central Evaluation
Unit and 16 Regional Marshall Offices in Poland (n=23).
Primary data collection:
• semi structured interviews -CATI- computer assisted telephone interview (n=802);
• questionnaire addressed to local development experts (n=6).

Structure of CATI respondents
(for purposes of theory-driven and particpatory evaluation designs)
Type of respondent
Total no of entities
in relevant area
No of interviews carried out
in relevant area
ENTERPRISES
• IROP Program 2004-2006 50 20
• ROP Program 2007-2013 220 85
• non-participants 73 000 365
NGOs
total 1800 300
LOCAL GOVERNANCE
local communities
poviats
37
5
29
3
TOTAL - 802

Type of respondent Name
NGOs by category
Economic • National Chamber of Commerce
• Cluster Assotiation
Society and Environment Center for Fostering Eco Development
Regional Governance by responsibilty
IROP 2004-2006
Regional Operational
Program (2007-2013)
Voivodship Office
Marshall Office
Scientific environment
--- Institute of Socio-Economic Geography & Spatial
Planning (Adam Mickiewicz University)
Structure of expert panel (for purpose of participatory evaluation)

Statistical-quantitative methods
• indictator analysis
• absolute beta-convergence
• principal component analysis
• structural equasion models
• multidimensional scalling
Social research methods
• textual analysis
• heuristic methods –concept mapping
• field research
Cartographic methods
• ArcGIS
Methods &
techniques
Concetptualization
logic models,
concept mapping
Observation
and analysis
structural equasion
models,
indicators,
multidimensional
scalling
Fig. Breakdown of methods according to the stage
of evaluation procedure
Source: Author’s own elaboration

Chapter I. Introduction
1.1. Scientific objective and scope of research
1.2. Outline of literature
1.3. Data sources and methods
Chapter II. Evaluation as a tool for invesitigating public interventions
in regional development
2.1. Definitions and characteristic features of evaluation
2.2. Types and functions
2.3. Assessment criteria
2.4. Methodology and research designs
2.5. Systemic conditions of evaluating regional programs under the framework
of EU cohesion policy in Poland
Structure of the content.

Chapter III. Regional development as the topic of evaluation research
3.1. Regional development in Regional Science. What important lessons can fuel
evaluation research?
3.2. Evaluation of regional development in perspective of policymakers. What do we think
we need to know?
3.3. Model of evaluation for regional programs
Chapter IV. Regional programs in Poland
4.1. Regional programs in the context of EU cohesion paradigm
4.2. Regional programs as planning documents. What is planned, where and why?
4.3. Regional programs in Poland
4.3.1 Integrated Regional Operational Program
4.3.2 Regional Operational Programs

Chapter V. Testing the evaluation model for regional programs in poznanski
subregion
5.1 Contextual diagnosis of regional development in poznanski subregion
5.1.1 Economic cohesion
5.1.2 Social cohesion
5.1.3 Territorial cohesion
5.1.4 Competetiveness
5.1.5 Living standards and quality of life
5.2 Empirical verification of the evaluation model
5.2.1 Objective-based evaluation design
5.2.2 Theory-driven evaluation design
5.2.3 Participatory evaluation design
5.3 Comparative analysis of the outcomes
Chapter VI. Conclusions and discussion

Evaluation in a narrow sense:
a single research process, seeking to determine the merit and value of public
intervention, characterized by high utility of evidence-based conclusions and reliable and
objective methods, or a product of such process.
[Scriven 1990, 1991; Stufflebeam et al..2000; Patton 2002; Alkin 2004, Such & Szcześniak 2006, Górniak 2007, Haber
2007, Olejniczak 2008]
Evaluation in a broader sense:
a body of normative research aimed at optimization of public interventions,
included in the applicable strands of various scientific disciplines, adequate to the
generic subject of evaluation research. In case of evaluating regional programs,
evaluation may be considered an applicable strand of socio-economic geography,
providing a respective scientific rigor of evaluation research is maintained;
[Chen 1990, Scriven 1991, Shadish 1991; Chojnicki 1999; Alkin 2004; Mathison 2004]
Basic terminology.

Basic terminology.
Meta-evaluation
analyis of the quality of evaluation research, carried out with respect to the following
criteria:
• Reliability: predictable, accurate and replicable procedure of research;
• Credibility: significant cause and effect chains;
• Generalizablity and transferability: universal conclusions;
• Objectivism: neutral attitude towards object of research, well-documented oppinions
• Responsiveness: utile conclusions for stakeholders and recipients.
[Weiss 1972, Guba & Lincoln 1989, Chen 1990]

Key factor differentiating designs is the way,
in which the effects of the intervention are
determined and priorititized against each other.
1. Objective-based evaluation
• explicit objectives as specified through indicators
or stated in the description of intervention
2. Theory-driven evaluation
• reconstruction of the logic behind the intervention
based on scientific theories
• reconstruction of political, behavioral and social
logic justifying the intervention
3. Participatory evaluation
• effects of the intervention as registered and
weighted in the perception of the stakeholders
Theoretical backgound.
Evaluation designs.
Hermeneutic circle
Logic model

Paradigm: universally recognized scientific
achievements that, for a time, provide
model problems and solutions for a community
of researchers
[Kuhn, 1968].
Paradigm of regional development:
• universally accepted hierarchy of values,
which are measures in meeting social needs
[Chojnicki 2008].
• economic doctrine explaining actual mega-
trends and mechanisms specified in recent
theories of regional development
[Kudłacz 1999 Stimson 2003, Churski 2008].
Theoretical backgound.
Paradigm of regional development.
Source: Author’s own elaboration.

Model of evaluation for regional
programs.
Model derived from:
• compilation of theoretical acpects of Regional
Science
based on paradigm and final approach to regional
development (i.e. considered a deliberate change
between two conditions of the system)
• analysis of 23 ToR’s for ex-post evaluations
drafted in the 2004-2010;
including measuring impact of intervention on regional
development and considering functionality of
interventions in terms of their morphology and
responsiveness to the actual needs (whether
geographical or social).

Outcomes of empirical
verification

Evaluation in poznanski subregion.
Contextual analysis.
Diagnosis of socio-economic development in poznanski subregion serves the purpose of
being a point of reference to outcomes of three evaluation designes in a further comparative
analysis.
It is of a macroeconomic character and refers to the following processes:
• economic, social and territorial cohesion;
• changes of competitiveness;
• changes in living standards and quality of life.
observed between 2003 (before intervention) and 2010 (final state of the regional system),
and carried out in three spatial scales, whenever possible:
(a) all NUTS 3 units in Europe;
(b) all NUTS 3 units in Poland;
(c) all NUTS 3 units in Poland with regards to the intraregional discrepancies.

Socio-economic diagnosis.
Economic cohesion: aboslute beta convergence
EU NUTS III units
(EU 27)
Model
No of observations 1166
Changeability 75%- high
p (significant for p <0,05 ) <0,00001
Co-efficient of determination R2 0,7
 -0,0464113
 (momentum of convergence) 0,16
Values of GDP per head in 2010 are
estimated in real prices from 2003 . This
allows to observe real changes in PPP. HCPI
index has been used as a GDP deflator.

Economic cohesion: aboslute beta convergence
Polish NUTS III Intraregional
discreapancies
Model
No of observations 66 66
Changeability 10,5% 41,7%- average
p (significant for p <0,05 ) 0,09026 0,0008
Determination co-efficient R2 - 0,16
 - -0,0402308
 (momentum of convergence) - 0,04

Social cohesion: spatial distribution of Gini coefficient measuring the intensity
of intraregional income disparties broken down into 7 income groups in each
subregion.
2005 2010

Territorial cohesion:
descriptive change determined through
outcomes of following research
[Korcelli 2007, Konecka-Szydłowska 2009, Churski et .2009
Sierpieński 2010, Komornicki 2012, Rosik 2008, 2012,
ESPON 1.13, ESPON 1.44, Spiekerman & Wengner 2006,
2008]: enocompassing:
• settlement patterns,
• spatial accessibility.
Source: Komornicki &Rosik, 2013 Source: Spiekerman &Wengner, 2006

Competitiveness: spatial distribution of V1 component (PCA) interpreted as
structure and effectiveness of employment accross Polish subregions.
Author’s own ellaboration.
2003 2010

Living standards and quality of life: spatial distribution of S1 component
(PCA) interpreted as accessibility to public services and effective land use
in subregion accross Polish subregions.
2003 2010

Outcomes of contextual diagnosis.
Strenghts of poznanski subregion:
- confirmed economic convergence with other EU NUTS III units (higher than
average momentum);
- confirmed economic convergence with other subregions in region against the
backdrop of all Polish regions (average momentum);
- low but fast growing competitiveness as compared to the EU;
- high competitiveness vs. other Polish subregions;
- high wages and employment level vs. other Polish subregions;
- sptatial accessibiity is low when confronting other EU NUTS III units, but one of
the best in country.
Weaknesses of poznanski subregion:
• relatively lower accessibility to public services rooted in violent suburbanization
process;
• high income polarization, high concentration of incomes in industry and
contsruction sectors, relatively low structural diversification of incomes.

RESEARCH PROCEDURE:
Conceptualization phase: logic model of
intervention based on aggregated indicators
describing products and results of IROP and RPO
programs.
Observation and analysis phase: indicator
analysis, regression analysis
Data: quantitative
Source of data: field work collecting
the data on projects stored by the
implementing institutions
148 projects of IROP with 700 indicators;
225 projects of RPO with 70 indc.
Interpretation of the effect caused
by intervention: material effect
Objective-based evaluation.

Lp
.
Wskaźnik produktu
Okres
Wskaźnik rezultatu
Okres
Nazwa
Jedn.
Wartość
Nazwa
Jedn.
Wartość
obornicki
poznański
szamotulski
średzki
śremski
RAZEM
2000-2006
2007-2013
obornicki
poznański
szamotulski
średzki
śremski
RAZEM
2000-2007
2007-2014
1 Długość wybudowanej drogi wojewódzkiej
km
- 16,02 - - 7,75 23,77 A x
Poprawa dostępu do terenów inwestycyjnych
ha
170 142 159 8 150 629 x -
2 Długość wybudowanej drogi powiatowej
km
- - - - - 0 A x
Liczba inwestycji na terenach inwestycyjnych
szt
- 23 2 - - 25 - x
3 Długość wybudowanej drogi gminnej
km
- 6,43 0,64 1,94 1,52 10,53 A x
Oszczędność czasu w przewozach pasażerskich
mln
zł
7,2 8,05 3,9 4,04 8,08 31,27 - x
4 Długość zmodernizowanej drogi
wojewódzkiej
km
11,53 25,17 36,36 0,45 43,32 116,83 A x
Oszczędność czasu w przewozach towarowych
mln
zł
1,8 17,29 16,6 1,46 20,4 57,55 - x
5 Długość zmodernizowanej drogi powiatowej
km
31,44 3,58 17,17 4,07 3,02 59,28 A x
6 Długość zmodernizowanej drogi gminnej
km
- 11,79 0 6,21 3,36 21,36 A x
7 Długość wybudowanych ciągów pieszo-
rowerowych
km
1,88 24,02 6,57 0,45 3,01 35,93 A -
8 Długość zmodernizowanych ciągów pieszo-
rowerowych
km
2,62 4,7 1,5 1,56 0,8 11,18 A -
In the course of aggregation 36 product and 17 result indicators
were identified from an extensive group of 700 IROP’s and 70 RPO’s
indicators to represent universal and trackable material effects of both
programs. Then indicators showing the biggest change versus their base
value were further examined with use of the regression analysis.

Outcomes of evaluation research:
The strongest impact has been confirmed for the following types of effects:
• number of supported SMEs (=0,66);
• number of modernized universities (=0,33)
• number of waste facilities (=0,33)
• modernized sewage system (=0, 028)
• modernized water system (=0,005)
Despite the magnitude of the impact, the greatest physical change vs. base indicators
has been observed for the infrastructural investments rather than in the investements
in economic realm.
These outcomes are partially compliant with socio-economic diagnosis:
• They confirm the actual investments were rightously targeted on existing territorial
deficits, such as accessibility to public (communal) services.
• But they do not reveal the actual impact on the scale of the deficits in 2010 (the
final state).

RESEARCH PROCEDURE:
Conceptualization phase: reconstruction of logic
model behind two regional programs based upon
regional development theories;
Observation and analysis phase: structural
model equasion (SEM);
Data: quantitative;
Source of data: field work, semi-
structured CATI interview (n=802);
by intervention: magnitude and vector (+/-)
of impact on basic parameters of regional
development.
Theory-driven evaluation.

Tab. Standarized estimants of SEM model displaying impact of EU funds on the level of
development of poznanski subregion (AGFI=0,923)
ue1 ue2 ue3 DP(s3) K WK1 WK2 PiWŻ
DP(s3) -0,033 -0,175 0,028 0 0 0 0 0
K 0,461 -0,299 0,296 0,358 0 0 0 0
WK1 0,084 -0,055 0,054 0,066 0,183 0 0 0
WK2 0,141 -0,092 0,091 0,11 0,307 0 0 0
PiWŻ 0,088 -0,092 -0,069 0,779 -0,466 0 0 0
Y (GDP) 0,342 -0,223 0,214 0,297 0,714 0 0 0,043
w5 0,053 -0,034 0,034 0,041 0,115 0,629 0 0
w1 0,021 -0,014 0,014 0,017 0,046 0,254 0 0
w4 0,046 -0,03 0,029 0,035 0,099 0,54 0 0
w2 0,059 -0,038 0,038 0,046 0,129 0 0,42 0
w6 0,049 -0,032 0,031 0,038 0,106 0 0,346 0
j6s7 0,061 -0,064 -0,047 0,537 -0,321 0 0 0,69
j5s6 0,061 -0,064 -0,048 0,543 -0,324 0 0 0,696
j3s5 0,066 -0,069 -0,051 0,585 -0,35 0 0 0,751
j2s2 0,068 -0,071 -0,053 0,604 -0,361 0 0 0,775
j1s1 0,077 -0,08 -0,06 0,678 -0,405 0 0 0,869

Outcomes of evaluation research:
• strong impact of investments in economic realm on competitiveness (while
indirectly influencing wages, employment and SMEs’ investments, observing
a weaker influence on favorable investment climate);
• investments in transport infrastructure noting positive influence on
competitiveness;
• investments in spatial accessibility noting only slight impact on quality of public
transport.
Side-effects:
• negative impact of investments in social realm on GDP;
• negative impact of investments in economic realm on spatial accessibility
(quality of public transport).
Fully compliant with socio-economic diagnosis:
• Intervention targeted at the competitiveness, model confirms the deficits as
diagnosed.

RESEARCH PROCEDURE:
Conceptualization phase: concept mapping
Observation and analysis phase: non-
structured interviews, multidimensional scalling,
assessment by an expert panel.
Data: qualitative
Source of data: field work, CATI in
poznanski subregion (n=802)
by intervention: qualitative effects
according to the perception of inhabitants
Participatory evaluation.

1. Determining
the effects
• Asking CATI respondents on the effects of
regional programs in unstructured way
2. Structuring
the effects
• Interpreting and ordering qualitative data
3. Assessing the
value of effects
and clustering
• Setting the panel of experts
• Expert assessment and clustering
4. Representing
the effects on
map of concepts
• Using statistical method to compute the
outcomes of qualitative research
5. Interpreting the
map of concepts
• Analysis of outcomes
• Labelling clusters of similar effects with
names
Procedure of concept
mapping

NAME OF EFFECT
1. Launching a new company
2. Enhancing productivity of local enterprises
3. Building competitiveness of local enterprises
4. Enhancing employment in local enterprises
5. Green investments in local enterprises
6. Improving quality of education:
7. Improving quality of public health services
8. Enhancing investments of local enterprises
9. Enhancing investments of public authorities
10. Enhancing investments of NGOs
11. Decline of public spendings
12. Improving the image of local territorial unit
13. Building sewage plant and systems ….
xx. ….
26. …. International integration of youth
Outcomes of the
following stages of
concept mapping
procedure:
1. determination
2. structurization
3. assessment
(clustering)
Assessment criteria:
relevance, urgence
ranging <0,3>

Fig. Similarity matrix (conceptual) Fig. Multidimensional scalling
Outcomes of following stages of the concept mapping procedure:
3. Assessment and clustering of the effects
4. Representation of the effects

GROUPS No of EFFECT LABELS FOR GROUPS OF EFFECTS RELEVAMCE URGENCE
GROUP 1 1,2,3,4,5,8 INVESTMENTS IN LOCAL FIRMS 2,53 2,23
GROUP 2 6,14,26 EDUCATION 2,06 1,56
GROUP 3 9,11 MAINATINING INVESTMENTS’ DYNAMICS BY PUBLIC AUTHORITIES 1,00 0,50
GROUP 4 12, 17 PROMOTION AND REVITAILIZATION 1,50 1,25
GROUP 5 16, 19, 18 LOCAL PUBLIC TRANSPORT AND TECHNICAL INFRASTRUCTURE 2,00 1,44
GROUP 6 7, 13, 15, 22 IMPROVING QUALITY OF LIFE 2,00 1,38
GROUP 7 24 25 IMPROVING LIVING CONDITIONS OF MARGINALIZED GROUPS 2,00 1,75
Single effects: 10 SUPPORT FOR NGOS 1,00 1,75
20 BROADBAND INFRUSTRUCTURE 2,50 2,00
21 EQUIPPING GREEN FIELDS WITH TECHNICAL INFRASTRUCTURE 2,00 2,17
23 SOCIAL ACTIVATION 2,33 1,83
Fig.: Labelling the clusters of similar effects with common names
Outcomes of the concept mapping procedure:
interpretation of clustering

Outcomes of the evaluation design:
In perception of the stakeholders of regional programs the greatest signifcance for the
regional development is attributed to the following effects:
• Investments in local enterprises,
• Investements in the broadband internet network,
• Social activation,
• Education (including modernization of educational infrastructure).
• Equipping green fields with technical infrastructure
Side-effects:
• Enhancing the enterpreneurs’ skills in planning investments
• Social activation of inhabitants
Partial compliance with socio-economic diagnosis:
intervention targeted in competitiveness.

Comparative analysis of evaluation designs.
Dimensions of comparative analysis of three independent evaluation designs
of regional development programs:
• cognitive dimension: degree, to which outcomes of respective designs
respond to evaluation questions specifying research issues following from
operationalized model of evaluation for regional programs;
• methodological dimension: degree to which criteria of meta-evaluation are
met in particular evaluation design.

Cognitive dimension of comparative
analysis.
Level of analysis Component of the model Analytical criteria
Outcomes
Planned effects
 Detailed description of effects
 Hierarchy of effects according to impact’s
magintude
 Hierarchy of effects according to impact’s
volume
Side-effects
 Sort of measured side-effects
Mechanisms Construction of intervention according to
policymakers
 Accuracy of impact’s measurement vs.
assumed level of spatial analysis
 Explanation of mechanism of exerting the
impact (i.e. mobilizing developmental factors)
 Explanation of origins and reasons of side-
effects
Theories of regional development
Needs
Needs of targeted geographical areas
 Adjustment of intervention to geographical
areas
 Identification of needs that have not been
satisfied
Needs of targeted social groups
 Adjustment of intervention to social groups
 Identification of needs that have not been
satisfied

Cognitive dimension of comparative
analysis.
General assumption: the more components of the evaluation model are considered in the
evaluation design the better.
• The best result is achieved by the theory-driven evaluation.
Limitations: not too detailed description of effects, low degree of determining the
correspondance of effects vs. needs of social groups.
• The average result is achieved by participatory evaluation.
Limitations: low chance to determine the hierarchy of effects in comparison to other designs,
does not explain the mechanisms of obtaining the effects.
• The weakest result is achieved by objective-based evaluation.
Limitations: does not identify side-effects, does not explain the mechanisms of intervention,
only to a small extent allows for assessing the adjustment of intervention to needs of social
groups, does not identify the unsatisfied needs.

Methodological dimension of comparative
analysis.
Criteria of analysis
By evaluation design
Objective-based Theory-driven Participatory
Reliability High Average Low
Credibility Average High Low
Generalizibility High Average Low
Objectivism High Average Average
Responisiveness Low High High

Summary. Conclusions. Discussion.
Verification of the model of evaluation regional programs including relevant theoretical and
practical aspects of assessing the impact of regional programs on the process of regional
development, was based on three independent evaluation designs, the procedures of which
have been adopted to theoretical and practical conditions of measuring the impact on regional
development process (subject of intervention) and carried out in real terms, exemplified by the
IROP and ROP programs with respect to the full context of their implementation.
The investigation has led to following achievements:
• Systemization and synthetization of issues regarding methodology of evaluation research
concerning specific interventions in regional development has enabled to place evaluation of
regional programs an applicable strand of research in socio-economic geography.
• Developed model for evaluation of regional programs is based on the concept of paradigm
of regional development and allows to apply regional science and policymakers’ perspectives
It serves a foundation to formulate requirements for more reliable evaluation studies.
• Paradigm of regional development is regarded overarching for scientific and practical
approach. It allows to combine contemporary views on regional development process as a
result of intended changes evoked mainly (though not entirely) by public intevention. The
components of paradigm evolve over time with changing values affecting perception of social
needs and various theories on how to effectively mobilise developmental factors. The concept
allows to measure cumulative effects of similar interventions, even if they were implemented
under formally different programs.
.

Summary. Conclusions. Discussion.
• The model extends the scope of standard evaluation of regional programs, because all
components of paradigm are examined with respect to their validity.
• The model respects the information needs of policymakers that refer to impact analysis of
intervention and assessment of intrisinc characteristics of intervention. The model seeks for
a balance between the strategic and operational requirements, when in real terms the
examination of intervention is given far more consideration.
• The model recognizes two basic needs i.e. deficits to be considered in the construct of
regional program: social and spatial needs of targeted groups and areas.
• The model considers that mechanisms of intervention are generally based on sound economic
theories of regional development, but they are very ecclectic and often take form of individual
constructs designed by policymakers.
• Verification of the model carried out with three various evaluation designes showcases their
different utility for purpose of evaluating regional programs. It singles out the theory-driven
evaluation characterized by the highest responsiveness in both cognitive and methodological
dimensions of analysis. This design corresponds best with the requirements of the model,
while the other give mixed results depending on dimension of analysis.
• It is recommended to re-orient the ToR’s evaluation questions towards identifying all
components of the model (especially the mechanism of exerting an impact by intervention),
while putting less politcal pressure on the description of outcomes. Strategic ex-post
evaluations should be carried out less frequently than they currently are, allowing to consider
longer time-series.

by Anna Borowczak
wuania@poczta.onet.pl

Application of various designs in evaluation of regional programs

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to Application of various designs in evaluation of regional programs

Similar to Application of various designs in evaluation of regional programs (20)

Recently uploaded

Recently uploaded (20)

Application of various designs in evaluation of regional programs