Assignment Writing Service http://StudyHub.vip/A-Review-Of-Program-And-Project-Evaluat 👈

1. Measuring Business Excellence
A review of program and proj ect evaluation models
Roberto Linzalone Giovanni Schiuma
Article information:
To cite this document:
Roberto Linzalone Giovanni Schiuma , (2015),"A review of program and project evaluation models", Measuring Business
Excellence, Vol. 19 Iss 3 pp. 90 - 99
Permanent link to this document:
http://dx.doi.org/10.1108/MBE-04-2015-0024
Downloaded on: 28 October 2015, At: 02:11 (PT)
References: this document contains references to 47 other documents.
To copy this document: permissions@emeraldinsight.com
The fulltext of this document has been downloaded 447 times since 2015*
Users who downloaded this article also downloaded:
Paolo Canonico, Ernesto De Nito, Vincenza Esposito, Marcello Martinez, Lorenzo Mercurio, Mario Pezzillo iacono,
(2015),"The boundaries of a performance management system between learning and control", Measuring Business
Excellence, Vol. 19 Iss 3 pp. 7-21 http://dx.doi.org/10.1108/MBE-04-2015-0021
Asbjørn Rolstadås, Iris Tommelein, Per Morten Schiefloe, Glenn Ballard, (2014),"Understanding project success through
analysis of project management approach", International Journal of Managing Projects in Business, Vol. 7 Iss 4 pp. 638-660
http://dx.doi.org/10.1108/IJMPB-09-2013-0048
Davide Aloini, Luisa Pellegrini, Valentina Lazzarotti, Raffaella Manzini, (2015),"Technological strategy, open innovation and
innovation performance: evidences on the basis of a structural-equation-model approach", Measuring Business Excellence,
Vol. 19 Iss 3 pp. 22-41 http://dx.doi.org/10.1108/MBE-04-2015-0018
Access to this document was granted through an Emerald subscription provided by editormbe
For Authors
If you would like to write for this, or any other Emerald publication, then please use our Emerald for Authors service
information about how to choose which publication to write for and submission guidelines are available for all. Please visit
www.emeraldinsight.com/ authors for more information.
About Emerald www.emeraldinsight.com
Emerald is a global publisher linking research and practice to the benefit of society. The company manages a portfolio of
more than 290 j ournals and over 2,350 books and book series volumes, as well as providing an extensive range of online
products and additional customer resources and services.
Emerald is both COUNTER 4 and TRANSFER compliant. The organization is a partner of the Committee on Publication
Ethics (COPE) and also works with Portico and the LOCKSS initiative for digital archive preservation.
*Related content and download information correct at time of download.
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

2. A review of program and project
evaluation models
Roberto Linzalone and Giovanni Schiuma
Roberto Linzalone is
based at the Department
of Mathematics,
Computer Sciences and
Economics, University of
Basilicata, Potenza, Italy,
and Institute of
Knowledge Asset
Management, IKAM
Centre for Education and
Research, Matera, Italy.
Giovanni Schiuma is
Professor at the
Department of
Mathematics, Computer
Sciences and Economics,
University of Basilicata,
Potenza, Italy, and
Innovation Insights Hub,
University of the Arts
London, Central Saint
Martins, London, UK.
Summary
Purpose – This paper aims to review Program and Project evaluation Models. The assessment of the
Evaluation Model (metaevaluation) is a critical step in Evaluation, as it is at the basis of a successful
Program/Project evaluation. A wide and effective review of EMs is a basic, as well as fundamental,
support in meta-evaluation that affects positively the overall evaluation efficacy and efficiency. Despite
a large number of reviews of EMs and a numerous population of EMs, developed in heterogeneous
projects and programs settings, the literature lacks comprehensive collections and reviews of EMs that
this paper addresses to provide a basis for the assessment of EMs.
Design/methodology/approach – Through a systematic literature review carried out via the Internet,
and querying search engines, several models addressing program or project evaluation have been
identified and analyzed. Following a process of normalization of the results gathered, they have been
analyzed and compared according to key descriptive issues. They have been, at the end, summarized
and rationalized in a comprehensive frame.
Findings – In recent years, evaluation studies have focused on the explanation of the mechanisms that
underlie the transformation of projects’ and programs’ outputs into socio-economic effects, arguing that
making them explicit allows to understand why a project or program is successful, as well as evaluating
its extent. To assess and explain program’s and project’s effects, a basic, although fundamental, role in
evaluation is played by the EM. A wide and heterogeneous set of 57 EMs has been identified, defined
and framed in typologies, according to a systematic review research.
Originality/value – The approach to the review of EMs and the definition of a boundary of interest for
management and economic researchers and practitioners represent an original issue of this paper.
Keywords Stakeholders, Model, Project evaluation
Paper type Research paper
1. Introduction
In the past decades, the project management (PM) approach allowed to increase project
and program efficacy in all fields. The research debate on PM has continued to be
passionate, while more and more companies and public administrations have managed
their projects by applying PM’s models and techniques.
The basic rationale of the PM approach is the delivery of the agreed outcomes of a project
in terms of time, cost and quality performances (Archibald, 2004; PMI, 2004). Although
effective for the management and achievement of project’s outcomes, the PM approach
does not allow to consider all the effects that occur along the time, because of the project
achievement.
Nowadays, more and more private and public organizations embark on projects and
programs with the aim of producing some effects and changes, in a target environment, on
a target beneficiary or on a stakeholder group. These effects and changes involve a variety
of dimensions and a number of variables, qualitative and quantitative, that make their
evaluation and management a complex, even though strategic, subject.
Received 27 April 2015
Revised 27 April 2015
Accepted 20 May 2015
PAGE 90 MEASURING BUSINESS EXCELLENCE VOL. 19 NO. 3 2015, pp. 90-99, © Emerald Group Publishing Limited, ISSN 1368-3047 DOI 10.1108/MBE-04-2015-0024
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

3. Therefore, if, on the one hand, PM has largely contributed to the development of effective
theories and models, enabling the effective achievement of projects’ outputs, on the other
hand, evaluation models (EMs) are becoming more and more attractive, as they enable the
evaluation of the effects of a project, both intended and unintended.
Looking at the literature on EMs, there is a large number of reviews which describe EMs
through different foci of analysis (e.g. theoretical stream, nature, approach, context, scope)
(Chen, 2005; Shadish et al., 1991, Rossi et al., 2003). However, there are a few literature
reviews resulting in an effective collection and identification of the EMs frameworks.
Building on a systematic literature review, this study presents a comprehensive analysis of
the existing frameworks of EMs.
The paper is organized as follows. Section 2 provides definitions of program, project and
their effects; Section 3 deals with the evaluation of programs and projects; Section 4
analyses the existent EMs and highlights the related research gaps; Section 5 reports the
methodology of the research; Section 6 presents the results and the findings, while
Section 7 describes the conclusions.
2. Program, Project, Effects
Several definitions of “project” can be found in the management literature. Each of them
highlights a peculiar aspect of it. While the Project Management Institute (PMI) states that
a “project is a temporary initiative undertaken to create a unique product or service” (PMI,
2004), Bowen (1996) argues that a project is a unique set of activities designed to produce
a definite result, with a clear start and end date, and a clear allocation of resources. Another
notable scholar, defines a project as a complex accomplishment, unique and of limited
duration, aimed at achieving a clear and agreed goal by a continuous process of planning
and control of different resources and interdependent constraints of time-cost-quality
(Archibald, 2004). Common and agreed characteristics of a project are clear and specific
goals, shared and clear project’s purpose and no technical ambiguity about the output to
undertake (Archibald, 2004; Cantamessa et al., 2007).
Regarding the concept of program, PMI (2008) defines it as a group of related projects
managed in a coordinated manner to achieve a level of benefits and a level of control that
would not be possible by managing them individually. Therefore, the program is an
elementary part of a complex strategic initiative, whereas the project is an elementary part
of a program.
Projects’ and programs’ results can be distinguished in outputs, outcomes and impacts
(WK Kellogg Foundation, 1998, 2004; Shapiro, 1996; Rossi et al., 2003). Outputs are the
products and/or services carried out from the project implementation. Outcomes and
impacts are both effects of the output that are observable along the time in the project
environment or on stakeholders. Outcomes are the specific changes in behavior,
knowledge, skills and the state and level of the activity/operation of the project target (i.e.
participants, beneficiaries, companies, processes, etc.). Outcomes reveal in the short-term
(from 1 to 3 years) or in the long-term (over a period of 4-6 years). Impact is the fundamental
change, wanted or unwanted, intended or unintended, that occurs in organizations,
communities or systems as a result of a project (it reveals in the long-term, within 7-10
years) (Kellogg Foundation, 2004). A “cause–effect” relation regulates the mechanism of
creation of outcomes and impacts, whose structure can be linear or systemic (complex).
Referring to the concepts of outputs, outcomes and impacts, programs differ from projects
because programs are focused on the consequences (outcomes) instead of results
(outputs) (Office of Government Commerce, 2005). Moreover Cantamessa et al. (2007)
argue that a project is usually linear in producing effects, while a program, has not linear
relations mechanism between outputs and effects.
VOL. 19 NO. 3 2015 MEASURING BUSINESS EXCELLENCE PAGE 91
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

4. 3. Program and project evaluation
Although there are documented evaluations of human interventions dating back to 2.200
BC (Shadish et al., 1991), the issue of project and program evaluation became especially
important in the USA in the 1960s, during the period of the social programs known as Great
Society, launched by Kennedy’s and Johnson’s administrations. Extraordinary public
investment in social programs was financed, but the impact of those investments remained
largely unknown.
With particular regard to projects and programs, evaluation is the assessment and the
analysis of the effectiveness of an activity; it involves the formulation of judgments about the
impact and progress. Evaluation is the comparison of the actual effects of a project, against
the agreed planned ones. Evaluation looks at what is planned to do, what has been
achieved and how it has been achieved (Shapiro, 1996; Archibald, 2012).
In a wider perspective, evaluation is a family of research methods that are used to
systematically investigate the effectiveness of policies, programs, projects and other types
of social intervention, with the aim of achieving improvement in the social, economic and
everyday conditions of people’s lives (Government Social Research Unit, 2007).
Many definitions of evaluation are linked to the concept of improvement, as evaluation
enables the review of policy and program development. Kahan and Goodstadt (2005)
conceive evaluation as a set of research questions and methods properly articulated to
review processes, activities and strategies, with the aim of achieving better results. So the
purpose of an evaluation is not just to find out what happened, but to use the information
to make the project better.
Evaluation of projects and programs can be implemented along the whole duration,
according to three stages: ex-ante, interim and ex-post evaluation. The first stage is aimed
to compare, select and finance alternative projects. The second one is aimed to improve
the strategy or the processes. The third stage is aimed to take lessons, insights, judgment
and awareness about taken decisions and projects.
A fundamental role in performing the projects and programs evaluation is played by EMs.
4. Programs’ and projects’ EMs
The terms approach and model, referred to evaluation, are used in an alternative way,
although there are some differences in meaning. Evaluation approach is the method, the
mental attitude or the particular perspective by which the evaluation is gathered. The model
represents the theoretical reconstruction or simulation of an abstract object, system or
concept, which describes more or less the structure or function of the subject to represent.
A model assists in the prediction of events, according to natural laws (social, economic,
organizational, etc.), and in increasing the understanding of phenomena.
EMs are approaches that assist evaluators in designing and carrying out useful, defensible
program evaluations (Stufflebeam and Shinkfield, 2007).
Many evaluation approaches and models exist. The literature suggests that no one
approach is best for all situations. Rather, the best approach varies according to factors
such as fit with basic values, the intent of the evaluation, the nature of key stakeholders and
the available resources. In addition, it is not necessary to stick strictly to one approach:
evaluations might quite reasonably combine elements of different approaches or adapt to
local conditions (Rogers and Fraser, 2003).
A variety of classifications of EMs can be found in the literature. Kahan (2008) distinguishes
EM in: goal based, goal free, theory based (logic model), utilization, collaborative,
balanced scorecard; appreciative inquiry; and external context, input, process, product
(CIPP). Stufflebeam and Shinkfield (2007) argue that EMs can be classified according to
the following three major types, with respective sub-types:
PAGE 92 MEASURING BUSINESS EXCELLENCE VOL. 19 NO. 3 2015
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

5. 1. Questions and/or methods-oriented:
 Questions-oriented; and
 Methods-oriented.
2. Improvement/accountability-oriented:
 Decision/accountability-oriented;
 Consumer-oriented; and
 Accreditation/certification.
3. Social agenda/advocacy approaches:
 Client-centered studies;
 Constructivist evaluation;
 Deliberative democratic evaluation; and
 Utilization-focused evaluation.
Hentschel (1999) proposes four main types of EMs, according to qualitative or quantitative
nature of the data and method of evaluation. This distinction gives rise to the following four
types of EMs: subjective welfare, standard household survey, ethnography and
econometric anthropology.
The adoption of an assessment model is required to represent and analyze the
mechanisms of the system, and allow an explicit analysis through analysis of the individual
components of the program (Dyehouse et al., 2009; Chen, 2005).
This is remarked by Chen (2005) who distinguishes an input–output or “black box”
evaluation and a white box evaluation. The first approach to evaluation does not consider
any model of evaluation: things go in and things come out. It is useful in identifying program
merits, but cannot capture the “transformation processes that turn interventions into
outcomes” (p. 231), and thus, evaluation findings lack robust explanatory power.
What is needed is the evaluation that considers the details of what occurs in “the black
box”. The function of the EM is to make clearer the system and allows for more explicit
analysis of the program through analysis of the components of the system, which is the
promise of a “white box” approach. Furthermore, this type of analysis of the inner
components and the logic of the system can enable the needed analyses, leading to
improvement of theoretical models (Dyehouse et al., 2009).
Despite a large number of classifications of EMs, that reflect theoretical dissertations, as
well as a numerous population of EMs developed in heterogeneous projects and programs
settings, the literature lacks comprehensive review of EMs. To overcome this limitation, a
review of EMs has been carried out.
5. Methodology
The research methodology used reflects the pattern of a systematic literature review
(Tranfield et al., 2003). According to it, the following five main phases characterize the
research:
1. planning the review;
2. identifying and evaluating studies;
3. extracting and synthesizing data;
4. reporting; and
5. utilizing the findings.
VOL. 19 NO. 3 2015 MEASURING BUSINESS EXCELLENCE PAGE 93
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

6. 5.1 Planning the review
As a first step, a review committee composed of two members has been set up. The
committee has mapped the field of study by identifying potential areas/sectors and players,
which use and apply evaluations models in projects and programs. Subsequently, a
double-stage sampling procedure was used to identify and retrieve EMs according to a
defined review protocol.
5.2 Identifying and evaluating studies
In the first stage, a broad scan of potential EMs has been carried out, through systematic
searches of application of EMs to programs and projects in many settings. Primary and
secondary data were collected.
Search terms used to identify potential case examples of program/project evaluations
included, but were not limited to, EM, program evaluation, project evaluation and “name”
of the EM (once identified). Searches were conducted using simple search methods as well
as Boolean operators (Reed and Baxter, 2009). The appearance of these terms was
searched using the Web search engines.
The sources collected were: scientific articles, reports on public and private projects,
websites of: associations, foundations, organizations of the third sector, research
institutions, national government agencies, government bodies, and development
agencies (of local, national and international levels).
The aim of this first and broad scan has been to identify the EMs used to evaluate
projects and programs with their main identification data. The review has been
organized according to three main steps: scanning, normalization and systematization
of results.
To be comprehensive and reduce the number of false-negative findings (i.e. relevant
sources not identified), the sampling was not limited only to evaluation-related articles
and/or Web sites (Miller and Campbell, 2006) but also included substantive sources in
areas where evaluations of programs and projects might also be found (i.e. urban
development, environment). Most of the sources were pertaining to the following settings:
education, health, social development, research, development, military and guardianship,
business, services and urban planning.
This stage yielded an initial population of 75 potential EMs that self-identified as being
project and/or program EMs.
The identification of the EMs was followed by the normalization of the data collected.
Each identified model has been analyzed and described according to a standard
sampling template, consisting of a data abstraction form (Miller and Campbell, 2006),
that is a card structured with the following fields: name of model, source, nature
(qualitative vs quantitative), field of application, methodology, pros and cons and
reference. The form’s schema was derived from the focal research questions and
developed during the final stages of the sampling process. It predominately consisted
of fixed items that were open-ended and were low inference items (requiring little
judgment).
All sampled EMs’ data were, therefore, normalized, that is decrypted according to the
form. The procedure of normalization acted also as a means of calibration, by which
each reviewer worked independently on a small subsample of cases without areas of
ambiguity (Wilson, 2009). After the calibration procedure, each model was analyzed.
5.3 Extracting and synthesizing data
After the first scan of EMs, a screening was carried out that aimed to select and discard the
results not pertaining to the socio-economic programs and projects. All 75 potential and
PAGE 94 MEASURING BUSINESS EXCELLENCE VOL. 19 NO. 3 2015
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

7. Table I Summary table of program/project EMs review
Typology EM Source Nature
Peer review (PR) Direct PR
Modified direct PR
Ancillary PR
Traditional PR
Indirect PR
Pre-emptive PR
Various authors from scientific
literature (most dated source:
Royal Society of Edinburgh,
1731)
Qualitative
Case study (CS) Prospective CS
Retrospective CS
Le Play (1829) Quali-quantitative
Technological forecasting Scenario planning Kahn (1950) Qualitative
Cross-impact matrices (or inter-dependency
matrices)
Gordon and Hayward (1968) Quali-quantitative
Morphological analysis Zwicky (1967) Qualitative
Financial methods Internal rate of return
Net present value
Payback period
Value-based management
literature
Quantitative/financial
Binomial option Pricing model
Trinomial option Pricing model
Cox et al. (1979) Quantitative
Economic-based methods Cost-benefit/Cost-effectiveness analysis Economic literature (most
dated source: Dupuit, 1844)
Quantitative/financial
Social accounting matrix Stone and Brown (1962) Quantitative
Experimental economics
Data
Instrumental variables
Computational methods
Structural econometrics
Economic literature Quantitative
Contingent valuation Ciriacy-Wantrup (1947) Quantitative
Technological-based methods Technology assessment
Technology dynamics
Technology forecasting
Technology management
literature
Ex-ante, quantitative
Narrative methods Storytelling Social sciences Qualitative
Impact narratives Social literature Qualitative
Most significant change Davies (1996) Qualitative
Ethnographic methods Ethnographic evaluation Social sciences Qualitative
Behavioral methods Outcome mapping International Development
Research Centre – Evaluation
Unit, 2001
Qualitative
Scoring methods Analytic hierarchy process Saaty, 1970s Quantitative
Earned value analysis/management US Department of Defense,
1960’s
Quantitative
Program assessment rating tool US Office of Management,
2002
Quantitative
Key performance indicators Management literature Quantitative
Scorecard methods Balanced scorecard Kaplan and Norton (1992) Quali-quantitative
Performance prism Neely et al. (2002) Quali-quantitative
Bibliometric methods Main science and technology indicators OECD (2013) Quantitative
World Bank (undated)
Pathways analysis Participatory impact pathways analysis
CPM/PERT
Critical path method/program evaluation and
review technique
Challenge program on water
and food
Qualitative
Catalytic Construction
Company, 1957
Quantitative
Booz, Allen and Hamilton,
Inc., 1958
Logic model/framework Logical framework approach Rosenberg (1969) Qualitative
Kellogg’s logic model Quigley M., for W.K. Kellogg
Foundation (1998)
Quali-Quantitative
CIPP evaluation framework Stufflebeam et al. (1966) Qualitative
Weaver’s triangle Weaver, undated Qualitative
TQM approach Malcom Baldridge Award/Model Malcolm Baldrige national
quality improvement
Quali-Quantitative
European Foundation for Quality
Management excellence model
European Foundation for
quality management
Quali-quantitative
(continued)
VOL. 19 NO. 3 2015 MEASURING BUSINESS EXCELLENCE PAGE 95
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

8. self-identified EMs were scrutinized to determine whether they met inclusion criteria. To be
considered for inclusion, the EMs had to:

not replicate other EMs with a different name; and

not be a personalization/derivation of other EMs.
EMs not meeting these criteria were excluded from the sample. Two reviewers
independently screened the 75 models to determine whether they met the inclusion criteria.
The comparison of the EMs, carried out with systematic comparison of the forms, revealed
cases of variations and “personalization” of the same original model. These variants were
discarded, leaving only the original model.
Consensus between both reviewers verified their inclusion or exclusion. The final sample
obtained from this procedure yielded 57 EMs[1].
6. Reporting results and findings
The EMs included in the final sample were subjected to a further comparison to recognize
and group models that share a common and predominant element (i.e. approach, the field
of application, the subject, methodology, etc.).
The results obtained have been summarized and reported in Table I. The collection
resulted in many models; some of them have common subjects, context/field of application,
evaluation or measure approach. These commonalities allowed to group them in
typologies.
The models have a different responsiveness and effectiveness to the evaluative needs of
programs and projects. The proposed classification of the EMs in Typologies helps identify
and associate with each model the main characteristic evaluation.
Quantitative models provide analytical relations between the elements, enabling the
evaluation reporting and the planning of achievable effects. The qualitative models support
and enable the construction of the program theory. EMs, according to the final results of the
review, appears as a wide spectrum of items differentiated by strategic, contextual,
methodological variables: what dimension is evaluated, what is the purpose of evaluation
purpose and how the evaluation is conducted.
Table I
Typology EM Source Nature
Strategic SWOT analysis Humphrey (1967) Qualitative
Strategy map Kaplan and Norton (1996) Qualitative
Critical success factor Daniel (1961) Qualitative
Rockart (1979)
Breakdown/tree structures Work breakdown structure US Department of Defense,
1957
Qualitative
Cost breakdown structure Management literature Quantitative
Problem tree analysis System analysis literature Qualitative
Statistical Six sigma Motorola (1981) Quali-quantitative
Multicriteria analysis Multicriteria decision analysis Early 1960s Quantitative
Impact assessment Environmental impact assessment 1960s – National
Environmental Policy Act,
USA, 1969
Quantitative
Social impact assessment 1960s – National
Environmental Policy Act, USA
1969
Quali-quantitative
Notes: CPM ⫽ critical path method; PERT ⫽ program evaluation and review technique; TQM ⫽ total quality management
PAGE 96 MEASURING BUSINESS EXCELLENCE VOL. 19 NO. 3 2015
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

9. 7. Conclusions
Effectiveness and efficiency of the evaluation are increasingly important for programs and
projects. In such a perspective, the assessment of EMs (meta-evaluation) is a key and
critical activity. The assessment and selection of the EMs is crucial for many different
purposes: finance of a project, assessment of a project’s efficacy and improvement of a
program’s performance. Any project’s/program’s stakeholder should consider EMs
characteristics as a rationale for an effective and efficient evaluation process. In fact, the
choice of an EM not consistent with the setting of the project/programs parameters can
determine distorted assessments and not defensible evaluation inefficiency. In short terms,
the choice of an EM influences the goodness of the whole projects/programs evaluation.
Based on a rigorous research approach, this paper offers a comprehensive overview of the
existing EMs that can support the decision-making process regarding the meta-evaluation
of programs and projects.
The paper represents a first step of a wider research aimed to understand how much
evaluation can benefit from meta-evaluation.
Note
1. The sources of the 57 EMs included in this review are available upon request.
References
Archibald, R. (2012), Project Management La Gestione di Progetti e Programmi Complessi, Franco
Angeli, Milano.
Archibald, R.D. (2004), Project Management La Gestione di Progetti e Programmi Complessi, Franco
Angeli, Milano.
Bowen, H.K. (1996), Project Management Manual, Harvard Business School Background Note
697-034, Boston, MA.
Cantamessa, M., Cobos, E. and Rafele, C. (2007), Il Project Managemen un Approccio Aistemico Alla
Gestione dei Progetti, Isedi, Torino.
Chen, H. (2005), Practical Program Evaluation: Assessing and Improving Planning, Implementation,
and Effectiveness, Sage Publications, Thousand Oaks, CA.
Ciriacy-Wantrup, S.V. (1947), “Capital returns from soil-conservation practices”, Journal of Farm
Economics, Vol. 29, pp. 1181-1196.
Cox, J.C., Ross, S.A. and Rubinstein, M. (1979), “Option pricing: a simplified approach”, Journal of
Financial Economics, Vol. 7 No. 3, pp. 229-263.
Daniel, D.R. (1961), “Management information crisis”, Harvard Business Review, Vol. 39 No. 5,
pp. 111-121.
Davies, R.J. (1996), “An evolutionary approach to facilitating organisational learning: an experiment by
the Christian commission for development in Bangladesh”, available at: www.mande.co.uk/docs/ccdb.
htm (accessed 26 September 2014).
Dupuit, A.J.E.J. (1844), “De la mesure de l’utilité des travaux publics”, Annales des ponts et
chaussées, Vol. 8.
Dyehouse, M., Bennett, D., Harbor, J., Childress, A. and Dark, M. (2009), “A comparison of linear and
systems thinking approaches for program evaluation illustrated using the Indiana interdisciplinary
GK-12”, Evaluation and Program Planning, Vol. 32 No. 3, pp. 187-196.
European Commission (1993), Project Cycle Management, An Integrated Approach, European
Commission, Brussels.
Farell, K.M., Kratzmann, S., McWilliam, N., Robinson, N., Saunders, S., Ticknor, J. and White, K. (2002),
“Evaluation made very easy accessible, and logical”, Atlantic Centre of Excellence for Women’s Health
(ACEWH), available at: www.acewh.dal.ca/eng/reports/EVAL.pdf (accessed 12 November 2014).
Gordon, T.J. and Hayward, H. (1968), “Initial experiments with the cross-impact matrix method of
forecasting”, Futures, Vol. 1, pp. 100-116.
VOL. 19 NO. 3 2015 MEASURING BUSINESS EXCELLENCE PAGE 97
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

10. Government Social Research Unit (2007), “Why do social experiments? Experiments and
quasi-experiments for evaluating government policies and programmes”, in HM Treasury, The
Magenta Book: Guidance Notes for Policy Evaluation and Analysis, HM Treasury, London.
Hentschel, J. (1999), “Contextuality and data collection methods: a framework and application to
health service utilization”, Journal of Development Studies, Vol. 35 No. 4, pp. 64-94.
Kahan, B. (2008), “Excerpts from review of evaluation frameworks”, Saskatchewan Ministry of
Education, available at: http://idmbestpractices.ca/pdf/evaluation-frameworks-review.pdf (accessed
24 September 2014).
Kahan, B. and Goodstadt, M. (2005), The IDM Manual Basics – 3rd Edition Centre for Health Promotion,
University of Toronto, Toronto.
Kaplan, R.S. and Norton, D.P. (1966), The Balanced Scorecard: Translating Strategy into Action,
Harvard Business School Press, Boston, MA.
Kaplan, R.S. and Norton, D.P. (1992), The Balanced Scorecard: Measures that Drive Performance,
Harvard Business Review, January/February, pp. 71-79.
Kellogg Foundation (1998), “Evaluation handbook”, available at: www.wkkf.org (accessed November
2013).
Kellogg Foundation (2004), “Logic model development guide”, available at: www.wkkf.org (accessed
January 2014).
Miller, R.L. and Campbell, R. (2006), “Taking stock of empowerment evaluation: an empirical review”,
American Journal of Evaluation, Vol. 27 No. 3, pp. 296-319.
Neely, A.D., Adams, C. and Kennerley, M. (2002), The Performance Prism: The Scorecard for
Measuring and Managing Stakeholder Relationships, Financial Times/Prentice Hall, London.
OECD (2013), Main Science and Technology Indicators, OECD Publishing, Paris, Vol. 2013 No. 1.
Office of Government Commerce (2005), Managing Successful Programmes: Delivering Business
Change in Multi-Project Environments, HM Stationary Office, London.
Project Management Institute (2004), A Guide to the Project Management Body of Knowledge, 3rd ed.,
Project Management Institute, Newton Square, PA.
Project Management Institute (2008), The Standard for Program Management, 2nd ed., Project
Management Institute, Newton Square, PA.
Reed, J.G. and Baxter, P.M. (2009), “Using reference databases”, in Cooper, H., Hedges, L.V. and
Valentine, J.C. (Eds), The Handbook of Research Synthesis and Meta-analysis, Russell Sage
Foundation, New York, NY, pp. 73-102.
Rockart, J.F. (1979), “Chief executives define their own data needs”, Harvard Business Review, Vol. 57
No. 2, pp. 81-93.
Rogers, P.J. and Fraser, D. (2003), “Appreciating appreciative inquiry”, in Preskill, H. and
Coghlan, A.T. (Eds), Using Appreciative Inquiry in Evaluation, Jossey-Bass, San Francisco, CA,
pp. 75-84.
Rosenberg, L. (1969), for the US Agency for International Development (USAID).
Rossi, P.H., Lipsey, M.W. and Freeman, H.E. (2003), Evaluation: A Systematic Approach, Sage
publications, Thousand Oaks, CA.
Shackman, G. (2012), What Is Program Evaluation: A Beginner’s Guide, The Global Social Change
Research Project, Albany, NY.
Shadish, W.R., Cook, T.D. and Leviton, L.C. (1991), Foundations of Program Evaluation: Theories of
Practice, Sage, Newbury Park, CA.
Shapiro, J. (1996), “Monitoring and evaluation”, available at: www.civicus.org/new/media/Monitoring
%20and%20Evaluation.pdf (accessed 24 April 2014).
Stone, R. and Brown, A. (1962), A Computable Model for Economic Growth, Cambridge Growth
Project, Cambridge.
Stufflebeam, D.L. (1966), “A depth study of the evaluation requirement”, Theory Into Practice, Vol. 5
No. 1, pp. 121-134.
PAGE 98 MEASURING BUSINESS EXCELLENCE VOL. 19 NO. 3 2015
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)

11. Stufflebeam, D.L. and Shinkfield, A.J. (2007), Evaluation Theory, Models, and Applications, Jossey-
Bass, San Francisco, CA.
Tranfield, D., Denyer, D. and Smart, P. (2003), “Towards a methodology for developing evidence-
informed management knowledge by means of systematic review”, British Journal of Management,
Vol. 14 No. 3, pp. 207-222.
Wilson, D.B. (2009), “Systematic coding”, in Cooper, H., Hedges, L.V. and Valentine, J.C. (Eds), The
Handbook of Research Synthesis and Meta-analysis, Russell Sage Foundation, New York, NY,
pp. 159-176.
Zwicky, F. (1967), in Zwicky, F. and Wilson, A. (Eds), New Methods of Thought and Procedure:
Contributions to the Symposium on Methodologies, Springer, Berlin.
Further reading
Coryn, C.L.S., Noakes, L.A., Westine, C.D. and Schroter, D.C. (2011), “A systematic review of
theory-driven evaluation practice from 1990 to 2009”, American Journal of Evaluation, Vol. 32 No. 2,
pp. 199-226.
Darabi, A. (2002), “Teaching program evaluation: using a systems approach”, American Journal of
Evaluation, Vol. 23 No. 2, pp. 219-228.
Office of Planning, Research and Evaluation (OPRE), Administration for Children and Families (ACF),
US Department of Health & Human Services (HHS) (2010), “What is program evaluation?”, in OPRE
(Ed.), The Program Manager’s Guide to Evaluation, Office of Planning, Research and Evaluation
(OPRE), Washington, DC, pp. 6-12.
Rossi, M. (2004), I Progetti di Sviluppo, Franco Angeli, Milano.
Scriven, M. (2007), “Key evaluation checklist”, available at: www.wmich.edu/evalctr/checklists
(accessed 8 march 2014).
About the authors
Dr Roberto Linzalone is Senior Research Fellow in Management Engineering at the
University of Basilicata. His research interests focus on knowledge management, new
product development, project management and evaluation. Roberto received his degree in
Civil Engineering from the University of Basilicata and PhD in Advanced Systems of
Production from the Polytechnic University of Bari. He is a regular speaker at national and
international conferences and has authored and co-authored about 30 publications,
including books, articles, research reports and working papers. Roberto Linzalone is the
corresponding author and can be contacted at: roberto.linzalone@unibas.it
Dr Giovanni Schiuma is Professor of Arts-based Management and Director of the
Innovation Insights Hub at the University of the Arts London. He is widely recognized as one
of the world’s leading experts in the arts in business and strategic knowledge management.
He is an inspiring speaker and facilitator, with extensive research management expertise
and excellent ability to coordinate complex projects and lead research teams. He is
creative and innovative, with international mind-set and openness to address and solve key
strategic research and organizational challenges. Giovanni is a leading international expert
on knowledge management and intellectual capital strategy and widely renowned for his
work on the use of the arts for business, as well as his work on assessing and managing
knowledge assets. He Chairs the International Forum on Knowledge Assets Dynamics.
For instructions on how to order reprints of this article, please visit our website:
www.emeraldgrouppublishing.com/licensing/reprints.htm
Or contact us for further details: permissions@emeraldinsight.com
VOL. 19 NO. 3 2015 MEASURING BUSINESS EXCELLENCE PAGE 99
Downloaded
by
Jos
van
Iwaarden
At
02:11
28
October
2015
(PT)