This paper aims to explore the interpretative value of macro-level indicators to better understand the relationships between the research performance and the environmental performance of nations. Using bibliometric tools developed by Science-Metrix as proxy indicators of environmental research performance of countries and the Environmental Performance Index (EPI) published jointly by Yale University and Columbia University, the authors present results and discuss the advantages and limitations of the use of such a macro-level approach to inform research evaluation process.

1. Does research performance influence
environment‐related outcomes of countries?
Lessons learned from a macro‐level evaluation using bibliometric
indicators and environmental performance indexes
Edmonton 2011 | CES Conference
Concurrent Session #6 A: Performance measurement and beyond | Room: Turner Valley
Wednesday, May 4, 2011 | 9:15 AM to 10:45 AM

2. Outline
 Background
 Need for a composite index of scientific performance
 Use of composite indexes for macro‐evaluation of national outcomes
 Environmental Performance Index (EPI)
 Objectives
 Develop a composite scientometric index
 Apply to macro‐evaluation
 Drive research
 Methods
 Composite Index of Scientific Performance (CISP)
 Relationship between the CISP and the EPI
 Preliminary results
 Next steps
2

3. Background: The challenge
How can evaluators investigate the impact of scientific
research on the environmental performance of countries?
 Macro‐indicators are often used to investigate the influence of economic and
non‐economic factors on the environmental performance of nations.
 The Environmental Performance Index (EPI) is commonly used for this.
 Literature shows examples of the influence of economic and non‐economic
factors on the environmental performance of nations.
 However, the role of national research performance, as a determinant of the
national environmental performance, has not been fully investigated.
 We need a new composite index that takes scientific research into account.
 We also, and primarily, need an approach to investigate the relationship
between scientific research and the environmental performance of nations.
3

4. Background: Need and opportunities
 Need for a composite index of scientific performance
 Multi‐criteria analysis is a synthesis tool used in scientometrics to inform
the decision‐making process in the science policy context
 When several dimensions characterizing the scientific performance of
countries are being measured for comparative purposes, it is often
difficult to determine the position of the countries being compared
relative to one another (i.e., A performs better than B or vice‐versa)
without a well‐structured ranking mechanism.
 Various methods have been developed to reduce numerous indicators to
a single composite indicator or multi‐rank.
 However, these methods are often sensitive to the composition of the
study sample: the position of two entities relative to one another can be
altered if entities are added or removed from the sample.
 A “similarity‐based approach to ranking multi‐criteria alternatives” was
adapted to provide a stable composite indicator for ranking [Deng, 2007]
in a bibliometric context.
4

5. Background: Environmental Performance Index (EPI)
 Performance‐oriented composite index developed by Yale and
Columbia Universities.
 Formally released in Davos, at the annual meeting of the World
Economic Forum in January 2006. Revised in 2008 and 2010.
 Measures progress toward a set of targets of desirable environmental
outcomes, taking into account a country's current policies.
 Ranks 163 countries on 25 performance indicators tracked across 10
policy categories for both environmental public health and ecosystem
vitality objectives.
 All variables are normalized on a scale from 0 to 100. The maximum
value of 100 is attributed to the target, as the zero value is credited the
worst player in the field.
5

6. Background: EPI’s Framework
6

7. Background: EPI’s Advantages and Limitations
Advantages Limitations
One‐dimensional metric to facilitate cross‐ Absence of broadly‐collected and
country comparisons and analysis methodologically‐consistent data [Saisana &
[Emerson et al., 2010] Saltelli, 2010]
Fails to meet fundamental scientific
Unambiguous yardstick against which a
requirements with respect to the three
country’s development can be measured
central steps of indices formation:
and even a cross‐country comparison can
normalization, weighting, and aggregation
be performed [Böhringer & Jochem, 2008]
[Böhringer & Jochem, 2008]
Facilitates the identification of leaders and Utilizes the best available global datasets on
laggards, highlights best policy practices, environmental performance, but overall
and identifies priorities for action [Samimi data quality and availability alarmingly poor
et al., 2010] [Emerson et al., 2010]
Intuitive methodology, possibility to drill
down into specific issues, global Lack of time series, focus too narrow
coverage, full data access and transparency [Pillarosetti & van den Bergh, 2010]
[Srebotnjak, 2010]
7

8. Examples of the use of the EPI
(Relation between two indices/indicators)
Question and data used Main findings
Impact of improvements in environment quality as
a determinant of economic growth in developing • Impact of EPI on economic growth in the
countries [Samimi, Erami, and Mehnatfar, 2010] countries under consideration is positive
and significant.
Data: EPI and Economic Growth
Trade or cross‐border investment • No strong support to the Pollution Haven
flows as a determinant of environmental Hypothesis (i.e. migration of pollution‐
degradation [Chakraborty & Mukherjeeo, 2010] intensive industries to the developing
Data: Relations between the EPI and the share of a world), but showed relationships between
country in the global export market and Foreign socio‐economic and socio‐political factors
Direct Investment inflow and national environmental performance.
• Democracy in itself is not a sufficient
Governance and social development as a precondition for good environmental
determinant of environmental performance and policies
capacity for climate change adaptation [Foa, 2009] • Strong evidence that engagement in local
Data: EPI and EM‐DAT database, Worldwide community can help improve environmental
Governance Indicators and Indices of Social performance
Development • Positive effect of gender equity upon
environmental performance
8

9. Objectives
 Develop a Composite Index of Scientific Performance (CISP):
Apply methods and scientometrics to improve the multi‐criteria
analysis of scientific performance of nations
 Apply to macro‐evaluation: Investigate the relationship
between the scientific and environmental performances of
countries using the CISP and the EPI to support the macro‐
evaluation of research outcomes
 Drive research: Provide the basis for further exploration of the
interpretative value of macro‐level indicators by better
understanding the links between the environmental research
performance and environmental outcomes of nations
9

10. Methods: Approach Overview
DEVELOPMENT OF A COMPOSITE INDEX OF SCIENTIFIC PERFORMANCE APPLICATION TO THE MACRO-EVALUATION OF
IN ENVIRONMENT RESEARCH NATIONAL-LEVEL OUTCOMES
IDENTIFICATION OF COMPUTATION OF COMPUTATION
SCIENTIFIC SCALE‐FREE COMPOSITE INDEX
OF COMPOSITE
JOURNALS SCIENTOMETRIC OF SCIENTIFIC
INDEX OF
SPECIALIZED IN INDICATORS PERFORMANCE
SCIENTIFIC
ENVIRONMENT 4 scientific (CISP) STATISTICAL
PERFORMANCE
RESEARCH performance Rank normalized on ANALYSIS OF THE
IN
# journals: ~ 650 indicators scale of 0 to 100 RELATIONSHIP
ENVIRONMENT
RESEARCH BETWEEN THE TWO
PERFORMANCE
INDEXES
CREATION OF A
DATASET OF ENVIRONMENTAL
DELINATION OF SCIENTIFIC PAPERS PERFORMANCE
ENVIRONMENT BY COUNTRY INDEX (EPI)
RESEARCH IN Rank normalized on
SCOPUS DATABASE # countries: 38 scale of 0 to 100
25 performance
Period: 2003‐2007 Threshold: Min. indicators tracked AVENUES FOR
1000 scientific across 10 policy FURTHER
# papers: 434,793 papers for the five‐ categories RESEARCH
year period
10

11. Methods: Delineation of Scientific Research
 Included the journals used in a previous scientometric study completed
for Environment Canada:
 Bertrand F. and Côté G. 25 Years of Canadian Environmental
Research: A Scientometric Analysis (1980‐2004). March 2006.
Science‐Metrix.
Link: http://www.science‐metrix.com/pdf/SM_2006_001_EC_Scientometrics_Environment_Full_Report.pdf
 Identified additional environmental research journals using the Ontology
Explorer and the Ontology and Journal Classification
11

12. Methods: CISP – Computation of Indicators
 Composed of four scale‐free scientometric indicators :
 Scientific Productivity: Scientific papers published by a country in
environment research relative to the number expected given its total
gross expenditure in R&D (GERD).
 Scientific Impact: Citations received by a country relative to the
number expected given the number of papers published in
environment research.
 International Collaboration: Number of co‐authored papers with a
foreign partner relative to the number expected given the number of
papers published in the country in environment research. An
indicator of collaboration propensity(*).
 Specialization: Papers published by a country in environment
research relative to the number expected given its total scientific
production (in all fields of science).
(*) International collaboration is associated with scientific impact [Katz and Hicks 1997]
12

13. Methods: CISP ‐ Computation of the CISP
 Adapted the similarity‐based approach to ranking multi‐criteria to
compute a composite index
 Equal weighting: Gives equal weight to all four indicators
 Vectorial calculation: Involves vectorial calculations in 4 dimensions
given that 4 indicators are used
 Index based on the vectorial calculation of ideal performance: The
composite index for each country is determined based on its
similarity with ideal performance solution
 Normalized for comparison with EPI: CISP scores normalized
between 0 and 100
 Insensitive to countries included in the ranking: The position of two
countries relative to one another does not change if countries are
added or removed from the sample
13

14. Results: Mapping of CISP scores
14

15. Results: Mapping of EPI scores
15

16. Results: Mapping of CISP and EPI scores
16

17. Results: Relationship between CISP and EPI
EPI Score
EPI = 47.641 + 0.43294 * CISP
Pearson Correlation
r: 0.51515
p‐value: < 0.001
CISP Score
17

18. Results: Regression‐based ranking of indexes
Country CISP EPI Regression EPI/ Country CISP EPI Regression EPI/
line Regression line line Regression line
Switzerland 58.0 89.1 72.8 1.22 Russia 31.2 61.2 61.1 1.00
Sweden 55.7 86 71.8 1.20 Brazil 36.7 63.4 63.5 1.00
Japan 31.2 72.5 61.2 1.19 Israel 36.2 62.4 63.3 0.99
Austria 46.5 78.1 67.8 1.15 Poland 37.9 63.1 64.1 0.99
Singapore 32.0 69.6 61.5 1.13 Turkey 34.6 60.4 62.6 0.96
France 49.6 78.2 69.1 1.13 N. Zealand 67.5 73.4 76.8 0.96
Czech Rep. 38.9 71.6 64.5 1.11 Denmark 59.2 69.2 73.3 0.94
Italy 42.9 73.1 66.2 1.10 Rep. Korea 31.3 57 61.2 0.93
Hungary 37.4 69.1 63.8 1.08 Thailand 44.5 62.2 66.9 0.93
Norway 63.3 81.1 75.0 1.08 Argentina 43.3 61 66.4 0.92
Chile 47.0 73.3 68.0 1.08 Netherlands 57.1 66.4 72.4 0.92
Finland 52.2 74.7 70.2 1.06 USA 50.2 63.5 69.4 0.92
Germany 49.3 73.2 69.0 1.06 Australia 57.9 65.7 72.7 0.90
Portugal 52.2 73 70.2 1.04 Greece 48.4 60.9 68.6 0.89
Iran 24.7 60 58.3 1.03 Canada 63.3 66.4 75.0 0.88
Ireland 41.5 67.1 65.6 1.02 Belgium 51.4 58.1 69.9 0.83
Spain 50.5 70.6 69.5 1.02 India 30.5 48.3 60.9 0.79
Mexico 43.9 67.3 66.6 1.01 China 34.0 49 62.4 0.79
UK 60.0 74.2 73.6 1.01 South Africa 49.2 50.8 68.9 0.74
Outliers in red – two possible explanations:
1) EPI is overestimated or CISP is underestimated for outliers above 1.15. EPI is underestimated or CISP is overestimated for
outliers below 0.80.
2) Outliers reflect a real effect due to other factors that come into play. Given the correlation coefficient of 0.52 (mid‐range
between no correlation and perfectly correlated), this is not unlikely.
18

19. Next steps (1)
 Further validate and improve the CISP:
 Compute principal components and/or factor analysis to
validate the selection of scientometric indicators
 Adjust or change the set of scientometric indicators used in the
composite index
 Additional testing of the limits/errors of the approach
(composite indicator and ranking in a bibliometric context)
19

20. Next steps (2)
 Further explore the interpretative value of macro‐level
indicators:
 Delineate environmental research at the subfield level to align
with EPI policy categories indicators (e.g., air pollution,
fisheries, environmental health, etc.)
 Further explore the value of macro‐level indicators with
multiple sources of evidence (other data on national policies
and programs) to explain macro‐evaluation results
 Test the experimental design (including the limits/errors of the
EPI)
20

21. Thank you for your time and feedback
Frédéric Bertrand, M.Sc.
Vice‐President, Evaluation | Science‐Metrix
frederic.bertrand@science‐metrix.com
David Campbell, M.Sc.
Senior Research Analyst | Science‐Metrix
david.campbell@science‐metrix.com
Grégoire Côté, B.Sc.
Vice‐President, Bibliometrics| Science‐Metrix
gregoire.cote@science‐metrix.com
Science‐Metrix Inc.
Address 1335, Mont‐Royal E.
Michelle Picard‐Aitken, M.Sc.
Montreal, Quebec
Senior Research Analyst | Science‐Metrix
Canada H2J 1Y6
m.picard‐aitken@science‐metrix.com
Toll‐free 1.800.299.8061
Michèle‐Odile Geoffroy, M.Sc. Phone 1.514.495.6505
Scientific Writer| Independent Email info@science‐metrix.com
michele‐odile.geoffroy@videotron.ca www.science‐metrix.com
21

22. References (1)
• Bertrand, F. and Côté, G. (2006) 25 Years of Canadian Environmental Research: A Scientometric
Analysis (1980‐2004). Science‐Metrix: http://www.science‐
metrix.com/pdf/SM_2006_001_EC_Scientometrics_Environment_Full_Report.pdf
• Böringher, C. & Jochem, P. (2008). Measuring the Immeasurable: A Survey of Sustainability
Indices. Centre for European Economic Research, Germany. Discussion Paper No. 06‐073.
• Chakraborty, D., & Mukherjeeo, S. (2010). Relationship between Trade, Investment and
Environment: A Review of Issues. MPRA Paper of the University Library of Munich, Germany,
No. 23333. Retrieved from http://mpra.ub.uni‐muenchen.de/23333/1/Trade_and_Environment‐
16.6.2010.pdf
• Deng, H. 2008. A similarity‐Based Approach to Ranking Multicriteria Alternatives. LNCS: Lecture
Notes in Computer Science, 2007, Volume 4682/2007, 253‐262.
• Emerson, J., D. C. Esty, M.A. Levy, C.H. Kim, V. Mara, A. de Sherbinin, and T. Srebotnjak (2010).
2010 Environmental Performance Index. New Haven: Yale Center for Environmental Law and
Policy
• Environmental Performance Index 2010: http://epi.yale.edu/
• Foa, R. (2009). Social and Governance Dimensions of Climate Change: Implications for Policy.
Background Policy Research Working Paper to the 2010 World Development Report, No. 4939.
Retrieved from http://www.iadb.org/intal/intalcdi/PE/2009/03543.pdf
• Katz, J.S. and Hicks, D. 1997. How much is a collaboration worth? A calibrated bibliometric
model. Scientometrics. 40 (3): 541‐554.
22

23. References (2)
• Pillarisetti, J. R., & van den Bergh, J. C. M. (2008). Sustainable Nations: What Do Aggregate
Indicators Tell Us? Tinbergen Institute Discussion Paper. Retrieved from
http://www.tinbergen.nl/discussionpapers/08012.pdf
• Samimi, A. J., Erami, N. E., & Mehnatfar, Y. (2010). Environmental Performance Index and
Economic Growth: Evidence from Some Developing Countries. Paper presented at the 12th
International BIOECON Conference, From the Wealth of Nations to the Wealth of Nature:
Rethinking Economic Growth, September 27, 2010, in Veneto, Italy. Retrieved from
http://www.ucl.ac.uk/bioecon/12th_2010/Samimi.pdf
• Science‐Metrix. Ontology and Journal Classification: http://www.science‐
metrix.com/SM_Ontology_100.xls
• Science‐Metrix. Ontology Explorer: http://www.science‐metrix.com/OntologyExplorer
• Srebotnjak, T. (2010). Assessing National Environmental Performance using Composite
Indicators: The Example of the 2010 Environmental Performance Index. Paper presented at the
IAOS/Scorus 2010 Conference on Official Statistics and the Environment: Approaches, Issues,
Challenges and Linkages, October 2010, Santiago, Chile. Retrieved from
http://encina.ine.cl/IAOS2010INGLES/portals/IAOS2010INGLES/Srebotnjak%20Paper_Short_6Au
gust2010.pdf
• Saisana, M., & Saltelli, A. (2010). Uncertainty and Sensitivity Analysis of the 2010
Environmental Performance Index. European Commission Joint Research Centre Scientific and
Technical Report. Retrieved from http://composite‐
indicators.jrc.ec.europa.eu/Document/Saisana_Saltelli_2010EPI_EUR.pdf
23

24. 24