The document discusses two projects to develop performance measurement frameworks for science-based organizations based on their organizational visions. For a provincial RTO aiming to be a world leader in innovation for its main client, the project used a focused and participatory approach, developing 25 indicators aligned with its strategic plan through workshops. For a federal RTO aiming to be a top five global RTO, the project took an exploratory and comparative approach, profiling over 100 indicators used by peer RTOs and developing a benchmarking strategy through international consultations. Key lessons highlighted how an organization's vision influences the project approach, methods and results, and implications for implementing and using the performance frameworks.

1. Valuing organizational vision in the development
of performance measurement frameworks
of science‐based organizations
Lessons learned from two projects
CES 2012 Conference Halifax
Tuesday, May 15, 2012 | 15:45‐17:15

2. Outline
 Context and objective
 Background on organizational vision and performance
 Overview of two performance framework development projects:
 Different organization types, visions and projects (tailored)
 Effect of organizational vision on the two projects:
 Approach
 Methods
 Results
 Project update on framework implementation
 Lessons learned
 For organizations
 For external evaluators/consultants
 Questions?
2

3. Context and objective
 Research and Technology Organizations (RTOs) increasingly make use
of performance measurement (PM) to monitor the achievement of
strategic objectives
 Approaches to develop performance measurement frameworks are
directly aligned with an organization’s:
 strategic vision/values
 competitive environment
 reporting requirements
 Using two recent projects, this paper demonstrates how
organizational values influenced:
 Approach: performance measurement strategies
 Methods: how the performance framework was developed
 Results: design and implementation of plan for the proposed framework
3

4. R&D performance measurement
Common goals:
 Monitor the scientific performance and effectiveness of the
organization on a regular basis
 Monitor the efficiency and economy of R&D project management and
other supporting processes
 Inform operational and strategic decisions
 Feed the production of relevant performance reports
 Provide reliable information to support effective evaluation
Common challenges:
 Culture of performance measurement and management
 Organizational structure and management
 Internal capacity for planning / performance measurement / evaluation
 Data availability and quality
 Buy‐in of data provider and user (championship/ communication)
4

5. Organizational vision and performance
 Many studies have examined the link between the shared vision,
values, strategy, goals and practices within organizations (often
referred to collectively as ‘organizational culture’) and organizational
performance.
 Given the difficulty of measuring organizational culture, the research
has been inconclusive about the exact nature of this relationship. (Lim,
1995)
 However, many researchers have asserted that organizational culture
affects all aspects of organizational interactions (Henri, 2006) and that
a strong culture is predictive of organizational performance. (Abu‐
Jarad, Yusof, & Nikbin, 2010)
 Very few studies, however, have looked specifically at how
organizational culture influences performance measurement
strategies.
5

6. Organizational vision and performance
 Evidence suggests that companies that exhibit more ‘flexible’ or
‘adaptive’ cultures tend to:
 have better long‐term performance overall (Abu‐Jarad, Yusof, &
Nikbin, 2010; Lim, 1995)
 use a greater number of performance measures and focus their
performance measurement strategy on supporting decision‐
making, justifying activities and establishing priorities. (Henri, 2006)
 R&D organizations, in particular, have unique needs and challenges
with respect to the capture of impacts
 e.g., contributions may be impossible to measure, even in the
medium‐long term.
6

7. Organizational vision and performance
 In general, studies suggest that organizations:
 clarify and translate their vision and values through their PM strategy;
 align their PM system with their specific R&D profile (basic vs. applied,
short vs. long‐term);
 use multiple performance measures, including both financial and non‐
financial indicators, to determine short‐term results as well as long‐
term processes, impacts and cause‐and‐effect functions;
 use their culture as a basis for allocating resources, establishing
milestones, and facilitating strategic review and feedback mechanisms.
(Abu‐Jarad, Yusof, & Nikbin, 2010; Jyoti, Banwet, & Deshmukt, 2006; Shahzad,
Luqman, Khan, & Shabbir, 2012)
 Not surprisingly, we observed the influence of organizational culture as
we conducted two projects using two completely different
performance measurement approaches that reflected each
organization’s vision and values.
7

8. Overview of two performance framework
development projects
Provincial RTO Project Federal RTO Project
Vision:
Vision:
Be a world leader in
Be among the top five
innovation aligned
leading RTOs in the
with strategic issues of
world
its main client
Approach: Approach:
Focused + Exploratory +
Participatory Comparative
8

9. Overview of two performance framework
development projects
Provincial RTO Federal RTO
Vision Be a world leader in innovation Be among the top five leading
aligned with strategic issues of its RTOs in the world
main client
Organization Provincial RTO that mainly serves Federal RTO that serves the
type the scientific and technological scientific and technological
needs of importance for a needs of importance for the
provincial crown owned company country
(main client)
Strategic • 5‐year plan New plan was in development
plan • 4 guiding principles at that time
• 5 priorities/11 action items
9

10. Overview of two performance framework
development projects
Provincial RTO Federal RTO
Vision Be a world leader in Be among the top five leading RTOs in
innovation aligned with the world
strategic issues of its client
Overall Goal: Goal:
project • Develop an internal Develop a comparative framework
goal and performance measurement strategy for assessing effectiveness and
approach framework to monitor the benchmarking performance against
strategic plan comparable international RTOs
Expected results: Expected results:
• Actionable plan • A conceptual framework strategy
• Plan developed in • Creation of linkages with the
consultation with key staff international RTO community (for
for buy‐in benchmarking)
10

11. Effect of organizational vision on project approach
Provincial RTO Federal RTO
Be a world leader in innovation Be among the top five leading RTOs in the
Vision
aligned with strategic issues of its world
main client
Focused Exploratory
• Focused on the strategic level • Identify best practices for both operational
• Focused on the identification of and strategic levels
limited/available number of • Explore multiple indicators and
indicators measurement approaches
Approach
Participatory Comparative
• Close collaboration with senior • Consult with other RTOs to compare/
management (RTO Director benchmark
General) • Examine how other RTOs are organized and
• Participatory and interactive deliver R&D (client‐focused and impact
approach with internal staff driven)
• PM capacity building and buy‐in • Engage a community of practice
11

12. Effect of organizational vision on project method
Provincial RTO Federal RTO
Be a world leader in innovation aligned Be among the top five leading
Vision
with strategic issues of its main client RTOs in the world
Secondary data collection Secondary data collection
• Focused on provincial RTO’s internal • Extensive scan and review of
literature and document/data document/data on the federal
• Identification of data RTO and on the comparable RTOs
availability/feasibility • Extensive literature review
Primary data Primary data
Methods
• 4 full‐day roundtable workshops with • Consultations with some senior
different levels of managers: managers from the Federal RTO
 Directors • Consultation with representatives
 Managers (involved in PM and corporate
 Lead scientists strategy) from 7 comparable RTOs
 Integration/wrap‐up • Presentation for validation
• Presentation for validation
12

13. Effect of organizational vision on the project results
Provincial RTO Federal RTO
Be a world leader in innovation Be among the top five leading RTOs in the
Vision
aligned with strategic issues of world
its main client
• Report on the • Profiles/analysis of performance measurement
proposed/selected indicators approaches and indicators used by other RTOs
and rationale for deployment • Characterization of operational and strategic
• Included alternative indicators indicators
for consideration • A strategy to assess the Federal RTO
• Total of 25 indicators (1‐3 per performance
Results
action item) • Step‐by‐step approach for engaging a
• 25 page report community of practice (for benchmarking)
• All levels of Provincial RTO • More than 100 indicators explored
management were aware of • 60 page report and 31 pages of technical
the project results material on international RTOs
• Federal RTO performance and evaluation staff
were aware of the project results
13

14. Current status of project implementation …
14

15. Lessons learned for organizations
Using a focused and participatory approach:
 Provides near‐to‐complete product for implementation
 Limited number of indicators
 Data is (almost) readily available
 Wider organizational awareness and buy‐in
 Provides a clear set of available indicators, but many are proxy
indicators at the strategic level
 Indicators inform on the progress towards strategic outcomes, but not
necessarily on how to improve performance
 Potential tension between staff during the definition/selection of
operational indicators to measure strategic performance
15

16. Lessons learned for organizations
Using an exploratory and comparative approach:
 Necessity to define and conceptualize a full framework, including
 Internal measures (operational/corporate‐level indicators)
 External measures (strategic/impact indicators)
 Need to distinguish indicators for internal performance measures and
indicators for international benchmarking (can’t always get both)
 Need to narrow down the number of potential indicators, especially for
international comparison
 Access/confidentiality/availability issues of comparable data for benchmark
 International representatives did not have the same conception of, or
experience with, internal and external measures
 Implementation requires further challenging steps (final set of indicators,
internal buy‐in, further engagement with RTOs, data sharing agreement, etc.)
 Organizational culture can impede/influence the selection of common
indicators and how performance is judged
16

17. Lessons learned for consultants
Using a focused and participatory approach:
 Data load and complexity more manageable
 End‐results/tools developed are more transferable to other
projects/RTOs
 Easy to access data (proximity, language and availability)
 Greater participation/input from a wider variety of internal
stakeholders
 Had to help operational managers to think in terms of strategic
indicators
 Had to mediate differences in performance values from different units
 No means to compare or calibrate selected indicators with other RTOs
17

18. Lessons learned for consultants
Using an exploratory and comparative approach:
 Learned and benefited from the experience of several organizations
 Further reflection and conceptualization of performance measurement
 Created links with the international community
 Challenges to select comparable RTOS and to reach the right
informants
 The “full comparative matrix syndrome”
 Expectation of data in all cells
 Multiplier effect (50 indicators x 6 dimensions x 8 RTOs = 2,400)
 Challenges in the definition and interpretation of collected indicators
 Language barrier with key PM documents/data sources
 Much more work compared to the other approach (for similar budget)
 Had to manage expectations given the ambitious nature of the
international and comparative dimensions
18

19. Thank you for your time and feedback
CONTACT INFO
Andréa Ventimiglia, BSc MJ
Research Analyst | Science‐Metrix
514‐495‐6505 x124 Questions?
andrea.ventimiglia@science‐metrix.com
Frédéric Bertrand, MSc CE
Vice‐President, evaluation | Science‐Metrix
514‐495‐6505 x117
frederic.bertrand@science‐metrix.com Science‐Metrix
1335, Mont‐Royal E.
Montreal, Quebec H2J 1Y6
Telephone: 514‐495‐6505
Fax: 514‐495‐6523
ACKNOWLEDGEMENTS E‐mail: info@science‐metrix.com
Emmanuel Trépanier MA (contributor)
Eric Archambault PhD (contributor) WEB SITE
Julie Caruso MLIS (contributor)
www.science‐metrix.com
19

20. References
 Abu‐Jarad, I. Y., Yusof, N., & Nikbin, D. (2010, December). A review paper on organizational
culture and organizational performance. International Journal of Business and Social Science,
1(3), 26‐46. Retrieved from
http://www.ijbssnet.com/journals/Vol._1_No._3_December_2010/4.pdf
 Henri, J. F. (2006). Organizational culture and performance measurement systems.
Accounting, Organizations and Society, 31, 77–103. Retrieved from
http://arafiki.edublogs.org/files/2011/05/Organizational‐culture‐and‐performance‐
measurement‐systems‐1zth1xx.pdf
 Jyoti, T., Banwet, D. K., & Deshmukt, S. G. (2006). Balanced scorecard for performance
evaluation of R&D organizations: A conceptual model. Journal of Scientific and Industrial
Research, 65, 879‐886. Retrieved from
http://nopr.niscair.res.in/bitstream/123456789/4952/1/JSIR%2065(11)%20879‐886.pdf
 Lim, B. (1995). Examining the organizational culture and organizational performance link.
Leadership & Organization Development Journal, 16(5), 16‐21. Retrieved from
http://www.emarketing.net.cn/upload/file/2008/06/25/211214400841549.pdf
 Shahzad, F. Luqman, R. A., Khan, A. R., & Shabbir, L. (2012, January). Impact of organizational
culture on organizational performance: An overview. Interdisciplinary Journal of
Contemporary Research in Business, 3(9), 975‐985. Retrieved from http://www.journal‐
archieves14.webs.com/975‐985.pdf
20

21. Theoretical model
Relationship between culture, use of PM systems and diversity of measurement
J.‐F. Henri / Accounting, Organizations and Society 31 (2006) 77–103
21