1. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
1
Multidimensional assessment of
performance of organizational units:
Integrating BSC and AHP methodologies
in a Brazilian telecom company
Alexandre Bentes, PUC-Rio, Brazil
Jorge Carneiro, PUC-Rio, Brazil
Herbert Kimura, Mackenzie, Brazil
Jorge Ferreira da Silva, PUC-Rio, Brazil
IAG Business School,
Pontifical Catholic University of Rio de Janeiro
March 24 – 26, 2010
Barcelona, Spain

2. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
2
Appropriate assessment of organizational performance is
essential for firms
The multidimensional nature of organizational performance
poses challenges to researchers and practitioners
The performance of functional areas within an organization
is an instance of the concept of organizational performance
in general
Several studies have
used BSC to assess the performance of firms and of
functional units within organizations
used AHP to rank decision alternatives
But few have used both This is what is new and
contributive in our study
Introduction

3. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
3
Verify whether an integrated approach – that combines the
multidimensional perspective of the Balanced Scorecard
(BSC) with the framework for multi-criteria ranking of decision
alternatives provided by the Analytic Hierarchy Process
(AHP) – can be useful for the assessment of the (relative)
performance of functional units within an organization.
Extend the external validity of both the BSC and the AHP
frameworks by applying those frameworks to another industry
(telecommunications) and another setting (Brazilian
environment and managers) not often researched
Objectives of the study
Multiple perspectives of
organizational performance
Multi-criteria ranking of
decision alternatives
Multidimensional assessment and
ranking of organizational performance

4. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
4
In this study the overall performance of three functional
areas of the financial department of a Brazilian
Telecommunications company was rank-ordered by taking
into consideration:
the relative degree of importance of (four) distinct
perspectives of organizational performance,
the relative degree of importance of performance indicators
within each perspective, and
the (relative) performance of the (three) functional areas in
each performance indicator
Objectives of the study (cont.)

5. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
5
Organizational performance and the Balanced Scorecard (BSC)
Comparison and decision-making based on multiple criteria
The Analytic Hierarchy Process (AHP)
Theoretical Framework

6. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
6
Organizational performance and the Balanced Scorecard (BSC)
Organizational performance is a complex and multi-faceted
phenomenon (Cameron, 1986; Chakravarthy, 1986; Venkatraman and
Ramanujam, 1986)
Kaplan and Norton (1996) proposed that organizational
performance be simultaneously assessed from distinct, but
complementary, perspectives, which were originally detailed as:
Financial
Customer
Internal Business Processes
Innovation and Learning
Theoretical Framework
Financial
Vision and
Strategy
Innovation and
Learning
InternalBusiness
Processes
Customer

7. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
7
Organizational performance and the Balanced Scorecard (BSC) (cont.)
While BSC provides a balance (Kaplan and Norton, 1992)
between short-term and long-term objectives,
between financial and non-financial measures,
between lagging and leading indicators, and
between internal and external performance perspectives”
BSC approaches, nonetheless, also present some challenges:
Overload of information
Necessity to reach some synthesized judgment
Theoretical Framework

8. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
8
Comparison and decision-making based on multiple criteria
Some decision-making problems involve the choice among
alternatives which must take into account multiple criteria which
may present some degree of inconsistency with one another
The values (performance) achieved in some criteria may
suggest a distinct choice (decision) from the values in some
other criteria
Improvement of results under a given criterion may be
obtainable only at the expense of another
This may be particularly true in the assessment of organizational
performance and the choice of the best performing unit (or the
rank-ordering of the units under evaluation)
Theoretical Framework

9. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
9
The Analytic Hierarchy Process (AHP)
A multi-criteria decision-making tool developed by Saaty (1980)
Decomposition, comparative judgment, and synthesis
Decomposition of a complex problem into a multilevel hierarchic
structure of objectives, criteria and sub-criteria, and alternatives
Series of pair-wise comparisons of judgments (of child items below
a parent node, i.e., the relative contributions of the child items to
the parent node)
Theoretical Framework
Objective
Criteria
Sub-criteria
Alternatives

10. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
10
The Analytic Hierarchy Process (AHP) (cont.)
Synthesis in an eigenvector (priority vector)
Judgments express the relative impact / importance of the
elements in the hierarchy and are translated into numbers, which
indicate the relative importance (weight, priority) of an element
over another in the determination of their respective higher
hierarchical level
Final weights of any given lowest element are calculated from a
weighted (by degree of importance) average over all levels of the
hierarchy and synthesize the multiple judgments (often not
mutually consistent)
Rank-ordering of alternatives
Overall importance represents the relative (percent) contribution
of a given indicator with respect to the global goal
Theoretical Framework

11. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
11
The Analytic Hierarchy Process (AHP) (cont.)
Given the particularities of the present study, AHP has some
advantages over other decision-making tools:
Synthesis of multiple viewpoints (criteria) into a single unified
result
A consistency index can be calculated to assess the transitivity
in judgments given by the paired comparisons
Involvement of decision-makers who have to explicitly reveal
their preferences
Communication among decision-makers tends to lead to better
decisions (reached through a process of conflict resolution until
some consensus or, at least, agreement is reached)
Theoretical Framework

12. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
12
Units of analysis: three functional areas of the financial department
of a Brazilian telecom company – Fraud unit, Collection unit, and
Revenue Assurance unit
Data collection procedures
Data collected in 2009
Interviewees and participants
three senior mangers (one for each unit; all with long experience
in the company),
three supervisors (one for each unit),
nine senior analysts (three for each unit), and
the financial operations director.
The diversity in hierarchical levels was deemed important in
order to reach a more consensual and “department-wide” opinion
about performance that would somehow better represent the
variety of views in the financial department.
Methods and data

13. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
13
Iterative and interactive process of data collection
For the selection of dimensions and measures of organizational
(functional) performance
− four dimensions and nine performance indicators chosen
For the definition of the weights (degree of importance) of each
performance dimension and of each indicator (within each dimension)
− Paired comparisons: How much more (less) important is this
dimension / indicator as compared to the other one?
Five rounds of discussion until some agreement was reached
Role played by the investigator
Convergent validity check
Methods and data

14. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
14
Identify the best
performing unit
objective
criteria
Financial
perspective
Ind A Ind B Ind C
Fraud unit
Customer
perspective
Innovation and
learning perspective
Internalbusiness
processperspective
Ind D Ind E Ind F Ind G Ind H Ind I
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
Fraud unit
Collection unit
Revenue
assurance unit
sub-criteria
alternatives
Methods and data
The AHP tree

15. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
15
Training and capability-building activitiesI
Improvement in employees’ capabilitiesH
Innovation and
Learning
Number of new projects fully delivered in
the year
G
Number of improvement projects
implemented in the year
F
Internal Business
Processes
Satisfaction level of external clientsE
Satisfaction level of internal clientsD
Customer
Decrease in operating costsC
ROI of implemented projectsB
Recovered value or avoided lossA
Financial
DefinitionIndicatorBSC Perspective
Methods and data
Indicators selected for the present study

16. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
16
Financial Ind A Ind B Ind C
Ind A 1 4 7
Ind B 1/4 1 3
Ind C 1/7 1/3 1
0.71
0.21
0.08
Customer Ind D Ind E
Ind D 1 1/7
Ind E 7 1
0.12
0.88
Processes Ind F Ind G
Ind F 1 3
Ind G 1/3 1
0.75
0.25
Innovation
and Learning
Ind H Ind I
Ind H 1 5
Ind I 1/5 1
0.83
0.17
CR = 0.03
CR = 0.00
CR = 0.00
CR = 0.00
Application of the Analytic Hierarchy Process
Relative importance,
Normalized weights, and
Consistency ratios
at the performance indicators (sub-criteria) level

17. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
17
BSC
perspectives
Finance Customer Processes
Innovation
and Learning
Finance 1 5 4 5
Customer 1/5 1 3 3
Processes 1/4 1/3 1 2
Innovation
and Learning
1/5 1/3 1/2 1
0.59
0.21
0.12
0.08
CR 0.09
Application of the Analytic Hierarchy Process
Relative importance,
Normalized weights, and
Consistency ratios
at the BSC perspectives (criteria) level

18. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
18
0.71
0.21
0.08
x 0.59
0.42
0.12
0.05
Indicators A,B and C
=
0.12
0.88
x 0.21 0.03
0.18
Indicators D and E
=
local
weights
global
weights
local
weights
global
weights
0.75
0.25
x 0.12 0.09
0.03
Indicators F and G
=
local
weights
global
weights
0.83
0.17
x 0.08 0.07
0.01
Indicators H and I
=
local
weights
global
weights
Application of the Analytic Hierarchy Process
From local weights to global weights of each indicator
weight of
parent
node
weight of
parent
node
weight of
parent
node
weight of
parent
node

19. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
19
Ind. A Fraud Collection
Revenue
assurance
Fraud 1 1/7 1/2
Collection 7 1 5
Revenue
assurance
2 1/5 1
0.09
0.74
0.17
Ind. B Fraud Collection
Revenue
assurance
Fraud
1 4 3
Collection
1/4 1 1/3
Revenue
assurance
1/3 3 1
0.61
0.12
0.27
Ind. C Fraud Collection
Revenue
assurance
Fraud
1 1/4 1/2
Collection
4 1 4
Revenue
assurance
2 1/4 1
0.13
0.66
0.21
CR = 0.01
CR = 0.06
CR = 0.05
Application of the Analytic Hierarchy Process
Relative degree of success in (performance indicators of) the
financial perspective by each functional unit

20. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
20
Ind D Fraud Collection
Revenue
assurance
Fraud
1 1/2 3
Collection
5 1 7
Revenue
assurance
1/3 1/7 1
0.19
0.73
0.08
Ind E Fraud Collection
Revenue
assurance
Fraud
1 1/4 2
Collection
4 1 3
Revenue
assurance
1/2 1/3 1
0.22
0.63
0.15
CR = 0.06 CR = 0.09
Application of the Analytic Hierarchy Process
Relative degree of success in (performance indicators of) the
customer perspective by each functional unit

21. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
21
Ind. F Fraud Collection
Revenue
assurance
Fraud
1 1/2 1/5
Collection
2 1 1/4
Revenue
assurance
5 4 1
0.12
0.20
0.68
Ind. G Fraud Collection
Revenue
assurance
Fraud
1 1/2 1/6
Collection
2 1 1/3
Revenue
assurance
6 3 1
0.11
0.22
0.67
CR = 0.02 CR = 0.00
Application of the Analytic Hierarchy Process
Relative degree of success in (performance indicators of) the
internal business process perspective by each functional unit

22. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
22
Ind. H Fraud Collection
Revenue
assurance
Fraud
1 1/5 1/3
Collection
5 1 3
Revenue
assurance
3 1/3 1
0.10
0.64
0.26
Ind. I Fraud Collection
Revenue
assurance
Fraud
1 1/4 1
Collection
4 1 2
Revenue
assurance
1 1/2 1
0.18
0.58
0.23
CR = 0.03 CR = 0.05
Application of the Analytic Hierarchy Process
Relative degree of success in (performance indicators of) the
innovation and learning perspective by each functional unit

23. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
23
Fraud 0.09 0.61 0.13
Collection 0.74 0.12 0.66
Revenue
assurance
0.17 0.27 0.21
Ind. A 0.42
Ind. B 0.12
Ind. C 0.05
x
Fraud 0.13
Collection 0.35
Revenue
assurance
0.11
vectorial product
Ind. A Ind. B Ind. C
0.59
=
Fraud
0.19 0.22
Collection
0.73 0.63
Revenue
assurance
0.08 0.15
Ind. D 0.03
Ind. E 0.18
Fraud
0.05
Collection
0.14
Revenue
assurance
0.02
Ind. D Ind. E
0.21
x =
vectorial product
Finance
Customer
Application of the Analytic Hierarchy Process
Partial contribution of each functional unit to the overall
performance objective

24. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
24
Fraud
0.12 0.11
Collection
0.20 0.22
Revenue
assurance
0.68 0.67
Ind. F 0.09
Ind. G 0.03
Fraud
0.01
Collection
0.03
Revenue
assurance
0.08
Ind. F Ind. G
0.12
x
vectorial product
=
Fraud
0.10 0.18
Collection
0.64 0.58
Revenue
assurance
0.26 0.23
Ind. H 0.07
Ind. I 0.01
Fraud
0.01
Collection
0.05
Revenue
assurance
0.02
Ind. H Ind. I
0.08
Processes
Innovation
and Learning
x =
vectorial product
Application of the Analytic Hierarchy Process
Partial contribution of each functional unit to the overall
performance objective (cont.)

25. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
25
0.240.240.020.080.020.11
Revenue assurance
unit
1.001.000.080.120.210.59Total
Finance Customers
Internal Business
Processes
Innovation and
Learning
TotalTotal
Fraud unit 0.13 0.05 0.01 0.01 0.190.19
Collection unit 0.35 0.14 0.03 0.05 0.570.57
Application of the Analytic Hierarchy Process
Final results for assessment of the best performing functional
unit

26. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
26
Results indicate that the Collection unit performed better than the
other units in three perspectives – finance, customer, and
innovation and learning – and had the highest contribution (57%)
to final objective.
The Revenue Assurance unit performed better in the internal
business processes perspective, but its contribution (23%) to the
final decision regarding the (overall) best performing unit is not too
different from the contribution of the Fraud unit (20%).
Discussion

27. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
27
BSC explicitly incorporates several perspectives (besides the usual
financial viewpoint) to organizational performance assessment
Evaluating performance requires some agreement on the relative
importance of performance perspectives and performance
indicators in the way to reach some aggregated (synthesized)
measure or ranking
AHP handles multiple dimensions and measures with distinct
degrees of importance and translates the overall result into a
unified metric, explicitly accounting for incongruence among
objectives and mutually inconsistent evaluations
AHP circumvents the pitfalls of having managers use a simpler, ad
hoc, weighing approach to make sense of the multiplicity of
performance measures from a balance scorecard
AHP provides more than an ordinal ranking; it also informs about
the magnitude of the difference between alternatives, that is, how
much more (less) a functional area is performing better (worse)
than another
Conclusions

28. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
28
AHP provides insights to managers as to which functional areas are
more likely to contribute to overall success and which deserve more
attention
but all in relative basis;
note that if all areas improve, some will still be (relative)
underperformers…)
so, both a relative and an absolute assessment should be
conducted
Results of the AHP + BSC can assist managers in deciding about:
bonus distribution, incentive schemes
identification (not covered in this study) of the possible reasons
for poor (or good) performance
Conclusions

29. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
29
Academic and managerial implications:
Such an integrated approach (between BSC and AHP) was shown
to have convergent validity with overall assessments by a senior
director while providing much more fine-grained information for
future managerial actions.
The interactive and iterative process employed in this study has
the additional advantage of leading managers to apprehend the
diverse perspectives of performance assessment and to
understand the possible trade-offs thereof.
Conclusion

30. Alexandre Bentes
Jorge Carneiro
Herbert Kimura
Jorge Ferreira da Silva
Multidimensional assessment of performance of organizational units: Integrating BSC and AHP methodologies
30
Questions and comments
are very welcome!
Alexandre Bentes (alexvb@globo.com)
Jorge Carneiro (jorgemtc@iag.puc-rio.br)
Herbert Kimura (herbert.kimura@gmail.com)
Jorge Ferreira da Silva (shopshop@iag.puc-rio.br)