This document proposes a seven-step framework for measuring the performance of an enterprise resource planning (ERP) system using a fuzzy integral approach. The framework aims to link performance indicators (PIs) to the objectives of the ERP implementation project. A fuzzy ERP performance index is used to account for ambiguities in evaluating ERP system performance. An example application in Taiwan demonstrates how the proposed framework can be applied in practice.

1. The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1741-038X.htm
An ERP
An ERP performance measurement
measurement framework framework
using a fuzzy integral approach
607
Chun-Chin Wei
Department of Industrial Engineering and Management, Ching Yun University, Received December 2006
Chung Li, Taiwan, Republic of China Revised July 2007
Accepted September 2007
Tian-Shy Liou
Department of Business Administration, Chen Shiu University, Niaosong,
Taiwan, Republic of China, and
Kuo-Liang Lee
Department of Industrial Engineering and Management, Ching Yun University,
Chung Li, Taiwan, Republic of China
Abstract
Purpose – The purpose of this paper is to propose a comprehensive framework for measuring the
performance of an enterprise resource planning (ERP) system to survey suitable performance
indicators (PIs) according to knowledge of the ERP implementation objectives set up at the
implementation phase and build consistent measurement standards for facilitating the complex ERP
performance evaluation process.
Design/methodology/approach – A seven-step ERP performance measurement framework based
on the objectives of ERP implementation is proposed. A fuzzy ERP performance index is used to
account for the ambiguities involved in evaluating the performance of the ERP system. The fuzzy ERP
performance index can be translated first into simple scores and then back to linguistic terms. An
actual example in Taiwan demonstrates the feasibility of applying the proposed framework.
Findings – The findings indicate that the PIs of ERP performance measurement should align with
the objectives of ERP implementation. The assessment results can represent the achievement of these
objectives and the directions for improving the adopted ERP system.
Originality/value – This study may be interesting to some academic researchers and practical
managers. The proposed framework can provide a procedure to link the objectives identified in the
ERP system implementation phase and the performance considerations in the ERP use phase.
Keywords Manufacturing resource planning, Fuzzy control, Decision theory,
Performance measurement (quality)
Paper type Research paper
1. Introduction
Owing to the highly severe market competition and the immense impact of advances in
information technology progress, a number of companies have widely implemented the
enterprise resource planning (ERP) systems. A comprehensive ERP system Journal of Manufacturing Technology
implementation project involves selecting an ERP software system and a cooperative Management
Vol. 19 No. 5, 2008
pp. 607-626
q Emerald Group Publishing Limited
The authors would like to thank the National Science Council of the Republic of China, Taiwan 1741-038X
for financially supporting this research under Contract No. NSC 94-2213-E-231-005. DOI 10.1108/17410380810877285

2. JMTM vendor, implementing the selected system, managing business processes change, and
19,5 examining the practicality of the adopted ERP system (Wei and Wang, 2004). That is,
completing ERP system implementation is not the final stop but a go live start. One of the
most significant challenges faced by information managers today is measuring the
performance of the adopted ERP system to justify its value-added contribution for
accomplishing the organization’s missions. Furthermore, managers would also like to
608 know which parts of their ERP system need to improve and whether the system’s overall
performance is enhancing over time.
Success has often been defined as a favorable or satisfactory result or outcome
(Saarinen, 1996). In reality, “the success of an ERP system” is achieved when
the organization is able to better perform all its business processes and when the
adopted ERP system really achieves the objectives that managers strive. That is, the
development of ERP performance measurement process should establish a feedback
mechanism between the desired objectives of ERP adoption and the substantial effects
of ERP execution (Mashari et al., 2003). Traditionally, a set of performance indicators
(PIs) is employed to determine the effectiveness and efficiency of an ERP system. The
key is to build up a process for determining the relationships between the objectives of
the ERP implementation project and the ERP PIs for measuring its performance, so that
they have identical guidance and evaluation standards during the entire project period.
Typically, there are many factors with many characteristics to consider in the ERP
performance evaluation: tangible, intangible, quantitative and qualitative. The
post-usage perception of an ERP system to a user is a subjective interaction. Personal
evaluation differs from one user to another depending on individual variance of
personal subjectivity, experience, and cognition. Furthermore, for many people, the
evaluation of a qualitative PI is a subjective and ambiguous concept hard to be
expressed, and not all people can concretely voice out their feelings on a scale of one to
five. The evaluators often express their ratings in natural language rather than in
numbers. The concept of a linguistic variable is very useful in dealing with situations
that are too ill-defined to be reasonably described in conventional quantitative
expressions (Chen and Hwang, 1992). Fuzzy set theory is developed for solving
problems in which descriptions of activities and observations are imprecise, vague, and
uncertain and widely used in the decision analysis problems, like selection (Liang and
Wang, 1994; Shamsuzzaman et al., 2003; Wei and Wang, 2004; Sharif Ullah, 2005; Chen
and Ben-Arieh, 2006) and performance assessment (Chan et al., 2002; Jain et al., 2004;
Ohdar and Ray, 2004; Chang et al., 2007). Thus, a fuzzy aggregative method is highly
effective in integrating linguistic assessments and weights to measure the performance
of an ERP system.
This paper aims to construct an ERP performance measurement framework to
elaborate the process of PI development for linking with the ERP implementation
objectives. According to the knowledge of the ERP implementation objectives, decision
makers can extend them to suitable PIs for measuring whether the objectives have
been achieved. A fuzzy ERP performance index is used to account for the ambiguities
involved in evaluating the performance of an ERP system. A method of translating the
fuzzy ERP performance index back to linguistics is also used to obtain the linguistic
achievement representation of the ERP implementation objectives and the overall ERP
system. An empirical case in Taiwan is described to demonstrate the practical viability
of the proposed method.

3. 2. Method review An ERP
Several methods have been proposed for measuring the performance of ERP systems measurement
or other information systems (IS). Traditionally, financial performance metrics such as
return on investment, net present value, or payback period could be used (Kivijarvi and framework
Saarinen, 1995; Murphy and Simon, 2001), but because of the unique nature of the IS
investment, they seldom suffice in practice. Instead, the evaluation of IS success has to
be supplemented by a subjective judgment and surrogate measures. 609
The system and data quality assessment of IS have been widely studied (Delone and
McLean, 1992; Palvia et al., 2001; Lee et al., 2002). The quality measurement reflects the
engineering-oriented performance characteristics of the system itself and the quality of
information and data. Data quality focuses on the IS output, namely, the quality of the
information that the system produces. Later, numerous information quality measures
have been included within the area of “User satisfaction.”
Information technologies cannot by itself influence the productivity of a company.
The main efficiency factor lies in the way people use these technologies. Related
studies about user satisfaction evaluated the IS performance using the experience
and perspective of various users, like employees, middle managers, top managers
and system engineers (Wu et al., 2002). Some IS user satisfaction measurement
questionnaires and methods have also been applied to real cases (Doll and Torkzadeh,
1988; Klenke, 1992; Saarinen, 1996; Wu et al., 2002).
Recently, some popular techniques have been used to measure the performance of
ERP systems or other IS, like analytic hierarchy process (AHP) (Chan et al., 2006; Chan
and Kumar, 2007), data envelopment analysis (Stensrud and Myrtveit, 2003),
importance-performance maps (Skok et al., 2001), and balanced scorecard (Michael and
Jens, 1999; Hagood and Friedman, 2002). These reports integrated the traditional PIs
with new techniques to build up performance measurement systems and offered some
useful applications in practice.
Many researchers stated that there is no best appraisal technique that addresses all
project considerations (Saarinen, 1996; Irani, 1999). Further, they argued that the reason
for this is the investments in IS are aggregates of complexity, and notably different from
each other. However, the most frequently adopted measures are to refer to the common
indices without developing tailor-made measures that echo the objectives of ERP
implementation for a specific company’s ERP system. Additionally, little research has
addressed the relationship between the ERP implementation stage and the ERP use stage.
This study develops a framework with fuzzy set theory to synthesize managers’ tangible
and intangible measures with respect to numerous PIs extended from the objectives of
ERP implementation to obtain an aggregated fuzzy ERP performance index. The
framework also can translate the fuzzy ERP performance index into simple scores and
then back to linguistic terms for indicating how the adopted ERP system is performing
and what actions the managers should undertake to improve the ERP system.
3. Procedure for measuring the ERP performance
Three principal themes are noted in the proposed ERP performance measurement
framework, including the PI structure construction, fuzzy group ERP performance
measurement, and result analysis and system improvement. To clearly present the
proposed ERP performance measurement framework, a step-wise procedure is first
described:

4. JMTM (1) extend the objectives of the ERP implementation project to appropriate PIs;
19,5 (2) add other crucial PIs into the PI set in an ERP output view;
(3) construct the proper PI structure;
(4) develop the detailed performance assessment method;
(5) assess the performance of the adopted ERP system;
610 (6) aggregate the assessments to determine the fuzzy ERP performance index; and
(7) analyze the results and improve the ERP system.
Figure 1 shows the flowchart of the proposed ERP performance measurement
framework. The details of each step are presented below.
3.1 Extend the objectives of the ERP implementation project to appropriate PIs
Clearly defined objectives were identified as the most important key to success. The
ERP implementation objectives generally indicate the direction in which the managers
should strive to do better. For evaluating ERP performance, it is important to
Extend the ERP implementation objectives to performance indicators
Discuss “How to No
Can this means-objective
PI structure evaluate whether the
be taken as a suitable
construction means-objective has
performance indicator?
been achieved?”
Yes
Generate a performance indicator
Add other crucial performance indicators
Construct the performance indicator structure
Fuzzy group ERP Develop the detailed performance evaluation contents
performance
measurement
Assess the performance indicators
Calculate the fuzzy ERP performance index
Result analysis and
Figure 1. system improvement
ERP performance
measurement framework Analyze the results and improve the ERP system

5. incorporate appropriate measures that are linked to the ERP system’s role and the An ERP
objectives of the ERP implementation project. The decision makers should transform measurement
the objectives into the suitable ERP PIs to link up the input factors of an ERP
implementation project with the output performance factors and indicate the gap framework
between what the managers want and what the ERP system performs. The objectives
of ERP implementation development method can refer to Wei et al. (2005).
The first step is to form an ERP performance measurement project team involving 611
critical managers, user representatives, system experts and consultants. Critical
managers formulate an ERP system performance assessment plan, identify suitable
PIs and develop consistent evaluation guidance. User representatives from different
departments in the team can be divided into research groups to gather and offer
managers the ERP system data based on their specialties and job responsibility.
Initially, the team members should extract the PIs to form a PI set from the objective
structure which has been established in the ERP implementation process. There are
two kinds of objectives in the ERP implementation objective structure (Wei et al., 2005).
The fundamental-objectives in the objective structure are those that are important to
specify the goal of the ERP implementation. They point out why the managers care
about the selection situation and what criteria the managers should be reviewing in the
alternatives (Clemen, 1996). Additionally, the means-objectives in the objective
structure highlight how to accomplish the desired fundamental-objectives. Having
sorted out them, the team members can rest assured that the team will be able to
evaluate alternatives whose performances are consistent with the company’s concerns.
Based on the definitions, this study finds that the fundamental-objectives indicate the
directions of ERP performance evaluation. Some means-objectives are suitable to be
PIs to evaluate whether the fundamental-objectives have been accomplished as
promised.
We can start from a means-objective in the means-objective network to discuss
whether it can be used to demonstrate a PI. After discussing, if the means-objective is a
suitable PI, then add it into the PI set. If it is not a suitable PI, the team members can
discuss, “How to evaluate whether this means-objective has been achieved?” The
answers can reveal some more detailed and new PIs. Add them to the PI set. If the PI
cannot completely evaluate the achievement of its corresponding means-objective,
members need to survey additional PIs to complement the PI. Go through all
means-objectives in the ERP objective structure, we can formulate an initial ERP PI set.
3.2 Add other crucial PIs into the PI set in an ERP output view
Whereas the initial PI set is expanded from the objectives of ERP implementation, the
set cannot entirely involve all PIs which are used to measure the ERP system
performance. The team members should survey some proper PIs based on the output
performance aspects of ERP system execution, like the impact of individual and
organization. Then, these critical PIs can be added into the PI set.
3.3 Construct the PI structure
Since the adopted ERP system is continuously working and improving over time and
across the organization in a complex exercise, the measurement effectiveness cannot be
simplified and understood from a single aspect only. After surveying the PIs, the team
members should organize them into a hierarchy to conduce the data analysis

6. JMTM in performance evaluation process. Structuring the PIs means organizing them so that
19,5 they describe in detail what the team members want to achieve and can be
incorporated in a proper method into the evaluation model. Additionally, a systematic
PI structure can guide the directions of ERP system improvement. In order to be
compatible with the ERP objective structure and consider the impact of individual and
organization, we classify the PIs into three main categories:
612 (1) System factors – indicators for evaluating the utilization of the ERP system.
(2) Vendor factors – indicators for assessing the performance of the ERP vendor.
(3) Impact factors – the impact of information on the organizational performance
and individual.
The team can review the indicators in the PI set and put them into perspective, the
three main categories, system, vendor, and impact factors. A certain degree of
arbitrariness may occur in some indicator classification, because they do not surely fit
into any one category or fit into several. If a PI is developed from the ERP
implementation objective structure, this PI would be classified into the same main
category as the corresponding objective belongs to. If the PI is not extracted from the
objective structure, team members must discuss which category the PI should put into.
For reducing duplicate and long-term discussions, the PI classification is well while
most of the members can achieve the consensus on the classification. And the group
discussion and classification can decrease the deviation of individual opinion.
Since too many indicators would make numerous evaluations, the process may
become very inefficient. The team should iteratively examine and modify the hierarchy
of selected PIs so that they are complete, decomposable, non-redundant, measurable
and minimal (Keeney and Raiffa, 1993). After specifying the PI hierarchy, they may
find themselves refining the context and modifying the performance evaluation
process. Refining the context several times and iterating through the corresponding set
of indicators are not sighs of poor decision making. They indicate that the decision
situation is being taken seriously, and that many different possibilities and
perspectives are being considered.
3.4 Develop the detailed performance measurement guidance
A PI is a measurable item whose value reflects the degree of achievement for a
particular fundamental-objective or an impact. It is important to have an explicit
knowledge and understanding of how a PI is measured. The members should
investigate what types of data they need to collect and how to collect the data for
evaluating each indicator. A standard form can help them to collect the data and
conduct the performance assessment. Additionally, the knowledge of the objective
structure cannot only help in identifying the PIs, but also the knowledge of the
objectives indicates how outcomes must be measured and what kinds of uncertainties
should be considered. The team also can examine the suitability of PIs in the PI
hierarchy when they discuss the detailed contents of every PI. If they find any
problems of PIs, they can revise the PI hierarchy.
After developing the detailed performance measurement guidance of PIs,
weightings associated with PIs can be assigning. The weight of each PI can be
determined by direct assignment or indirect pairwise comparisons like the AHP

7. (Chang and Chen, 1994; Saaty, 1980). Then, we can obtain a weighting vector, W. The An ERP
values in vector W have the domain range (0, 1). measurement
3.5 Assess the PIs framework
Even some PIs can be easily quantified, it is possible that the rest of the majority may
be hardly measured. The quantitative indicators are evaluated using marginal value
function in terms of direct and inverse linear relationship. The rating rises as the value 613
of the PI rises in direct relationship. Contrarily, in inverse relationship, rating rises
as the value of PI lowers. A baseline of each PI which the team members hope to
achieve can be setting. Then, the team members can easily analyze the gap in what is
being collected the ERP system was performing versus what they want to achieve.
Define:
ðvi 2 v0 Þ
i
ri ¼ : ð1Þ
ðv* 2 v0 Þ
i i
where vi is the value of PI i which the evaluators assess the performance of current ERP
system is performing. v0 is the worst value of PI i which the team believes the ERP
i
system should perform. v* is the maximum value of PI i which the team expects the
i
best possible performance they believed that the ERP system might achieve. Then, ri
(0 # ri # 1) denotes a dimensionless value to ensure that the value is compatible with
the linguistic ratings of the qualitative PIs. Assume that the crisp rating of ri is r, its
triangular fuzzy number (TFN) is (r, r, r).
On the other hand, the members assess the qualitative PIs using a simple rating
questionnaire or form to rate each PI. Subjective assessments are given in linguistic
terms to determine the degree of the adopted ERP system performing against qualitative
PIs. Linguistic terms have been found intuitively easy to use in expressing the
subjectiveness and imprecision of the decision makers’ assessments (Omero et al., 2005;
Chang and Chen, 1994; Liou and Wang, 1994). Then, linguistic terms must first be
transformed into fuzzy numbers by using appropriate conversion scale. To facilitate the
making of subjective assessments in evaluating the qualitative PIs’ performance, a
numerical approximation system proposed by Chen and Hwang (1992) is used to
systematically convert linguistic terms to their corresponding fuzzy numbers. L ¼ {VP,
P, F, G, VG}, VP – very poor, P – poor, F – fair, G – good, and VG – very good. Table I
specifies the TFNs for these linguistic values. If some decision makers do not agree with
the assumed numerical approximation system, they can define their own ratings and the
corresponding TFNs to express the individual perception of the linguistic terms. Since
the values of the quantitative PIs are converted into dimensionless ratings, the ratings
Rating TFN
Very poor (0, 0, 0.3)
Poor (0, 0.3, 0.5) Table I.
Fair (0.2, 0.5, 0.8) Linguistic variables
Good (0.5, 0.7, 1.0) describing values
Very good (0.7, 1.0, 1.0) of ratings

8. JMTM ~
are compatible with the ratings of the qualitative PIs. A fuzzy vector R of PI ratings can
19,5 be obtained combined the both quantitative and qualitative indicators.
3.6 Aggregate the assessments to determine the fuzzy ERP performance index
Define:
~ ~
S ¼ R^W T ð2Þ
614
~
Based on the extension principle, the values in the fuzzy vector S are still TFNs. For
each corresponding fundamental-objective, a fuzzy performance index can be obtained.
Then, roll them up into the fuzzy ERP performance index of each main category and
the entire system using equation (2).
A score is easy for the managers to understand and communicate to each other. In
this study, a fuzzy integral value method with an optimism index proposed by Liou
and Wang (1992, 1994) is applied.
Suppose the fuzzy performance index of a fundamental-objective or the entire
system is c with the left membership function f L and the right membership function f R
~ ~
c ~
c
divided by the highest membership value 1. Define that g L and g R are the inverse
~
c ~
c
functions of f L and f R , respectively. Then the left integral value of c is defined as:
~
c ~
c ~
Z 1
I L ð~Þ ¼
c g L ð yÞdy;
~
c
0
and the right integral value of c is defined as:
~
Z 1
I R ð~Þ ¼
c g R ð yÞdy:
~
c
0
Then, the total integral value with an optimism index u is defined as:
I u ð~Þ ¼ uI R ð~Þ þ ð1 2 uÞI L ð~Þ; u [ ½0; 1Š:
T c c c ð3Þ
The total integral value of a fuzzy performance index is a crisp value and is used to be
the performance score. The performance scores of overall ERP system or the different
objectives can be easy to understand and communicate with others. The trends of these
scores can indicate which parts of the ERP system are in need of resource and attention
for improving the associated performance.
However, a performance score only indicates an absolute position of the adopted
ERP system’s performance, it cannot show a relative perception how well the ERP
system is performing and serving the needs of company. Since linguistic terms can
easily express the condition of the ERP system against each fundamental-objective and
main category and the decision makers use linguistic terms to measure the qualitative
PIs, the decision makers can translate the results into linguistic terms. To avoid losing
some precision to transform cardinal information to ordinal information, this study
directly translates the fuzzy ERP performance index into linguistic terms.
To translate the membership function of a fuzzy number back to linguistic terms is a
rather sophisticated problem. Given the conditions that the interested fuzzy number, the
fuzzy ERP performance index, is convex and normal. In this study, the optimism index
using in the prior fuzzy integral value method (Liou and Wang, 1992, 1994) is applied.

9. The linguistic term set L ¼ {VP, P, F, G, VG}. Then, a fuzzy performance index c ~ An ERP
~ ~ ~
should be the elements of L. Suppose LD ¼ {d1 ; . . . ; dp }; it is a subset of L, where di [ L
be arranged from VP to VG, p denotes the number of linguistic terms in the set L. The
measurement
~
order of the total integral value of di should be: framework
~ ~ ~
I u ðd1 Þ , I u ðd2 Þ , · · · , I u ðdp Þ:
T T T
615
Then there exists a j such that:
~ ~
I u ðdj Þ # I u ð~Þ , I u ðdjþ1 Þ; j ¼ 1; 2; . . . ; p 2 1:
T T c T
Define:
& '
u  Ã
M ¼ min I ð~Þ 2 I u ðdj Þ; I u ð~Þ 2 1 I u ðdj Þ þ I u ðdjþ1 Þ ; I u ð~Þ 2 I u ðdjþ1 Þ :
~ ~ ~ ~
T c T Tc Tc
2 T T T
ð4Þ
The linguistic term translation rules are:
.
if M ¼ I u ð~Þ 2 I u ðdj Þ, the linguistic term is dj ;
T c T
~ ~
u
. I ð~Þ 2 I u ðdjþ1 Þ, the linguistic term is djþ1 ; and
if M ¼ T c ~ ~
T
u u ~ u ~
.
if M ¼ jI T ð~Þ 2 1=2½I T ðdj Þ þ I T ðdjþ1 ÞŠj, the linguistic term is between dj and
c
~
djþ1 .
3.7 Analyze the results and improve the ERP system
The organization can only absorb a limited amount of change during a finite time
period. Changes are an on-going process; successful companies understand this and
encourage their employees to use the system and continue to improve the system. After
assessment, graphs and reports can be built to show the achievement of each
fundamental-objective and show whether the overall ERP system is making progress
or losing ground. By studying the trends of scores, the managers can set meaningful
targets and plans for improvement.
Owing to inevasible changes in the ERP system and its environment, the ERP
performance measurement framework is dynamic. Periodic ERP performance
assessments should be undertaken to provide a basis for the practice of continuous
improvement. Additionally, this framework is conducted whenever the need for a new
PI is realized. The values of v0 and v* about those quantitative PIs are not fixed forever,
i i
they would be changed over time after a cautious discussion of the team.
4. Practical example
The case company used in this study is in the business of various modular microwave
communication systems design, manufacturing, and sale to USA, Europe, and
Mainland China. The sales cycle of exportations and the need to maintain good
customer service put great pressure on the company. The company seeks to maintain
its competitive advantage in the highly dynamic business environment by improving
the effectiveness of its global logistics. Additionally, the legacy IS were disparate. The
fragmented modules and systems limited the efficiency of the company’s operations,
caused much duplication of efforts, and put the business process into turmoil.

10. JMTM Adopting an ERP application was expected to be the logical solution that could replace
19,5 and integrate their legacy IS.
Then, an objective structure of the ERP implementation project including the
fundamental-objective hierarchy and means-objective network has been constructed
during the ERP project implementation phase. There were two major aspects in the
objective structure, namely, the ERP system dimension and the ERP vendor
616 dimension. Figure 2 shows the fundamental-objective hierarchy. For details, readers
can refer to Wei et al. (2005).
After adopting the ERP system, the information managers hoped to know how the
ERP system is currently performing and how it should be performing at a future point
in time. Additionally, they want to justify the success and the value-added contribution
of the ERP system to accomplish the objectives of the ERP system implementation
project. The stepwise procedure is presented in the following.
4.1 Step 1
An ERP performance measurement project team with some members was formed,
including critical managers, IS experts, user representatives and consultants. Five
major managers and the information manager was responsible to formulate the project
plan, integrate the resources, identify the appropriate PIs, develop the consistent
evaluation guideline of each PI and measure the performance of the adopted ERP
system. Other critical user representatives also were selected to form some research
groups to assist the managers in collecting data, offering their use experience and
discussing the detailed evaluation considerations. All managers and user
representatives had experienced the ERP system selection and implementation in
the company.
The objectives of ERP implementation have been developed and discussed in detail
in Wei et al. (2005). The members started from an existed means-objective of a bottom
level fundamental-objective in the objective structure to discuss whether it was
suitable to be a PI following the systematic discussion process. Go through all the
means-objectives, the results of this process were the derivation of a set of PIs that need
to be supported in the performance measurement mechanism.
Significantly, once the ERP implementation project is complete, some
fundamental-objectives and relative critical problems, like project cost and
implementation time, should be examined immediately. However, these objectives
need not to be evaluated again when the ERP system has executed smoothly. Initially,
total 39 PIs converted from the means-objective network were joined into the PI set.
4.2 Step 2
We recommended some additional PIs for which data would need to be collected in an
ERP output view. After surveying the PIs presented by prior literatures and examining
the necessity of these indicators with the members, there were 23 PIs added into the PI
set, and then the number of selected PIs came to 62.
4.3 Step 3
As a result of some reviews, PIs were added, deleted, and revised. Based on the objective
structure of the ERP implementation project, the remaining 34 PIs were constructed a
hierarchy based on the three main categories, system, vendor, and impact factors.

11. An ERP
Price measurement
Minimizing total framework
Maintenance cost
cost
Consultant expense
Minimizing time of 617
Infrastructure cost
implementation
Module completion
Having complete
Function-fitness
function
Security
Having user-friendly Easy to operate
interface and
operations Easy to learn
Choosing
a suitable Upgrade ability
ERP Being excellent system
system flexibility Easy to integrate
Easy to develop
Choosing a in - house
suitable
ERP Being high system Stability
system and reliability
vendor Recovery ability
Financial position
Owning proud
reputation Scale of vendor
Market share ratio
RD capability
Selecting
a good Providing good
Technical support
ERP technical ability
ability
vendor
Implementing ability
Warranties
Supplying satisfying Consulting service
service ability
Training service
Service speed
Figure 2.
Source: Wei et al. (2005) The fundamental
objectives hierarchy

12. JMTM This process of reviewing was repeated until agreement was reached. After discussing,
19,5 the ERP PI hierarchy of this case was shown in Table II. For aligning with the
fundamental-objective hierarchy (Figure 2), the first column indicates the three main
categories, namely, system, vendor, and impact factors. The fundamental-objectives of
each main factor in the objective structure were shows in the third column. From the
knowledge of means-objective network and the prior systematic PI discussion process,
618 the project team identified the corresponding PIs of each fundamental-objective and
listed them in the fifth column.
Main
category Weight Fundamental-objective Weight PI Weight
System 0.540 Module completion 0.220 System completion 0.50
Global task performance 0.50
Function fitness 0.311 Degree of workflow support 0.48
Information timeliness 0.24
Information aggregation 0.18
Frequency of special function
requests 0.11
Security 0.043 System and database protection 0.75
Permission management 0.25
Ease of operation 0.106 UI friendliness 0.50
e-Guidebook usefulness 0.25
Acceptance of reports 0.25
Ease of learning 0.020 Online learning 1.00
Upgradation ability 0.023 Upgrade service performance 1.00
Ease of integration 0.071 Ease of integration with other
systems 0.50
Ease of communication with other
platforms 0.50
Ease of in-house 0.014 Ease of maintenance 0.75
development Ease of modification 0.25
Stability 0.159 Frequency of system error 0.50
Data error rate 0.50
Recovery ability 0.033 Mean recovery time 1.00
Vendor 0.163 Technology support 0.279 Diverse product introduction 1.00
Training support 0.072 Effective training lessons 1.00
Service ability 0.649 Solving problem ability 0.33
Consultant service ability 0.33
Service speed 0.34
Impact 0.297 Organization 0.297 Management enhancement 0.12
Cycle time reduction 0.20
Workflow standardization 0.27
Efficiency of system 0.41
Individual 0.163 Quality of decision making 0.25
Personal productivity improvement 0.59
Employ satisfaction 0.16
Table II. Customer 0.540 Response time to customer 0.33
ERP PI structure On time delivery 0.67

13. 4.4 Step 4 An ERP
Initially, the project team discussed how to measure every PI and how to collect its data measurement
of the ERP system performed. They first investigated what types of measurement
data were already being collected to establish a baseline and determine whether any framework
data existed that could be used to determine the overall success of the adopted ERP
system. Then, the project team reviewed the available information whether this is
currently being collected for PIs or objectives. Additionally, they also paid attention on 619
the reliability of each data, its usefulness, as well as the correspondence with certain PI.
For quantitative PIs, the lowest and maximum values which the members believed
the ERP system should and can perform were set.
On the other hand, for qualitative PIs, the detailed evaluation guidance and an
assessment questionnaire also were developed. For example, Table III presents the PIs’
detailed descriptions of a fundamental-objective, “function fitness.”
The weight of each PI can be determined by direct assignment or indirect pairwise
comparisons. For reducing the loading of the PIs’ importance comparison process, this
case followed the AHP methodology. Paired comparisons of PIs relative importance
were made and converted to a numerical scale of one to nine. The software Expert
Choice was then used to determine the normalized weights. Then, the relative weights
of each main category, fundamental-objective and PI using AHP method are also listed
in the second, fourth and sixth column of Table II, respectively.
4.5 Step 5
The managers measured the current performance of the ERP system to determine the
rating of each PI based on the data gathered by user research groups. For example, in
Table III, for the quantitative PI “frequency of special function requests,” the best
possible number of times (maximum value v* ) and the worst value (minimum value v0 )
i i
were 3 and 50 within a specified timeframe. The current performance rating vi was 16.
By the equation (1), the rating of this quantitative PI was 0.7234. That is:
Fundamental-objective: function fitness
Degree of workflow Information Information Frequency of special
PI suppose timeliness aggregation function requests
Qualitative PI: Qualitative PI: Qualitative PI: Quantitative PI:
PI character average value based average value based average value based number of special
on ratings made in on ratings made in on ratings made in function requests
the linguistic set L the linguistic set L the linguistic set L within specified
timeframe max: 3;
min: 50
Rating G G F 16
Weight 0.48 0.24 0.18 0.11
Fuzzy
performance
index (0.4756, 0.6736, 0.9436)
Score 0.6916
Linguistic Table III.
term G Examples of PI details

14. JMTM r¼
16 2 50
¼ 0:7234
19,5 3 2 50
On the other hand, the members evaluated the performance of the ERP system with
respect to the qualitative PIs by using the linguistic ratings in the scale set L. For
example, Table III shows the measurement result at a certain time about the
620 corresponding PIs of the fundamental-objective “function fitness.” The linguistic
ratings were obtained by assessing the major members through a subjective
assessment process and translated into the fuzzy numbers based on Table I. The
precision with which decision makers could provide measurements was limited by
their knowledge, experience, and even cognitive biases, as well as by the complexity of
the ERP system. Thus, to avoid inconsistency among semantic descriptions and score
assignments to the PIs, it is necessary to train the decision makers to understand the
details, strengths, and limitations of the proposed method. During the evaluation
process, consistency checks were conducted. The decision makers in some cases were
asked to provide reasons and detailed explications to justify and refine their
assessments.
4.6 Step 6
Aggregated the quantitative and qualitative measurements with the corresponding
weights of PIs in Table II to yield the fuzzy performance index of the
fundamental-objective “function fitness” by equation (2):
2 3
ð0:5; 0:7; 1:0Þ
6 7
6 ð0:5; 0:7; 1:0Þ 7
6 7
6 7^½0:48; 0:24; 0:18; 0:11Š ¼ ½0:4756; 0:6736; 0:9436Š:
6 ð0:2; 0:5; 0:8Þ 7
4 5
ð0:7234; 0:7234; 0:7234Þ
The fuzzy performance index of “function fitness” was (0.4756, 0.6736, 0.9436). Assume
c ¼ ð0:4756; 0:6736; 0:9436Þ. Then, its membership function is:
~
8 x20:4756
0:1980 ; 0:4756 # x # 0:6736
1; x ¼ 0:6736
f c ðxÞ ¼ x20:9436
~
20:2700 ; 0:6736 # x # 0:9436
: 0; otherwise
The left integral value of c is defined as:
~
Z 1
I L ð~Þ ¼
c 0:198y þ 0:4756 dy ¼ 0:5746;
0
and the right integral value of c is defined as:
~
Z 1
I R ð~Þ ¼
c 2 0:27y þ 0:9436 dy ¼ 0:8086:
0

15. Then, the total integral value of the fuzzy performance index were obtained by using An ERP
the fuzzy integral value method with u ¼ 0.5 (equation (3)):
measurement
I 0:5 ð~Þ ¼ 0:5 £ 0:5746 þ 0:5 £ 0:8086 ¼ 0:6916:
T c
framework
The integral value 0.6916 was regarded as the performance score of “function fitness.”
Finally, the project team translated the fuzzy performance index back to linguistics.
Since: 621
I 0:5 ðd3 ¼ FÞ ¼ 0:5 , I 0:5 ð~Þ ¼ 0:6916 , I 0:5 ðd4 ¼ GÞ ¼ 0:725;
T
~
T c T
~
then:
M ¼ min{j0:6916 2 0:5j; j0:6916 2 0:6125j; j0:6916 2 0:725j} ¼ 0:0334:
Following the linguistic term translation rules to get M ¼ 0.0334 of rule (2) was
minimum. As d4 ¼ G, the linguistic description of “function fitness” was “Good.”
4.7 Step 7
Went through all the fundamental-objectives by using the proposed fuzzy aggregative
method to obtain their fuzzy performance index and performance scores. Rolled them
up to gain the fuzzy performance index and performance scores of the three main
categories. Following the linguistic term translation rules, the linguistics of all
fundamental-objectives and main categories could be obtained. Using the same
algorithm, the performance score and the linguistic term of the entire ERP system
could be obtained. The final linguistic term of the adopted ERP system performance at
the certain time was “between fair and good.”
We helped them to collect the data and track the performance scores six months
after the ERP performance measurement system establishing. Figure 3 shows the score
trends of the system, vendor, and impact categories. A significant progress on the
system and impact categories of the ERP performance had been made. However,
the scores of ERP vendor indicator category had not improved over time. Figure 4
shows the detailed score records of vendor PIs. Obviously, the fundamental-objectives
1.0
0.8
0.6
score
0.4
0.2
system
vendor
impact Figure 3.
0.0 Score trend of the three
1 2 3 4 5 6 7
main PI categories
month

16. JMTM 1.0
19,5
0.8
622 0.6
score 0.4
0.2 Technology support
Training support
Figure 4. Service ability
Score trend of the 0.0
1 2 3 4 5 6 7
vendor PIs
month
“training support” and “service ability” related PIs had made regression. The
managers hoped that the ERP vendor could provide more support and service to
continuously improve the ERP functions and reports. They decided to strengthen the
relationship with the ERP vendor. A problem feedback mechanism and a solving
problem process were also established immediately with the ERP vendor.
The relative stability of the ERP PI hierarchy is very important. After discussing,
PIs only change if any service aims change, major business processes or system
change, and any PI is found unsatisfactory or needs to add.
5. Conclusion
An ERP system implementation project needs to invest enormous money, labor, and
time for a company. Hence, managers must understand what benefits the system has
contributed and what aspects the system should be improved. The PIs reflect whether
the input resources and efforts in an ERP system implementation project have
achieved the objectives which managers want to gain. This study presents a
framework to measure the performance of an adopted ERP system under fuzzy
environment. The proposed framework developed an ERP PI structure according to the
knowledge of ERP implementation objectives. Since humans are difficult in giving
quantitative ratings exactly, where some PIs are comparatively efficient in linguistic
expressions. An integration model that uses the fuzzy operation and fuzzy integral
method was proposed to obtain a fuzzy ERP performance index. Then, the fuzzy ERP
performance index can be translated into a performance score and back to a linguistic
term. The evaluation results can truly reflect the current situation of the adopted ERP
system and the accomplishment of the ERP implementation objectives.
It must be noted that the evaluation results do really not be used to punish someone
or any department in order to avoid the resistance and misunderstanding of employees.
The results point out the functionality and service of the ERP system can be trusted
and the high-system performance standards can be maintained. The key point is how
to improve the performance of ERP system. The PIs are also aligned with the objective

17. structure of the ERP system implementation and the framework can ensure the An ERP
inclusion of the concept of continuous improvement. measurement
The proposed framework offers the following advantages in the ERP performance
measurement processes for the companies: framework
.
It provides a comprehensive and systematic method to extend the objectives of
an ERP implementation project to suitable PIs of an ERP performance
measurement mechanism. Managers can easily assess the achievement of the 623
ERP implementation objectives by following the stepwise procedure.
.
The proposed algorithm considers not only quantitative data but also linguistic
data. Managers can assess the performance of their adopted ERP system against
various PIs, particularly in an ill-defined situation, by using linguistic or
quantitative values in the ERP performance evaluation.
. The fuzzy ERP performance index can be translated back into to linguistic terms.
The linguistic results provide a semantic and impressional description about the
current condition of the ERP system.
.
Additionally, the fuzzy ERP performance index can be calculated to obtain a
crisp score. The trends of ERP performance scores of each main category,
fundamental-objective and PI can indicate whether the system’s performance is
enhancing or descending over time. Managers can recognize the directions of
ERP system improvement and the strategies of corporate IS in the future.
.
The proposed framework can also be applied to other enterprise information
systems (EIS) performance evaluation problems. However, because the
characteristics and roles of various EIS are different in a company, the
framework should be revised as it is applied to other EIS.
References
Buckley, J.J. (1985), “Fuzzy hierarchical analysis”, Fuzzy Sets and Systems, Vol. 17, pp. 233-47.
Chan, D.C.K., Yung, K.L. and Ip, A.W.H. (2002), “An application of fuzzy sets to process
performance evaluation”, Integrated Manufacturing Systems, Vol. 13 No. 4, pp. 237-46.
Chan, F.T.S. and Kumar, N. (2007), “Global supplier development considering risk factors using
fuzzy extended AHP-based approach”, Omega, Vol. 35 No. 4, pp. 417-31.
Chan, F.T.S., Chan, H.K., Lau, H.C.W. and Ip, R.W.L. (2006), “An AHP approach in benchmarking
logistics performance of the postal industry”, International Journal of Benchmarking,
Vol. 13 No. 6, pp. 636-61.
Chang, P.L. and Chen, Y.C. (1994), “A fuzzy multi-criteria decision making method for technology
transfer strategy selection in biotechnology”, Fuzzy Sets and Systems, Vol. 63, pp. 131-9.
Chang, S.L., Wang, R.C. and Wang, S.Y. (2007), “Applying a direct multi-granularity linguistic
and strategy-oriented aggregation approach on the assessment of supply performance”,
European Journal of Operational Research, Vol. 177 Nos 2/1, pp. 1013-25.
Chen, S.J. and Hwang, C.L. (1992), Fuzzy Multiple Attribute Decision Making: Methods and
Applications, Springer-Verlag, New York, NY.
Chen, Z. and Ben-Arieh, D. (2006), “On the fusion of multi-granularity linguistic label sets in
group decision making”, Computers Industrial Engineering, Vol. 51 No. 3, pp. 526-41.
Clemen, R.T. (1996), Making Hard Decisions: An Introduction to Decision Analysis, Duxbury
Press, Pacific Grove, CA.

18. JMTM Delone, W.H. and McLean, E.R. (1992), “Information systems success: the quest for the dependent
variable”, Information Systems Research, Vol. 3, pp. 60-95.
19,5
Doll, W.J. and Torkzadeh, G. (1988), “The measurement of end-user computing satisfaction”, MIS
Quarterly, Vol. 12 No. 2, pp. 259-74.
Dubois, D. and Prade, H. (1978), “Operations on fuzzy numbers”, International Journal of Systems
Science, Vol. 9, pp. 613-26.
624 Hagood, W.O. and Friedman, L. (2002), “Using the balanced scorecard to measure the
performance of your HR information system”, Public Personnel Management, Vol. 31 No. 4,
pp. 543-57.
Irani, Z. (1999), “IT/IS investment decision making”, Logistics and Information Management,
Vol. 12 No. 1, pp. 8-11.
Jain, V., Tiwari, M.K. and Chan, F.T.S. (2004), “Evaluation of the supplier performance using an
evolutionary fuzzy-based approach”, Journal of Manufacturing Technology Management,
Vol. 15 No. 8, pp. 735-44.
Kaufmann, A. and Gupta, M.M. (1991), Introduction to Fuzzy Arithmetic: Theory and Application,
Van Nostrand Reinhold, New York, NY.
Keeney, R.L. and Raiffa, H. (1993), Decisions with Multiple Objectives: Preferences and Value
Tradeoffs, Cambridge University Press, New York, NY.
Kivijarvi, H. and Saarinen, T. (1995), “Investment in information systems and the financial
performance of the firm”, Information Management, Vol. 28, pp. 143-63.
Klenke, K. (1992), “Construct and critique of user satisfaction and user involvement instructions”,
Information, Vol. 3 No. 4, pp. 325-48.
Lee, Y.W., Strong, D.M., Kahn, B.K. and Wang, R.Y. (2002), “AIMQ: a methodology for
information quality assessment”, Information Management, Vol. 40, pp. 133-46.
Liang, G.S. and Wang, M.J.J. (1994), “Personnel selection using fuzzy MCDM algorithm”,
European Journal of Operational Research, Vol. 78, pp. 22-33.
Liou, T.S. and Wang, M.J.J. (1992), “Ranking fuzzy number with integral value”, Fuzzy Sets and
Systems, Vol. 50, pp. 247-55.
Liou, T.S. and Wang, M.J.J. (1994), “Subjective assessment of mental workload – a fuzzy
linguistic multi-criteria approach”, Fuzzy Sets and Systems, Vol. 62, pp. 155-65.
Mashari, M.A., Mudimigh, A.A. and Zairi, M. (2003), “Enterprise resource planning: a taxonomy
of critical factors”, European Journal of Operational Research, Vol. 146, pp. 352-64.
Michael, R. and Jens, W. (1999), “Measuring the performance of ERP software: a balanced
scorecard approach”, Proceeding of the 10th Australasian Conference on Information
Systems, pp. 773-84.
Murphy, K.E. and Simon, S.J. (2001), “Using cost benefit analysis for enterprise resource planning
project evaluation: a case for including intangibles”, Proceedings of the 34th Hawaii
International Conference on System Sciences, pp. 1-11.
Ohdar, R. and Ray, P.K. (2004), “Performance measurement and evaluation of suppliers in supply
chain: an evolutionary fuzzy-based approach”, Journal of Manufacturing Technology
Management, Vol. 15 No. 8, pp. 723-34.
Omero, M., D’Ambrosio, L., Pesenti, R. and Ukovich, W. (2005), “Multiple-attribute decision
support system based on fuzzy logic for performance assessment”, European Journal of
Operational Research, Vol. 160, pp. 710-25.

19. Palvia, S.C., Sharma, R.S. and Conrath, D.W. (2001), “A socio-technical framework for quality An ERP
assessment of computer information systems”, Industrial Management Data Systems,
Vol. 101 No. 5, pp. 237-51. measurement
Saarinen, T. (1996), “An expanded instrument for evaluating information system success”, framework
Information Management, Vol. 31, pp. 103-18.
Saaty, T.L. (1980), The Analytic Hierarchy Process, McGraw-Hill, New York, NY.
Shamsuzzaman, M., Sharif Ullah, A.M.M. and Bohez, E.L.J. (2003), “Applying linguistic criteria in 625
FMS selection: fuzzy-set-AHP approach”, Integrated Manufacturing Systems, Vol. 14 No. 3,
pp. 247-54.
Sharif Ullah, A.M.M. (2005), “A fuzzy decision model for conceptual design”, Systems
Engineering, Vol. 8 No. 4, pp. 296-308.
Skok, W., Kophamel, A. and Richardson, I. (2001), “Diagnosing information system success:
importance-performance maps in the health club industry”, Information Management,
Vol. 38, pp. 409-19.
Stensrud, E. and Myrtveit, I. (2003), “Identifying high performance ERP projects”, IEEE
Transaction on Software Engineering, Vol. 29 No. 5, pp. 398-416.
Wei, C.C. and Wang, M.J.J. (2004), “A comprehensive framework for selecting an ERP system”,
International Journal of Project Management, Vol. 22, pp. 161-9.
Wei, C.C., Chien, C.F. and Wang, M.J.J. (2005), “An AHP-based approach to ERP system
selection”, International Journal of Production Economics, Vol. 96, pp. 47-62.
Wu, J.H., Wang, Y.M., Chang-Chien, M.C. and Tai, W.C. (2002), “An examination of ERP user
satisfaction in Taiwan”, Proceedings of the 35th Hawaii International Conference on
System Sciences.
Zadeh, L.A. (1965), “Fuzzy sets”, Information and Control, Vol. 8, pp. 338-53.
Appendix
Fuzzy set theory was developed by Zadeh (1965). Some definitions of fuzzy sets, TFNs and
linguistic variables introduced by Dubois and Prade (1978), Buckley (1985) and Kaufmann and
Gupta (1991) are applied throughout this paper and illustrated as below.
~
Definition 1. In a universe of discourse X, a fuzzy set A of X is characterized by a
membership function uA ðxÞ which associates with each element x in X a real number in the interval
~
[0, 1]. The function value uA ðxÞ represents the grade of membership of x in A.
~
~
Definition 2. A fuzzy number A ~ is described as a fuzzy subset of discourse X, whose
membership function uA ðxÞ specifies a mapping from R to a closed interval [0, 1]. A fuzzy
~
number has the following characteristics:
.
uA ðxÞ ¼ 0; ;x [ ð21; aŠ ½d; 1Þ;
~
.
uA ðxÞ is strictly increasing on [a, b ] and strictly decreasing on [g, d ]; and
~
.
uA ðxÞ ¼ 1; ;x [ ½b; gŠ:
~
Definition 3. ~
A fuzzy number A is a TFN if its membership function uA is given by:
~
8
ðx 2 aÞ=ðb 2 aÞ; a # x # b;
1; b # x # c;
uA ðxÞ ¼ ðx 2 cÞ=ðb 2 cÞ; c # x # d;
~
: 0; otherwise
~
The TFN A can be denoted by (a, b, c).

20. JMTM By the extension principle, the fuzzy sum % and fuzzy subtraction * of any two TFNs are
also TFNs. But the multiplication ^ of any two TFNs is only an approximate TFN. That is, if
19,5 ~ ~
A1 ¼ ða1 ; b1 ; c1 Þ and A2 ¼ ða2 ; b2 ; c2 Þ then:
~ ~
A1 %A2 ¼ ða1 þ a2 ; b1 þ b2 ; c1 þ c2 Þ;
~ ~
A1 *A2 ¼ ða1 þ a2 ; b1 þ b2 ; c1 þ c2 Þ;
626
~ ~
A1 ^A2 ø ða1 a2 ; b1 b2 ; c1 c2 Þ;
~
k^A ¼ ðka; kb; kcÞ; k [ R:
Corresponding author
Chun-Chin Wei can be contacted at: d887801@cyu.edu.tw
To purchase reprints of this article please e-mail: reprints@emeraldinsight.com
Or visit our web site for further details: www.emeraldinsight.com/reprints