1. An Erp Performance Measurement Framework Using A Fuzzy Integral Approach


Published on

Published in: Technology, Business
1 Comment
  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

1. An Erp Performance Measurement Framework Using A Fuzzy Integral Approach

  1. 1. The current issue and full text archive of this journal is available at www.emeraldinsight.com/1741-038X.htm An ERP An ERP performance measurement measurement framework framework using a fuzzy integral approach 607 Chun-Chin Wei Department of Industrial Engineering and Management, Ching Yun University, Received December 2006 Chung Li, Taiwan, Republic of China Revised July 2007 Accepted September 2007 Tian-Shy Liou Department of Business Administration, Chen Shiu University, Niaosong, Taiwan, Republic of China, and Kuo-Liang Lee Department of Industrial Engineering and Management, Ching Yun University, Chung Li, Taiwan, Republic of China Abstract Purpose – The purpose of this paper is to propose a comprehensive framework for measuring the performance of an enterprise resource planning (ERP) system to survey suitable performance indicators (PIs) according to knowledge of the ERP implementation objectives set up at the implementation phase and build consistent measurement standards for facilitating the complex ERP performance evaluation process. Design/methodology/approach – A seven-step ERP performance measurement framework based on the objectives of ERP implementation is proposed. A fuzzy ERP performance index is used to account for the ambiguities involved in evaluating the performance of the ERP system. The fuzzy ERP performance index can be translated first into simple scores and then back to linguistic terms. An actual example in Taiwan demonstrates the feasibility of applying the proposed framework. Findings – The findings indicate that the PIs of ERP performance measurement should align with the objectives of ERP implementation. The assessment results can represent the achievement of these objectives and the directions for improving the adopted ERP system. Originality/value – This study may be interesting to some academic researchers and practical managers. The proposed framework can provide a procedure to link the objectives identified in the ERP system implementation phase and the performance considerations in the ERP use phase. Keywords Manufacturing resource planning, Fuzzy control, Decision theory, Performance measurement (quality) Paper type Research paper 1. Introduction Owing to the highly severe market competition and the immense impact of advances in information technology progress, a number of companies have widely implemented the enterprise resource planning (ERP) systems. A comprehensive ERP system Journal of Manufacturing Technology implementation project involves selecting an ERP software system and a cooperative Management Vol. 19 No. 5, 2008 pp. 607-626 q Emerald Group Publishing Limited The authors would like to thank the National Science Council of the Republic of China, Taiwan 1741-038X for financially supporting this research under Contract No. NSC 94-2213-E-231-005. DOI 10.1108/17410380810877285
  2. 2. JMTM vendor, implementing the selected system, managing business processes change, and 19,5 examining the practicality of the adopted ERP system (Wei and Wang, 2004). That is, completing ERP system implementation is not the final stop but a go live start. One of the most significant challenges faced by information managers today is measuring the performance of the adopted ERP system to justify its value-added contribution for accomplishing the organization’s missions. Furthermore, managers would also like to 608 know which parts of their ERP system need to improve and whether the system’s overall performance is enhancing over time. Success has often been defined as a favorable or satisfactory result or outcome (Saarinen, 1996). In reality, “the success of an ERP system” is achieved when the organization is able to better perform all its business processes and when the adopted ERP system really achieves the objectives that managers strive. That is, the development of ERP performance measurement process should establish a feedback mechanism between the desired objectives of ERP adoption and the substantial effects of ERP execution (Mashari et al., 2003). Traditionally, a set of performance indicators (PIs) is employed to determine the effectiveness and efficiency of an ERP system. The key is to build up a process for determining the relationships between the objectives of the ERP implementation project and the ERP PIs for measuring its performance, so that they have identical guidance and evaluation standards during the entire project period. Typically, there are many factors with many characteristics to consider in the ERP performance evaluation: tangible, intangible, quantitative and qualitative. The post-usage perception of an ERP system to a user is a subjective interaction. Personal evaluation differs from one user to another depending on individual variance of personal subjectivity, experience, and cognition. Furthermore, for many people, the evaluation of a qualitative PI is a subjective and ambiguous concept hard to be expressed, and not all people can concretely voice out their feelings on a scale of one to five. The evaluators often express their ratings in natural language rather than in numbers. The concept of a linguistic variable is very useful in dealing with situations that are too ill-defined to be reasonably described in conventional quantitative expressions (Chen and Hwang, 1992). Fuzzy set theory is developed for solving problems in which descriptions of activities and observations are imprecise, vague, and uncertain and widely used in the decision analysis problems, like selection (Liang and Wang, 1994; Shamsuzzaman et al., 2003; Wei and Wang, 2004; Sharif Ullah, 2005; Chen and Ben-Arieh, 2006) and performance assessment (Chan et al., 2002; Jain et al., 2004; Ohdar and Ray, 2004; Chang et al., 2007). Thus, a fuzzy aggregative method is highly effective in integrating linguistic assessments and weights to measure the performance of an ERP system. This paper aims to construct an ERP performance measurement framework to elaborate the process of PI development for linking with the ERP implementation objectives. According to the knowledge of the ERP implementation objectives, decision makers can extend them to suitable PIs for measuring whether the objectives have been achieved. A fuzzy ERP performance index is used to account for the ambiguities involved in evaluating the performance of an ERP system. A method of translating the fuzzy ERP performance index back to linguistics is also used to obtain the linguistic achievement representation of the ERP implementation objectives and the overall ERP system. An empirical case in Taiwan is described to demonstrate the practical viability of the proposed method.
  3. 3. 2. Method review An ERP Several methods have been proposed for measuring the performance of ERP systems measurement or other information systems (IS). Traditionally, financial performance metrics such as return on investment, net present value, or payback period could be used (Kivijarvi and framework Saarinen, 1995; Murphy and Simon, 2001), but because of the unique nature of the IS investment, they seldom suffice in practice. Instead, the evaluation of IS success has to be supplemented by a subjective judgment and surrogate measures. 609 The system and data quality assessment of IS have been widely studied (Delone and McLean, 1992; Palvia et al., 2001; Lee et al., 2002). The quality measurement reflects the engineering-oriented performance characteristics of the system itself and the quality of information and data. Data quality focuses on the IS output, namely, the quality of the information that the system produces. Later, numerous information quality measures have been included within the area of “User satisfaction.” Information technologies cannot by itself influence the productivity of a company. The main efficiency factor lies in the way people use these technologies. Related studies about user satisfaction evaluated the IS performance using the experience and perspective of various users, like employees, middle managers, top managers and system engineers (Wu et al., 2002). Some IS user satisfaction measurement questionnaires and methods have also been applied to real cases (Doll and Torkzadeh, 1988; Klenke, 1992; Saarinen, 1996; Wu et al., 2002). Recently, some popular techniques have been used to measure the performance of ERP systems or other IS, like analytic hierarchy process (AHP) (Chan et al., 2006; Chan and Kumar, 2007), data envelopment analysis (Stensrud and Myrtveit, 2003), importance-performance maps (Skok et al., 2001), and balanced scorecard (Michael and Jens, 1999; Hagood and Friedman, 2002). These reports integrated the traditional PIs with new techniques to build up performance measurement systems and offered some useful applications in practice. Many researchers stated that there is no best appraisal technique that addresses all project considerations (Saarinen, 1996; Irani, 1999). Further, they argued that the reason for this is the investments in IS are aggregates of complexity, and notably different from each other. However, the most frequently adopted measures are to refer to the common indices without developing tailor-made measures that echo the objectives of ERP implementation for a specific company’s ERP system. Additionally, little research has addressed the relationship between the ERP implementation stage and the ERP use stage. This study develops a framework with fuzzy set theory to synthesize managers’ tangible and intangible measures with respect to numerous PIs extended from the objectives of ERP implementation to obtain an aggregated fuzzy ERP performance index. The framework also can translate the fuzzy ERP performance index into simple scores and then back to linguistic terms for indicating how the adopted ERP system is performing and what actions the managers should undertake to improve the ERP system. 3. Procedure for measuring the ERP performance Three principal themes are noted in the proposed ERP performance measurement framework, including the PI structure construction, fuzzy group ERP performance measurement, and result analysis and system improvement. To clearly present the proposed ERP performance measurement framework, a step-wise procedure is first described:
  4. 4. JMTM (1) extend the objectives of the ERP implementation project to appropriate PIs; 19,5 (2) add other crucial PIs into the PI set in an ERP output view; (3) construct the proper PI structure; (4) develop the detailed performance assessment method; (5) assess the performance of the adopted ERP system; 610 (6) aggregate the assessments to determine the fuzzy ERP performance index; and (7) analyze the results and improve the ERP system. Figure 1 shows the flowchart of the proposed ERP performance measurement framework. The details of each step are presented below. 3.1 Extend the objectives of the ERP implementation project to appropriate PIs Clearly defined objectives were identified as the most important key to success. The ERP implementation objectives generally indicate the direction in which the managers should strive to do better. For evaluating ERP performance, it is important to Extend the ERP implementation objectives to performance indicators Discuss “How to No Can this means-objective PI structure evaluate whether the be taken as a suitable construction means-objective has performance indicator? been achieved?” Yes Generate a performance indicator Add other crucial performance indicators Construct the performance indicator structure Fuzzy group ERP Develop the detailed performance evaluation contents performance measurement Assess the performance indicators Calculate the fuzzy ERP performance index Result analysis and Figure 1. system improvement ERP performance measurement framework Analyze the results and improve the ERP system
  5. 5. incorporate appropriate measures that are linked to the ERP system’s role and the An ERP objectives of the ERP implementation project. The decision makers should transform measurement the objectives into the suitable ERP PIs to link up the input factors of an ERP implementation project with the output performance factors and indicate the gap framework between what the managers want and what the ERP system performs. The objectives of ERP implementation development method can refer to Wei et al. (2005). The first step is to form an ERP performance measurement project team involving 611 critical managers, user representatives, system experts and consultants. Critical managers formulate an ERP system performance assessment plan, identify suitable PIs and develop consistent evaluation guidance. User representatives from different departments in the team can be divided into research groups to gather and offer managers the ERP system data based on their specialties and job responsibility. Initially, the team members should extract the PIs to form a PI set from the objective structure which has been established in the ERP implementation process. There are two kinds of objectives in the ERP implementation objective structure (Wei et al., 2005). The fundamental-objectives in the objective structure are those that are important to specify the goal of the ERP implementation. They point out why the managers care about the selection situation and what criteria the managers should be reviewing in the alternatives (Clemen, 1996). Additionally, the means-objectives in the objective structure highlight how to accomplish the desired fundamental-objectives. Having sorted out them, the team members can rest assured that the team will be able to evaluate alternatives whose performances are consistent with the company’s concerns. Based on the definitions, this study finds that the fundamental-objectives indicate the directions of ERP performance evaluation. Some means-objectives are suitable to be PIs to evaluate whether the fundamental-objectives have been accomplished as promised. We can start from a means-objective in the means-objective network to discuss whether it can be used to demonstrate a PI. After discussing, if the means-objective is a suitable PI, then add it into the PI set. If it is not a suitable PI, the team members can discuss, “How to evaluate whether this means-objective has been achieved?” The answers can reveal some more detailed and new PIs. Add them to the PI set. If the PI cannot completely evaluate the achievement of its corresponding means-objective, members need to survey additional PIs to complement the PI. Go through all means-objectives in the ERP objective structure, we can formulate an initial ERP PI set. 3.2 Add other crucial PIs into the PI set in an ERP output view Whereas the initial PI set is expanded from the objectives of ERP implementation, the set cannot entirely involve all PIs which are used to measure the ERP system performance. The team members should survey some proper PIs based on the output performance aspects of ERP system execution, like the impact of individual and organization. Then, these critical PIs can be added into the PI set. 3.3 Construct the PI structure Since the adopted ERP system is continuously working and improving over time and across the organization in a complex exercise, the measurement effectiveness cannot be simplified and understood from a single aspect only. After surveying the PIs, the team members should organize them into a hierarchy to conduce the data analysis
  6. 6. JMTM in performance evaluation process. Structuring the PIs means organizing them so that 19,5 they describe in detail what the team members want to achieve and can be incorporated in a proper method into the evaluation model. Additionally, a systematic PI structure can guide the directions of ERP system improvement. In order to be compatible with the ERP objective structure and consider the impact of individual and organization, we classify the PIs into three main categories: 612 (1) System factors – indicators for evaluating the utilization of the ERP system. (2) Vendor factors – indicators for assessing the performance of the ERP vendor. (3) Impact factors – the impact of information on the organizational performance and individual. The team can review the indicators in the PI set and put them into perspective, the three main categories, system, vendor, and impact factors. A certain degree of arbitrariness may occur in some indicator classification, because they do not surely fit into any one category or fit into several. If a PI is developed from the ERP implementation objective structure, this PI would be classified into the same main category as the corresponding objective belongs to. If the PI is not extracted from the objective structure, team members must discuss which category the PI should put into. For reducing duplicate and long-term discussions, the PI classification is well while most of the members can achieve the consensus on the classification. And the group discussion and classification can decrease the deviation of individual opinion. Since too many indicators would make numerous evaluations, the process may become very inefficient. The team should iteratively examine and modify the hierarchy of selected PIs so that they are complete, decomposable, non-redundant, measurable and minimal (Keeney and Raiffa, 1993). After specifying the PI hierarchy, they may find themselves refining the context and modifying the performance evaluation process. Refining the context several times and iterating through the corresponding set of indicators are not sighs of poor decision making. They indicate that the decision situation is being taken seriously, and that many different possibilities and perspectives are being considered. 3.4 Develop the detailed performance measurement guidance A PI is a measurable item whose value reflects the degree of achievement for a particular fundamental-objective or an impact. It is important to have an explicit knowledge and understanding of how a PI is measured. The members should investigate what types of data they need to collect and how to collect the data for evaluating each indicator. A standard form can help them to collect the data and conduct the performance assessment. Additionally, the knowledge of the objective structure cannot only help in identifying the PIs, but also the knowledge of the objectives indicates how outcomes must be measured and what kinds of uncertainties should be considered. The team also can examine the suitability of PIs in the PI hierarchy when they discuss the detailed contents of every PI. If they find any problems of PIs, they can revise the PI hierarchy. After developing the detailed performance measurement guidance of PIs, weightings associated with PIs can be assigning. The weight of each PI can be determined by direct assignment or indirect pairwise comparisons like the AHP
  7. 7. (Chang and Chen, 1994; Saaty, 1980). Then, we can obtain a weighting vector, W. The An ERP values in vector W have the domain range (0, 1). measurement 3.5 Assess the PIs framework Even some PIs can be easily quantified, it is possible that the rest of the majority may be hardly measured. The quantitative indicators are evaluated using marginal value function in terms of direct and inverse linear relationship. The rating rises as the value 613 of the PI rises in direct relationship. Contrarily, in inverse relationship, rating rises as the value of PI lowers. A baseline of each PI which the team members hope to achieve can be setting. Then, the team members can easily analyze the gap in what is being collected the ERP system was performing versus what they want to achieve. Define: ðvi 2 v0 Þ i ri ¼ : ð1Þ ðv* 2 v0 Þ i i where vi is the value of PI i which the evaluators assess the performance of current ERP system is performing. v0 is the worst value of PI i which the team believes the ERP i system should perform. v* is the maximum value of PI i which the team expects the i best possible performance they believed that the ERP system might achieve. Then, ri (0 # ri # 1) denotes a dimensionless value to ensure that the value is compatible with the linguistic ratings of the qualitative PIs. Assume that the crisp rating of ri is r, its triangular fuzzy number (TFN) is (r, r, r). On the other hand, the members assess the qualitative PIs using a simple rating questionnaire or form to rate each PI. Subjective assessments are given in linguistic terms to determine the degree of the adopted ERP system performing against qualitative PIs. Linguistic terms have been found intuitively easy to use in expressing the subjectiveness and imprecision of the decision makers’ assessments (Omero et al., 2005; Chang and Chen, 1994; Liou and Wang, 1994). Then, linguistic terms must first be transformed into fuzzy numbers by using appropriate conversion scale. To facilitate the making of subjective assessments in evaluating the qualitative PIs’ performance, a numerical approximation system proposed by Chen and Hwang (1992) is used to systematically convert linguistic terms to their corresponding fuzzy numbers. L ¼ {VP, P, F, G, VG}, VP – very poor, P – poor, F – fair, G – good, and VG – very good. Table I specifies the TFNs for these linguistic values. If some decision makers do not agree with the assumed numerical approximation system, they can define their own ratings and the corresponding TFNs to express the individual perception of the linguistic terms. Since the values of the quantitative PIs are converted into dimensionless ratings, the ratings Rating TFN Very poor (0, 0, 0.3) Poor (0, 0.3, 0.5) Table I. Fair (0.2, 0.5, 0.8) Linguistic variables Good (0.5, 0.7, 1.0) describing values Very good (0.7, 1.0, 1.0) of ratings
  8. 8. JMTM ~ are compatible with the ratings of the qualitative PIs. A fuzzy vector R of PI ratings can 19,5 be obtained combined the both quantitative and qualitative indicators. 3.6 Aggregate the assessments to determine the fuzzy ERP performance index Define: ~ ~ S ¼ R^W T ð2Þ 614 ~ Based on the extension principle, the values in the fuzzy vector S are still TFNs. For each corresponding fundamental-objective, a fuzzy performance index can be obtained. Then, roll them up into the fuzzy ERP performance index of each main category and the entire system using equation (2). A score is easy for the managers to understand and communicate to each other. In this study, a fuzzy integral value method with an optimism index proposed by Liou and Wang (1992, 1994) is applied. Suppose the fuzzy performance index of a fundamental-objective or the entire system is c with the left membership function f L and the right membership function f R ~ ~ c ~ c divided by the highest membership value 1. Define that g L and g R are the inverse ~ c ~ c functions of f L and f R , respectively. Then the left integral value of c is defined as: ~ c ~ c ~ Z 1 I L ð~Þ ¼ c g L ð yÞdy; ~ c 0 and the right integral value of c is defined as: ~ Z 1 I R ð~Þ ¼ c g R ð yÞdy: ~ c 0 Then, the total integral value with an optimism index u is defined as: I u ð~Þ ¼ uI R ð~Þ þ ð1 2 uÞI L ð~Þ; u [ ½0; 1Š: T c c c ð3Þ The total integral value of a fuzzy performance index is a crisp value and is used to be the performance score. The performance scores of overall ERP system or the different objectives can be easy to understand and communicate with others. The trends of these scores can indicate which parts of the ERP system are in need of resource and attention for improving the associated performance. However, a performance score only indicates an absolute position of the adopted ERP system’s performance, it cannot show a relative perception how well the ERP system is performing and serving the needs of company. Since linguistic terms can easily express the condition of the ERP system against each fundamental-objective and main category and the decision makers use linguistic terms to measure the qualitative PIs, the decision makers can translate the results into linguistic terms. To avoid losing some precision to transform cardinal information to ordinal information, this study directly translates the fuzzy ERP performance index into linguistic terms. To translate the membership function of a fuzzy number back to linguistic terms is a rather sophisticated problem. Given the conditions that the interested fuzzy number, the fuzzy ERP performance index, is convex and normal. In this study, the optimism index using in the prior fuzzy integral value method (Liou and Wang, 1992, 1994) is applied.
  9. 9. The linguistic term set L ¼ {VP, P, F, G, VG}. Then, a fuzzy performance index c ~ An ERP ~ ~ ~ should be the elements of L. Suppose LD ¼ {d1 ; . . . ; dp }; it is a subset of L, where di [ L be arranged from VP to VG, p denotes the number of linguistic terms in the set L. The measurement ~ order of the total integral value of di should be: framework ~ ~ ~ I u ðd1 Þ , I u ðd2 Þ , · · · , I u ðdp Þ: T T T 615 Then there exists a j such that: ~ ~ I u ðdj Þ # I u ð~Þ , I u ðdjþ1 Þ; j ¼ 1; 2; . . . ; p 2 1: T T c T Define: & ' u  à M ¼ min I ð~Þ 2 I u ðdj Þ; I u ð~Þ 2 1 I u ðdj Þ þ I u ðdjþ1 Þ ; I u ð~Þ 2 I u ðdjþ1 Þ : ~ ~ ~ ~ T c T Tc Tc 2 T T T ð4Þ The linguistic term translation rules are: . if M ¼ I u ð~Þ 2 I u ðdj Þ, the linguistic term is dj ; T c T ~ ~ u . I ð~Þ 2 I u ðdjþ1 Þ, the linguistic term is djþ1 ; and if M ¼ T c ~ ~ T u u ~ u ~ . if M ¼ jI T ð~Þ 2 1=2½I T ðdj Þ þ I T ðdjþ1 ފj, the linguistic term is between dj and c ~ djþ1 . 3.7 Analyze the results and improve the ERP system The organization can only absorb a limited amount of change during a finite time period. Changes are an on-going process; successful companies understand this and encourage their employees to use the system and continue to improve the system. After assessment, graphs and reports can be built to show the achievement of each fundamental-objective and show whether the overall ERP system is making progress or losing ground. By studying the trends of scores, the managers can set meaningful targets and plans for improvement. Owing to inevasible changes in the ERP system and its environment, the ERP performance measurement framework is dynamic. Periodic ERP performance assessments should be undertaken to provide a basis for the practice of continuous improvement. Additionally, this framework is conducted whenever the need for a new PI is realized. The values of v0 and v* about those quantitative PIs are not fixed forever, i i they would be changed over time after a cautious discussion of the team. 4. Practical example The case company used in this study is in the business of various modular microwave communication systems design, manufacturing, and sale to USA, Europe, and Mainland China. The sales cycle of exportations and the need to maintain good customer service put great pressure on the company. The company seeks to maintain its competitive advantage in the highly dynamic business environment by improving the effectiveness of its global logistics. Additionally, the legacy IS were disparate. The fragmented modules and systems limited the efficiency of the company’s operations, caused much duplication of efforts, and put the business process into turmoil.
  10. 10. JMTM Adopting an ERP application was expected to be the logical solution that could replace 19,5 and integrate their legacy IS. Then, an objective structure of the ERP implementation project including the fundamental-objective hierarchy and means-objective network has been constructed during the ERP project implementation phase. There were two major aspects in the objective structure, namely, the ERP system dimension and the ERP vendor 616 dimension. Figure 2 shows the fundamental-objective hierarchy. For details, readers can refer to Wei et al. (2005). After adopting the ERP system, the information managers hoped to know how the ERP system is currently performing and how it should be performing at a future point in time. Additionally, they want to justify the success and the value-added contribution of the ERP system to accomplish the objectives of the ERP system implementation project. The stepwise procedure is presented in the following. 4.1 Step 1 An ERP performance measurement project team with some members was formed, including critical managers, IS experts, user representatives and consultants. Five major managers and the information manager was responsible to formulate the project plan, integrate the resources, identify the appropriate PIs, develop the consistent evaluation guideline of each PI and measure the performance of the adopted ERP system. Other critical user representatives also were selected to form some research groups to assist the managers in collecting data, offering their use experience and discussing the detailed evaluation considerations. All managers and user representatives had experienced the ERP system selection and implementation in the company. The objectives of ERP implementation have been developed and discussed in detail in Wei et al. (2005). The members started from an existed means-objective of a bottom level fundamental-objective in the objective structure to discuss whether it was suitable to be a PI following the systematic discussion process. Go through all the means-objectives, the results of this process were the derivation of a set of PIs that need to be supported in the performance measurement mechanism. Significantly, once the ERP implementation project is complete, some fundamental-objectives and relative critical problems, like project cost and implementation time, should be examined immediately. However, these objectives need not to be evaluated again when the ERP system has executed smoothly. Initially, total 39 PIs converted from the means-objective network were joined into the PI set. 4.2 Step 2 We recommended some additional PIs for which data would need to be collected in an ERP output view. After surveying the PIs presented by prior literatures and examining the necessity of these indicators with the members, there were 23 PIs added into the PI set, and then the number of selected PIs came to 62. 4.3 Step 3 As a result of some reviews, PIs were added, deleted, and revised. Based on the objective structure of the ERP implementation project, the remaining 34 PIs were constructed a hierarchy based on the three main categories, system, vendor, and impact factors.
  11. 11. An ERP Price measurement Minimizing total framework Maintenance cost cost Consultant expense Minimizing time of 617 Infrastructure cost implementation Module completion Having complete Function-fitness function Security Having user-friendly Easy to operate interface and operations Easy to learn Choosing a suitable Upgrade ability ERP Being excellent system system flexibility Easy to integrate Easy to develop Choosing a in - house suitable ERP Being high system Stability system and reliability vendor Recovery ability Financial position Owning proud reputation Scale of vendor Market share ratio RD capability Selecting a good Providing good Technical support ERP technical ability ability vendor Implementing ability Warranties Supplying satisfying Consulting service service ability Training service Service speed Figure 2. Source: Wei et al. (2005) The fundamental objectives hierarchy
  12. 12. JMTM This process of reviewing was repeated until agreement was reached. After discussing, 19,5 the ERP PI hierarchy of this case was shown in Table II. For aligning with the fundamental-objective hierarchy (Figure 2), the first column indicates the three main categories, namely, system, vendor, and impact factors. The fundamental-objectives of each main factor in the objective structure were shows in the third column. From the knowledge of means-objective network and the prior systematic PI discussion process, 618 the project team identified the corresponding PIs of each fundamental-objective and listed them in the fifth column. Main category Weight Fundamental-objective Weight PI Weight System 0.540 Module completion 0.220 System completion 0.50 Global task performance 0.50 Function fitness 0.311 Degree of workflow support 0.48 Information timeliness 0.24 Information aggregation 0.18 Frequency of special function requests 0.11 Security 0.043 System and database protection 0.75 Permission management 0.25 Ease of operation 0.106 UI friendliness 0.50 e-Guidebook usefulness 0.25 Acceptance of reports 0.25 Ease of learning 0.020 Online learning 1.00 Upgradation ability 0.023 Upgrade service performance 1.00 Ease of integration 0.071 Ease of integration with other systems 0.50 Ease of communication with other platforms 0.50 Ease of in-house 0.014 Ease of maintenance 0.75 development Ease of modification 0.25 Stability 0.159 Frequency of system error 0.50 Data error rate 0.50 Recovery ability 0.033 Mean recovery time 1.00 Vendor 0.163 Technology support 0.279 Diverse product introduction 1.00 Training support 0.072 Effective training lessons 1.00 Service ability 0.649 Solving problem ability 0.33 Consultant service ability 0.33 Service speed 0.34 Impact 0.297 Organization 0.297 Management enhancement 0.12 Cycle time reduction 0.20 Workflow standardization 0.27 Efficiency of system 0.41 Individual 0.163 Quality of decision making 0.25 Personal productivity improvement 0.59 Employ satisfaction 0.16 Table II. Customer 0.540 Response time to customer 0.33 ERP PI structure On time delivery 0.67
  13. 13. 4.4 Step 4 An ERP Initially, the project team discussed how to measure every PI and how to collect its data measurement of the ERP system performed. They first investigated what types of measurement data were already being collected to establish a baseline and determine whether any framework data existed that could be used to determine the overall success of the adopted ERP system. Then, the project team reviewed the available information whether this is currently being collected for PIs or objectives. Additionally, they also paid attention on 619 the reliability of each data, its usefulness, as well as the correspondence with certain PI. For quantitative PIs, the lowest and maximum values which the members believed the ERP system should and can perform were set. On the other hand, for qualitative PIs, the detailed evaluation guidance and an assessment questionnaire also were developed. For example, Table III presents the PIs’ detailed descriptions of a fundamental-objective, “function fitness.” The weight of each PI can be determined by direct assignment or indirect pairwise comparisons. For reducing the loading of the PIs’ importance comparison process, this case followed the AHP methodology. Paired comparisons of PIs relative importance were made and converted to a numerical scale of one to nine. The software Expert Choice was then used to determine the normalized weights. Then, the relative weights of each main category, fundamental-objective and PI using AHP method are also listed in the second, fourth and sixth column of Table II, respectively. 4.5 Step 5 The managers measured the current performance of the ERP system to determine the rating of each PI based on the data gathered by user research groups. For example, in Table III, for the quantitative PI “frequency of special function requests,” the best possible number of times (maximum value v* ) and the worst value (minimum value v0 ) i i were 3 and 50 within a specified timeframe. The current performance rating vi was 16. By the equation (1), the rating of this quantitative PI was 0.7234. That is: Fundamental-objective: function fitness Degree of workflow Information Information Frequency of special PI suppose timeliness aggregation function requests Qualitative PI: Qualitative PI: Qualitative PI: Quantitative PI: PI character average value based average value based average value based number of special on ratings made in on ratings made in on ratings made in function requests the linguistic set L the linguistic set L the linguistic set L within specified timeframe max: 3; min: 50 Rating G G F 16 Weight 0.48 0.24 0.18 0.11 Fuzzy performance index (0.4756, 0.6736, 0.9436) Score 0.6916 Linguistic Table III. term G Examples of PI details
  14. 14. JMTM r¼ 16 2 50 ¼ 0:7234 19,5 3 2 50 On the other hand, the members evaluated the performance of the ERP system with respect to the qualitative PIs by using the linguistic ratings in the scale set L. For example, Table III shows the measurement result at a certain time about the 620 corresponding PIs of the fundamental-objective “function fitness.” The linguistic ratings were obtained by assessing the major members through a subjective assessment process and translated into the fuzzy numbers based on Table I. The precision with which decision makers could provide measurements was limited by their knowledge, experience, and even cognitive biases, as well as by the complexity of the ERP system. Thus, to avoid inconsistency among semantic descriptions and score assignments to the PIs, it is necessary to train the decision makers to understand the details, strengths, and limitations of the proposed method. During the evaluation process, consistency checks were conducted. The decision makers in some cases were asked to provide reasons and detailed explications to justify and refine their assessments. 4.6 Step 6 Aggregated the quantitative and qualitative measurements with the corresponding weights of PIs in Table II to yield the fuzzy performance index of the fundamental-objective “function fitness” by equation (2): 2 3 ð0:5; 0:7; 1:0Þ 6 7 6 ð0:5; 0:7; 1:0Þ 7 6 7 6 7^½0:48; 0:24; 0:18; 0:11Š ¼ ½0:4756; 0:6736; 0:9436Š: 6 ð0:2; 0:5; 0:8Þ 7 4 5 ð0:7234; 0:7234; 0:7234Þ The fuzzy performance index of “function fitness” was (0.4756, 0.6736, 0.9436). Assume c ¼ ð0:4756; 0:6736; 0:9436Þ. Then, its membership function is: ~ 8 x20:4756 0:1980 ; 0:4756 # x # 0:6736 1; x ¼ 0:6736 f c ðxÞ ¼ x20:9436 ~ 20:2700 ; 0:6736 # x # 0:9436 : 0; otherwise The left integral value of c is defined as: ~ Z 1 I L ð~Þ ¼ c 0:198y þ 0:4756 dy ¼ 0:5746; 0 and the right integral value of c is defined as: ~ Z 1 I R ð~Þ ¼ c 2 0:27y þ 0:9436 dy ¼ 0:8086: 0
  15. 15. Then, the total integral value of the fuzzy performance index were obtained by using An ERP the fuzzy integral value method with u ¼ 0.5 (equation (3)): measurement I 0:5 ð~Þ ¼ 0:5 £ 0:5746 þ 0:5 £ 0:8086 ¼ 0:6916: T c framework The integral value 0.6916 was regarded as the performance score of “function fitness.” Finally, the project team translated the fuzzy performance index back to linguistics. Since: 621 I 0:5 ðd3 ¼ FÞ ¼ 0:5 , I 0:5 ð~Þ ¼ 0:6916 , I 0:5 ðd4 ¼ GÞ ¼ 0:725; T ~ T c T ~ then: M ¼ min{j0:6916 2 0:5j; j0:6916 2 0:6125j; j0:6916 2 0:725j} ¼ 0:0334: Following the linguistic term translation rules to get M ¼ 0.0334 of rule (2) was minimum. As d4 ¼ G, the linguistic description of “function fitness” was “Good.” 4.7 Step 7 Went through all the fundamental-objectives by using the proposed fuzzy aggregative method to obtain their fuzzy performance index and performance scores. Rolled them up to gain the fuzzy performance index and performance scores of the three main categories. Following the linguistic term translation rules, the linguistics of all fundamental-objectives and main categories could be obtained. Using the same algorithm, the performance score and the linguistic term of the entire ERP system could be obtained. The final linguistic term of the adopted ERP system performance at the certain time was “between fair and good.” We helped them to collect the data and track the performance scores six months after the ERP performance measurement system establishing. Figure 3 shows the score trends of the system, vendor, and impact categories. A significant progress on the system and impact categories of the ERP performance had been made. However, the scores of ERP vendor indicator category had not improved over time. Figure 4 shows the detailed score records of vendor PIs. Obviously, the fundamental-objectives 1.0 0.8 0.6 score 0.4 0.2 system vendor impact Figure 3. 0.0 Score trend of the three 1 2 3 4 5 6 7 main PI categories month
  16. 16. JMTM 1.0 19,5 0.8 622 0.6 score 0.4 0.2 Technology support Training support Figure 4. Service ability Score trend of the 0.0 1 2 3 4 5 6 7 vendor PIs month “training support” and “service ability” related PIs had made regression. The managers hoped that the ERP vendor could provide more support and service to continuously improve the ERP functions and reports. They decided to strengthen the relationship with the ERP vendor. A problem feedback mechanism and a solving problem process were also established immediately with the ERP vendor. The relative stability of the ERP PI hierarchy is very important. After discussing, PIs only change if any service aims change, major business processes or system change, and any PI is found unsatisfactory or needs to add. 5. Conclusion An ERP system implementation project needs to invest enormous money, labor, and time for a company. Hence, managers must understand what benefits the system has contributed and what aspects the system should be improved. The PIs reflect whether the input resources and efforts in an ERP system implementation project have achieved the objectives which managers want to gain. This study presents a framework to measure the performance of an adopted ERP system under fuzzy environment. The proposed framework developed an ERP PI structure according to the knowledge of ERP implementation objectives. Since humans are difficult in giving quantitative ratings exactly, where some PIs are comparatively efficient in linguistic expressions. An integration model that uses the fuzzy operation and fuzzy integral method was proposed to obtain a fuzzy ERP performance index. Then, the fuzzy ERP performance index can be translated into a performance score and back to a linguistic term. The evaluation results can truly reflect the current situation of the adopted ERP system and the accomplishment of the ERP implementation objectives. It must be noted that the evaluation results do really not be used to punish someone or any department in order to avoid the resistance and misunderstanding of employees. The results point out the functionality and service of the ERP system can be trusted and the high-system performance standards can be maintained. The key point is how to improve the performance of ERP system. The PIs are also aligned with the objective
  17. 17. structure of the ERP system implementation and the framework can ensure the An ERP inclusion of the concept of continuous improvement. measurement The proposed framework offers the following advantages in the ERP performance measurement processes for the companies: framework . It provides a comprehensive and systematic method to extend the objectives of an ERP implementation project to suitable PIs of an ERP performance measurement mechanism. Managers can easily assess the achievement of the 623 ERP implementation objectives by following the stepwise procedure. . The proposed algorithm considers not only quantitative data but also linguistic data. Managers can assess the performance of their adopted ERP system against various PIs, particularly in an ill-defined situation, by using linguistic or quantitative values in the ERP performance evaluation. . The fuzzy ERP performance index can be translated back into to linguistic terms. The linguistic results provide a semantic and impressional description about the current condition of the ERP system. . Additionally, the fuzzy ERP performance index can be calculated to obtain a crisp score. The trends of ERP performance scores of each main category, fundamental-objective and PI can indicate whether the system’s performance is enhancing or descending over time. Managers can recognize the directions of ERP system improvement and the strategies of corporate IS in the future. . The proposed framework can also be applied to other enterprise information systems (EIS) performance evaluation problems. However, because the characteristics and roles of various EIS are different in a company, the framework should be revised as it is applied to other EIS. References Buckley, J.J. (1985), “Fuzzy hierarchical analysis”, Fuzzy Sets and Systems, Vol. 17, pp. 233-47. Chan, D.C.K., Yung, K.L. and Ip, A.W.H. (2002), “An application of fuzzy sets to process performance evaluation”, Integrated Manufacturing Systems, Vol. 13 No. 4, pp. 237-46. Chan, F.T.S. and Kumar, N. (2007), “Global supplier development considering risk factors using fuzzy extended AHP-based approach”, Omega, Vol. 35 No. 4, pp. 417-31. Chan, F.T.S., Chan, H.K., Lau, H.C.W. and Ip, R.W.L. (2006), “An AHP approach in benchmarking logistics performance of the postal industry”, International Journal of Benchmarking, Vol. 13 No. 6, pp. 636-61. Chang, P.L. and Chen, Y.C. (1994), “A fuzzy multi-criteria decision making method for technology transfer strategy selection in biotechnology”, Fuzzy Sets and Systems, Vol. 63, pp. 131-9. Chang, S.L., Wang, R.C. and Wang, S.Y. (2007), “Applying a direct multi-granularity linguistic and strategy-oriented aggregation approach on the assessment of supply performance”, European Journal of Operational Research, Vol. 177 Nos 2/1, pp. 1013-25. Chen, S.J. and Hwang, C.L. (1992), Fuzzy Multiple Attribute Decision Making: Methods and Applications, Springer-Verlag, New York, NY. Chen, Z. and Ben-Arieh, D. (2006), “On the fusion of multi-granularity linguistic label sets in group decision making”, Computers Industrial Engineering, Vol. 51 No. 3, pp. 526-41. Clemen, R.T. (1996), Making Hard Decisions: An Introduction to Decision Analysis, Duxbury Press, Pacific Grove, CA.
  18. 18. JMTM Delone, W.H. and McLean, E.R. (1992), “Information systems success: the quest for the dependent variable”, Information Systems Research, Vol. 3, pp. 60-95. 19,5 Doll, W.J. and Torkzadeh, G. (1988), “The measurement of end-user computing satisfaction”, MIS Quarterly, Vol. 12 No. 2, pp. 259-74. Dubois, D. and Prade, H. (1978), “Operations on fuzzy numbers”, International Journal of Systems Science, Vol. 9, pp. 613-26. 624 Hagood, W.O. and Friedman, L. (2002), “Using the balanced scorecard to measure the performance of your HR information system”, Public Personnel Management, Vol. 31 No. 4, pp. 543-57. Irani, Z. (1999), “IT/IS investment decision making”, Logistics and Information Management, Vol. 12 No. 1, pp. 8-11. Jain, V., Tiwari, M.K. and Chan, F.T.S. (2004), “Evaluation of the supplier performance using an evolutionary fuzzy-based approach”, Journal of Manufacturing Technology Management, Vol. 15 No. 8, pp. 735-44. Kaufmann, A. and Gupta, M.M. (1991), Introduction to Fuzzy Arithmetic: Theory and Application, Van Nostrand Reinhold, New York, NY. Keeney, R.L. and Raiffa, H. (1993), Decisions with Multiple Objectives: Preferences and Value Tradeoffs, Cambridge University Press, New York, NY. Kivijarvi, H. and Saarinen, T. (1995), “Investment in information systems and the financial performance of the firm”, Information Management, Vol. 28, pp. 143-63. Klenke, K. (1992), “Construct and critique of user satisfaction and user involvement instructions”, Information, Vol. 3 No. 4, pp. 325-48. Lee, Y.W., Strong, D.M., Kahn, B.K. and Wang, R.Y. (2002), “AIMQ: a methodology for information quality assessment”, Information Management, Vol. 40, pp. 133-46. Liang, G.S. and Wang, M.J.J. (1994), “Personnel selection using fuzzy MCDM algorithm”, European Journal of Operational Research, Vol. 78, pp. 22-33. Liou, T.S. and Wang, M.J.J. (1992), “Ranking fuzzy number with integral value”, Fuzzy Sets and Systems, Vol. 50, pp. 247-55. Liou, T.S. and Wang, M.J.J. (1994), “Subjective assessment of mental workload – a fuzzy linguistic multi-criteria approach”, Fuzzy Sets and Systems, Vol. 62, pp. 155-65. Mashari, M.A., Mudimigh, A.A. and Zairi, M. (2003), “Enterprise resource planning: a taxonomy of critical factors”, European Journal of Operational Research, Vol. 146, pp. 352-64. Michael, R. and Jens, W. (1999), “Measuring the performance of ERP software: a balanced scorecard approach”, Proceeding of the 10th Australasian Conference on Information Systems, pp. 773-84. Murphy, K.E. and Simon, S.J. (2001), “Using cost benefit analysis for enterprise resource planning project evaluation: a case for including intangibles”, Proceedings of the 34th Hawaii International Conference on System Sciences, pp. 1-11. Ohdar, R. and Ray, P.K. (2004), “Performance measurement and evaluation of suppliers in supply chain: an evolutionary fuzzy-based approach”, Journal of Manufacturing Technology Management, Vol. 15 No. 8, pp. 723-34. Omero, M., D’Ambrosio, L., Pesenti, R. and Ukovich, W. (2005), “Multiple-attribute decision support system based on fuzzy logic for performance assessment”, European Journal of Operational Research, Vol. 160, pp. 710-25.
  19. 19. Palvia, S.C., Sharma, R.S. and Conrath, D.W. (2001), “A socio-technical framework for quality An ERP assessment of computer information systems”, Industrial Management Data Systems, Vol. 101 No. 5, pp. 237-51. measurement Saarinen, T. (1996), “An expanded instrument for evaluating information system success”, framework Information Management, Vol. 31, pp. 103-18. Saaty, T.L. (1980), The Analytic Hierarchy Process, McGraw-Hill, New York, NY. Shamsuzzaman, M., Sharif Ullah, A.M.M. and Bohez, E.L.J. (2003), “Applying linguistic criteria in 625 FMS selection: fuzzy-set-AHP approach”, Integrated Manufacturing Systems, Vol. 14 No. 3, pp. 247-54. Sharif Ullah, A.M.M. (2005), “A fuzzy decision model for conceptual design”, Systems Engineering, Vol. 8 No. 4, pp. 296-308. Skok, W., Kophamel, A. and Richardson, I. (2001), “Diagnosing information system success: importance-performance maps in the health club industry”, Information Management, Vol. 38, pp. 409-19. Stensrud, E. and Myrtveit, I. (2003), “Identifying high performance ERP projects”, IEEE Transaction on Software Engineering, Vol. 29 No. 5, pp. 398-416. Wei, C.C. and Wang, M.J.J. (2004), “A comprehensive framework for selecting an ERP system”, International Journal of Project Management, Vol. 22, pp. 161-9. Wei, C.C., Chien, C.F. and Wang, M.J.J. (2005), “An AHP-based approach to ERP system selection”, International Journal of Production Economics, Vol. 96, pp. 47-62. Wu, J.H., Wang, Y.M., Chang-Chien, M.C. and Tai, W.C. (2002), “An examination of ERP user satisfaction in Taiwan”, Proceedings of the 35th Hawaii International Conference on System Sciences. Zadeh, L.A. (1965), “Fuzzy sets”, Information and Control, Vol. 8, pp. 338-53. Appendix Fuzzy set theory was developed by Zadeh (1965). Some definitions of fuzzy sets, TFNs and linguistic variables introduced by Dubois and Prade (1978), Buckley (1985) and Kaufmann and Gupta (1991) are applied throughout this paper and illustrated as below. ~ Definition 1. In a universe of discourse X, a fuzzy set A of X is characterized by a membership function uA ðxÞ which associates with each element x in X a real number in the interval ~ [0, 1]. The function value uA ðxÞ represents the grade of membership of x in A. ~ ~ Definition 2. A fuzzy number A ~ is described as a fuzzy subset of discourse X, whose membership function uA ðxÞ specifies a mapping from R to a closed interval [0, 1]. A fuzzy ~ number has the following characteristics: . uA ðxÞ ¼ 0; ;x [ ð21; aŠ ½d; 1Þ; ~ . uA ðxÞ is strictly increasing on [a, b ] and strictly decreasing on [g, d ]; and ~ . uA ðxÞ ¼ 1; ;x [ ½b; gŠ: ~ Definition 3. ~ A fuzzy number A is a TFN if its membership function uA is given by: ~ 8 ðx 2 aÞ=ðb 2 aÞ; a # x # b; 1; b # x # c; uA ðxÞ ¼ ðx 2 cÞ=ðb 2 cÞ; c # x # d; ~ : 0; otherwise ~ The TFN A can be denoted by (a, b, c).
  20. 20. JMTM By the extension principle, the fuzzy sum % and fuzzy subtraction * of any two TFNs are also TFNs. But the multiplication ^ of any two TFNs is only an approximate TFN. That is, if 19,5 ~ ~ A1 ¼ ða1 ; b1 ; c1 Þ and A2 ¼ ða2 ; b2 ; c2 Þ then: ~ ~ A1 %A2 ¼ ða1 þ a2 ; b1 þ b2 ; c1 þ c2 Þ; ~ ~ A1 *A2 ¼ ða1 þ a2 ; b1 þ b2 ; c1 þ c2 Þ; 626 ~ ~ A1 ^A2 ø ða1 a2 ; b1 b2 ; c1 c2 Þ; ~ k^A ¼ ðka; kb; kcÞ; k [ R: Corresponding author Chun-Chin Wei can be contacted at: d887801@cyu.edu.tw To purchase reprints of this article please e-mail: reprints@emeraldinsight.com Or visit our web site for further details: www.emeraldinsight.com/reprints