Using Multi-Criteria Decision and knowledge representation methodologies for evaluation of Innovation Projects
Using Multi-Criteria Decision and knowledge representation methodologies for evaluation of Innovation Projects Tania C. D. Bueno1,a, Claudia de O. Bueno1,b, Angela I. Zotti1,c,Vinicius Mirapalheta1,d , Thiago P. de Oliveira1,e, Hugo C. Hoeschl1,f 1 Instituto de Governo Eletrônico, Inteligências e Sistemas – i3G Florianópolis – SC Brasil a email@example.com, firstname.lastname@example.org, email@example.com, d firstname.lastname@example.org, email@example.com, firstname.lastname@example.orgKeywords: Analytic Hierarchy Process (AHP), knowledge representation, multicriteria decision,projects evaluation, decision making.Abstract. The performance of R&D sector has a high importance degree as a competitiveadvantage in energy market, so there is the application of a large amount of investments whichcauses a great number of offered projects. Therefore, it has been necessary to propose a set ofcriteria in order to help on R&D projects prioritization for Electric Power Companies in Brazil. Thenormalization of criteria for projects selection on the Innovation field will guarantee quality on thepromotion of technological innovation in electric sector, which has a fundamental role on theintegration among economic, social and environmental factors inside the company. This work usedaspects of knowledge representation methodologies and multicriteria decision problem to supportthe process of quality measurement of R&D Projects.IntroductionThe evaluation process for R&D projects is the main bottleneck for the Electric Energy institutionsin Brazil because of its subjectivity and lack of relevant and adequate criteria for each situation. Tohelp on this demand, the Electric Energy National Agency (ANEEL) [1,2] has defined somecriteria, so that the companies in electric sector could evaluate R&D projects under legal Acts.These criteria, however, are too broad and subjective, becoming difficult the creation of patterns forthe evaluation process for the electric energy companies. The absence of solid criteria comes outwith delays, and in many cases, might cause the selection of inappropriate projects. The selection problem of R&D projects has being seen as a multicriteria decision problem. Infact, due to a number of criteria ranging from financial investments and materials to possiblecompetitive advantages, we intend to prioritize projects according to their importance, to beimplemented at first moment . On every evaluation system it is fundamental the correct selectionof criteria, relevant and adequate for each situation. The normalization of such criteria – forinnovation project selection – will guarantee quality on the promotion of technological innovationin electric sector, which plays a key role for the integration of economic, social and environmentalfactors inside Electric Power Companies. Therefore, a project evaluation model was developed.This model used aspects from Albertyn [4,5], Saaty [6,7] and Bueno at al [8,9] to support theprocess of quality measurement of R&D Projects. In order to turn this process evident, this workwas divided on the following sections: on Introduction is presented the subject of the present work,followed by the presentation of the R&D projects evaluation process. The third item explains themethodology. The fourth and fifth items present the Results and Conclusion, respectively. Finally,the sixth item presents the References consulted for this study.The R&D Projects Evaluation ProcessThe Brazilian Electric Energy National Agency – ANEEL, in initial evaluation, defines that theproposed Research and Development (R&D) projects should be evaluated solely based on the
content of the XML file based on predefined criteria. In this criterion, the state of the art, challengesand proposed advances in scientific and/or technology terms must be analyzed, considering themain product of the project. The problem to be solved and the absence or the high cost solutionavailable in the market must be considered when is relevant. Since the law creation, the electricpower companies have difficulty in sustaining a significant portfolio of investments in innovation.Among the main factors is the delay in approving the projects by ANEEL [1,2]. Criteria by ANEEL. The criteria established and adopted by ANEEL are described in sequence,which currently parameterize the selection and evaluation process of the companies. The evaluationof a R&D project is performed based on the criteria: Originality, Applicability, Relevance andReasonableness of Costs and the main parameters analyzed in each criterion. The projects areevaluated at the end, necessarily, and there is also the possibility of an initial evaluation, ifrequested, not obligatory though. Nevertheless, the initial evaluation is highly recommended inorder to "evaluate the framework of the project as R & D activity, its relevance to the technologicalchallenges facing the sector and the reasonableness of the planned investments in the face ofexpected results and benefits". Currently, the electric power company submits the project proposal to the initial evaluation ofANEEL for analysis of the following criteria: Originality, Applicability, Relevance andReasonableness of Costs. Criteria by the Electric Power Company. The project proposal is reviewed by the Companyarea where the knowledge to be gained in the research, and/or the potential outcomes may be used.If there is a favorable analysis by the area, the R&D Commission reviews the project proposal,based on parameters and evaluation criteria provided in the Research and Development GuideProgram. If the Commission considers relevant, the proposals coordinators are invited for a facepresentation . After this step and after the Regulatory Agency (ANNEL) initial evaluation, the R&DCommission establishes a conclusive record containing a list of the projects considered "able" tojoin the R&D Program of Energy. Projects can only be considered "able" if they answer thefollowing premises: a) To obtain at least the concept “3” in Originality criterion given by ANEEL; b) To achieve grade equal or higher than 2.8, the average applied to the concepts that guide theproject (Originality, Applicability, Relevance and Reasonableness of Costs); c) The favorable assent of the R & D Commission. In final evaluation in case of inadequacy asan R&D activity the project is disapproved. The Originality criterion has eliminatory character, sothe project must have grade equal to or higher than three (3) and still be characterized as R&Dproject. Otherwise the value of the project should be reversed for the account of R&D .MethodologyThe structure of the Electric Power Company´s R&D Management System allowed the inclusion ofan evaluation process by the following organization. This organization is a result of the addition ofa meta-model proposed by Albertyn [4,5]. The reason for joining was to realize the occurrence ofsome classes of the metamodel that present connections with classes pointing to quality aspectsadapted to the Electric Power Company R&D Project. Before starting the analysis, we defined thepurpose and expected outputs from the process. The knowledge engineering team determined that the objective pursued by the application of themodel was to identify a set of questions and expressions which could fit the criteria set by ANEELthrough knowledge engineering methodology proposed by Bueno at all [8,9] where the definition ofthe relevance of the expressions is related to frequency of use of these expressions in context. Set the goal of the analysis - it was the choice of the projects that were using the process - wechose to work on projects that were running. In the case of a system of Multi-Criteria Decision to
formalize and validate the process of evaluation and selection of projects aimed to a practice thatwas structured in line with the Mind Engineering Methodology® . The MCDA worked byAlbertyn  is a method to support evaluators who are faced with many different and conflictingsolutions to a problem. Deriving this method, we chose to focus the evaluation on the application of the AHP, becausethe method is based on the innate human ability to make judgments about different problems.The method is characterized by decomposing the problem into descendant hierarchical levels,starting with the overall goal, criteria, subcriteria and alternatives in successive levels, until itreaches a prioritization of its indicators, approaching a better response. The AHP has the advantageof allowing comparison of quantitative and qualitative criteria. At each stage (criteria andsubcriteria) pairwise comparisons are made to determine the relative importance of each criterion toreach the goal. After this hierarchical structure phase, AHP includes other two important steps: thejudgment of value and priority, where the evaluator establishes a peer comparison of elements ofthe various hierarchical levels, prioritizing them and in sequence, the analysis of consistency ofthese trials. This pairwise comparison is done according to Saaty Scale [6,7] that allows theconversion of the analysis on a scale from 1 to 9, ending in an array for each level of criteria andsub-criteria, showing the result of the comparisons made in pairs. Through pairwise comparisons, the priorities calculated by AHP capture subjective and objectivemeasures and demonstrate the strength domain of a criterion over another or one alternative overanother. Through synchronization meetings, the teams began the process of knowledge engineering,with the task of standardizing the language that would apply in the development of Knowledge-Based System - KMS. We identified the main concepts used in the domain of evaluation andselection of R&D projects and their understanding and determination of national and regionalcontext of the procedures adopted so far the Electric Power Company. Prospecting criteria, inagreement with the criteria established by ANEEL, and definition of the Energy Power Companystrategic criteria were issues discussed. The elements used by the domain experts, knowledge engineers and programmers could pass ontheir knowledge in a better structured way in order to build a KMS based on the Mind EngineeringMethodology® . Inventories of contents, processes and people were conducted following thepremises of knowledge sharing, visualization and definition of relevance. It is the synchronizationof these factors that enables the knowledge understanding or expertise in a particular field. There isa need to change the formal model of innovation management in the Electric Power Company tosupport and become innovation more effectively within the parameters of ANEEL and thusadapting it to the national context. To do so, the innovation management models from other companies in power sector werestudied and the proposal of a new model came out from the synergic teamwork. This process tooktwo months and four meetings were held for sharing knowledge among six knowledge engineersand three domain experts.Results - AHP Application for R&D EvaluationWhile inventories allowed the selection of the questions available to the R&D Projects EvaluationSystem and therefore the adoption of criteria to be evaluated and prioritized, the application of AHPmethod allowed calculating the weights for each alternative of evaluation project in relation to theproposed criteria. At this stage, based on the questionnaire already prepared, we performed pairwisecomparisons using the intensities of importance by Saaty Scale. The pairwise comparison followedthe next hierarchy: pairwise evaluation among criteria, pairwise evaluation among questions withinthe criteria, pairwise subcriteria evaluation. Thus, the intensities of importance were converted intonumbers, according to that scale. The test calculations were performed in Excel software. Thefollowing tables present the values resulting from the evaluation of researchers and the matrixobtained from calculations that make up the method.
Pairwise evaluation among criteriaFigure 1 represents the comparison among all the criteria (Originality, Applicability, Relevance andReasonableness of Costs). According to the data presented the Originality criterion is the one thathas a greater impact over the other criteria defined by the company. This result quantifies theguidance established by the energy national agency and the extreme importance of the originalitycriterion for the evaluation and selection of R&D projects. Fig. 1. Matrix with the pairwise criteria evaluationPairwise evaluation among questions within the criteriaFigure 2 presents the comparison among the questions that form the Originality criterion. It showsthe “Unpublished” as the most important. Fig. 2. Matrix of Pairwise evaluation of alternatives within the criteria OriginalityPairwise Subcriteria EvaluationThe alternatives inside each question of each criterion were considered as subcriteria. Figure 3presents the result of pairwise comparison made among the subcriteria in question 4 of Originality
criterion that evaluates the innovative character of the proposed project (Innovativeness). The mostrelevant subcriterion was “It is well defined and innovative, generating new product, process andmethodology”. Fig. 3. Matrix of pairwise comparison for the subcriteria in question 4 (Originality)ConclusionThis methodology allowed the identification of the project that best meets the criteria defined by thecompany, considering the different weights for each alternative. As main result it produced a toolthat allowed simulating scenarios depending on the questions to evaluate R&D projects. Every new scenario allows using the same methodology to support decision making in othersituations, specific to each edict or companys proposal. It means that depending on the proposal,another criterion as Relevance can be considered the most important. Therefore, each scenariocreated can permit the use of the methodology applied in this study as support for decision makingin other circumstances. Thus, for each new scenario will be necessary to compare the questions according to each edictadopted, calculating the weights again and subsequently, ranking criteria, and producing a newresult. Since ANEEL´s R&D Program is recent, results from this research have contributed effectivelyfor the Electric Power Company intention, on the challenge for an evaluation and selection ofsubmitted projects with good quality, about a task previously assumed by ANEEL. Although there is already a model for textual knowledge representation inside this ElectricPower Company´s knowledge base, this base is not filled with enough documents forexperimentation. In future jobs, this knowledge base will be completed with a considerable set ofproposed R&D projects, in order to do more tests about knowledge representation by usingontologies.Acknowledgments. Our thanks to Centrais Elétricas de Santa Catarina – CELESC, especially toCelesc Team: Mr. Amaro Koneski Filho, Mr. Luiz Afonso Pereira Athayde Filho e Mrs. DéboraSimoni Ramlow. Special thanks to i3G Team: Mrs. Alessandra Zoucas and Ms. Angela Iara Zottiwho helped us to finish this job.
References ANEEL - Agência Nacional de Energia Elétrica (2002). Atlas de Energia Elétrica do Brasil.ANEEL, Brasília: ANEEL, 2002. Retrieved from:<www.aneel.gov.br/arquivos/PDF/livro_atlas.pdf> ANEEL - Agência Nacional de Energia Elétrica (2010). Guia do avaliador de projetos de P&D:Programa de Pesquisa e Desenvolvimento Tecnológico do Setor de Energia Elétrica. Brasília:ANEEL, 2010. 30f. A. P. H Gusmão. Modelo Multicritério para Priorização de Portfólio de Projetos de Pesquisa e Desenvolvimento. Recife, 2005. 101p. Dissertação (Mestrado em Engenharia de Produção) – Universidade Federal de Pernambuco. F. Albertyn. E-Process Selection using Case-Based Reasoning Techniques. IADIS international conference E-Society 2007. Lisbon, Portugal. (http://www.esocietyconf.org/) part of the IADIS MultiConference on Computer Science and Information Systems (MCCSIS 2007) Lisbon, Portugal, 3 – 8 July 2007 (http://www.mccsis.org). F. Alberty. A Decision Making Approach: a meta-model for e-process selection. In: IADIS International Conference e-Commerce, 2010, Freiburg, Germany, 26-30 July. ISBN (Book): 978-972-8939-24-3. T. L. Saaty, (2005). Theory and Applications of the Analytic Network Process: Decision Making with Benefits, Opportunities, Costs, and Risks. Pittsburgh: RWS Publications.). T. L. Saaty, (2009). Extending the Measurement of Tangibles to Intangibles. International Journal of Information Technology & Decision Making, Vol. 8, N. 1, p. 7-27, 2009. Available at SSRN: http://ssrn.com/abstract=1483438. T. C. D. Bueno , H. C. Hoeschl , A. Bortolon , R. M. Barcia. Engineering of Minds: The syncrhonicity between Artificial Intelligence and Management of knowlegde in collaborative networked organazations. Conference Proceedings IADIS International Conference WWW/Internet - ICWI IADIS International Conference WWW/Internet 2004. ISBN: 972- 99353-0-0. T. C. D. Bueno, C. Stradiotto, S. Bedin, V. Mirapalheta, H. Hoeschl, T. Oliveira, L. A. Filho, P. A. Junior. Case-based System for Innovation Management in Electric Power Enterprises. 2nd Workshop on Applications of Knowledge-Based Technologies in Business (AKTB 2010) in 13th International Conference on Business Information Systems (BIS 2010). Berlin, Germany. May 3-5, 2010. Celesc (2010). Centrais Elétricas de Santa Catarina S.A.. Relatório Anual de Responsabilidade Socioambiental 2010. CELESC. 64f. J. R. Grandzol. Improving the Faculty Selection Process in Higher Education: a case for the analytic hierarchy process. IR Applications, v. 6, 13 p., 2005. Available in: <http://airweb.org/images/IRApp6.pdf>