An in-depth reseaarch based on fuzzy logic methodology how define demands and requirements for a corporate PPMIS software.

1. See discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/235644567
Evaluation of project and portfolio
Management Information Systems with the use
of a hybrid IFS-TOPSIS method
ARTICLE in INTELLIGENT DECISION TECHNOLOGIES · JANUARY 2013
DOI: 10.3233/IDT-120153
CITATIONS
3
READS
35
3 AUTHORS:
Vassilis C. Gerogiannis
Technological Educational Institute of Thes…
63 PUBLICATIONS 168 CITATIONS
SEE PROFILE
Panos Fitsilis
Technological Educational Institute of Thes…
57 PUBLICATIONS 147 CITATIONS
SEE PROFILE
Achilles Kameas
Hellenic Open University
147 PUBLICATIONS 697 CITATIONS
SEE PROFILE
All in-text references underlined in blue are linked to publications on ResearchGate,
letting you access and read them immediately.
Available from: Vassilis C. Gerogiannis
Retrieved on: 06 December 2015

2. Intelligent Decision Technologies 7 (2013) 91–105 91
DOI 10.3233/IDT-120153
IOS Press
Evaluation of project and portfolio
Management Information Systems with the
use of a hybrid IFS-TOPSIS method
Vassilis C. Gerogiannisa,∗, Panos Fitsilisa and Achilles D. Kameasb
a
Project Management Department, Technological Education Institute of Larissa, Larissa, Hellas
b
Hellenic Open University, Patras, Hellas
Abstract. Contemporary Project and Portfolio Management Information Systems (PPMIS) have embarked from single-user,
single-project management systems to web-based, collaborative, multi-project, multi-operational information systems which of-
fer organization-wide management support. The variety of offered functionalities, along with the variation among each orga-
nization needs and the plethora of PPMIS available in the market, make the selection of an appropriate PPMIS a complicate,
multi-criteria decision problem. The problem complexity is further augmented since the multi stakeholders involved in the eval-
uation/selection process cannot often rate precisely their preferences and the performances of candidate PPMIS on them. To
meet these challenges, this paper presents a PPMIS selection/evaluation approach that applies a hybrid group decision making
method based on TOPSIS and Intuitionistic Fuzzy Sets (IFS). The approach considers the vagueness of assessors’ judgments
when evaluating PPMIS and the uncertainty of users when they judge their needs. The approach is demonstrated through a case
study aiming to support the Hellenic Open University to select a suitable PPMIS.
Keywords: Project and portfolio management information systems, multi-criteria decision making, group decision making, tech-
nique for order preference by similarity to ideal solution, intuitionistic fuzzy sets
1. Introduction
The adoption of an appropriate Project and Port-
folio Management Information System (PPMIS) of-
fers a lot of benefits for an organization that under-
takes projects, project programs and project portfo-
lios to implement business process changes and de-
velop new products or services. Research studies [20]
present that increasing organizational requirements for
the management of the entire life-cycle of complex
projects, programs and portfolios motivate the fur-
ther exploitation of powerful PPMIS from organiza-
tions of any type and size. PPMIS have embarked
∗Corresponding author: Vassilis C. Gerogiannis, Project Man-
agement Department, Technological Education Institute of Larissa,
Larissa 41110, Hellas. Tel.: +30 2410 684585; Fax: +30 2410
684580; E-mail: gerogian@teilar.gr.
from stand-alone, single-user, single-project manage-
ment systems to multi-user, multi-functional, collabo-
rative, web-based and enterprise-wide software tools
which offer integrated project, program and portfolio
management solutions, not limited to scope, budget
and time management/control. Contemporary PPMIS
can support, through a range of functionalities, most
processes in all knowledge areas of the “Project Man-
agement Body of Knowledge” [25], by covering an ex-
pansive view of the “integration management” knowl-
edge area that includes alignment and control of multi-
project programs and portfolios. The market of PP-
MIS is rapidly growing and includes many commer-
cial and open source software tools offering a number
of functionalities such as time, resource and cost man-
agement, reporting features and support for change,
risk, communication, contract and stakeholder man-
agement. Interested readers are referred to [22] where
detailed information is given for 24 commercial lead-
ISSN 1872-4981/13/$27.50 c 2013 – IOS Press and the authors. All rights reserved

3. 92 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
ing PPMIS. In this report, each presented PPMIS is
evaluated upon approximately 270 functional and non-
functional features. Apart from commercial systems,
for organizations which do not require the complete
range of functionalities of a commercial tool or inter-
ested in reducing the total cost of software ownership,
there is available a variety of open source PPMIS or
Software as a Service (SaaS) products.
This variety of offered functionalities, along with the
variation among each organization needs and the large
number of powerful PPMIS in the market, make their
evaluation and selection a complicate multi-criteria de-
cision problem. The problem is often approached in
practice by ad hoc procedures based only on personal
preferences of users/evaluators or any marketing infor-
mation available [27]. Such an approach may lead to
a final selection that does not reflect adequately the
organization needs or, even worse, to an unsuitable
PPMIS. Therefore, a systematic technique from the
multi-criteria decision making (MCDM) domain can
be useful to support the PPMIS evaluation/selection
process.
The main objective of this paper is to present a pos-
sible solution for this multi-criteria decision problem.
The paper presents a PPMIS evaluation/selection ap-
proach that applies a hybrid group decision making
method based on the Technique for Order Preference
by Similarity to Ideal Solution (TOPSIS) [11] and Intu-
itionistic Fuzzy Sets (IFS) [4]. The aim of the approach
is to consider the vagueness of assessors’ judgments
when evaluating PPMIS and the uncertainty of users
when they judge their needs. The approach is demon-
strated through a case study aiming to support the Hel-
lenic Open University to select a suitable PPMIS.
The outline of the paper is structured as follows.
Section 2 briefly reviews the relevant literature in the
field of MCDM techniques for the evaluation of soft-
ware products. In Section 3, we discuss the aspects of
the PPMIS evaluation problem and we justify how, in
our case study, the PPMIS selection criteria were de-
termined. In Section 4, we present an overview of the
characteristics of the presented approach and justify
the selection of IFS. In Section 5, the basic concepts of
IFS are briefly discussed. Section 6 presents the steps
of the approach and Section 7 presents conclusions and
future work.
2. MCDM methods for evaluating software
packages
Although there is no a generic MCDM approach
for selecting a software package of any type, available
literature reviews in software products evaluation [19]
suggest that users and evaluators can receive a lot of
benefits if they decide to adopt a MCDM method. Re-
view surveys [18,19,29] reveal that the Analytic Hier-
archy Process method (AHP) and its variations/exten-
sions have been widely and successfully used in eval-
uating several types of software packages (e.g., MRP
systems, ERP systems, simulation software, CAD sys-
tems and knowledge management systems). This ex-
tensive application of AHP is due to the method ad-
vantages, since it supports the hierarchical decomposi-
tion of a decision problem, allows decision making to
be held by a group of stakeholders as well as it han-
dles both qualitative and quantitative selection criteria.
Although AHP presents wide applicability in evaluat-
ing various types of software products, little work has
been done in the field of evaluating PPMIS. For exam-
ple, in [2] the authors admit that their work is rather
indicative with main objective to expose a representa-
tive case for illustrating the PPMIS selection process
and not to create a definitive set of criteria that should
be taken into account in practice. This lack of applica-
bility of AHP in the PPMIS selection problem domain
can be attributed to the fact that, despite its advantages,
the method main limitation is the large number of pair-
wise comparisons required. The time needed for com-
parisons increases geometrically with the increase of
criteria and alternatives involved, making AHP appli-
cation practically prohibitive for complicate decisions,
such as the selection of a PPMIS.
As a response to this problem, in the recent past, we
presented an approach for evaluating alternative PP-
MIS that combines group-based AHP with a simple
scoring model [15]. According to this approach, PP-
MIS evaluators (decision makers) use a scoring model
to evaluate the performances of candidate systems with
respect to an extensive list of functional-oriented cri-
teria (organized into criteria clusters), while PPMIS
users follow the AHP method to determine the over-
all weights of the criteria clusters based on the needs
of their organization. This group-AHP scoring model,
although practical and easy to use, does not consider
the vagueness or even the unawareness of users, when
they evaluate their preferences from a PPMIS by rat-
ing their requirements. Also the approach does not take
into account the uncertainty of evaluators, when they
judge the performance ratings of alternative PPMIS on
the selected criteria, expressed as user requirements.
In case of evaluating a PPMIS, these uncertainties are
more evident when the software product does not of-
fer a certain functionality by default (in the product

4. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 93
standard version) and the desired functionality can be
fulfilled – at a certain degree – through configuration,
customization, use of workarounds (other functionali-
ties that act as substitutes) or use of interfaces to other
software products.
Treating with these ambiguities in the PPMIS evalu-
ation and consideration of incomplete available infor-
mation expose the need to adopt a fuzzy-based deci-
sion making approach [9]. Fuzzy-based methods pro-
vide the intuitive advantage to utilize, instead of crisp
values, linguistic terms to evaluate performance of the
alternatives and criteria weights. A linguistic term (i.e.,
a variable whose value is a natural language phrase)
can be particularly useful to express qualitative assess-
ments. A fuzzy-based approach can be even more ben-
eficial when it is combined with other decision mak-
ing techniques. For example, fuzzy AHP [10] is pro-
posed to handle the inherent imprecision in the pair-
wise comparison process, while fuzzy TOPSIS (Tech-
nique for Order Preference by Similarity to Ideal Solu-
tion) [11] can be used to jointly consider both positive
(benefit/functional oriented) and negative (cost/effort
oriented) selection criteria. Fuzzy-based MCDM tech-
niques have been used to select various types of soft-
ware products (see, for example [9,13,21]), but in the
relevant literature there is lack of a structured fuzzy-
based approach for the selection of PPMIS under un-
certain knowledge.
This paper presents such a fuzzy-based approach
that in comparison with other MCDM approaches for
software product selection [9,13,21] mainly differs in
three aspects: i) the approach involves both decision
makers and PPMIS users in the decision making pro-
cess and aggregates their weighted opinions (through
fuzzy weighted averaging operators) to support agree-
ment upon the final selection, ii) the approach handles
the degree of indeterminacy that characterizes both de-
cision makers and users in their evaluations, iii) both
positive (benefit) and negative (cost) criteria are con-
sidered in the evaluation. These aspects are supported
by the approach underling method that is a hybrid
group decision making method based on TOPSIS and
Intuitionistic Fuzzy Sets.
3. PPMIS adoption, evaluation and selection
criteria
Empirical studies [26] demonstrate that a number of
project managers from the business community indi-
cate a strong impact of PPMIS usage upon success-
ful implementation of their business projects, while
others do not. These findings indicate that “unsatis-
fied” project managers are depended upon a PPMIS
that produces information of low quality. Hence project
managers use the system less and consequently get
less support in their management tasks. It is impor-
tant, therefore, for an enterprise to select a proper PP-
MIS that covers technical, managerial and organisa-
tional needs. The importance of project management
techniques and tools is also gaining recognition in aca-
demic institutions and Higher Education organizations
as a valuable tool, especially for supporting university
Information Technology (IT) projects. The results of a
survey in IT departments of US universities show that
project planning, monitoring and status/budget report-
ing are of crucial importance for the majority of uni-
versity projects [34].
The PPMIS selection process can be supported by
referencing to available market surveys [27]. In the
past, for example, the Project Management Institute
has published an extensive survey [24] that compared
more than 200 products by considering classic project
management dimensions like scheduling, cost, risk, re-
source and communication management. These com-
parisons, however, focus rather on factors which rep-
resent vendors’ perspectives and any such assessment
should be utilized with care by considering specific
project management needs within the context of indi-
vidual organizations. Furthermore, PPMIS of today of-
fer support for the entire project life-cycle, including
portfolio planning and monitoring and thus, a PPMIS
evaluation process based only on classic single-project
management functionalities is very limited.
Support in setting up a PPMIS system can be also
gained by considering the users’ perceptions and sat-
isfaction from a PPMIS usage. A representative exam-
ple is the one presented in [3]. This work surveyed 497
PPMIS users and the final result was a general index
for measuring the effectiveness of PPMIS according to
four, user-oriented, dimensions (i.e., information qual-
ity, system functionality, ease of use, performance im-
pact). However, respective PPMIS users, when evalu-
ate a PPMIS system, often express their perceived sat-
isfaction and not their knowledge on potential benefits
that can be obtained from a PPMIS.
Detailed assistance in evaluating PPMIS is provided
by evaluation frameworks which propose to consider
an extensive list of system characteristics. These char-
acteristics can be either functional or process-oriented
selection criteria. NASA, for example, in the past has
convened a working group to evaluate alternative PP-

5. 94 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
Fig. 1. M-Model architecture – source: [6]. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/IDT-120153)
MIS for NASA’s departments, upon a number of func-
tional requirements. In the group’s report [16] thir-
teen clusters of functional requirements are identified,
namely: 1) open database connectivity and architec-
ture, 2) workgroup capabilities, 3) networking capa-
bilities, 4) ease of use, 5) project scheduling method-
ology, 6) project task/field features, 7) baselining and
tracking project progress, 8) resource features, 9) cal-
endar features, 10) cost management features, 11) risk
management features, 12) project reporting, and 13)
management reporting. Each cluster further includes a
set of functional features and, in total, more than one
hundred functional criteria are identified to be eval-
uated. This vast number of criteria prevents decision
makers from utilizing a typical hierarchical MCDM
approach like AHP.
As far as process oriented evaluation is concerned,
evaluators may use as reference the set of process-
oriented criteria offered by a conceptual software ar-
chitecture for PPMIS, like, for example, is the M-
Model (Fig. 1) [1]. An abstract software architecture
may be used to handle the selection problem from
a business process reengineering perspective, since it
embraces all tasks performed during a project/program
life-cycle (initiation, planning, execution and termina-
tion). The M-Model specifies the project phases/tasks
supported by PPMIS which are mapped into differ-
ent management levels (project, program and port-
folio management). The model was used in [22] to
evaluate commercial PPMIS according to the project
phases/tasks supported by a PPMIS and the corre-
sponding required functionalities (Table 1). Each PP-
MIS was evaluated according to the extent that it of-
fers the required functionality and the overall support
for the corresponding project phase/task was specified
with a “4-stars” score. Yellow and grey stars at each
score indicate (see Fig. 1) if the corresponding sup-
port is offered as standard functionality (yellow stars)
or it can be provided by customizing the PPMIS (grey
stars). The authors admit in their evaluation report
that the stars (i.e., the performance rating) assigned at
each PPMIS upon each criterion (i.e., the level of pro-
vided support for the corresponding project phase/task)
are obtained only by counting the number of func-
tionalities offered by the standard system version or
through applying simple customisations (without ex-

6. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 95
Table 1
Evaluation criteria – source: [22]
Phase/task Required functionality
1. Idea generation/lead management (IGLM) Creativity techniques, idea/project classification, lead management (Mgmt.),
project status/project process mgmt
2. Idea evaluation (IE) Estimation of effort, resource needs specification, risk estimation, profitability
analysis, project budgeting, offer mgmt
3. Portfolio planning (PP1) Organizational budgeting, project assessment, project portfolio optimization,
project portfolio configuration
4. Program planning (PP2) Project templates, resource master data, resource assignment workflow, resource
allocation
5. Project planning (PP3) Work breakdown structure planning, scope/product planning, network planning,
scheduling, resource leveling, risk planning, cost planning
6. Project controlling (PC1) Change request Mgmt., (travel) expense Mgmt., timesheet, cost controlling,
meeting support
7. Program controlling (PC2) Status reporting, deviation/earned value analysis, quality controlling, versioning,
milestone controlling
8. Portfolio controlling (PC3) Performance measurement, dashboard, organizational budget controlling
9. Program termination (PT1) Knowledge portal, competence database/yellow pages, project archiving, searching
10. Project termination (PT2) Invoicing, document Mgmt., supplier and claim Mgmt.
11. Administration/configuration (AC) Workflow Mgmt., access control, report development, form development,
user-defined data structures, MS office project interface,
application programming interface, offline usage
tensive code development). Thus, any workarounds,
interfaces and/or use of programming (in case of open
source PPMIS) are not taken into account in this eval-
uation report. In case of selecting a proper PPMIS for
a specific organization, the consideration of these pa-
rameters or lack of knowledge upon them will certainly
affect the uncertainty of the final performance rating
for each candidate PPMIS.
In Section 6 of the paper, we show how these 11
project phases/tasks (Table 1) were included in a list of
selection criteria for evaluating alternative PPMIS for
the case organization. This decision supported users
(members of the case organization) to rate the impor-
tance of their requirements by considering the pro-
cesses supported by PPMIS, without need for know-
ing technical capabilities and functionalities of each
candidate system. This decision also helped decision
makers (evaluators) to perform cross-checking (i.e.,
comparisons) of their linguistic assessments on can-
didate PPMIS against the scores presented in [22]. In
addition, the adoption of intuitionistic fuzzy numbers
for the evaluation of PPMIS helped decision makers
to consider in the evaluation not only the availabil-
ity/unavailability of a functionality to support a crite-
rion but also the degree that the unavailability can be
“relaxed” though admitting that there can be also other
solutions, not offered by the “out of the box” PPMIS
version.
4. Overview of the suggested PPMIS
evaluation/selection approach
The suggested approach for PPMIS involves both
users and evaluators (decision makers) in the deci-
sion making process and tries to exploit the inter-
est/expertise of each one in order to strengthen the fi-
nal evaluation results. This is achieved by aggregating
all weights of criteria (requirements) and all ratings of
performance of the alternative systems, as they are ex-
pressed, by individual stakeholders, in linguistic terms.
The approach is based on Intuitionistic Fuzzy Sets
(IFS), an extension of fuzzy sets proposed by Atana-
ssov [4] that has been successfully used in many deci-
sion making problems, such as medical diagnosis [14],
web services selection [33] and supplier selection [7,
12,23]. An IFS includes the membership and the non-
membership function of an element to a set as well
as a third function that is called the hesitation degree.
This third function is useful to express lack of knowl-
edge and hesitancy concerning membership and non-
membership of an element to a set.
Expression of hesitation is particularly helpful for
both decision makers and PPMIS users when they se-
lect a software product for an organization such as, in
our case, a PPMIS. On one hand, decision makers of-
ten cannot have a full knowledge upon all function-
alities included in the newest version of each candi-
date system. Thus, they base their ratings only on ex-

7. 96 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
perience from using previous system versions as well
by referencing to system assessments which can be
found in products survey reports [22]. Furthermore,
a “negative” PPMIS performance rating – that is as-
signed when the system does not provide standard sup-
port for a required functionally (i.e., the functional-
ity is not available at the standard version of the sys-
tem) – can be even more hesitant, since the software
may offer the functionality through configuration, cus-
tomization, use of workarounds (other functionalities
that act as substitutes) or interfaces to other available
software products. On the other hand, PPMIS users are
often unfamiliar with how a system can support project
management processes and tasks and, therefore, can-
not precisely express which tasks require more to be
supported by a PPMIS.
It should be noted here that the presented approach
mainly utilizes a hybrid method presented in [7] which
combines IFS with TOPSIS for supporting supplier se-
lection problems. The advantage of this combination
in case of PPMIS evaluation is that we can distinguish
between benefit criteria (e.g., functionalities/tasks sup-
ported by the PPMIS) and cost criteria (e.g., effort for
system customisation and price for ownership). The
PPMIS that is closest to the positive ideal solution and
most far from the negative ideal solution could be prob-
ably the most appropriate PPMIS to cover the organi-
zation needs. The approach not only validates the orig-
inal method in a new application field that is the eval-
uation of PPMIS (where other MCDM approaches are
rather limited in the literature), but also considers a
more extensive list of benefit and cost oriented criteria,
suitable for PPMIS selection. In addition, final results
are verified by applying sensitivity analysis.
5. Intuitionistic fuzzy sets: Basic concepts
Before proceeding to describe how the PPMIS se-
lection problem was tackled, we briefly introduce some
necessary introductory concepts of IFS. An IFS A in a
finite set X can be defined as [4]:
A = {< x, μA(x), vA(x) > |x ∈ X} (1)
where μA : X → [0, 1], vA : X → [0, 1], and 0
μA(x) + vA(x) 1 ∀x ∈ X.
μA(x) and vA(x) denote respectively the degree of
membership and non-membership of x to A. For each
IFS A in X, πA(x) = 1 − μA(x) − vA(x) is called
the hesitation degree of whether x belongs to A. If the
hesitation degree is small then knowledge whether x
Table 2
Linguistic terms for the importance of stakeholders and criteria
Level of stakeholder
expertise (1)
Importance of
selection criteria (2)
IFN(3)
Master Very important (VI) [0.90,0.10]
Expert Important (I) [0.75,0.20]
Proficient Medium (M) [0.50,0.45]
Practitioner Unimportant (U) [0.25,0.70]
Beginner Very unimportant (VU) [0.10,0.90]
belongs to A is more certain, while if it is large then
knowledge on that is more uncertain. Thus, an ordinary
fuzzy set can be written as:
{< x, μA(x), 1 − μA(x) > |x ∈ X} (2)
In the evaluation approach we will use linguistic
terms [17] to express: i) the importance of decision
stakeholders (users/decision makers), ii) judgements of
decision makers on the performance of each PPMIS
and iii) perceptions of users on the importance of each
selection criterion. These linguistic terms can be trans-
formed into intuitionistic fuzzy numbers (IFNs) in the
form of [μ(x), v(x)]. For example, an IFN [0.50, 0.45]
represents membership μ = 0.5, non-membership v =
0.45 and hesitation degree π = 0.05.
In the approach, we will also use addition and mul-
tiplication operators for IFNs. Let a1 = (μa1, va1) and
a2 = (μa2, va2) be two IFNs. Then these operators can
be defined as follows [4,30,31]:
a1 ⊕ a2=(μa1 + μa2 − μa1 · μa2, va1 · va2)
a1 ⊗ a2=(μa1 ·μa2, va1 +va2−va1·va2) (3)
λ · a1=(1 − (1 − μa1)λ
, vλ
a1), λ > 0
6. Evaluation of PPMIS with intuitionistic fuzzy
sets and TOPSIS
In this section we describe how an intuitionistic
fuzzy MCDM approach was applied with the overall
goal to select the most appropriate PPMIS system to
cover needs of the Hellenic Open University (HOU)
(www.eap.gr) in facilitating, supporting and providing
project management for university-industry collabora-
tion in research and development (R&D). HOU is a
university that undertakes various types of national and
international R&D projects and programs, particularly
in the field of continuous adult education. The univer-
sity does not maintain an integrated project/portfolio
management infrastructure. In order to increase project
management maturity, effectiveness and productivity,
the management of HOU has decided to investigate the

8. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 97
Table 3
Linguistic terms for rating the performance of PPMIS
Level of performance/support IFN Degree of hesitation (π) Final IFN
Extremely high (EH) [1.00,0.00] 0 [1.00,0.00]
Very very high (VVH) [0.90,0.10 − π] 0 [0.90,0.10]
Very high (VH) [0.80,0.20 − π] 0.1 [0.80,0.10]
High (H) [0.70,0.30 − π] 0.1 [0.70,0.20]
Medium high (MH) [0.60,0.40 − π] 0.1 [0.60,0.30]
Medium (M) [0.50,0.50 − π] 0.1 [0.50,0.40]
Medium low (ML) [0.40,0.60 − π] 0.1 [0.40,0.50]
Low (L) [0.30,0.70 − π] 0.1 [0.30,0.60]
Very low (VL) [0.20,0.80 − π] 0 [0.20,0.80]
Very very low (VVL) [0.10,0.90 − π] 0 [0.10,0.90]
Fig. 2. Steps of the PPMIS evaluation approach. (Colours are visible in the online version of the article; http://dx.doi.org/10.3233/IDT-120153)
adoption of a collaborative PPMIS. The Department of
Project Management (DPM) (dde.teilar.gr) at the Tech-
nological Education Institute of Larissa in Greece was
appointed to act as an experienced consultant and aid
this decision making process.
Three experts D1, D2 and D3 (decision makers/
evaluators) from DPM, with an average of seven years
teaching/professional experience in using different PP-
MIS, were involved in this process, aiming to iden-
tify HOU requirements from a PPMIS and to select an
appropriate system that will cover these requirements.
Three project officers/managers U1, U2 and U3 (users)
from the HOU site were also involved in the decision
making. These persons have high expertise in contract
management, multi-project coordination and planning
of R&D projects and portfolios, but they present low
experience in systematically using PPMIS.
The application of the approach for selecting an ap-
propriate PPMIS for the case organization (HOU) has
been conducted in eight steps (Fig. 2) presented as fol-
lows.
Step 1: Determine the weight of importance of decision
makers and users
In this first step, the expertise of both decision mak-
ers and users was analysed by specifying correspond-
ing weights. In a joint meeting, the three decision mak-
ers D1, D2, D3 agreed to qualify their experience
in using PPMIS as “Master”, “Proficient” and “Ex-
pert”, respectively. The three users U1, U2, U3 also
agreed that their level of expertise in managing large
projects can be characterized as “Master”, “Proficient”
and “Expert”, respectively. These linguistic terms were
assigned to IFNs by using the relationships presented
in Table 2 between values in column 1 and values in
column 3.
If there are l stakeholders in the decision process,
each one with a level of expertise rated equal to the IFN
[μk, vk, πk], the weight of importance of k stakeholder
can be calculated as [7]:
λk =
μk + πk
μk
μk + vk
l
k=1
μk + πk
μk
μk + vk
(4)
where λk ∈ [0, 1] and l
k=1 λk = 1.
By applying Eq. (4) the weights of decision mak-
ers were calculated as follows: λD1 = 0.406, λD2 =
0.238, λD3 = 0.356. Since users were assigned to the
same linguistic values, their weights were respectively
the same: λU1 = 0.406, λU2 = 0.238, λU3 = 0.356.
It should be noted here that the heuristic of Eq. (4) for

9. 98 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
Table 4
The ratings of the alternative PPMIS
Criteria Decision makers PPMIS
A1 A2 A3 A4 A5
IGLM D1 VH VH H MH H
D2 H VH MH H H
D3 H H H H MH
IE D1 H M VH M M
D2 MH M H H H
D3 M MH H MH H
PP1 D1 MH H VVH VH VH
D2 MH MH VH MH VH
D3 MH MH H H VH
PP2 D1 MH MH VH VH VH
D2 MH H VH MH VH
D3 H M H MH H
PP3 D1 VH H VH VH VH
D2 H H MH H VH
D3 VH VH H MH MH
PC1 D1 H VH VH VH H
D2 MH H H H H
D3 H H H MH MH
PC2 D1 H MH VH H VH
D2 MH M H H VH
D3 H MH H MH H
PC3 D1 H VH MH H VH
D2 MH H M H VH
D3 H MH H VH M
PT1 D1 H H VH VH H
D2 MH H VH H MH
D3 H MH H MH M
PT2 D1 H H VH H VH
D2 H H VVH MH VH
D3 H MH H MH H
AC D1 MH H H H M
D2 M M MH MH MH
D3 H MH H MH M
PO D1 MH MH H H VH
D2 M M MH MH H
D3 MH MH H MH H
CC D1 M MH MH H VH
D2 M M MH MH VH
D3 H MH M MH H
calculating weights has been also adopted in other se-
lection methods (see, for example [7,32,35]).
Step 2: Determine the level of support provided by each
alternative PPMIS
Though there is a large number of available PPMIS,
decision makers were queried to express their gen-
eral opinion on ten commercial PPMIS which in mar-
ket survey results [27] are characterised as leaders and
challengers in this segment of enterprise software mar-
ket. Five from these systems were excluded for two
reasons. First, since they do not have presence in the
national market and, second, because decision mak-
ers were persuaded that their usage was inappropriate
for the specific case, mainly due to lack of technical
support and non-availability of training services. This
first-level screening resulted in a list of five powerful,
widespread PPMIS with strong presence (i.e., techni-
cal/training support) in the national market. For confi-
dentiality reasons and aiming at avoiding the commer-
cial promotion of any software package, we will refer
to these PPMIS as A1, A2, A3, A4 and A5.
In order to evaluate the candidate PPMIS in a man-
ageable and reliable way, decision makers (evalua-
tors) rated the performance of each system with re-
spect to the criteria previously identified. Each deci-
sion maker was asked to carefully rate the support
provided by each system on each of the 11 criteria
(project phases/tasks) presented in Table 1. In addi-
tion to these 11 “positive” (benefit oriented) criteria,
two “negative” (cost oriented criteria) were decided to
be included in the list. These are the total price for
purchasing/ownership (PO) and the effort required to
customise/configure the PPMIS (CC). Thus, 13 cri-
teria in total were adopted. All decision makers pro-
vided a short written justification for every rating they
gave in linguistic terms. For their ratings decision mak-
ers used the linguistic terms presented in Table 3. For
the construction of Table 3, the so-called “Positive-
Confidence Approach” [33] was adopted, according to
which the degree of support offered by an evaluated
system to a certain criterion is made firm (i.e, the mem-
bership value), and the associated hesitation degree is
subtracted from the degree that the system does not
support the criterion (i.e, the non-membership value).
Decision makers expressed in a joint meeting that they
are rather confident in their judgements and they de-
cided hesitation degrees equal to 0 and 0.1 for “strong”
judgments (i.e., EH, VVH, VL, VVL) and “medium”
judgments (VH, H, MH, M, ML, L), respectively. De-
cision makers justified this agreement upon the hesi-
tation degrees by commenting that: i) they have expe-
rience in utilizing these 5 candidate PPMIS, and thus
they feel quite determinant in their judgments and ii)
the candidate systems are commercial tools (and not
open source products) and the level of functionality
that can be easily implemented (by configuration) to
achieve a not- supported functionality is low. To check
the validity of the ratings, decision makers were also
asked to cross-check their marks, according to the cor-
responding “4-stars” scores, as they are listed for each
tool in [22]. All ratings finally given by the three de-
cision makers to the five PPMIS alternatives are pre-
sented in Table 4.

10. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 99
Table 5
Aggregated intuitionistic fuzzy decision matrix
A1 A2 A2 A4 A5
IGLM 0.746 0.769 0.679 0.663 0.668
0.151 0.128 0.220 0.236 0.231
0.104 0.103 0.101 0.101 0.101
IE 0.615 0.538 0.746 0.591 0.631
0.282 0.361 0.151 0.306 0.265
0.103 0.101 0.104 0.103 0.104
PP1 0.600 0.644 0.826 0.728 0.800
0.300 0.254 0.128 0.166 0.100
0.100 0.101 0.046 0.106 0.100
PP2 0.639 0.596 0.769 0.698 0.769
0.260 0.302 0.128 0.192 0.128
0.101 0.103 0.103 0.110 0.103
PP3 0.780 0.740 0.728 0.718 0.744
0.118 0.156 0.166 0.174 0.148
0.102 0.103 0.106 0.108 0.108
PC1 0.679 0.746 0.746 0.718 0.668
0.220 0.151 0.151 0.174 0.231
0.101 0.104 0.104 0.108 0.101
PC2 0.679 0.578 0.746 0.668 0.769
0.220 0.321 0.151 0.231 0.128
0.101 0.101 0.104 0.101 0.103
PC3 0.679 0.718 0.619 0.740 0.723
0.220 0.174 0.278 0.156 0.164
0.101 0.108 0.103 0.103 0.113
PT1 0.679 0.668 0.769 0.718 0.615
0.220 0.231 0.128 0.174 0.282
0.101 0.101 0.103 0.108 0.103
PT2 0.700 0.668 0.804 0.644 0.769
0.200 0.231 0.128 0.254 0.128
0.100 0.101 0.068 0.101 0.103
AC 0.619 0.625 0.679 0.644 0.526
0.278 0.272 0.220 0.254 0.374
0.103 0.103 0.101 0.101 0.101
PO 0.578 0.578 0.679 0.644 0.746
0.321 0.321 0.220 0.254 0.151
0.101 0.101 0.101 0.101 0.104
CC 0.583 0.578 0.567 0.644 0.769
0.312 0.321 0.332 0.254 0.128
0.104 0.101 0.101 0.101 0.103
Based on these ratings and the weights of deci-
sion makers, the aggregated intuitionistic fuzzy de-
cision matrix (AIFDM) was calculated by applying
the intuitionistic fuzzy weighted averaging (IFWA) op-
erator [31]. The basic steps of the IFWA operator
are that it first weights all given IFNs by a normal-
ized weight vector, and then aggregates these weighted
IFNs by addition. Each result derived by using the
IFWA operator is an IFN. If A = {A1, A2, . . . , Am}
is the set of alternatives and X = {X1, X2, . . . , Xn}
is the set of criteria, then AIFDM R is an m × n
matrix with elements IFNs in the form of rij =
[μAi (xj), vAi (xj), πAi (xj)], where i = 1, 2, . . ., m
and j = 1, 2, . . ., n.
By considering weights λk(k = 1, 2, . . . , l) of l de-
cision makers, the elements rij of the AIFDM can be
calculated using IFWA as follows:
rij = IFWAλ(r
(1)
ij , r
(2)
ij , . . . , r
(l)
ij )
= λ1r
(1)
ij ⊕ λ2r
(2)
ij ⊕ λ3r
(3)
ij ⊕ . . . ⊕ λlr
(l)
ij
= 1 −
l
k=1
(1 − μ
(k)
ij )λk
,
l
k=1
(v
(k)
ij )λk
,
l
k=1
(1 − μ
(k)
ij )λk
−
l
k=1
(v
(k)
ij )λk
(5)
The AIFDM for the case problem is shown in Table 5.
The matrix IFNs were calculated by substituting in
Eq. (5) the weights of the three (l = 3) decision mak-
ers (λD1 = 0.406, λD2 = 0.238, λD3 = 0.356) and
the IFNs (μ
(k)
ij , v
(k)
ij , π
(k)
ij ) produced by using the re-
lationships of Table 3 (i.e., these IFNs correspond to
ratings given by the k decision maker on each sys-
tem Ai (i = 1, 2, . . . , 5) with respect to each criterion
j (j = 1, 2, . . ., 13).
For example, in Table 5, the IFN [0.769, 0.128,
0.103], shown in bold, is the aggregated score of PP-
MIS A2 on criterion IGLM (Idea Generation/Lead
Mgmt.), while the IFN [0.600, 0.300, 0.100], also
shown in bold, is the aggregated score of PPMIS A1
on criterion PP1 (Portfolio Planning).
Step 3: Determine the weights of the selection criteria
To analyse users’ requirements from a PPMIS we
disseminated to the three users/members of HOU a
structured questionnaire, asking from them to evaluate
the 13 selection criteria and express their perceptions
on the relative importance of each one criterion with
respect to the overall performance and benefits pro-
vided from a candidate PPMIS. Each of the 3 users was
requested to answer 13 questions by denoting a grade
for the importance of each criterion in a linguistic term,
as it is shown in column 2 of Table 2. Opinions of
users U1, U2 and U3 on the importance of the crite-
ria are presented in Table 6. These preferences are as-
signed to corresponding IFNs by using the relation-
ships between values in column 2 and values in col-
umn 3 of Table 2.
The IFWA operator was also used to calculate the
weights of criteria by aggregating the opinions of the
users. Let w
(k)
j = (μ
(k)
j , v
(k)
j , π
(k)
j ) be the IFN as-
signed to criterion j (j = 1, 2, . . ., n) by the k user
(k = 1, 2, . . . , l). Then the weight of j can be calcu-
lated as follows:

11. 100 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
Table 6
Importance values of the criteria
Criteria
Users
U1 U2 U3
IGLM VI I M
IE M VI I
PP1 M VI VI
PP2 VI VI VI
PP3 I VI VI
PC1 M VI VI
PC2 M VI I
PC3 M M VI
PT1 I VI VI
PT2 VI M I
AC VI I I
PO VI VI M
CC I M VI
Table 7
Weights of the criteria
Criteria
Weights
μ v π
IGLM 0.779 0.201 0.019
IE 0.734 0.236 0.031
PP1 0.808 0.184 0.008
PP2 0.900 0.100 0.000
PP3 0.855 0.133 0.013
PC1 0.808 0.184 0.008
PC2 0.734 0.236 0.031
PC3 0.718 0.263 0.018
PT1 0.855 0.133 0.013
PT2 0.797 0.183 0.020
AC 0.828 0.151 0.021
PO 0.823 0.171 0.007
CC 0.787 0.189 0.023
wj =IFWAλ(w
(1)
j , w
(2)
j , . . . , w
(l)
j )
=λ1w
(1)
j ⊕λ2w
(2)
j ⊕λ3w
(3)
j ⊕. . .⊕λlw
(l)
j
= 1 −
l
k=1
(1 − μ
(k)
j )λk
,
l
k=1
(v
(k)
j )λk
,
l
k=1
(1 − μ
(k)
j )λk
−
l
k=1
(v
(k)
j )λk
(6)
Thus, a vector of criteria weights is obtained W =
[w1, w2, . . . , wj], where each weight wj is an IFN in
the form [μj, vj, πj] (j = 1, 2, . . . , n). In the case
problem, substituting in Eq. (6) the weights of three
users (λU1 = 0.406, λU2 = 0.238, λU3 = 0.356) and
using the IFNs which correspond to linguistic values of
Table 6 yielded the criteria weights shown in Table 7.
Step 4: Compose the aggregated weighted intuitionistic
fuzzy decision matrix
In this step, the aggregated weighted intuitionistic
fuzzy decision (AWIFDM) matrix R is composed by
considering the aggregated intuitionistic fuzzy deci-
sion matrix (i.e., table R produced in step 2) and the
Table 8
Aggregated weighted intuitionistic fuzzy decision matrix
A1 A2 A3 A4 A5
IGLM 0.581 0.599 0.529 0.517 0.520
0.322 0.304 0.377 0.390 0.386
0.097 0.097 0.094 0.094 0.094
IE 0.451 0.395 0.547 0.433 0.463
0.451 0.512 0.351 0.470 0.438
0.098 0.094 0.102 0.097 0.099
PP1 0.485 0.520 0.667 0.588 0.646
0.429 0.392 0.289 0.320 0.266
0.086 0.088 0.044 0.093 0.088
PP2 0.575 0.536 0.692 0.628 0.692
0.334 0.372 0.215 0.273 0.215
0.091 0.092 0.093 0.099 0.093
PP3 0.667 0.633 0.622 0.614 0.636
0.235 0.268 0.277 0.284 0.261
0.099 0.099 0.101 0.102 0.103
PC1 0.548 0.602 0.602 0.580 0.539
0.364 0.307 0.307 0.326 0.373
0.088 0.090 0.090 0.094 0.088
PC2 0.498 0.424 0.547 0.490 0.564
0.404 0.481 0.351 0.412 0.334
0.098 0.095 0.102 0.098 0.102
PC3 0.488 0.516 0.445 0.532 0.519
0.426 0.392 0.468 0.378 0.384
0.087 0.092 0.087 0.090 0.097
PT1 0.580 0.571 0.657 0.614 0.525
0.324 0.333 0.244 0.284 0.377
0.096 0.096 0.099 0.102 0.097
PT2 0.558 0.532 0.641 0.513 0.613
0.346 0.372 0.288 0.391 0.288
0.096 0.096 0.072 0.096 0.100
AC 0.513 0.517 0.562 0.533 0.435
0.387 0.382 0.338 0.367 0.468
0.100 0.101 0.100 0.100 0.097
PO 0.476 0.476 0.558 0.530 0.613
0.437 0.437 0.353 0.382 0.296
0.087 0.087 0.088 0.088 0.091
CC 0.459 0.455 0.446 0.507 0.605
0.443 0.450 0.459 0.396 0.293
0.098 0.095 0.095 0.097 0.101
vector of the criteria weights (i.e., table W produced
in step 3). Step 4 is necessary to synthesize the ratings
of both decision makers and users. In particular, ele-
ments of the AWIFDM can be calculated by using the
multiplication operator of IFS as follows:
R ⊗ W = {< x, μAi (x) · μW (x), vAi (x)
+vW (x) − vAi (x) · vW (x) > |x ∈ X}
(7)
R is an m×n matrix composed with IFNs in the form
of rij = [μAiW (xj), vAiW (xj), πAiW (xj)], where:
μAiW (xj), vAiW (xj) are values derived by apply-
ing Eq. (7). The hesitation degree can be computed
each time by subtracting the sum of these two values

12. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 101
Table 9
Separation measures and relative closeness coefficient of each
PPMIS
PPMIS S∗ (1) S− (2) C∗ (3)
A1 0.076 0.074 0.495
A2 0.091 0.074 0.448
A3 0.041 0.116 0.737
A4 0.069 0.074 0.520
A5 0.088 0.085 0.490
(μAiW (xj), vAiW (xj)) from 1:
πAiW (x)=1− vAi (x)−vW (x)
−μAi (x)·μW (x)+vAi (x)·vW (x)
(8)
In the case problem, substituting in Eq. (7) the IFNs
of Table 5 (table R) and IFNs of Table 7 (table W)
yielded the IFNs of the AWIFDM (table R ) pre-
sented in Table 8. For example, in Table 8, the IFN
[0.599, 0.304, 0.097], shown in bold, is the aggregated
weighted score of PPMIS A2 on criterion IGLM (Idea
Generation/Lead Mgmt.), while the IFN [0.485, 0.429,
0.086], also shown in bold, is the aggregated weighted
score of PPMIS A1 on criterion PP1 (Portfolio Plan-
ning).
Step 5: Compute the intuitionistic fuzzy positive ideal
solution and the intuitionistic fuzzy negative ideal so-
lution
To apply the TOPSIS method the intuitionistic fuzzy
positive ideal solution (IFPIS) A∗
and the intuitionis-
tic fuzzy negative ideal solution (IFNIS) A−
have to
be determined. Both solutions are vectors of IFN ele-
ments and they are derived from the AWIFDM matrix
as follows. Let B and C be the sets of benefit and cost
criteria, respectively. Then A∗
and A−
are equal to:
A∗
= (μA∗W (xj), vA∗W (xj))
and
A−
= (μA−W (xj), vA−W (xj))
where
μA∗W (xj) = ((max
i
μAiW (xj)|j ∈ B),
(min
i
μAiW (xj)|j ∈ C))
vA∗W (xj) = ((min
i
vAiW (xj)|j ∈ B),
(max
i
vAiW (xj)|j ∈ C))
μA−W (xj) = ((min
i
μAiW (xj)|j ∈ B),
(max
i
μAiW (xj)|j ∈ C))
vA−W (xj) = ((max
i
vAiW (xj)|j ∈ B),
(min
i
vAiW (xj)|j ∈ C))
(9)
In the case problem, B = {IGLM, IE, PP1, PP2, PP3,
PC1, PC2, PC3, PT1, PT2, AC} and C = {PO, CC}.
To obtain IFPIS and IFNIS, Eq. (9) was applied on the
IFNs of the AWIFDM decision matrix. The IFPIS and
IFNIS were determined as follows:
A∗
= ([0.599, 0.304, 0.097], [0.547, 0.351, 0.102],
[0.667, 0.289, 0.044], [0.692, 0.215, 0.093],
[0.667, 0.235, 0.099], [0.602, 0.307, 0.090],
[0.564, 0.334, 0.102], [0.532, 0.378, 0.090],
[0.657, 0.244, 0.099], [0.641, 0.288, 0.072],
[0.562, 0.338, 0.100], [0.476, 0.437, 0.087],
[0.446, 0.459, 0.095])
A−
= ([0.517, 0.390, 0.094], [0.395, 0.512, 0.094],
[0.485, 0.429, 0.086], [0.536, 0.372, 0.092],
[0.614, 0.284, 0.102], [0.539, 0.373, 0.088],
[0.424, 0.481, 0.095], [0.445, 0.468, 0.087],
[0.525, 0.377, 0.097], [0.513, 0.391, 0.096],
[0.435, 0.468, 0.097], [0.613, 0.296, 0.091],
[0.605, 0.293, 0.101])
Step 6: Calculate the separation between the alterna-
tive PPMIS
Next, the separation measures Si∗ and Si− can be
calculated for each candidate system Ai from the IF-
PPIS and the IFNIS, respectively. As a distance mea-
sure, the normalized Euclidean distance was adopted,
since it has been proved to be a reliable distance
measure that takes into account not only membership
and non-membership but also the hesitation part of
IFNs [28]. For each alternative system these two sepa-
ration values can be calculated as follows:
S∗
=
1
2n
n
j=1
[(μAiW (xj) − μA∗W (xj))2
+(vAiW (xj) − vA∗W (xj))2
+(πAiW (xj) − πA∗W (xj))2
]
S−
=
1
2n
n
j=1
[(μAiW (xj) − μA−W (xj))2
+(vAiW (xj) − vA−W (xj))2
+(πAiW (xj) − πA−W (xj))2
]
(10)
By utilizing these Eq. (10), the positive and nega-
tive separation measures for the five alternative PPMIS
were calculated. These are shown in columns (1) and
(2) of Table 9.

13. 102 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
Table 10
Sensitivity analysis results (based on criteria weights)
Exp. Criteria weights Scores of PPMIS Ranking
A1 A2 A3 A4 A5
1 w1−13 = [0.10, 0.90] 0.497 0.447 0.730 0.514 0.494 A3 > A4 > A1 > A5 > A2
2 w1−13 = [0.25, 0.70] 0.502 0.451 0.728 0.519 0.495 A3 > A4 > A1 > A5 > A2
3 w1−13 = [0.50, 0.45] 0.499 0.449 0.729 0.517 0.494 A3 > A4 > A1 > A5 > A2
4 w1−13 = [0.75, 0.20] 0.498 0.448 0.729 0.516 0.494 A3 > A4 > A1 > A5 > A2
5 w1−13 = [0.90, 0.10] 0.497 0.447 0.730 0.514 0.494 A3 > A4 > A1 > A5 > A2
6 w1 = [0.90, 0.10], w2−13 = [0.10, 0.90] 0.674 0.712 0.389 0.247 0.278 A2 > A1 > A3 > A5 > A4
7 w2 = [0.90, 0.10], w1,3−13 = [0.10, 0.90] 0.388 0.147 0.910 0.291 0.458 A3 > A5 > A1 > A4 > A2
8 w3 = [0.90, 0.10], w1−2,4−13 = [0.10, 0.90] 0.152 0.257 0.909 0.595 0.763 A3 > A5 > A4 > A2 > A1
9 w4 = [0.90, 0.10], w1−3,5−13 = [0.10, 0.90] 0.294 0.171 0.896 0.597 0.804 A3 > A5 > A4 > A2 > A1
10 w5 = [0.90, 0.10], w1−4,6−13 = [0.10, 0.90] 0.650 0.413 0.516 0.349 0.477 A1 > A3 > A5 > A2 > A4
11 w6 = [0.90, 0.10], w1−5,7−13 = [0.10, 0.90] 0.333 0.656 0.821 0.601 0.320 A3 > A2 > A4 > A1 > A5
12 w7 = [0.90, 0.10], w1−6,8−13 = [0.10, 0.90] 0.521 0.158 0.850 0.473 0.819 A3 > A5 > A1 > A4 > A2
13 w8 = [0.90, 0.10], w1−7,9−13 = [0.10, 0.90] 0.487 0.683 0.318 0.784 0.718 A4 > A5 > A2 > A1 > A3
14 w9 = [0.90, 0.10], w1−8,10−13 = [0.10, 0.90] 0.426 0.363 0.885 0.651 0.210 A3 > A4 > A1 > A2 > A5
15 w10 = [0.90, 0.10], w1−9,11−13 = [0.10, 0.90] 0.400 0.250 0.881 0.197 0.703 A3 > A5 > A1 > A2 > A4
16 w11 = [0.90, 0.10], w1−10,12−13 = [0.10, 0.90] 0.593 0.603 0.885 0.717 0.211 A3 > A4 > A2 > A1 > A5
17 w12 = [0.90, 0.10], w1−11,13 = [0.10, 0.90] 0.821 0.791 0.464 0.592 0.196 A1 > A2 > A4 > A3 > A5
18 w13 = [0.90, 0.10], w1−12 = [0.10, 0.90] 0.818 0.806 0.908 0.606 0.170 A3 > A1 > A2 > A4 > A5
Table 11
Sensitivity analysis results (based on performance ratings)
Exp. Performance ratings Scores of PPMIS Ranking
A1 A2 A3 A4 A5
1 VVH=[0.9,0.1,0] VH=[0.7,0.3,0] H=[0.5,0.5,0] 0.477 0.436 0.744 0.505 0.491 A3 > A4 > A5 > A1 > A2
MH=[0.3,0.7,0] M=[0.1,0.9,0]
Positive-confidence scale
2 VVH=[0.9,0,0.1] VH=[0.7,0.2,0.1] 0.445 0.414 0.758 0.473 0.477 A3 > A5 > A4 > A1 > A2
H=[0.5,0.4,0.1] MH=[0.3,0.6,0.1] M=[0.1,0.8,0.1]
3 VVH=[0.9,0,0.1] VH=[0.7,0.1,0.2] 0.467 0.433 0.748 0.505 0.489 A3 > A4 > A5 > A1 > A2
H=[0.5,0.3,0.2] MH=[0.3,0.5,0.2] M=[0.1,0.7,0.2]
4 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] 0.481 0.462 0.731 0.560 0.483 A3 > A4 > A5 > A1 > A2
H=[0.5,0.2,0.3] MH=[0.3,0.4,0.3] M=[0.1,0.6,0.3]
5 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] 0.497 0.452 0.737 0.532 0.485 A3 > A4 > A1 > A5 > A2
H=[0.5,0.1,0.4] MH=[0.3,0.3,0.4] M=[0.1,0.5,0.4]
6 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] 0.498 0.452 0.727 0.517 0.494 A3 > A4 > A1 > A5 > A2
H=[0.5,0,0.5] MH=[0.3,0.2,0.5] M=[0.1,0.4,0.5]
7 VVH=[0.9,0,0.1] VH=[0.7,0,0.3] 0.477 0.433 0.738 0.506 0.497 A3 > A4 > A5 > A1 > A2
H=[0.5,0,0.5] MH=[0.3,0.1,0.6] M=[0.1,0.3,0.6]
Negative-confidence scale
8 VVH=[0.8,0.1,0.1] VH=[0.6,0.3,0.1] 0.480 0.437 0.743 0.506 0.491 A3 > A4 > A5 > A1 > A2
H=[0.4,0.5,0.1] MH=[0.2,0.7,0.1] M=[0,0.9,0.1]
9 VVH=[0.7,0.1,0.2] VH=[0.5,0.3,0.2] 0.470 0.436 0.741 0.500 0.496 A3 > A4 > A5 > A1 > A2
H=[0.3,0.5,0.2] MH=[0.1,0.7,0.2] M=[0,0.9,0.1]
10 VVH=[0.6,0.1,0.3] VH=[0.4,0.3,0.3] 0.464 0.437 0.745 0.500 0.502 A3 > A5 > A4 > A1 > A2
H=[0.2,0.5,0.3] MH=[0,0.7,0.3] M=[0,0.9,0.1]
11 VVH=[0.5,0.1,0.4] VH=[0.3,0.3,0.4] 0.472 0.435 0.746 0.504 0.496 A3 > A4 > A5 > A1 > A2
H=[0.1,0.5,0.4] MH=[0,0.7,0.3] M=[0,0.9,0.1]
12 VVH=[0.4,0.1,0.5] VH=[0.2,0.3,0.5] 0.426 0.435 0.677 0.434 0.458 A3 > A5 > A2 > A4 > A1
H=[0,0.5,0.5] MH=[0,0.7,0.3] M=[0,0.9,0.1]
13 VVH=[0.3,0.1,0.6] VH=[0.1,0.3,0.6] 0.436 0.436 0.682 0.436 0.449 A3 > A5 > A4 > A1 > A2
H=[0,0.5,0.5] MH=[0,0.7,0.3] M=[0,0.9,0.1]
Step 7: Determine the final ranking of PPMIS
The final score of each system was derived by calcu-
lating the corresponding relative closeness coefficient
with respect to the intuitionistic fuzzy ideal solution.
For each alternative Ai, the relative closeness coeffi-
cient Ci∗ with respect to the IFPIS is defined as fol-

14. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 103
Fig. 3. Screenshots of the method implementation in spreadsheets. (Colours are visible in the online version of the article; http://dx.doi.org/
10.3233/IDT-120153)
lows:
Ci∗ =
Si−
Si∗ + Si−
(11)
where 0 Ci∗ 1.
Equation (11) was used to calculate these coeffi-
cients (final scores) listed in column (3) of Table 9. The
alternative PPMIS were ranked in a descending order
of these scores as A3 > A4 > A1 > A5 > A2, from
where it can be deduced that alternative A3 is the most
dominant PPMIS for the present case study.
Step 8: Sensitivity analysis
Sensitivity analysis is concerned with ‘what-if’ kind
of scenarios to determine if the final answer (ranking)
is stable to changes (experiments) in the inputs, either
judgments of alternatives or weights of criteria. In the
present case, sensitivity analysis was first performed
by examining the impact of criteria weights (i.e., the
weights of users’ requirements from a PPMIS) on the
final PPMIS ranking. Of special interest was to see if
criteria weights’ changes alter the order of the alter-
natives. 18 experiments were conducted in a similar
way with the approach presented in [5]. The details
of all experiments are shown in Table 10, where w1,
w2, . . . , w13 denote respectively the weights of crite-
ria IGLM, IE, PP1, PP2, PP3, PC1, PC2, PC3, PT1,
PT2, AC, PO, CC. In Exps 1–5, weights of all criteria
were set equal to [0.10, 0.90], [0.25, 0.70], [0.50, 0.45],
[0.75, 0.20] and [0.90, 0.10], respectively. These IFNs
correspond to the linguistic terms VU, U, M, I and VI,
respectively (see Table 2). In Exps 6–18, the weight of
each of the 13 criteria was set equal to the highest IFN
[0.90,0.10], one by one, and the weights of the rest of
criteria were set all equal to the lowest IFN [0.10,0.90].
The results show that PPMIS A3 remains the domi-
nant alternative in 14 out of the 18 experiments (this
represents a clear “majority” equal to 77.77%). PPMIS
A1 was first in 2/18 experiments, namely in Exps 10
and 17, where the highest weights were assigned, re-
spectively, to criterion PP3 (project planning) and cri-
terion PO (total price for purchasing/ownership). Sys-
tem A2 had the highest score in Exp. 6, where the high-
est weight was assigned to criterion IGLM (Idea Gen-
eration/Lead Management), while system A4 had the
highest score in Exp. 13, where the highest value was
assigned to the weight of PC3 (portfolio controlling).
Further sensitivity analysis on the final ranking can
be performed by changing the IFNs presented in Ta-
ble 2 and Table 3. For example, we can notice (in Ta-
ble 4) that decision makers have utilized specific lin-
guistic terms (i.e., VVH, VH, H, MH and M) to ex-
press their judgments on the performances of the al-
ternative PPMIS with respect to the evaluation criteria.
Table 11 shows 13 additional experiments applied to
study the sensitivity of the final ranking with different
values of IFNs for the utilized linguistic terms (VVH,
VH, H, MH and M). Each experiment is associated
with a different degree of hesitation. Table 11 presents
the rankings finally produced by: i) considering that
hesitation degrees are all equal to zero (Exp. 1), ii) in-
creasing gradually the hesitation degrees and consider-
ing that hesitation is subtracted from non-membership
(Exps 2–7), iii) increasing gradually the hesitation de-

15. 104 V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems
grees and considering that hesitation is subtracted from
membership (Exps 8–13). From Table 11, it can be
seen that the “best” and “worst” PPMIS are not sensi-
tive to changing hesitation degrees. PPMIS A3 was the
most preferable alternative in all experiments, while
PPMIS A2 was the least preferable alternative in 12
out of the 13 experiments. Thus, by applying sensitiv-
ity analysis we can conclude, with a high confidence,
that system A3 is the most suitable PPMIS.
Generalization and further validation of the pre-
sented approach require the use of a fully parame-
terised form of the hesitation degree. This can be per-
formed in two ways: i) by asking users/evaluators (de-
cision makers) to express also a different hesitation de-
gree for each assessment, based on either a Positive-
Confidence or a Negative-Confidence approach [33]
or ii) by asking users/evaluators to express their judg-
ments by utilizing interval-valued intutionistic fuzzy
numbers [23]. We have plans to investigate these two
solutions in a future research. In addition, we intend
to apply the decision making approach in software se-
lection problems which involve large number of stake-
holders and decision makers.
7. Conclusions
The paper presented, through a case study, the ap-
plication of a group-based multi criteria decision mak-
ing (MCDM) method for the evaluation and final se-
lection of an appropriate Project and Portfolio Man-
agement Information System (PPMIS). The applied
method jointly synthesized intuitionistic fuzzy sets and
TOPSIS. The benefit from this combination in a PP-
MIS selection approach is twofold: First, the approach
actively involves decision makers and PPMIS users
in the decision making process and aggregates their
opinions to support agreement upon the final selection.
Second, the approach considers that they both express
their judgments under inherent uncertainty. More sig-
nificantly, the approach handles adequately the degree
of indeterminacy that characterizes both decision mak-
ers and users in their evaluations. This is very impor-
tant when an organization needs to decide upon the se-
lection of any new, multi-functional information sys-
tem, as in our case is a suitable PPMIS, since decision
makers often cannot have full knowledge of the extend
that each candidate system will (or will not) support
the user requirements. System users, on the other hand,
can be unfamiliar with the processes supported by the
required system, and thus, they cannot judge with cer-
tainty the importance of their needs.
The presented approach not only validated the me-
thod, as it was originally defined in [7], in a new ap-
plication field that is the evaluation of PPMIS (where
other MCDM approaches are rather limited in the lit-
erature), but also considered a more extensive list of
benefit and cost-oriented criteria, suitable for PPMIS
selection. In addition, final results were verified by
applying sensitivity analysis. We should mention that
the method underlying computations are not transpar-
ent to the problem stakeholders which utilise linguis-
tic terms to state evaluations/preferences. Actually, we
implemented the method in a spreadsheet program that
helps to effectively and practically apply the approach
with a variety of inputs. Example screenshots of this
spreadsheet are shown in Fig. 3. Figure 3(a) presents
an excerpt of user opinions on the importance of the
criteria (an excerpt of the input data shown in Ta-
ble 6). Figure 3(b) presents an excerpt of the criteria
weights (an excerpt of the data shown in Table 7). Fig-
ure 3(c) presents excerpts of: i) the aggregated intu-
itionistic fuzzy decision matrix (Table 5), ii) the ag-
gregated weighted intuitionistic fuzzy decision matrix
(Table 8), iii) the intuitionistic fuzzy positive ideal and
negative ideal solutions (step 5 of the method).
The approach raises several issues that could spark
further research. For example, an interesting idea could
be to validate the approach applicability in address-
ing the selection of other types of software packages.
We are now investigating the selection of e-learning
management systems for the case organization (i.e., the
Hellenic Open University). In addition, treating more
with uncertainties would further strengthen the pro-
posed approach in deriving more precise results. We
have also plans to examine the utilization of more pow-
erful methods in the same domain, such as the interval-
valued intutionistic fuzzy sets [12,23].
Acknowledgments
The authors would like to thank the anonymous re-
viewers for their helpful suggestions, as well as Ilias
Maglogiannis, Lazaros Iliadis and Harry Papadopou-
los for their kind invitation to participate in the special
issue of the Intelligent Decision Technologies Jour-
nal. This paper is an updated and extended version of
an article presented in the 12th EANN/7th AIAI 2011
Conference. The research presented in this paper has
been co-financed by the European Union (European
Social Fund) and Greek national funds through the
Operational Program “Education and Lifelong Learn-
ing” of the National Strategic Reference Framework.

16. V.C. Gerogiannis et al. / Evaluation of project and portfolio Management Information Systems 105
In particular, the research work has been co-financed
by the funding program “MIS 296121 – Hellenic Open
University” and the R&D program “SPRINT SMEs –
ARCHIMEDES III”.
References
[1] F. Ahlemann, Towards a conceptual reference model for
project management information systems, International Jour-
nal of Project Management 27(1) (2009), 19–30.
[2] N. Ahmad and P. Laplante, Software project management
systems: Making a practical decision using AHP, in: 30th
Annual IEEE/NASA Software Engineering Workshop, IEEE
Press, 2006, pp. 76–84.
[3] A.S.B. Ali and W.H. Money, A study of project management
system acceptance, in: 38th Hawaii International Conference
on System Sciences, IEEE Press, 2005, pp. 234–244.
[4] K.T. Atanassov, Intuitionistic fuzzy sets, Fuzzy Sets and Sys-
tems 20(1) (1986), 87–96.
[5] A. Awasthi, S.S. Chauhan and S.K. Goyal, A fuzzy multi-
criteria approach for evaluating environmental performance
of suppliers, International Journal of Production Economics
126(2) (2010), 370–378.
[6] BARC (2010), Project management software systems report,
Business Application Research Center (BARC), Retrieved
May 2, 2011, from http://www.pm-software-report.com.
[7] F.E. Boran, S. Genc, M. Kurt and D. Akay, A multi-criteria
intuitionistic fuzzy group decision making for supplier selec-
tion with TOPSIS method, Expert Systems with Applications
36(8) (2009), 11363–11368.
[8] F.E. Boran, S. Genc, M. Kurt and D. Akay, Personnel selec-
tion based on intuitionistic fuzzy sets, Human Factors and
Ergonomics in Manufacturing and Service Industries 21(5)
(2011), 493–503.
[9] C.E. Bozdag, C. Kahraman and D. Ruan, Fuzzy group deci-
sion making for selection among computer integrated manu-
facturing systems, Computers in Industry 51(1) (2003), 3–29.
[10] D.Y. Chang, Application of extent analysis method on fuzzy
AHP, European Journal of Operational Research 95(3)
(1996), 649–655.
[11] C.T. Chen, Extensions of the TOPSIS for group decision-
making under fuzzy environment, Fuzzy Sets and Systems
114(1) (2000), 1–9.
[12] T.Y. Chen, H.P. Wang and Y.Y. Lu, A multicriteria group
decision-making approach based on interval-valued intuition-
istic fuzzy sets: A comparative perspective, Expert Systems
with Applications 38(6) (2011), 7647–7658.
[13] J.K. Cochran and H.N. Chen, Fuzzy multi-criteria selection
of object-oriented simulation software for production system
analysis, Computers and Operations Research 32(1) (2005),
153–168.
[14] S.K. De, R. Biswas and A.R. Roy, An application of intuition-
istic fuzzy sets in medical diagnosis, Fuzzy Sets and Systems
117(2) (2001), 209–213.
[15] V.C. Gerogiannis, P. Fitsilis, D. Voulgaridou, K.A. Kirytopou-
los and E. Sachini, A case study for project and portfolio man-
agement information system selection: A group AHP-scoring
model approach, International Journal of Project Organisa-
tion and Management 2(4) (2010), 361–381.
[16] D.C. Gomez, A. Alexander, D. Anderson, D. Cook, K. Poole
and O. Findlay, NASA Project Management Tool Analysis
and Recommendations White Paper (Tech Rep), 2004, Project
Management Tool Working Group, NASA Glenn Research
Center, Retrieved May 2, 2011, from http://km.nasa.gov/
pdf/54927main_pm-tool-paper.pdf.
[17] F. Herrera and E. Herrera-Viedma, Linguistic decision anal-
ysis: Steps for solving decision problems under linguistic in-
formation, Fuzzy Sets and Systems 115(1) (2000), 67–82.
[18] W. Ho, Integrated analytic hierarchy process and its applica-
tions – a literature review, European Journal of Operational
Research 186(1) (2008), 211–218.
[19] A.S. Jadhav and R.M. Sonar, Evaluating and selecting soft-
ware packages: A review, Information and Software Technol-
ogy 51(3) (2009), 555–563.
[20] M.J. Liberatore and B. Pollack-Johnson, Factors influenc-
ing the usage and selection of project management soft-
ware, IEEE Transactions on Engineering Management 50(2)
(2003), 164–174.
[21] H.Y. Lin, P.Y. Hsu and G.J. Sheen, A fuzzy-based decision-
making procedure for data warehouse system selection, Ex-
pert System with Applications 32(3) (2007), 939–953.
[22] M.M. Meyer and F. Ahlemann, Project Management Software
Systems, 6th edition, Business Application Research Center,
Wurzburg, Germany, 2010.
[23] J.H. Park, I.Y. Park, Y.C. Kwun and X. Tan, Extension of the
TOPSIS method for decision making problems under interval-
valued intuitionistic fuzzy environment, Applied Mathemati-
cal Modelling 35(5) (2011), 2544–2556.
[24] PMI, Project management software survey, Project Manage-
ment Institute (PMI), Newtown Square, 1999.
[25] PMI, A guide to the project management body of knowledge,
4th edition, Project Management Institute (PMI), Newtown
Square, 2008.
[26] L. Raymond and F. Bergeron, Project management infor-
mation systems: An empirical study of their impact on
project managers and project success, International Journal
of Project Management 26(2) (2008), 213–220.
[27] D.B. Stang, Magic quadrant for IT project and portfolio man-
agement, Gartner RAS Core Research Note, Gartner Re-
search, 2010.
[28] E. Szmidt and J. Kacprzyk, Distances between intuitionistic
fuzzy sets, Fuzzy Sets and Systems 114(3) (2000), 505–518.
[29] O.S. Vaidya and S. Kumar, Analytic hierarchy process: An
overview of applications, European Journal of Operational
Research 169(1) (2006), 1–29.
[30] Z.S. Xu and R.R. Yager, Some geometric aggregation opera-
tors based on intuitionistic fuzzy sets, International Journal
of General Systems 35(4) (2006), 417–433.
[31] Z.S. Xu, Intuitionistic fuzzy aggregation operators, IEEE
Transactions on Fuzzy Systems 15(6) (2007), 1179–1187.
[32] M. Xu, Q. Chen and C. Ligang, An improved approach for
supplier selection in project material bidding procurement,
in: Computational Risk Management, Modeling Risk Man-
agement in Sustainable Construction, Part 1, Springer, 2011,
pp. 3–10.
[33] P. Wang, QoS-aware web services selection with intuitionistic
fuzzy set under consumer’s vague perception, Expert Systems
with Applications 36(3) (2009), 4460–4466.
[34] D. Wierschem and C. Johnston, The role of project manage-
ment in university computing resource departments, Interna-
tional Journal of Project Management 23(8) (2005), 640–649.
[35] S. Zhang and S. Liu, A GRA-based intuitionistic fuzzy multi-
criteria group decision making method for personnel selec-
tion, Expert Systems with Applications 38(9) (2011), 11401–
11405.