The diagnosis of automation systems in an existing production system through the analysis of the interoperability level between its components represents the first step in its evaluation for improving it towards alignment with Industry 4.0. The analysis of the level of interoperability finds in the multi-criteria decision making methods an auxiliary tool to evaluate and classify the solution. This work uses the AHP method for this evaluation, drawing criteria from the literature and expert experience. The application of the method is illustrated, showing that it can direct investment decisions.

1. 2351-9789 © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee of the 27th International Conference on Flexible Automation and Intelligent Manufacturing
doi:10.1016/j.promfg.2017.07.321
Procedia Manufacturing 11 (2017) 1837 – 1845
Available online at www.sciencedirect.com
ScienceDirect
27th International Conference on Flexible Automation and Intelligent Manufacturing, FAIM2017,
27-30 June 2017, Modena, Italy
Evaluation of interoperability between automation systems using
multi-criteria methods
Maicon Saturnoa,b
*, Luiz Felipe Pierin Ramosa
, Fabrício Polatoa
, Fernando Deschampsa,c
,
Eduardo de Freitas Rocha Louresa,d
ªGraduate Program in Industrial and Systems Engineering (PPGEPS), Polytechnic School, Pontifical Catholic University of Parana (PUCPR),
Rua Imaculada Conceição, 1155 Curitiba, Paraná, Brazil
b
Dominus – Automação, Sistemas e Acionamentos, Avenida Manoel Ribas, 8.120 Curitiba, Paraná, Brazil
c
Department of Mechanical Engineering (DEMEC), Federal University of Paraná (UFPR), Rua Coronel Francisco Heráclito dos Santos, 230
Curitiba, Paraná, Brazil
d
Department of Electro-tecnology (DAEL), Federal University of Technology – Paraná (UTFPR), Avenida Sete de Setembro, 3033 Curitiba,
Paraná, Brazil
Abstract
The diagnosis of automation systems in an existing production system through the analysis of the interoperability level between
its components represents the first step in its evaluation for improving it towards alignment with Industry 4.0. The analysis of the
level of interoperability finds in the multi-criteria decision making methods an auxiliary tool to evaluate and classify the solution.
This work uses the AHP method for this evaluation, drawing criteria from the literature and expert experience. The application of
the method is illustrated, showing that it can direct investment decisions.
© 2017 The Authors. Published by Elsevier B.V.
Peer-review under responsibility of the scientific committee of the 27th International Conference on Flexible Automation and
Intelligent Manufacturing.
Keywords: Automation technology (AT), interoperability, diagnosis, automation systems, multi-criteria methods.
* Corresponding author. Tel.: +55 41 32748304.
E-mail address: maicon@dominus-eng.com
© 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee of the 27th International Conference on Flexible Automation and
Intelligent Manufacturing

2. 1838 Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
1. Introduction
The industrial automation area is responsible for joining technology suppliers and end-consumers concerned
about the continuous reduction of costs and losses in order to contribute to a company’s production of goods or
provision of services. The convergence of Automation Technology (AT) and Information Technology (IT) boosted
the development of these new solutions offering more flexibility and facilitating the integration of systems
composed by several manufacturers. This integration among different systems needs a set of interoperable
equipment and efficient inter communication to allow reliability and security in the planned system [1].
The objective of this study is to evaluate interoperability among systems within an existing automation platform
compared to a standard solution composed by an integrated system in a connected industry (Industry 4.0). The ISA
95 standard is used as a background pattern to help formulate the definitions that are evaluated in a consistent
automation system. This way, Multi-Criteria Decision Making (MCDM) methods are used to increase the credibility
and assertiveness of the solution [2, 3]. The criteria to be evaluated were identified using the concept of intelligent
and integrated solutions in which complex decisions are needed and the MCDM methods are used as an important
support tool for these decisions. The evaluation of interoperability among different automation systems goes toward
the implementation of new technological solutions and, according to Keith Nosbusch (2015), “the first step is to
have a standardized network with open protocols, sensors and intelligent instruments”.
Section 2 of this paper presents the literature review focused on interoperability and Industry 4.0. In sections 3
and 4, the AHP method is presented as a proposed evaluation model of automation systems. Its decision hierarchy
and the final results are also shown in these sections. Conclusions and future research can be found in section 5.
2. Interoperability and industry 4.0
2.1. ISA 95 Automation Framework: AT versus IT convergence
The codes and standards for technologies in the area of industrial automation are fundamental for choosing
criteria to evaluate an automation system. There is a great variety of technological solutions and manufacturers in
the market, and choosing them is a challenge for company’s investment, particularly considering all the different
advantages that each one offers to the customer. To ensure that the automation of a plant is aligned with the
company’s business strategy, the technological evolution and extensibility must be a requirement when evaluating
the existing system [4].
a) b)
Fig. 1. (a) ISA Activity Levels, (b) Interoperability between AT and IT
The ANSI/ISA 95 standard presents an architecture split into five levels of activities in a manufacturing
organization. The control and automation part is represented by levels 1 and 2. Manufacturing operations
management systems (MES, LIMS, WMS) are represented by level 3. Level 4 represents the Enterprise Resource
Planning (ERP) systems. All these representations are shown in Fig. 1(a).
The convergence between AT and IT systems has narrowed the distance between new technological evolutions,
consequently, decreasing the interoperability barriers and allowing an integration between these technologies. Fig.

3. 1839
Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
1(b) shows the space between AT and IT groups where convergence has increased interoperability between the
systems. The proposed evaluation method to measure the potential of interoperability between AT and IT levels
helps in decision-making that aims at the evolution of installed technology.
2.1.1. Industry 4.0: Elements of Evaluation
According to Weyer [5], standardization is the crucial challenge for highly modular production systems with lots
of suppliers. The central aspects of Industry 4.0 can be specified through three models: Smart Product, Smart
Machine and Enhanced Operator. The idea behind Smart Product model is to extend the role of a tool in the work
environment to an active part of the system. Products receive a memory in which the operational data and
requirements are stocked as an individual construction plan. This way, the product demands the necessary resources
and manages the production processes to finalize itself. This is a pre-requisite to allow self-configuration processes
in highly modular production systems.
Inside the vision of these three models as central aspects of Industry 4.0, Weyer provides, in his study, some of
the main elements to construct a technological model organized as Industry 4.0:
i) Plug and produce – Modules that can be easily removed or added during plant operation. Interfaces with
universal standardized connectors for Ethernet, compressed air, on/off power and electrical connections.
ii) Vertical integration – The main technologies to allow the vertical integration inside a factory, with
interaction that is free of problems of different modules beyond the limits imposed by the supplier.
iii) Control architecture – Modulated production lines request decentralized control architectures. Together
with standard PLC control systems, modules that already offer technologies such as SOA (Service Oriented
Architecture) for an efficient, consistent, secure and standardized communication are necessary.
iv) Manual Work Station – The concept follows the Increased Operator model and allows a flexible and
modular integration in automated production lines.
2.1.2. Maturity in Industrial Automation
Industry 4.0 concepts as Internet of Things, Cyber-Physical Systems and Production Based on Cloud face an
undesired obstacle when the necessity of technological evolution comes from an existing automation system. The
proposal of this analysis is to allow the maturity evaluation of the automation components applied to levels 1 and 2
of the ISA 95 model. This way, it will be possible to measure the automation degree of maturity of an installed plant
classifying the existing elements in comparison with the technological elements of Industry 4.0 [6].
The classification of maturity is divided into three levels, comparing the elements pointed as fundamental for
Industry 4.0. Level 3 (higher level) considers the evaluated elements of the actual system as “adapted” to Industry
4.0 requirements. Level 2 (intermediate level) classifies the evaluated elements as “inclined” to Industry 4.0
requirements. Level 1 (lower level) considers the evaluated elements as “isolated”, meaning it does not satisfy the
requirements necessary for Industry 4.0. The requirements necessary for a solution to meet the concept of Industry
4.0 considered in Level 3 were defined using the current academic literature about Industry 4.0. Levels 1 and 2 of
maturity were defined using the academic literature about automation systems plus the opinion of experienced
specialists of industrial automation. Table 1 presents the classification levels and the requirements necessary for the
proposed evaluation.
2.2. Interoperability with a View to Connected Industry
Interoperability is the capacity of a system or product to work with other systems or products without any special
effort produced by the customer [7]. Interoperability between systems can be measured or evaluated through its
potential, compatibility and performance. This paper approaches interoperability potential of a system to identify the
capacity of the AT level (Levels 1 and 2) to communicate in an interoperable manner with the IT level (Levels 3 and
4) of the ISA 95 model.

4. 1840 Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
The criteria for evaluation are defined according to the evaluation of the potential of the interoperability
architecture of the systems of the AT level to ensure efficient integration with the IT level. The concepts of Industry
4.0 pointed to an analysis of interoperability in the following areas:
i) Infrastructure – Define a single standard for networks and communication protocols between the
interoperable systems facilitating the communication between different suppliers for the same solution.
ii) Architecture standard – An architecture that meets the international standards, with the application of
open protocols and easy of access to decrease the complexity and barriers in the integration of this
architecture.
iii) Software platform – Flexible software platforms with easy remote access and availability of access by
Web Services.
iv) Technology update – Define the potential of future integration with other systems. Software updates
facilitated and hardware components exchange occurs in a modular way.
Table 1. Levels of automation maturity and requirements.
# $ %
%!$,!)!
!!$%$!$)!
%+% %/$%!%!!'
!!$%$''!'%$+/
!$,!!!$%$!$
)!%+% %!(%/

5. '$
%+% %)%+ !!!/!$,
#$ '!$'!($%/

6. '%!%+$ ($
'$!#$!#!/ !$%)
'!($%!!$%%!$,$$
!!!%.$!. #$%%$!
#)$!0/!($%%!$%+% /

7. '!!)!(%
)$! '!!
#$%$!)$
'!!/ %%+% %/
'!!)!(%$'
$!)'##%%%/
'!!#$/
'!!$'$! (9)
#$/$($!+!
$ $' !+/
%)%!%
)' '!!)$
%+% %/!(+!)$
'!!$*$!%%/
$'$)%!!$%+% %
#$!!!$!)
#!%+% %/%%+!%!
2$($!$'$3 '%$
!$!)!%+% %/
!.$!%!$,
'!!$'-% '%/
%%+% (

8. $'!
%!%$/$%%#)$%!$,%1
$!,!/
!+#$%%)/
%%#$'!$ !!
#$%%#$!!$ !/
+% )$$$%%%
#$'!/
$+#$'!)+%!%!/
!$!!$+
!$! !$!%+% %/
,! %+% %$$+
(%!#$'/#$$%'##$
+($'!%$'!%!!#!$'
$% $%%%/
The evaluation of the interoperability potential is proposed in a division of three levels of maturity. This
evaluation considers Industry 4.0 requirements for interoperability of systems. Level 3 (higher level) considers the
integration between systems of the current platform as “adapted” to Industry 4.0 requirements. Level 2 (intermediate
level) classifies the integration between systems of the current platform as “inclined” to the Industry 4.0
requirements. Level 1 (lower level) considers the integration between systems of the current platform as “isolated”,
in other words, it does not allow interoperability between AT and IT. Level 3 represents the requirements that are
necessary for a solution to meet the concept of Industry 4.0 and to make the system interoperable. This level was
defined on the basis of the scientific literature about automation systems plus opinions of experienced specialists in
industrial automation. Table 2 presents the classification levels and the requirements for evaluation of the
interoperability between AT and IT levels.
3. Application methods
3.1. Definition of the method
For support in decision-making where many elements are to be analyzed, Multi-Criteria Decision Making
(MCDM) methods can be used as a complementary strategic alternative. As the Multi-Criteria Decision Making

9. 1841
Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
methods are diverse and vary among themselves in the areas of application, it is important to evaluate the best
method for each situation [2].
The AHP method was firstly developed by Thomas L. Saaty in the 1970s [2]. It is considered the most used
method among scientists and professionals of the management area. To apply AHP, one must first determine the
objective to be reached, then, identify the available alternatives to solve the problem, the criteria and sub criteria that
will assist in the choice of the best alternative, determine the weights of each criterion regarding the ideal solution,
compare the criteria or sub criteria pair to pair following the pre-defined scale from 1 to 9 provided by Saaty and,
finally, calculate the Consistency Index for the comparisons, making sure that the Index has a value lower than 10%
[8]. This way, in the end, the alternative that receives the highest weight will be considered the best alternative to
solve the problem under analysis.
Table 2. Levels of interoperability potential between AT and IT levels.
# $ %
!$%$''$$
'!!)!%+% %/
'%!!)$%$
'!!)'%
$+/
!$,!%!%$
!$!!!)!(%!
% ($'$/
!)!%!$%$!)$!
)#%%+ '!!
)!(%$!
%'##$%'%$+/
!%!$$ '!!
)! #!!%
%+% /+%%$''$!
!'!)$%)!!
'!!/*%!
'!!)%'#$$
(%$! ;8
$'$/
)$$'$)'
!%!$%/
'!!#$%/
!1
!$!)$#+/
%%+% %/
!!)$$'$/
!+!$!!
' !$'$/*%!
'!!#$%/
$'$$!
!$!!%!$%2.
96676. ;8. ::3/#!
'!!#$%/
! )' !/%!
%/%

10. %/$$$%
'!!)!%+% %/
+% % !$!!$!
' !$'$/%
)#%%+ '!!
)%'#$$(%$'
%!$ '!!#$%/
*#$ )!$
'!!/ !$%+%
!)#%%+
$(%%/
''#*%!%)$
!$)$/%!
!%/
%%+'#*%!
%)$!$)$!%
'$'$)
!$!)!%+% %/
*%%%+% %'#/
!$''$!$!)
*$!(%$' !$!
!%2 3/#!!%)
#%%+ +/
The evaluation of automation and interoperability maturity of systems in the automation technology environment
will be carried through the application of the AHP method. The AHP method was chosen to evaluate the defined
criteria, since its evaluation technique allows a comparison of pairs between the elements analysed. In this first step,
this comparison is understood as the most appropriate option to analyse the existing components, providing a more
direct view of the elements in comparison with each other.
3.2. Evaluation Criteria
The ISA 95 standard and the concepts of Industry 4.0 were used in the definition of the evaluation criteria. Levels
1 and 2 of the ISA 95 model that correspond to Automation Technology (AT) were used in the evaluation of the
degree of automation maturity. The result of this evaluation of automation maturity collaborates with a second
evaluation regarding interoperability between AT and IT levels. Industry 4.0 puts face to face the consistent ISA 95
standard with the growing necessity of an industry that is increasingly integrated and flexible, allowing easy access
to information in all levels with open protocols and independent systems.
The matrices were generated with the organization of the group of evaluation criteria and sub criteria (4 criteria)
and alternatives (3 for each goal) to support the construction of the decision hierarchy. The pairwise comparison is
fulfilled in every item through the matrix inside the same hierarchical level of the AHP method. The value of global
priority for each criterion in the hierarchy was calculated according to the method presented by Saaty (1997).

11. 1842 Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
According to Saaty's method, the values are applied in a scale of values between 1 and 9, where the value 1 is
considered for items of equal importance. As higher values are applied, they show within the scale the degree of
importance of one item relative to another compared. These values were then used to decide the final priority of
each alternative. So, a process was performed to verify the consistency of the evaluation of priority and, this way, to
reach a conclusion after the end of these three applied methods.
3.3. Application of AHP model – Horizontal evaluation of maturity in automation
The first evaluation proposed by this paper is the degree of maturity of automation systems in a working plant.
This evaluation has as premise the identification of which degree the automation equipment is adapted or with
possibility of inclination to the technological solutions of Industry 4.0. The first evaluation will be performed in the
same level of sensors and actuators (level 1) and a second evaluation in the level of control and automation systems
(level 2) of architecture ISA 95.
The diagrams of Fig. 2(a) and (b) present the structure of evaluation divided into the levels according to ISA 95
standard. The evaluation criteria were defined according to the concepts of Industry 4.0 and the barriers were
defined according to the maturity stages showed in table 1. The first level in each structure shows the objective of
evaluation, it means that this level classifies in which state the criteria are inside each cluster (level) to evaluate the
degree of maturity in automation between the systems of a working automation platform. The second and third level
represent the perspectives (criteria) and barriers (attributes) related to the subject of this evaluation. The evaluation
of maturity of the automation devices and systems in a horizontal way compares the areas of Plug and Produce,
Vertical Integration, Control Architecture and Manual Work Station allowing to have a clearer vision of availability
of components inside the automation level to interact with higher levels.
a) b)
Fig. 2. AHP model: maturity evaluation - Level 1 (a) and Level 2 (b)
The results supplied by this evaluation have the intention of evaluating the actual degree of maturity of a plant in
order to identify the flexibility of this architecture in meeting the requirements of Industry 4.0. The evaluation will
contribute as a support to decision-making, orienting future investments that aim the technological improvement in
AT levels.
3.4. Application of the AHP model – Vertical evaluation for interoperability potential
The interoperability between the AT and IT levels inside the industrial environment raised interest with the
arrival of the concepts of Industry 4.0. The evaluation of automation maturity inside the levels 1 and 2 of ISA 95
model proposed on the last section becomes a significant support for the evaluation of the interoperability potential
of the AT level for integration with the IT level for presenting the barriers and technological aspects of equipment in
inclination to the interoperability potential.
The vertical evaluation analyses the interoperability potential of the AT level that is responsible for the control
and automation of the process for integration with the IT level responsible for the monitoring of operational data and
business processes. An example of vertical integration is Cyber-Physical Systems (CPS) that integrates the
dynamics of physical processes with network software. Computer systems and intelligent networks monitor physical
processes in an integrated way in a loop that feeds them with real-time information again. Innovative abstraction
) )

12. 1843
Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
approaches and architectures that enable seamless integration of control, communication and computation for the
rapid design and deployment of CPS [4] should be developed. Fig. 3 shows an example of interoperability between
systems oriented to Industry 4.0.
Fig. 3. Example of interoperability in Industry 4.0
The evaluation criteria (concerns) were defined according to the concepts of Industry 4.0 and the barriers were
defined according to the stage of interoperability shown in Table 2. The first level in this structure presents the
objective of evaluation. This means that this level classifies in which state the criteria are inside each cluster (level)
to evaluate the interoperability potential between the AT and IT levels of a working architecture. The second and the
third level represent the perspectives (criteria) and barriers (attributes) related to the subject of this evaluation. The
choice of criteria has its basis on concepts of Industry 4.0 with orientation to the areas related to the concept of
interoperability between systems.
Fig. 4 presents the evaluation structure of the AHP method divided into the levels oriented through the ISA 95
architecture for interoperability of AT systems and IT industrial and corporative systems. The standardization of the
model for the criteria Infrastructure, Architecture Standard, Software Platform and Technology Update in an
automation platform is because these criteria advance the efficiency of communication with the higher IT levels.
Fig. 4. AHP model: Assessment of interoperability potential
The results supplied by this evaluation have the intention of contributing with the targeting of investments in the
automation environment of an existing platform. Evaluating the degree of interoperability allows the identification
of existing potential points for integration between the systems and the detection of deficiencies (barriers) to be
overcome. This evaluation will contribute for the development of a strategic plan for the evolution of factory
technology in order to plan interoperability according to the concepts of the connected industry. The development of
this investment plan does not make part of the area of analysis of this paper.
4. Results and analysis
The evaluation script was constructed with the software Super Decision, used to apply the AHP method. The
vertical evaluation of potential was elaborated for application in the area of AT in the industrial environment. This
way, this study is not supported in any specific section of process or product. The use of evaluation methods was
applied to evaluators with professional experience and specialist profiles of the industrial automation area. The

13. 1844 Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
evaluation criteria and the construction of the evaluation structure were organized according to the answers collected
in the interview phase. Fig. 5 below shows the result of this evaluation extracting the potential degree in each
criterion of interoperability according to Industry 4.0 requirements.
Fig. 5. Results of the evaluation of interoperability potential
Fig. 5. Result of potential degree of interoperability for each criteria.
The evaluation shows the limitations of the architecture of a running plant through the analysis of the criteria
raised in the previous steps considered necessary to support the evolution to Industry 4.0. Looking at the graphs you
can see the main barriers of a plant to evolve into a smart factory. The interoperability potential is measured by
comparing the current plant stage between the Infrastructure, Architecture Standard, Software Platform and
Technology Update criteria. The obtained results show the potential of the four criteria to meet Industry 4.0
requirements in the current automation systems architecture. Although the criterion Software Platform is in level 2
(intermediate), it is closer to level 3 (adapted). On the other hand, the criterion Infrastructure is more distant from
level 3 (adapted). Therefore, it is possible to conclude that the starting point for this architecture will be the
investment in Infrastructure to increase the potential level of interoperability of this solution.
5. Conclusions
Interoperability between several systems and devices that compose an automation architecture is the fundamental
link for the evolution of the AT environment with respect to intelligent industry.
To support and offer more reliability in the process of evaluation of an automation plant, the AHP method was
used in the construction of this application. The use of this MCDM method has as an important factor the integration
of the ISA 95 architecture and the concepts of Industry 4.0 for evaluation of a given automation system. The
application of the AHP method was realized in two phases. On the first phase for evaluation of maturity in
automation inside Levels 1 and 2 of AT in the ISA 95 model. On the second phase for evaluation of the
interoperability potential of AT levels with the industrial IT levels. Both evaluations are linked, as orienting the
evolution of maturity in automation has direct influence with the increase of interoperability potential between
systems. Forecasting an integrated architecture between all the levels of a plant considering the requirements that
must be met within an intelligent industry context is the main objective in an evaluation of an industrial automation
plant. This was the explored objective in this paper. Future research can be conducted for classification of the
requirements of Industry 4.0 criteria using other Multi-Criteria Decision Making methods.
References
[1] Leitão, Paulo, Armando Walter Colombo, and Stamatis Karnouskos. Industrial automation based on cyber-physical systems technologies:
Prototype implementations and challenges. Computers in Industry 81 (2016): 11-25.
[2] Hendriks, Margriet MWB, Jan H. de Boer, Age K. Smilde, and Durk A. Doornbos. Multicriteria decision making. Chemometrics and
Intelligent Laboratory Systems 16, no. 3 (1992): 175-191.
[3] Saaty, Thomas L. How to make a decision: the analytic hierarchy process. European journal of operational research 48, no. 1 (1990): 9-26.
[4] Baheti, Radhakisan, and Helen Gill. Cyber-physical systems. The impact of control technology 12 (2011): 161-166.
[5] Weyer, Stephan, Mathias Schmitt, Moritz Ohmer, and Dominic Gorecky. Towards Industry 4.0-Standardization as the crucial challenge for
highly modular, multi-vendor production systems. IFAC-PapersOnLine 48, no. 3 (2015): 579-584.
[6] Schumacher, Andreas, Selim Erol, and Wilfried Sihn. A Maturity Model for Assessing Industry 4.0 Readiness and Maturity of
Manufacturing Enterprises. Procedia CIRP 52 (2016): 161-166.
Fig. 5. Results of the evaluatio
o
o
o
on
n
n
n
n of interoperability potential

14. 1845
Maicon Saturno et al. / Procedia Manufacturing 11 (2017) 1837 – 1845
[7] Chen, David, Guy Doumeingts, and François Vernadat. Architectures for enterprise integration and interoperability: Past, present and
future. Computers in industry 59, no. 7 (2008): 647-659.
[8] Liao, Yongxin, Eduardo Rocha Loures, Osiris Junior Canciglieri, and Hervé Panetto. A novel approach for ontological representation of
analytic hierarchy process. Advanced Materials Research 945 (2014): 675-682.