.

1. XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Software Testing Automation: A Comparative Study
on Productivity Rate of Open Source Automated
Software Testing Tools For Smart Manufacturing
Ashok Sivaji, Rosnisa Abdul Razak, Nur Faezah Mohamad, Nurshakirin Sazali, Afiqah Musa, Norzam Mohd Bajuri, Fazil
Zainal Abidin, Aslinda Md Hashim, Mohd Solehuddin Abdullah, Nur Diyana Joha, Nadia Ellyani Azis, Anjana Devi N
Kuppusamy, Azlan Deniel, Ngip Khean Chuan, ,
MIMOS Technology Solutions Sdn. Bhd., Technology Park Malaysia, Kuala Lumpur, Malaysia
ashok.sivaji@mimos.my
School
Abstract— With the growth in emerging technologies,
manual software testing possesses a challenge to project team in
terms of test planning, test case creation, execution and
reporting. Software testing automation with open source tools
promises productivity savings for software test engineers. The
objective of this study is to compare the level of productivity of
open source Smart Manufacturing toolkits for web application
testing. Together with literature analysis, subject matter experts
in software testing were interviewed and were requested to
experiment and compare the productivity rate of two tools in use
for the various steps in the software development lifecycle.
Katalon Studio 7.0 was found to be a more smart and productive
tool than its predecessor, Robot Framework 3.0 in terms of
installation, scripting, verification and maintenance for web
application testing. By employing a Wilcoxon Matched Pairs
Signed-Rank Test, results revealed that the null hypothesis of "
no statistically significant differences between SME
productivity rate in using RF and KS to performing software
test automation in terms of time taken” can be rejected , Z= -
6.399, p < 0.05. Future work could replicate similar studies for
selecting mobile manufacturing toolkits that will be suitable and
improve productivity rate.
Keywords—Software Testing Automation, Smart Tools,
Productivity Rate, Software Test Engineers, Open Source Tools.
I. INTRODUCTION
Smart manufacturing (SM) toolkits have the potential to
monitor, predict and alert machine health to ensure smooth
manufacturing operations [1-3]. This is particularly true for
SM toolkits of advanced statistics process control systems
with capabilities in data science such as automated gauge
repeatability and reproducibility and measurement
uncertainties[4]. In order to eliminate waste and ensure a
smart manufacturing process with these SM toolkits, they
needs to be verified and validated using software testing
automation tools. Thus, software testing automation is a key
part of any SM toolkit. SM toolkits automation encompasses
design and simulation of embedded software/firmware for (a)
new production introduction (NPI), (b) robotic and automatic
tools that perform repetitive work via shop floor control
systems hosted on smart cloud manufacturing framework, and
(c) automated statistical process control and monitoring
systems [1-4]. Software test automation has three (3) benefits,
namely enabling resource and effort estimation [5], providing
an opportunity to test engineers to upskill from manual testing
to automation testing, and reducing lengthy test execution and
reporting time [6] that will benefit smart manufacturing.
In this study, the use of open source software testing
automation tools is explored in terms of comparing the
productivity among software testing engineers using the tools
[7]. Open source tools refers to software for which the source
code is made available for the users to access and customize
while for proprietary tools the source codes are copyrighted
and owned by the organization [8]. The comparison of tools
can be used for future software test engineers to select suitable
tools to improve the productivity rate in smart manufacturing
[1, 7].
II. REVIEW OF LITERATURE
The literature review comprise of the benefits of software
test automation, open source test automation tools, and a
comparison of tools based on survey from software testing
professionals.
A. Benefits of Software Test Automation
In this section, the various benefits of software test
automation is reviewed. With the adoption of Agile process,
most organization strive to use automation testing with the aim
to ensure that consistent and streamlined methods are applied
by the software testing teams. Automation software test tools
also ‘promises’ higher productivity, as stable test cases [5] can
be regressed and retested via a ‘record and play’ mode in the
tools [9].
There are three (3) main challenges to productivity that
software testing automation solutions will help overcome.
Firstly, software testing automation can improve the current
inaccurate estimation of resource and effort estimation
especially for new testing domains such as smart
manufacturing, artificial intelligence (AI), block chain, data
analytics and internet of things. Secondly, software testing
automation can reduce the lack of skilled test engineers in
white and grey box testing. Thirdly, which is highly related to
this paper, software testing automation can minimize time on
the currently lengthy test planning, test case creation, test
execution and reporting due to the overreliance on manual
testing that inhibit productivity [6-7].

2. Compared to the use of standalone word processors and
spreadsheets for preparing test plans, test cases and reports,
the use of a software test automation toolkit [6, 9] ensures real
time traceability and updates between test assets between test
engineers, developers, manufacturing operators and
customers.
B. Open Source Software Test Automation Tools for Smart
Manufacturing
Open source software is software developed with the aim
to promote collaboration among the community. It can be
used, copied, studied, changed and openly shared with
minimal concerns on copyright by the owner [8]. SM
automated software testing tools can be open source test tools
and proprietary test tools [8]. Many open source tools are
inexpensive and collaborative to use. They enable sharing of
software libraries between various manufacturing lines in
large factories to streamline operations and processes. By
using software libraries hosted in the cloud, test engineers are
able to select and customize further libraries that are suitable
for their production lines [10]. This speeds up cycle time
software development and testing to ensure quality
improvement.
C. Selenium and Appium Automation Framework
A 2016 survey conducted with Finnish software
professionals found that they as a group used 133 different
software test automation tools. The researchers shortlisted the
tools based on ISO/IEC 25010 system and software
characteristics such as usability, functional suitability and
maintainability [11-12]. Table I, column 4 shows the ranking
of tools based on their usage frequency (%). The tool with the
highest usage percentage was Robot Framework (RF),
followed by Selenium and Jenkins. These belonged to the
open source category. This was the reason why this research
compared two open source tools. A survey conducted in 2018
showed that RF was the highest evaluated tool among
software professionals followed by Jenkins. Appium gained
popularity in the 2018 survey compared to the 2016 survey. It
overtook Selenium in the ranks of tools, probably mostly due
to its suitability in supporting mobile software test automation.
Mobile software test automation is however beyond the scope
of this study. Selenium is an automated software testing tool
that supports different types of testing in web applications.
Appium is automated software testing tool for mobile
platforms like such as Android, iOS, and Windows.
TABLE I. RANKING OF USAGE FREQUENCY BY SOFTWARE TESTING
PROFESSIONALS [11,13]
No. Tool
Vendor
Usage
Frequency
(%) 2016
Survey
[11]
Top 6 Most
Evaluated
Tools in 2018
Survey
(Rank 1-
Highest to
Rank 6
Lowest) [13]
1
Robot
Framework
Open Source 69 1
2 Selenium Open Source 47 4
3 Jenkins Open Source 26 2
4 UFT & QTP
Hewlett
Packard
19 N/A
No. Tool
Vendor
Usage
Frequency
(%) 2016
Survey
[11]
Top 6 Most
Evaluated
Tools in 2018
Survey
(Rank 1-
Highest to
Rank 6
Lowest) [13]
5 Soap UI Open Source 19 N/A
6 JIRA Atlassian 17 6
7 jMeter Open Source 14 5
8 Appium Open Source 9 3
D. Robot Framework
Being a generic open source automation framework,
Robot Framework 3.0 (RF) is compatible across various
operating systems such as Microsoft Window (PC), MAC OS
Fig. 1. Virtual Oscilloscope (ISUT) [14]
and Linux [11, 13]. For SM, tools like RF is useful to test
virtual engineering equipment (instrumentation software
under test (ISUT)) that have user interface and are PC based.
One such example is the virtual oscilloscope [14] software
system that is based on ISO/IEC 25010 and instrumentation
standards [12]. Refer to Figure 1.
E. Katalon Studio
Integrated, flexible, and high-in-usability automated
software testing tools such as Katalon Studio (KS) [9] have
emerged. It is built on top of Selenium and Appium
framework and can be used for API, mobile and web services
automation [11]. Katalon Studio provides a flexibility in terms
of dual interchangeable interface for scripting; a Manual
Mode for the less technical users (with Recorder and point and
click GUI) and Script Mode towards experienced user to write
automation test with syntax and intelligent code completion
(with Groovy editor) [9]. KS is also compatible across
Microsoft Windows, MAC OS and Linux and comes with
user-friendly interface as per the usability characteristics in
ISO/IEC 25010 [9, 12]. The use of UI in the form of either a
web application or desktop environment is common in
manufacturing lines. Hence, the research motivation to

3. compare these SM toolkits (RF and KS) for manufacturing is
of high interest [1, 3, 6-7].
F. Research Objective and Scope: Test Automation in
Software Engineering Development Process
Similar to benefits realized from smart manufacturing in
other fields such as in electronic equipment [2] and
semiconductor [4] manufacturing, software test engineers
have started to apply smart techniques, continuous software
engineering and ‘shift left’ thinking by engaging in earlier
development cycles such as in requirement engineering
phases in an iterative manner [7, 15]. Figure 2 shows the
software engineering processes that engineers engage in with
opportunities to employ software testing automation [6, 11,
13]. Based on the problem statement and literature reviews,
the objective of this research are summarized in Table II. The
objective follows previous research on identifying suitable
tools that are smart with the maturity and culture of an
organization [1, 7]. Since both RF and KS are based on
Selenium framework for software testing automation, it will
be of high interest to SM test engineers to compare the
suitability of these tools for SM software engineering process
improvements.
TABLE II. OBJECTIVE AND EXPECTED RESULTS
No. Objective Expected Results
1
To recruit subject matter experts
(SME) in the areas of software
test automation with experience
in Robot Framework 3.0.
Refer to Results Section
A, Demography of SME
in Web Application
Automation.
2
To identify the processes in the
software engineering
development process that are
suitable for automation.
Refer to Figure 2 –
processes that indicate
Selenium automation.
3
To gather data from SME on
their productivity rate (time) in
using Robot Framework 3.0 for
the identified processes in
Objective 1.
Refer to Table III Column
3.
4
To conduct a training for SME
on using Katalon Studio 7.0.
Refer to Methodology
Section B, Training and
Expert Review on
Katalon Studio 7.0.
5
To measure the effectiveness of
the training.
Refer to Results Section
C, Pre-post quiz on
Katalon Studio 7.0.
6
To gather data from SME on
their productivity rate (time) in
using Katalon Studio 7.0 for the
identified processes in Objective
1.
Refer to Table IV
Column 4.
7
To compare the data from
Objective 6 against baseline data
in Objective 3.
Refer to Table IV
Column 5.
8
To test the level of significance
in terms of productivity
improvement.
Refer to Table III and V –
Wilcoxon signed ranked
test
III. METHODOLOGY
A. Expert Recruitment, Interview and Expert Review on
Robot Framework 3.0
The comparison of the software testing automation was
carried out by recruiting ten (10) experts with experience in
software testing. The experts were interviewed on possible
areas of web application test automation with the use of RF,
based on Figure 2. Based on the identified areas suitable for
automation, experts were requested to fill up a data sheet on
the time taken to perform testing activities using RF.
B. Training and Expert Review on Katalon Studio 7.0
An intervention was introduced in the form of a training
for all 10 experts on the use of a new tool, known as Katalon
Studio [9] for purpose of software testing automation. A pre
and post quiz was conducted to measure the level of
knowledge acquired by the experts. The quiz was
administered using Mi-TestLab [10]. After the completion of
the training, the experts were requested to explore the KS tool
and compare various features of the tools against their
previous experience in using RF in a data sheet on the time
taken to perform testing activities.
C. Comparison of Time Taken to Use RF and KS Wilcoxon
signed-rank test
In order to compare and validate the productivity between
RF and KS, a Wilcoxon signed-ranked test is used. The
differences between the means for both sets of scores are
calculated using IBM SPSS Version 26 [15]. As shown in
Table III, the hypothesis statements are presented for this
study.
TABLE III. RESEARCH HYPOTHESIS IN TERMS OF TIME
TAKEN BY SME TO USE RF AND KF FOR CARRYING OUT SOFTWARE TEST
AUTOMATION
Hypothesis Statements
Ho There are no statistically significant differences
between SME productivity rate in using RF and KS
to performing software test automation in terms of
time taken.
Ha There are statistically significant differences
between SME productivity rate in using RF and KS
to performing software test automation in terms of
time taken.
IV. RESULTS
A. Demography of Subject Matter Experts in Web
Application Automation
The comparison of the software testing automation is
carried out by 10 SME who have between 3 to 20 years of
experience. These SME have acquired knowledge and skills
in using the RF in carrying out the software testing projects
for various systems such as in SM [1 - 4, 6, 14 - 15], cloud
manufacturing [7, 10] and ISO/IEC 25010 quality
measurement characteristics [12]. They have acquired
experience in working in a software testing laboratory that is
accredited by MS ISO 17025:2017 [15]. Experts are familiar
with various standard test methods such as:
 ISO/IEC/IEEE 29148:2018 - Systems and software
engineering - Life cycle processes - Requirements
engineering

4.  ISO/IEC 25010:2011 - Systems and software
engineering - Systems and software Quality
Requirements and Evaluation (SQuaRE) - System
and software quality models
 ISO/IEC 25023:2016 - Systems and software
engineering - Systems and software Quality
Requirements and Evaluation (SQuaRE) -
Measurement of system and software product quality
 ISO/IEC/IEEE 29119-1:2013 - Software and systems
engineering — Software testing - Part 1: Concepts
and definitions
 ISO/IEC/IEEE 29119-2:2013 - Software and systems
engineering — Software testing - Part 2: Test
processes
 ISO/IEC/IEEE 29119-3:2013 - Software and systems
engineering — Software testing - Part 3: Test
documentation
 ISO/IEC/IEEE 29119-4:2015 - Software and systems
engineering — Software testing - Part 4: Test
techniques
 Capability Maturity Model Integration (CMMi) Level
5
B. Identified Processes for Web Application Automation
Based on the interview, the SME identified 7 out of the 10
sub-process that were suitable to be studied for web
application automation. These areas are marked with ‘Se√’ in
Figure 2 and comprise of :
 Step 4: Installation
 Step 5:Scripting
 Step 6:Verification
 Step 7:Execution
 Step 8:Reporting
 Step 9: Script Maintenance
 Step 10:Regression Testing
C. Pre-post quiz on Katalon Studio 7.0
In order to gauge the effectiveness of training conducted,
a quiz was conducted before and after the training. The
average results for the pre-training was 70%. After the
training, the knowledge level among SME increased and the
average score was 91%.
D. Productivity Rate Comparison between Robot
Framework 3.0 and Katalon Studio 7.0
The result of the comparison between experts are
summarized in Table III. Since KS is a newer tool introduced
to the market with a more usability characteristics [16, 17], it
is hypothesized that there would be a time savings in terms of
automation rates for all the steps when compared to the Robot
Framework. Time savings in hours is calculated by averaging
the time taken by all ten (10) experts in using the following
equation:
Fig. 2. Test automation in software engineering process for SM
Time Savings = Time take using Robot Framework 3.0 – Time
taken using Katalon Studio 7.0
(1)
TABLE IV. AVERAGE TIME TAKEN BY TEST ENGINEER FOR ROBOT
FRAMEWORK, KATALON STUDIO AND TIME SAVINGS REALIZED
No. Step
Robot
Framework
3.0
(Hours)
Katalon
Studio
7.0
(Hours)
Time
Savings
(Hours)
1 4 - Installation 5.25 0.44 4.81
2 5- Scripting 0.44 0.23 0.21
3 6-Verification 0.29 0.13 0.16
4 7-Execution 0.18 0.11 0.07
5 8-Reporting 0.04 0.03 0.01
6 9-Maintenance 0.38 0.22 0.16
7 10-Regression 0.24 0.17 0.07

5. Table V shows the mean time taken by SMEs to use RF
and KS tool for smart manufacturing. Since the normality
assumptions were violated, hence the equivalent non-
parametric Wilcoxon Matched Pair test was employed [15,
18]. Out of the 70 test activities undertaken by SMEs (N=70),
54 activities took lesser time when KS was used as compared
to RF. Moreover, there were no time differences between 16
activities. By using the newer tool, i.e. KS, there was a
improvement from 0.9740 hours to 0.1889 hours by SMEs in
carrying out testing activities for web based smart
manufacturing.
The output indicates that there is significant difference
(improvement) between SMEs productivity rate in terms of
using KS as compared to RF, , Z = -6.399, p < .05, with higher
score after attending the KS training and performing testing
activities using the KS.
TABLE V. WILCOXON SIGNRED RANKED TEST FOR
COMPARISON OF PRODUCTIVITY AMONG RF AND KS [15]
Descriptive Statistics
Tool N
Mean
(Hours)
Std
Deviation
(Hours)
Min.
(Hours)
Max.
(Hours)
RF 70 0.9740 2.52582 0.00 16.00
KS 70 0.1899 0.26002 0.00 2.00
Wilcoxon Signed Ranks Test
KS - RF N
Mean
Rank
Sums of
Ranks
Negative Ranks 54 27.50 1485.00
Positive Ranks 0 0.00 0.00
Ties 16
KS - RF
Test Statistics
Wilcoxon Signed Ranks Test
Z (based on positive
ranks)
-6.399
Asymp. Sig (2-tailed) .000
V. DISCUSSION
In this research, in order to realize the benefits of the open
source test automation tools in SM, the features of the two
tools are compared in terms of test planning, execution,
reporting and regression [6]. The average time savings taken
by experts are highest for installation activity, scripting, and
verification activities, Figure 2 and Table III.
A. Installation Step
The highest amount of time savings which is 4.81 hours is
realized at the installation step. This is because RF requires
various packages, drivers and library to be downloaded from
various web applications and installed individually, which can
be time consuming. After the installation, there are various
other steps such as setting of environment variables and paths.
In contrast, KS provides a hassle-free download and
installation from a ‘ZIP’ file. This is in line with smart
production practices that results in labor productivity and
cycle time reduction [7] for the test engineers. For SM
engineers whose focus is on yield optimization with good
quality, it is possible for them to automate testing without the
need of advanced coding skills which is a skill that is primarily
important for software and system developer (of the virtual
oscilloscope [14]) but of secondary importance to software
test engineer, who has the additional responsibility on
monitoring yield, quality improvements [7] and ensuring
customer satisfaction for the virtual oscilloscope.
These findings are similar to that in previous studies in
semiconductor manufacturing, whereby with the use of
software based SM toolkits to detect bias and variance, the
various components affecting the readiness of the
measurement system in terms of environment, equipment and
operator skills is assured to be stable before mass
manufacturing [4]. While in previous study [4], the SM
toolkits were targeted to produce good quality semiconductors
for the automotive industry as per the ISO/TS 16949 standard,
in this study, KS was found to be suitable to produce quality
software as per ISO/IEC 25010 system and software
characteristics [12]. In the former study, the measurement
characteristics used to determine the readiness of the
measurement system to test semiconductor includes stability,
bias, linearity, repeatability, and reproducibility [4]; while in
this study, the measurement characteristics used to test the
web or desktop based system includes functional suitability,
performance efficiency, compatibility, usability,
maintainability and portability [12].
B. Test Case Scripting, Verification and Maintenance
Experts found that there is a potential of average 0.21
hours of time saving while scripting one test case in KS as
compared to RF. Although the time savings for scripting looks
small at an individual test case level, at a project level (master
suite comprising of many test cases) that could comprise of
about 1,415 over test cases [5], this could scale and result in
297 hours of savings, which is equivalent to 50 days, assuming
a 6 hours of testing hours / day allocated for testing work [5].
RF is a powerful open-source tool, which offers advanced
features and is suitable for more advanced users. However, it
does not have any built-in recorder utility. Test engineers need
to install third-party browser extension such as
RobotRecorder or ChromeRobot, which is limited on Chrome
browser. In order to playback-recorded scripts, user need to
manually download, edit and run in RF integrated
development environment (RIDE). Some of these
disadvantages are resolved in KS. KS makes it possible to
automate web application without the need of coding skills.
KS provides powerful built-in record and playback capability
in major browsers including Chrome, Firefox and Internet
Explorer [9]. It captures user’s actions and generated codes
automatically to create runnable scripts. As the package
bundles browser drivers, it allows users to playback recorded
steps using single script on various browsers [9]. With these
benefits and the time savings of 50 days, KS meets the
requirements influencing the selection of smart tools and
techniques. Hence continuous support form senior
management is important to carry out similar research such as
this study [16]. This will eventually help senior management
to realize the short and long term goals of the SM organization
[7, 16].
Similarly, for test case verification and maintenance, the
time savings realized by using KS is 0.16 hours per test case.
This productivity savings make the selection of KS as a smart
tool that is provides high value to an organization [7]. The time
savings is expected to scale for larger projects [5].

6. C. Reporting
KS provides intuitive reports with friendly and readable
message. The screenshots captured at the time of failure, can
be used to communicate between manufacturing operators,
software test engineers and product developers easily to
identify issues during the manufacturing process [12]. The
generated reports in KS contain all the test steps and test
results, with screenshots and logs appended, which enables
multiple parties to reproduce the issues at various locations
and provide resolutions for smooth manufacturing operations
[7]. Although the time savings from reporting both tools are
almost negligible, the reporting format from KS enables both
technical and non-technical employees to comprehend and
make sound decision.
D. Execution and Regression
The time savings realized from test case execution and
regression by using KS over RF is almost negligible. Both
tools are suitable for execution and regression as per the SM
toolkit [1], that will support the reduction of repetitive work
in a collaborative manner, as the scripts created by the more
experienced and skills test engineers can now be reused by
lower labor and skilled workforce [7]. In other words, the
script created by the test engineers with complex software
scripting skills can be reused by manufacturing operators to
ensure smooth manufacturing operations in a consistent
manner.
E. Limitation
This study is limited to web application testing. Future
studies should include comparing similar tools for mobile
manufacturing toolkit and the potential for collaborative
robots with AI.
VI. CONCLUSION
The study has met all of the 8 objectives that is shown in
Table II. From this study, it can be concluded that the use of
an integrated, flexible, and high-in-usability software testing
automation of Katalon Studio 7.0 realizes a much better
productivity rate in software testing automation than a generic
open source automation framework of Robot Framework 3.0
for Smart Manufacturing. Based on 10 subject matter experts
in the field of software testing for web application, Katalon
Studio 7.0 is the smart tool of choice especially during the
installation, scripting, verification and maintenance steps. The
user friendliness of Katalon Studio 7.0 enables users from
both technical and non-technical background to ensure
smooth manufacturing operations.
Future work could replicate the methods used in this study
for selecting mobile manufacturing toolkits that will be
suitable and improve productivity rate, as shown in Figure 2,
Table IV and V.
ACKNOWLEDGEMENT
The authors would like to thank all subject matter experts
who took the time to experiment both software test automation
tools. This research is fully supported by Cloud Based
Software Application Evaluation Platform program, under the
Eleventh Malaysia plan, funded by Economic Planning Unit,
Ministry of Economic Affairs grant. We extend our
acknowledgement to Mr Faisal Ahmad, Mr Thillai Raj T
Ramanathan and Mr Redzuan Abdullah in providing guidance
in this research.
REFERENCES
[1] S. Mittal, D. Romero, and T. Wuest, “Towards a smart
manufacturing toolkit for SMEs,” IFIP International Conference
on Product Lifecycle Management, 2018, pp. 476–487.
[2] C. K. Koh, J. F. Chin, and S. Kamaruddin, “Modified short-run
statistical process control for test and measurement process,” The
International Journal of Advanced Manufacturing Technology.,
vol. 100, no. 5–8, pp. 1531–1548, 2019
[3] R. Rahmat et al., “Improving electronic document control
approval process through e-certification,” Journal of Advanced
Manufacturing Technology, vol. 13, no. 1, pp. 33–44, 2019.
[4] Shaji, A. (2006, January). Measurement system analysis. In Third
IEEE International Workshop on Electronic Design, Test and
Applications (DELTA'06) (pp. 4-pp). IEEE.
[5] L. Inozemtseva and R. Holmes, “Coverage is not strongly
correlated with test suite effectiveness,” in Proceedings of the
36th International Conference on Software Engineering, 2014,
pp. 435–445.
[6] L. M. Sei, “Automating Test Activities: Test Cases Creation, Test
Execution, and Test Reporting ,” International Journal Computer
and Information Engineering, vol. 9, no. 10, pp. 2213–2216,
2015.
[7] M. S. Yahya, M. Mohammad, B. Omar and B. Sulistyo, “Factors
Influencing Selection of Lean Tools and Techniques in
Malaysian Organisation,” Journal of Advanced Manufacturing
Technology, vol. 13, no. 2 (1), 2019.
[8] Z. Jin, “Open Models: Beyond the Open Source Software
Development,” ACM SIGSOFT Software Engineering Notes,
vol. 43, no. 4, pp. 9–12, 2019.
[9] Z. Ereiz, “Automating Web Application Testing Using Katalon
Studio,” Zb. Rad. Medunarodne naucne Konf. o Digit. Ekon.
DIEC, vol. 2, no. 2, pp. 87–97, 2019.
[10] A. Sivaji et al., “Test in the Cloud Evaluation Framework
Proposal for Independent Online Dispute Resolution System,”
IEEE Conference on Open Systems (ICOS), 2018, pp. 80–85.
[11] P. Raulamo-Jurvanen et al., “Using surveys and web-scraping to
select tools for software testing consultancy,” International
Conference on Product-Focused Software Process Improvement,
2016, pp. 285–300.
[12] A. Sivaji et al., “Measuring Public Value UX based on ISO/IEC
25010 Quality Attributes,” in 3rd International Conference on
User Science and Engineering 2014 (i-USEr 2014), 2014.
[13] P. Raulamo-Jurvanen, S. Hosio, and M. V Mäntylä, “Practitioner
evaluations on software testing tools,” in Proceedings of the
Evaluation and Assessment on Software Engineering, 2019, pp.
57–66.
[14] A. U. Jibia and N. D. Robinson, “A PC-Based Multifunctional
Virtual Oscilloscope,” in 2019 2nd International Conference of
the IEEE Nigeria Computer Chapter (NigeriaComputConf),
2019, pp. 1–9.
[15] A. Sivaji, A. Deniel, A.D. Kuppusamy, et al. "Validation of Early
Testing Method for E-Government Projects by Requirement
Engineering," in 2019 IEEE Conference on Open Systems
(ICOS) (pp. 23-27). IEEE.
[16] Y. Inal, T. Clemmensen, D. Rajanen, N. Iivari, K. Rızvanoğlu
and A. Sivaji, “Positive Developments but Challenges Still
Ahead – A Survey Study on UX Professionals’ Work Practices,”
Journal of Usability Studies, 2020
[17] B. Oliinyk and V. Oleksiuk, “Automation in software testing, can
we automate anything we want", in Proceedings of the 2nd
Student Workshop on Computer Science & Software Engineering
(CS&SE@ SW 2019), Kryvyi Rih, Ukraine (pp. 224-234).
[18] M. M. Shahimin, N. Saad, S. Kaur, A. Sivaji, S. Soo, and N.-K.
Chuan, “Online sunglasses purchasing : Where do people look ?,”
in 3rd Conference on User Science & Engineering 2014 (i-USEr
2014), 2014.