.

1. See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/337950933
Validation of Early Testing Method for E- Government Projects by
Requirement Engineering
Conference Paper · December 2019
CITATIONS
0
READS
22
12 authors, including:
Some of the authors of this publication are also working on these related projects:
Usability Testing for Low Income Earners (B40) User Segment View project
behavioral culture product aesthetic View project
A. Sivaji
Malaysian Institute of Microelectronic Systems
44 PUBLICATIONS 315 CITATIONS
SEE PROFILE
Afiqah Musa
Malaysian Institute of Microelectronic Systems
4 PUBLICATIONS 2 CITATIONS
SEE PROFILE
N.K. Chuan
Malaysian Institute of Microelectronic Systems
16 PUBLICATIONS 81 CITATIONS
SEE PROFILE
All content following this page was uploaded by A. Sivaji on 16 December 2019.
The user has requested enhancement of the downloaded file.

2. Validation of Early Testing Method for E-
Government Projects by Requirement Engineering
Ashok Sivaji, Azlan Deniel , Anjana Devi N Kuppusamy, Aslinda Md Hashim, Fazil Zainal Abidin, Norzamzarini Mohd
Bajuri, Nurshakirin Sazali, Afiqah Musa, Mohd Solehudin Abdullah, Ngip Khean Chuan, Nadia Ellyani Azis
MIMOS Technology Solutions Sdn. Bhd., Technology Park Malaysia, Kuala Lumpur, Malaysia
ashok.sivaji@mimos.my
Abstract— e-Government System and Software projects are costly. The success of projects has high impact to the level of
effectiveness of service delivered by government to citizens. However, poorly managed system and software requirements by project
team could lead to project failures. Therefore, validation of requirements at early stages is important. The objective of this paper is
to validate the Early Requirement Testing Method for e-Government projects. A ‘Wilcoxon Matched Pairs Signed-Rank Test’ was
used, whereby test engineers were subjected to performing Early Requirement Testing Method across six(6) e-Government projects
before and after an intervention has been introduced to the e-Government employees and the project team (Business
Analyst/Requirement Engineer). Results revealed that the null hypothesis of "no statistically significant differences between
employing the method in e-Government projects” can be rejected and therefore conclude that method reported was useful in reducing
the requirement related defects, as the difference is negative (reduction of defects) Z= -2.892, p=0.004.
Keywords—Validation of Requirement, Requirement Engineering, System and Software Projects.
I. INTRODUCTION
The Standish Group report[1] shows that three(3) major success factor of a software project are attributed to user
involvement, executive management support and clear statement of requirements. In addition, it was found that projects with
incomplete requirements are ranked the most frequent reasons provided for cancellation of projects[1]. Studies have shown that
Requirement Engineering (RE) process is crucial for the success of system and software design and development [2]. As part
of the RE process, validation of the requirements plays an important role. Established models such as the Kano Model classifies
customer ( citizen or e-Government employee for e-Goverment system) requirements into five categories namely ‘Must-be
Quality’, ‘One-dimensional Quality’, ‘Attractive Quality’, ‘Indifferent Quality’ and ‘Reverse Quality’ [3].
Also known as the Kano methodology, researchers have used this models to classify and validate requirements as those that
delight customer and those that are basic needs that can impact the level of customer satisfaction [4]. The earlier the requirement
is validated, the earlier the stakeholders are aware of the stability of the requirement and this facilitates dependent activities such
as design, development, and testing (validation). The conventional approach of performing testing after the development is
insufficient, hence this paper attempts to bring forward the validation effort at a requirement level. This is achieved by deriving
and validating a requirement checklist on e-Government projects based on ISO/IEC/IEEE 29148 standards for requirement
engineering [5].
II. LITERATURE REVIEW
A. Requirement Engineering
Requirement Engineering (RE) comprise of requirement elicitation (gathering) and development, documentation of
requirements, validation and verification of requirements and requirement management and planning[6]. It is important to
perform the RE in an iterative manner throughout the software lifecycle with various stakeholders.
Conventionally, requirement engineering is performed at the start of the software system development. This may be suitable
for smaller projects, but for e-Government projects which are large in nature and extend or many years to serve the demands of
the citizens, it is almost impossible to have stable requirements across the long period of time [6], [7]. Hence, an iterative
approach at various cycles are necessary.
There exists many methods to elicit and classify requirements. Indirect methods such as surveys and literature review have
been used to gather qualitative feedback. Based on these feedback, previous researcher have performed pain storming or
statistical analysis to derive requirements that lead to proposing a framework for a system development [4], [8]–[10]. Direct pain
storming approach with stakeholders is more common as stakeholders are given an opportunity to voice their opinion and
challenges faced with the existing system and what new requirements are needed to build the future system. It is also common
for the stakeholders to demonstrate the existing system to the requirement elicitation and software system development team to
access the current state and shortcomings and to propose future improvements to the system[11]–[15].
B. Requirement Engineering in e-Government
The next section describes some of the challenges in requirement engineering found in e-Government.
1) Lack of Best Practices

3. Previous research have found that there are lack of requirement engineering best practices in e-Government projects. This
increases the challenges faced by the project team as the nature of e-Government itself is already complicated and prone to
political and legislative requirements[16].
For instance, the need to continually address citizen centric requirements and satisfaction[11], requirements for legacy system
support and data migration to latest technology[6], requirements for usability such as system error prevention, good navigation,
visual clarity and consistency of user interface[12], [16], security and performance efficiency requirements[17]; studies have
also shown the importance of
employing a series of quality requirements such that
specified in the ISO25010[11], [18].
2) Bureaucratic Procedural Barriers
Another challenge faced by Government IT departments
is the bureaucratic procedural barriers, especially in the
procurement process[16]. This has led to an increase in
investment of Commercial Off-the-Shelf (COTS) products,
which can be costly to maintain over long period of time.
Another disadvantage of COTS products are dependence on
proprietary technology which has poor interoperatbility due
to vendor lock-in strategy. One way to overcome this is for
eGoverment to develop customised or bespoke solutions.
However, studies has shown that Government projects are
highly impacted by scope creep, by decision that are risk
averse and by employees turn over that could change the
project goals[16].
C. Cultural Impact in Requirement Engineering
Cultural issues between requirement engineer and
customer such as resolving conflicts by compromising,
establishing trust, recognition of uncertainty, openness and
honesty, taking ownership and responsibility can positively
help reduce the risk of project failures[19]. In contrast,
negative issues such as hidden agenda, managerial influence,
and technological focused solution dictated by customer
which can directly impact project deliverables were found to
be common and challenging for the Australian requirement
engineering practitioners. Some of these issues could have a
potential impact on the RE activities. Hence, early
requirement validation is important to be carried out at various
cycles of the project[19].
D. Requirements Validation Technique
Studies have found that there are six(6) techniques of
requirement validation namely, requirement preview, review,
prototyping, model-based validation, testing based and
viewpoint oriented validation [20]. The scope of this paper is
on the requirement review based on the application of check-
list based reading technique. Based on a checklist developed
by an organization, a group of Test Engineers (TE) reviewed
the requirement document also known as Customer
Requirement Document (CRD) or System Requirement
Specification (SRS). The checklist comprise of a set of
statements and questions to guide the TE to ensure the
requirements corresponds to the items in the checklist[21].
E. Development of Requirement Validation Checklist
In order to develop a requirement validation checklist, the
TEs studied the ISO/IEC/IEEE 29148 standard [5] and
derived a checklist of characteristics such as Incomplete
Requirement, Documentation Errors, Ambiguous
Requirement to facilitate the requirement validation process.
Hence, one of the objective of this study is to validate the
requirement validation method developed. In order to achieve
this objective, the requirement validation is employed to six
(6) e-Government projects and data collected in the form of
defects. Defects here, refers to when there is an error
committed by the SRS document originator. In these projects,
the document is created by the Business Analyst (BA) team
while eliciting the requirements from e-Government
employees. The result of the validation will shed some light
on areas for improvements on the method.
III. EARLY REQUIREMENT TESTING METHOD
AND HYPOTHESIS
The Early Requirement Testing Method (ERTM) is
developed by MIMOS Technology Solution Sdn. Bhd.
Software Test Lab at Technology Park Malaysia, Kuala
Lumpur, Malaysia. The lab is the first to be accredited by the
Department of Standards Malaysia for under the MS ISO/IEC
17025 (formerly ISO/IEC Guide 25) in the field of Software
Testing since 2013 in Malaysia[22]. In its effort to add value
to the growth in software development in Malaysia, the lab has
continued its journey expanding the scope in the area of
validation of software and system requirement.
A. Early Requirement Testing Method(ERTM)
The ERTM developed comprise of the performing a
review on SRS document produced by the software vendors
based on the requirement validation checklist, followed by
raising defects based on the errors made by the business
analyst, followed by conducting an awareness training session
with the BA team on how the requirements could be improved
based on the defects reported.
B. Hypothesis
In order to validate the effectiveness of the ERTM method,
this paper attempts to test whether there are any differences in
defects encountered before and after performing the ERTM
method. As shown in Table I, the hypothesis statements are
presented. Figure 1 depicts the ERTM method.
Fig. 1. Early Requirement Testing Method (ERTM)
1. Receive SRS from
eGovernment
stakeholder
2. TE to perform
testing using the
requirment checkist
3. TE to raise defect
and catergorise
based on errors
made by the BAs
4. TE to provide
recommendation
awarness training
based on defects
found
5. Repeat Step 2
and 3

4. TABLE I. HYPOTHESIS
Hypothesis Statements
Ho
There are no statistically significant differences
between employing the ERT method in e-Government
projects in terms of mean number of defects found.
Ha
There are statistically significant differences between
employing the ERT method in e-Government projects
in terms of mean number of defects found.
IV. METHODOLOGY
Based on Figure 1, the study was conducted using seventy
(70) test cases on six (6) e-Government projects using the
ERTM method developed. The group of engineers who
performed the testing, also known as the Test Engineers were
trained on the ERTM method. In order to validate the ERTM
method, the number of requirement related defects found after
the first round of test was compared to that found during the
second round of test after a series of intervention activities.
The number of defects reported in the both rounds were
compared using the ‘Wilcoxon Matched Pairs Test’ similar to
previous studies[13], [23]–[25]. During the study, the Test
Engineers (TE) and the Business Analysts (BA) were not
aware that the intention of this study is to compare the
differences between the numbers of defects before and after
the intervention activities. Thus, there is no reason to suspect
that the results are biased by the BA’s and TE’s
expectations[13].
A. Intervention Activities between Round 1 and 2
The stakeholders involved in this studies includes the TEs,
BAs and the end user, which is the government employees.
The intervention introduced by the TEs between the two(2)
rounds of testing includes categorizing the defects found,
providing recommendation and awareness training on the
defects found to the Business Analyst (BA) and government
employees, as per ISO/IEC/IEEE 29148 standard[5]. Based
on the level of improved understanding, the BA may reach a
level of consensus with the government employees to address
the gaps in the requirements related defects. After which, a
second round of ERT was performed to compare the
effectiveness of the ERT method.
B. Tools and Instruments for Validating the ERT Method
The number of defects found from round one (1) and two
(2) from the ERT was systematically categorized and recorded
into IBM SPSS Version 25 for further analysis[24].
V. RESULTS
This section presents the overview of the six (6) projects
and the results of the ‘Wilcoxon Matched Pairs Test’.
A. Overview of e-Government Project
Based on the six (6) e-Government projects that were
subjected to ERT, Table II shows the number of TEs assigned
together with the number modules for each project. The same
group of TEs were involved in the round one (1) and round
two (2) of the testing to ensure the data collection used for
validation is performed in a systematic and unbiased manner.
Due to the qualitative nature of software, the higher number
of modules is not directly related to the actual project size.
After the completion of the ERT for round one (1) (Before
Intervention Activities) and two (2) (After Intervention
Activities), the number of defects were recorded as below.
TABLE II. NUMBER OF TEST ENGINEERS, MODULES AND DEFECTS
REPORTED PER PROJECT ID
Project
ID
No. of TEs
No. of
Modules
No. of
Defects
Before
No. of
Defects
After
1 5 5 451 8
2 4 4 96 11
3 4 5 84 38
4 3 10 26 16
5 3 3 45 1
6 4 8 200 336
B. Validation of ERT Method Across e-Government
Projects
Due to the violations of assumptions for parametric
analysis (data was not normal), the ‘Wilcoxon Matched Pairs
Test’ was employed to test and validate whether there were
significant differences in the defects found using the ERT
method, which also comprise of the intervention. Using IBM
SPSS Statistics Version 25, Table III shows that in overall,
there are significant differences in defects found before and
after the ERT method for across e-Government projects,
Z= -2.892, p=0.004. Based on this, the null hypothesis(Ho) is
rejected, whereby there are ‘no differences between
employing early testing methodologies in e-Government
projects’, while the alternative hypothesis (Ha) is
accepted[25].
A reduction in the defects found between round one (1)
and two (2) shows that the ERT method comprising of Early
Requirement Testing and the intervention activities that
follows has added value and created awareness among the
BAs and the government employees to document the
requirements as per the ISO/IEC/IEEE 29148, Systems and
Software Engineering, Life Cycle Processes,” Requirement
Engineering. Out of seventy (70) test cases for the e-
Government projects, a majority of thirty three (33) test cases
has seen a reduction in defects found from round one (1) and
two (2). As shown in Table II, this result was applicable in
general to project one (1) to five (5).
In contrary to the above findings, eleven (11) test cases for
the e-Government projects had an increase in number of
defects found. This is shown in Table II above, whereby there
was an increase from two hundred (200) defects to three
hundred and thirty six (336) defects. It is important to note that
the capability of finding two hundred (200) and three hundred
and thirty six (336) defects goes on to show that ERT method
is still relevant in finding defects. It is however surprising that
the number of defects has increased from round one (1) and
two (2). Upon investigation with the project team, the
additional defects stemmed from additional requirements
(‘requirement creep’) from the e-Government employees.
TABLE III. DESCRIPTIVE AND TEST STATITICS
Description (N=70) Mean Maximum
Defect Round 2
- Defect Round
1
Defect Round 1 12.89 171

5. Description (N=70) Mean Maximum
Defect Round 2
- Defect Round
1
Defect Round 2 5.86 90
Z (based on positive
ranks)
-2.892
Asymp. Sig (2-tailed) 0.004
Negative Ranks
(Defects Round 2 <
Round 1)
33
Description (N=70) Mean Maximum
Defect Round 2
- Defect Round
1
Positive Ranks
(Defects Round 2 >
Round 1)
11
Ties
(Defects Round 2 =
Round 1)
26
VI. DISCUSSION
A. Validation of Survey Instruments for JKSM
As shown in Table II, the number of defects for Project ID one (1) and two (2) is four hundred and fifty one (451) and
ninety six (96) respectively before the ERT Method. Similar to previous literature, the highest defect category was found to be
Incomplete Requirements[1], [2]. For both of these projects, the requirement engineers had to meet the quality and citizen
centric requirements [11], [18], requirements for legacy system support and data migration to latest technology[6], as supported
in previous literatures[12][16].
In Project ID three (3), TE found that Documentation Error was the highest defect. This was attributed to lack of cultural
understanding of requirement engineers who although were experienced in private sector, they lacked working on large and
complex e-Government projects[16]. It is important to employ and educate requirement engineers to adopt to working on e-
Government projects.
For Project ID one (1), two (2), three (3) and six (6), project team experienced various challenges such as scope creep, by
decision that are risk averse and by employee turn over that eventually change the project goals and impacted the timeline[16].
Surprisingly for Project ID six (6), there was an increase in defects found even after the second round of employing the
ERT method. Upon further interview with the customer and project team, it was found that there were resistance to accepting
to changes by the customer who insisted on a technological focused solution [19]. This shows that similar cultural behavior as
experienced by Australian requirement engineers was also experienced by their Malaysian counterpart[19].
VII. CONCLUSION
In conclusion, from employing the ERD method on the six (6) government projects, it was found that there was a significant
impact in improving the validation of the requirements. This shows that the requirement checklist and methods employed is
useful for e-Government projects. Due to the complex nature of e-Government projects, it can be said that future projects should
employ the ERD method in a more frequent cycles, rather than two (2) cycles currently employed in this study. It was also
observed that the level of cooperation among the project team and e-Government employees improved in terms of establishing
of trust over time, resolution of conflicts by compromising and prioritizing requirements. In terms of communication, in general
it was observed in some of the projects that the level of openness and honesty has improved to reduce the risk of project failures.
ACKNOWLEDGEMENT
This research is fully supported by Cloud Based Software Application Evaluation Platform program, under the Eleventh
Malaysia plan, funded by Economic Planning Unit, Ministry of Economic Affairs. The authors fully acknowledge Ministry of
Economic Affairs and MIMOS Berhad for the approved fund which makes this important research viable and effective. We
extend our acknowledgement to Ms. Noor Jehan Mohd Nizam from MIMOS Berhad in providing project management support
for this research.
REFERENCES
[1] T. Clancy, “The standish group report,” Chaos Rep., 1995.
[2] A. Hussain, E. O. C. Mkpojiogu, and F. M. Kamal, “The role of requirements in the success or failure of software projects,” Int. Rev. Manag.
Mark., vol. 6, no. 7S, pp. 306–311, 2016.
[3] N. Kano, “Attractive quality and must-be quality,” Hinshitsu (Quality, J. Japanese Soc. Qual. Control., vol. 14, pp. 39–48, 1984.
[4] A.-M. Kranzbühler, M. H. P. Kleijnen, and P. W. J. Verlegh, “Outsourcing the pain, keeping the pleasure: effects of outsourced touchpoints in the
customer journey,” J. Acad. Mark. Sci., vol. 47, no. 2, pp. 308–327, 2019.
[5] I. E. C. ISO, “IEEE: ISO/IEC/IEEE 29148, Systems and Software Engineering, Life Cycle Processes,” Requir. Eng., 2011.
[6] D. Pandey, U. Suman, and A. K. Ramani, “An effective requirement engineering process model for software development and requirements
management,” in 2010 International Conference on Advances in Recent Technologies in Communication and Computing, 2010, pp. 287–291.
[7] W. W. Royce, “Managing the development of large software systems. proceedings of IEEE WESCON,” Los Angeles, pp. 328–388, 1970.
[8] A. Sivaji et al., “Test in the Cloud Evaluation Framework Proposal for Independent Online Dispute Resolution System,” in 2018 IEEE Conference
on Open Systems (ICOS), 2018, pp. 80–85.
[9] O. Sohaib, H. Lu, and W. Hussain, “Internet of Things (IoT) in E-commerce: For people with disabilities,” in 2017 12th IEEE Conference on
Industrial Electronics and Applications (ICIEA), 2017, pp. 419–423.

6. [10] A. S. Hashim, W. F. W. Ahmad, and A. Sarlan, “A study on acceptance of mobileschool at secondary schools in Malaysia: Urban vs rural,” in AIP
Conference Proceedings, 2017, vol. 1891, no. 1, p. 20050.
[11] A. Sivaji et al., “Measuring Public Value UX based on ISO/IEC 25010 Quality Attributes,” in 3rd International Conference on User Science and
Engineering 2014 (i-USEr 2014), 2014.
[12] A. Sivaji, A. Abdullah, and A. G. Downe, “Usability testing methodology: Effectiveness of heuristic evaluation in E-government website
development,” in Proceedings - AMS 2011: Asia Modelling Symposium 2011 - 5th Asia International Conference on Mathematical Modelling and
Computer Simulation, IEEEXplore, 2011, pp. 68–72.
[13] A. Sivaji, T. Clemmensen, and S. F. Nielsen, “Comparison of CTA and Textual Feedback in Usability Testing for Malaysian Users,” in The ACM
Conference on Human Factors in Computing Systems. CHI 2015, 2015.
[14] A. Sivaji and S. Soo, “Understanding, Enhancing and Automating HCI Work Practices: Malaysian Case Studies,” Procedia - Soc. Behav. Sci., vol.
97, pp. 656–665, 2013.
[15] I. Hussein, M. Mahmud, T. Mat, and R. A. Razak, “Citizen experience towards humanizing digital government: a case in Malaysia,” in
Proceedings of the 29th Australian Conference on Computer-Human Interaction, 2017, pp. 538–542.
[16] A. Alexandrova, L. Rapanotti, and A. S. Meehan, “Requirements practices in e-government solution development,” in Proceedings of the
International Conference on e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE), 2011, p. 1.
[17] M. D. M. Suffian and F. R. Fahrurazi, “Performance testing: Analyzing differences of response time between performance testing tools,” in 2012
International Conference on Computer & Information Science (ICCIS), 2012, vol. 2, pp. 919–923.
[18] S. ISO/IEC 25010:2011 (E), Ed., Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) —
System and software quality Models, International Standard ISO/IEC 25010. International Standard ISO/IEC 25010, 2011.
[19] M. S. and J. H. Tawfeeq Alsanoosy, “Cultural Influence on the Requirements Engineering Process: Australian Practitioners’ View,” in In
Proceedings of the 28th International Conference on Information Systems Development (ISD 2019), 2019.
[20] S. B. Saqi and S. Ahmed, “Requirements Validation Techniques practiced in industry: Studies of six companies.” 2008.
[21] O. Laitenberger and J.-M. DeBaud, “An encompassing life cycle centric survey of software inspection,” J. Syst. Softw., vol. 50, no. 1, pp. 5–31,
2000.
[22] “Standards Malaysia Recognises MIMOS As The First Malaysian Lab To Receive MS ISO/IEC Accreditation Certificate For Software Testing,”
BERNAMA, 2014. [Online]. Available: http://www.bernama.com/bernama/v7/newsindex.php?id=1011837.
[23] “How to Use SPSS: Wilcoxon Matched Pairs Test,” TheRMUoHP Biostatistics Resource Channel, 2012. [Online]. Available:
https://www.youtube.com/watch?v=OaZO-_wzPpY. [Accessed: 02-Aug-2019].
[24] S. J. Coakes, SPSS version 20.0 for Windows, Analysis without Anguish. 2013.
[25] Q. Xie and A. M. Memon, “Designing and comparing automated test oracles for GUI-based software applications,” ACM Trans. Softw. Eng.
Methodol., vol. 16, no. 1, p. 4, 2007.
View publication stats
View publication stats