Importance of software quality assurance to prevent and reduce software failures: Document Management System In Defence Industry Case Study by Seval Çapraz

1. Importance of software quality assurance to prevent and reduce software failures:
Document Management System In Defence Industry Case Study
Seval Çapraz, capraz.seval@gmail.com
BIL/CMP 781 – Software Quality Management, Fall 2018
Structured Abstract
Context: Document management systems are very beneficial since they speed up processes and reduce paper and
postal costs because every document can be signed and sent via online network securely. Users do not need to print the
documents if there is a digital document management system. On the other hand, online systems may cause some
defects and accidents which can be incredibly expensive and fatal. In this study we inspected a document management
system (DMS) which is written in Java language for defence industry of Turkey. It is used by two customers which have
more than 1000 active users daily. The defects are very important for this system because they may cause danger for
human life because secret documents of military services are the heart of a country. Therefore, passing a certain level of
quality is crucial for this software program.
Purpose: This study aims to discuss the incidents when the lack of software quality management and the proposed
solutions based on the document management systems such as DMS. Software quality seeks to meet functionality,
usability, performance, serviceability, capability, installability, maintainability, and documentation. We researched that if
these properties are provided by the DMS or not. This paper focuses on representing the importance of software quality
assurance and its roles in document management systems to prevent and reduce software failure as well as avoiding
any human damages.
Method: We analyzed DMS as a case study. Firstly, we used defects which are recored from the start of the project until
now. We compared task numbers and defect numbers. We tried to analyze the modules which have more defects and
cause big problems. Secondly, we explore research in software quality assurance in the case of the DMS and discuss
how lack of proper software quality assurance resulted in incidents.
Results: We recommend Software Quality Assurance (SQA) as a proper solution in various stage of the software
development process especially for document management software such as DMS. After the review of DMS, the
recommended actions and tools are given in results section.
Conclusion: Software quality assurance is crucial for document management systems if it is used for military purposes.
We can prevent defects and accidents before they happen if there is a proper software quality assurance.
1. INTRODUCTION
Computer software products can contain various defects which affect the software quality very much. These
defects may cause not only financial damage but also many life losts especially in defence sector. Therefore
software quality assurance has importance in order to reduce defects and improve the quality.
DMS; is an electronic document management system that is in compliance with internal guidelines and
regulations to produce digital documents which is mainly used in Turkish defence industry. In this study, we
discuss how lack of proper software quality management(SQM) resulted in incidents based on DMS project.
We also recommend SQM as a proper solution in various stage of the software development process
especially for document management systems such as DMS.
DMS is the product of one of the pioneer companies in Turkey which develops many tools and software
solutions for Turkish defence sector. DMS manages all processes for digital documents from creation phase
to disposal phase.
DMS has more than 1000 KLOC and it exists more than for 4 years. In this 4 years, the team successed to
deliver product to two customers. We analyzed bug numbers of the system by comparing them before it is
delivered to customers and after the delivery. Incidents occurred and developers tried to close the defects as
soon as possible. There is no software assuarance system in the DMS project. This paper aims to discuss
the already found incidents and possible incidents and how to overwhelm them by creating a categorical
view perspective.
1

2. 2. BACKGROUND AND RELATED WORKS
2.1. Background
Document management systems are very important for a country and military service. They are very fast and
secure to communicate. If something goes wrong in document management system, it may cause
miscommunication or leak in the system. For example, the secret documents should not be read by non-
authorized people. Permission and authorization failures can not be tolerated in defence industry because
enemies can use any information against the government. A lot of bugs occur in the DMS about permission
and authorization system however they can be avoided before the disasters by using proper software quality
management.
DMS is discussed in detail below. DMS has 2 main modules: document management module and archive
module.
Document Management Module. This module allows to create documents first. Then it runs processes one
by one to complete all e-signatures of signers. After the all signatures are done, there are postal and
approval pages in other to do later works of documents. Document management module is:
1. In compliance with TS13298 (Turkish Standards Institution – 13298),
2. In compliance with the e-signature Regulation,
3. In compliance with the Regulation on Procedures and Principles,
4. Supports power of act attorney procedures,
5. Supports E-mail integration,
6. Has ability to keep different metadatas for each document type,
7. Has ability to view the differences in versions of the documents,
8. Supports dynamic workflow process.
Archive Module.This module allows the documents that expired their life time to be archived in the system.
In addition to that, it allows physical archive by letting the users enter the metadatas for physical documents.
Benefits of DMS:
1. Is web-based,
2. Is easy to configure,
3. Supports electronic and mobile signature,
4. Supports integration with different systems to enable digital document transfer with REST/WEB
services,
5. Contribution to knowledge management,
6. Contribution to corporate memory accumulation,
7. Saving work time and paper,
8. Compliance with standards and guidelines,
9. Supports authority management, e-signature, encryption and logging management,
10. Quick and easy access to documents,
11. Standardization of processes and institutionalization,
12. Mobile signature,
13. Tracking changes by document versioning,
14. Tracking approval processes.
2

3. To understand the how big this system is we can use KLOC as a metric. KLOC (thousands of lines of code)
is a traditional measure of how large a computer program is or how long or how many people it will take to
write it. The DMS project includes 1000 KLOC. Productivity for the organization is 1 KLOC per person per
month since this project is 1040 person month. The archive module is 30.4 KLOC and document
management module 969.7 KLOC.
Modules of DMS KLOC (thousands of lines of code) Person Month
Document Management Module ~969.7 KLOC 832
Archive Module ~30.4 KLOC 208
TOTAL ~1000 KLOC 1040
Table 1. Line of codes and person month information of modules
2.2. Related Work
In 1999, first patent about document management systems are taken by [1] as computer-based document
management system. After that there are a few different patterns about this topic emerged like “Internet
document management system and methods” [2] in 2003.
Earlier of the patents, the system design is done in 1992 in the study of a multiple presentation document
management system [3]. First patent is taken as document management and production system in later year,
1993 [4].
In literature, there are no study which mentions the problems and solutions of document management
systems in software quality perspective. In 1995, there is a research study [3] which purposes are to show
the value of new technology for managing documents, to illustrate the variety of ways this value can be
realized, to develop some structure for understanding this rapidly evolving field, and to suggest some actions
I/S managers can take now to prepare for this revolution in information management.
ISO/TR 14105:2011 provides a framework for understanding the basic issues and concepts of organizational
and human factors associated with implementing EDMS technologies [4]. It describes the principles of
human factors and ergonomics in their application to usability criteria for the planning and implementation of
EDMS technologies, to environmental and implementation issues, and to training for long-term productivity
benefits. [4] discusses change management for successful electronic document management system
(EDMS) implementation.
[5] proposes achievements and opportunities for information resources & document management systems in
1994. The problem is still the same, however, the conjuncture is changed a lot. There is mobile phones and
touch screens now. The printing technologies are also changed a lot.
In 1995, Electronic Document Management (EDM) [6], [7], [8] and [9] are proposed for document flows and
management. It promises major productivity and performance. However this study is not about software
quality, too.
In recent years, the software management issues are included in research papers. In 2011, [10] is published
and the auther discusses change management for successful electronic document management system
(EDMS) implementation in the study. The mobility of document management systems is disscussed in 2013
[11].
The most similar study is done in 2017 by D. Baek, et al. That is “Defect Management Method for Content-
based Document Artifact Test in Software R&D Project,” [12]. In order to improve the quality of software R&D
project, studies on the content-based document artifact test (CbDAT), which considers the heterogeneity of
the research phase compared to development phase, have been introduced in study [12]. The CbDAT
manages the research phase by testing the document artifacts which could not be performed by the
conventional software test. However, the existing studies on the CbDAT have a limitation in managing
because they only consider test planning and test execution. To solve this problem, the authors analyze the
characteristics of the CbDAT and derive six major differences. Based on these, they proposed activities that
3

4. researchers should perform for the defect management. In addition, they presented an incident report and
defect severity reflecting the characteristics of the CbDAT. The defect management method proposed in that
paper enables the researcher to manage defects identified from the CbDAT and to improve the quality of
software R&D project.
This case study can be done on different domain with different systems. There are this kind of case studies
in different industries for different software products like Therac-25 medical device [13].
3. RESEARCH DESIGN
This research is done in order to give details and descriptions of incidents and the proposed solutions when
the lack of software quality management when there is a bug in the document management system. To
implement this aim, we choose research questions and tried to answer them.
3.1. Research Questions (RQs)
The research questions are:
1. What are the incidents and solutions of the main cause of defects?
2. Which problems arise when the actual practice comes into play?
3. Why software quality management is important for DMS?
4. How can we decrease bugs?
3.2. Research Method
The research method is a case study that we will follow in answering the research questions that we claimed
above.
4. RESEARCH IMPLEMENTATION
Description of your research implementation.
4.1. Process of Implementation
The software development is done by iterative development. The sprints are done in one week. Approximtely
23 people work in the project. 12 of them are software developers. 4 of them are looking after deployment
while other 4 for team lead. 3 of them are testers.
The main software quality problems of this project are:
• There are no deployment automation tool.
• The product delivery is done when the system is ready, not based on a calendar. So it always
delays.
• Bugs are not analyzed and categorized before. We do not now the root cause of bugs.
• There are no review meetings with customers.
• There is a tool (TFS) to track the bugs and tasks.
• The product tests are done by testers. There is no test automation tool.
• The products are not measured for quality.
This project was started more than 4 years ago in order to change the existing document management
system for the armed forces of Turkey. 62 developers have involved since project started on September 19,
2014. However they joined to the project and left from it after a while. Therefore the team has changed a lot.
4

5. The total count of developers are 20 on average over 52 months (from September 2014 to January 2019).
Therefore the effort is 1040 person month. The start of the project took a few months with meetings, so we
assume that actual development with the team started on January 1, 2015. Therefore we analyzed the whole
4 years between 2015 and 2018. The total count of defects are calculated based on years in Table 2.
Year Number of Tasks Number of Defects Defects Percentage
2015 1277 69 5,4
2016 2338 1462 62,5
2017 3084 2220 71,9
2018 1278 2880 225,3
TOTAL 7977 6631 83,1
Tablo 2. Number of tasks and number of defects based on years
There is a huge increase in number of bugs while number of tasks decreases. The total number of tasks
decreased 58% in 2018 according to 2017. On the other hand, the total number of bugs increased 30% in
2018 according to 2017. In first year of he product, it is designed for one customer (Customer 1) which has
800 users. The product is delivered to Customer 1 in September 2017. So the whole product was developed
in 3 years with 20 developers. After this delivery, the product was ready for adaptation to other customers.
We used the project base code by altering it for new customers after September 2017.
In September 2017, the developers are divided into two groups in order to implement the tasks and bugs of a
new customer (Customer 2) which has more than 40.000 users. The first group was decreased to 3 to 4
people in a few months because Customer 1 was using the product and 3 or 4 developers are enough for
maintenance for the delivered product. The other team focused on Customer 2 and started to implement
project iteratively. The next year, two new customers (Customer 3 and Customer 4) bought the product.
Customer 3 has 360 active users (among 1500 users), Customer 4 have more than 4.000 users. We
delivered the product to Customer 3 on December 10, 2018. If we analyze this customer, there were 151
bugs.
Before delivery:
Month Tasks & Defects
July 6 tasks
August 1 tasks
September 5 bugs
October 30 bugs
November 23 bugs
December 7 bugs
Table 3. How many tasks and defects are opened based on years for Customer 3 before delivery.
After delivery on alive system:
Month Tasks & Defects
5

6. December 79 bugs
January 38 bugs
Table 2. How many tasks and defects are opened based on years for Customer 3 after delivery.
The defects found from the actual users are greater than the total bugs developers found. This means it is
critical to measure the software quality and use proper quality standard to reduce defects which are found by
users on alive system.
There are a lot of bugs/defects which have not detected yet. They can go undetected for a long time. We
have to be sure the system is robust and not buggy. Some of bugs caused big incidents in the past which are
mentioned below.
4.1.1. A Short Review Of The Incidents
This section focuses on the occurred failures (six incidents) plus some assumptions based on analyzing the
accidents were given. DMS involved in a lot of accidents between the years 2014 to 2018. We summurized 3
of them. In all failures, the users does not recognize the defect and they signed the document which resulted
in the highly deep injuries and some death cases.
Software quality is more important in defence industry because huge incidents may occur when the lack of
software quality management. In first incident, the system had a bug with transparency of the watermarks on
pdf documents in old releases. When the watermark on the secret documents is not half-transparent, the
soldiers can not read the document clearly. It caused that a soldier bombed wrong destination because the
nontransparent watermark was published on top of the coordinates of the correct location on the document.
This defect caused life losts of innocent civilians, so document management system is very important in
defence industry. Recently this defect is resolved and closed.
In second example, there was a bug in the system which allows users to see any document from the system
at any time. Only administrators and autorized people can see the content of documents. The opponents
found a way to see whole documents in the system. If they add an extra document to any document, the
extra documents become visible to users. Malicious people found this bug and use secret document
information as against to our military service at the terrible events of 2015. We assume they got the user lists
too by using this defects. Consequently, they attacked the important people by using this bug.
In third example, there was a bug at the postal system. There were two postal records which were not visible
to post office users at the screens however they should see them to sent as a post. We found that the
nonvisible documents are belong to Board of Directors and they were very secret and urgent. The post office
users never know there were posts which are still waiting. After we detect the defect, we solved it. These
posts are delivered recently.
4.2. Results of Implementation
A lot of bugs may occur in the DMS about usage, permission and authorization system however they can be
avoided before the disasters by using proper software quality management. We can diminish the software
delivery time by using software quality management standard. ISO/IEC 25000 standard is very suitable,
ultimate and the best solution for this project.
There are many reasons why this organization should implement a software quality management based on
the ISO/IEC 25000 series of standards. Among the most important would be:
• Differentiating from competitors,
• meeting delivery deadlines
• and assuring that product defects are reduced.
Being able to establish service level agreements, defining specific quality parameters that the product must
meet before being delivered. Detecting defects in the software product and proceed to their removal before
6

7. its delivery, saving costs in the subsequent maintenance phase. Evaluating and monitoring the performance
of the software product being developed, ensuring that it provides results given the existing time and
resource constraints. Ensuring that the software product being developed meets the required levels for
security features (confidentiality , integrity, authenticity , non-repudiation , etc.). Checking that the product
being developed can be deployed in the production environment without compromising other systems and
maintaining compatibility with the necessary interfaces.
The opportunities of software quality management for this companies that develop software:
• Control product quality
• Improve the characteristics of the product
• Quarantee software quality to their clients
• Increate sales of their product
• Reduce defects detected after deployment
• Reduce software acquisition costs
Because of the opportunities which we mentioned above, this company should use ISO/IEC 25000
standard.
This project is developed by 1-week iterative software development life cycle. This can be changed with Agile
metodologies to ensure the software quality.
5. DISCUSSION
The 6 incidents are mentioned in this study and the proposed solution is given. In earlier related studies
there are no mention about the incidents of document management modules. In this viewpoint, this study is
unique.
6. POTENTIAL THREATS TO VALIDITY
There are potential threats to validity of this study. First of all, the proposed software quality assuarance
standard may cause performance issues while solving other issues like security issues. Because if we add
more business control operators a lot like if-clause and while-clause, the system slows down. So it is a trade
off. Secondly, the review meetings also take time of developers. If they leave their time for meetings rather
than developing and resolving defects, the final products are delivered lately.
7. CONCLUSIONS AND FUTURE WORK
We recommend Software Quality Assurance (SQA) as a proper solution in various stage of the software
development process especially for document management software such as DMS. After the review of DMS,
the recommended actions and tools are given in results section. Software quality assurance is crucial for
document management systems if it is used for military purposes. We can prevent defects and accidents
before they happen if there is a proper software quality assurance like ISO 25000. In order to improve
quality, the company can change the software development life cycle from iterative to incremental. They
need to migrate from 1 week iteration to 2 weeks sprints. Moreover, the regular review meetings with
customers can also improve quality.
In future work, the proposed methods and tools can be implemented in the project so that we can compare
the new results with existing one.
8. REFERENCES
[1] Ying-Jye James Chen, et al. "Computer-based document management system." U.S. Patent No.
6,009,442. 28 Dec. 1999.
7

8. [2] M. Michael Serbinis, et al. "Internet document management system and methods." U.S. Patent No.
6,584,466. 24 Jun. 2003.
[3] A. Celentano, S. Pozzi, and D. Toppeta, “A multiple presentation document management system,” in
Proceedings of the 10th annual international conference on Systems documentation - SIGDOC ’92, 1992,
pp. 63–71.
[4] R. M. Smith, D. M. T. Ting, J. H. Boer, and M. Mendelssohn, “Document management and production
system.” Google Patents, 01-Jan-1993.
[5] J. S. Deogun, E. Fox, and V. V. Raghavan, “INFORMATION RESOURCES & DOCUMENT
MANAGEMENT SYSTEMS: ACHIEVEMENTS AND OPPORTUNITIES,” 1994.
[6] Ralph H. Sprague Jr, “Electronic document management: Challenges and opportunities for information
systems managers.” MIS Quarterly, 1995, 29-49.
[7] R. Knowles, “Electronic document management systems: how to select and implement the right
documentation management system for your organisation,” Electron. Libr., vol. 13, no. 6, Jun. 1995.
[8] B. C. Bjork, “Electronic document management in construction - research issues and results.” 2003.
[9] F. Amato, A. Mazzeo, A. Penta, and A. Picariello, “A semantic document management system for legal
applications,” Int. J. Web Grid Serv., vol. 4, no. 3, 2008.
[10] “Document management. Change management for successful electronic document management
system (EDMS) implementation.” BSI British Standards, 31-Aug-2011.
[11] T. Leikums, “Document Management System Mobility – Main Trend in Future Document Management,”
Int. J. Comput. Commun. Eng., pp. 505–509, 2013.
[12] D. Baek, J.-H. Shin, B. Lee, and J.-W. Lee, “Defect Management Method for Content-based Document
Artifact Test in Software R&D Project,” in Advances in Computer Science and Ubiquitous Computing: CSA-
CUTE 17, Springer Singapore, 2017, pp. 1192–1198.
[13] Madadipouya, Kasra. "Importance of software quality assurance to prevent and reduce software failures
in medical devices: Therac-25 case study."
8