SOFTWARE RELIABILITY AND QUALITY ASSURANCE Reliability issues Reliability metrics Reliability growth modeling Software quality ISO 9000 certification for software industry SEI capability maturity model comparison between ISO and SEI CMM

1. Software Engineering
Dr. Kamal Gulati
Part 8 : SOFTWARE RELIABILITY AND QUALITY
ASSURANCE

2. Contents for SOFTWARE RELIABILITY
AND QUALITY ASSURANCE
1. Reliability issues
2. Reliability metrics
3. Reliability growth modeling
4. Software quality
5. ISO 9000 certification for software industry
6. SEI capability maturity model
7. comparison between ISO and SEI CMM

3. SOFTWARE RELIABILITY AND
QUALITY ASSURANCE

4. Quality
Quality Refers to any measurable characteristics such as
• Correctness
• Maintainability
• Portability
• Testability
• Usability
• Reliability
• Efficiency
• Integrity
• Reusability and
• Interoperability.

5. Software Quality Assurance (SQA)
• Consists of a means of monitoring the
software engineering processes and methods
used to ensure quality.
• The methods by which this is accomplished
are many and varied, &
• May include ensuring conformance to one or
more standards, such as ISO 9000 or CMM.

6. SQA Encompasses
• Quality management
• Effective SE technology (methods and tools)
• Formal technical reviews.
• Multi-tier testing strategy.
• Ongoing documentation.
• A procedure to ensure compliance with
software development
• Standards.
• Measurement and reporting

7. Verification and Validation
• Verification and validation is the process of
checking that a product, service, or system
meets specifications and that it fulfills its
intended purpose.
• These are critical components of a quality
management system such as ISO 9000.
• Ensure the validation is performed by a
disinterested third party.

8. • Verification is a Quality control process that is
used to evaluate whether or not a product,
service, or system complies with regulations,
specifications, or conditions imposed at the start
of a development phase. Verification can be in
development, scale-up, or production. This is
often an internal process.
• Validation is Quality assurance process of
establishing evidence that provides a high degree
of assurance that a product, service, or system
accomplishes its intended requirements. This
often involves acceptance of fitness for purpose
with end users and other product stakeholders.

9. Software Quality Assurance
Software quality assurance (SQA) consists of a
means of monitoring the software
engineering processes and methods used to
ensure quality.
The methods by which this is accomplished
are many and varied, and may include
ensuring conformance to one or more
standards, such as ISO 9000 or a model such
as CMM.

10. ISO
• ISO is the International Organization for
Standards and produces international
standards.
• A network of standards institutes forms the
ISO, of which there currently are 163
countries represented by these institutes.
• Since the ISO is the world’s largest developer
of standards, it would be beneficial to follow
these standards.

11. What is ISO 9000?
• ISO 9000 is a group of standards from ISO for
quality management systems (QMS).
• This standard began as a standard specifically
for manufacturing but has been used in other
businesses.
• If a business wants to sell a product or do
business internationally, being ISO 9000
certified will mean that it adheres to a high
level of quality standards that are understood
the world over and will attract more clients.

12. • ISO 9000 certification means that a company can
meet requirements related to the quality of a
product and also customer service.
• It provides a set of rules that have been followed
time and again successfully to manage the
various processes of an organization to turn out a
successful product.
• The standard explains what requirements for
quality a company must meet to be effective.
• An organization can conduct self-audits or checks
on itself to ensure that ISO 9000’s steps to
success are working or it can obtain someone
from outside the organization to perform an
audit, or even a client.

13. • ISO 9000 is a family of standards for quality
management systems.
• ISO 9000 is maintained by ISO, the
International Organization for
Standardization and is administered by
accreditation and certification bodies. The
rules are updated, as the requirements
motivate changes over time.

14. Some of the requirements in ISO 9001:2008
(which is one of the standards in the ISO 9000
family) include
• a set of procedures that cover all key processes in
the business
• monitoring processes to ensure they are effective
• keeping adequate records
• checking output for defects, with appropriate and
corrective action where necessary
• regularly reviewing individual processes and the
quality system itself for effectiveness; and
• facilitating continual improvement

15. Benefits of ISO 9000 certification
• Focus on "how they do business”
• Documented processes
• Corrective and Preventative
• Employee morale
• Customer satisfaction
• Reduced problems
• Better products and services
• quality
• Improved profit levels
• Improved communications

16. • 1. ISO-9000 forces an organization to focus on
"how they do business". Each procedure and
work instruction must be documented and
thus, becomes the springboard for Continuous
Improvement.
•
2. Documented processes are the basis for
repetition and help eliminate variation within
the process. As variation is eliminated,
efficiency improves. As efficiency improves,
the cost of quality is reduced.

17. 3. With the development of solid Corrective and
Preventative measures, permanent, company-wide
solutions to quality problems are found.
4. Employee morale is increased as they are asked to
take control of their processes and document their
work processes.
5. Customer satisfaction, and more importantly
customer loyalty, grows. As a company transforms
from a reactive organization to a pro-active,
preventative organization, it becomes a company
people want to do business with.

18. • 6. Reduced problems resulting from increased
employee participation, involvement,
awareness and systematic employee training.
7. Better products and services result from
Continuous Improvement processes.
8. Fosters the understanding that quality, in
and of itself, is not limited to a quality
department but is everyone's responsibility.

19. 9. Improved profit levels result as productivity
improves and rework costs are reduced.
10. Improved communications both internally
and externally which improves quality,
efficiency, on time delivery and
customer/supplier relations.

20. ISO 9126
• Product quality is an international standard for
the evaluation of software quality.
• The fundamental objective of this standard is
to address some of the well known human
biases that can adversely affect the delivery
and perception of a software development
project.

21. • These biases include changing priorities after
the start of a project or not having any clear
definitions of "success".
• By clarifying, then agreeing on the project
priorities and subsequently converting
abstract priorities (compliance) to measurable
values (output data can be validated against
schema X with zero intervention), ISO/IEC
9126 tries to develop a common
understanding of the project's objectives and
goals.
International Organization for Standardization (ISO)
International Electrotechnical Commission (IEC)

22. • ISO/IEC 27002 is an information security
standard published by the International
Organization for Standardization (ISO) and by
the International Electrotechnical Commission
(IEC), titled Information technology – Security
techniques – Code of practice for information
security management.

23. The standard is divided into four parts:
• Quality Model
• External Metrics
• Internal Metrics
• Quality in Use Metrics.
http://www.cse.dcu.ie/essiscope/sm2/9126ref.html

24. CMM
• The Capability Maturity Model (CMM) is a service
mark owned by Carnegie Mellon
University (CMU) and refers to a development
model elicited from actual data.
• The data were collected from organizations that
contracted with the U.S. Department of Defense,
who funded the research, and they became the
foundation from which CMU created the
Software Engineering Institute (SEI).
• Like any model, it is an abstraction of an existing
system.

25. • When it is applied to an existing
organization's software development
processes, it allows an effective approach
toward improving them.
• Eventually it became clear that the
model could be applied to other
processes.
• This gave rise to a more general concept
that is applied to business processes and
to developing people.

26. CMM Levels
• Initial
• Managed
• Defined
• Quantitatively managed
• Optimizing

27. • Level 1 - Initial (Chaotic)
It is characteristic of processes at this level that they are
(typically) undocumented and in a state of dynamic change,
tending to be driven in an ad hoc, uncontrolled and reactive
manner by users or events. This provides a chaotic or
unstable environment for the processes.
• Level 2 - Repeatable
It is characteristic of processes at this level that some
processes are repeatable, possibly with consistent results.
Process discipline is unlikely to be rigorous, but where it
exists it may help to ensure that existing processes are
maintained during times of stress.
• Level 3 - Defined
It is characteristic of processes at this level that there are
sets of defined and documented standard processes
established and subject to some degree of improvement
over time. These standard processes are in place (i.e., they
are the AS-IS processes) and used to establish consistency
of process performance across the organization.

28. • Level 4 - Managed
It is characteristic of processes at this level that,
using process metrics, management can
effectively control the AS-IS process (e.g., for
software development ). In particular,
management can identify ways to adjust and
adapt the process to particular projects without
measurable losses of quality or deviations from
specifications. Process Capability is established
from this level.
• Level 5 - Optimizing
It is a characteristic of processes at this level that
the focus is on continually improving process
performance through both incremental and
innovative technological changes/improvements.

29. The Software Engineering Institute (SEI) constantly analyzes the
results of CMM usage by different companies and perfects the
model taking into account accumulated experience.

30. Software Reliability
• Software quality measures how well software
is designed (quality of design), and how well
the software conforms to that design (quality
of conformance).
• It is often described as the 'fitness for
purpose' of a piece of software.

31. Software Reliability
1. Software requirements are the foundations from which
quality is measured. Lack of conformance to requirement
is lack of quality.
2. Specified standards define a set of development criteria
that guide the manager is software engineering. If criteria
are not followed lack of quality will almost result.
3. A set of implicit requirements often goes unmentioned,
like for example ease of use, maintainability etc.
If software confirms to its explicit requirement but fails to
meet implicit requirements, software quality is suspected.

32. SOFTWARE RELIABILITY
• Defined as the probability of failure free
operation of a computer program in a specified
environment for a specified time period.
• Can be measured directly and estimated using
historical and developmental data (unlike many
other software quality factors)
• Software reliability problems can usually be
traced back to errors in design or
implementation.

33. Software Reliability Metrics
• Reliability metrics are units of measure for system
reliability.
• System reliability is measured by counting the
number of operational failures and relating these
to demands made on the system at the time of
failure.
• A long-term measurement program is required to
assess the reliability of critical systems.

34. Reliability Metrics - part 1
• Probability of Failure on Demand (POFOD)
– POFOD = 0.001
– For one in every 1000 requests the service fails per time
unit
• Rate of Fault Occurrence (ROCOF)
– ROCOF = 0.02
– Two failures for each 100 operational time units of
operation

35. Reliability Metrics - part 2
• Mean Time to Failure (MTTF)
– average time between observed failures (aka
MTBF)
• Availability = MTBF / (MTBF+MTTR)
– MTBF = Mean Time Between Failure
– MTTR = Mean Time to Repair
• Reliability = MTBF / (1+MTBF)

36. Time Units
• Raw Execution Time
– non-stop system
• Calendar Time
– If the system has regular usage patterns
• Number of Transactions
– demand type transaction systems

37. Classification of failures
A possible classification of failures of software
products into five different types is as follows:
1. Transient
2. Permanent
3. Recoverable
4. Unrecoverable
5. Cosmetic

38. • Transient. Transient failures occur only for
certain input values while invoking a function
of the system.
• Permanent. Permanent failures occur for all
input values while invoking a function of the
system.
• Recoverable. When recoverable failures occur,
the system recovers with or without operator
intervention.

39. • Unrecoverable. In unrecoverable failures, the
system may need to be restarted.
• Cosmetic These classes of failures cause only
minor irritations, and do not lead to incorrect
results. An example of a cosmetic failure is the
case where the mouse button has to be
clicked twice instead of once to invoke a given
function through the graphical user interface.

40. Reverse Software Engineering
• Reverse engineering is the process of
analyzing software with the objective of
recovering its design and specification.
• The program itself is unchanged by the
reverse engineering process.
• The software source code is usually available
as the input to the reverse engineering
process.
• Sometimes, however, even this has been lost
and the reverse engineering must start with
the executable code.

41. • Reverse engineering is not the same thing as
re-engineering.
• The objective of reverse engineering is to
derive the design or specification of a system
from its source code.
• Reverse engineering is used during the
software re-engineers process to recover the
program design which engineers use to help
them understand a program before
reorganizing its structure.

42. Re-Engineering Process
• The input to the process is a legacy program
and the output is a structured, modularized
version of the same program.
• At the time as program re-engineering, the
data for the system may also be re-
engineered.

43. The activities in this re-engineering
process are:
• Source code translation:
The program is converted from an old programming
language to a more modern version of the same
language or to a different language.
• Reverse Engineering:
The program is analyzed and information extracted
from it which helps to document its organization and
functionality.
• Program structure improvement:
The control structure of the program is analyzed and
modified to make it easier to read and understand.

44. • Program modularization:
Related parts of the program are grouped
together and, where appropriate, redundancy
is removed.
• Data re-engineering:
The data processed by the program is changed
to reflect program changes.

46. ThankYou&
KeepLearning!
?Follow us :
You can find me at
https://www.linkedin.com/in/drkamalgulati/
https://mybigdataanalytics.in/
https://www.facebook.com/DrKamalGulatiBig
DATA/