Process Improvement for better Software Technical Quality under Global Crisis Scenario

Process Improvement for better
Software Technical Quality under
Global Crisis scenario
Luis Rodriguez Berzosa –
lrodriguez@optimyth.com

Madrid, 4th-7th of June 2012

Software Quality under a Global Crisis Scenario?

• Current situation: Economic crisis
– Decrease in new SW development projects investment. Higher % of IT
budget for Maintenance (corrective and perfective).
– Existing software has to work better and must admit change with less cost.
– Less resources for software quality.
• Software is more and more ubiquitous
– Who doesn‟t use a mobile device and mobile apps, or a social network, or
electronic transaction systems with government agencies or their bank?
– How many process are NOT supported by software?
– How many key infrastructure of a country are NOT based on software
systems?
– How many organizations could shutdown their information systems and
not disappear?
• Even though software is one of the most widely used products of
human civilization, it shows shocking failure rates, more than any other
human invention mainly due to lack of quality.


Recipes to tackle the crisis. Manager‟s view
Subprime crisis global financial crisis  real economy crisis  recession  ?

• Well known recipes.
Size reduction
• Important: accept the
situation, analyze the
causes, accurate
diagnosis, and propose
effective measures in short
term plan.
• Objective: Stay alive when
the crisis is over.


Software Quality trends (2011)
1. Quality is not a software development add-on, but an integral part of the
process from the beginning. It is not an expense: High quality levels are
related with shorter than average development time and cost.
2. Software development/maintenance creates business risk that should be
deleted by independent quality assurance.
3. Higher use of open source frameworks.
4. Use of cloud solutions to substitute ‘in-house’ solutions.
5. Quality = Testing + … not only testing.
• The # of defects is a quality indicator but not the only one
• Wider angle approach: verify the compliance with standards and
normatives, avoid common coding and configuration , detect design anti-
patterns defects or not compliance with architectural normatives, make
software easier to maintain, and in general technically measure other
characteristics (efficiency, portability, usability, reliability…)
QA InfoTech, 2011 - http://www.prweb.com/releases/2011/01/prweb4959164.htm


The gap widens…
• … between current software systems complexity and software quality
assurance technology “in the broad sense”
– Test automation has been evolving for 20 years, but hasn‟t been able to
advance to cope with current software platforms. The cost to maintain the
automated tests has not diminish, on the contrary1.
– Static code analysis technology has been around for even longer, but the
number of languages, frameworks, middleware and architectural models
have grown exponentially. The static analyzers haven‟t been able to keep
the pace.
– The knowledge on software quality metrics, security
vulnerabilities, techniques to facilitate future maintenance and so
on, haven‟t changed much in the past 20 years. It is true that the
acknowledgement that there are quality problems has grown, but that is
not enough.
– Organization's software complexity has grown, but not at the same rate as
the use of tools to generate a software element catalog and the
dependencies between them.
(1) http://www.origsoft.com/_assets/client/docs/pdf/whitepapers/throw_away_test_automation.pdf


Maintenance: What kind of changes can you
afford?

Fuente: Boehm B., “Software Engineering Economics”, Prentice Hall (1981).


www.dilbert.com


Analyzing Software Quality issues

 Possible framework
 Situation analysis
 Diagnostics, by quality
characteristic


Let‟s try to define Software Quality

• Software quality is much more than the absence of known defects (the
goal of testing)
• Different types classes: functional quality, technical quality, process
quality, quality in-use, service quality (support), subjective quality (user
satisfaction), quality against standards (compliancy), legal quality
(SLA, contracts)
• Key aspects that our definition must include:
– Ease to accommodate changes (in the business, in the environment, in the
technology, in the load, in the regulation…)
– Must be predictable, measurable (in economic terms), accommodate new
characteristics and environments (virtualization and cloud).
– It must include the benefit of quality for the business, not only the implicit cost
of its treatment.


Classic framework: ISO 9126 / 25000 (SQuaRe)

Este marco sigue vigente, pero se cambia el énfasis en otras características.


Situation analysis: “soul searching”
• What are the software quality issues?
– Software does not fulfill user nor business expectations (functionality).
– Bad requirements, not up to date, badly defined (functionality)).
– Software has security issues (Functionality)).
– Software is not stable. Under specific circumstances, issues arise or the
system stops being reliable (reliability).
– Software is difficult to use and users and less productive (usability).
– Users only use a reduce set of functionality and are prompt to errors when
they try other characteristics (usability).
– Software requires frequent hardware updates or response times soar
under heavy loads (efficiency).
– Software changes are costly and there is a high risk of regression errors
(maintainability).
– To test the changes you need a lot of resources (maintainability).
– Software has to be maintained for a legacy platform and it is hard to
integrate with other systems (portability).
– Software also has “toxic asset”.


Diagnostic: root cause
• We are looking for root causes of the software quality
problems:
• Functionality problems: Badly designed requirements or
not up to date. Not reviewed requirements. No user
acceptance tests. No product security audits. Development
teams don‟t have software security knowledge…
• Reliability problems: Concurrency defects (race conditions
or sync failures), transactional control failures, memory
leaks, wrong error control, failover environments not
supported, no technical tests to detect these problems…
• Usability problems: Product definition did not consider user
interface, wrong user interface components used, lack of
documentation, UI not intuitive enough, complex and/or
limited configuration, no user testing.


Diagnostic (2)
• Efficiency problems: In efficient DB queries, unused available physical
resources (memory, CPU), unnecessary contention, excessive
granularity in external interfaces, no performance related requirements,
no performance tests, development teams don‟t have optimization
techniques knowledge…
• Maintainability problems: Not well documented code, unnecessary
complexity or excessive coupling, no public well documented API, the
initial design is no longer valid due to continuous patches and
modifications, no regression tests, lack of a development normative to
assure future maintenance…
• Portability problems: Excessive coupling with the execution
environment, lack of API to ease integration with other systems, no
portability requirements, the design doesn't allow to replace
environment components, lack of platform migration documentation…


Defect injection: where, how, when…

• The diagnosis has to identify the causes that inject quality defects.
Under what circumstances, when in the development lifecycle, what is
so particular about the process, the methodology, the teams, the tools
or the approach that makes some defects to be injected in the software
ending up in the product.
• Example: If I‟m suffering performance problems in the SQL sentences,
how do we get to this problem? How can I prevent it?
• The true causes are in the process, the commitment and knowledge of
the software quality teams, the development and QA practices applied.
• This analysis yields to propose specific actions to improve the quality
process.


Technical quality issues

 Reliability
 Efficiency
 Maintainability


Technical quality: Reliability
(robustness)
 Reliable software = Software that is not prone to errors or problems (crashes,
infinite loops, performance degradation, sporadic errors, etc.)
 Usual suspects: defects (design, implementation) that yield to memory leaks,
deadlocks, race conditions, data inconsistency, inefficient use of resources,
inadequate error management.
 Process causes: Reactive approach, lack of reliability testing and detection
techniques, regression errors, development and QA teams inexperience
Trends:
• Better detection and testing tools.
• Frameworks with better
monitoring, transaction support and fault
tolerance capabilities.
• Easy to use APIs.

http://www.ece.cmu.edu/~koopman/des_s99/sw_reliability/


Technical quality: efficiency
• Specification issues: If there are non functional requirements specifying the
expected load, the response times nor the expected operation rates, there is
nothing to test or to comply to… until the user complaints start coming.
• Design flaws: excessive granularity in the remote interfaces, lack of
optimization like cache use, unnecessary I/O.
• Implementations flaws: too many locks or concurrency serialization, incorrect
use of the APIs. An unpredicted infinite loop is simultaneously an availability
and performance problem.
• Database problems (lack of indices or inefficient SQL). DB programming needs
specialists.
• To assure efficiency you need knowledge about performance and the
optimization of the software technology used, performance
requirements, specific tests (load, stress, performance) and other techniques
(like profiling and design and code reviews).


Technical quality: Maintainability
• A hidden characteristic for the final user.. Unless the user has to pay
the corrective/perfective maintenance cost
• A significant amount of the software cost is due to maintenance.
Corrective maintenance is important but statistically perfective
maintenance consumes a bigger portion of the cake.
• Any maintainability improvement will have an important impact on
software cost and the response to demand changes.
• Software maintenance depends on the organization, the infrastructure
and how the software is built.
• “The software is not usually designed for change. Even when it
complies with the operational specification, its design and
documentation is usually poor for maintenance”. Review “The
Lehmann laws” [*]
• In times of crisis, is it useful to do preventive maintenance to improve
this characteristic and reduce cost?

[*] http://en.wikipedia.org/wiki/Software_evolution


How are the software cost distributed
• The % of the overall development cost used for maintenance has
increased from 50% in the „70s to up to 90% today. By maintenance
type:
– Corrective: Fix software problems (20%)
– Adaptative: Adapt the software to a changing environment (25%)
– Perfective: Update the software with new functionality (50%)
– Preventive: Improve the software make it easier to maintain, or detect
problems not identified by users (5%)
• Almost half of the maintenance time is used in the analysis (SWEBOK)
– 50% understanding/analysis, 10% design, 10% documentation, 5% implementation, 25% testing

• Functional changes: 7% of the size (FP) / year
• Problems concentrate: Typically it takes x5 times more effort to
maintain problematic modules.


Maintenance problems root causes

• Software systems are built under deadline pressure, functionality is
the priority and, if there is time, performance, reliability and usability
• Design does not have change into account. The necessary features
for a later software modification are overlooked:
– Requirements rarely take this into account.
– Logical architecture is not designed for change.
– Maintainability is not measured, hence it is not improved before delivery.
– Inadequate approach of the maintenance process.
– Poor documentation about the evolution of the software (beyond the
historical change log), of the design or software structure.
– Knowledge loss due to people rotation.
– “Maintenance = mechanic task for junior developers”.


What can you do?
 Action plan
 Technical quality “upsell”


Effectiveness/efficiency quality metrics
• Let‟s use a classic metric (IBM, 1973): Defect removal (DRR)
– % of detected, identified and removed defects before a given phase (or release).
The average DDR is around 85% in USA (IBM).
– Defect detection rate (DDR) is a little above higher (92%)
Potential defects per FP, by size (FP)
Defects source
10 100 1,000 10,000 100,000
Requirements 0.25 0.75 1.00 1.25 1.50
Architecture 0.05 0.10 0.25 0.50 0.75
Design 0.50 1.00 1.25 1.50 1.75
Source code 1.65 1.70 1.75 2.00 2.10
Tests 1.25 1.50 1.85 2.00 2.10
Documents 0.60 0.65 0.70 0.75 0.80
Database 1.50 2.00 2.75 3.00 3.20
Web content 1.25 1.50 1.75 2.00 2.25
TOTAL 7.05 9.20 11.30 13.00 14.45

Source: Applied Software Measurement, C. Jones

• Target (measurable, even reachable): DRR > 95%
• The metric needs an estimation of the potential defects . There are models that
offer a defect estimation by system size (in FP or other volume metric), the
defects fixed in each phase can be counted directly. This way you cam
measure DRR/RDD.
Potential defects = FPE, where E depends on the project and team


Action plan (in times of economic
crisis…)
• Actions have to be necessarily short term and prioritized:
– Outsource part of the quality management.
– Centralize quality management (look for scale economy).
– Train the development teams to avoid repeating errors, or be more efficient
solving the quality problems during maintenance.
– Apply cheap technical solutions (technical quality analyzers).
– Refactor software with higher priority, where the immediate benefit is higher.
– Detect and remove “dead code”, to reduce volume of code to maintain.
– Introduce efficient techniques to solve quality problems with higher impact:
Design / code inspections, requirements review, special testing, coding
standards, etc.
– Forget about reaching a high test automatization rates (or try to automate
some type of tests if its future maintenance cost allows it).


How much do the defects cost
• The cost of a defect grows as we move along in the life
cycle. This fact encourages a strategy of early detection
and correction, or even prevention.
• The quality problems underlying cost is not only the cost
of correcting them…
– But sorter development time,
– Less development and maintenance cost,
– Less project failure probability,
– Less contractual problems,
– Higher user satisfaction
• Cf. The Economics of Software Quality - Capers Jones &
Olivier Bonsignour.


Scale economy for quality techniques
• The bigger the size of the software, the bigger the impact due to lack of
quality. Size increases the unitary cost (FP), but using prevention
defect removal techniques before testing, yield to higher cost reduction
for bigger sizes.
Cost reduction
Size (FP) Low quality Medium quality High quality
(BC)->AC)

10 688 625 594 14%

100 886 787 748 16%

1,000 1,040 920 847 19%

10,000 2,393 2,380 1,872 22%

100,000 5,078 4,340 3,819 25%

Cost in $ per function point.
Source: The Economics of Software Quality, C. Jones, O. Bonsignour

• All types of testing reach up to 85% effectiveness (15% of defects end
up in the released version). High quality demand prevention and
correction techniques before testing (> 95% effectiveness).


Quality toolbox: prevention techniques

• Reuse of certified components
• [*] Training and software quality awareness
• [*] Formal inspections or reviews. http://www.gilb.com/Inspection
• Specialized techniques: Poka-Yoke (ポカヨケ “error proof”), QFD
• [*] Prototyping
• Patterns (design, requirements, code, architecture, documentation…)
• [*] Quality analysis (and management) tools
• [*] Include quality score in status reports
• Agile development(TDD, pair programming)

In these times of economic crisis, implement practices that
produce value in the short/mid term [*]


“Quality debt”: How to “upsell” technical
quality
• Quality debt: Measures the necessary cost and effort to fix detected
quality problems (acquired debt, that has to be returned in the future with
accrued interests)
• State the lack of technical quality in economic terms is not very
accurate, but it gets a positive effect and help justify the preventive work

(*) Similar term: technical debt, introduced by Ward Cunningham in1992.
http://www.martinfowler.com/bliki/TechnicalDebt.html


Estimate software quality is a difficult task


www.dilbert.com (2)


Final recommendations
• Soul search. Where do you have higher impact quality problems, their
root cause, how were they introduced and what can you do to
detect/fix/prevent them.
– After this evaluation, build an action prioritized action plan that takes into
account the organization‟s principles
• Up the ladder awareness: “upsell” the immediate and tangible benefits
that quality improvement offers, and that it should be a priority one
objective in times of economic crisis.
– Historical data, quality debt metrics, benchmarking, learn from others
– The metric “cost per defect” is not useful in economic terms
• Quality innovation. Old approaches may not be valid in the actual
context..
– Other classical practices, like inspection, are still valid; if the soul search
dictates them, use them.
– You can even propose a complete change in the paradigm:
http://googletesting.blogspot.com/2011/01/how-google-tests-software.html


Situations that reduce quality
• Not structures requirement definition; lack of volume metrics
• Inexistence of any kind of risk analysis
• Lack of explicit quality rules; not include internal quality in the project
status reports
• High requirement change rate + inadequate requirement change
management
• Excessive final user or project managers pressure
• Inexperience on how to avoid common defects (i.e. reliability,
performance, security, maintainability…)
• The attitude “Only the quality we can afford” (usually nothing)

Any improvement actions you may implement will easily outweighed by
a nasty combination of these “negative forces”


Process Improvement for better Software Technical Quality under Global Crisis Scenario

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Process Improvement for better Software Technical Quality under Global Crisis Scenario

Similar to Process Improvement for better Software Technical Quality under Global Crisis Scenario (20)

Recently uploaded

Recently uploaded (20)

Process Improvement for better Software Technical Quality under Global Crisis Scenario