By Rodrigo Guzmán, Pilar Barrio la Iglesia & Raúl Martínez
Software Product Quality
Can software quality be seen as practical and business ori-ented,
and with indicators showing that quality can have
the effect of improving products and business?
Can we go beyond the idea of measuring software quality
through defects of the software product, and move on to
also consider other parameters?
What does a software quality model look like in practice?
By analyzing an actual case, this paper shows how the ab-stract
concepts of software quality were implemented, using
applicable quality models and quality indicators, and how
a path to future cycles of optimization could be achieved
while pursuing a continuous cycle of business and software
To do this, the intuitive process followed by the Organiza-tion
is first described, and then a series of actions to further
improve what has already been achieved are proposed and
explained. These actions could be implemented by differ-ent
organizations to improve the quality of their software
Software development and evolution processes are considered essential
to obtaining and maintaining quality in software products. For a long
time, several models have been proposed to help understanding while
increasingly improving how software products are built and maintained.
From the point of view of the software product itself, software quality
models, perhaps less known and used in practice, help us to understand
how quality requirements can be found, describe their quality charac-teristics,
and then evaluate the quality of the software built, based on
This paper shows how an Organization that develops and maintains
a large product, which is central to the company business, has moved
towards the idea of product quality and has identified the need for a
new organization in its Quality Assurance area and for a change to the
It also discusses how to define business indicators closely related to the
software product, so that the indicators are actionable, taking into ac-count
the observed behavior of users and customers.
62 Testing Experience – 22/2013
An e-commerce software product and the evolution
of its quality
The original product
For several years the e-commerce software product was a closed mono-lithic
platform. Both database and source code were centralized. The
design of this architecture was doubtless intended to generate multiple
competitive advantages, but high costs, loss of flexibility, and high risks
were incurred when changes were required.
The impact of bugs
Under this monolithic, centralized, and closed scheme, defects introduced
as a result of a mistake during development had a high probability of
impacting on different elements of the application and, therefore, having
direct and important consequences on the business.
The risk associated with a latent defect was very high and the unintended
consequences on the business could be significant. As a result, for several
years the focus was on correcting defects. The QA Team was committed
to covering the widest possible set of code with tests building a very
large safety net before any implementation.
The product development process
Consequently, processes designed under these assumptions have had
very slow development and testing cycles, taken extensive time, been
difficult to scale up and had a “time-to-market” far from usual in the
The introduction of a defect generated an impact, not only often unpre-dictable,
but also difficult to solve due to the slow and costly processes
involved. Development and release cycles were significantly long, weeks/
months were devoted to testing, and the release rate was sometimes
biweekly or monthly. Exceptionally, a release was implemented to solve
critical failures, but only in cases of emergency and after deliberation
and approval by a committee.
This all created organizational dynamics with a low tolerance for defects,
and with incentives aligned to avoid and prevent any type and size of
defect. Errors were frowned on and, therefore, the ability to produce
and iterate tended to decline.
Consequently, the task of testing became critical, and motivations and
challenges revolved around improving testing practices so they were
scalable in effort and time, increasing the coverage of testing, and im-proving
the design of test cases, defect reports, etc.
Testing Experience – 22/2013 63
Indicators and incentives
The concept of product quality was simple and was defined only in terms
of detected errors. Metrics and goals were designed around these con-cepts
and focused on measuring and controlling the number of failures in
production. Incentives for team members were aligned with these goals.
This biased approach introduced important frictions into the Organiza-tion
and also introduced business constraints. Assuming that success and
product quality were dependent only on the software failures variable
was like not seeing the wood for the trees.
Figure 1. This stage can be visualized as the base of a pyramid. The effort is put into
a flawless product and the rest of the features are assumed to have already been
implemented. Indicators of success are related to a product with a low error rate. 
The new product
Architecture and its influence on quality and processes
A dramatic change happened when the e-commerce product became
an open platform for e-commerce instead of a website.
This had strong implications for all the variables of the Organization
and its business. The change meant redesigning architecture, open-ing
the platform, and decentralizing both the database and the source
code, allowing separate departments distribution of the e-commerce
Definition and design of the new architecture succeeded in eliminating
friction as well as functional and technical dependencies which existed
among the different modules in the monolithic model. Consequently, risk
was distributed and this became the most important change factor in
the Organization and the business.
Limiting the impact of errors and reducing business risk resulted in
diminished tension regarding processes and considerable improvement
in satisfying the Organization needs.
Additionally, responsibility for product quality control was distributed
throughout the various areas.
Agile processes for developing and testing were established, enabling
an increase in test frequency, the amount and parallelism of changes,
improved deployment speed and defect correction, and increasing de-ployment
frequency from one every fifteen days to, in some cases, more
than eighty releases to production in one day.
Agile practices reduced cycles of design, development, testing, and im-plementation,
and weekly sprints with constant delivery of software
artifacts were achieved. Project duration was also significantly reduced.
Implementing a change in this context is now simpler, solving a problem
is quicker than before, the defects life cycle has been significantly reduced,
the impact of errors has diminished, and risk has therefore decreased.
This new perspective has introduced a radical change into the orga-nizational
dynamics, focusing and aligning teams on the search for
continuous improvement and constant change rather than attempting
to prevent all defects and on achieving a good enough quality product
which does not affect the business indicators.
Development teams have increased their capacity to build, test, and
quickly solve problems.
In this new scenario, the goals and competences of the original QA Team
became obsolete, and a radically new paradigm was necessary.
The effect and impact of product defects is no longer the main cause of
friction with the business. Other variables have been exposed that affect
it, which formerly were not explicitly considered part of the definition
of product internal quality. These variables had always existed but had
never come to the fore.
The incentive of the new QA Team is no longer based on reducing errors
in the production environment, but on being committed to continuous
improvement, constant change, and identifying aspects of the product
that generate friction, and directly affect the business and the comple-tion
of transactions on the platform.
This is how the evolution of the concept of product quality has enabled
the e-commerce company platform to support business needs.
New indicators and incentives
The evolution in the definition of product quality can encompass aspects
beyond the platform operation, highlighting other issues such as overall
product performance, usability, user’s understandability, usefulness,
degree of transaction conversion, success, etc., which certainly affect
the evolution and growth of the business.
It is no longer enough to avoid product errors and ensure platform per-formance
and availability. It is necessary to give the users what they need
and the way they need it. The new goals are to fulfill their expectations
at the right time, and create an experience that differentiates the e-com-merce
product from its competition, with chance of mistakes but with
very streamlined and responsive processes for change and correction.
Indicators at several levels monitor the behavior of the new variables:
Level 1 – Scalability
Traffic light status indicators track platform availability; “health” of the
infrastructure, networks and databases; downtime detection; opera-tional
capacity; performance; etc.
Level 2 – Core Operation
Malfunction-oriented indicators on product operation, detecting inci-dents,
security issues, integration issues, code coverage level in testing,
Level 3 – Usability
Indicators focusing on detection of transactional flow frictions and
navigational friction within the platform, as well as understanding,
comprehension, and user adoption, detection of “drop outs”, “bounce”,
response times, etc.
Level 4 – Customer Satisfaction
Indicators focusing on the level of complaints, claims, and “noise” deter-mination
arising from users’ problems on the platform, understanding,
failures, adoption, etc.
Level 5 – Profitability
Indicators to measure degree of flows conversion within the platform,
visits vs. transactions, and other business and money metrics.
Level 6 – Competitiveness
Indicators that analyze product positioning vs. competitive products,
platform level of traffic, market share, new “features”, etc.
Figure 2. This stage can be visualized when completing the pyramid in Figure 1 with two
other levels: a useful product for users and, once this is filled in, a successful product for the
business. In the lower levels, success indicators relate to low rates of errors, whereas, as they
move towards the vertex, they are defined in relation to KPIs of interest to the business, not
forgetting client satisfaction.
While several decisions mentioned in the preceding paragraphs were
possibly adopted in an intuitive way, they were undoubtedly intended to
solve problems that arose in the operational context, and/or to improve it.
Some questions arising from the original product description are:
▪▪ What motivates the e-commerce Company …
▪▪ … to face a change in architecture?
▪▪ … to face an organizational change?
▪▪ What is the meaning of the new indicators generated?
Possible reasons behind the decisions and changes are analyzed below.
Stakeholders and their expectations
The Company and their expectations:
▪▪ An agile product, easy to expand
▪▪ With a competitive level of functionality
64 Testing Experience – 22/2013
▪▪ With acceptable development and operation costs
▪▪ With indicators that show business performance through customer
behavior in using the product.
The visitor and their expectations:
▪▪ A mature and reliable product
▪▪ A fully fledged product compared to others of its kind
▪▪ Simple to use
▪▪ Highly secure
▪▪ With adequate performance
Groups responsible for product development and their expectations:
▪▪ A product with adequate time-to-market
▪▪ Analyzable and modifiable with minimum impact
▪▪ Testable before releasing changes
Although expectations were initially fulfilled by the original product, they
changed as new competition appeared in the market and the product
evolved. Stakeholders began to feel disappointed, perceiving the product
as difficult to maintain and test. The Organization was seen as rigid and
with few indicators for tracking product/business performance.
These warnings forced the e-commerce Company to face radical changes,
▪▪ A very important change in architecture
▪▪ A new product development process
▪▪ A new product development and support organization
▪▪ A new set of indicators
As mentioned previously, the original architecture was monolithic, with
technical problems and organizational accountability issues, and conflict
situations were difficult to solve.
The architectural change was conceived to fulfill management expecta-tions
and the product groups’ expectations.
Visitors/users still observed the same or even improved product behavior.
But the cost of that behavior was very different when comparing the
original architecture with the new one.
This was not visible to visitors, but was very significant to the business.
The product development process
The original development and product maintenance processes were pos-sibly
based on the product structure, and included all the steps needed
to develop and test a large monolithic product with the deliverables
and controls required to perform a comprehensive test and subsequent
implementation of a highly coupled product.
The consequences were mentioned earlier in this paper.
The processes were changed to fulfill management and product groups’
Implementation of Agile practices seems appropriate for a decentralized
architecture as described for the new product, assigning smaller teams
complete responsibility for some aspect of it, thus providing improved
clarity and definitions.
with the decision
and enjoy test
case design in
the cloud today!
CaseMaker SaaS systematically
supports test case design by
covering the techniques taught in
the ISTQB® Certi ed Tester program
and standardized within the British
Standard BS 7925. The implemented
techniques are: equivalence
partitioning, boundary check, error
guessing, decision tables, pairwise
testing, and risk-based testing.
CaseMaker SaaS is your perfect t
between requirement management
and test management/test
and enjoy a 14-day
trial period for free!
Your license starts at 75 €/month (+ VAT).
The original organization that developed and maintained the e-com-merce
product evolved in parallel with the new processes.
The organizational changes were focused on fulfilling the expectations
of management and of the groups responsible for the product relative to
the “time-to-market”, responsibilities, and maintenance cost.
The development of new indicators showing the relationship between
product behavior and business performance was also addressed.
The new organizational model dissolved the centralized QA Team and
distributed its members and responsibilities in the new product teams,
now responsible for planning and implementing new features or changes
until these reach the production or live stage.
A new area called Business Assurance was created, with no similarities to
the previous QA Team but in charge of operational control of the product
and analyzing its impact on business through performance indicators.
Due to the product’s original monolithic structure and organization, it
was difficult to detect which item to act on in a conflict situation. Also,
indicators related to the base of the pyramid in Figure 1 were not useful
for the business.
Management’s expectations of monitoring evolved into actionable in-dicators,
meaning that action can now be taken on any portion of the
product or the business to correct or enhance a situation once it has
been detected. Not all indicators are necessarily financial.
Therefore, new processes, architecture, indicators, and organization al-low
a more direct measurement of the behavior of a given feature and
enable it to be acted on it if necessary.
The new indicators defined by levels are intended to detect the behavior
of the product in use by the customers (or other interested visitors) and
define interventions, if needed, into parts of the product. See Figure 2.
While not simple, the indicators enable the e-commerce Company to
establish whether or not business objectives related to the software
product have been achieved. They also form the basis for evaluating
incentives given to those responsible for the various aspects of its de-velopment
Possible next steps
Current and new stakeholders
Continue with stakeholder detection. In addition to site visitors and
merchants, reach other possible stakeholders: external developers who
extend the product, auditors, information security, etc. Understand
their influence and propose communication channels with them to
Expectations discovery tools
For the detected interested communities, find out what their expecta-tions
are in respect of the product characteristics. This can be done
through instant surveys, online questions to visitors/buyers and mer-chants,
or through meetings with other stakeholders, e.g. with external
66 Testing Experience – 22/2013
Formalization of quality model
The evolution of quality concepts in the Organization justifies a formal-ization
of the quality model allowing transference to the Company’s own
staff and third parties, such as developers, that generate extensions to
the e-commerce platform through APIs.
This will permit understanding by all developers of the expected quality
level and a possible assessment by the e-commerce Company, regarding
the level of quality offered. User, visitors, or buyers will perceive a product
with homogeneous quality.
Committed external developers
Explain to external developers the expectations of the e-commerce Com-pany
regarding quality through a visible, accurate, and easily understood
model. This will guide their development work and help to obtain their
commitment to quality.
These KPIs are derived from measures taken after a release of changes
to the production environment.
In this way, external and internal developers will easily associate a busi-ness
KPI with the portions of the product affecting or determining that
KPI and take action on the product if necessary.
Product development KPIs
These are indicators derived from measures obtained prior to the release
of a new feature of the product.
Develop and/or improve the KPIs that measure the quality of product
development and testing, so that final quality can be predicted and their
potential impact on the production environment measured.
Business and product KPIs and how to obtain them must be proposed,
discussed and agreed with technical and business stakeholders.
These measures must form another component of the quality model
The steps described close the loop on an improvement cycle that should
be repeated continuously in order to improve product quality and make
the business more and more competitive and successful every single day.
The case has shown, from a practical point of view, how an organization
has evolved in relating business success to the software product quality.
It also explained how the Organization changed its structure and work-ing
models in the area of software to adapt to market demands on times
After the description of the case, an analysis was made of the steps
followed intuitively by the Organization, creating a parallel with the
development of a software quality model.
Finally, a proposal was made to the Organization to further improve the
quality of its product and, thus, the business results.
Testing Experience – 22/2013 67
The example, although just a case, can be used by other enterprises and
organizations who want to achieve a stronger and better relationship
between their software product and their business.
Bibliography and references
 Factors in software quality; NTIS, 1977, J. McCall.
 Software Quality: The elusive target; IEEE, 1996, B. Kitchenham S.
 What does “Product Quality” really mean?; Sloan Management
Review, Fall 1984, D. Garvin.
 A model for software product quality; Australian Software Quality
Research Institute, October, 1994, G. Dromey.
 Relating Business Goals to Architecturally Significant Requirements
for Software Systems; CMU/SEI-2010-TN-018, 2010, Bass, Clements.
 Quality Attribute Workshops (QAWs); Third Edition, CMU/SEI-2003-
TR-016, 2003, Barbacci.
 Software Architecture in Practice; 2nd ed., 2003, Bass, Clements,
 ISO/IEC 25000 Software engineering: Software product Quality
Requirements and Evaluation (SQuaRE) – Quality model; 2005, ISO/
 ISO/IEC 25010 Software engineering: Software product Quality Re-quirements
and Evaluation (SQuaRE) – Quality model; 2011, ISO/IEC.
 Cornering the chimera; IEEE SOFTWARE, 1996, G. Dromey.
 Competing on the Eight Dimensions of Quality; HBR, 1987, D.
 Software Quality Models in Practice; Umfrage-Ergebnisse, 2010,
 In Application Projects, ‘Success’ Needs Many Definitions; 2011,
 Application Quality Assurance for Nonfunctional Requirements;
 Redefining-software-quality; www.gojko.net, 2012, Gojko Adzic.
 Norms and Standards in SAP’s Development Process Framework;
 Attractive quality and must-be quality; ASQC, 1996, N. Kano, N.
Seraku, F. Takahashi, S. Tsuji. ◼
about the authors
Rodrigo Guzmán is Quality Assurance Senior Man-ager
at MercadoLibre.com a Latin American leading
e-commerce technology company. He joined the com-pany
in 2004 and is responsible for defining, imple-menting,
and managing software quality policies that
enable IT to ensure and control the operation of the
website in the 12 countries of Latin America. He is also
responsible for improving the IT processes as well as
Rodrigo has a degree in Business Administration, a postgraduate degree in
Quality Management, and is a Certified Scrum Master. He has nineteen
years’ experience in IT, mainly in processes, projects, and quality.
Before joining MercadoLibre.com, he worked for 10 years in the IT area at
Telecom Argentina, a telecommunications company.
Among other things, in 2007 he implemented the quality management
system for the QA department, currently certified under ISO-9001:2008
standards. He also implemented test automation, project management
process improvement, and Agile practices in IT projects.
Pilar Barrio la Iglesia has specialized in Quality Man-agement,
having been responsible for those areas at
consulting firms. She has extensive practical experi-ence,
as well as a solid theoretical background in soft-ware
development and the technology-related aspects
of business. She has collaborated in the development
and implementation of tools to support IT process
improvement, and participated in, coached, and man-aged
diverse projects in Argentina and abroad.
Pilar is a founding partner of RMyA SRL, Argentina, a firm established 13
years ago which is dedicated to providing software product testing and
process improvement services to IT organizations and to medium-to-large
customers in the areas of project management and quality management.
She is an instructor on quality-related courses in various areas related to
computer technology and is a former Professor at Universidad de Buenos
Aires in the Databases and Operative Systems areas.
Pilar is a member of the Quality Commission – CESSI.
Raúl Martínez specializes in IT Project Management
and software engineering.
He has a theoretical background and practical experi-ence
in software development in leading local and
international companies and has worked with clients’
IT divisions to develop innovative, reliable, and cost-effective
Raúl is a founding partner of RMyA SRL, Argentina,
a firm established 13 years ago which is dedicated to providing software
product testing and process improvement services to IT organizations and
to medium-to-large customers in the areas of project management and
He has been a Regular Associate Professor in the area of Software Engineer-ing
at the Faculty of Engineering, Universidad de Buenos Aires for over 30
years, teaching Project Management courses for future System Engineers
Raúl is a member of the Curriculum Committee at the Faculty of Engineering
(UBA), for the undergraduate degree in Systems Engineering. He is also a
member of the Quality subcommittee in Information Technology and Project
Management at IRAM and an IRAM appointed expert to ISO/IEC JTC1/SC7/
WG6 on the subject of Product Quality ISO/IEC 25000 Standards (SQuaRE).
He is a member of the Quality Commission – CESSI.