Software Product Testing in
The new economic reality puts spotlight on agile software product development. Agile provides the opportunities to
reduce cost of development and achieve payback on the investment sooner than in the traditional approach by
allowing decisions on the product future before making the full investment. Software Product Testing in an agile
environment offers a completely different perspective and is packed with multiple challenges.
Proteans was part of a product development effort, spanning over 3 years where in we gained considerable
understanding of product testing in an agile environment. The lesson we learnt and applied has been brought into
this White Paper.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com
Proteans’ approach towards implementing an agile Testing method is very different from the traditional testing
practices. We gained knowledge of testing products in an agile environment through a recent client engagement
that gave us responsibility to develop, enhance, and maintain a suite of Add-on products for Microsoft Dynamics
CRM versions 1.2, 3.0, and 4.0. The following sections of this paper identify Proteans methodology towards
successful release and support of this suite of products.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 2/8
To Test a suite of 18 Add-on products and synchronize worldwide releases
Support all previous released versions (via upgrades) of the products in multiple languages [English, French,
German, and Dutch], which are deployed across different environments
The product was used by end Users from wide array of business vertical s– viz. Airlines, NBA teams, ISV’s,
Professional Services firms, Investment Banks etc - all aggregating to 4000+ clients with a installed user base of
The depth and breadth of complexity of the products and deployment options translated to 400+ Testing scenarios
Proteans, set-up a small group of Product Management Team (Client) to generate the Product Backlog together
with prioritization of the features based on maximum business value delivered
We scheduled releases, every 9 weeks - broken down into three iterations, each of 3 weeks duration. The first
two iterations are the’ construction phase’ with third being the ‘Product stabilization’ iteration
We defined each feature or defect as ‘User Story’ and assigned priority to it
We organized the engagement team into two development teams and one QA Team. Each development team has
a QA resource, called the ‘Integrated QA’ and four developers. A team of 7 QA resources formed the ‘Regression
The ‘Integrated QA’ is involved in the iteration planning meetings where each User Story is estimated taking
into account both the development effort as well as the testing effort. Testing is ‘story’ focused and context
based – we did not test allied features but focused only on the feature/defect defined in the User Story
At the end of the iteration, the Regression QA team picks up the build and does a regression of the product. This
takes care of Integration Testing and Regression Testing of the products. The ‘Regression QA’ team was assigned
the task to create multiple environments, respond to Client Support issues, update Product Test Suites, and
update the outstanding Defect Sheet which is base-lined
Proteans used RALLY for Project Management to maintain all project artifacts viz. User Stories, Defects, Test
Cases, Product Test Suite. Besides project management, Rally was useful in Agile Process Management and
towards defining and measuring project metrics
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 3/8
Proteans Software Testing Approach
At Proteans, testing of the software is operationalised and automated even before the project begins unlike in the
traditional approach where you come to know of the glitches only close to the end of the project. Software Product
Testing is all about efficient management of Product Releases. To do this we listed down a series of questions and
tried to find answers:
In a suite of 18 products, with 12 existing versions in the market, how do we ascertain what is an existing defect
or a new defect?
How do you handle 400 + test scenarios?
How do you certify that a Product is good enough to be released with minimum testing?
How do you manage Test Suites over a Products Life Cycle?
In answer to the questions, we adopted the following steps:
A. The quintessential QA artifacts
1. The Product Test Suite
The Product Test Suite has Test Cases, and categorized as:
- Setups ( Installer)
- Vital functionality
- L10n and I18n
- User Interface Test Cases
- MSCRM version Specific
2. The Product Baselined Defect Sheet
All the existing defects in the product are categorized and logged into this sheet. This sheet acts as ready
reference to ascertain the existence of a defect. This also prevents duplicate defect logging and acts as beacon
pointing to existing defective functionality.
3. The Product Regression Testing Estimates
A log sheet records the time taken to execute Regression Test Case per product on a particular Deployment
Scenario and this forms basis for Test Execution Planning.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 4/8
B. The Story of the Integrated QA; user acceptance testing of story cards
The Integrated QA writes test cases for the defects/ stories / features (worked upon in the iteration) and
executes them. Post execution the results are updated into RALLY and the Story Card is sent for acceptance to
Product Management team. The Story Card is accepted based on test case results, screenshots provided, and by
user acceptance testing on the client test server. Largely, user acceptance testing of every story card has been a
keystone in achieving customer satisfaction.
A story card may or may not pass the test, for every minor defect detected. Each defect is logged into RALLY and
the Product Baseline Defect Sheet is updated accordingly.
At the end of the iteration, the Integrated QA will update the Product Test Suite with the new test cases. This is
the most important step in managing the Test Suites.
C. The story of the Regression QA; the regression testing process and defined exit criteria
The Regression QA tests all the products in multiple environments using the product test suite to verify
defects/story cards / features worked in the previous iterations. They execute the test cases from the Product
Test Suite in its entirety
On their assigned priority - the ‘vital’ test cases are executed first and then the other test cases
Follow Exit criteria - Release Candidate has to pass the ‘Vital’ test cases and test cases written for the
priority defects/ stories designated for the Release. In addition, the Release Candidate has to pass the Setup
test cases in all applicable scenarios
Post execution of the test cases, the team first compare the defects found with those on the Product Baseline
Defect Sheet. If the defect exists there, it is ignored, else it is logged into RALLY, and the Product Baseline
Defect Sheet is updated with defect ID.
At the end of Regression iteration, the updated Product Baseline Defect Sheet is shared with Product Management
This approach helps in optimum utilization of available time and work force, and decreases stress on the team,
leading to sustainable productivity in the end.
D. What happens to a defect? – The Defect Life Cycle
• A defect is identified in Regression Testing or from the Support Team
• A defect is always logged into RALLY
• A defect is always updated in the Product Baseline Defect sheet
• A defect has a priority and a severity attached to it. The Product Management team identifies the priority
and severity of the defect. Based on the above two parameters a defect is scheduled to be resolved in a
• Test cases are created; the defect is fixed and tested
• The Integrated QA marks it as fixed in RALLY and updates the defect status on the Product Baseline Defect
• The Regression QA verifies the defect in the regression iteration and marks it as ‘To be closed’
• The Product Management team / Support Team mark it as closed in RALLY
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 5/8
E. Test planning
Plan for Regression Testing is mostly context based. Factors like number of products to be released and their
complexity, number of setup scenarios, target markets involved, etc, are taken into consideration. For example,
priority of Non English language test environments to be included in the test plan depends on the products being
released in that region. The review and acceptance of Test plan by the product management team ensures clarity
on the testing goals.
F. Use of Virtualization Technology
The testing involved high level of complexity. The products, including setups, required to be tested on a multiple
environments. Hence, we created test environments that mimic customer environments. This includes
major/minor upgrades, l10n/i18n environments [French, German, Dutch, Japanese, etc], Outlook clients
[desktop, laptop], NLB, etc along with customizations as implemented by the customers.
Virtualization is extremely cost effective and versatile technology. By creating Virtual PC’s and Virtual Server, we
were able to reduce capital costs incurred in development with the added advantage of creating mixed
deployment scenarios with almost negligible incremental effort.
G. Cross product training/responsibilities
Each QA member is trained on all the products and there is absolute Product ownership. All QA members are
responsible for testing, documentation and preparation/maintenance of test environments. This ensures that non-
availability of resources at times does not hinder the scheduled testing plan.
H. Human factor: Team spirit
The whole project team – Developers and QA worked hand –in-hand. Unlike in most projects, QA is not
somebody’s “step-child". Developers and testers work in close co-ordination, which helped in minimizing conflict
and keeping at bay, the all too common ‘developer-tester divide’. The members of the team trusted each other
and look upon each other’s roles and responsibilities as complementary rather than adversarial. Effective
mentoring and knowledge transfer among team members are other key factors that contributed to our success.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 6/8
Benefits of Proteans Software Product Testing Approach
We managed the defect lifecycle efficiently. To simplify understanding of features, we used story cards and shared
the same with the Product Management team. The user acceptance of every story card resulted in enhancing our
testing effort and helps avoid waste in terms of time and resources for the documentation process.
Reduced capital costs
We used virtualization technology. By creating Virtual PC’s and Virtual Server we were able to reduce capital costs
incurred in development with the added advantage of creating mixed deployment scenarios with almost negligible
The classification and assigning priority to every identified defect played a vital role in achieving the testing goals.
Further, this approach helped in optimum utilization of available time and work force and decrease stress on the
team, leading to sustainable productivity in the end.
We ensured our QA team members are trained on all products and inculcated a sense of product ownership in each
of them. This ensured adhering to schedule inspite of any inadequacy in resources, if any. Thus, we were able to
manage the testing activities efficiently and hence, give a green signal for the product to go live despite the tight
deadlines for testing.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 7/8
Proteans is one of the leading outsourced Software Product Development Company and subsidiary of Norway based
CAMO ASA (www.camogroup.com ). We specialize in software product research and development services and help
ISVs, SaaS providers, and Software Development Organizations of global 1000 companies worldwide to bring
software products faster to market while reducing R&D costs. For more information, visit our website –
If you are considering improving your software product testing effort in agile environment, call us today and we
will be glad to offer you free assessment of your software testing and test automation needs.
Phone: +1-(410) 312 7625 Phone: +44 (1708) 440 319
Phone: +1-(410) 800-4679
Phone: +47- 223 963 00 Phone - +46 -707296481
Phone - +91 80 6618 6555
All registered trademarks acknowledged.
Copyright 2009 Proteans Software Solutions Pvt Ltd. All rights reserved. www.proteans.com 8/8