This is the abstract published in the IEEE S/W Engineering Conference proceedings in the industry experience sharing section of the Intl Workshop on Combinatorial Testing. We share the experience and serendipitous findings of how CT can be used for early requirements validation.

1. XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Combinatorial Test Design – A smarter way to
connect with the business
Saritha Route
IBM Business Services
IBM India Pvt. Ltd.
Bangalore, India
saritha.route@in.ibm.com
Sudheer Pendela
IBM Business Services
IBM India Pvt. Ltd.
Bangalore, India
sudheerpendela@in.ibm.com
Abstract— A formal quality gate called the Test Readiness
Review (TRR) is a typical pre-requisite for commencing System
Integration Testing. For a large program that is currently
underway, we were in the test design phase. There were
continued delays in the Test Readiness sign off from the client.
The main reason was the 300 plus open questions related to
requirements clarity that was impacting the test preparation
completion. To address this impasse, we introduced
Combinatorial Test (CT). The aim was to provide coverage
assurance and uncover gaps in tests and requirements using CT.
The interactive CT design process would ensure that open
questions, undocumented requirements and system interactions
are highlighted using scientific methods without bias. Thus,
aiding speedy closure of gaps in both requirements and test cases.
We used the IBM IGNITE Method for deploying Combinatorial
Test Design, which engages stakeholders and implements CT
techniques with the Focus tool.
Keywords—CTD Implementation, IBM Focus, IGNITE, Test
coverage, Test optimization, Test Design, CTD for agile, CTD for
requirements clarification, Test sign-off
I. INTRODUCTION
A Typical Test Readiness Review determines, amongst
other aspects, whether the test cases and scenarios are ready,
well-documented, tagged to requirements and provide
quantified test coverage across the requirements. The test cases
are reviewed and signed off by the business users and client
representatives for completeness and coverage against the
scope and requirements. The test case sign-off is a long-drawn
out process and is conducted in an examination-style, with the
clients marking correct and incorrect tests and highlighting
coverage aspects against a matrix.
The estimated size of the SIT for a program underway is
over 3500 test cases encompassing nine functional modules.
Due to the delays in freezing of the design, the test schedule
was in danger of being greatly extended due to late
identification of critical defects, higher dev-fix effort and lack
of early visibility on software quality. Given the criticality of
the program, the client was concerned about the coverage of the
test cases as well as meeting a non-negotiable go-live date.
Over 300 questions were still open due to inadequate
requirements documentation and design changes. The existing
business processes were also being redefined even as the
functionality was being coded. The project was envisaged as a
platform change rather than a business change and therefore
documentation had more of a technical focus as opposed to
elaborating the functionality. A good part of the business
requirements was inferred based on current usage and were not
fully documented. So overall the test cases were developed
from system design documents and not from defined functional
requirements specifications.
The functionality was primarily about testing business rules
covering four product brands across different market segments
in the retail selling space. The functional interactions spanned
different core systems that either provided customer data, credit
scores, and interfaces for pulling customer financials, eligibility
checks and so forth. The interfaces were UI based, API based
and data transfers that needed to happen seamlessly for the
business flow to be completed.
II. APPROACH: ESTABLISHING CT MODELS AS THE SIGN OFF
CRITEREON FOR TEST DESIGN
We used Combinatorial Test Design (CTD) to establish a
channel with the right stakeholders upfront. We created CTD
models and worked with application SMEs to detail variables
and interactions that were not documented in the system design
document. The draft models were then reviewed with the
business analysts and clients to get concurrence on the
understanding of the functionality. Questions on points of
variation and interactions were addressed in the online reviews
of the CTD models. Instead of sharing test cases in excel
sheets, we opened the CT tool and shared the representation of
attributes, values and restrictions governing these values. Then
built the tests based on the interactions desired in front of the
client. We used the IBM Focus tool, which is an integral part
of the IGNITE Quality Platform solution from IBM.
We adopted a hybrid approach to reuse the tests already
constructed manually using the following thumb rule:
• CT based design for functionality that had many points
of variation as well as a high number of interactions
• Direct manual test design approach for functionality that
was performed as single click operations or tested as
plain vanilla verification steps
This work has been sponsored and financially support by IBM Services.
The author Saritha Route, is the Global Program Leader for Test
Automation and Sudheer Pendela is a certified CTD Coach.
IBM Functional Coverage Unified Solution (IBM FOCUS) is an advanced test
planning tool. IBM FOCUS uses Combinatorial Test Design (CTD) to generate an
efficient test plan that provides consistent coverage across the test space at a known
depth, while significantly reducing the required resources. IBM focus is accessible
from the IGNITE Quality Platform IGNITE CTD seamlessly extends to test
automation leveraging open source solutions on the IGNITE Quality Platform
with IGNITE Optimized Test Flow Automation.

2. The hybrid approach enabled coverage prioritization, i.e.
higher level of coverage on the attributes which are critical to
testing and lower coverage for those which are not so critical
yet important to be covered in the tests. The manually
designed tests and the CTD models were shared with the client
for a collaborative review and final sign off.
III. FINDINGS FROM INTERACTIVE REVIEW SESSIONS
In the CTD review session, we observed that the clients and
business analysts were able to understand the CTD model
construct with more ease than the manual test cases. During
the very first functional module review, where we reviewed
two CTD models covering 122 test cases and 61 manual test
cases; the clients were able to provide insights on the attributes
and interactions. They also shared inputs on the levels of
interactions they would require. We were able to modify the
model, add the attributes and change the interaction levels and
share the coverage impact.
The first model originally had six attributes and had 16
restrictions (business rules). It resulted in 49 good path tests
and one bad path test. Based on the client review – we added
two more attributes to the original six that were understood
from the requirements and system design specifications. This
was done real-time in the tool during the review and the test
combinations were regenerated instantaneously in the CTD
tool for client approval and sign off. This greatly influenced
their confidence in the test solution and coverage.
The client review of the single-click verification steps for
which we had adopted a parallel manual design process was
not so smooth. Interestingly, the client was not able to easily
visualize the steps and the test actions and found it difficult to
distinguish between precursor steps and final validations. They
wanted test cases to be decoupled / broken down to provide
more clarity. The manual tests also tended to follow screen
actions instead of functionality. The tests also required
elaboration increasing the number of test cases required to
provide the structure, clarity and coverage required. Even
though the content was present, it required rewriting as the
flow was not evident.
We observer that the CTD model on the other hand could
clearly articulate at a glance the approach taken to testing the
products / brands irrespective of how the data and responses
where provisioned and irrespective of the technical landscape.
Clients were able to follow the flow of the tests as a business
process viewed as a customer’s record journey.
The business analysts / clients could quickly
• Comprehend the combinations from a business
standpoint
• Relate the CTD model output to the requirements and
business behavior
• Directly add value in terms of commonly expected
actions from end-customers
• Provide review inputs and recommend changes in an
online review mode
• Establish areas of increased test weightage to skew
the tests towards functionality that would typically be
used by end customers.
Overall, using CTD resulted in getting rapid consensus on the
features and functionality. It helped in rapidly building client
confidence in the test content and approach. The necessary test
coverage was established and the sign off on the Test
Readiness Review was obtained in time for the SIT schedule.
IV. CONCLUSION
Using a tool based approach for test design leveraging
Combinatorial Testing ensured active participation and
collaboration between the client, business analysts and testers
to build the right tests. Functional gaps in the requirements are
naturally highlighted without referencing back to inadequate
requirements, open questions and uncertain application
behavior, interface specifications or related queries that were
being unanswered in the past. Thus, helped to create a strong
interactive and collaborative environment.
Given the rapidity with which functional queries and
clarifications were addressed, it not only becomes obvious that
CT models should be the default review and Test Readiness
signoff criterion. More importantly, CT can be used as an
early shift left lever to not just clarify requirements but to also
define requirements and functional actions upfront. Based on
this experience, we expect that CT can also become a powerful
tool for driving Agile Development. CT tools can be used to
craft and elaborate user stories, establish requirements
interaction and complexity ranking. CT will enable business to
achieve early quality and speed to market by enabling for Test
Driven Business Definition and Requirements Elaboration.
ACKNOWLEDGMENT
We thank the team, the managers and the account leaders of
the project experience shared – Shaji K Namath and Sanjiv
Gupta. Special thanks to Reenav Gala and Dalibor Stavek,
business analysts from the IBM team who helped with the
model design and owned explaining the functionality to the
clients. Roopa Vasan’s planning and her team for developing
the CTD models. Our deepest gratitude to Phil Stead for
driving this agenda with the client and for being that bridge to
make it succeed. He defined and set the context upfront.
Thanks to Andrew Williams of the IBM IGNITE Leadership
team for being encouraging as always! Finally, without
Amitava Sharma, who gave us the great freedom to explore and
do things differently, we would not have experienced this great
step change in how CT modelling can be used.
REFERENCES
[1] Krishnan, R and Krishna, S Murali and Nandhan, P Siva, “Combinatorial
testing: learnings from our experience,” ACM SIGSOFT Software
Engineering Notes, vol. 32, no. 3, pp. 100-108, 2007