This is the abstract published in the IEEE S/W Engineering Conference proceedings in the industry experience sharing section of the Intl Workshop on Combinatorial Testing. We share the experience and serendipitous findings of how CT can be used for early requirements validation.
2. The hybrid approach enabled coverage prioritization, i.e.
higher level of coverage on the attributes which are critical to
testing and lower coverage for those which are not so critical
yet important to be covered in the tests. The manually
designed tests and the CTD models were shared with the client
for a collaborative review and final sign off.
III. FINDINGS FROM INTERACTIVE REVIEW SESSIONS
In the CTD review session, we observed that the clients and
business analysts were able to understand the CTD model
construct with more ease than the manual test cases. During
the very first functional module review, where we reviewed
two CTD models covering 122 test cases and 61 manual test
cases; the clients were able to provide insights on the attributes
and interactions. They also shared inputs on the levels of
interactions they would require. We were able to modify the
model, add the attributes and change the interaction levels and
share the coverage impact.
The first model originally had six attributes and had 16
restrictions (business rules). It resulted in 49 good path tests
and one bad path test. Based on the client review – we added
two more attributes to the original six that were understood
from the requirements and system design specifications. This
was done real-time in the tool during the review and the test
combinations were regenerated instantaneously in the CTD
tool for client approval and sign off. This greatly influenced
their confidence in the test solution and coverage.
The client review of the single-click verification steps for
which we had adopted a parallel manual design process was
not so smooth. Interestingly, the client was not able to easily
visualize the steps and the test actions and found it difficult to
distinguish between precursor steps and final validations. They
wanted test cases to be decoupled / broken down to provide
more clarity. The manual tests also tended to follow screen
actions instead of functionality. The tests also required
elaboration increasing the number of test cases required to
provide the structure, clarity and coverage required. Even
though the content was present, it required rewriting as the
flow was not evident.
We observer that the CTD model on the other hand could
clearly articulate at a glance the approach taken to testing the
products / brands irrespective of how the data and responses
where provisioned and irrespective of the technical landscape.
Clients were able to follow the flow of the tests as a business
process viewed as a customer’s record journey.
The business analysts / clients could quickly
• Comprehend the combinations from a business
standpoint
• Relate the CTD model output to the requirements and
business behavior
• Directly add value in terms of commonly expected
actions from end-customers
• Provide review inputs and recommend changes in an
online review mode
• Establish areas of increased test weightage to skew
the tests towards functionality that would typically be
used by end customers.
Overall, using CTD resulted in getting rapid consensus on the
features and functionality. It helped in rapidly building client
confidence in the test content and approach. The necessary test
coverage was established and the sign off on the Test
Readiness Review was obtained in time for the SIT schedule.
IV. CONCLUSION
Using a tool based approach for test design leveraging
Combinatorial Testing ensured active participation and
collaboration between the client, business analysts and testers
to build the right tests. Functional gaps in the requirements are
naturally highlighted without referencing back to inadequate
requirements, open questions and uncertain application
behavior, interface specifications or related queries that were
being unanswered in the past. Thus, helped to create a strong
interactive and collaborative environment.
Given the rapidity with which functional queries and
clarifications were addressed, it not only becomes obvious that
CT models should be the default review and Test Readiness
signoff criterion. More importantly, CT can be used as an
early shift left lever to not just clarify requirements but to also
define requirements and functional actions upfront. Based on
this experience, we expect that CT can also become a powerful
tool for driving Agile Development. CT tools can be used to
craft and elaborate user stories, establish requirements
interaction and complexity ranking. CT will enable business to
achieve early quality and speed to market by enabling for Test
Driven Business Definition and Requirements Elaboration.
ACKNOWLEDGMENT
We thank the team, the managers and the account leaders of
the project experience shared – Shaji K Namath and Sanjiv
Gupta. Special thanks to Reenav Gala and Dalibor Stavek,
business analysts from the IBM team who helped with the
model design and owned explaining the functionality to the
clients. Roopa Vasan’s planning and her team for developing
the CTD models. Our deepest gratitude to Phil Stead for
driving this agenda with the client and for being that bridge to
make it succeed. He defined and set the context upfront.
Thanks to Andrew Williams of the IBM IGNITE Leadership
team for being encouraging as always! Finally, without
Amitava Sharma, who gave us the great freedom to explore and
do things differently, we would not have experienced this great
step change in how CT modelling can be used.
REFERENCES
[1] Krishnan, R and Krishna, S Murali and Nandhan, P Siva, “Combinatorial
testing: learnings from our experience,” ACM SIGSOFT Software
Engineering Notes, vol. 32, no. 3, pp. 100-108, 2007