Problem Statement:One of the common concerns from the customers is that how to effectively optimize the testing given the multiple integration points in a distributed/composite system environments, which does expose at least the below pain points: 1. Avoid Exhausted testing 2. Meet all the boundary conditions 3. Limited time to execute 100% test execution 4. Include all the critical business functions 5. Efficient Regression Testing and the list goes on... Resolution: The solution is detailed in the attachment and have effectively implemented in various client places.

1. 1
Orthogonal Array approach
in Testing
Presenter – Karthikeyan Rajendran
Enterprise Testing Services

2. Agenda-
 Introduction-Current scenario & Customer
Expectations
 Normal approach in testing
 Testing – The only validation Method
 Challenges while designing test cases
 Orthogonal Array Testing – Basics &
Terminologies
 Implementing OA techniques
 Case Study I – Mobile application
 Case Study II – GradeGuru application
 Benefits of OA
 Limitations of OA
 Mistakes to avoid while implementing OA
 Conclusion
2

3. Introduction: Current Scenario
Tight Budget
Schedule Pressure
Demanding Customer
3

4. Zero Defect
Lower Execution Cost
Faster Execution Efficiency
Customer Satisfaction
Introduction: Expectations
4

5. Normal Approach in Testing
Study ‘Requirement
Specification’.
Identify Test
Parameter & their
Levels
Remove Redundancy
& Invalid Test Cases
Design Test CaseExecute Test CaseLog Errors
This is repeated for each Test Case
More the Test Cases: More Time & Effort
5

6. Testing: The only Validation Method!
Whether the product is the “Right Product”!
The challenge in identifying Test Cases:
 Are all the paths covered?
(Unit Testing or white box testing)
 Are all interfaces tested?
(Integration Testing)
 Is the system functionality validated?
(System Testing)
 Are all single mode and double mode faults detected?
6

7. Challenges while Designing Test Cases
Endless executing tests don’t increase
Confidence Level in the System
May not have time to execute 100%
combinations of Test Cases
All boundary conditions may
not be tested.
Repeated test cycles incase
of Regression may not be
feasible.
7

8. Orthogonal Array Testing:
An Efficient and Effective Test Strategy
Covers all the Boundary Conditions
Optimize the test cases that are likely to
uncover most of the bugs
Increase the confidence level in the system by
executing concise and well defined sets of tests.
Orthogonal Array is the tool used to generate the Test Cases
8

9. OA Terminologies-
OA is a 2-Dimensional Array consisting of:
 Runs:
 number of rows in the array or
 number of test cases that will be generated by the OA technique.
 Factors:
 number of columns in an array or
 the parameter/ variables that need to be tested in the system.
 Levels:
 the maximum number of values that can be taken on by any single
factor.
 the values in orthogonal array
 Orthogonal arrays are represented by:
L Runs
(Levels Factors
)
9

10. Implementing OA Techniques-
1. Decide the independent variables will be tested for interaction. This
will map to the Factors of the array.
2. Decide the maximum number of values that each independent
variable/ factor will take on. This will map to the Levels of the array.
3. Find a suitable orthogonal array with the smallest number of Runs.
4. Calculate Degrees of freedom (DOF) using formula:
DOF = 1+ factors (level – 1)
The number of experimental runs (test cases) should be >= DOF.
10

11. Case Study I-
Consider a mobile application -
 Three call types (Free, National, International)
 Three call methods (Phone Book, Voice Activated, Quick Dial)
 Two account types (Postpaid, Prepaid)
11

12. All possible combinations-
Call Types Call Methods Account Types
Free PB Prepaid
Free PB Postpaid
Free VA Prepaid
Free VA Postpaid
Free QD Prepaid
Free QD Postpaid
National PB Prepaid
National PB Postpaid
National VA Prepaid
National VA Postpaid
National QD Prepaid
National QD Postpaid
International PB Prepaid
International PB Postpaid
International VA Prepaid
International VA Postpaid
International QD Prepaid
International QD Postpaid
12

13. Applying OA-
Call Types Call Methods Account Types
Free PB Postpaid
Free VA Postpaid
Free QD Prepaid
National PB Postpaid
National VA Prepaid
National QD Postpaid
International PB Prepaid
International VA Prepaid
International QD Postpaid
13

14. Case Study I contd…
 Test cases count before applying OA =18
 Test Cases count after applying OA=9
 Reduction percentage = 50%
14

15. Case Study II-
HPI GradeGuru application:
 Notes – 17 formats
 Users – 3 types
 Operation – 4 types
Total possible combinations – 204
Applying OA it is reduced to 85.
15

16. Case Study II contd…
 The Challenge:
 Optimize the number of Test Cases.
 Reduce the Test Case Execution time.
 Cover 100% Functionality.
 The Solution:
 Orthogonal Array Test Strategy to generate Test Cases.
 The Benefits:
 Execution Time is reduced.
 Result Analysis time is reduced.
 Code Coverage is increased.
 Increase in Productivity.
16

17. Benefits of OA-
 Helps in productivity improvement with cycle time
reduction.
 Helps in improving test coverage.
 Orthogonal Array test cases can be customized
based on available time and known problems.
 Independent of platforms and domains.
17

18. Limitations of OA-
Conclusion
 To reveal problems by OA’s the error probability and
the dimension of the OA need to be in balance;
 Rare failures require many tests (low L-index),
 Frequent failures will almost always be revealed by
OA’s.
Challenge
 To know the levels and critical combinations in
advance
 To know the probability of errors in advance.
18

19. Mistakes to avoid while implementing-
 Applying OATS manually
 Focusing the testing effort on the wrong area of the
application
 Using OATS for minimal testing efforts
 Picking the wrong parameters to combine
19

20. Conclusion-
 The Orthogonal Array Testing is a systematic, statistical way of testing
the software.
 Creates an efficient and concise test set with many fewer test cases.
 A uniform distributed coverage of all variable pair combinations to be
tested.
 Highly effective for the detection of region faults with a relatively small
number of tests.
 Is simpler to generate and less error prone than test sets created by
hand.
20

21. References…
1. Bret Pettichord, “A Unified Theory of Software Testing”,
Feb 2003.
< http://testingeducation.org/conference/ >
2. Perry, William.E. “Effective Methods for Software Testing”
Wiley Press, 2nd
Edition, Oct 1999.
3. Jeremy M. Harrell, Quality Assurance Manager, Seilevel,
Inc., Orthogonal Array Testing Strategy (OATS) Technique.
http://www.cvc.uab.es/shared/teach/a21291/apunts/Prova%20orientada%20o
4. Phadke, M.S. "Quality Engineering Using Robust Design"
Prentice Hall, Englewood Cliff, NJ. November 1989.
5. Sloane, Neil J. A. “A Library of Orthogonal Arrays”.
Information Sciences Research Center, AT&T Shannon
Labs. Aug 2001 <http://www.research.att.com/~njas/oadir/
>
21

22. THANK YOU
22