Combinatorial software test design beyond pairwise testing


Published on

Pairwise and combinatorial testing explained. Orthogonal array-based testing, pair-wise software testing, and other more thorough n-way combinatorial test design strategies are proven to be efficient and effective. Unfortunately, much of the material on the internet about these test design techniques are dense, impenetrable tomes filled with long Greek-letter-infused equations that only a mathematician could love. This presentation aims to explain the principles behind this powerful but under-appreciated software test design approach.

Published in: Technology
  • This is real take it serious, who will believe that a herb can cure herpes, i navel believe that this will work i have spend a lot when getting drugs from the hospital to keep me healthy, what i was waiting for is death because i was broke, one day i hard about this great man who is well know of HIV and cancer cure, i decided to email him, unknowingly to me that this will be the end of the herpes in my body, he prepare the herb for me, and give me instruction on how to take it, at the end of the one month, he told me to go to the hospital for a check up, and i went, surprisingly after the test the doctor confirm me negative, i thought it was a joke, i went to other hospital was also negative, then i took my friend who was also herpes positive to the Dr Agumagu, after the treatment she was also confirm negative . He also have the herb to cure cancer. please i want every one with this virus to be free, that is why am dropping his email address, or do email him he is a great man. the government is also interested in this DR, thank you for saving my life, and I promise I will always testify for your good his number +233200116937
    Are you sure you want to  Yes  No
    Your message goes here
  • 'After being in relationship with Wilson for seven years,he broke up with me, I did everything possible to bring him back but all was in vain, I wanted him back so much because of the love I have for him, I begged him with everything, I made promises but he refused. I explained my problem to someone online and she suggested that I should contact a spell caster that could help me cast a spell to bring him back but I am the type that don't believed in spell, I had no choice than to try it, I meant a spell caster called Dr Zuma zuk and I email him, and he told me there was no problem that everything will be okay before three days, that my ex will return to me before three days, he cast the spell and surprisingly in the second day, it was around 4pm. My ex called me, I was so surprised, I answered the call and all he said was that he was so sorry for everything that happened, that he wanted me to return to him, that he loves me so much. I was so happy and went to him, that was how we started living together happily again. Since then, I have made promise that anybody I know that have a relationship problem, I would be of help to such person by referring him or her to the only real and powerful spell caster who helped me with my own problem and who is different from all the fake ones out there. Anybody could need the help of the spell caster, his email: or call him +2349055637784 you can email him if you need his assistance in your relationship or anything. CONTACT HIM NOW FOR SOLUTION TO ALL YOUR PROBLEMS'
    Are you sure you want to  Yes  No
    Your message goes here
  • Thanks for the kind words, Claude!

    Please let us know if you have any questions as you design your tests. I'll ask someone from our Customer Success Team to activate the 'Live Test Design Support' feature in your account (in the lower left hand side of your screen after you login).
    Are you sure you want to  Yes  No
    Your message goes here
  • Just signed up for a trial of Hexawise and found this gem. Impeccable presentation! Fantastic supporting images for the 'plot'. Educational and entertaining. A++
    Are you sure you want to  Yes  No
    Your message goes here
  • @ssehlhorst Thank you Scott. I appreciate it. I believe it's an extremely important topic for software testers. I've tried to make it accessible without dumbing it down.
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Combinatorial software test design beyond pairwise testing

  1. 1. Combinatorial Software Test Design (Beyond Pairwise Testing) 1
  2. 2. Presented at STPCon October, 2010 By Justin Hunter CEO of Hexawise Cover slide photo credit: "Mike" Michael L. Baird, 2
  3. 3. L E M O B P R 3
  4. 4. The Problem You can’t test everything. 2
  5. 5. The Problem “With so many possible tests to choose from...” 2
  6. 6. The Problem “Things might seem uncontrollable...” 2
  7. 7. The Problem ... discouraging, even. 2
  8. 8. The Problem So, how can you select just a few tests... 8
  9. 9. The Problem ... and still achieve adequate coverage? 9
  10. 10. The Problem Wait. So this is a test design method to achieve greater coverage in fewer tests. Right? 10
  11. 11. Not Just Fewer, Better Tests If you prioritize But for those testers who speed, yes. You prioritize thoroughness, can achieve this method can also be greater coverage used to increase in fewer tests. thoroughness dramatically. It can help you prioritize around either goal. 11
  12. 12. Not Just Fewer, Better Tests W ay The choice is fewer yours to make. tes ts! etter ligh tly b ith s than we (W rage cove e now.) hav 12
  13. 13. Not Just Fewer, Better Tests How can you create “curiously strong” test cases that will allow you to achieve either of these testing objectives? 13
  14. 14. I O N L UT S O 14
  15. 15. The Solution This presentation describes an approach that will help you create powerful test cases. 15
  16. 16. Structured Variation To do so, use combinatorial test design methods that build structured variation into your test cases. 16
  17. 17. Structured Variation? “Test One” 17
  18. 18. Structured Variation? This is not what I mean by structured variation. “Test Two” 18
  19. 19. Structured Variation? This is not what I mean by structured variation. “Test Three” 19
  20. 20. Structured Variation? So if that’s not structured variation, what is? 20
  21. 21. Structured Variation Structured variation means 21
  22. 22. Structured Variation ...each test will be different 22
  23. 23. Structured Variation ...than the tests before it, and 23
  24. 24. Structured Variation ...testers will do their testing in different ways, 24
  25. 25. Structured Variation ... and explore from more angles. 25
  26. 26. Structured Variation “OK. Maximum variation is better than redundantly repeated redundant repetition. Got it. That it? Am I good to go now?” 26
  27. 27. Coverage No. There’s more. Much more. Remember coverage? 27
  28. 28. Coverage Accidentally leaving gaps in coverage can lead to painful results. 28
  29. 29. Coverage Structured variation focuses on covering as much as possible... 29
  30. 30. Coverage ... with as few resources as possible... 30
  31. 31. Coverage ...and in the most efficiently structured way possible. 31
  32. 32. F O & S U N C IO O T F U L E L ONA O I S T R A 32
  33. 33. Single-Mode Faults Most software defects in production today can be triggered by just one test input. 33
  34. 34. Dual-Mode Faults Software bugs caused by the interaction of two test inputs are very common too. 34
  35. 35. Triple-Mode Faults Defects that require three specific test inputs to trigger them are rare. 35
  36. 36. Quadruple-Mode Faults Defects that require four more test inputs to trigger them are extremely rare. 36
  37. 37. Implication on Testing Strategy Aha! So, to find bugs efficiently, we should focus first on testing all possible combinations every set of TWO test inputs! 37
  38. 38. Implication on Testing Strategy That’s right, two. Focus on covering all possible two-way defects first. In fact, that’s so important, can you say it a second time? 38
  39. 39. Implication on Testing Strategy To find bugs efficiently we should focus first on testing all possible combinations of TWO test inputs! 39
  40. 40. I C S AN E C H M 40
  41. 41. Combinations of Test Inputs The key words here are test inputs and combinations. 41
  42. 42. Combinations of Test Inputs Test inputs will vary dramatically from project to project. 42
  43. 43. Combinatorial Software Test Design Well, duh! They would, wouldn’t they? But how does it all work? 43
  44. 44. Combinatorial Software Test Design Here’s how it works: First, people are required to identify what test inputs should be tested. 44
  45. 45. Combinatorial Software Test Design Second, computer algorithms will take those inputs and quickly calculate the smallest possible number of tests required to meet the tester’s specified coverage objectives. 45
  46. 46. Combinatorial Software Test Design ... and again, the algorithms will W ay create different solutions fewer depending upon tes ts! your testing objectives. etter ligh tly b ith s than we (W rage cove e now.) hav 46
  47. 47. Combinatorial Software Test Design W ay The woman wanting fewer fewer tests should tes ts! create 2-way tests r bette htly Wi ig th sl n we ha ve (AKA ( ha age t .) c over now “pairwise” or “AllPairs” tests) 47
  48. 48. Combinatorial Software Test Design This guy, wanting more coverage, should create more thorough combinatorial tests. (e.g., 3-way, 4-way, or multi-strength tests). 48
  49. 49. OT Y N STS H W NT ? E S G ND I A E D YH B 49
  50. 50. Advantages of Combinatorial Software Test Design Test Design: Man vs. Machine (Once test inputs are known) Speed: Coverage: Efficiency: Human Weeks Gaps Way More Tests Computer Seconds 100%* Way Fewer Tests *100% of what? 100% of the desired test input combinations (e.g., all pairs of values for 2-way tests solutions, all triplets of possible values for 3-way solutions, etc.).
  51. 51. Coverage Advantages Whatever your test inputs, combinatorial testing makes it easy and fast to cover all of the specific combinations of test inputs that you suspect are relatively important. 51
  52. 52. Advantages of Combinatorial Software Test Design OK, but wouldn’t I have tested those relatively important combinations on my own?
  53. 53. Advantages of Combinatorial Software Test Design Perhaps, but maybe you... Forget Run out of time Use more tests Accidentally than needed repeat yourself 53
  54. 54. Coverage Advantages Plus, using combinatorial testing, you’ll also get additional coverage from combinations that wouldn’t have made your list, including... 54
  55. 55. Coverage Advantages ...specialized use cases, No, its way cooler than pareto. 55
  56. 56. Coverage Advantages ... unexpected combinations, 56
  57. 57. Coverage Advantages ...notable, combinations that capture your attention, 57
  58. 58. Coverage Advantages ...even combinations that rarely, if ever, actually occur, 58
  59. 59. Coverage Advantages ... and... combinations that will take you into different parts of the application (and different states) than you would have otherwise thought to explore. 59
  60. 60. Coverage Advantages Quickly testing unusual pairs is often surprisingly efficient and effective. Many unexpected two-way interactions cause problems. 60
  61. 61. Constraint Handling Combinations that can’t ever appear in reality? Yes. No problem. You can prevent them from appearing in your plans with “Invalid Pairs.” 61
  62. 62. E T ? Y Y E AD R 62
  63. 63. Ready? I’m ready already! Let’s do this! 63
  64. 64. Hazards Not so fast. Hazards exist. To succeed, you must use common sense. 64
  65. 65. Limitations And let’s not kid ourselves. Not just anyone can do this stuff. 65
  66. 66. Prerequisites It also requires good critical thinking skills. 66
  67. 67. Prerequisites To thrive, you must be able to develop models about how the systems you’re exploring 67 work.
  68. 68. Practice Makes Perfect Training and practice will help you succeed.
  69. 69. Questions ? 69
  70. 70. Ready? Let’s get started! 70
  71. 71. D I X P E N AP 71
  72. 72. Thank you Special thanks to the folks at: and its talented contributing photographers, including: Mike Baird ... and the inimitable John Carleton (who, let the record show, had no input into the cartoonish statements we added next to his striking expressions). 72
  73. 73. Why 2-way Testing is Effective Multiple thorough studies show approximately 85% of defects in production could have been detected by simply testing all possible pairs of values. Text Source: “Combinatorial Testing” IEEE Computer, Aug, 2009. Rick Kuhn, Raghu Kacker, Jeff Lei, Justin Hunter
  74. 74. Additional Information Excellent introductory articles and instructional videos: Results of a ten-project study of effectiveness: Free trials of our firm’s combinatorial test design tool: (which stay free indefinitely for companies using 1-4 licenses). Also, please feel free to contact me if you have any questions. I’d be happy to quickly review a test plan or two, answer your questions, give quick pointers to help you run a pilot, etc. Seriously.... I enjoy helping people get started with this test design approach. Please don’t hesitate to reach out. There’s no charge and no catch. 74
  75. 75. Practice Tips: 4-Step Process Four-Step Process to Design Efficient and Effective Tests Scope Level  of  Detail Passive  Field  Treatment 1.  Plan Be  clear  about  single  or  mul3ple: Acceptable  op3ons:   Dis3nguish  between  important  fields  (par3cularly  those  that   will  trigger  business  rules)  and  unimportant  fields  in  the   -­‐    Features  /  Func3ons  /  Capabili3es -­‐    High  level  “search  for  something” applica3on. -­‐    User  types -­‐    Medium  level  “search  for  a  book” Quickly  document  what  your  approach  will  be  towards   -­‐    Business  Units -­‐    Detailed  “search  for  ‘Catcher  in  the  Rye’   passive  fields.  You  might  consider:  ignore  them  (e.g.,  don’t   -­‐    H/W  or  S/W  Configura3ons            by  its  3tle” select  any  Values  in  your  plan)  or  a  3  Value  approach  such  as   “valid”  “Invalid  (then  fix)”  and  “Blank  (then  fix)” Configura3ons Users Ac3ons 2.  Create First  add  hardware  configura3ons Next,  add  mul3ple  types  of  users  (e.g.,   Start  with  Big  Common  Ac3ons  made  by  users administrator,  customer,  special  customer) AZer  comple3ng  Big  Common  Ac3ons,  circle  back  and  add   Next  add  soZware  configura3ons Consider  permission  /  authority  levels  of  admin   Small  Ac3ons  and  Excep3ons users  as  well  as  business  rules  that  different   Remember  some  ac3ons  may  be  system-­‐generated users  might  trigger Business  Rules Gap  Filling Itera3on 3.  Refine Select  Values  to  trigger  bus.  rules Iden3fy  gaps  by  analyzing  different  states,   Refine  longest  lists  of  Values;  reduce  their  numbers  by  using   equivalence  classes,  etc. orders  of  opera3ons,  decision-­‐tree  outcomes   Iden3fy  equivalence  classes sought  vs.  delivered  by  tests,  “gap  hun3ng”   Create  Tests  with  and  w/out  borderline  Values;  consider  cost/ conversa3ons  w/  SME’s,  etc. benefit  tradeoffs  of  addi3onal  test  design  refinements Test  for  boundary  values Fill  gaps  by  either  (i)  adding  Parameters  and/or   Consider  stopping  tes3ng  aZer  reaching  ~80%  coverage   Mark  constraints  /  invalid  pairs Values  or  (ii)  crea3ng  “one-­‐off”  tests.   Consider  2-­‐way,  3-­‐way,  and  Mixed-­‐Strength  op3ons Auto-­‐Scrip3ng Expected  Results Con3nuous  Improvement 4.  Execute       Add  auto-­‐scrip3ng  instruc3ons  once;   Export  the  tests  into  Excel  when  you’re  done   If  possible  measure  defects  found  per  tester  hour  “with  and   without  Hexawise”  and  share  the  results apply  those  instruc3ons  to  all  of  the  tests   itera3ng  the  plan in  your  plan  instantly Add  inputs  based  on  undetected  defects Add  complex  Expected  Results  in  Excel  post-­‐ Don’t  include  complex  Expected  Results   export Share  good,  proven,  plan  templates  with  others in  auto-­‐scripts   © Hexawise, 2010. All rights reserved. 26
  76. 76. Practice Tips: Warning Signs “You might be headed for trouble if...” Scope Level  of  Detail Passive  Field  Treatment 1.  Plan ...  You  cannot  clearly  describe  both  the   ...  Parameters  with  the  most  Values  have  more   ...  You  cannot  clearly  describe  your  strategy  to  deal  with   scope  of  your  test  plan  and  what  will  be   Values  than  they  require.    365  values  for  “days   unimportant  details.    If  Values  will  impact  business  rules,  focus   leZ  out  of  scope. of  the  year”  is  bad.    Instead,  use  equivalence   on  them.    If  Values  don’t  impact  business  rules,  consider   class  Values  like  “weekend”  &  “weekday.”    When   ignoring  them.   in  doubt,  choose  more  Parameters  and  fewer   Values. Configura3ons Users Ac3ons 2.  Create ...  You  have  ignored  hardware  and   ...  You  have  not  included  all  the  different  types   ...  You  start  entering  Small  Ac3ons  (e.g.,  “search  for  a   soZware  configura3ons    without  first   of  users  necessary  to  trigger  different  business   hardback  science  book  by  author  name”)  before  you  enter  Big   confirming  this  approach  with   rules.    What  user  types  might  create  different   Ac3ons  (e.g.,  “Put  Something  in  Cart.    Buy  it.”)    First  go  from   stakeholders. outcomes?    Authority  level?  Age?  Loca3on?     beginning  to  end  at  a  high  level.    AZer  you’ve  done  that,  feel   Income?    Customer  status? free  to  add  more  details.   Business  Rules Gap  Filling Itera3on 3.  Refine ...  You  forget  to  iden3fy  invalid  pairs.          -­‐   ...  You  assume  that  the  test  condi3ons  coming   ...  You  forget  to  look  at  the  Coverage  Analysis  charts.    If  you   or  -­‐    ...  You  rely  only  on  Func3onal   out  of  Hexawise  will  be  100%  of  the  tests  you   achieve  80%  coverage  in  the  first  quarter  of  the  tests,  you   Requirements  and  Tech  Specs  w/out   should  run.    There  might  well  be  addi3onal   should  measure  the  cost/benefit  implica3ons  of  execu3ng  the   thinking  hard  yourself  and  asking   “one-­‐off”  things  that  you  should  test  and/or  a   last  3/4  of  the  tests. ques3ons  to  SME’s  about  business  rules   few  nega3ve  tests  to  design  by  hand. and  outcomes  that  are  not  yet  triggered. Auto-­‐Scrip3ng Expected  Results Con3nuous  Improvement 4.  Execute ...  You  add  detailed  Expected  Results  in   ...  You  invest  a  lot  of  3me  in  calcula3ng  and   ...  You  don’t  ask  (when  defects  that  the  tests  missed  are  found   the  tests.                                                                                                          -­‐   documen3ng  Expected  Results  before  you  have   post-­‐tes3ng)  “What  input  could  have  been  added  to  the  test   or  -­‐  ...  You  forget  that  this  feature  exists   plan  to  detected  this?”  “Should  I  add  that  input  to  the   and  find  yourself  typing  out  test-­‐by-­‐test   determined  your  “final  version”  Parameters  and   instruc3ons  one-­‐by-­‐one. Values.    Last  minute  addi3ons  to  inputs  will   Hexawise  test  plan  now  to  improve  it  in  advance  of  the  next   jumble  up  test  condi3ons  for  most  test  cases. 3me  it  is  used?”   © Hexawise, 2010. All rights reserved. 27